2024-04-21T23:00:42.028 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-21T23:00:42.028 DEBUG:teuthology.run:Teuthology command: teuthology --name teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/fsx}} --owner scheduled_teuthology@teuthology --verbose --archive /home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667597 -- /home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667597/orig.config.yaml 2024-04-21T23:00:42.090 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-21T23:00:42.182 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667597 branch: squid description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/fsx}} email: ceph-qa@ceph.com first_in_suite: false job_id: '7667597' kernel: client: sha1: distro kdb: 1 sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true name: teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_export_ephemeral_random_max: 0.1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - slow metadata IO - SLOW_OPS - slow request sha1: 820a2f8ced66152ed3d765c31444b8076ea28c6b subvols: create: 2 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount - mds.dir_split install: ceph: flavor: default sha1: 820a2f8ced66152ed3d765c31444b8076ea28c6b kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=crc - nowsync syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: 430e09df97c8fc7dc2b2ae424f68ed11366c540f owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 676 sha1: 820a2f8ced66152ed3d765c31444b8076ea28c6b sleep_before_teardown: 0 subset: 46/512 suite: fs suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_430e09df97c8fc7dc2b2ae424f68ed11366c540f/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 430e09df97c8fc7dc2b2ae424f68ed11366c540f syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi040.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNT+hzD/V5Q4zEFRYEAvJrUNAYM0ssmmtHkEPvHajAavMOIliQ1IVX1WGg1jcHBZZTXml+Le35lkmhPf97rHsS0= smithi111.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEqzOEpC4iF6MYQz/A+OjWQlHSxszVS/H13hLtt9oQaTUBzQRs9yeoDkxt5lbrSeDflEC6Haneqcu+tNpxds2D0= smithi174.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDr2h/lwQ+5x1MozouN3F0vf5TFKKT9TfI0KrG+lrE4mK3z0Uoia+My2pbUQR3QkptY/XHF4CVxNVHlrmQGcA40= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs qa random 0.10 - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - suites/fsx.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-21_21:24:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-21T23:00:42.183 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_430e09df97c8fc7dc2b2ae424f68ed11366c540f/qa; will attempt to use it 2024-04-21T23:00:42.183 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_430e09df97c8fc7dc2b2ae424f68ed11366c540f/qa/tasks 2024-04-21T23:00:42.184 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-21T23:00:42.185 INFO:teuthology.task.internal:Checking packages... 2024-04-21T23:00:42.208 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '820a2f8ced66152ed3d765c31444b8076ea28c6b' 2024-04-21T23:00:42.208 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-21T23:00:42.209 INFO:teuthology.packaging:ref: None 2024-04-21T23:00:42.209 INFO:teuthology.packaging:tag: None 2024-04-21T23:00:42.209 INFO:teuthology.packaging:branch: squid 2024-04-21T23:00:42.209 INFO:teuthology.packaging:sha1: 820a2f8ced66152ed3d765c31444b8076ea28c6b 2024-04-21T23:00:42.209 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-04-21T23:00:42.526 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2407.g430e09df 2024-04-21T23:00:42.528 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-21T23:00:42.535 INFO:teuthology.task.internal:no buildpackages task found 2024-04-21T23:00:42.535 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-21T23:00:42.570 INFO:teuthology.task.internal:Saving configuration 2024-04-21T23:00:42.593 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-21T23:00:42.614 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-21T23:00:42.639 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi040.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667597', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-21 22:55:06.386214', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNT+hzD/V5Q4zEFRYEAvJrUNAYM0ssmmtHkEPvHajAavMOIliQ1IVX1WGg1jcHBZZTXml+Le35lkmhPf97rHsS0='} 2024-04-21T23:00:42.663 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi111.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667597', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-21 22:55:06.387758', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEqzOEpC4iF6MYQz/A+OjWQlHSxszVS/H13hLtt9oQaTUBzQRs9yeoDkxt5lbrSeDflEC6Haneqcu+tNpxds2D0='} 2024-04-21T23:00:42.686 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi174.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667597', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-21 22:55:06.384410', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDr2h/lwQ+5x1MozouN3F0vf5TFKKT9TfI0KrG+lrE4mK3z0Uoia+My2pbUQR3QkptY/XHF4CVxNVHlrmQGcA40='} 2024-04-21T23:00:42.686 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-21T23:00:42.693 INFO:teuthology.task.internal:roles: ubuntu@smithi040.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-04-21T23:00:42.693 INFO:teuthology.task.internal:roles: ubuntu@smithi111.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-04-21T23:00:42.693 INFO:teuthology.task.internal:roles: ubuntu@smithi174.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-04-21T23:00:42.693 INFO:teuthology.run_tasks:Running task console_log... 2024-04-21T23:00:42.816 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f9132918160>, signals=[15]) 2024-04-21T23:00:42.816 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-21T23:00:42.823 INFO:teuthology.task.internal:Opening connections... 2024-04-21T23:00:42.823 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi040.front.sepia.ceph.com 2024-04-21T23:00:42.825 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-21T23:00:42.907 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi111.front.sepia.ceph.com 2024-04-21T23:00:42.907 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-21T23:00:42.981 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi174.front.sepia.ceph.com 2024-04-21T23:00:42.981 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-21T23:00:43.058 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-21T23:00:43.065 DEBUG:teuthology.orchestra.run.smithi040:> uname -m 2024-04-21T23:00:43.084 INFO:teuthology.orchestra.run.smithi040.stdout:x86_64 2024-04-21T23:00:43.084 DEBUG:teuthology.orchestra.run.smithi040:> cat /etc/os-release 2024-04-21T23:00:43.142 INFO:teuthology.orchestra.run.smithi040.stdout:NAME="CentOS Stream" 2024-04-21T23:00:43.142 INFO:teuthology.orchestra.run.smithi040.stdout:VERSION="9" 2024-04-21T23:00:43.142 INFO:teuthology.orchestra.run.smithi040.stdout:ID="centos" 2024-04-21T23:00:43.142 INFO:teuthology.orchestra.run.smithi040.stdout:ID_LIKE="rhel fedora" 2024-04-21T23:00:43.142 INFO:teuthology.orchestra.run.smithi040.stdout:VERSION_ID="9" 2024-04-21T23:00:43.142 INFO:teuthology.orchestra.run.smithi040.stdout:PLATFORM_ID="platform:el9" 2024-04-21T23:00:43.142 INFO:teuthology.orchestra.run.smithi040.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-21T23:00:43.142 INFO:teuthology.orchestra.run.smithi040.stdout:ANSI_COLOR="0;31" 2024-04-21T23:00:43.142 INFO:teuthology.orchestra.run.smithi040.stdout:LOGO="fedora-logo-icon" 2024-04-21T23:00:43.142 INFO:teuthology.orchestra.run.smithi040.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-21T23:00:43.143 INFO:teuthology.orchestra.run.smithi040.stdout:HOME_URL="https://centos.org/" 2024-04-21T23:00:43.143 INFO:teuthology.orchestra.run.smithi040.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-21T23:00:43.143 INFO:teuthology.orchestra.run.smithi040.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-21T23:00:43.143 INFO:teuthology.orchestra.run.smithi040.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-21T23:00:43.143 INFO:teuthology.lock.ops:Updating smithi040.front.sepia.ceph.com on lock server 2024-04-21T23:00:43.165 DEBUG:teuthology.orchestra.run.smithi111:> uname -m 2024-04-21T23:00:43.183 INFO:teuthology.orchestra.run.smithi111.stdout:x86_64 2024-04-21T23:00:43.183 DEBUG:teuthology.orchestra.run.smithi111:> cat /etc/os-release 2024-04-21T23:00:43.240 INFO:teuthology.orchestra.run.smithi111.stdout:NAME="CentOS Stream" 2024-04-21T23:00:43.240 INFO:teuthology.orchestra.run.smithi111.stdout:VERSION="9" 2024-04-21T23:00:43.240 INFO:teuthology.orchestra.run.smithi111.stdout:ID="centos" 2024-04-21T23:00:43.240 INFO:teuthology.orchestra.run.smithi111.stdout:ID_LIKE="rhel fedora" 2024-04-21T23:00:43.240 INFO:teuthology.orchestra.run.smithi111.stdout:VERSION_ID="9" 2024-04-21T23:00:43.240 INFO:teuthology.orchestra.run.smithi111.stdout:PLATFORM_ID="platform:el9" 2024-04-21T23:00:43.240 INFO:teuthology.orchestra.run.smithi111.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-21T23:00:43.240 INFO:teuthology.orchestra.run.smithi111.stdout:ANSI_COLOR="0;31" 2024-04-21T23:00:43.240 INFO:teuthology.orchestra.run.smithi111.stdout:LOGO="fedora-logo-icon" 2024-04-21T23:00:43.240 INFO:teuthology.orchestra.run.smithi111.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-21T23:00:43.241 INFO:teuthology.orchestra.run.smithi111.stdout:HOME_URL="https://centos.org/" 2024-04-21T23:00:43.241 INFO:teuthology.orchestra.run.smithi111.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-21T23:00:43.241 INFO:teuthology.orchestra.run.smithi111.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-21T23:00:43.241 INFO:teuthology.orchestra.run.smithi111.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-21T23:00:43.241 INFO:teuthology.lock.ops:Updating smithi111.front.sepia.ceph.com on lock server 2024-04-21T23:00:43.265 DEBUG:teuthology.orchestra.run.smithi174:> uname -m 2024-04-21T23:00:43.283 INFO:teuthology.orchestra.run.smithi174.stdout:x86_64 2024-04-21T23:00:43.283 DEBUG:teuthology.orchestra.run.smithi174:> cat /etc/os-release 2024-04-21T23:00:43.339 INFO:teuthology.orchestra.run.smithi174.stdout:NAME="CentOS Stream" 2024-04-21T23:00:43.339 INFO:teuthology.orchestra.run.smithi174.stdout:VERSION="9" 2024-04-21T23:00:43.339 INFO:teuthology.orchestra.run.smithi174.stdout:ID="centos" 2024-04-21T23:00:43.339 INFO:teuthology.orchestra.run.smithi174.stdout:ID_LIKE="rhel fedora" 2024-04-21T23:00:43.340 INFO:teuthology.orchestra.run.smithi174.stdout:VERSION_ID="9" 2024-04-21T23:00:43.340 INFO:teuthology.orchestra.run.smithi174.stdout:PLATFORM_ID="platform:el9" 2024-04-21T23:00:43.340 INFO:teuthology.orchestra.run.smithi174.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-21T23:00:43.340 INFO:teuthology.orchestra.run.smithi174.stdout:ANSI_COLOR="0;31" 2024-04-21T23:00:43.340 INFO:teuthology.orchestra.run.smithi174.stdout:LOGO="fedora-logo-icon" 2024-04-21T23:00:43.340 INFO:teuthology.orchestra.run.smithi174.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-21T23:00:43.340 INFO:teuthology.orchestra.run.smithi174.stdout:HOME_URL="https://centos.org/" 2024-04-21T23:00:43.340 INFO:teuthology.orchestra.run.smithi174.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-21T23:00:43.340 INFO:teuthology.orchestra.run.smithi174.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-21T23:00:43.340 INFO:teuthology.orchestra.run.smithi174.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-21T23:00:43.341 INFO:teuthology.lock.ops:Updating smithi174.front.sepia.ceph.com on lock server 2024-04-21T23:00:43.365 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-21T23:00:43.378 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-21T23:00:43.412 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-21T23:00:43.412 DEBUG:teuthology.orchestra.run.smithi040:> test '!' -e /home/ubuntu/cephtest 2024-04-21T23:00:43.415 DEBUG:teuthology.orchestra.run.smithi111:> test '!' -e /home/ubuntu/cephtest 2024-04-21T23:00:43.417 DEBUG:teuthology.orchestra.run.smithi174:> test '!' -e /home/ubuntu/cephtest 2024-04-21T23:00:43.433 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-21T23:00:43.439 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-21T23:00:43.439 DEBUG:teuthology.orchestra.run.smithi040:> test -z $(ls -A /var/lib/ceph) 2024-04-21T23:00:43.473 DEBUG:teuthology.orchestra.run.smithi111:> test -z $(ls -A /var/lib/ceph) 2024-04-21T23:00:43.476 DEBUG:teuthology.orchestra.run.smithi174:> test -z $(ls -A /var/lib/ceph) 2024-04-21T23:00:43.583 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-21T23:00:43.662 INFO:teuthology.run_tasks:Running task kernel... 2024-04-21T23:00:43.678 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': 1, 'sha1': 'distro'} 2024-04-21T23:00:43.678 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-04-21T23:00:43.678 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-04-21T23:00:43.679 DEBUG:teuthology.orchestra.run.smithi040:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-21T23:00:43.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T23:00:43.696 DEBUG:teuthology.orchestra.run.smithi040:> uname -r 2024-04-21T23:00:43.754 INFO:teuthology.orchestra.run.smithi040.stdout:5.14.0-437.el9.x86_64 2024-04-21T23:00:43.754 INFO:teuthology.task.kernel:Running kernel on smithi040: 5.14.0-437.el9.x86_64 2024-04-21T23:00:43.754 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum install -y kernel 2024-04-21T23:00:46.124 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.0 MB 00:00 2024-04-21T23:00:48.722 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - AppStream 12 MB/s | 19 MB 00:01 2024-04-21T23:00:52.890 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.2 MB 00:00 2024-04-21T23:00:54.374 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - Extras packages 60 kB/s | 16 kB 00:00 2024-04-21T23:00:55.992 INFO:teuthology.orchestra.run.smithi040.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-04-21T23:01:01.259 INFO:teuthology.orchestra.run.smithi040.stdout:lab-extras 43 kB/s | 1.7 kB 00:00 2024-04-21T23:01:03.265 INFO:teuthology.orchestra.run.smithi040.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-21T23:01:03.265 INFO:teuthology.orchestra.run.smithi040.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-21T23:01:03.364 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-04-21T23:01:03.365 INFO:teuthology.orchestra.run.smithi040.stdout:Nothing to do. 2024-04-21T23:01:03.365 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-04-21T23:01:03.473 DEBUG:teuthology.orchestra.run.smithi040:> echo no | sudo yum reinstall kernel || true 2024-04-21T23:01:03.993 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:00:02 ago on Sun 21 Apr 2024 11:01:01 PM UTC. 2024-04-21T23:01:04.125 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-04-21T23:01:04.134 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-21T23:01:04.135 INFO:teuthology.orchestra.run.smithi040.stdout: Package Architecture Version Repository Size 2024-04-21T23:01:04.135 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-21T23:01:04.135 INFO:teuthology.orchestra.run.smithi040.stdout:Reinstalling: 2024-04-21T23:01:04.135 INFO:teuthology.orchestra.run.smithi040.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-21T23:01:04.135 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:01:04.135 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-04-21T23:01:04.135 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-21T23:01:04.135 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:01:04.135 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 255 k 2024-04-21T23:01:04.135 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 0 2024-04-21T23:01:04.136 INFO:teuthology.orchestra.run.smithi040.stderr:Operation aborted. 2024-04-21T23:01:04.198 INFO:teuthology.orchestra.run.smithi040.stdout:Is this ok [y/N]: 2024-04-21T23:01:04.198 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum reinstall -y kernel || true 2024-04-21T23:01:04.713 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:00:03 ago on Sun 21 Apr 2024 11:01:01 PM UTC. 2024-04-21T23:01:04.834 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-04-21T23:01:04.834 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-21T23:01:04.835 INFO:teuthology.orchestra.run.smithi040.stdout: Package Architecture Version Repository Size 2024-04-21T23:01:04.835 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-21T23:01:04.835 INFO:teuthology.orchestra.run.smithi040.stdout:Reinstalling: 2024-04-21T23:01:04.835 INFO:teuthology.orchestra.run.smithi040.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-21T23:01:04.835 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:01:04.835 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-04-21T23:01:04.835 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-21T23:01:04.835 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:01:04.835 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 255 k 2024-04-21T23:01:04.835 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 0 2024-04-21T23:01:04.835 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-04-21T23:01:06.627 INFO:teuthology.orchestra.run.smithi040.stdout:kernel-5.14.0-437.el9.x86_64.rpm 432 kB/s | 255 kB 00:00 2024-04-21T23:01:06.627 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-04-21T23:01:06.627 INFO:teuthology.orchestra.run.smithi040.stdout:Total 142 kB/s | 255 kB 00:01 2024-04-21T23:01:06.645 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-04-21T23:01:06.842 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-04-21T23:01:06.843 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-04-21T23:01:06.848 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-04-21T23:01:06.848 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-04-21T23:01:06.977 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-04-21T23:01:07.069 INFO:teuthology.orchestra.run.smithi040.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-21T23:01:07.070 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-21T23:01:07.795 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-21T23:01:07.795 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-21T23:01:08.543 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-21T23:01:08.543 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:01:08.543 INFO:teuthology.orchestra.run.smithi040.stdout:Reinstalled: 2024-04-21T23:01:08.543 INFO:teuthology.orchestra.run.smithi040.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-21T23:01:08.544 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:01:08.544 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-04-21T23:01:08.743 DEBUG:teuthology.orchestra.run.smithi040:> rpm -q kernel | sort -rV | head -n 1 2024-04-21T23:01:08.790 INFO:teuthology.orchestra.run.smithi040.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-21T23:01:08.791 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-21T23:01:08.791 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-21T23:01:08.792 DEBUG:teuthology.parallel:result is None 2024-04-21T23:01:08.792 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-21T23:01:08.799 INFO:teuthology.task.internal:Creating test directory... 2024-04-21T23:01:08.799 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-21T23:01:08.801 DEBUG:teuthology.orchestra.run.smithi111:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-21T23:01:08.804 DEBUG:teuthology.orchestra.run.smithi174:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-21T23:01:08.822 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-21T23:01:08.844 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-21T23:01:08.868 INFO:teuthology.task.internal:Creating archive directory... 2024-04-21T23:01:08.868 DEBUG:teuthology.orchestra.run.smithi040:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-21T23:01:08.870 DEBUG:teuthology.orchestra.run.smithi111:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-21T23:01:08.872 DEBUG:teuthology.orchestra.run.smithi174:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-21T23:01:08.983 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-21T23:01:08.990 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-21T23:01:08.990 DEBUG:teuthology.orchestra.run.smithi111:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-21T23:01:09.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T23:01:09.007 DEBUG:teuthology.orchestra.run.smithi174:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-21T23:01:09.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T23:01:09.024 DEBUG:teuthology.orchestra.run.smithi040:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-21T23:01:09.026 DEBUG:teuthology.orchestra.run.smithi111:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-21T23:01:09.049 DEBUG:teuthology.orchestra.run.smithi174:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-21T23:01:09.090 INFO:teuthology.orchestra.run.smithi040.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-21T23:01:09.097 INFO:teuthology.orchestra.run.smithi111.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-21T23:01:09.098 INFO:teuthology.orchestra.run.smithi174.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-21T23:01:09.106 INFO:teuthology.orchestra.run.smithi040.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-21T23:01:09.111 INFO:teuthology.orchestra.run.smithi174.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-21T23:01:09.114 INFO:teuthology.orchestra.run.smithi111.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-21T23:01:09.116 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-21T23:01:09.123 INFO:teuthology.task.internal:Configuring sudo... 2024-04-21T23:01:09.123 DEBUG:teuthology.orchestra.run.smithi040:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-21T23:01:09.125 DEBUG:teuthology.orchestra.run.smithi111:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-21T23:01:09.127 DEBUG:teuthology.orchestra.run.smithi174:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-21T23:01:09.179 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-21T23:01:09.188 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-21T23:01:09.188 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-21T23:01:09.190 DEBUG:teuthology.orchestra.run.smithi111:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-21T23:01:09.193 DEBUG:teuthology.orchestra.run.smithi174:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-21T23:01:09.234 DEBUG:teuthology.orchestra.run.smithi040:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-21T23:01:09.285 DEBUG:teuthology.orchestra.run.smithi040:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-21T23:01:09.331 DEBUG:teuthology.orchestra.run.smithi040:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-21T23:01:09.413 DEBUG:teuthology.orchestra.run.smithi040:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-21T23:01:09.452 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-21T23:01:09.452 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-21T23:01:09.516 DEBUG:teuthology.orchestra.run.smithi111:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-21T23:01:09.557 DEBUG:teuthology.orchestra.run.smithi111:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-21T23:01:09.606 DEBUG:teuthology.orchestra.run.smithi111:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-21T23:01:09.693 DEBUG:teuthology.orchestra.run.smithi111:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-21T23:01:09.731 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-04-21T23:01:09.731 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-21T23:01:09.797 DEBUG:teuthology.orchestra.run.smithi174:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-21T23:01:09.839 DEBUG:teuthology.orchestra.run.smithi174:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-21T23:01:09.905 DEBUG:teuthology.orchestra.run.smithi174:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-21T23:01:09.947 DEBUG:teuthology.orchestra.run.smithi174:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-21T23:01:09.989 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-04-21T23:01:09.990 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-21T23:01:10.055 DEBUG:teuthology.orchestra.run.smithi040:> sudo service rsyslog restart 2024-04-21T23:01:10.057 DEBUG:teuthology.orchestra.run.smithi111:> sudo service rsyslog restart 2024-04-21T23:01:10.060 DEBUG:teuthology.orchestra.run.smithi174:> sudo service rsyslog restart 2024-04-21T23:01:10.094 INFO:teuthology.orchestra.run.smithi111.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-21T23:01:10.102 INFO:teuthology.orchestra.run.smithi040.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-21T23:01:10.128 INFO:teuthology.orchestra.run.smithi174.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-21T23:01:10.492 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-21T23:01:10.499 INFO:teuthology.task.internal:Starting timer... 2024-04-21T23:01:10.499 INFO:teuthology.run_tasks:Running task pcp... 2024-04-21T23:01:10.546 INFO:teuthology.run_tasks:Running task selinux... 2024-04-21T23:01:10.572 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-04-21T23:01:10.572 DEBUG:teuthology.orchestra.run.smithi040:> sudo service auditd rotate 2024-04-21T23:01:10.638 INFO:teuthology.orchestra.run.smithi040.stdout:Rotating logs: 2024-04-21T23:01:10.639 DEBUG:teuthology.orchestra.run.smithi111:> sudo service auditd rotate 2024-04-21T23:01:10.695 INFO:teuthology.orchestra.run.smithi111.stdout:Rotating logs: 2024-04-21T23:01:10.697 DEBUG:teuthology.orchestra.run.smithi174:> sudo service auditd rotate 2024-04-21T23:01:10.753 INFO:teuthology.orchestra.run.smithi174.stdout:Rotating logs: 2024-04-21T23:01:10.756 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-21T23:01:10.756 DEBUG:teuthology.orchestra.run.smithi040:> /usr/sbin/getenforce 2024-04-21T23:01:10.779 INFO:teuthology.orchestra.run.smithi040.stdout:Permissive 2024-04-21T23:01:10.779 DEBUG:teuthology.orchestra.run.smithi111:> /usr/sbin/getenforce 2024-04-21T23:01:10.806 INFO:teuthology.orchestra.run.smithi111.stdout:Permissive 2024-04-21T23:01:10.806 DEBUG:teuthology.orchestra.run.smithi174:> /usr/sbin/getenforce 2024-04-21T23:01:10.833 INFO:teuthology.orchestra.run.smithi174.stdout:Permissive 2024-04-21T23:01:10.833 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi040.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi111.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi174.front.sepia.ceph.com': 'permissive'} 2024-04-21T23:01:10.834 DEBUG:teuthology.orchestra.run.smithi040:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-21T23:01:10.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T23:01:10.860 DEBUG:teuthology.orchestra.run.smithi111:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-21T23:01:10.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T23:01:10.889 DEBUG:teuthology.orchestra.run.smithi174:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-21T23:01:10.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T23:01:10.914 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-21T23:01:10.914 DEBUG:teuthology.orchestra.run.smithi040:> sudo /usr/sbin/setenforce permissive 2024-04-21T23:01:10.939 DEBUG:teuthology.orchestra.run.smithi111:> sudo /usr/sbin/setenforce permissive 2024-04-21T23:01:10.965 DEBUG:teuthology.orchestra.run.smithi174:> sudo /usr/sbin/setenforce permissive 2024-04-21T23:01:11.000 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-21T23:01:11.009 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-21T23:01:11.015 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-21T23:01:11.165 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-21T23:01:11.182 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-21T23:01:11.184 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi040.front.sepia.ceph.com,smithi111.front.sepia.ceph.com,smithi174.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-21T23:04:59.795 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi040.front.sepia.ceph.com'), Remote(name='ubuntu@smithi111.front.sepia.ceph.com'), Remote(name='ubuntu@smithi174.front.sepia.ceph.com')] 2024-04-21T23:04:59.796 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi040.front.sepia.ceph.com' 2024-04-21T23:04:59.797 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-21T23:04:59.876 DEBUG:teuthology.orchestra.run.smithi040:> true 2024-04-21T23:04:59.954 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi040.front.sepia.ceph.com' 2024-04-21T23:04:59.954 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2024-04-21T23:04:59.954 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-21T23:05:00.028 DEBUG:teuthology.orchestra.run.smithi111:> true 2024-04-21T23:05:00.109 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi111.front.sepia.ceph.com' 2024-04-21T23:05:00.109 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-04-21T23:05:00.110 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-21T23:05:00.184 DEBUG:teuthology.orchestra.run.smithi174:> true 2024-04-21T23:05:00.217 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-04-21T23:05:00.217 INFO:teuthology.run_tasks:Running task clock... 2024-04-21T23:05:00.229 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-21T23:05:00.229 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-21T23:05:00.229 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-21T23:05:00.232 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-21T23:05:00.232 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-21T23:05:00.234 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-21T23:05:00.234 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-21T23:05:00.259 INFO:teuthology.orchestra.run.smithi111.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-21T23:05:00.260 INFO:teuthology.orchestra.run.smithi040.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-21T23:05:00.272 INFO:teuthology.orchestra.run.smithi111.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-21T23:05:00.275 INFO:teuthology.orchestra.run.smithi040.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-21T23:05:00.285 INFO:teuthology.orchestra.run.smithi174.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-21T23:05:00.294 INFO:teuthology.orchestra.run.smithi111.stderr:sudo: ntpd: command not found 2024-04-21T23:05:00.299 INFO:teuthology.orchestra.run.smithi040.stderr:sudo: ntpd: command not found 2024-04-21T23:05:00.299 INFO:teuthology.orchestra.run.smithi174.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-21T23:05:00.304 INFO:teuthology.orchestra.run.smithi111.stdout:506 Cannot talk to daemon 2024-04-21T23:05:00.310 INFO:teuthology.orchestra.run.smithi040.stdout:506 Cannot talk to daemon 2024-04-21T23:05:00.317 INFO:teuthology.orchestra.run.smithi111.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-21T23:05:00.321 INFO:teuthology.orchestra.run.smithi174.stderr:sudo: ntpd: command not found 2024-04-21T23:05:00.324 INFO:teuthology.orchestra.run.smithi040.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-21T23:05:00.330 INFO:teuthology.orchestra.run.smithi111.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-21T23:05:00.331 INFO:teuthology.orchestra.run.smithi174.stdout:506 Cannot talk to daemon 2024-04-21T23:05:00.337 INFO:teuthology.orchestra.run.smithi040.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-21T23:05:00.344 INFO:teuthology.orchestra.run.smithi174.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-21T23:05:00.357 INFO:teuthology.orchestra.run.smithi174.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-21T23:05:00.385 INFO:teuthology.orchestra.run.smithi111.stderr:bash: line 1: ntpq: command not found 2024-04-21T23:05:00.388 INFO:teuthology.orchestra.run.smithi111.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-21T23:05:00.388 INFO:teuthology.orchestra.run.smithi111.stdout:=============================================================================== 2024-04-21T23:05:00.388 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T23:05:00.388 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T23:05:00.388 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T23:05:00.389 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T23:05:00.394 INFO:teuthology.orchestra.run.smithi040.stderr:bash: line 1: ntpq: command not found 2024-04-21T23:05:00.397 INFO:teuthology.orchestra.run.smithi040.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-21T23:05:00.397 INFO:teuthology.orchestra.run.smithi040.stdout:=============================================================================== 2024-04-21T23:05:00.397 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T23:05:00.397 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T23:05:00.397 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T23:05:00.397 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T23:05:00.411 INFO:teuthology.orchestra.run.smithi174.stderr:bash: line 1: ntpq: command not found 2024-04-21T23:05:00.415 INFO:teuthology.orchestra.run.smithi174.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-21T23:05:00.415 INFO:teuthology.orchestra.run.smithi174.stdout:=============================================================================== 2024-04-21T23:05:00.415 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T23:05:00.415 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T23:05:00.415 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T23:05:00.415 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T23:05:00.415 INFO:teuthology.run_tasks:Running task install... 2024-04-21T23:05:00.423 DEBUG:teuthology.task.install:project ceph 2024-04-21T23:05:00.423 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '820a2f8ced66152ed3d765c31444b8076ea28c6b'}} 2024-04-21T23:05:00.423 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '820a2f8ced66152ed3d765c31444b8076ea28c6b'} 2024-04-21T23:05:00.423 INFO:teuthology.task.install:Using flavor: default 2024-04-21T23:05:00.428 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-21T23:05:00.429 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-21T23:05:00.429 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '820a2f8ced66152ed3d765c31444b8076ea28c6b', 'tag': None, 'wait_for_package': False} 2024-04-21T23:05:00.429 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=820a2f8ced66152ed3d765c31444b8076ea28c6b 2024-04-21T23:05:00.431 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '820a2f8ced66152ed3d765c31444b8076ea28c6b', 'tag': None, 'wait_for_package': False} 2024-04-21T23:05:00.431 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=820a2f8ced66152ed3d765c31444b8076ea28c6b 2024-04-21T23:05:00.432 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '820a2f8ced66152ed3d765c31444b8076ea28c6b', 'tag': None, 'wait_for_package': False} 2024-04-21T23:05:00.432 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=820a2f8ced66152ed3d765c31444b8076ea28c6b 2024-04-21T23:05:00.575 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/centos/9/flavors/default/ 2024-04-21T23:05:00.575 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2405.g820a2f8c 2024-04-21T23:05:00.579 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/centos/9/flavors/default/ 2024-04-21T23:05:00.580 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2405.g820a2f8c 2024-04-21T23:05:00.588 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/centos/9/flavors/default/ 2024-04-21T23:05:00.588 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2405.g820a2f8c 2024-04-21T23:05:00.686 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-21T23:05:00.686 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-21T23:05:00.686 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-21T23:05:00.696 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-21T23:05:00.696 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-04-21T23:05:00.696 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-21T23:05:00.711 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-21T23:05:00.711 DEBUG:teuthology.orchestra.run.smithi040:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/820a2f8ced66152ed3d765c31444b8076ea28c6b/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-21T23:05:00.721 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-21T23:05:00.721 DEBUG:teuthology.orchestra.run.smithi111:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/820a2f8ced66152ed3d765c31444b8076ea28c6b/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-21T23:05:00.749 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-21T23:05:00.749 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-04-21T23:05:00.749 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-21T23:05:00.774 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-21T23:05:00.775 DEBUG:teuthology.orchestra.run.smithi174:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/820a2f8ced66152ed3d765c31444b8076ea28c6b/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-21T23:05:00.776 DEBUG:teuthology.orchestra.run.smithi040:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-21T23:05:00.786 DEBUG:teuthology.orchestra.run.smithi111:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-21T23:05:00.839 DEBUG:teuthology.orchestra.run.smithi174:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-21T23:05:00.850 DEBUG:teuthology.orchestra.run.smithi040:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-21T23:05:00.858 DEBUG:teuthology.orchestra.run.smithi111:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-21T23:05:00.912 DEBUG:teuthology.orchestra.run.smithi174:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-21T23:05:00.913 INFO:teuthology.orchestra.run.smithi040.stdout:check_obsoletes = 1 2024-04-21T23:05:00.915 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum clean all 2024-04-21T23:05:00.920 INFO:teuthology.orchestra.run.smithi111.stdout:check_obsoletes = 1 2024-04-21T23:05:00.922 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum clean all 2024-04-21T23:05:00.975 INFO:teuthology.orchestra.run.smithi174.stdout:check_obsoletes = 1 2024-04-21T23:05:00.977 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum clean all 2024-04-21T23:05:01.145 INFO:teuthology.orchestra.run.smithi111.stdout:48 files removed 2024-04-21T23:05:01.148 INFO:teuthology.orchestra.run.smithi040.stdout:48 files removed 2024-04-21T23:05:01.172 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-21T23:05:01.176 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-21T23:05:01.199 INFO:teuthology.orchestra.run.smithi174.stdout:48 files removed 2024-04-21T23:05:01.226 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-21T23:05:01.833 INFO:teuthology.orchestra.run.smithi111.stdout:ceph packages for x86_64 208 kB/s | 83 kB 00:00 2024-04-21T23:05:01.891 INFO:teuthology.orchestra.run.smithi174.stdout:ceph packages for x86_64 205 kB/s | 83 kB 00:00 2024-04-21T23:05:01.960 INFO:teuthology.orchestra.run.smithi040.stdout:ceph packages for x86_64 163 kB/s | 83 kB 00:00 2024-04-21T23:05:02.191 INFO:teuthology.orchestra.run.smithi111.stdout:ceph noarch packages 37 kB/s | 11 kB 00:00 2024-04-21T23:05:02.308 INFO:teuthology.orchestra.run.smithi174.stdout:ceph noarch packages 31 kB/s | 11 kB 00:00 2024-04-21T23:05:02.327 INFO:teuthology.orchestra.run.smithi040.stdout:ceph noarch packages 36 kB/s | 11 kB 00:00 2024-04-21T23:05:02.542 INFO:teuthology.orchestra.run.smithi111.stdout:ceph source packages 5.9 kB/s | 1.8 kB 00:00 2024-04-21T23:05:02.661 INFO:teuthology.orchestra.run.smithi040.stdout:ceph source packages 6.2 kB/s | 1.8 kB 00:00 2024-04-21T23:05:02.717 INFO:teuthology.orchestra.run.smithi174.stdout:ceph source packages 4.9 kB/s | 1.8 kB 00:00 2024-04-21T23:05:03.478 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.0 MB 00:00 2024-04-21T23:05:03.484 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 9 - BaseOS 8.9 MB/s | 8.0 MB 00:00 2024-04-21T23:05:05.506 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - AppStream 20 MB/s | 19 MB 00:00 2024-04-21T23:05:05.917 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 9 - BaseOS 2.5 MB/s | 8.0 MB 00:03 2024-04-21T23:05:09.043 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 9 - AppStream 9.2 MB/s | 19 MB 00:02 2024-04-21T23:05:10.838 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - CRB 3.8 MB/s | 6.2 MB 00:01 2024-04-21T23:05:12.480 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 16 kB 00:00 2024-04-21T23:05:13.462 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 9 - CRB 7.1 MB/s | 6.2 MB 00:00 2024-04-21T23:05:15.157 INFO:teuthology.orchestra.run.smithi040.stdout:Extra Packages for Enterprise Linux 8.2 MB/s | 21 MB 00:02 2024-04-21T23:05:15.186 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 9 - Extras packages 27 kB/s | 16 kB 00:00 2024-04-21T23:05:18.947 INFO:teuthology.orchestra.run.smithi174.stdout:Extra Packages for Enterprise Linux 5.7 MB/s | 21 MB 00:03 2024-04-21T23:05:20.332 INFO:teuthology.orchestra.run.smithi040.stdout:lab-extras 37 kB/s | 1.7 kB 00:00 2024-04-21T23:05:22.551 INFO:teuthology.orchestra.run.smithi040.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-21T23:05:22.551 INFO:teuthology.orchestra.run.smithi040.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-21T23:05:22.563 INFO:teuthology.orchestra.run.smithi040.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-21T23:05:22.563 INFO:teuthology.orchestra.run.smithi040.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-21T23:05:22.564 INFO:teuthology.orchestra.run.smithi040.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-21T23:05:22.565 INFO:teuthology.orchestra.run.smithi040.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-21T23:05:22.567 INFO:teuthology.orchestra.run.smithi040.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-21T23:05:22.569 INFO:teuthology.orchestra.run.smithi040.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-21T23:05:22.570 INFO:teuthology.orchestra.run.smithi040.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-21T23:05:22.635 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-04-21T23:05:22.644 INFO:teuthology.orchestra.run.smithi040.stdout:======================================================================================= 2024-04-21T23:05:22.645 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2024-04-21T23:05:22.645 INFO:teuthology.orchestra.run.smithi040.stdout:======================================================================================= 2024-04-21T23:05:22.645 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2024-04-21T23:05:22.645 INFO:teuthology.orchestra.run.smithi040.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-21T23:05:22.645 INFO:teuthology.orchestra.run.smithi040.stdout: ceph x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 6.6 k 2024-04-21T23:05:22.645 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-base x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 5.4 M 2024-04-21T23:05:22.645 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-fuse x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 889 k 2024-04-21T23:05:22.645 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 145 k 2024-04-21T23:05:22.645 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 1.5 M 2024-04-21T23:05:22.645 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 141 k 2024-04-21T23:05:22.645 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 3.5 M 2024-04-21T23:05:22.645 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 7.4 M 2024-04-21T23:05:22.645 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-rook noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 49 k 2024-04-21T23:05:22.646 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-radosgw x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 10 M 2024-04-21T23:05:22.646 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-test x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 48 M 2024-04-21T23:05:22.646 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-volume noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 264 k 2024-04-21T23:05:22.646 INFO:teuthology.orchestra.run.smithi040.stdout: cephadm noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 762 k 2024-04-21T23:05:22.646 INFO:teuthology.orchestra.run.smithi040.stdout: cephfs-mirror x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 222 k 2024-04-21T23:05:22.646 INFO:teuthology.orchestra.run.smithi040.stdout: cephfs-shell x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 27 k 2024-04-21T23:05:22.646 INFO:teuthology.orchestra.run.smithi040.stdout: cephfs-top noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 18 k 2024-04-21T23:05:22.646 INFO:teuthology.orchestra.run.smithi040.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-21T23:05:22.646 INFO:teuthology.orchestra.run.smithi040.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-21T23:05:22.646 INFO:teuthology.orchestra.run.smithi040.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-21T23:05:22.646 INFO:teuthology.orchestra.run.smithi040.stdout: libcephfs-devel x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 32 k 2024-04-21T23:05:22.647 INFO:teuthology.orchestra.run.smithi040.stdout: libcephfs2 x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 745 k 2024-04-21T23:05:22.647 INFO:teuthology.orchestra.run.smithi040.stdout: librados-devel x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 127 k 2024-04-21T23:05:22.647 INFO:teuthology.orchestra.run.smithi040.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-21T23:05:22.647 INFO:teuthology.orchestra.run.smithi040.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-21T23:05:22.647 INFO:teuthology.orchestra.run.smithi040.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-04-21T23:05:22.647 INFO:teuthology.orchestra.run.smithi040.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-21T23:05:22.647 INFO:teuthology.orchestra.run.smithi040.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-21T23:05:22.647 INFO:teuthology.orchestra.run.smithi040.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-21T23:05:22.647 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cephfs x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 161 k 2024-04-21T23:05:22.647 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rados x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 321 k 2024-04-21T23:05:22.647 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rbd x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 298 k 2024-04-21T23:05:22.647 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rgw x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 99 k 2024-04-21T23:05:22.647 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-fuse x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 86 k 2024-04-21T23:05:22.648 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-mirror x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 3.1 M 2024-04-21T23:05:22.648 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-nbd x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 171 k 2024-04-21T23:05:22.648 INFO:teuthology.orchestra.run.smithi040.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-21T23:05:22.648 INFO:teuthology.orchestra.run.smithi040.stdout:Upgrading: 2024-04-21T23:05:22.648 INFO:teuthology.orchestra.run.smithi040.stdout: librados2 x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 3.4 M 2024-04-21T23:05:22.648 INFO:teuthology.orchestra.run.smithi040.stdout: librbd1 x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 3.1 M 2024-04-21T23:05:22.648 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2024-04-21T23:05:22.648 INFO:teuthology.orchestra.run.smithi040.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-21T23:05:22.648 INFO:teuthology.orchestra.run.smithi040.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-21T23:05:22.648 INFO:teuthology.orchestra.run.smithi040.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-21T23:05:22.648 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-common x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 21 M 2024-04-21T23:05:22.648 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 32 k 2024-04-21T23:05:22.648 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mds x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 2.4 M 2024-04-21T23:05:22.648 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 249 k 2024-04-21T23:05:22.649 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mon x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 4.7 M 2024-04-21T23:05:22.649 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-osd x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 17 M 2024-04-21T23:05:22.649 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 15 k 2024-04-21T23:05:22.649 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-selinux x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 25 k 2024-04-21T23:05:22.649 INFO:teuthology.orchestra.run.smithi040.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-21T23:05:22.649 INFO:teuthology.orchestra.run.smithi040.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-21T23:05:22.649 INFO:teuthology.orchestra.run.smithi040.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-21T23:05:22.649 INFO:teuthology.orchestra.run.smithi040.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-21T23:05:22.649 INFO:teuthology.orchestra.run.smithi040.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-21T23:05:22.649 INFO:teuthology.orchestra.run.smithi040.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-21T23:05:22.649 INFO:teuthology.orchestra.run.smithi040.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-21T23:05:22.649 INFO:teuthology.orchestra.run.smithi040.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-21T23:05:22.650 INFO:teuthology.orchestra.run.smithi040.stdout: libcephsqlite x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 164 k 2024-04-21T23:05:22.650 INFO:teuthology.orchestra.run.smithi040.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-21T23:05:22.650 INFO:teuthology.orchestra.run.smithi040.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-21T23:05:22.650 INFO:teuthology.orchestra.run.smithi040.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-21T23:05:22.650 INFO:teuthology.orchestra.run.smithi040.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-21T23:05:22.650 INFO:teuthology.orchestra.run.smithi040.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-21T23:05:22.650 INFO:teuthology.orchestra.run.smithi040.stdout: libradosstriper1 x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 501 k 2024-04-21T23:05:22.650 INFO:teuthology.orchestra.run.smithi040.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-21T23:05:22.650 INFO:teuthology.orchestra.run.smithi040.stdout: librgw2 x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 5.3 M 2024-04-21T23:05:22.650 INFO:teuthology.orchestra.run.smithi040.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-21T23:05:22.650 INFO:teuthology.orchestra.run.smithi040.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-21T23:05:22.650 INFO:teuthology.orchestra.run.smithi040.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-21T23:05:22.650 INFO:teuthology.orchestra.run.smithi040.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-21T23:05:22.651 INFO:teuthology.orchestra.run.smithi040.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-21T23:05:22.651 INFO:teuthology.orchestra.run.smithi040.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-21T23:05:22.651 INFO:teuthology.orchestra.run.smithi040.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-21T23:05:22.651 INFO:teuthology.orchestra.run.smithi040.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-21T23:05:22.651 INFO:teuthology.orchestra.run.smithi040.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-21T23:05:22.651 INFO:teuthology.orchestra.run.smithi040.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-21T23:05:22.651 INFO:teuthology.orchestra.run.smithi040.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-21T23:05:22.651 INFO:teuthology.orchestra.run.smithi040.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-21T23:05:22.651 INFO:teuthology.orchestra.run.smithi040.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-21T23:05:22.651 INFO:teuthology.orchestra.run.smithi040.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-21T23:05:22.651 INFO:teuthology.orchestra.run.smithi040.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-21T23:05:22.651 INFO:teuthology.orchestra.run.smithi040.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-21T23:05:22.651 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-21T23:05:22.652 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ceph-argparse x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 45 k 2024-04-21T23:05:22.652 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ceph-common x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 133 k 2024-04-21T23:05:22.652 INFO:teuthology.orchestra.run.smithi040.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-21T23:05:22.652 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-21T23:05:22.652 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-21T23:05:22.652 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-21T23:05:22.652 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-21T23:05:22.652 INFO:teuthology.orchestra.run.smithi040.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-21T23:05:22.652 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-21T23:05:22.652 INFO:teuthology.orchestra.run.smithi040.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-21T23:05:22.652 INFO:teuthology.orchestra.run.smithi040.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-21T23:05:22.652 INFO:teuthology.orchestra.run.smithi040.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-21T23:05:22.652 INFO:teuthology.orchestra.run.smithi040.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-21T23:05:22.653 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-21T23:05:22.653 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-21T23:05:22.653 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-21T23:05:22.653 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-21T23:05:22.653 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-21T23:05:22.653 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-21T23:05:22.653 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-21T23:05:22.653 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-21T23:05:22.653 INFO:teuthology.orchestra.run.smithi040.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-21T23:05:22.653 INFO:teuthology.orchestra.run.smithi040.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-21T23:05:22.653 INFO:teuthology.orchestra.run.smithi040.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-21T23:05:22.653 INFO:teuthology.orchestra.run.smithi040.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-21T23:05:22.653 INFO:teuthology.orchestra.run.smithi040.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-21T23:05:22.653 INFO:teuthology.orchestra.run.smithi040.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-21T23:05:22.654 INFO:teuthology.orchestra.run.smithi040.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-21T23:05:22.654 INFO:teuthology.orchestra.run.smithi040.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-21T23:05:22.654 INFO:teuthology.orchestra.run.smithi040.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-21T23:05:22.654 INFO:teuthology.orchestra.run.smithi040.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-21T23:05:22.654 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-21T23:05:22.654 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-21T23:05:22.654 INFO:teuthology.orchestra.run.smithi040.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-21T23:05:22.654 INFO:teuthology.orchestra.run.smithi040.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-21T23:05:22.654 INFO:teuthology.orchestra.run.smithi040.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-21T23:05:22.654 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-21T23:05:22.654 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-21T23:05:22.654 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-21T23:05:22.654 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-21T23:05:22.655 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-21T23:05:22.655 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-21T23:05:22.655 INFO:teuthology.orchestra.run.smithi040.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-21T23:05:22.655 INFO:teuthology.orchestra.run.smithi040.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-21T23:05:22.655 INFO:teuthology.orchestra.run.smithi040.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-21T23:05:22.655 INFO:teuthology.orchestra.run.smithi040.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-21T23:05:22.655 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-21T23:05:22.655 INFO:teuthology.orchestra.run.smithi040.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-21T23:05:22.655 INFO:teuthology.orchestra.run.smithi040.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-21T23:05:22.655 INFO:teuthology.orchestra.run.smithi040.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-21T23:05:22.655 INFO:teuthology.orchestra.run.smithi040.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-21T23:05:22.655 INFO:teuthology.orchestra.run.smithi040.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-21T23:05:22.656 INFO:teuthology.orchestra.run.smithi040.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-21T23:05:22.656 INFO:teuthology.orchestra.run.smithi040.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-21T23:05:22.656 INFO:teuthology.orchestra.run.smithi040.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-21T23:05:22.656 INFO:teuthology.orchestra.run.smithi040.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-21T23:05:22.656 INFO:teuthology.orchestra.run.smithi040.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-21T23:05:22.656 INFO:teuthology.orchestra.run.smithi040.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-21T23:05:22.656 INFO:teuthology.orchestra.run.smithi040.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-21T23:05:22.656 INFO:teuthology.orchestra.run.smithi040.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-21T23:05:22.656 INFO:teuthology.orchestra.run.smithi040.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-21T23:05:22.656 INFO:teuthology.orchestra.run.smithi040.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-21T23:05:22.656 INFO:teuthology.orchestra.run.smithi040.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-21T23:05:22.656 INFO:teuthology.orchestra.run.smithi040.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-21T23:05:22.656 INFO:teuthology.orchestra.run.smithi040.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-21T23:05:22.657 INFO:teuthology.orchestra.run.smithi040.stdout:Installing weak dependencies: 2024-04-21T23:05:22.657 INFO:teuthology.orchestra.run.smithi040.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-21T23:05:22.657 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:05:22.657 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-04-21T23:05:22.657 INFO:teuthology.orchestra.run.smithi040.stdout:======================================================================================= 2024-04-21T23:05:22.657 INFO:teuthology.orchestra.run.smithi040.stdout:Install 148 Packages 2024-04-21T23:05:22.657 INFO:teuthology.orchestra.run.smithi040.stdout:Upgrade 2 Packages 2024-04-21T23:05:22.657 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:05:22.658 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 216 M 2024-04-21T23:05:22.658 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-04-21T23:05:23.966 INFO:teuthology.orchestra.run.smithi040.stdout:(1/150): ceph-19.0.0-2405.g820a2f8c.el9.x86_64. 42 kB/s | 6.6 kB 00:00 2024-04-21T23:05:24.013 INFO:teuthology.orchestra.run.smithi174.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-04-21T23:05:24.150 INFO:teuthology.orchestra.run.smithi040.stdout:(2/150): ceph-fuse-19.0.0-2405.g820a2f8c.el9.x8 4.7 MB/s | 889 kB 00:00 2024-04-21T23:05:24.225 INFO:teuthology.orchestra.run.smithi040.stdout:(3/150): ceph-immutable-object-cache-19.0.0-240 1.9 MB/s | 145 kB 00:00 2024-04-21T23:05:24.484 INFO:teuthology.orchestra.run.smithi040.stdout:(4/150): ceph-mds-19.0.0-2405.g820a2f8c.el9.x86 9.1 MB/s | 2.4 MB 00:00 2024-04-21T23:05:24.718 INFO:teuthology.orchestra.run.smithi040.stdout:(5/150): ceph-mgr-19.0.0-2405.g820a2f8c.el9.x86 6.4 MB/s | 1.5 MB 00:00 2024-04-21T23:05:25.318 INFO:teuthology.orchestra.run.smithi040.stdout:(6/150): ceph-common-19.0.0-2405.g820a2f8c.el9. 14 MB/s | 21 MB 00:01 2024-04-21T23:05:25.451 INFO:teuthology.orchestra.run.smithi040.stdout:(7/150): ceph-mon-19.0.0-2405.g820a2f8c.el9.x86 6.4 MB/s | 4.7 MB 00:00 2024-04-21T23:05:26.165 INFO:teuthology.orchestra.run.smithi174.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-21T23:05:26.166 INFO:teuthology.orchestra.run.smithi174.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-21T23:05:26.177 INFO:teuthology.orchestra.run.smithi174.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-21T23:05:26.177 INFO:teuthology.orchestra.run.smithi174.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-21T23:05:26.178 INFO:teuthology.orchestra.run.smithi174.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-21T23:05:26.179 INFO:teuthology.orchestra.run.smithi174.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-21T23:05:26.180 INFO:teuthology.orchestra.run.smithi174.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-21T23:05:26.183 INFO:teuthology.orchestra.run.smithi174.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-21T23:05:26.184 INFO:teuthology.orchestra.run.smithi174.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-21T23:05:26.243 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-04-21T23:05:26.270 INFO:teuthology.orchestra.run.smithi174.stdout:======================================================================================= 2024-04-21T23:05:26.270 INFO:teuthology.orchestra.run.smithi174.stdout: Package Arch Version Repository Size 2024-04-21T23:05:26.270 INFO:teuthology.orchestra.run.smithi174.stdout:======================================================================================= 2024-04-21T23:05:26.270 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-04-21T23:05:26.270 INFO:teuthology.orchestra.run.smithi174.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-21T23:05:26.270 INFO:teuthology.orchestra.run.smithi174.stdout: ceph x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 6.6 k 2024-04-21T23:05:26.270 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-base x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 5.4 M 2024-04-21T23:05:26.270 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-fuse x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 889 k 2024-04-21T23:05:26.270 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 145 k 2024-04-21T23:05:26.270 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 1.5 M 2024-04-21T23:05:26.271 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 141 k 2024-04-21T23:05:26.271 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 3.5 M 2024-04-21T23:05:26.271 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 7.4 M 2024-04-21T23:05:26.271 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-rook noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 49 k 2024-04-21T23:05:26.271 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-radosgw x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 10 M 2024-04-21T23:05:26.271 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-test x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 48 M 2024-04-21T23:05:26.271 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-volume noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 264 k 2024-04-21T23:05:26.271 INFO:teuthology.orchestra.run.smithi174.stdout: cephadm noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 762 k 2024-04-21T23:05:26.271 INFO:teuthology.orchestra.run.smithi174.stdout: cephfs-mirror x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 222 k 2024-04-21T23:05:26.271 INFO:teuthology.orchestra.run.smithi174.stdout: cephfs-shell x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 27 k 2024-04-21T23:05:26.271 INFO:teuthology.orchestra.run.smithi174.stdout: cephfs-top noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 18 k 2024-04-21T23:05:26.271 INFO:teuthology.orchestra.run.smithi174.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-21T23:05:26.271 INFO:teuthology.orchestra.run.smithi174.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-21T23:05:26.271 INFO:teuthology.orchestra.run.smithi174.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-21T23:05:26.272 INFO:teuthology.orchestra.run.smithi174.stdout: libcephfs-devel x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 32 k 2024-04-21T23:05:26.272 INFO:teuthology.orchestra.run.smithi174.stdout: libcephfs2 x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 745 k 2024-04-21T23:05:26.272 INFO:teuthology.orchestra.run.smithi174.stdout: librados-devel x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 127 k 2024-04-21T23:05:26.272 INFO:teuthology.orchestra.run.smithi174.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-21T23:05:26.272 INFO:teuthology.orchestra.run.smithi174.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-21T23:05:26.272 INFO:teuthology.orchestra.run.smithi174.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-04-21T23:05:26.272 INFO:teuthology.orchestra.run.smithi174.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-21T23:05:26.272 INFO:teuthology.orchestra.run.smithi174.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-21T23:05:26.272 INFO:teuthology.orchestra.run.smithi174.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-21T23:05:26.272 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cephfs x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 161 k 2024-04-21T23:05:26.272 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rados x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 321 k 2024-04-21T23:05:26.272 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rbd x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 298 k 2024-04-21T23:05:26.272 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rgw x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 99 k 2024-04-21T23:05:26.272 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-fuse x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 86 k 2024-04-21T23:05:26.273 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-mirror x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 3.1 M 2024-04-21T23:05:26.273 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-nbd x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 171 k 2024-04-21T23:05:26.273 INFO:teuthology.orchestra.run.smithi174.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-21T23:05:26.273 INFO:teuthology.orchestra.run.smithi174.stdout:Upgrading: 2024-04-21T23:05:26.273 INFO:teuthology.orchestra.run.smithi174.stdout: librados2 x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 3.4 M 2024-04-21T23:05:26.273 INFO:teuthology.orchestra.run.smithi174.stdout: librbd1 x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 3.1 M 2024-04-21T23:05:26.273 INFO:teuthology.orchestra.run.smithi174.stdout:Installing dependencies: 2024-04-21T23:05:26.273 INFO:teuthology.orchestra.run.smithi174.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-21T23:05:26.273 INFO:teuthology.orchestra.run.smithi174.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-21T23:05:26.274 INFO:teuthology.orchestra.run.smithi174.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-21T23:05:26.274 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-common x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 21 M 2024-04-21T23:05:26.274 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 32 k 2024-04-21T23:05:26.274 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mds x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 2.4 M 2024-04-21T23:05:26.274 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 249 k 2024-04-21T23:05:26.274 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mon x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 4.7 M 2024-04-21T23:05:26.274 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-osd x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 17 M 2024-04-21T23:05:26.274 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 15 k 2024-04-21T23:05:26.274 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-selinux x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 25 k 2024-04-21T23:05:26.274 INFO:teuthology.orchestra.run.smithi174.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-21T23:05:26.274 INFO:teuthology.orchestra.run.smithi174.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-21T23:05:26.275 INFO:teuthology.orchestra.run.smithi174.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-21T23:05:26.275 INFO:teuthology.orchestra.run.smithi174.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-21T23:05:26.275 INFO:teuthology.orchestra.run.smithi174.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-21T23:05:26.275 INFO:teuthology.orchestra.run.smithi174.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-21T23:05:26.275 INFO:teuthology.orchestra.run.smithi174.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-21T23:05:26.275 INFO:teuthology.orchestra.run.smithi174.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-21T23:05:26.275 INFO:teuthology.orchestra.run.smithi174.stdout: libcephsqlite x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 164 k 2024-04-21T23:05:26.275 INFO:teuthology.orchestra.run.smithi174.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-21T23:05:26.275 INFO:teuthology.orchestra.run.smithi174.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-21T23:05:26.276 INFO:teuthology.orchestra.run.smithi174.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-21T23:05:26.276 INFO:teuthology.orchestra.run.smithi174.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-21T23:05:26.276 INFO:teuthology.orchestra.run.smithi174.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-21T23:05:26.276 INFO:teuthology.orchestra.run.smithi174.stdout: libradosstriper1 x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 501 k 2024-04-21T23:05:26.276 INFO:teuthology.orchestra.run.smithi174.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-21T23:05:26.276 INFO:teuthology.orchestra.run.smithi174.stdout: librgw2 x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 5.3 M 2024-04-21T23:05:26.276 INFO:teuthology.orchestra.run.smithi174.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-21T23:05:26.276 INFO:teuthology.orchestra.run.smithi174.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-21T23:05:26.277 INFO:teuthology.orchestra.run.smithi174.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-21T23:05:26.277 INFO:teuthology.orchestra.run.smithi174.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-21T23:05:26.277 INFO:teuthology.orchestra.run.smithi174.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-21T23:05:26.277 INFO:teuthology.orchestra.run.smithi174.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-21T23:05:26.277 INFO:teuthology.orchestra.run.smithi174.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-21T23:05:26.277 INFO:teuthology.orchestra.run.smithi174.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-21T23:05:26.277 INFO:teuthology.orchestra.run.smithi174.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-21T23:05:26.277 INFO:teuthology.orchestra.run.smithi174.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-21T23:05:26.277 INFO:teuthology.orchestra.run.smithi174.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-21T23:05:26.278 INFO:teuthology.orchestra.run.smithi174.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-21T23:05:26.278 INFO:teuthology.orchestra.run.smithi174.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-21T23:05:26.278 INFO:teuthology.orchestra.run.smithi174.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-21T23:05:26.278 INFO:teuthology.orchestra.run.smithi174.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-21T23:05:26.278 INFO:teuthology.orchestra.run.smithi174.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-21T23:05:26.278 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-21T23:05:26.278 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ceph-argparse x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 45 k 2024-04-21T23:05:26.279 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ceph-common x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 133 k 2024-04-21T23:05:26.279 INFO:teuthology.orchestra.run.smithi174.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-21T23:05:26.279 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-21T23:05:26.279 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-21T23:05:26.279 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-21T23:05:26.279 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-21T23:05:26.279 INFO:teuthology.orchestra.run.smithi174.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-21T23:05:26.279 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-21T23:05:26.279 INFO:teuthology.orchestra.run.smithi174.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-21T23:05:26.280 INFO:teuthology.orchestra.run.smithi174.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-21T23:05:26.280 INFO:teuthology.orchestra.run.smithi174.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-21T23:05:26.280 INFO:teuthology.orchestra.run.smithi174.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-21T23:05:26.280 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-21T23:05:26.280 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-21T23:05:26.280 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-21T23:05:26.280 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-21T23:05:26.280 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-21T23:05:26.281 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-21T23:05:26.281 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-21T23:05:26.281 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-21T23:05:26.281 INFO:teuthology.orchestra.run.smithi174.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-21T23:05:26.281 INFO:teuthology.orchestra.run.smithi174.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-21T23:05:26.281 INFO:teuthology.orchestra.run.smithi174.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-21T23:05:26.281 INFO:teuthology.orchestra.run.smithi174.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-21T23:05:26.282 INFO:teuthology.orchestra.run.smithi174.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-21T23:05:26.282 INFO:teuthology.orchestra.run.smithi174.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-21T23:05:26.282 INFO:teuthology.orchestra.run.smithi174.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-21T23:05:26.282 INFO:teuthology.orchestra.run.smithi174.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-21T23:05:26.282 INFO:teuthology.orchestra.run.smithi174.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-21T23:05:26.282 INFO:teuthology.orchestra.run.smithi174.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-21T23:05:26.282 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-21T23:05:26.282 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-21T23:05:26.282 INFO:teuthology.orchestra.run.smithi174.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-21T23:05:26.283 INFO:teuthology.orchestra.run.smithi174.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-21T23:05:26.283 INFO:teuthology.orchestra.run.smithi174.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-21T23:05:26.283 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-21T23:05:26.283 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-21T23:05:26.283 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-21T23:05:26.283 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-21T23:05:26.283 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-21T23:05:26.283 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-21T23:05:26.284 INFO:teuthology.orchestra.run.smithi174.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-21T23:05:26.284 INFO:teuthology.orchestra.run.smithi174.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-21T23:05:26.284 INFO:teuthology.orchestra.run.smithi174.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-21T23:05:26.284 INFO:teuthology.orchestra.run.smithi174.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-21T23:05:26.284 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-21T23:05:26.284 INFO:teuthology.orchestra.run.smithi174.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-21T23:05:26.284 INFO:teuthology.orchestra.run.smithi174.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-21T23:05:26.284 INFO:teuthology.orchestra.run.smithi174.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-21T23:05:26.285 INFO:teuthology.orchestra.run.smithi174.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-21T23:05:26.285 INFO:teuthology.orchestra.run.smithi174.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-21T23:05:26.285 INFO:teuthology.orchestra.run.smithi174.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-21T23:05:26.285 INFO:teuthology.orchestra.run.smithi174.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-21T23:05:26.285 INFO:teuthology.orchestra.run.smithi174.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-21T23:05:26.285 INFO:teuthology.orchestra.run.smithi174.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-21T23:05:26.285 INFO:teuthology.orchestra.run.smithi174.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-21T23:05:26.285 INFO:teuthology.orchestra.run.smithi174.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-21T23:05:26.285 INFO:teuthology.orchestra.run.smithi174.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-21T23:05:26.286 INFO:teuthology.orchestra.run.smithi174.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-21T23:05:26.286 INFO:teuthology.orchestra.run.smithi174.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-21T23:05:26.286 INFO:teuthology.orchestra.run.smithi174.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-21T23:05:26.286 INFO:teuthology.orchestra.run.smithi174.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-21T23:05:26.286 INFO:teuthology.orchestra.run.smithi174.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-21T23:05:26.286 INFO:teuthology.orchestra.run.smithi174.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-21T23:05:26.286 INFO:teuthology.orchestra.run.smithi174.stdout:Installing weak dependencies: 2024-04-21T23:05:26.286 INFO:teuthology.orchestra.run.smithi174.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-21T23:05:26.287 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-04-21T23:05:26.287 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-04-21T23:05:26.287 INFO:teuthology.orchestra.run.smithi174.stdout:======================================================================================= 2024-04-21T23:05:26.287 INFO:teuthology.orchestra.run.smithi174.stdout:Install 148 Packages 2024-04-21T23:05:26.287 INFO:teuthology.orchestra.run.smithi174.stdout:Upgrade 2 Packages 2024-04-21T23:05:26.287 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-04-21T23:05:26.288 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 216 M 2024-04-21T23:05:26.288 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-04-21T23:05:26.452 INFO:teuthology.orchestra.run.smithi040.stdout:(8/150): ceph-osd-19.0.0-2405.g820a2f8c.el9.x86 15 MB/s | 17 MB 00:01 2024-04-21T23:05:26.577 INFO:teuthology.orchestra.run.smithi040.stdout:(9/150): ceph-radosgw-19.0.0-2405.g820a2f8c.el9 9.1 MB/s | 10 MB 00:01 2024-04-21T23:05:26.593 INFO:teuthology.orchestra.run.smithi040.stdout:(10/150): ceph-selinux-19.0.0-2405.g820a2f8c.el 178 kB/s | 25 kB 00:00 2024-04-21T23:05:26.652 INFO:teuthology.orchestra.run.smithi040.stdout:(11/150): cephfs-mirror-19.0.0-2405.g820a2f8c.e 3.7 MB/s | 222 kB 00:00 2024-04-21T23:05:26.711 INFO:teuthology.orchestra.run.smithi040.stdout:(12/150): cephfs-shell-19.0.0-2405.g820a2f8c.el 466 kB/s | 27 kB 00:00 2024-04-21T23:05:26.769 INFO:teuthology.orchestra.run.smithi040.stdout:(13/150): libcephfs-devel-19.0.0-2405.g820a2f8c 544 kB/s | 32 kB 00:00 2024-04-21T23:05:26.869 INFO:teuthology.orchestra.run.smithi040.stdout:(14/150): libcephfs2-19.0.0-2405.g820a2f8c.el9. 7.3 MB/s | 745 kB 00:00 2024-04-21T23:05:26.937 INFO:teuthology.orchestra.run.smithi040.stdout:(15/150): libcephsqlite-19.0.0-2405.g820a2f8c.e 2.4 MB/s | 164 kB 00:00 2024-04-21T23:05:27.003 INFO:teuthology.orchestra.run.smithi040.stdout:(16/150): librados-devel-19.0.0-2405.g820a2f8c. 1.9 MB/s | 127 kB 00:00 2024-04-21T23:05:27.087 INFO:teuthology.orchestra.run.smithi040.stdout:(17/150): libradosstriper1-19.0.0-2405.g820a2f8 5.9 MB/s | 501 kB 00:00 2024-04-21T23:05:27.521 INFO:teuthology.orchestra.run.smithi040.stdout:(18/150): librgw2-19.0.0-2405.g820a2f8c.el9.x86 12 MB/s | 5.3 MB 00:00 2024-04-21T23:05:27.579 INFO:teuthology.orchestra.run.smithi040.stdout:(19/150): python3-ceph-argparse-19.0.0-2405.g82 779 kB/s | 45 kB 00:00 2024-04-21T23:05:27.646 INFO:teuthology.orchestra.run.smithi040.stdout:(20/150): python3-ceph-common-19.0.0-2405.g820a 1.9 MB/s | 133 kB 00:00 2024-04-21T23:05:27.656 INFO:teuthology.orchestra.run.smithi174.stdout:(1/150): ceph-19.0.0-2405.g820a2f8c.el9.x86_64. 12 kB/s | 6.6 kB 00:00 2024-04-21T23:05:27.713 INFO:teuthology.orchestra.run.smithi040.stdout:(21/150): python3-cephfs-19.0.0-2405.g820a2f8c. 2.4 MB/s | 161 kB 00:00 2024-04-21T23:05:27.789 INFO:teuthology.orchestra.run.smithi040.stdout:(22/150): python3-rados-19.0.0-2405.g820a2f8c.e 4.2 MB/s | 321 kB 00:00 2024-04-21T23:05:27.815 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 9 - AppStream 837 kB/s | 19 MB 00:23 2024-04-21T23:05:27.872 INFO:teuthology.orchestra.run.smithi040.stdout:(23/150): python3-rbd-19.0.0-2405.g820a2f8c.el9 3.5 MB/s | 298 kB 00:00 2024-04-21T23:05:27.955 INFO:teuthology.orchestra.run.smithi040.stdout:(24/150): python3-rgw-19.0.0-2405.g820a2f8c.el9 1.2 MB/s | 99 kB 00:00 2024-04-21T23:05:28.023 INFO:teuthology.orchestra.run.smithi174.stdout:(2/150): ceph-fuse-19.0.0-2405.g820a2f8c.el9.x8 2.4 MB/s | 889 kB 00:00 2024-04-21T23:05:28.031 INFO:teuthology.orchestra.run.smithi040.stdout:(25/150): rbd-fuse-19.0.0-2405.g820a2f8c.el9.x8 1.1 MB/s | 86 kB 00:00 2024-04-21T23:05:28.090 INFO:teuthology.orchestra.run.smithi174.stdout:(3/150): ceph-immutable-object-cache-19.0.0-240 2.1 MB/s | 145 kB 00:00 2024-04-21T23:05:28.323 INFO:teuthology.orchestra.run.smithi040.stdout:(26/150): ceph-base-19.0.0-2405.g820a2f8c.el9.x 1.2 MB/s | 5.4 MB 00:04 2024-04-21T23:05:28.382 INFO:teuthology.orchestra.run.smithi040.stdout:(27/150): rbd-mirror-19.0.0-2405.g820a2f8c.el9. 8.8 MB/s | 3.1 MB 00:00 2024-04-21T23:05:28.449 INFO:teuthology.orchestra.run.smithi040.stdout:(28/150): ceph-grafana-dashboards-19.0.0-2405.g 473 kB/s | 32 kB 00:00 2024-04-21T23:05:28.516 INFO:teuthology.orchestra.run.smithi040.stdout:(29/150): ceph-mgr-cephadm-19.0.0-2405.g820a2f8 2.1 MB/s | 141 kB 00:00 2024-04-21T23:05:28.542 INFO:teuthology.orchestra.run.smithi040.stdout:(30/150): rbd-nbd-19.0.0-2405.g820a2f8c.el9.x86 783 kB/s | 171 kB 00:00 2024-04-21T23:05:28.682 INFO:teuthology.orchestra.run.smithi174.stdout:(4/150): ceph-mds-19.0.0-2405.g820a2f8c.el9.x86 4.0 MB/s | 2.4 MB 00:00 2024-04-21T23:05:29.207 INFO:teuthology.orchestra.run.smithi174.stdout:(5/150): ceph-mgr-19.0.0-2405.g820a2f8c.el9.x86 2.8 MB/s | 1.5 MB 00:00 2024-04-21T23:05:29.292 INFO:teuthology.orchestra.run.smithi040.stdout:(31/150): ceph-mgr-diskprediction-local-19.0.0- 9.9 MB/s | 7.4 MB 00:00 2024-04-21T23:05:29.375 INFO:teuthology.orchestra.run.smithi040.stdout:(32/150): ceph-mgr-modules-core-19.0.0-2405.g82 2.9 MB/s | 249 kB 00:00 2024-04-21T23:05:29.442 INFO:teuthology.orchestra.run.smithi040.stdout:(33/150): ceph-mgr-rook-19.0.0-2405.g820a2f8c.e 742 kB/s | 49 kB 00:00 2024-04-21T23:05:29.466 INFO:teuthology.orchestra.run.smithi174.stdout:(6/150): ceph-base-19.0.0-2405.g820a2f8c.el9.x8 2.3 MB/s | 5.4 MB 00:02 2024-04-21T23:05:29.509 INFO:teuthology.orchestra.run.smithi040.stdout:(34/150): ceph-prometheus-alerts-19.0.0-2405.g8 230 kB/s | 15 kB 00:00 2024-04-21T23:05:29.593 INFO:teuthology.orchestra.run.smithi040.stdout:(35/150): ceph-volume-19.0.0-2405.g820a2f8c.el9 3.1 MB/s | 264 kB 00:00 2024-04-21T23:05:29.693 INFO:teuthology.orchestra.run.smithi040.stdout:(36/150): cephadm-19.0.0-2405.g820a2f8c.el9.noa 7.4 MB/s | 762 kB 00:00 2024-04-21T23:05:29.760 INFO:teuthology.orchestra.run.smithi040.stdout:(37/150): cephfs-top-19.0.0-2405.g820a2f8c.el9. 274 kB/s | 18 kB 00:00 2024-04-21T23:05:29.966 INFO:teuthology.orchestra.run.smithi174.stdout:(7/150): ceph-mon-19.0.0-2405.g820a2f8c.el9.x86 6.2 MB/s | 4.7 MB 00:00 2024-04-21T23:05:30.227 INFO:teuthology.orchestra.run.smithi040.stdout:(38/150): ceph-test-19.0.0-2405.g820a2f8c.el9.x 13 MB/s | 48 MB 00:03 2024-04-21T23:05:30.252 INFO:teuthology.orchestra.run.smithi040.stdout:(39/150): fuse-2.9.9-15.el9.x86_64.rpm 163 kB/s | 80 kB 00:00 2024-04-21T23:05:30.394 INFO:teuthology.orchestra.run.smithi040.stdout:(40/150): libquadmath-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 195 kB 00:00 2024-04-21T23:05:30.478 INFO:teuthology.orchestra.run.smithi040.stdout:(41/150): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.0 MB/s | 253 kB 00:00 2024-04-21T23:05:30.520 INFO:teuthology.orchestra.run.smithi040.stdout:(42/150): libgfortran-11.4.1-3.el9.x86_64.rpm 2.7 MB/s | 803 kB 00:00 2024-04-21T23:05:30.578 INFO:teuthology.orchestra.run.smithi040.stdout:(43/150): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 106 kB 00:00 2024-04-21T23:05:30.637 INFO:teuthology.orchestra.run.smithi040.stdout:(44/150): python3-pycparser-2.20-6.el9.noarch.r 2.3 MB/s | 135 kB 00:00 2024-04-21T23:05:30.696 INFO:teuthology.orchestra.run.smithi040.stdout:(45/150): python3-requests-2.25.1-8.el9.noarch. 2.1 MB/s | 125 kB 00:00 2024-04-21T23:05:30.788 INFO:teuthology.orchestra.run.smithi040.stdout:(46/150): python3-cryptography-36.0.1-4.el9.x86 4.0 MB/s | 1.2 MB 00:00 2024-04-21T23:05:30.813 INFO:teuthology.orchestra.run.smithi040.stdout:(47/150): python3-urllib3-1.26.5-5.el9.noarch.r 1.8 MB/s | 215 kB 00:00 2024-04-21T23:05:30.880 INFO:teuthology.orchestra.run.smithi040.stdout:(48/150): ceph-mgr-dashboard-19.0.0-2405.g820a2 1.5 MB/s | 3.5 MB 00:02 2024-04-21T23:05:30.933 INFO:teuthology.orchestra.run.smithi174.stdout:(8/150): ceph-radosgw-19.0.0-2405.g820a2f8c.el9 11 MB/s | 10 MB 00:00 2024-04-21T23:05:30.974 INFO:teuthology.orchestra.run.smithi174.stdout:(9/150): ceph-selinux-19.0.0-2405.g820a2f8c.el9 610 kB/s | 25 kB 00:00 2024-04-21T23:05:30.996 INFO:teuthology.orchestra.run.smithi040.stdout:(49/150): automake-1.16.2-8.el9.noarch.rpm 3.2 MB/s | 689 kB 00:00 2024-04-21T23:05:31.047 INFO:teuthology.orchestra.run.smithi040.stdout:(50/150): boost-program-options-1.75.0-8.el9.x8 641 kB/s | 107 kB 00:00 2024-04-21T23:05:31.072 INFO:teuthology.orchestra.run.smithi040.stdout:(51/150): elfutils-libelf-devel-0.190-2.el9.x86 286 kB/s | 21 kB 00:00 2024-04-21T23:05:31.123 INFO:teuthology.orchestra.run.smithi040.stdout:(52/150): flexiblas-3.0.4-8.el9.x86_64.rpm 653 kB/s | 33 kB 00:00 2024-04-21T23:05:31.156 INFO:teuthology.orchestra.run.smithi040.stdout:(53/150): flex-2.6.4-9.el9.x86_64.rpm 2.8 MB/s | 313 kB 00:00 2024-04-21T23:05:31.190 INFO:teuthology.orchestra.run.smithi040.stdout:(54/150): bison-3.7.4-5.el9.x86_64.rpm 2.4 MB/s | 943 kB 00:00 2024-04-21T23:05:31.240 INFO:teuthology.orchestra.run.smithi040.stdout:(55/150): libaio-devel-0.3.111-13.el9.x86_64.rp 223 kB/s | 11 kB 00:00 2024-04-21T23:05:31.265 INFO:teuthology.orchestra.run.smithi040.stdout:(56/150): flexiblas-openblas-openmp-3.0.4-8.el9 168 kB/s | 18 kB 00:00 2024-04-21T23:05:31.324 INFO:teuthology.orchestra.run.smithi040.stdout:(57/150): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.9 MB/s | 160 kB 00:00 2024-04-21T23:05:31.349 INFO:teuthology.orchestra.run.smithi040.stdout:(58/150): librabbitmq-0.11.0-7.el9.x86_64.rpm 540 kB/s | 45 kB 00:00 2024-04-21T23:05:31.525 INFO:teuthology.orchestra.run.smithi040.stdout:(59/150): libtool-2.4.6-45.el9.x86_64.rpm 3.3 MB/s | 582 kB 00:00 2024-04-21T23:05:31.558 INFO:teuthology.orchestra.run.smithi040.stdout:(60/150): librdkafka-1.6.1-102.el9.x86_64.rpm 2.8 MB/s | 662 kB 00:00 2024-04-21T23:05:31.584 INFO:teuthology.orchestra.run.smithi040.stdout:(61/150): libuuid-devel-2.37.4-18.el9.x86_64.rp 435 kB/s | 25 kB 00:00 2024-04-21T23:05:31.609 INFO:teuthology.orchestra.run.smithi040.stdout:(62/150): libzstd-devel-1.5.1-2.el9.x86_64.rpm 946 kB/s | 47 kB 00:00 2024-04-21T23:05:31.692 INFO:teuthology.orchestra.run.smithi040.stdout:(63/150): lua-5.4.4-4.el9.x86_64.rpm 2.2 MB/s | 188 kB 00:00 2024-04-21T23:05:31.718 INFO:teuthology.orchestra.run.smithi040.stdout:(64/150): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.1 MB/s | 292 kB 00:00 2024-04-21T23:05:31.743 INFO:teuthology.orchestra.run.smithi040.stdout:(65/150): openblas-0.3.21-2.el9.x86_64.rpm 702 kB/s | 35 kB 00:00 2024-04-21T23:05:31.826 INFO:teuthology.orchestra.run.smithi040.stdout:(66/150): flexiblas-netlib-3.0.4-8.el9.x86_64.r 4.2 MB/s | 3.0 MB 00:00 2024-04-21T23:05:31.883 INFO:teuthology.orchestra.run.smithi174.stdout:(10/150): ceph-osd-19.0.0-2405.g820a2f8c.el9.x8 7.0 MB/s | 17 MB 00:02 2024-04-21T23:05:31.950 INFO:teuthology.orchestra.run.smithi174.stdout:(11/150): cephfs-mirror-19.0.0-2405.g820a2f8c.e 3.3 MB/s | 222 kB 00:00 2024-04-21T23:05:32.000 INFO:teuthology.orchestra.run.smithi174.stdout:(12/150): cephfs-shell-19.0.0-2405.g820a2f8c.el 543 kB/s | 27 kB 00:00 2024-04-21T23:05:32.050 INFO:teuthology.orchestra.run.smithi174.stdout:(13/150): libcephfs-devel-19.0.0-2405.g820a2f8c 637 kB/s | 32 kB 00:00 2024-04-21T23:05:32.152 INFO:teuthology.orchestra.run.smithi040.stdout:(67/150): postgresql-13.14-1.el9.x86_64.rpm 4.9 MB/s | 1.6 MB 00:00 2024-04-21T23:05:32.184 INFO:teuthology.orchestra.run.smithi174.stdout:(14/150): libcephfs2-19.0.0-2405.g820a2f8c.el9. 5.5 MB/s | 745 kB 00:00 2024-04-21T23:05:32.243 INFO:teuthology.orchestra.run.smithi174.stdout:(15/150): libcephsqlite-19.0.0-2405.g820a2f8c.e 2.8 MB/s | 164 kB 00:00 2024-04-21T23:05:32.318 INFO:teuthology.orchestra.run.smithi174.stdout:(16/150): librados-devel-19.0.0-2405.g820a2f8c. 1.7 MB/s | 127 kB 00:00 2024-04-21T23:05:32.327 INFO:teuthology.orchestra.run.smithi040.stdout:(68/150): postgresql-contrib-13.14-1.el9.x86_64 5.1 MB/s | 911 kB 00:00 2024-04-21T23:05:32.386 INFO:teuthology.orchestra.run.smithi040.stdout:(69/150): postgresql-private-libs-13.14-1.el9.x 2.2 MB/s | 133 kB 00:00 2024-04-21T23:05:32.418 INFO:teuthology.orchestra.run.smithi174.stdout:(17/150): libradosstriper1-19.0.0-2405.g820a2f8 4.9 MB/s | 501 kB 00:00 2024-04-21T23:05:32.619 INFO:teuthology.orchestra.run.smithi040.stdout:(70/150): openssl-devel-3.0.7-27.el9.x86_64.rpm 4.7 MB/s | 4.1 MB 00:00 2024-04-21T23:05:32.702 INFO:teuthology.orchestra.run.smithi174.stdout:(18/150): ceph-common-19.0.0-2405.g820a2f8c.el9 3.8 MB/s | 21 MB 00:05 2024-04-21T23:05:32.760 INFO:teuthology.orchestra.run.smithi174.stdout:(19/150): python3-ceph-argparse-19.0.0-2405.g82 779 kB/s | 45 kB 00:00 2024-04-21T23:05:32.820 INFO:teuthology.orchestra.run.smithi174.stdout:(20/150): python3-ceph-common-19.0.0-2405.g820a 2.2 MB/s | 133 kB 00:00 2024-04-21T23:05:32.844 INFO:teuthology.orchestra.run.smithi040.stdout:(71/150): protobuf-3.14.0-13.el9.x86_64.rpm 4.5 MB/s | 1.0 MB 00:00 2024-04-21T23:05:32.875 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 9 - CRB 4.1 MB/s | 6.2 MB 00:01 2024-04-21T23:05:32.878 INFO:teuthology.orchestra.run.smithi174.stdout:(21/150): python3-cephfs-19.0.0-2405.g820a2f8c. 2.7 MB/s | 161 kB 00:00 2024-04-21T23:05:32.903 INFO:teuthology.orchestra.run.smithi040.stdout:(72/150): python3-attrs-20.3.0-7.el9.noarch.rpm 1.5 MB/s | 87 kB 00:00 2024-04-21T23:05:32.970 INFO:teuthology.orchestra.run.smithi174.stdout:(22/150): python3-rados-19.0.0-2405.g820a2f8c.e 3.4 MB/s | 321 kB 00:00 2024-04-21T23:05:33.087 INFO:teuthology.orchestra.run.smithi040.stdout:(73/150): openblas-openmp-0.3.21-2.el9.x86_64.r 3.5 MB/s | 4.7 MB 00:01 2024-04-21T23:05:33.095 INFO:teuthology.orchestra.run.smithi174.stdout:(23/150): python3-rbd-19.0.0-2405.g820a2f8c.el9 2.3 MB/s | 298 kB 00:00 2024-04-21T23:05:33.171 INFO:teuthology.orchestra.run.smithi174.stdout:(24/150): python3-rgw-19.0.0-2405.g820a2f8c.el9 1.3 MB/s | 99 kB 00:00 2024-04-21T23:05:33.203 INFO:teuthology.orchestra.run.smithi040.stdout:(74/150): python3-devel-3.9.18-3.el9.x86_64.rpm 2.0 MB/s | 244 kB 00:00 2024-04-21T23:05:33.254 INFO:teuthology.orchestra.run.smithi174.stdout:(25/150): rbd-fuse-19.0.0-2405.g820a2f8c.el9.x8 1.0 MB/s | 86 kB 00:00 2024-04-21T23:05:33.321 INFO:teuthology.orchestra.run.smithi040.stdout:(75/150): python3-jinja2-2.11.3-5.el9.noarch.rp 2.1 MB/s | 248 kB 00:00 2024-04-21T23:05:33.387 INFO:teuthology.orchestra.run.smithi040.stdout:(76/150): postgresql-server-13.14-1.el9.x86_64. 5.8 MB/s | 5.8 MB 00:01 2024-04-21T23:05:33.413 INFO:teuthology.orchestra.run.smithi040.stdout:(77/150): python3-jsonpatch-1.21-16.el9.noarch. 279 kB/s | 26 kB 00:00 2024-04-21T23:05:33.430 INFO:teuthology.orchestra.run.smithi174.stdout:(26/150): librgw2-19.0.0-2405.g820a2f8c.el9.x86 5.3 MB/s | 5.3 MB 00:01 2024-04-21T23:05:33.438 INFO:teuthology.orchestra.run.smithi040.stdout:(78/150): python3-jsonpointer-2.0-4.el9.noarch. 382 kB/s | 19 kB 00:00 2024-04-21T23:05:33.488 INFO:teuthology.orchestra.run.smithi040.stdout:(79/150): python3-markupsafe-1.1.1-12.el9.x86_6 695 kB/s | 35 kB 00:00 2024-04-21T23:05:33.513 INFO:teuthology.orchestra.run.smithi040.stdout:(80/150): python3-mako-1.1.4-6.el9.noarch.rpm 1.7 MB/s | 172 kB 00:00 2024-04-21T23:05:33.530 INFO:teuthology.orchestra.run.smithi174.stdout:(27/150): rbd-nbd-19.0.0-2405.g820a2f8c.el9.x86 1.7 MB/s | 171 kB 00:00 2024-04-21T23:05:33.614 INFO:teuthology.orchestra.run.smithi174.stdout:(28/150): ceph-grafana-dashboards-19.0.0-2405.g 381 kB/s | 32 kB 00:00 2024-04-21T23:05:33.630 INFO:teuthology.orchestra.run.smithi040.stdout:(81/150): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.1 MB/s | 250 kB 00:00 2024-04-21T23:05:33.705 INFO:teuthology.orchestra.run.smithi174.stdout:(29/150): ceph-mgr-cephadm-19.0.0-2405.g820a2f8 1.5 MB/s | 141 kB 00:00 2024-04-21T23:05:33.739 INFO:teuthology.orchestra.run.smithi040.stdout:(82/150): python3-oauthlib-3.1.1-5.el9.noarch.r 2.0 MB/s | 222 kB 00:00 2024-04-21T23:05:33.798 INFO:teuthology.orchestra.run.smithi040.stdout:(83/150): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 77 kB 00:00 2024-04-21T23:05:33.915 INFO:teuthology.orchestra.run.smithi040.stdout:(84/150): python3-prettytable-0.7.2-27.el9.noar 362 kB/s | 42 kB 00:00 2024-04-21T23:05:33.990 INFO:teuthology.orchestra.run.smithi040.stdout:(85/150): python3-babel-2.9.1-2.el9.noarch.rpm 5.5 MB/s | 6.0 MB 00:01 2024-04-21T23:05:34.049 INFO:teuthology.orchestra.run.smithi040.stdout:(86/150): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.7 MB/s | 159 kB 00:00 2024-04-21T23:05:34.074 INFO:teuthology.orchestra.run.smithi040.stdout:(87/150): python3-protobuf-3.14.0-13.el9.noarch 1.7 MB/s | 269 kB 00:00 2024-04-21T23:05:34.150 INFO:teuthology.orchestra.run.smithi040.stdout:(88/150): python3-numpy-1.20.1-5.el9.x86_64.rpm 7.7 MB/s | 5.1 MB 00:00 2024-04-21T23:05:34.174 INFO:teuthology.orchestra.run.smithi040.stdout:(89/150): python3-pytz-2021.1-5.el9.noarch.rpm 508 kB/s | 51 kB 00:00 2024-04-21T23:05:34.200 INFO:teuthology.orchestra.run.smithi040.stdout:(90/150): python3-pyasn1-modules-0.4.8-6.el9.no 1.8 MB/s | 279 kB 00:00 2024-04-21T23:05:34.225 INFO:teuthology.orchestra.run.smithi040.stdout:(91/150): python3-requests-oauthlib-1.3.0-12.el 711 kB/s | 54 kB 00:00 2024-04-21T23:05:34.250 INFO:teuthology.orchestra.run.smithi040.stdout:(92/150): python3-toml-0.10.2-6.el9.noarch.rpm 831 kB/s | 42 kB 00:00 2024-04-21T23:05:34.276 INFO:teuthology.orchestra.run.smithi040.stdout:(93/150): python3-wcwidth-0.2.5-8.el9.noarch.rp 880 kB/s | 44 kB 00:00 2024-04-21T23:05:34.301 INFO:teuthology.orchestra.run.smithi040.stdout:(94/150): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 221 kB 00:00 2024-04-21T23:05:34.326 INFO:teuthology.orchestra.run.smithi040.stdout:(95/150): qatlib-service-24.02.0-1.el9.x86_64.r 714 kB/s | 36 kB 00:00 2024-04-21T23:05:34.352 INFO:teuthology.orchestra.run.smithi040.stdout:(96/150): qatzip-libs-1.2.0-1.el9.x86_64.rpm 948 kB/s | 47 kB 00:00 2024-04-21T23:05:34.385 INFO:teuthology.orchestra.run.smithi040.stdout:(97/150): socat-1.7.4.1-5.el9.x86_64.rpm 5.1 MB/s | 305 kB 00:00 2024-04-21T23:05:34.410 INFO:teuthology.orchestra.run.smithi040.stdout:(98/150): uuid-1.6.2-55.el9.x86_64.rpm 986 kB/s | 58 kB 00:00 2024-04-21T23:05:34.436 INFO:teuthology.orchestra.run.smithi040.stdout:(99/150): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.8 MB/s | 94 kB 00:00 2024-04-21T23:05:34.461 INFO:teuthology.orchestra.run.smithi040.stdout:(100/150): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-04-21T23:05:34.486 INFO:teuthology.orchestra.run.smithi040.stdout:(101/150): zlib-devel-1.2.11-41.el9.x86_64.rpm 893 kB/s | 45 kB 00:00 2024-04-21T23:05:34.489 INFO:teuthology.orchestra.run.smithi174.stdout:(30/150): rbd-mirror-19.0.0-2405.g820a2f8c.el9. 2.5 MB/s | 3.1 MB 00:01 2024-04-21T23:05:34.600 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 9 - Extras packages 28 kB/s | 16 kB 00:00 2024-04-21T23:05:34.670 INFO:teuthology.orchestra.run.smithi040.stdout:(102/150): lua-devel-5.4.4-4.el9.x86_64.rpm 107 kB/s | 22 kB 00:00 2024-04-21T23:05:34.681 INFO:teuthology.orchestra.run.smithi174.stdout:(31/150): ceph-mgr-dashboard-19.0.0-2405.g820a2 3.6 MB/s | 3.5 MB 00:00 2024-04-21T23:05:34.789 INFO:teuthology.orchestra.run.smithi174.stdout:(32/150): ceph-mgr-modules-core-19.0.0-2405.g82 2.2 MB/s | 249 kB 00:00 2024-04-21T23:05:34.898 INFO:teuthology.orchestra.run.smithi174.stdout:(33/150): ceph-mgr-rook-19.0.0-2405.g820a2f8c.e 454 kB/s | 49 kB 00:00 2024-04-21T23:05:34.945 INFO:teuthology.orchestra.run.smithi040.stdout:(103/150): protobuf-compiler-3.14.0-13.el9.x86_ 1.8 MB/s | 863 kB 00:00 2024-04-21T23:05:34.998 INFO:teuthology.orchestra.run.smithi174.stdout:(34/150): ceph-prometheus-alerts-19.0.0-2405.g8 153 kB/s | 15 kB 00:00 2024-04-21T23:05:35.174 INFO:teuthology.orchestra.run.smithi174.stdout:(35/150): ceph-volume-19.0.0-2405.g820a2f8c.el9 1.5 MB/s | 264 kB 00:00 2024-04-21T23:05:35.195 INFO:teuthology.orchestra.run.smithi040.stdout:(104/150): abseil-cpp-20211102.0-3.el9.x86_64.r 1.0 MB/s | 548 kB 00:00 2024-04-21T23:05:35.296 INFO:teuthology.orchestra.run.smithi040.stdout:(105/150): grpc-data-1.46.7-10.el9.noarch.rpm 194 kB/s | 19 kB 00:00 2024-04-21T23:05:35.349 INFO:teuthology.orchestra.run.smithi174.stdout:(36/150): cephadm-19.0.0-2405.g820a2f8c.el9.noa 4.3 MB/s | 762 kB 00:00 2024-04-21T23:05:35.394 INFO:teuthology.orchestra.run.smithi111.stdout:Extra Packages for Enterprise Linux 31 MB/s | 21 MB 00:00 2024-04-21T23:05:35.396 INFO:teuthology.orchestra.run.smithi040.stdout:(106/150): gperftools-libs-2.9.1-2.el9.x86_64.r 686 kB/s | 309 kB 00:00 2024-04-21T23:05:35.441 INFO:teuthology.orchestra.run.smithi174.stdout:(37/150): cephfs-top-19.0.0-2405.g820a2f8c.el9. 199 kB/s | 18 kB 00:00 2024-04-21T23:05:35.496 INFO:teuthology.orchestra.run.smithi040.stdout:(107/150): libarrow-doc-9.0.0-10.el9.noarch.rpm 260 kB/s | 26 kB 00:00 2024-04-21T23:05:35.549 INFO:teuthology.orchestra.run.smithi174.stdout:(38/150): fuse-2.9.9-15.el9.x86_64.rpm 744 kB/s | 80 kB 00:00 2024-04-21T23:05:35.588 INFO:teuthology.orchestra.run.smithi040.stdout:(108/150): liboath-2.6.7-2.el9.x86_64.rpm 530 kB/s | 49 kB 00:00 2024-04-21T23:05:35.655 INFO:teuthology.orchestra.run.smithi040.stdout:(109/150): libarrow-9.0.0-10.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-04-21T23:05:35.658 INFO:teuthology.orchestra.run.smithi174.stdout:(39/150): libgfortran-11.4.1-3.el9.x86_64.rpm 7.2 MB/s | 803 kB 00:00 2024-04-21T23:05:35.680 INFO:teuthology.orchestra.run.smithi040.stdout:(110/150): libunwind-1.6.2-1.el9.x86_64.rpm 732 kB/s | 67 kB 00:00 2024-04-21T23:05:35.708 INFO:teuthology.orchestra.run.smithi174.stdout:(40/150): libquadmath-11.4.1-3.el9.x86_64.rpm 3.8 MB/s | 195 kB 00:00 2024-04-21T23:05:35.756 INFO:teuthology.orchestra.run.smithi040.stdout:(111/150): luarocks-3.9.2-1.el9.noarch.rpm 1.5 MB/s | 151 kB 00:00 2024-04-21T23:05:35.758 INFO:teuthology.orchestra.run.smithi174.stdout:(41/150): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.0 MB/s | 253 kB 00:00 2024-04-21T23:05:35.864 INFO:teuthology.orchestra.run.smithi040.stdout:(112/150): python3-asyncssh-2.13.2-5.el9.noarch 5.0 MB/s | 548 kB 00:00 2024-04-21T23:05:35.876 INFO:teuthology.orchestra.run.smithi174.stdout:(42/150): python3-cryptography-36.0.1-4.el9.x86 11 MB/s | 1.2 MB 00:00 2024-04-21T23:05:35.923 INFO:teuthology.orchestra.run.smithi040.stdout:(113/150): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.4 MB/s | 838 kB 00:00 2024-04-21T23:05:35.926 INFO:teuthology.orchestra.run.smithi174.stdout:(43/150): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-04-21T23:05:35.948 INFO:teuthology.orchestra.run.smithi040.stdout:(114/150): python3-bcrypt-3.2.2-1.el9.x86_64.rp 520 kB/s | 43 kB 00:00 2024-04-21T23:05:35.976 INFO:teuthology.orchestra.run.smithi174.stdout:(44/150): python3-pycparser-2.20-6.el9.noarch.r 2.6 MB/s | 135 kB 00:00 2024-04-21T23:05:36.007 INFO:teuthology.orchestra.run.smithi040.stdout:(115/150): python3-cachetools-4.2.4-1.el9.noarc 386 kB/s | 32 kB 00:00 2024-04-21T23:05:36.026 INFO:teuthology.orchestra.run.smithi174.stdout:(45/150): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-04-21T23:05:36.042 INFO:teuthology.orchestra.run.smithi040.stdout:(116/150): python3-certifi-2021.10.8-2.el9.noar 159 kB/s | 15 kB 00:00 2024-04-21T23:05:36.077 INFO:teuthology.orchestra.run.smithi174.stdout:(46/150): python3-urllib3-1.26.5-5.el9.noarch.r 4.2 MB/s | 215 kB 00:00 2024-04-21T23:05:36.101 INFO:teuthology.orchestra.run.smithi040.stdout:(117/150): python3-cheroot-8.6.0-4.el9.noarch.r 1.8 MB/s | 172 kB 00:00 2024-04-21T23:05:36.143 INFO:teuthology.orchestra.run.smithi040.stdout:(118/150): python3-cherrypy-18.6.1-2.el9.noarch 3.5 MB/s | 358 kB 00:00 2024-04-21T23:05:36.202 INFO:teuthology.orchestra.run.smithi040.stdout:(119/150): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.0 MB/s | 303 kB 00:00 2024-04-21T23:05:36.235 INFO:teuthology.orchestra.run.smithi040.stdout:(120/150): python3-colorama-0.4.4-12.el9.noarch 385 kB/s | 35 kB 00:00 2024-04-21T23:05:36.294 INFO:teuthology.orchestra.run.smithi040.stdout:(121/150): python3-google-auth-2.28.2-1.el9.noa 2.2 MB/s | 203 kB 00:00 2024-04-21T23:05:36.369 INFO:teuthology.orchestra.run.smithi174.stdout:(47/150): automake-1.16.2-8.el9.noarch.rpm 2.3 MB/s | 689 kB 00:00 2024-04-21T23:05:36.394 INFO:teuthology.orchestra.run.smithi040.stdout:(122/150): python3-grpcio-tools-1.46.7-10.el9.x 1.4 MB/s | 144 kB 00:00 2024-04-21T23:05:36.520 INFO:teuthology.orchestra.run.smithi040.stdout:(123/150): python3-jaraco-8.2.1-3.el9.noarch.rp 85 kB/s | 11 kB 00:00 2024-04-21T23:05:36.712 INFO:teuthology.orchestra.run.smithi040.stdout:(124/150): python3-jaraco-classes-3.2.1-5.el9.n 92 kB/s | 18 kB 00:00 2024-04-21T23:05:36.761 INFO:teuthology.orchestra.run.smithi174.stdout:(48/150): bison-3.7.4-5.el9.x86_64.rpm 2.3 MB/s | 943 kB 00:00 2024-04-21T23:05:36.812 INFO:teuthology.orchestra.run.smithi040.stdout:(125/150): python3-jaraco-collections-3.0.0-8.e 231 kB/s | 23 kB 00:00 2024-04-21T23:05:36.844 INFO:teuthology.orchestra.run.smithi174.stdout:(49/150): boost-program-options-1.75.0-8.el9.x8 1.3 MB/s | 107 kB 00:00 2024-04-21T23:05:36.886 INFO:teuthology.orchestra.run.smithi174.stdout:(50/150): elfutils-libelf-devel-0.190-2.el9.x86 516 kB/s | 21 kB 00:00 2024-04-21T23:05:36.946 INFO:teuthology.orchestra.run.smithi040.stdout:(126/150): python3-jaraco-functools-3.5.0-2.el9 146 kB/s | 19 kB 00:00 2024-04-21T23:05:36.995 INFO:teuthology.orchestra.run.smithi174.stdout:(51/150): flex-2.6.4-9.el9.x86_64.rpm 2.8 MB/s | 313 kB 00:00 2024-04-21T23:05:37.037 INFO:teuthology.orchestra.run.smithi174.stdout:(52/150): flexiblas-3.0.4-8.el9.x86_64.rpm 787 kB/s | 33 kB 00:00 2024-04-21T23:05:37.046 INFO:teuthology.orchestra.run.smithi040.stdout:(127/150): python3-jaraco-text-3.2.0-6.el9.noar 196 kB/s | 20 kB 00:00 2024-04-21T23:05:37.163 INFO:teuthology.orchestra.run.smithi040.stdout:(128/150): python3-kubernetes-26.1.0-2.el9.noar 8.8 MB/s | 1.0 MB 00:00 2024-04-21T23:05:37.263 INFO:teuthology.orchestra.run.smithi040.stdout:(129/150): python3-logutils-0.3.5-21.el9.noarch 463 kB/s | 46 kB 00:00 2024-04-21T23:05:37.313 INFO:teuthology.orchestra.run.smithi040.stdout:(130/150): python3-grpcio-1.46.7-10.el9.x86_64. 1.9 MB/s | 2.0 MB 00:01 2024-04-21T23:05:37.321 INFO:teuthology.orchestra.run.smithi174.stdout:(53/150): ceph-mgr-diskprediction-local-19.0.0- 2.6 MB/s | 7.4 MB 00:02 2024-04-21T23:05:37.355 INFO:teuthology.orchestra.run.smithi040.stdout:(131/150): python3-more-itertools-8.12.0-2.el9. 858 kB/s | 79 kB 00:00 2024-04-21T23:05:37.404 INFO:teuthology.orchestra.run.smithi174.stdout:(54/150): flexiblas-openblas-openmp-3.0.4-8.el9 219 kB/s | 18 kB 00:00 2024-04-21T23:05:37.406 INFO:teuthology.orchestra.run.smithi040.stdout:(132/150): python3-natsort-7.1.1-5.el9.noarch.r 628 kB/s | 58 kB 00:00 2024-04-21T23:05:37.454 INFO:teuthology.orchestra.run.smithi174.stdout:(55/150): libaio-devel-0.3.111-13.el9.x86_64.rp 223 kB/s | 11 kB 00:00 2024-04-21T23:05:37.456 INFO:teuthology.orchestra.run.smithi040.stdout:(133/150): python3-pecan-1.4.2-3.el9.noarch.rpm 2.7 MB/s | 272 kB 00:00 2024-04-21T23:05:37.498 INFO:teuthology.orchestra.run.smithi040.stdout:(134/150): python3-portend-3.1.0-2.el9.noarch.r 179 kB/s | 16 kB 00:00 2024-04-21T23:05:37.548 INFO:teuthology.orchestra.run.smithi040.stdout:(135/150): python3-pyOpenSSL-21.0.0-1.el9.noarc 978 kB/s | 90 kB 00:00 2024-04-21T23:05:37.580 INFO:teuthology.orchestra.run.smithi174.stdout:(56/150): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.3 MB/s | 160 kB 00:00 2024-04-21T23:05:37.607 INFO:teuthology.orchestra.run.smithi040.stdout:(136/150): python3-pyperclip-1.8.2-2.el9.noarch 259 kB/s | 28 kB 00:00 2024-04-21T23:05:37.638 INFO:teuthology.orchestra.run.smithi174.stdout:(57/150): librabbitmq-0.11.0-7.el9.x86_64.rpm 779 kB/s | 45 kB 00:00 2024-04-21T23:05:37.640 INFO:teuthology.orchestra.run.smithi040.stdout:(137/150): python3-repoze-lru-0.7-16.el9.noarch 335 kB/s | 31 kB 00:00 2024-04-21T23:05:37.707 INFO:teuthology.orchestra.run.smithi040.stdout:(138/150): python3-routes-2.5.1-5.el9.noarch.rp 1.8 MB/s | 188 kB 00:00 2024-04-21T23:05:37.733 INFO:teuthology.orchestra.run.smithi040.stdout:(139/150): python3-rsa-4.9-2.el9.noarch.rpm 643 kB/s | 59 kB 00:00 2024-04-21T23:05:37.800 INFO:teuthology.orchestra.run.smithi040.stdout:(140/150): python3-tempora-5.0.0-2.el9.noarch.r 389 kB/s | 36 kB 00:00 2024-04-21T23:05:37.820 INFO:teuthology.orchestra.run.smithi174.stdout:(58/150): flexiblas-netlib-3.0.4-8.el9.x86_64.r 3.8 MB/s | 3.0 MB 00:00 2024-04-21T23:05:37.825 INFO:teuthology.orchestra.run.smithi040.stdout:(141/150): python3-typing-extensions-4.4.0-2.el 555 kB/s | 51 kB 00:00 2024-04-21T23:05:37.856 INFO:teuthology.orchestra.run.smithi174.stdout:(59/150): librdkafka-1.6.1-102.el9.x86_64.rpm 3.0 MB/s | 662 kB 00:00 2024-04-21T23:05:37.900 INFO:teuthology.orchestra.run.smithi040.stdout:(142/150): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-04-21T23:05:37.925 INFO:teuthology.orchestra.run.smithi040.stdout:(143/150): python3-websocket-client-1.2.3-2.el9 895 kB/s | 90 kB 00:00 2024-04-21T23:05:38.009 INFO:teuthology.orchestra.run.smithi040.stdout:(144/150): python3-werkzeug-2.0.3-3.el9.1.noarc 3.8 MB/s | 427 kB 00:00 2024-04-21T23:05:38.035 INFO:teuthology.orchestra.run.smithi040.stdout:(145/150): python3-zc-lockfile-2.0-10.el9.noarc 183 kB/s | 20 kB 00:00 2024-04-21T23:05:38.110 INFO:teuthology.orchestra.run.smithi040.stdout:(146/150): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2024-04-21T23:05:38.289 INFO:teuthology.orchestra.run.smithi174.stdout:(60/150): ceph-test-19.0.0-2405.g820a2f8c.el9.x 6.6 MB/s | 48 MB 00:07 2024-04-21T23:05:38.315 INFO:teuthology.orchestra.run.smithi174.stdout:(61/150): libuuid-devel-2.37.4-18.el9.x86_64.rp 55 kB/s | 25 kB 00:00 2024-04-21T23:05:38.340 INFO:teuthology.orchestra.run.smithi174.stdout:(62/150): libtool-2.4.6-45.el9.x86_64.rpm 1.1 MB/s | 582 kB 00:00 2024-04-21T23:05:38.432 INFO:teuthology.orchestra.run.smithi174.stdout:(63/150): libzstd-devel-1.5.1-2.el9.x86_64.rpm 334 kB/s | 47 kB 00:00 2024-04-21T23:05:38.457 INFO:teuthology.orchestra.run.smithi174.stdout:(64/150): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 188 kB 00:00 2024-04-21T23:05:38.482 INFO:teuthology.orchestra.run.smithi174.stdout:(65/150): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.7 MB/s | 292 kB 00:00 2024-04-21T23:05:38.508 INFO:teuthology.orchestra.run.smithi174.stdout:(66/150): openblas-0.3.21-2.el9.x86_64.rpm 467 kB/s | 35 kB 00:00 2024-04-21T23:05:38.677 INFO:teuthology.orchestra.run.smithi040.stdout:(147/150): librados2-19.0.0-2405.g820a2f8c.el9. 6.0 MB/s | 3.4 MB 00:00 2024-04-21T23:05:38.844 INFO:teuthology.orchestra.run.smithi040.stdout:(148/150): thrift-0.15.0-2.el9.x86_64.rpm 2.0 MB/s | 1.6 MB 00:00 2024-04-21T23:05:38.875 INFO:teuthology.orchestra.run.smithi174.stdout:(67/150): postgresql-13.14-1.el9.x86_64.rpm 4.3 MB/s | 1.6 MB 00:00 2024-04-21T23:05:39.083 INFO:teuthology.orchestra.run.smithi174.stdout:(68/150): postgresql-contrib-13.14-1.el9.x86_64 4.3 MB/s | 911 kB 00:00 2024-04-21T23:05:39.142 INFO:teuthology.orchestra.run.smithi174.stdout:(69/150): postgresql-private-libs-13.14-1.el9.x 2.2 MB/s | 133 kB 00:00 2024-04-21T23:05:39.211 INFO:teuthology.orchestra.run.smithi040.stdout:(149/150): librbd1-19.0.0-2405.g820a2f8c.el9.x8 5.9 MB/s | 3.1 MB 00:00 2024-04-21T23:05:39.350 INFO:teuthology.orchestra.run.smithi174.stdout:(70/150): openssl-devel-3.0.7-27.el9.x86_64.rpm 4.8 MB/s | 4.1 MB 00:00 2024-04-21T23:05:39.493 INFO:teuthology.orchestra.run.smithi174.stdout:(71/150): openblas-openmp-0.3.21-2.el9.x86_64.r 4.6 MB/s | 4.7 MB 00:01 2024-04-21T23:05:39.542 INFO:teuthology.orchestra.run.smithi174.stdout:(72/150): python3-attrs-20.3.0-7.el9.noarch.rpm 1.7 MB/s | 87 kB 00:00 2024-04-21T23:05:39.601 INFO:teuthology.orchestra.run.smithi174.stdout:(73/150): protobuf-3.14.0-13.el9.x86_64.rpm 4.0 MB/s | 1.0 MB 00:00 2024-04-21T23:05:39.678 INFO:teuthology.orchestra.run.smithi040.stdout:(150/150): python3-scipy-1.6.2-8.el9.x86_64.rpm 3.0 MB/s | 16 MB 00:05 2024-04-21T23:05:39.681 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-04-21T23:05:39.681 INFO:teuthology.orchestra.run.smithi040.stdout:Total 13 MB/s | 216 MB 00:17 2024-04-21T23:05:39.684 INFO:teuthology.orchestra.run.smithi174.stdout:(74/150): python3-devel-3.9.18-3.el9.x86_64.rpm 2.9 MB/s | 244 kB 00:00 2024-04-21T23:05:39.768 INFO:teuthology.orchestra.run.smithi174.stdout:(75/150): python3-jinja2-2.11.3-5.el9.noarch.rp 2.9 MB/s | 248 kB 00:00 2024-04-21T23:05:39.827 INFO:teuthology.orchestra.run.smithi174.stdout:(76/150): python3-jsonpatch-1.21-16.el9.noarch. 441 kB/s | 26 kB 00:00 2024-04-21T23:05:39.877 INFO:teuthology.orchestra.run.smithi174.stdout:(77/150): python3-jsonpointer-2.0-4.el9.noarch. 383 kB/s | 19 kB 00:00 2024-04-21T23:05:39.944 INFO:teuthology.orchestra.run.smithi174.stdout:(78/150): python3-mako-1.1.4-6.el9.noarch.rpm 2.5 MB/s | 172 kB 00:00 2024-04-21T23:05:39.994 INFO:teuthology.orchestra.run.smithi174.stdout:(79/150): python3-markupsafe-1.1.1-12.el9.x86_6 696 kB/s | 35 kB 00:00 2024-04-21T23:05:40.220 INFO:teuthology.orchestra.run.smithi174.stdout:(80/150): postgresql-server-13.14-1.el9.x86_64. 5.4 MB/s | 5.8 MB 00:01 2024-04-21T23:05:40.312 INFO:teuthology.orchestra.run.smithi174.stdout:(81/150): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.7 MB/s | 250 kB 00:00 2024-04-21T23:05:40.370 INFO:teuthology.orchestra.run.smithi174.stdout:(82/150): python3-oauthlib-3.1.1-5.el9.noarch.r 3.7 MB/s | 222 kB 00:00 2024-04-21T23:05:40.421 INFO:teuthology.orchestra.run.smithi174.stdout:(83/150): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-04-21T23:05:40.443 INFO:teuthology.orchestra.run.smithi111.stdout:lab-extras 21 kB/s | 1.7 kB 00:00 2024-04-21T23:05:40.479 INFO:teuthology.orchestra.run.smithi174.stdout:(84/150): python3-prettytable-0.7.2-27.el9.noar 726 kB/s | 42 kB 00:00 2024-04-21T23:05:40.546 INFO:teuthology.orchestra.run.smithi174.stdout:(85/150): python3-babel-2.9.1-2.el9.noarch.rpm 5.9 MB/s | 6.0 MB 00:01 2024-04-21T23:05:40.605 INFO:teuthology.orchestra.run.smithi174.stdout:(86/150): python3-protobuf-3.14.0-13.el9.noarch 2.1 MB/s | 269 kB 00:00 2024-04-21T23:05:40.630 INFO:teuthology.orchestra.run.smithi174.stdout:(87/150): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 159 kB 00:00 2024-04-21T23:05:40.697 INFO:teuthology.orchestra.run.smithi174.stdout:(88/150): python3-numpy-1.20.1-5.el9.x86_64.rpm 7.3 MB/s | 5.1 MB 00:00 2024-04-21T23:05:40.722 INFO:teuthology.orchestra.run.smithi174.stdout:(89/150): python3-pytz-2021.1-5.el9.noarch.rpm 555 kB/s | 51 kB 00:00 2024-04-21T23:05:40.747 INFO:teuthology.orchestra.run.smithi174.stdout:(90/150): python3-pyasn1-modules-0.4.8-6.el9.no 1.9 MB/s | 279 kB 00:00 2024-04-21T23:05:40.775 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-04-21T23:05:40.798 INFO:teuthology.orchestra.run.smithi174.stdout:(91/150): python3-toml-0.10.2-6.el9.noarch.rpm 834 kB/s | 42 kB 00:00 2024-04-21T23:05:40.848 INFO:teuthology.orchestra.run.smithi174.stdout:(92/150): python3-wcwidth-0.2.5-8.el9.noarch.rp 885 kB/s | 44 kB 00:00 2024-04-21T23:05:40.873 INFO:teuthology.orchestra.run.smithi174.stdout:(93/150): python3-requests-oauthlib-1.3.0-12.el 305 kB/s | 54 kB 00:00 2024-04-21T23:05:40.914 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-04-21T23:05:40.915 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-04-21T23:05:40.932 INFO:teuthology.orchestra.run.smithi174.stdout:(94/150): qatlib-service-24.02.0-1.el9.x86_64.r 617 kB/s | 36 kB 00:00 2024-04-21T23:05:40.957 INFO:teuthology.orchestra.run.smithi174.stdout:(95/150): qatlib-24.02.0-1.el9.x86_64.rpm 2.0 MB/s | 221 kB 00:00 2024-04-21T23:05:40.991 INFO:teuthology.orchestra.run.smithi174.stdout:(96/150): qatzip-libs-1.2.0-1.el9.x86_64.rpm 809 kB/s | 47 kB 00:00 2024-04-21T23:05:41.049 INFO:teuthology.orchestra.run.smithi174.stdout:(97/150): uuid-1.6.2-55.el9.x86_64.rpm 992 kB/s | 58 kB 00:00 2024-04-21T23:05:41.108 INFO:teuthology.orchestra.run.smithi174.stdout:(98/150): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.6 MB/s | 94 kB 00:00 2024-04-21T23:05:41.166 INFO:teuthology.orchestra.run.smithi174.stdout:(99/150): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-04-21T23:05:41.192 INFO:teuthology.orchestra.run.smithi174.stdout:(100/150): socat-1.7.4.1-5.el9.x86_64.rpm 1.3 MB/s | 305 kB 00:00 2024-04-21T23:05:41.225 INFO:teuthology.orchestra.run.smithi174.stdout:(101/150): zlib-devel-1.2.11-41.el9.x86_64.rpm 765 kB/s | 45 kB 00:00 2024-04-21T23:05:41.292 INFO:teuthology.orchestra.run.smithi174.stdout:(102/150): lua-devel-5.4.4-4.el9.x86_64.rpm 222 kB/s | 22 kB 00:00 2024-04-21T23:05:41.459 INFO:teuthology.orchestra.run.smithi174.stdout:(103/150): protobuf-compiler-3.14.0-13.el9.x86_ 3.6 MB/s | 863 kB 00:00 2024-04-21T23:05:41.518 INFO:teuthology.orchestra.run.smithi174.stdout:(104/150): abseil-cpp-20211102.0-3.el9.x86_64.r 2.4 MB/s | 548 kB 00:00 2024-04-21T23:05:41.560 INFO:teuthology.orchestra.run.smithi174.stdout:(105/150): grpc-data-1.46.7-10.el9.noarch.rpm 467 kB/s | 19 kB 00:00 2024-04-21T23:05:41.744 INFO:teuthology.orchestra.run.smithi174.stdout:(106/150): gperftools-libs-2.9.1-2.el9.x86_64.r 1.1 MB/s | 309 kB 00:00 2024-04-21T23:05:41.786 INFO:teuthology.orchestra.run.smithi174.stdout:(107/150): libarrow-doc-9.0.0-10.el9.noarch.rpm 622 kB/s | 26 kB 00:00 2024-04-21T23:05:41.828 INFO:teuthology.orchestra.run.smithi174.stdout:(108/150): liboath-2.6.7-2.el9.x86_64.rpm 1.1 MB/s | 49 kB 00:00 2024-04-21T23:05:41.886 INFO:teuthology.orchestra.run.smithi174.stdout:(109/150): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-04-21T23:05:41.953 INFO:teuthology.orchestra.run.smithi174.stdout:(110/150): luarocks-3.9.2-1.el9.noarch.rpm 2.2 MB/s | 151 kB 00:00 2024-04-21T23:05:42.145 INFO:teuthology.orchestra.run.smithi174.stdout:(111/150): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.3 MB/s | 838 kB 00:00 2024-04-21T23:05:42.279 INFO:teuthology.orchestra.run.smithi174.stdout:(112/150): python3-asyncssh-2.13.2-5.el9.noarch 4.0 MB/s | 548 kB 00:00 2024-04-21T23:05:42.321 INFO:teuthology.orchestra.run.smithi174.stdout:(113/150): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.0 MB/s | 43 kB 00:00 2024-04-21T23:05:42.363 INFO:teuthology.orchestra.run.smithi174.stdout:(114/150): python3-cachetools-4.2.4-1.el9.noarc 772 kB/s | 32 kB 00:00 2024-04-21T23:05:42.405 INFO:teuthology.orchestra.run.smithi174.stdout:(115/150): python3-certifi-2021.10.8-2.el9.noar 359 kB/s | 15 kB 00:00 2024-04-21T23:05:42.488 INFO:teuthology.orchestra.run.smithi174.stdout:(116/150): libarrow-9.0.0-10.el9.x86_64.rpm 4.8 MB/s | 4.4 MB 00:00 2024-04-21T23:05:42.514 INFO:teuthology.orchestra.run.smithi174.stdout:(117/150): python3-cheroot-8.6.0-4.el9.noarch.r 1.5 MB/s | 172 kB 00:00 2024-04-21T23:05:42.572 INFO:teuthology.orchestra.run.smithi174.stdout:(118/150): python3-cherrypy-18.6.1-2.el9.noarch 4.2 MB/s | 358 kB 00:00 2024-04-21T23:05:42.611 INFO:teuthology.orchestra.run.smithi111.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-21T23:05:42.612 INFO:teuthology.orchestra.run.smithi111.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-21T23:05:42.614 INFO:teuthology.orchestra.run.smithi174.stdout:(119/150): python3-colorama-0.4.4-12.el9.noarch 845 kB/s | 35 kB 00:00 2024-04-21T23:05:42.622 INFO:teuthology.orchestra.run.smithi111.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-21T23:05:42.623 INFO:teuthology.orchestra.run.smithi111.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-21T23:05:42.624 INFO:teuthology.orchestra.run.smithi111.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-21T23:05:42.625 INFO:teuthology.orchestra.run.smithi111.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-21T23:05:42.626 INFO:teuthology.orchestra.run.smithi111.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-21T23:05:42.629 INFO:teuthology.orchestra.run.smithi111.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-21T23:05:42.630 INFO:teuthology.orchestra.run.smithi111.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-21T23:05:42.639 INFO:teuthology.orchestra.run.smithi174.stdout:(120/150): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.4 MB/s | 303 kB 00:00 2024-04-21T23:05:42.681 INFO:teuthology.orchestra.run.smithi174.stdout:(121/150): python3-google-auth-2.28.2-1.el9.noa 3.0 MB/s | 203 kB 00:00 2024-04-21T23:05:42.689 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-04-21T23:05:42.699 INFO:teuthology.orchestra.run.smithi111.stdout:======================================================================================= 2024-04-21T23:05:42.699 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2024-04-21T23:05:42.699 INFO:teuthology.orchestra.run.smithi111.stdout:======================================================================================= 2024-04-21T23:05:42.699 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-04-21T23:05:42.699 INFO:teuthology.orchestra.run.smithi111.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-21T23:05:42.699 INFO:teuthology.orchestra.run.smithi111.stdout: ceph x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 6.6 k 2024-04-21T23:05:42.699 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-base x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 5.4 M 2024-04-21T23:05:42.699 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-fuse x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 889 k 2024-04-21T23:05:42.699 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 145 k 2024-04-21T23:05:42.699 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 1.5 M 2024-04-21T23:05:42.700 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 141 k 2024-04-21T23:05:42.700 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 3.5 M 2024-04-21T23:05:42.700 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 7.4 M 2024-04-21T23:05:42.700 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-rook noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 49 k 2024-04-21T23:05:42.700 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-radosgw x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 10 M 2024-04-21T23:05:42.700 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-test x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 48 M 2024-04-21T23:05:42.700 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-volume noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 264 k 2024-04-21T23:05:42.700 INFO:teuthology.orchestra.run.smithi111.stdout: cephadm noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 762 k 2024-04-21T23:05:42.700 INFO:teuthology.orchestra.run.smithi111.stdout: cephfs-mirror x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 222 k 2024-04-21T23:05:42.700 INFO:teuthology.orchestra.run.smithi111.stdout: cephfs-shell x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 27 k 2024-04-21T23:05:42.700 INFO:teuthology.orchestra.run.smithi111.stdout: cephfs-top noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 18 k 2024-04-21T23:05:42.700 INFO:teuthology.orchestra.run.smithi111.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-21T23:05:42.701 INFO:teuthology.orchestra.run.smithi111.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-21T23:05:42.701 INFO:teuthology.orchestra.run.smithi111.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-21T23:05:42.701 INFO:teuthology.orchestra.run.smithi111.stdout: libcephfs-devel x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 32 k 2024-04-21T23:05:42.701 INFO:teuthology.orchestra.run.smithi111.stdout: libcephfs2 x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 745 k 2024-04-21T23:05:42.701 INFO:teuthology.orchestra.run.smithi111.stdout: librados-devel x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 127 k 2024-04-21T23:05:42.701 INFO:teuthology.orchestra.run.smithi111.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-21T23:05:42.701 INFO:teuthology.orchestra.run.smithi111.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-21T23:05:42.701 INFO:teuthology.orchestra.run.smithi111.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-04-21T23:05:42.701 INFO:teuthology.orchestra.run.smithi111.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-21T23:05:42.701 INFO:teuthology.orchestra.run.smithi111.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-21T23:05:42.701 INFO:teuthology.orchestra.run.smithi111.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-21T23:05:42.701 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cephfs x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 161 k 2024-04-21T23:05:42.702 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rados x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 321 k 2024-04-21T23:05:42.702 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rbd x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 298 k 2024-04-21T23:05:42.702 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rgw x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 99 k 2024-04-21T23:05:42.702 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-fuse x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 86 k 2024-04-21T23:05:42.702 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-mirror x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 3.1 M 2024-04-21T23:05:42.702 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-nbd x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 171 k 2024-04-21T23:05:42.702 INFO:teuthology.orchestra.run.smithi111.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-21T23:05:42.702 INFO:teuthology.orchestra.run.smithi111.stdout:Upgrading: 2024-04-21T23:05:42.702 INFO:teuthology.orchestra.run.smithi111.stdout: librados2 x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 3.4 M 2024-04-21T23:05:42.702 INFO:teuthology.orchestra.run.smithi111.stdout: librbd1 x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 3.1 M 2024-04-21T23:05:42.702 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2024-04-21T23:05:42.703 INFO:teuthology.orchestra.run.smithi111.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-21T23:05:42.703 INFO:teuthology.orchestra.run.smithi111.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-21T23:05:42.703 INFO:teuthology.orchestra.run.smithi111.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-21T23:05:42.703 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-common x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 21 M 2024-04-21T23:05:42.703 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 32 k 2024-04-21T23:05:42.703 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mds x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 2.4 M 2024-04-21T23:05:42.703 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 249 k 2024-04-21T23:05:42.703 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mon x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 4.7 M 2024-04-21T23:05:42.703 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-osd x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 17 M 2024-04-21T23:05:42.703 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 15 k 2024-04-21T23:05:42.703 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-selinux x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 25 k 2024-04-21T23:05:42.703 INFO:teuthology.orchestra.run.smithi111.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-21T23:05:42.704 INFO:teuthology.orchestra.run.smithi111.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-21T23:05:42.704 INFO:teuthology.orchestra.run.smithi111.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-21T23:05:42.704 INFO:teuthology.orchestra.run.smithi111.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-21T23:05:42.704 INFO:teuthology.orchestra.run.smithi111.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-21T23:05:42.704 INFO:teuthology.orchestra.run.smithi111.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-21T23:05:42.704 INFO:teuthology.orchestra.run.smithi111.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-21T23:05:42.704 INFO:teuthology.orchestra.run.smithi111.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-21T23:05:42.704 INFO:teuthology.orchestra.run.smithi111.stdout: libcephsqlite x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 164 k 2024-04-21T23:05:42.704 INFO:teuthology.orchestra.run.smithi111.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-21T23:05:42.704 INFO:teuthology.orchestra.run.smithi111.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-21T23:05:42.704 INFO:teuthology.orchestra.run.smithi111.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-21T23:05:42.704 INFO:teuthology.orchestra.run.smithi111.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-21T23:05:42.705 INFO:teuthology.orchestra.run.smithi111.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-21T23:05:42.705 INFO:teuthology.orchestra.run.smithi111.stdout: libradosstriper1 x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 501 k 2024-04-21T23:05:42.705 INFO:teuthology.orchestra.run.smithi111.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-21T23:05:42.705 INFO:teuthology.orchestra.run.smithi111.stdout: librgw2 x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 5.3 M 2024-04-21T23:05:42.705 INFO:teuthology.orchestra.run.smithi111.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-21T23:05:42.705 INFO:teuthology.orchestra.run.smithi111.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-21T23:05:42.705 INFO:teuthology.orchestra.run.smithi111.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-21T23:05:42.705 INFO:teuthology.orchestra.run.smithi111.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-21T23:05:42.705 INFO:teuthology.orchestra.run.smithi111.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-21T23:05:42.705 INFO:teuthology.orchestra.run.smithi111.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-21T23:05:42.705 INFO:teuthology.orchestra.run.smithi111.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-21T23:05:42.705 INFO:teuthology.orchestra.run.smithi111.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-21T23:05:42.706 INFO:teuthology.orchestra.run.smithi111.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-21T23:05:42.706 INFO:teuthology.orchestra.run.smithi111.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-21T23:05:42.706 INFO:teuthology.orchestra.run.smithi111.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-21T23:05:42.706 INFO:teuthology.orchestra.run.smithi111.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-21T23:05:42.706 INFO:teuthology.orchestra.run.smithi111.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-21T23:05:42.706 INFO:teuthology.orchestra.run.smithi111.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-21T23:05:42.706 INFO:teuthology.orchestra.run.smithi111.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-21T23:05:42.706 INFO:teuthology.orchestra.run.smithi111.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-21T23:05:42.706 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-21T23:05:42.706 INFO:teuthology.orchestra.run.smithi111.stdout: python3-ceph-argparse x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 45 k 2024-04-21T23:05:42.706 INFO:teuthology.orchestra.run.smithi111.stdout: python3-ceph-common x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 133 k 2024-04-21T23:05:42.706 INFO:teuthology.orchestra.run.smithi111.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-21T23:05:42.707 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-21T23:05:42.707 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-21T23:05:42.707 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-21T23:05:42.707 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-21T23:05:42.707 INFO:teuthology.orchestra.run.smithi111.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-21T23:05:42.707 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-21T23:05:42.707 INFO:teuthology.orchestra.run.smithi111.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-21T23:05:42.707 INFO:teuthology.orchestra.run.smithi111.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-21T23:05:42.707 INFO:teuthology.orchestra.run.smithi111.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-21T23:05:42.707 INFO:teuthology.orchestra.run.smithi111.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-21T23:05:42.707 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-21T23:05:42.707 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-21T23:05:42.707 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-21T23:05:42.708 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-21T23:05:42.708 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-21T23:05:42.708 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-21T23:05:42.708 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-21T23:05:42.708 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-21T23:05:42.708 INFO:teuthology.orchestra.run.smithi111.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-21T23:05:42.708 INFO:teuthology.orchestra.run.smithi111.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-21T23:05:42.708 INFO:teuthology.orchestra.run.smithi111.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-21T23:05:42.708 INFO:teuthology.orchestra.run.smithi111.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-21T23:05:42.708 INFO:teuthology.orchestra.run.smithi111.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-21T23:05:42.708 INFO:teuthology.orchestra.run.smithi111.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-21T23:05:42.708 INFO:teuthology.orchestra.run.smithi111.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-21T23:05:42.709 INFO:teuthology.orchestra.run.smithi111.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-21T23:05:42.709 INFO:teuthology.orchestra.run.smithi111.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-21T23:05:42.709 INFO:teuthology.orchestra.run.smithi111.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-21T23:05:42.709 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-21T23:05:42.709 INFO:teuthology.orchestra.run.smithi111.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-21T23:05:42.709 INFO:teuthology.orchestra.run.smithi111.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-21T23:05:42.709 INFO:teuthology.orchestra.run.smithi111.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-21T23:05:42.709 INFO:teuthology.orchestra.run.smithi111.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-21T23:05:42.709 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-21T23:05:42.709 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-21T23:05:42.709 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-21T23:05:42.709 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-21T23:05:42.709 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-21T23:05:42.709 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-21T23:05:42.710 INFO:teuthology.orchestra.run.smithi111.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-21T23:05:42.710 INFO:teuthology.orchestra.run.smithi111.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-21T23:05:42.710 INFO:teuthology.orchestra.run.smithi111.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-21T23:05:42.710 INFO:teuthology.orchestra.run.smithi111.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-21T23:05:42.710 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-21T23:05:42.710 INFO:teuthology.orchestra.run.smithi111.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-21T23:05:42.710 INFO:teuthology.orchestra.run.smithi111.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-21T23:05:42.710 INFO:teuthology.orchestra.run.smithi111.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-21T23:05:42.710 INFO:teuthology.orchestra.run.smithi111.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-21T23:05:42.710 INFO:teuthology.orchestra.run.smithi111.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-21T23:05:42.710 INFO:teuthology.orchestra.run.smithi111.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-21T23:05:42.710 INFO:teuthology.orchestra.run.smithi111.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-21T23:05:42.711 INFO:teuthology.orchestra.run.smithi111.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-21T23:05:42.711 INFO:teuthology.orchestra.run.smithi111.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-21T23:05:42.711 INFO:teuthology.orchestra.run.smithi111.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-21T23:05:42.711 INFO:teuthology.orchestra.run.smithi111.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-21T23:05:42.711 INFO:teuthology.orchestra.run.smithi111.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-21T23:05:42.711 INFO:teuthology.orchestra.run.smithi111.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-21T23:05:42.711 INFO:teuthology.orchestra.run.smithi111.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-21T23:05:42.711 INFO:teuthology.orchestra.run.smithi111.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-21T23:05:42.711 INFO:teuthology.orchestra.run.smithi111.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-21T23:05:42.711 INFO:teuthology.orchestra.run.smithi111.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-21T23:05:42.711 INFO:teuthology.orchestra.run.smithi111.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-21T23:05:42.711 INFO:teuthology.orchestra.run.smithi111.stdout:Installing weak dependencies: 2024-04-21T23:05:42.711 INFO:teuthology.orchestra.run.smithi111.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-21T23:05:42.712 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-21T23:05:42.712 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-04-21T23:05:42.712 INFO:teuthology.orchestra.run.smithi111.stdout:======================================================================================= 2024-04-21T23:05:42.712 INFO:teuthology.orchestra.run.smithi111.stdout:Install 148 Packages 2024-04-21T23:05:42.712 INFO:teuthology.orchestra.run.smithi111.stdout:Upgrade 2 Packages 2024-04-21T23:05:42.712 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-21T23:05:42.712 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 216 M 2024-04-21T23:05:42.712 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-04-21T23:05:42.730 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-04-21T23:05:42.730 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-04-21T23:05:42.740 INFO:teuthology.orchestra.run.smithi174.stdout:(122/150): python3-grpcio-tools-1.46.7-10.el9.x 2.4 MB/s | 144 kB 00:00 2024-04-21T23:05:42.782 INFO:teuthology.orchestra.run.smithi174.stdout:(123/150): python3-jaraco-8.2.1-3.el9.noarch.rp 257 kB/s | 11 kB 00:00 2024-04-21T23:05:42.824 INFO:teuthology.orchestra.run.smithi174.stdout:(124/150): python3-jaraco-classes-3.2.1-5.el9.n 425 kB/s | 18 kB 00:00 2024-04-21T23:05:42.874 INFO:teuthology.orchestra.run.smithi174.stdout:(125/150): python3-jaraco-collections-3.0.0-8.e 463 kB/s | 23 kB 00:00 2024-04-21T23:05:42.933 INFO:teuthology.orchestra.run.smithi174.stdout:(126/150): python3-jaraco-functools-3.5.0-2.el9 333 kB/s | 19 kB 00:00 2024-04-21T23:05:42.975 INFO:teuthology.orchestra.run.smithi174.stdout:(127/150): python3-jaraco-text-3.2.0-6.el9.noar 474 kB/s | 20 kB 00:00 2024-04-21T23:05:43.225 INFO:teuthology.orchestra.run.smithi174.stdout:(128/150): python3-scipy-1.6.2-8.el9.x86_64.rpm 6.5 MB/s | 16 MB 00:02 2024-04-21T23:05:43.259 INFO:teuthology.orchestra.run.smithi174.stdout:(129/150): python3-kubernetes-26.1.0-2.el9.noar 3.6 MB/s | 1.0 MB 00:00 2024-04-21T23:05:43.301 INFO:teuthology.orchestra.run.smithi174.stdout:(130/150): python3-more-itertools-8.12.0-2.el9. 1.9 MB/s | 79 kB 00:00 2024-04-21T23:05:43.342 INFO:teuthology.orchestra.run.smithi174.stdout:(131/150): python3-grpcio-1.46.7-10.el9.x86_64. 2.9 MB/s | 2.0 MB 00:00 2024-04-21T23:05:43.368 INFO:teuthology.orchestra.run.smithi174.stdout:(132/150): python3-logutils-0.3.5-21.el9.noarch 326 kB/s | 46 kB 00:00 2024-04-21T23:05:43.393 INFO:teuthology.orchestra.run.smithi174.stdout:(133/150): python3-natsort-7.1.1-5.el9.noarch.r 627 kB/s | 58 kB 00:00 2024-04-21T23:05:43.418 INFO:teuthology.orchestra.run.smithi174.stdout:(134/150): python3-portend-3.1.0-2.el9.noarch.r 328 kB/s | 16 kB 00:00 2024-04-21T23:05:43.444 INFO:teuthology.orchestra.run.smithi174.stdout:(135/150): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.7 MB/s | 90 kB 00:00 2024-04-21T23:05:43.469 INFO:teuthology.orchestra.run.smithi174.stdout:(136/150): python3-pyperclip-1.8.2-2.el9.noarch 561 kB/s | 28 kB 00:00 2024-04-21T23:05:43.494 INFO:teuthology.orchestra.run.smithi174.stdout:(137/150): python3-repoze-lru-0.7-16.el9.noarch 611 kB/s | 31 kB 00:00 2024-04-21T23:05:43.519 INFO:teuthology.orchestra.run.smithi174.stdout:(138/150): python3-pecan-1.4.2-3.el9.noarch.rpm 1.5 MB/s | 272 kB 00:00 2024-04-21T23:05:43.553 INFO:teuthology.orchestra.run.smithi174.stdout:(139/150): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2024-04-21T23:05:43.561 INFO:teuthology.orchestra.run.smithi111.stdout:(1/150): ceph-19.0.0-2405.g820a2f8c.el9.x86_64. 30 kB/s | 6.6 kB 00:00 2024-04-21T23:05:43.578 INFO:teuthology.orchestra.run.smithi174.stdout:(140/150): python3-routes-2.5.1-5.el9.noarch.rp 1.7 MB/s | 188 kB 00:00 2024-04-21T23:05:43.604 INFO:teuthology.orchestra.run.smithi174.stdout:(141/150): python3-tempora-5.0.0-2.el9.noarch.r 427 kB/s | 36 kB 00:00 2024-04-21T23:05:43.629 INFO:teuthology.orchestra.run.smithi174.stdout:(142/150): python3-typing-extensions-4.4.0-2.el 677 kB/s | 51 kB 00:00 2024-04-21T23:05:43.654 INFO:teuthology.orchestra.run.smithi174.stdout:(143/150): python3-websocket-client-1.2.3-2.el9 1.8 MB/s | 90 kB 00:00 2024-04-21T23:05:43.679 INFO:teuthology.orchestra.run.smithi174.stdout:(144/150): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-04-21T23:05:43.704 INFO:teuthology.orchestra.run.smithi174.stdout:(145/150): python3-zc-lockfile-2.0-10.el9.noarc 398 kB/s | 20 kB 00:00 2024-04-21T23:05:43.730 INFO:teuthology.orchestra.run.smithi174.stdout:(146/150): python3-werkzeug-2.0.3-3.el9.1.noarc 4.1 MB/s | 427 kB 00:00 2024-04-21T23:05:43.755 INFO:teuthology.orchestra.run.smithi174.stdout:(147/150): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-04-21T23:05:43.944 INFO:teuthology.orchestra.run.smithi111.stdout:(2/150): ceph-fuse-19.0.0-2405.g820a2f8c.el9.x8 2.3 MB/s | 889 kB 00:00 2024-04-21T23:05:44.053 INFO:teuthology.orchestra.run.smithi111.stdout:(3/150): ceph-immutable-object-cache-19.0.0-240 1.3 MB/s | 145 kB 00:00 2024-04-21T23:05:44.114 INFO:teuthology.orchestra.run.smithi174.stdout:(148/150): thrift-0.15.0-2.el9.x86_64.rpm 3.9 MB/s | 1.6 MB 00:00 2024-04-21T23:05:44.253 INFO:teuthology.orchestra.run.smithi111.stdout:(4/150): ceph-base-19.0.0-2405.g820a2f8c.el9.x8 5.9 MB/s | 5.4 MB 00:00 2024-04-21T23:05:44.446 INFO:teuthology.orchestra.run.smithi111.stdout:(5/150): ceph-mgr-19.0.0-2405.g820a2f8c.el9.x86 7.8 MB/s | 1.5 MB 00:00 2024-04-21T23:05:44.822 INFO:teuthology.orchestra.run.smithi174.stdout:(149/150): librbd1-19.0.0-2405.g820a2f8c.el9.x8 2.9 MB/s | 3.1 MB 00:01 2024-04-21T23:05:44.871 INFO:teuthology.orchestra.run.smithi111.stdout:(6/150): ceph-mon-19.0.0-2405.g820a2f8c.el9.x86 11 MB/s | 4.7 MB 00:00 2024-04-21T23:05:44.889 INFO:teuthology.orchestra.run.smithi174.stdout:(150/150): librados2-19.0.0-2405.g820a2f8c.el9. 2.9 MB/s | 3.4 MB 00:01 2024-04-21T23:05:44.892 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-04-21T23:05:44.892 INFO:teuthology.orchestra.run.smithi174.stdout:Total 12 MB/s | 216 MB 00:18 2024-04-21T23:05:44.929 INFO:teuthology.orchestra.run.smithi111.stdout:(7/150): ceph-mds-19.0.0-2405.g820a2f8c.el9.x86 2.7 MB/s | 2.4 MB 00:00 2024-04-21T23:05:45.913 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-04-21T23:05:46.013 INFO:teuthology.orchestra.run.smithi111.stdout:(8/150): ceph-osd-19.0.0-2405.g820a2f8c.el9.x86 15 MB/s | 17 MB 00:01 2024-04-21T23:05:46.037 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-04-21T23:05:46.037 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-04-21T23:05:46.071 INFO:teuthology.orchestra.run.smithi111.stdout:(9/150): ceph-selinux-19.0.0-2405.g820a2f8c.el9 433 kB/s | 25 kB 00:00 2024-04-21T23:05:47.750 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-04-21T23:05:47.751 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-04-21T23:05:48.363 INFO:teuthology.orchestra.run.smithi111.stdout:(10/150): ceph-common-19.0.0-2405.g820a2f8c.el9 4.2 MB/s | 21 MB 00:05 2024-04-21T23:05:48.472 INFO:teuthology.orchestra.run.smithi111.stdout:(11/150): cephfs-mirror-19.0.0-2405.g820a2f8c.e 2.0 MB/s | 222 kB 00:00 2024-04-21T23:05:48.539 INFO:teuthology.orchestra.run.smithi111.stdout:(12/150): cephfs-shell-19.0.0-2405.g820a2f8c.el 406 kB/s | 27 kB 00:00 2024-04-21T23:05:48.599 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-04-21T23:05:48.672 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : thrift-0.15.0-2.el9.x86_64 1/152 2024-04-21T23:05:48.689 INFO:teuthology.orchestra.run.smithi111.stdout:(13/150): ceph-radosgw-19.0.0-2405.g820a2f8c.el 2.7 MB/s | 10 MB 00:03 2024-04-21T23:05:48.697 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 2/152 2024-04-21T23:05:48.706 INFO:teuthology.orchestra.run.smithi111.stdout:(14/150): libcephfs-devel-19.0.0-2405.g820a2f8c 190 kB/s | 32 kB 00:00 2024-04-21T23:05:48.741 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 3/152 2024-04-21T23:05:48.781 INFO:teuthology.orchestra.run.smithi111.stdout:(15/150): libcephsqlite-19.0.0-2405.g820a2f8c.e 2.1 MB/s | 164 kB 00:00 2024-04-21T23:05:48.865 INFO:teuthology.orchestra.run.smithi111.stdout:(16/150): librados-devel-19.0.0-2405.g820a2f8c. 1.5 MB/s | 127 kB 00:00 2024-04-21T23:05:48.982 INFO:teuthology.orchestra.run.smithi111.stdout:(17/150): libcephfs2-19.0.0-2405.g820a2f8c.el9. 2.5 MB/s | 745 kB 00:00 2024-04-21T23:05:49.089 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 4/152 2024-04-21T23:05:49.214 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : librados2-2:19.0.0-2405.g820a2f8c.el9.x86_64 5/152 2024-04-21T23:05:49.322 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librados2-2:19.0.0-2405.g820a2f8c.el9.x86_64 5/152 2024-04-21T23:05:49.357 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libcephfs2-2:19.0.0-2405.g820a2f8c.el9.x86_64 6/152 2024-04-21T23:05:49.399 INFO:teuthology.orchestra.run.smithi111.stdout:(18/150): ceph-test-19.0.0-2405.g820a2f8c.el9.x 15 MB/s | 48 MB 00:03 2024-04-21T23:05:49.424 INFO:teuthology.orchestra.run.smithi111.stdout:(19/150): libradosstriper1-19.0.0-2405.g820a2f8 897 kB/s | 501 kB 00:00 2024-04-21T23:05:49.450 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: libcephfs2-2:19.0.0-2405.g820a2f8c.el9.x86_64 6/152 2024-04-21T23:05:49.458 INFO:teuthology.orchestra.run.smithi111.stdout:(20/150): python3-ceph-argparse-19.0.0-2405.g82 775 kB/s | 45 kB 00:00 2024-04-21T23:05:49.491 INFO:teuthology.orchestra.run.smithi111.stdout:(21/150): python3-ceph-common-19.0.0-2405.g820a 1.9 MB/s | 133 kB 00:00 2024-04-21T23:05:49.492 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-rados-2:19.0.0-2405.g820a2f8c.el9.x86_64 7/152 2024-04-21T23:05:49.516 INFO:teuthology.orchestra.run.smithi111.stdout:(22/150): python3-cephfs-19.0.0-2405.g820a2f8c. 2.7 MB/s | 161 kB 00:00 2024-04-21T23:05:49.520 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/152 2024-04-21T23:05:49.546 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/152 2024-04-21T23:05:49.583 INFO:teuthology.orchestra.run.smithi111.stdout:(23/150): python3-rbd-19.0.0-2405.g820a2f8c.el9 4.4 MB/s | 298 kB 00:00 2024-04-21T23:05:49.609 INFO:teuthology.orchestra.run.smithi111.stdout:(24/150): python3-rados-19.0.0-2405.g820a2f8c.e 2.7 MB/s | 321 kB 00:00 2024-04-21T23:05:49.642 INFO:teuthology.orchestra.run.smithi111.stdout:(25/150): python3-rgw-19.0.0-2405.g820a2f8c.el9 1.6 MB/s | 99 kB 00:00 2024-04-21T23:05:49.668 INFO:teuthology.orchestra.run.smithi111.stdout:(26/150): rbd-fuse-19.0.0-2405.g820a2f8c.el9.x8 1.4 MB/s | 86 kB 00:00 2024-04-21T23:05:49.712 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 10/152 2024-04-21T23:05:49.726 INFO:teuthology.orchestra.run.smithi111.stdout:(27/150): rbd-nbd-19.0.0-2405.g820a2f8c.el9.x86 2.9 MB/s | 171 kB 00:00 2024-04-21T23:05:49.760 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : postgresql-13.14-1.el9.x86_64 11/152 2024-04-21T23:05:49.787 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libcephsqlite-2:19.0.0-2405.g820a2f8c.el9.x86_64 12/152 2024-04-21T23:05:49.802 INFO:teuthology.orchestra.run.smithi111.stdout:(28/150): ceph-grafana-dashboards-19.0.0-2405.g 423 kB/s | 32 kB 00:00 2024-04-21T23:05:49.860 INFO:teuthology.orchestra.run.smithi111.stdout:(29/150): rbd-mirror-19.0.0-2405.g820a2f8c.el9. 14 MB/s | 3.1 MB 00:00 2024-04-21T23:05:49.886 INFO:teuthology.orchestra.run.smithi111.stdout:(30/150): ceph-mgr-cephadm-19.0.0-2405.g820a2f8 1.6 MB/s | 141 kB 00:00 2024-04-21T23:05:49.893 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2405.g820a2f8c.el9.x86_64 12/152 2024-04-21T23:05:49.922 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libradosstriper1-2:19.0.0-2405.g820a2f8c.el9.x86 13/152 2024-04-21T23:05:49.976 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2405.g820a2f8c.el9.x86 13/152 2024-04-21T23:05:49.998 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-04-21T23:05:50.020 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-04-21T23:05:50.128 INFO:teuthology.orchestra.run.smithi111.stdout:(31/150): ceph-mgr-dashboard-19.0.0-2405.g820a2 13 MB/s | 3.5 MB 00:00 2024-04-21T23:05:50.173 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 15/152 2024-04-21T23:05:50.203 INFO:teuthology.orchestra.run.smithi111.stdout:(32/150): ceph-mgr-modules-core-19.0.0-2405.g82 3.2 MB/s | 249 kB 00:00 2024-04-21T23:05:50.206 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 15/152 2024-04-21T23:05:50.270 INFO:teuthology.orchestra.run.smithi111.stdout:(33/150): ceph-mgr-rook-19.0.0-2405.g820a2f8c.e 741 kB/s | 49 kB 00:00 2024-04-21T23:05:50.318 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : re2-1:20211101-3.el9.x86_64 16/152 2024-04-21T23:05:50.328 INFO:teuthology.orchestra.run.smithi111.stdout:(34/150): ceph-prometheus-alerts-19.0.0-2405.g8 262 kB/s | 15 kB 00:00 2024-04-21T23:05:50.404 INFO:teuthology.orchestra.run.smithi111.stdout:(35/150): ceph-volume-19.0.0-2405.g820a2f8c.el9 3.4 MB/s | 264 kB 00:00 2024-04-21T23:05:50.417 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 17/152 2024-04-21T23:05:50.455 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 18/152 2024-04-21T23:05:50.492 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : liboath-2.6.7-2.el9.x86_64 19/152 2024-04-21T23:05:50.512 INFO:teuthology.orchestra.run.smithi111.stdout:(36/150): cephadm-19.0.0-2405.g820a2f8c.el9.noa 6.9 MB/s | 762 kB 00:00 2024-04-21T23:05:50.568 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 20/152 2024-04-21T23:05:50.579 INFO:teuthology.orchestra.run.smithi111.stdout:(37/150): librgw2-19.0.0-2405.g820a2f8c.el9.x86 3.3 MB/s | 5.3 MB 00:01 2024-04-21T23:05:50.596 INFO:teuthology.orchestra.run.smithi111.stdout:(38/150): cephfs-top-19.0.0-2405.g820a2f8c.el9. 218 kB/s | 18 kB 00:00 2024-04-21T23:05:50.607 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 21/152 2024-04-21T23:05:50.651 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 22/152 2024-04-21T23:05:50.692 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 23/152 2024-04-21T23:05:50.727 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : lua-5.4.4-4.el9.x86_64 24/152 2024-04-21T23:05:50.802 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 25/152 2024-04-21T23:05:50.861 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 26/152 2024-04-21T23:05:50.889 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 27/152 2024-04-21T23:05:50.933 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 28/152 2024-04-21T23:05:50.964 INFO:teuthology.orchestra.run.smithi111.stdout:(39/150): ceph-mgr-diskprediction-local-19.0.0- 6.9 MB/s | 7.4 MB 00:01 2024-04-21T23:05:51.013 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 29/152 2024-04-21T23:05:51.014 INFO:teuthology.orchestra.run.smithi111.stdout:(40/150): fuse-2.9.9-15.el9.x86_64.rpm 185 kB/s | 80 kB 00:00 2024-04-21T23:05:51.043 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-ceph-common-2:19.0.0-2405.g820a2f8c.el9. 30/152 2024-04-21T23:05:51.087 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-ceph-argparse-2:19.0.0-2405.g820a2f8c.el 31/152 2024-04-21T23:05:51.115 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cephfs-2:19.0.0-2405.g820a2f8c.el9.x86_6 32/152 2024-04-21T23:05:51.191 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 33/152 2024-04-21T23:05:51.247 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : luarocks-3.9.2-1.el9.noarch 34/152 2024-04-21T23:05:51.328 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 35/152 2024-04-21T23:05:51.355 INFO:teuthology.orchestra.run.smithi111.stdout:(41/150): libgfortran-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 803 kB 00:00 2024-04-21T23:05:51.381 INFO:teuthology.orchestra.run.smithi111.stdout:(42/150): python3-cffi-1.14.5-5.el9.x86_64.rpm 690 kB/s | 253 kB 00:00 2024-04-21T23:05:51.473 INFO:teuthology.orchestra.run.smithi111.stdout:(43/150): libquadmath-11.4.1-3.el9.x86_64.rpm 384 kB/s | 195 kB 00:00 2024-04-21T23:05:51.474 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 36/152 2024-04-21T23:05:51.498 INFO:teuthology.orchestra.run.smithi111.stdout:(44/150): python3-ply-3.11-14.el9.noarch.rpm 912 kB/s | 106 kB 00:00 2024-04-21T23:05:51.534 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 37/152 2024-04-21T23:05:51.581 INFO:teuthology.orchestra.run.smithi111.stdout:(45/150): python3-pycparser-2.20-6.el9.noarch.r 1.2 MB/s | 135 kB 00:00 2024-04-21T23:05:51.607 INFO:teuthology.orchestra.run.smithi111.stdout:(46/150): python3-requests-2.25.1-8.el9.noarch. 1.1 MB/s | 125 kB 00:00 2024-04-21T23:05:51.707 INFO:teuthology.orchestra.run.smithi111.stdout:(47/150): python3-urllib3-1.26.5-5.el9.noarch.r 1.7 MB/s | 215 kB 00:00 2024-04-21T23:05:51.741 INFO:teuthology.orchestra.run.smithi111.stdout:(48/150): python3-cryptography-36.0.1-4.el9.x86 3.2 MB/s | 1.2 MB 00:00 2024-04-21T23:05:51.816 INFO:teuthology.orchestra.run.smithi111.stdout:(49/150): automake-1.16.2-8.el9.noarch.rpm 3.2 MB/s | 689 kB 00:00 2024-04-21T23:05:51.858 INFO:teuthology.orchestra.run.smithi111.stdout:(50/150): elfutils-libelf-devel-0.190-2.el9.x86 516 kB/s | 21 kB 00:00 2024-04-21T23:05:51.883 INFO:teuthology.orchestra.run.smithi111.stdout:(51/150): boost-program-options-1.75.0-8.el9.x8 752 kB/s | 107 kB 00:00 2024-04-21T23:05:51.909 INFO:teuthology.orchestra.run.smithi111.stdout:(52/150): flex-2.6.4-9.el9.x86_64.rpm 6.1 MB/s | 313 kB 00:00 2024-04-21T23:05:51.934 INFO:teuthology.orchestra.run.smithi111.stdout:(53/150): flexiblas-3.0.4-8.el9.x86_64.rpm 651 kB/s | 33 kB 00:00 2024-04-21T23:05:51.976 INFO:teuthology.orchestra.run.smithi111.stdout:(54/150): bison-3.7.4-5.el9.x86_64.rpm 3.4 MB/s | 943 kB 00:00 2024-04-21T23:05:52.001 INFO:teuthology.orchestra.run.smithi111.stdout:(55/150): flexiblas-openblas-openmp-3.0.4-8.el9 272 kB/s | 18 kB 00:00 2024-04-21T23:05:52.026 INFO:teuthology.orchestra.run.smithi111.stdout:(56/150): libaio-devel-0.3.111-13.el9.x86_64.rp 223 kB/s | 11 kB 00:00 2024-04-21T23:05:52.052 INFO:teuthology.orchestra.run.smithi111.stdout:(57/150): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 160 kB 00:00 2024-04-21T23:05:52.077 INFO:teuthology.orchestra.run.smithi111.stdout:(58/150): librabbitmq-0.11.0-7.el9.x86_64.rpm 900 kB/s | 45 kB 00:00 2024-04-21T23:05:52.109 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-rsa-4.9-2.el9.noarch 38/152 2024-04-21T23:05:52.135 INFO:teuthology.orchestra.run.smithi111.stdout:(59/150): flexiblas-netlib-3.0.4-8.el9.x86_64.r 13 MB/s | 3.0 MB 00:00 2024-04-21T23:05:52.161 INFO:teuthology.orchestra.run.smithi111.stdout:(60/150): librdkafka-1.6.1-102.el9.x86_64.rpm 6.0 MB/s | 662 kB 00:00 2024-04-21T23:05:52.186 INFO:teuthology.orchestra.run.smithi111.stdout:(61/150): libuuid-devel-2.37.4-18.el9.x86_64.rp 508 kB/s | 25 kB 00:00 2024-04-21T23:05:52.200 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 39/152 2024-04-21T23:05:52.211 INFO:teuthology.orchestra.run.smithi111.stdout:(62/150): libzstd-devel-1.5.1-2.el9.x86_64.rpm 948 kB/s | 47 kB 00:00 2024-04-21T23:05:52.232 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 40/152 2024-04-21T23:05:52.245 INFO:teuthology.orchestra.run.smithi111.stdout:(63/150): libtool-2.4.6-45.el9.x86_64.rpm 3.4 MB/s | 582 kB 00:00 2024-04-21T23:05:52.270 INFO:teuthology.orchestra.run.smithi111.stdout:(64/150): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.4 MB/s | 292 kB 00:00 2024-04-21T23:05:52.276 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 41/152 2024-04-21T23:05:52.295 INFO:teuthology.orchestra.run.smithi111.stdout:(65/150): lua-5.4.4-4.el9.x86_64.rpm 2.2 MB/s | 188 kB 00:00 2024-04-21T23:05:52.314 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : librados-devel-2:19.0.0-2405.g820a2f8c.el9.x86_6 42/152 2024-04-21T23:05:52.321 INFO:teuthology.orchestra.run.smithi111.stdout:(66/150): openblas-0.3.21-2.el9.x86_64.rpm 468 kB/s | 35 kB 00:00 2024-04-21T23:05:52.346 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 43/152 2024-04-21T23:05:52.405 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 44/152 2024-04-21T23:05:52.446 INFO:teuthology.orchestra.run.smithi111.stdout:(67/150): postgresql-13.14-1.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-04-21T23:05:52.479 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 45/152 2024-04-21T23:05:52.514 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 46/152 2024-04-21T23:05:52.521 INFO:teuthology.orchestra.run.smithi111.stdout:(68/150): postgresql-contrib-13.14-1.el9.x86_64 12 MB/s | 911 kB 00:00 2024-04-21T23:05:52.551 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 47/152 2024-04-21T23:05:52.580 INFO:teuthology.orchestra.run.smithi111.stdout:(69/150): openblas-openmp-0.3.21-2.el9.x86_64.r 15 MB/s | 4.7 MB 00:00 2024-04-21T23:05:52.599 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 48/152 2024-04-21T23:05:52.605 INFO:teuthology.orchestra.run.smithi111.stdout:(70/150): postgresql-private-libs-13.14-1.el9.x 1.6 MB/s | 133 kB 00:00 2024-04-21T23:05:52.633 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 49/152 2024-04-21T23:05:52.664 INFO:teuthology.orchestra.run.smithi111.stdout:(71/150): openssl-devel-3.0.7-27.el9.x86_64.rpm 12 MB/s | 4.1 MB 00:00 2024-04-21T23:05:52.678 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 50/152 2024-04-21T23:05:52.714 INFO:teuthology.orchestra.run.smithi111.stdout:(72/150): python3-attrs-20.3.0-7.el9.noarch.rpm 1.7 MB/s | 87 kB 00:00 2024-04-21T23:05:52.720 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 51/152 2024-04-21T23:05:52.748 INFO:teuthology.orchestra.run.smithi111.stdout:(73/150): protobuf-3.14.0-13.el9.x86_64.rpm 7.1 MB/s | 1.0 MB 00:00 2024-04-21T23:05:52.840 INFO:teuthology.orchestra.run.smithi111.stdout:(74/150): postgresql-server-13.14-1.el9.x86_64. 22 MB/s | 5.8 MB 00:00 2024-04-21T23:05:52.865 INFO:teuthology.orchestra.run.smithi111.stdout:(75/150): python3-devel-3.9.18-3.el9.x86_64.rpm 2.0 MB/s | 244 kB 00:00 2024-04-21T23:05:52.869 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 52/152 2024-04-21T23:05:52.890 INFO:teuthology.orchestra.run.smithi111.stdout:(76/150): python3-jinja2-2.11.3-5.el9.noarch.rp 4.8 MB/s | 248 kB 00:00 2024-04-21T23:05:52.907 INFO:teuthology.orchestra.run.smithi111.stdout:(77/150): python3-jsonpatch-1.21-16.el9.noarch. 612 kB/s | 26 kB 00:00 2024-04-21T23:05:52.917 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 53/152 2024-04-21T23:05:52.924 INFO:teuthology.orchestra.run.smithi111.stdout:(78/150): python3-jsonpointer-2.0-4.el9.noarch. 571 kB/s | 19 kB 00:00 2024-04-21T23:05:52.949 INFO:teuthology.orchestra.run.smithi111.stdout:(79/150): python3-mako-1.1.4-6.el9.noarch.rpm 4.0 MB/s | 172 kB 00:00 2024-04-21T23:05:52.954 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 54/152 2024-04-21T23:05:52.966 INFO:teuthology.orchestra.run.smithi111.stdout:(80/150): python3-markupsafe-1.1.1-12.el9.x86_6 830 kB/s | 35 kB 00:00 2024-04-21T23:05:52.996 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 55/152 2024-04-21T23:05:53.067 INFO:teuthology.orchestra.run.smithi111.stdout:(81/150): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 6.0 MB 00:00 2024-04-21T23:05:53.092 INFO:teuthology.orchestra.run.smithi111.stdout:(82/150): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.0 MB/s | 250 kB 00:00 2024-04-21T23:05:53.097 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 56/152 2024-04-21T23:05:53.117 INFO:teuthology.orchestra.run.smithi111.stdout:(83/150): python3-oauthlib-3.1.1-5.el9.noarch.r 4.3 MB/s | 222 kB 00:00 2024-04-21T23:05:53.129 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 57/152 2024-04-21T23:05:53.142 INFO:teuthology.orchestra.run.smithi111.stdout:(84/150): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-04-21T23:05:53.167 INFO:teuthology.orchestra.run.smithi111.stdout:(85/150): python3-prettytable-0.7.2-27.el9.noar 845 kB/s | 42 kB 00:00 2024-04-21T23:05:53.169 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 58/152 2024-04-21T23:05:53.193 INFO:teuthology.orchestra.run.smithi111.stdout:(86/150): python3-protobuf-3.14.0-13.el9.noarch 5.3 MB/s | 269 kB 00:00 2024-04-21T23:05:53.203 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 59/152 2024-04-21T23:05:53.218 INFO:teuthology.orchestra.run.smithi111.stdout:(87/150): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.1 MB/s | 159 kB 00:00 2024-04-21T23:05:53.227 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 60/152 2024-04-21T23:05:53.243 INFO:teuthology.orchestra.run.smithi111.stdout:(88/150): python3-pyasn1-modules-0.4.8-6.el9.no 5.4 MB/s | 279 kB 00:00 2024-04-21T23:05:53.268 INFO:teuthology.orchestra.run.smithi111.stdout:(89/150): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-04-21T23:05:53.294 INFO:teuthology.orchestra.run.smithi111.stdout:(90/150): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 54 kB 00:00 2024-04-21T23:05:53.308 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 61/152 2024-04-21T23:05:53.336 INFO:teuthology.orchestra.run.smithi111.stdout:(91/150): python3-toml-0.10.2-6.el9.noarch.rpm 1.0 MB/s | 42 kB 00:00 2024-04-21T23:05:53.403 INFO:teuthology.orchestra.run.smithi111.stdout:(92/150): python3-numpy-1.20.1-5.el9.x86_64.rpm 11 MB/s | 5.1 MB 00:00 2024-04-21T23:05:53.428 INFO:teuthology.orchestra.run.smithi111.stdout:(93/150): python3-wcwidth-0.2.5-8.el9.noarch.rp 482 kB/s | 44 kB 00:00 2024-04-21T23:05:53.441 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 62/152 2024-04-21T23:05:53.453 INFO:teuthology.orchestra.run.smithi111.stdout:(94/150): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 221 kB 00:00 2024-04-21T23:05:53.476 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 63/152 2024-04-21T23:05:53.478 INFO:teuthology.orchestra.run.smithi111.stdout:(95/150): qatlib-service-24.02.0-1.el9.x86_64.r 713 kB/s | 36 kB 00:00 2024-04-21T23:05:53.504 INFO:teuthology.orchestra.run.smithi111.stdout:(96/150): qatzip-libs-1.2.0-1.el9.x86_64.rpm 946 kB/s | 47 kB 00:00 2024-04-21T23:05:53.505 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 64/152 2024-04-21T23:05:53.537 INFO:teuthology.orchestra.run.smithi111.stdout:(97/150): socat-1.7.4.1-5.el9.x86_64.rpm 5.1 MB/s | 305 kB 00:00 2024-04-21T23:05:53.539 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 65/152 2024-04-21T23:05:53.563 INFO:teuthology.orchestra.run.smithi111.stdout:(98/150): uuid-1.6.2-55.el9.x86_64.rpm 986 kB/s | 58 kB 00:00 2024-04-21T23:05:53.584 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : uuid-1.6.2-55.el9.x86_64 66/152 2024-04-21T23:05:53.588 INFO:teuthology.orchestra.run.smithi111.stdout:(99/150): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.8 MB/s | 94 kB 00:00 2024-04-21T23:05:53.614 INFO:teuthology.orchestra.run.smithi111.stdout:(100/150): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-04-21T23:05:53.622 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 67/152 2024-04-21T23:05:53.639 INFO:teuthology.orchestra.run.smithi111.stdout:(101/150): zlib-devel-1.2.11-41.el9.x86_64.rpm 893 kB/s | 45 kB 00:00 2024-04-21T23:05:53.661 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 68/152 2024-04-21T23:05:53.690 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 69/152 2024-04-21T23:05:53.712 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-04-21T23:05:53.725 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 70/152 2024-04-21T23:05:53.758 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 71/152 2024-04-21T23:05:53.791 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : thrift-0.15.0-2.el9.x86_64 1/152 2024-04-21T23:05:53.801 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 72/152 2024-04-21T23:05:53.834 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 73/152 2024-04-21T23:05:53.872 INFO:teuthology.orchestra.run.smithi111.stdout:(102/150): python3-scipy-1.6.2-8.el9.x86_64.rpm 27 MB/s | 16 MB 00:00 2024-04-21T23:05:53.917 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 2/152 2024-04-21T23:05:53.927 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 74/152 2024-04-21T23:05:53.975 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 75/152 2024-04-21T23:05:54.084 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 76/152 2024-04-21T23:05:54.113 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 3/152 2024-04-21T23:05:54.123 INFO:teuthology.orchestra.run.smithi111.stdout:(103/150): abseil-cpp-20211102.0-3.el9.x86_64.r 2.1 MB/s | 548 kB 00:00 2024-04-21T23:05:54.190 INFO:teuthology.orchestra.run.smithi111.stdout:(104/150): gperftools-libs-2.9.1-2.el9.x86_64.r 4.5 MB/s | 309 kB 00:00 2024-04-21T23:05:54.240 INFO:teuthology.orchestra.run.smithi111.stdout:(105/150): grpc-data-1.46.7-10.el9.noarch.rpm 389 kB/s | 19 kB 00:00 2024-04-21T23:05:54.273 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 77/152 2024-04-21T23:05:54.298 INFO:teuthology.orchestra.run.smithi111.stdout:(106/150): lua-devel-5.4.4-4.el9.x86_64.rpm 33 kB/s | 22 kB 00:00 2024-04-21T23:05:54.389 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-packaging-20.9-5.el9.noarch 78/152 2024-04-21T23:05:54.399 INFO:teuthology.orchestra.run.smithi111.stdout:(107/150): libarrow-doc-9.0.0-10.el9.noarch.rpm 260 kB/s | 26 kB 00:00 2024-04-21T23:05:54.441 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 79/152 2024-04-21T23:05:54.442 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 4/152 2024-04-21T23:05:54.466 INFO:teuthology.orchestra.run.smithi111.stdout:(108/150): libarrow-9.0.0-10.el9.x86_64.rpm 20 MB/s | 4.4 MB 00:00 2024-04-21T23:05:54.491 INFO:teuthology.orchestra.run.smithi111.stdout:(109/150): liboath-2.6.7-2.el9.x86_64.rpm 530 kB/s | 49 kB 00:00 2024-04-21T23:05:54.516 INFO:teuthology.orchestra.run.smithi111.stdout:(110/150): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-04-21T23:05:54.551 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : librados2-2:19.0.0-2405.g820a2f8c.el9.x86_64 5/152 2024-04-21T23:05:54.593 INFO:teuthology.orchestra.run.smithi111.stdout:(111/150): parquet-libs-9.0.0-10.el9.x86_64.rpm 11 MB/s | 838 kB 00:00 2024-04-21T23:05:54.617 INFO:teuthology.orchestra.run.smithi111.stdout:(112/150): luarocks-3.9.2-1.el9.noarch.rpm 1.2 MB/s | 151 kB 00:00 2024-04-21T23:05:54.659 INFO:teuthology.orchestra.run.smithi111.stdout:(113/150): python3-asyncssh-2.13.2-5.el9.noarch 8.0 MB/s | 548 kB 00:00 2024-04-21T23:05:54.672 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: librados2-2:19.0.0-2405.g820a2f8c.el9.x86_64 5/152 2024-04-21T23:05:54.676 INFO:teuthology.orchestra.run.smithi111.stdout:(114/150): python3-bcrypt-3.2.2-1.el9.x86_64.rp 741 kB/s | 43 kB 00:00 2024-04-21T23:05:54.701 INFO:teuthology.orchestra.run.smithi111.stdout:(115/150): python3-cachetools-4.2.4-1.el9.noarc 771 kB/s | 32 kB 00:00 2024-04-21T23:05:54.702 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libcephfs2-2:19.0.0-2405.g820a2f8c.el9.x86_64 6/152 2024-04-21T23:05:54.718 INFO:teuthology.orchestra.run.smithi111.stdout:(116/150): python3-certifi-2021.10.8-2.el9.noar 358 kB/s | 15 kB 00:00 2024-04-21T23:05:54.751 INFO:teuthology.orchestra.run.smithi111.stdout:(117/150): python3-cheroot-8.6.0-4.el9.noarch.r 3.3 MB/s | 172 kB 00:00 2024-04-21T23:05:54.793 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: libcephfs2-2:19.0.0-2405.g820a2f8c.el9.x86_64 6/152 2024-04-21T23:05:54.810 INFO:teuthology.orchestra.run.smithi111.stdout:(118/150): python3-cmd2-2.4.2-2.el9.noarch.rpm 5.1 MB/s | 303 kB 00:00 2024-04-21T23:05:54.835 INFO:teuthology.orchestra.run.smithi111.stdout:(119/150): python3-cherrypy-18.6.1-2.el9.noarch 3.0 MB/s | 358 kB 00:00 2024-04-21T23:05:54.836 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-rados-2:19.0.0-2405.g820a2f8c.el9.x86_64 7/152 2024-04-21T23:05:54.852 INFO:teuthology.orchestra.run.smithi111.stdout:(120/150): python3-colorama-0.4.4-12.el9.noarch 845 kB/s | 35 kB 00:00 2024-04-21T23:05:54.864 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/152 2024-04-21T23:05:54.886 INFO:teuthology.orchestra.run.smithi111.stdout:(121/150): python3-google-auth-2.28.2-1.el9.noa 3.9 MB/s | 203 kB 00:00 2024-04-21T23:05:54.890 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/152 2024-04-21T23:05:54.928 INFO:teuthology.orchestra.run.smithi111.stdout:(122/150): python3-grpcio-tools-1.46.7-10.el9.x 3.4 MB/s | 144 kB 00:00 2024-04-21T23:05:54.970 INFO:teuthology.orchestra.run.smithi111.stdout:(123/150): python3-grpcio-1.46.7-10.el9.x86_64. 17 MB/s | 2.0 MB 00:00 2024-04-21T23:05:54.987 INFO:teuthology.orchestra.run.smithi111.stdout:(124/150): python3-jaraco-8.2.1-3.el9.noarch.rp 182 kB/s | 11 kB 00:00 2024-04-21T23:05:55.013 INFO:teuthology.orchestra.run.smithi111.stdout:(125/150): python3-jaraco-classes-3.2.1-5.el9.n 423 kB/s | 18 kB 00:00 2024-04-21T23:05:55.029 INFO:teuthology.orchestra.run.smithi111.stdout:(126/150): python3-jaraco-collections-3.0.0-8.e 552 kB/s | 23 kB 00:00 2024-04-21T23:05:55.044 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 10/152 2024-04-21T23:05:55.054 INFO:teuthology.orchestra.run.smithi111.stdout:(127/150): python3-jaraco-functools-3.5.0-2.el9 463 kB/s | 19 kB 00:00 2024-04-21T23:05:55.080 INFO:teuthology.orchestra.run.smithi111.stdout:(128/150): python3-jaraco-text-3.2.0-6.el9.noar 391 kB/s | 20 kB 00:00 2024-04-21T23:05:55.116 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : postgresql-13.14-1.el9.x86_64 11/152 2024-04-21T23:05:55.132 INFO:teuthology.orchestra.run.smithi111.stdout:(129/150): python3-kubernetes-26.1.0-2.el9.noar 13 MB/s | 1.0 MB 00:00 2024-04-21T23:05:55.149 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libcephsqlite-2:19.0.0-2405.g820a2f8c.el9.x86_64 12/152 2024-04-21T23:05:55.157 INFO:teuthology.orchestra.run.smithi111.stdout:(130/150): python3-logutils-0.3.5-21.el9.noarch 600 kB/s | 46 kB 00:00 2024-04-21T23:05:55.208 INFO:teuthology.orchestra.run.smithi111.stdout:(131/150): python3-more-itertools-8.12.0-2.el9. 1.0 MB/s | 79 kB 00:00 2024-04-21T23:05:55.233 INFO:teuthology.orchestra.run.smithi111.stdout:(132/150): python3-natsort-7.1.1-5.el9.noarch.r 767 kB/s | 58 kB 00:00 2024-04-21T23:05:55.243 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 80/152 2024-04-21T23:05:55.252 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2405.g820a2f8c.el9.x86_64 12/152 2024-04-21T23:05:55.275 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libradosstriper1-2:19.0.0-2405.g820a2f8c.el9.x86 13/152 2024-04-21T23:05:55.283 INFO:teuthology.orchestra.run.smithi111.stdout:(133/150): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-04-21T23:05:55.295 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 81/152 2024-04-21T23:05:55.308 INFO:teuthology.orchestra.run.smithi111.stdout:(134/150): python3-portend-3.1.0-2.el9.noarch.r 218 kB/s | 16 kB 00:00 2024-04-21T23:05:55.329 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 82/152 2024-04-21T23:05:55.334 INFO:teuthology.orchestra.run.smithi111.stdout:(135/150): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.8 MB/s | 90 kB 00:00 2024-04-21T23:05:55.337 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2405.g820a2f8c.el9.x86 13/152 2024-04-21T23:05:55.359 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-04-21T23:05:55.367 INFO:teuthology.orchestra.run.smithi111.stdout:(136/150): python3-pyperclip-1.8.2-2.el9.noarch 482 kB/s | 28 kB 00:00 2024-04-21T23:05:55.382 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-04-21T23:05:55.392 INFO:teuthology.orchestra.run.smithi111.stdout:(137/150): python3-repoze-lru-0.7-16.el9.noarch 527 kB/s | 31 kB 00:00 2024-04-21T23:05:55.436 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 83/152 2024-04-21T23:05:55.443 INFO:teuthology.orchestra.run.smithi111.stdout:(138/150): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-04-21T23:05:55.485 INFO:teuthology.orchestra.run.smithi111.stdout:(139/150): protobuf-compiler-3.14.0-13.el9.x86_ 468 kB/s | 863 kB 00:01 2024-04-21T23:05:55.488 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 84/152 2024-04-21T23:05:55.532 INFO:teuthology.orchestra.run.smithi111.stdout:(140/150): python3-rsa-4.9-2.el9.noarch.rpm 423 kB/s | 59 kB 00:00 2024-04-21T23:05:55.560 INFO:teuthology.orchestra.run.smithi111.stdout:(141/150): python3-tempora-5.0.0-2.el9.noarch.r 305 kB/s | 36 kB 00:00 2024-04-21T23:05:55.567 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 15/152 2024-04-21T23:05:55.580 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 85/152 2024-04-21T23:05:55.585 INFO:teuthology.orchestra.run.smithi111.stdout:(142/150): python3-typing-extensions-4.4.0-2.el 972 kB/s | 51 kB 00:00 2024-04-21T23:05:55.609 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 15/152 2024-04-21T23:05:55.619 INFO:teuthology.orchestra.run.smithi111.stdout:(143/150): python3-websocket-client-1.2.3-2.el9 1.5 MB/s | 90 kB 00:00 2024-04-21T23:05:55.622 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 86/152 2024-04-21T23:05:55.669 INFO:teuthology.orchestra.run.smithi111.stdout:(144/150): python3-zc-lockfile-2.0-10.el9.noarc 401 kB/s | 20 kB 00:00 2024-04-21T23:05:55.695 INFO:teuthology.orchestra.run.smithi111.stdout:(145/150): python3-werkzeug-2.0.3-3.el9.1.noarc 3.8 MB/s | 427 kB 00:00 2024-04-21T23:05:55.716 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : re2-1:20211101-3.el9.x86_64 16/152 2024-04-21T23:05:55.720 INFO:teuthology.orchestra.run.smithi111.stdout:(146/150): python3-webob-1.8.7-6.el9.noarch.rpm 1.2 MB/s | 230 kB 00:00 2024-04-21T23:05:55.745 INFO:teuthology.orchestra.run.smithi111.stdout:(147/150): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-04-21T23:05:55.787 INFO:teuthology.orchestra.run.smithi111.stdout:(148/150): thrift-0.15.0-2.el9.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2024-04-21T23:05:55.800 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : openblas-0.3.21-2.el9.x86_64 87/152 2024-04-21T23:05:55.822 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 17/152 2024-04-21T23:05:55.828 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 88/152 2024-04-21T23:05:55.858 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 18/152 2024-04-21T23:05:55.893 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : liboath-2.6.7-2.el9.x86_64 19/152 2024-04-21T23:05:55.909 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 89/152 2024-04-21T23:05:55.957 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 20/152 2024-04-21T23:05:55.992 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 21/152 2024-04-21T23:05:56.035 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 22/152 2024-04-21T23:05:56.068 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 23/152 2024-04-21T23:05:56.096 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : lua-5.4.4-4.el9.x86_64 24/152 2024-04-21T23:05:56.146 INFO:teuthology.orchestra.run.smithi111.stdout:(149/150): librbd1-19.0.0-2405.g820a2f8c.el9.x8 7.8 MB/s | 3.1 MB 00:00 2024-04-21T23:05:56.174 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 25/152 2024-04-21T23:05:56.227 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 26/152 2024-04-21T23:05:56.265 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 27/152 2024-04-21T23:05:56.317 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 28/152 2024-04-21T23:05:56.394 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 29/152 2024-04-21T23:05:56.428 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-ceph-common-2:19.0.0-2405.g820a2f8c.el9. 30/152 2024-04-21T23:05:56.471 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-ceph-argparse-2:19.0.0-2405.g820a2f8c.el 31/152 2024-04-21T23:05:56.517 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cephfs-2:19.0.0-2405.g820a2f8c.el9.x86_6 32/152 2024-04-21T23:05:56.587 INFO:teuthology.orchestra.run.smithi111.stdout:(150/150): librados2-19.0.0-2405.g820a2f8c.el9. 3.9 MB/s | 3.4 MB 00:00 2024-04-21T23:05:56.589 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 33/152 2024-04-21T23:05:56.591 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-04-21T23:05:56.591 INFO:teuthology.orchestra.run.smithi111.stdout:Total 16 MB/s | 216 MB 00:13 2024-04-21T23:05:56.621 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 90/152 2024-04-21T23:05:56.640 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : luarocks-3.9.2-1.el9.noarch 34/152 2024-04-21T23:05:56.715 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 35/152 2024-04-21T23:05:56.858 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 36/152 2024-04-21T23:05:56.909 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 37/152 2024-04-21T23:05:56.918 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 91/152 2024-04-21T23:05:57.445 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-rsa-4.9-2.el9.noarch 38/152 2024-04-21T23:05:57.530 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 39/152 2024-04-21T23:05:57.567 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 40/152 2024-04-21T23:05:57.610 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 41/152 2024-04-21T23:05:57.648 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-04-21T23:05:57.648 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : librados-devel-2:19.0.0-2405.g820a2f8c.el9.x86_6 42/152 2024-04-21T23:05:57.681 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 43/152 2024-04-21T23:05:57.737 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 44/152 2024-04-21T23:05:57.771 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-04-21T23:05:57.771 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-04-21T23:05:57.809 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 45/152 2024-04-21T23:05:57.840 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 46/152 2024-04-21T23:05:57.876 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 47/152 2024-04-21T23:05:57.923 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 48/152 2024-04-21T23:05:57.967 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 49/152 2024-04-21T23:05:58.011 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 50/152 2024-04-21T23:05:58.053 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 51/152 2024-04-21T23:05:58.191 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 52/152 2024-04-21T23:05:58.241 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 53/152 2024-04-21T23:05:58.271 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 54/152 2024-04-21T23:05:58.273 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 92/152 2024-04-21T23:05:58.313 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 55/152 2024-04-21T23:05:58.349 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 93/152 2024-04-21T23:05:58.378 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 94/152 2024-04-21T23:05:58.408 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 56/152 2024-04-21T23:05:58.410 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 95/152 2024-04-21T23:05:58.446 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 57/152 2024-04-21T23:05:58.494 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 58/152 2024-04-21T23:05:58.528 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 59/152 2024-04-21T23:05:58.561 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 60/152 2024-04-21T23:05:58.645 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 61/152 2024-04-21T23:05:58.748 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 96/152 2024-04-21T23:05:58.772 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 62/152 2024-04-21T23:05:58.789 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : librbd1-2:19.0.0-2405.g820a2f8c.el9.x86_64 97/152 2024-04-21T23:05:58.802 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 63/152 2024-04-21T23:05:58.830 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 64/152 2024-04-21T23:05:58.864 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 65/152 2024-04-21T23:05:58.873 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librbd1-2:19.0.0-2405.g820a2f8c.el9.x86_64 97/152 2024-04-21T23:05:58.903 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-rbd-2:19.0.0-2405.g820a2f8c.el9.x86_64 98/152 2024-04-21T23:05:58.908 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : uuid-1.6.2-55.el9.x86_64 66/152 2024-04-21T23:05:58.939 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 99/152 2024-04-21T23:05:58.946 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 67/152 2024-04-21T23:05:58.977 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 68/152 2024-04-21T23:05:59.007 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 69/152 2024-04-21T23:05:59.050 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 70/152 2024-04-21T23:05:59.091 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 71/152 2024-04-21T23:05:59.141 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 72/152 2024-04-21T23:05:59.193 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 73/152 2024-04-21T23:05:59.281 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 74/152 2024-04-21T23:05:59.331 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 75/152 2024-04-21T23:05:59.362 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 76/152 2024-04-21T23:05:59.409 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 77/152 2024-04-21T23:05:59.438 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 100/152 2024-04-21T23:05:59.468 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-04-21T23:05:59.469 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-04-21T23:05:59.476 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : librgw2-2:19.0.0-2405.g820a2f8c.el9.x86_64 101/152 2024-04-21T23:05:59.510 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-packaging-20.9-5.el9.noarch 78/152 2024-04-21T23:05:59.556 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librgw2-2:19.0.0-2405.g820a2f8c.el9.x86_64 101/152 2024-04-21T23:05:59.565 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 79/152 2024-04-21T23:05:59.586 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-rgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 102/152 2024-04-21T23:06:00.313 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 80/152 2024-04-21T23:06:00.361 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 81/152 2024-04-21T23:06:00.395 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 82/152 2024-04-21T23:06:00.497 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 83/152 2024-04-21T23:06:00.553 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 84/152 2024-04-21T23:06:00.641 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 85/152 2024-04-21T23:06:00.705 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 86/152 2024-04-21T23:06:00.881 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : openblas-0.3.21-2.el9.x86_64 87/152 2024-04-21T23:06:00.911 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 88/152 2024-04-21T23:06:00.988 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 89/152 2024-04-21T23:06:01.646 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 90/152 2024-04-21T23:06:01.760 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 91/152 2024-04-21T23:06:01.999 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-common-2:19.0.0-2405.g820a2f8c.el9.x86_64 103/152 2024-04-21T23:06:02.032 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-common-2:19.0.0-2405.g820a2f8c.el9.x86_64 103/152 2024-04-21T23:06:02.606 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-common-2:19.0.0-2405.g820a2f8c.el9.x86_64 103/152 2024-04-21T23:06:02.644 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-base-2:19.0.0-2405.g820a2f8c.el9.x86_64 104/152 2024-04-21T23:06:02.736 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-base-2:19.0.0-2405.g820a2f8c.el9.x86_64 104/152 2024-04-21T23:06:02.736 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-21T23:06:02.736 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-21T23:06:02.736 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:06:02.770 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-selinux-2:19.0.0-2405.g820a2f8c.el9.x86_64 105/152 2024-04-21T23:06:03.013 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 92/152 2024-04-21T23:06:03.089 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 93/152 2024-04-21T23:06:03.117 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 94/152 2024-04-21T23:06:03.150 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 95/152 2024-04-21T23:06:03.461 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 96/152 2024-04-21T23:06:03.496 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : librbd1-2:19.0.0-2405.g820a2f8c.el9.x86_64 97/152 2024-04-21T23:06:03.587 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: librbd1-2:19.0.0-2405.g820a2f8c.el9.x86_64 97/152 2024-04-21T23:06:03.618 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-rbd-2:19.0.0-2405.g820a2f8c.el9.x86_64 98/152 2024-04-21T23:06:03.671 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 99/152 2024-04-21T23:06:04.147 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 100/152 2024-04-21T23:06:04.191 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : librgw2-2:19.0.0-2405.g820a2f8c.el9.x86_64 101/152 2024-04-21T23:06:04.261 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: librgw2-2:19.0.0-2405.g820a2f8c.el9.x86_64 101/152 2024-04-21T23:06:04.284 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-rgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 102/152 2024-04-21T23:06:05.211 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-04-21T23:06:05.298 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : thrift-0.15.0-2.el9.x86_64 1/152 2024-04-21T23:06:05.332 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 2/152 2024-04-21T23:06:05.383 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 3/152 2024-04-21T23:06:05.724 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 4/152 2024-04-21T23:06:05.850 INFO:teuthology.orchestra.run.smithi111.stdout: Upgrading : librados2-2:19.0.0-2405.g820a2f8c.el9.x86_64 5/152 2024-04-21T23:06:05.980 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librados2-2:19.0.0-2405.g820a2f8c.el9.x86_64 5/152 2024-04-21T23:06:06.017 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libcephfs2-2:19.0.0-2405.g820a2f8c.el9.x86_64 6/152 2024-04-21T23:06:06.109 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: libcephfs2-2:19.0.0-2405.g820a2f8c.el9.x86_64 6/152 2024-04-21T23:06:06.151 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-rados-2:19.0.0-2405.g820a2f8c.el9.x86_64 7/152 2024-04-21T23:06:06.180 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/152 2024-04-21T23:06:06.206 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/152 2024-04-21T23:06:06.364 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 10/152 2024-04-21T23:06:06.412 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : postgresql-13.14-1.el9.x86_64 11/152 2024-04-21T23:06:06.439 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libcephsqlite-2:19.0.0-2405.g820a2f8c.el9.x86_64 12/152 2024-04-21T23:06:06.543 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2405.g820a2f8c.el9.x86_64 12/152 2024-04-21T23:06:06.565 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libradosstriper1-2:19.0.0-2405.g820a2f8c.el9.x86 13/152 2024-04-21T23:06:06.581 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-common-2:19.0.0-2405.g820a2f8c.el9.x86_64 103/152 2024-04-21T23:06:06.628 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2405.g820a2f8c.el9.x86 13/152 2024-04-21T23:06:06.630 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-common-2:19.0.0-2405.g820a2f8c.el9.x86_64 103/152 2024-04-21T23:06:06.650 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-04-21T23:06:06.670 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-04-21T23:06:06.850 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 15/152 2024-04-21T23:06:06.883 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 15/152 2024-04-21T23:06:06.989 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : re2-1:20211101-3.el9.x86_64 16/152 2024-04-21T23:06:07.080 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 17/152 2024-04-21T23:06:07.115 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 18/152 2024-04-21T23:06:07.150 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : liboath-2.6.7-2.el9.x86_64 19/152 2024-04-21T23:06:07.167 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-common-2:19.0.0-2405.g820a2f8c.el9.x86_64 103/152 2024-04-21T23:06:07.216 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-base-2:19.0.0-2405.g820a2f8c.el9.x86_64 104/152 2024-04-21T23:06:07.225 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 20/152 2024-04-21T23:06:07.266 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 21/152 2024-04-21T23:06:07.308 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-base-2:19.0.0-2405.g820a2f8c.el9.x86_64 104/152 2024-04-21T23:06:07.308 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-21T23:06:07.308 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-21T23:06:07.308 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-04-21T23:06:07.309 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 22/152 2024-04-21T23:06:07.334 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-selinux-2:19.0.0-2405.g820a2f8c.el9.x86_64 105/152 2024-04-21T23:06:07.342 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 23/152 2024-04-21T23:06:07.370 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : lua-5.4.4-4.el9.x86_64 24/152 2024-04-21T23:06:07.440 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 25/152 2024-04-21T23:06:07.493 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 26/152 2024-04-21T23:06:07.523 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 27/152 2024-04-21T23:06:07.566 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 28/152 2024-04-21T23:06:07.644 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 29/152 2024-04-21T23:06:07.677 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-ceph-common-2:19.0.0-2405.g820a2f8c.el9. 30/152 2024-04-21T23:06:07.720 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-ceph-argparse-2:19.0.0-2405.g820a2f8c.el 31/152 2024-04-21T23:06:07.749 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cephfs-2:19.0.0-2405.g820a2f8c.el9.x86_6 32/152 2024-04-21T23:06:07.822 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 33/152 2024-04-21T23:06:07.864 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : luarocks-3.9.2-1.el9.noarch 34/152 2024-04-21T23:06:07.940 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 35/152 2024-04-21T23:06:08.084 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 36/152 2024-04-21T23:06:08.142 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 37/152 2024-04-21T23:06:08.684 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-rsa-4.9-2.el9.noarch 38/152 2024-04-21T23:06:08.783 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 39/152 2024-04-21T23:06:08.816 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 40/152 2024-04-21T23:06:08.859 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 41/152 2024-04-21T23:06:08.897 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : librados-devel-2:19.0.0-2405.g820a2f8c.el9.x86_6 42/152 2024-04-21T23:06:08.930 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 43/152 2024-04-21T23:06:08.986 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 44/152 2024-04-21T23:06:09.060 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 45/152 2024-04-21T23:06:09.098 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 46/152 2024-04-21T23:06:09.134 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 47/152 2024-04-21T23:06:09.181 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 48/152 2024-04-21T23:06:09.217 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 49/152 2024-04-21T23:06:09.260 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 50/152 2024-04-21T23:06:09.302 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 51/152 2024-04-21T23:06:09.442 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 52/152 2024-04-21T23:06:09.482 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 53/152 2024-04-21T23:06:09.512 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 54/152 2024-04-21T23:06:09.554 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 55/152 2024-04-21T23:06:09.651 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 56/152 2024-04-21T23:06:09.695 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 57/152 2024-04-21T23:06:09.735 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 58/152 2024-04-21T23:06:09.761 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 59/152 2024-04-21T23:06:09.786 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 60/152 2024-04-21T23:06:09.863 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 61/152 2024-04-21T23:06:09.990 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 62/152 2024-04-21T23:06:10.026 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 63/152 2024-04-21T23:06:10.055 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 64/152 2024-04-21T23:06:10.089 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 65/152 2024-04-21T23:06:10.133 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : uuid-1.6.2-55.el9.x86_64 66/152 2024-04-21T23:06:10.179 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 67/152 2024-04-21T23:06:10.219 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 68/152 2024-04-21T23:06:10.248 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 69/152 2024-04-21T23:06:10.283 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 70/152 2024-04-21T23:06:10.316 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 71/152 2024-04-21T23:06:10.368 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 72/152 2024-04-21T23:06:10.401 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 73/152 2024-04-21T23:06:10.489 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 74/152 2024-04-21T23:06:10.531 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 75/152 2024-04-21T23:06:10.570 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 76/152 2024-04-21T23:06:10.618 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 77/152 2024-04-21T23:06:10.718 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-packaging-20.9-5.el9.noarch 78/152 2024-04-21T23:06:10.773 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 79/152 2024-04-21T23:06:11.523 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 80/152 2024-04-21T23:06:11.570 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 81/152 2024-04-21T23:06:11.603 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 82/152 2024-04-21T23:06:11.707 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 83/152 2024-04-21T23:06:11.762 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 84/152 2024-04-21T23:06:11.852 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 85/152 2024-04-21T23:06:11.889 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 86/152 2024-04-21T23:06:12.065 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : openblas-0.3.21-2.el9.x86_64 87/152 2024-04-21T23:06:12.095 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 88/152 2024-04-21T23:06:12.163 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 89/152 2024-04-21T23:06:12.817 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 90/152 2024-04-21T23:06:12.944 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 91/152 2024-04-21T23:06:14.178 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 92/152 2024-04-21T23:06:14.257 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 93/152 2024-04-21T23:06:14.285 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 94/152 2024-04-21T23:06:14.318 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 95/152 2024-04-21T23:06:14.628 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 96/152 2024-04-21T23:06:14.635 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2405.g820a2f8c.el9.x86_64 105/152 2024-04-21T23:06:14.635 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /sys 2024-04-21T23:06:14.635 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /proc 2024-04-21T23:06:14.635 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /mnt 2024-04-21T23:06:14.636 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /var/tmp 2024-04-21T23:06:14.636 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /home 2024-04-21T23:06:14.636 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /root 2024-04-21T23:06:14.636 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /tmp 2024-04-21T23:06:14.636 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:06:14.664 INFO:teuthology.orchestra.run.smithi111.stdout: Upgrading : librbd1-2:19.0.0-2405.g820a2f8c.el9.x86_64 97/152 2024-04-21T23:06:14.766 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librbd1-2:19.0.0-2405.g820a2f8c.el9.x86_64 97/152 2024-04-21T23:06:14.794 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-rbd-2:19.0.0-2405.g820a2f8c.el9.x86_64 98/152 2024-04-21T23:06:14.815 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-volume-2:19.0.0-2405.g820a2f8c.el9.noarch 106/152 2024-04-21T23:06:14.830 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 99/152 2024-04-21T23:06:14.831 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-volume-2:19.0.0-2405.g820a2f8c.el9.noarch 106/152 2024-04-21T23:06:14.831 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T23:06:14.831 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-21T23:06:14.831 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:06:15.299 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 100/152 2024-04-21T23:06:15.326 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : librgw2-2:19.0.0-2405.g820a2f8c.el9.x86_64 101/152 2024-04-21T23:06:15.396 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librgw2-2:19.0.0-2405.g820a2f8c.el9.x86_64 101/152 2024-04-21T23:06:15.419 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-rgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 102/152 2024-04-21T23:06:16.376 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-osd-2:19.0.0-2405.g820a2f8c.el9.x86_64 107/152 2024-04-21T23:06:16.403 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-osd-2:19.0.0-2405.g820a2f8c.el9.x86_64 107/152 2024-04-21T23:06:16.403 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T23:06:16.403 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-21T23:06:16.403 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-21T23:06:16.404 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-21T23:06:16.404 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:06:16.645 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mds-2:19.0.0-2405.g820a2f8c.el9.x86_64 108/152 2024-04-21T23:06:16.672 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mds-2:19.0.0-2405.g820a2f8c.el9.x86_64 108/152 2024-04-21T23:06:16.672 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T23:06:16.672 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-21T23:06:16.673 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-21T23:06:16.673 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-21T23:06:16.673 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:06:17.137 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mon-2:19.0.0-2405.g820a2f8c.el9.x86_64 109/152 2024-04-21T23:06:17.161 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mon-2:19.0.0-2405.g820a2f8c.el9.x86_64 109/152 2024-04-21T23:06:17.161 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T23:06:17.162 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-21T23:06:17.162 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-21T23:06:17.162 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-21T23:06:17.162 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:06:17.277 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : automake-1.16.2-8.el9.noarch 110/152 2024-04-21T23:06:17.336 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-ply-3.11-14.el9.noarch 111/152 2024-04-21T23:06:17.399 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 112/152 2024-04-21T23:06:17.604 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 113/152 2024-04-21T23:06:17.656 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 114/152 2024-04-21T23:06:17.664 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-common-2:19.0.0-2405.g820a2f8c.el9.x86_64 103/152 2024-04-21T23:06:17.699 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-common-2:19.0.0-2405.g820a2f8c.el9.x86_64 103/152 2024-04-21T23:06:17.738 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 115/152 2024-04-21T23:06:17.843 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 116/152 2024-04-21T23:06:18.005 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 117/152 2024-04-21T23:06:18.047 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 118/152 2024-04-21T23:06:18.079 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 119/152 2024-04-21T23:06:18.106 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : fuse-2.9.9-15.el9.x86_64 120/152 2024-04-21T23:06:18.235 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-common-2:19.0.0-2405.g820a2f8c.el9.x86_64 103/152 2024-04-21T23:06:18.277 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-base-2:19.0.0-2405.g820a2f8c.el9.x86_64 104/152 2024-04-21T23:06:18.367 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-base-2:19.0.0-2405.g820a2f8c.el9.x86_64 104/152 2024-04-21T23:06:18.368 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-21T23:06:18.368 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-21T23:06:18.368 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-21T23:06:18.386 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-selinux-2:19.0.0-2405.g820a2f8c.el9.x86_64 105/152 2024-04-21T23:06:18.570 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: cephadm-2:19.0.0-2405.g820a2f8c.el9.noarch 121/152 2024-04-21T23:06:18.597 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : cephadm-2:19.0.0-2405.g820a2f8c.el9.noarch 121/152 2024-04-21T23:06:18.598 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2405.g820a2f8c.el9.x86_64 105/152 2024-04-21T23:06:18.599 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /sys 2024-04-21T23:06:18.599 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /proc 2024-04-21T23:06:18.599 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /mnt 2024-04-21T23:06:18.599 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /var/tmp 2024-04-21T23:06:18.599 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /home 2024-04-21T23:06:18.599 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /root 2024-04-21T23:06:18.599 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /tmp 2024-04-21T23:06:18.599 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-04-21T23:06:18.629 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2405.g820a2f8c.e 122/152 2024-04-21T23:06:18.694 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2405.g820a2f8c. 123/152 2024-04-21T23:06:18.720 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2405.g820a2f8c.el9.noa 124/152 2024-04-21T23:06:18.770 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-volume-2:19.0.0-2405.g820a2f8c.el9.noarch 106/152 2024-04-21T23:06:18.785 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-volume-2:19.0.0-2405.g820a2f8c.el9.noarch 106/152 2024-04-21T23:06:18.785 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T23:06:18.785 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-21T23:06:18.785 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-04-21T23:06:19.816 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2405.g820a2f8c.el9.noa 124/152 2024-04-21T23:06:19.846 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2405.g820a2f8c.el9.n 125/152 2024-04-21T23:06:20.281 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-osd-2:19.0.0-2405.g820a2f8c.el9.x86_64 107/152 2024-04-21T23:06:20.306 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-osd-2:19.0.0-2405.g820a2f8c.el9.x86_64 107/152 2024-04-21T23:06:20.306 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T23:06:20.306 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-21T23:06:20.306 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-21T23:06:20.306 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-21T23:06:20.307 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-04-21T23:06:20.531 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mds-2:19.0.0-2405.g820a2f8c.el9.x86_64 108/152 2024-04-21T23:06:20.553 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mds-2:19.0.0-2405.g820a2f8c.el9.x86_64 108/152 2024-04-21T23:06:20.553 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T23:06:20.553 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-21T23:06:20.553 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-21T23:06:20.554 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-21T23:06:20.554 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-04-21T23:06:20.866 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2405.g820a2f8c.el9.n 125/152 2024-04-21T23:06:20.897 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2405.g820 126/152 2024-04-21T23:06:20.990 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mon-2:19.0.0-2405.g820a2f8c.el9.x86_64 109/152 2024-04-21T23:06:21.010 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2405.g820 126/152 2024-04-21T23:06:21.012 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mon-2:19.0.0-2405.g820a2f8c.el9.x86_64 109/152 2024-04-21T23:06:21.012 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T23:06:21.012 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-21T23:06:21.013 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-21T23:06:21.013 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-21T23:06:21.013 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-04-21T23:06:21.112 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : automake-1.16.2-8.el9.noarch 110/152 2024-04-21T23:06:21.168 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-ply-3.11-14.el9.noarch 111/152 2024-04-21T23:06:21.184 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2405.g820a2f8c.el 127/152 2024-04-21T23:06:21.216 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-2:19.0.0-2405.g820a2f8c.el9.x86_64 128/152 2024-04-21T23:06:21.224 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 112/152 2024-04-21T23:06:21.240 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2405.g820a2f8c.el9.x86_64 128/152 2024-04-21T23:06:21.240 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T23:06:21.240 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-21T23:06:21.240 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-21T23:06:21.240 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-21T23:06:21.240 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:06:21.291 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-rook-2:19.0.0-2405.g820a2f8c.el9.noarch 129/152 2024-04-21T23:06:21.303 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2405.g820a2f8c.el9.noarch 129/152 2024-04-21T23:06:21.405 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-2:19.0.0-2405.g820a2f8c.el9.x86_64 130/152 2024-04-21T23:06:21.426 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 113/152 2024-04-21T23:06:21.458 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-fuse-2:19.0.0-2405.g820a2f8c.el9.x86_64 131/152 2024-04-21T23:06:21.474 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 114/152 2024-04-21T23:06:21.552 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 115/152 2024-04-21T23:06:21.666 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 116/152 2024-04-21T23:06:21.797 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 117/152 2024-04-21T23:06:21.849 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 118/152 2024-04-21T23:06:21.881 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 119/152 2024-04-21T23:06:21.908 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : fuse-2.9.9-15.el9.x86_64 120/152 2024-04-21T23:06:22.319 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: cephadm-2:19.0.0-2405.g820a2f8c.el9.noarch 121/152 2024-04-21T23:06:22.349 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : cephadm-2:19.0.0-2405.g820a2f8c.el9.noarch 121/152 2024-04-21T23:06:22.380 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2405.g820a2f8c.e 122/152 2024-04-21T23:06:22.406 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libtool-2.4.6-45.el9.x86_64 132/152 2024-04-21T23:06:22.436 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-radosgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 133/152 2024-04-21T23:06:22.442 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2405.g820a2f8c. 123/152 2024-04-21T23:06:22.460 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 133/152 2024-04-21T23:06:22.460 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T23:06:22.460 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-21T23:06:22.460 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-21T23:06:22.461 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-21T23:06:22.461 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:06:22.471 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2405.g820a2f8c.el9.noa 124/152 2024-04-21T23:06:22.503 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2405.g820a2 134/152 2024-04-21T23:06:22.527 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2405.g820a2 134/152 2024-04-21T23:06:22.527 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T23:06:22.527 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-21T23:06:22.527 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:06:22.578 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : cephfs-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 135/152 2024-04-21T23:06:22.602 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 135/152 2024-04-21T23:06:22.602 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T23:06:22.603 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-21T23:06:22.603 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-21T23:06:22.603 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-21T23:06:22.603 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:06:22.921 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : rbd-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 136/152 2024-04-21T23:06:22.945 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 136/152 2024-04-21T23:06:22.945 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T23:06:22.945 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-21T23:06:22.945 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-21T23:06:22.945 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-21T23:06:22.945 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:06:23.501 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2405.g820a2f8c.el9.noa 124/152 2024-04-21T23:06:23.532 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2405.g820a2f8c.el9.n 125/152 2024-04-21T23:06:24.492 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2405.g820a2f8c.el9.n 125/152 2024-04-21T23:06:24.515 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2405.g820 126/152 2024-04-21T23:06:24.618 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2405.g820 126/152 2024-04-21T23:06:24.787 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2405.g820a2f8c.el 127/152 2024-04-21T23:06:24.817 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-2:19.0.0-2405.g820a2f8c.el9.x86_64 128/152 2024-04-21T23:06:24.839 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2405.g820a2f8c.el9.x86_64 128/152 2024-04-21T23:06:24.840 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T23:06:24.840 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-21T23:06:24.840 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-21T23:06:24.840 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-21T23:06:24.840 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-04-21T23:06:24.893 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-rook-2:19.0.0-2405.g820a2f8c.el9.noarch 129/152 2024-04-21T23:06:24.904 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2405.g820a2f8c.el9.noarch 129/152 2024-04-21T23:06:25.002 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-2:19.0.0-2405.g820a2f8c.el9.x86_64 130/152 2024-04-21T23:06:25.058 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-fuse-2:19.0.0-2405.g820a2f8c.el9.x86_64 131/152 2024-04-21T23:06:25.951 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libtool-2.4.6-45.el9.x86_64 132/152 2024-04-21T23:06:25.979 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-radosgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 133/152 2024-04-21T23:06:26.002 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 133/152 2024-04-21T23:06:26.002 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T23:06:26.002 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-21T23:06:26.002 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-21T23:06:26.002 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-21T23:06:26.002 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-04-21T23:06:26.047 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2405.g820a2 134/152 2024-04-21T23:06:26.068 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2405.g820a2 134/152 2024-04-21T23:06:26.069 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T23:06:26.069 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-21T23:06:26.069 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-04-21T23:06:26.114 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : cephfs-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 135/152 2024-04-21T23:06:26.136 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 135/152 2024-04-21T23:06:26.136 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T23:06:26.136 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-21T23:06:26.136 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-21T23:06:26.136 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-21T23:06:26.136 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-04-21T23:06:26.439 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : rbd-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 136/152 2024-04-21T23:06:26.462 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 136/152 2024-04-21T23:06:26.462 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T23:06:26.462 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-21T23:06:26.462 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-21T23:06:26.462 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-21T23:06:26.462 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-04-21T23:06:27.733 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-test-2:19.0.0-2405.g820a2f8c.el9.x86_64 137/152 2024-04-21T23:06:27.775 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : rbd-fuse-2:19.0.0-2405.g820a2f8c.el9.x86_64 138/152 2024-04-21T23:06:27.817 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : rbd-nbd-2:19.0.0-2405.g820a2f8c.el9.x86_64 139/152 2024-04-21T23:06:27.847 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 140/152 2024-04-21T23:06:27.875 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 141/152 2024-04-21T23:06:28.062 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : cephfs-shell-2:19.0.0-2405.g820a2f8c.el9.x86_64 142/152 2024-04-21T23:06:28.104 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 143/152 2024-04-21T23:06:28.133 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libcephfs-devel-2:19.0.0-2405.g820a2f8c.el9.x86_ 144/152 2024-04-21T23:06:28.776 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-21T23:06:28.814 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-21T23:06:28.835 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-21T23:06:29.676 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2405.g820a2f8c.el9.x86_64 105/152 2024-04-21T23:06:29.676 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /sys 2024-04-21T23:06:29.676 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /proc 2024-04-21T23:06:29.676 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /mnt 2024-04-21T23:06:29.677 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /var/tmp 2024-04-21T23:06:29.677 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /home 2024-04-21T23:06:29.677 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /root 2024-04-21T23:06:29.677 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /tmp 2024-04-21T23:06:29.677 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-21T23:06:29.718 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : cephfs-top-2:19.0.0-2405.g820a2f8c.el9.noarch 146/152 2024-04-21T23:06:29.802 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 147/152 2024-04-21T23:06:29.836 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 148/152 2024-04-21T23:06:29.857 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-volume-2:19.0.0-2405.g820a2f8c.el9.noarch 106/152 2024-04-21T23:06:29.873 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-volume-2:19.0.0-2405.g820a2f8c.el9.noarch 106/152 2024-04-21T23:06:29.873 INFO:teuthology.orchestra.run.smithi111.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T23:06:29.874 INFO:teuthology.orchestra.run.smithi111.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-21T23:06:29.874 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-21T23:06:29.945 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : flex-2.6.4-9.el9.x86_64 149/152 2024-04-21T23:06:29.990 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : bison-3.7.4-5.el9.x86_64 150/152 2024-04-21T23:06:29.990 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-04-21T23:06:30.074 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-04-21T23:06:30.074 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 152/152 2024-04-21T23:06:31.001 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-test-2:19.0.0-2405.g820a2f8c.el9.x86_64 137/152 2024-04-21T23:06:31.051 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : rbd-fuse-2:19.0.0-2405.g820a2f8c.el9.x86_64 138/152 2024-04-21T23:06:31.100 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : rbd-nbd-2:19.0.0-2405.g820a2f8c.el9.x86_64 139/152 2024-04-21T23:06:31.132 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 140/152 2024-04-21T23:06:31.159 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 141/152 2024-04-21T23:06:31.345 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : cephfs-shell-2:19.0.0-2405.g820a2f8c.el9.x86_64 142/152 2024-04-21T23:06:31.362 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-osd-2:19.0.0-2405.g820a2f8c.el9.x86_64 107/152 2024-04-21T23:06:31.387 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-osd-2:19.0.0-2405.g820a2f8c.el9.x86_64 107/152 2024-04-21T23:06:31.387 INFO:teuthology.orchestra.run.smithi111.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T23:06:31.387 INFO:teuthology.orchestra.run.smithi111.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-21T23:06:31.387 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-21T23:06:31.387 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-21T23:06:31.387 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-21T23:06:31.388 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 143/152 2024-04-21T23:06:31.418 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libcephfs-devel-2:19.0.0-2405.g820a2f8c.el9.x86_ 144/152 2024-04-21T23:06:31.611 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mds-2:19.0.0-2405.g820a2f8c.el9.x86_64 108/152 2024-04-21T23:06:31.634 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mds-2:19.0.0-2405.g820a2f8c.el9.x86_64 108/152 2024-04-21T23:06:31.634 INFO:teuthology.orchestra.run.smithi111.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T23:06:31.634 INFO:teuthology.orchestra.run.smithi111.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-21T23:06:31.634 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-21T23:06:31.634 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-21T23:06:31.634 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-21T23:06:32.009 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-21T23:06:32.048 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-21T23:06:32.068 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-21T23:06:32.078 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mon-2:19.0.0-2405.g820a2f8c.el9.x86_64 109/152 2024-04-21T23:06:32.101 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mon-2:19.0.0-2405.g820a2f8c.el9.x86_64 109/152 2024-04-21T23:06:32.101 INFO:teuthology.orchestra.run.smithi111.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T23:06:32.101 INFO:teuthology.orchestra.run.smithi111.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-21T23:06:32.101 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-21T23:06:32.101 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-21T23:06:32.101 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-21T23:06:32.208 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : automake-1.16.2-8.el9.noarch 110/152 2024-04-21T23:06:32.266 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-ply-3.11-14.el9.noarch 111/152 2024-04-21T23:06:32.321 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 112/152 2024-04-21T23:06:32.517 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 113/152 2024-04-21T23:06:32.571 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 114/152 2024-04-21T23:06:32.650 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 115/152 2024-04-21T23:06:32.747 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 116/152 2024-04-21T23:06:32.846 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : cephfs-top-2:19.0.0-2405.g820a2f8c.el9.noarch 146/152 2024-04-21T23:06:32.887 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 117/152 2024-04-21T23:06:32.927 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 147/152 2024-04-21T23:06:32.929 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 118/152 2024-04-21T23:06:32.961 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 119/152 2024-04-21T23:06:32.976 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 148/152 2024-04-21T23:06:32.989 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : fuse-2.9.9-15.el9.x86_64 120/152 2024-04-21T23:06:33.059 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : flex-2.6.4-9.el9.x86_64 149/152 2024-04-21T23:06:33.132 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : bison-3.7.4-5.el9.x86_64 150/152 2024-04-21T23:06:33.132 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-04-21T23:06:33.207 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-04-21T23:06:33.207 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 152/152 2024-04-21T23:06:33.399 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: cephadm-2:19.0.0-2405.g820a2f8c.el9.noarch 121/152 2024-04-21T23:06:33.429 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : cephadm-2:19.0.0-2405.g820a2f8c.el9.noarch 121/152 2024-04-21T23:06:33.461 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2405.g820a2f8c.e 122/152 2024-04-21T23:06:33.523 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2405.g820a2f8c. 123/152 2024-04-21T23:06:33.552 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2405.g820a2f8c.el9.noa 124/152 2024-04-21T23:06:34.584 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2405.g820a2f8c.el9.noa 124/152 2024-04-21T23:06:34.620 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2405.g820a2f8c.el9.n 125/152 2024-04-21T23:06:35.585 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2405.g820a2f8c.el9.n 125/152 2024-04-21T23:06:35.612 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2405.g820 126/152 2024-04-21T23:06:35.715 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2405.g820 126/152 2024-04-21T23:06:35.726 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 152/152 2024-04-21T23:06:35.726 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-2:19.0.0-2405.g820a2f8c.el9.x86_64 1/152 2024-04-21T23:06:35.726 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-base-2:19.0.0-2405.g820a2f8c.el9.x86_64 2/152 2024-04-21T23:06:35.726 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-common-2:19.0.0-2405.g820a2f8c.el9.x86_64 3/152 2024-04-21T23:06:35.727 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-fuse-2:19.0.0-2405.g820a2f8c.el9.x86_64 4/152 2024-04-21T23:06:35.727 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2405.g820a2 5/152 2024-04-21T23:06:35.727 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mds-2:19.0.0-2405.g820a2f8c.el9.x86_64 6/152 2024-04-21T23:06:35.727 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-2:19.0.0-2405.g820a2f8c.el9.x86_64 7/152 2024-04-21T23:06:35.727 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mon-2:19.0.0-2405.g820a2f8c.el9.x86_64 8/152 2024-04-21T23:06:35.727 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-osd-2:19.0.0-2405.g820a2f8c.el9.x86_64 9/152 2024-04-21T23:06:35.727 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-radosgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 10/152 2024-04-21T23:06:35.727 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-selinux-2:19.0.0-2405.g820a2f8c.el9.x86_64 11/152 2024-04-21T23:06:35.727 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-test-2:19.0.0-2405.g820a2f8c.el9.x86_64 12/152 2024-04-21T23:06:35.727 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : cephfs-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 13/152 2024-04-21T23:06:35.727 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : cephfs-shell-2:19.0.0-2405.g820a2f8c.el9.x86_64 14/152 2024-04-21T23:06:35.727 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libcephfs-devel-2:19.0.0-2405.g820a2f8c.el9.x86_ 15/152 2024-04-21T23:06:35.727 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libcephfs2-2:19.0.0-2405.g820a2f8c.el9.x86_64 16/152 2024-04-21T23:06:35.728 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libcephsqlite-2:19.0.0-2405.g820a2f8c.el9.x86_64 17/152 2024-04-21T23:06:35.728 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librados-devel-2:19.0.0-2405.g820a2f8c.el9.x86_6 18/152 2024-04-21T23:06:35.728 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libradosstriper1-2:19.0.0-2405.g820a2f8c.el9.x86 19/152 2024-04-21T23:06:35.728 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librgw2-2:19.0.0-2405.g820a2f8c.el9.x86_64 20/152 2024-04-21T23:06:35.728 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2405.g820a2f8c.el 21/152 2024-04-21T23:06:35.728 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-ceph-common-2:19.0.0-2405.g820a2f8c.el9. 22/152 2024-04-21T23:06:35.728 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cephfs-2:19.0.0-2405.g820a2f8c.el9.x86_6 23/152 2024-04-21T23:06:35.728 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-rados-2:19.0.0-2405.g820a2f8c.el9.x86_64 24/152 2024-04-21T23:06:35.728 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-rbd-2:19.0.0-2405.g820a2f8c.el9.x86_64 25/152 2024-04-21T23:06:35.728 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-rgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 26/152 2024-04-21T23:06:35.728 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : rbd-fuse-2:19.0.0-2405.g820a2f8c.el9.x86_64 27/152 2024-04-21T23:06:35.740 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : rbd-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 28/152 2024-04-21T23:06:35.740 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : rbd-nbd-2:19.0.0-2405.g820a2f8c.el9.x86_64 29/152 2024-04-21T23:06:35.741 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2405.g820a2f8c. 30/152 2024-04-21T23:06:35.741 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2405.g820a2f8c.el9.noa 31/152 2024-04-21T23:06:35.741 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2405.g820a2f8c.el9.n 32/152 2024-04-21T23:06:35.741 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2405.g820 33/152 2024-04-21T23:06:35.741 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2405.g820a2f8c.el 34/152 2024-04-21T23:06:35.741 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2405.g820a2f8c.el9.noarch 35/152 2024-04-21T23:06:35.741 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2405.g820a2f8c.e 36/152 2024-04-21T23:06:35.741 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-volume-2:19.0.0-2405.g820a2f8c.el9.noarch 37/152 2024-04-21T23:06:35.741 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : cephadm-2:19.0.0-2405.g820a2f8c.el9.noarch 38/152 2024-04-21T23:06:35.741 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : cephfs-top-2:19.0.0-2405.g820a2f8c.el9.noarch 39/152 2024-04-21T23:06:35.741 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/152 2024-04-21T23:06:35.741 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/152 2024-04-21T23:06:35.741 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/152 2024-04-21T23:06:35.741 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/152 2024-04-21T23:06:35.741 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/152 2024-04-21T23:06:35.742 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/152 2024-04-21T23:06:35.742 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/152 2024-04-21T23:06:35.742 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/152 2024-04-21T23:06:35.742 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/152 2024-04-21T23:06:35.742 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/152 2024-04-21T23:06:35.742 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/152 2024-04-21T23:06:35.742 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/152 2024-04-21T23:06:35.742 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/152 2024-04-21T23:06:35.742 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/152 2024-04-21T23:06:35.742 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/152 2024-04-21T23:06:35.742 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/152 2024-04-21T23:06:35.742 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/152 2024-04-21T23:06:35.742 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/152 2024-04-21T23:06:35.742 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/152 2024-04-21T23:06:35.743 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/152 2024-04-21T23:06:35.743 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/152 2024-04-21T23:06:35.743 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/152 2024-04-21T23:06:35.743 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/152 2024-04-21T23:06:35.743 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/152 2024-04-21T23:06:35.743 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/152 2024-04-21T23:06:35.743 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/152 2024-04-21T23:06:35.743 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/152 2024-04-21T23:06:35.743 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/152 2024-04-21T23:06:35.743 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 68/152 2024-04-21T23:06:35.743 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/152 2024-04-21T23:06:35.743 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/152 2024-04-21T23:06:35.743 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/152 2024-04-21T23:06:35.743 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/152 2024-04-21T23:06:35.743 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/152 2024-04-21T23:06:35.744 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/152 2024-04-21T23:06:35.744 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/152 2024-04-21T23:06:35.744 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/152 2024-04-21T23:06:35.744 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/152 2024-04-21T23:06:35.744 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/152 2024-04-21T23:06:35.744 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/152 2024-04-21T23:06:35.744 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/152 2024-04-21T23:06:35.744 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/152 2024-04-21T23:06:35.744 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/152 2024-04-21T23:06:35.744 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/152 2024-04-21T23:06:35.744 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/152 2024-04-21T23:06:35.744 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/152 2024-04-21T23:06:35.744 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/152 2024-04-21T23:06:35.744 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/152 2024-04-21T23:06:35.745 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/152 2024-04-21T23:06:35.745 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/152 2024-04-21T23:06:35.745 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/152 2024-04-21T23:06:35.745 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/152 2024-04-21T23:06:35.745 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/152 2024-04-21T23:06:35.745 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/152 2024-04-21T23:06:35.745 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/152 2024-04-21T23:06:35.745 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/152 2024-04-21T23:06:35.745 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/152 2024-04-21T23:06:35.745 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/152 2024-04-21T23:06:35.745 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/152 2024-04-21T23:06:35.745 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/152 2024-04-21T23:06:35.745 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/152 2024-04-21T23:06:35.745 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/152 2024-04-21T23:06:35.746 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/152 2024-04-21T23:06:35.746 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/152 2024-04-21T23:06:35.746 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/152 2024-04-21T23:06:35.746 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/152 2024-04-21T23:06:35.746 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/152 2024-04-21T23:06:35.746 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/152 2024-04-21T23:06:35.746 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/152 2024-04-21T23:06:35.746 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/152 2024-04-21T23:06:35.746 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/152 2024-04-21T23:06:35.746 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/152 2024-04-21T23:06:35.746 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/152 2024-04-21T23:06:35.746 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/152 2024-04-21T23:06:35.746 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/152 2024-04-21T23:06:35.746 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/152 2024-04-21T23:06:35.747 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/152 2024-04-21T23:06:35.747 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/152 2024-04-21T23:06:35.747 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/152 2024-04-21T23:06:35.747 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/152 2024-04-21T23:06:35.747 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/152 2024-04-21T23:06:35.747 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/152 2024-04-21T23:06:35.747 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 122/152 2024-04-21T23:06:35.747 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/152 2024-04-21T23:06:35.747 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/152 2024-04-21T23:06:35.747 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/152 2024-04-21T23:06:35.747 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/152 2024-04-21T23:06:35.747 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/152 2024-04-21T23:06:35.747 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/152 2024-04-21T23:06:35.747 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/152 2024-04-21T23:06:35.748 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/152 2024-04-21T23:06:35.748 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/152 2024-04-21T23:06:35.748 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/152 2024-04-21T23:06:35.748 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/152 2024-04-21T23:06:35.748 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/152 2024-04-21T23:06:35.748 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/152 2024-04-21T23:06:35.748 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/152 2024-04-21T23:06:35.748 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/152 2024-04-21T23:06:35.748 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/152 2024-04-21T23:06:35.748 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/152 2024-04-21T23:06:35.748 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/152 2024-04-21T23:06:35.748 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/152 2024-04-21T23:06:35.748 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/152 2024-04-21T23:06:35.748 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/152 2024-04-21T23:06:35.748 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/152 2024-04-21T23:06:35.748 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/152 2024-04-21T23:06:35.748 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/152 2024-04-21T23:06:35.749 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/152 2024-04-21T23:06:35.749 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/152 2024-04-21T23:06:35.749 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librados2-2:19.0.0-2405.g820a2f8c.el9.x86_64 149/152 2024-04-21T23:06:35.749 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/152 2024-04-21T23:06:35.749 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librbd1-2:19.0.0-2405.g820a2f8c.el9.x86_64 151/152 2024-04-21T23:06:35.876 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2405.g820a2f8c.el 127/152 2024-04-21T23:06:35.935 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-2:19.0.0-2405.g820a2f8c.el9.x86_64 128/152 2024-04-21T23:06:35.959 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2405.g820a2f8c.el9.x86_64 128/152 2024-04-21T23:06:36.007 INFO:teuthology.orchestra.run.smithi111.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T23:06:36.007 INFO:teuthology.orchestra.run.smithi111.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-21T23:06:36.007 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-21T23:06:36.007 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-21T23:06:36.007 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-21T23:06:36.008 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-rook-2:19.0.0-2405.g820a2f8c.el9.noarch 129/152 2024-04-21T23:06:36.018 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2405.g820a2f8c.el9.noarch 129/152 2024-04-21T23:06:36.118 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-2:19.0.0-2405.g820a2f8c.el9.x86_64 130/152 2024-04-21T23:06:36.163 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-fuse-2:19.0.0-2405.g820a2f8c.el9.x86_64 131/152 2024-04-21T23:06:37.053 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libtool-2.4.6-45.el9.x86_64 132/152 2024-04-21T23:06:37.076 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-radosgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 133/152 2024-04-21T23:06:37.098 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 133/152 2024-04-21T23:06:37.098 INFO:teuthology.orchestra.run.smithi111.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T23:06:37.098 INFO:teuthology.orchestra.run.smithi111.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-21T23:06:37.098 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-21T23:06:37.098 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-21T23:06:37.098 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-21T23:06:37.151 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2405.g820a2 134/152 2024-04-21T23:06:37.173 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2405.g820a2 134/152 2024-04-21T23:06:37.173 INFO:teuthology.orchestra.run.smithi111.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T23:06:37.174 INFO:teuthology.orchestra.run.smithi111.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-21T23:06:37.174 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-21T23:06:37.219 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : cephfs-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 135/152 2024-04-21T23:06:37.241 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 135/152 2024-04-21T23:06:37.241 INFO:teuthology.orchestra.run.smithi111.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T23:06:37.241 INFO:teuthology.orchestra.run.smithi111.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-21T23:06:37.241 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-21T23:06:37.241 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-21T23:06:37.241 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-21T23:06:37.544 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : rbd-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 136/152 2024-04-21T23:06:37.567 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 136/152 2024-04-21T23:06:37.567 INFO:teuthology.orchestra.run.smithi111.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T23:06:37.567 INFO:teuthology.orchestra.run.smithi111.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-21T23:06:37.567 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-21T23:06:37.567 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-21T23:06:37.567 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-21T23:06:37.750 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/152 2024-04-21T23:06:37.751 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:06:37.751 INFO:teuthology.orchestra.run.smithi040.stdout:Upgraded: 2024-04-21T23:06:37.751 INFO:teuthology.orchestra.run.smithi040.stdout: librados2-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:37.751 INFO:teuthology.orchestra.run.smithi040.stdout: librbd1-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:37.751 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2024-04-21T23:06:37.751 INFO:teuthology.orchestra.run.smithi040.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-21T23:06:37.751 INFO:teuthology.orchestra.run.smithi040.stdout: automake-1.16.2-8.el9.noarch 2024-04-21T23:06:37.751 INFO:teuthology.orchestra.run.smithi040.stdout: bison-3.7.4-5.el9.x86_64 2024-04-21T23:06:37.751 INFO:teuthology.orchestra.run.smithi040.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-21T23:06:37.751 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:37.751 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-base-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:37.751 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-common-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:37.751 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-fuse-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:37.751 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-grafana-dashboards-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T23:06:37.752 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-immutable-object-cache-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:37.752 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mds-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:37.752 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:37.752 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-cephadm-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T23:06:37.752 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-dashboard-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T23:06:37.752 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T23:06:37.752 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-modules-core-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T23:06:37.752 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-rook-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T23:06:37.752 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mon-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:37.752 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-osd-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:37.752 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-prometheus-alerts-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T23:06:37.752 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-radosgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:37.752 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-selinux-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:37.752 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-test-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:37.752 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-volume-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T23:06:37.753 INFO:teuthology.orchestra.run.smithi040.stdout: cephadm-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T23:06:37.753 INFO:teuthology.orchestra.run.smithi040.stdout: cephfs-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:37.753 INFO:teuthology.orchestra.run.smithi040.stdout: cephfs-shell-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:37.753 INFO:teuthology.orchestra.run.smithi040.stdout: cephfs-top-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T23:06:37.753 INFO:teuthology.orchestra.run.smithi040.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-21T23:06:37.753 INFO:teuthology.orchestra.run.smithi040.stdout: flex-2.6.4-9.el9.x86_64 2024-04-21T23:06:37.753 INFO:teuthology.orchestra.run.smithi040.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-21T23:06:37.753 INFO:teuthology.orchestra.run.smithi040.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-21T23:06:37.753 INFO:teuthology.orchestra.run.smithi040.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-21T23:06:37.753 INFO:teuthology.orchestra.run.smithi040.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-21T23:06:37.753 INFO:teuthology.orchestra.run.smithi040.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-21T23:06:37.753 INFO:teuthology.orchestra.run.smithi040.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-21T23:06:37.753 INFO:teuthology.orchestra.run.smithi040.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-21T23:06:37.753 INFO:teuthology.orchestra.run.smithi040.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-21T23:06:37.753 INFO:teuthology.orchestra.run.smithi040.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-21T23:06:37.754 INFO:teuthology.orchestra.run.smithi040.stdout: libcephfs-devel-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:37.754 INFO:teuthology.orchestra.run.smithi040.stdout: libcephfs2-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:37.754 INFO:teuthology.orchestra.run.smithi040.stdout: libcephsqlite-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:37.754 INFO:teuthology.orchestra.run.smithi040.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-21T23:06:37.754 INFO:teuthology.orchestra.run.smithi040.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-21T23:06:37.754 INFO:teuthology.orchestra.run.smithi040.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-21T23:06:37.754 INFO:teuthology.orchestra.run.smithi040.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-21T23:06:37.754 INFO:teuthology.orchestra.run.smithi040.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-21T23:06:37.754 INFO:teuthology.orchestra.run.smithi040.stdout: librados-devel-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:37.754 INFO:teuthology.orchestra.run.smithi040.stdout: libradosstriper1-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:37.754 INFO:teuthology.orchestra.run.smithi040.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-21T23:06:37.754 INFO:teuthology.orchestra.run.smithi040.stdout: librgw2-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:37.754 INFO:teuthology.orchestra.run.smithi040.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-21T23:06:37.755 INFO:teuthology.orchestra.run.smithi040.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-21T23:06:37.755 INFO:teuthology.orchestra.run.smithi040.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-21T23:06:37.755 INFO:teuthology.orchestra.run.smithi040.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-21T23:06:37.755 INFO:teuthology.orchestra.run.smithi040.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-21T23:06:37.755 INFO:teuthology.orchestra.run.smithi040.stdout: lua-5.4.4-4.el9.x86_64 2024-04-21T23:06:37.755 INFO:teuthology.orchestra.run.smithi040.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-21T23:06:37.755 INFO:teuthology.orchestra.run.smithi040.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-21T23:06:37.755 INFO:teuthology.orchestra.run.smithi040.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-21T23:06:37.755 INFO:teuthology.orchestra.run.smithi040.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-21T23:06:37.755 INFO:teuthology.orchestra.run.smithi040.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-04-21T23:06:37.755 INFO:teuthology.orchestra.run.smithi040.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-21T23:06:37.755 INFO:teuthology.orchestra.run.smithi040.stdout: postgresql-13.14-1.el9.x86_64 2024-04-21T23:06:37.755 INFO:teuthology.orchestra.run.smithi040.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-21T23:06:37.755 INFO:teuthology.orchestra.run.smithi040.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-21T23:06:37.756 INFO:teuthology.orchestra.run.smithi040.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-21T23:06:37.756 INFO:teuthology.orchestra.run.smithi040.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-21T23:06:37.756 INFO:teuthology.orchestra.run.smithi040.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-21T23:06:37.756 INFO:teuthology.orchestra.run.smithi040.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-21T23:06:37.756 INFO:teuthology.orchestra.run.smithi040.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-21T23:06:37.756 INFO:teuthology.orchestra.run.smithi040.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-21T23:06:37.756 INFO:teuthology.orchestra.run.smithi040.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-21T23:06:37.756 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-21T23:06:37.756 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ceph-argparse-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:37.756 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ceph-common-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:37.756 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cephfs-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:37.756 INFO:teuthology.orchestra.run.smithi040.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-21T23:06:37.756 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-21T23:06:37.756 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-21T23:06:37.756 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-21T23:06:37.757 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-21T23:06:37.757 INFO:teuthology.orchestra.run.smithi040.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-21T23:06:37.757 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-21T23:06:37.757 INFO:teuthology.orchestra.run.smithi040.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-21T23:06:37.757 INFO:teuthology.orchestra.run.smithi040.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-21T23:06:37.757 INFO:teuthology.orchestra.run.smithi040.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-21T23:06:37.757 INFO:teuthology.orchestra.run.smithi040.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-21T23:06:37.757 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-21T23:06:37.757 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-21T23:06:37.757 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-21T23:06:37.757 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-21T23:06:37.757 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-21T23:06:37.757 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-21T23:06:37.757 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-21T23:06:37.757 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-21T23:06:37.757 INFO:teuthology.orchestra.run.smithi040.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-21T23:06:37.758 INFO:teuthology.orchestra.run.smithi040.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-21T23:06:37.758 INFO:teuthology.orchestra.run.smithi040.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-21T23:06:37.758 INFO:teuthology.orchestra.run.smithi040.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-21T23:06:37.758 INFO:teuthology.orchestra.run.smithi040.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-21T23:06:37.758 INFO:teuthology.orchestra.run.smithi040.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-21T23:06:37.758 INFO:teuthology.orchestra.run.smithi040.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-21T23:06:37.758 INFO:teuthology.orchestra.run.smithi040.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-21T23:06:37.758 INFO:teuthology.orchestra.run.smithi040.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-21T23:06:37.758 INFO:teuthology.orchestra.run.smithi040.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-21T23:06:37.758 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-21T23:06:37.759 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ply-3.11-14.el9.noarch 2024-04-21T23:06:37.759 INFO:teuthology.orchestra.run.smithi040.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-21T23:06:37.759 INFO:teuthology.orchestra.run.smithi040.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-21T23:06:37.759 INFO:teuthology.orchestra.run.smithi040.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-21T23:06:37.759 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-21T23:06:37.761 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-21T23:06:37.762 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-21T23:06:37.762 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-21T23:06:37.762 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-21T23:06:37.762 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-21T23:06:37.762 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rados-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:37.762 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rbd-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:37.762 INFO:teuthology.orchestra.run.smithi040.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-21T23:06:37.762 INFO:teuthology.orchestra.run.smithi040.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-21T23:06:37.762 INFO:teuthology.orchestra.run.smithi040.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-21T23:06:37.762 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:37.762 INFO:teuthology.orchestra.run.smithi040.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-21T23:06:37.763 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-21T23:06:37.763 INFO:teuthology.orchestra.run.smithi040.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-21T23:06:37.763 INFO:teuthology.orchestra.run.smithi040.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-21T23:06:37.763 INFO:teuthology.orchestra.run.smithi040.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-21T23:06:37.763 INFO:teuthology.orchestra.run.smithi040.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-21T23:06:37.763 INFO:teuthology.orchestra.run.smithi040.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-21T23:06:37.763 INFO:teuthology.orchestra.run.smithi040.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-21T23:06:37.763 INFO:teuthology.orchestra.run.smithi040.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-21T23:06:37.763 INFO:teuthology.orchestra.run.smithi040.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-21T23:06:37.763 INFO:teuthology.orchestra.run.smithi040.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-21T23:06:37.763 INFO:teuthology.orchestra.run.smithi040.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-21T23:06:37.763 INFO:teuthology.orchestra.run.smithi040.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-21T23:06:37.763 INFO:teuthology.orchestra.run.smithi040.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-21T23:06:37.763 INFO:teuthology.orchestra.run.smithi040.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-21T23:06:37.763 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-fuse-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:37.763 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:37.764 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-nbd-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:37.764 INFO:teuthology.orchestra.run.smithi040.stdout: re2-1:20211101-3.el9.x86_64 2024-04-21T23:06:37.764 INFO:teuthology.orchestra.run.smithi040.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-21T23:06:37.764 INFO:teuthology.orchestra.run.smithi040.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-21T23:06:37.764 INFO:teuthology.orchestra.run.smithi040.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-21T23:06:37.764 INFO:teuthology.orchestra.run.smithi040.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-21T23:06:37.764 INFO:teuthology.orchestra.run.smithi040.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-21T23:06:37.764 INFO:teuthology.orchestra.run.smithi040.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-21T23:06:37.764 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:06:37.764 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-04-21T23:06:38.157 DEBUG:teuthology.parallel:result is None 2024-04-21T23:06:39.205 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 152/152 2024-04-21T23:06:39.205 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-2:19.0.0-2405.g820a2f8c.el9.x86_64 1/152 2024-04-21T23:06:39.205 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-base-2:19.0.0-2405.g820a2f8c.el9.x86_64 2/152 2024-04-21T23:06:39.205 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-common-2:19.0.0-2405.g820a2f8c.el9.x86_64 3/152 2024-04-21T23:06:39.205 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-fuse-2:19.0.0-2405.g820a2f8c.el9.x86_64 4/152 2024-04-21T23:06:39.205 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2405.g820a2 5/152 2024-04-21T23:06:39.205 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mds-2:19.0.0-2405.g820a2f8c.el9.x86_64 6/152 2024-04-21T23:06:39.205 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-2:19.0.0-2405.g820a2f8c.el9.x86_64 7/152 2024-04-21T23:06:39.205 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mon-2:19.0.0-2405.g820a2f8c.el9.x86_64 8/152 2024-04-21T23:06:39.205 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-osd-2:19.0.0-2405.g820a2f8c.el9.x86_64 9/152 2024-04-21T23:06:39.205 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-radosgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 10/152 2024-04-21T23:06:39.206 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-selinux-2:19.0.0-2405.g820a2f8c.el9.x86_64 11/152 2024-04-21T23:06:39.206 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-test-2:19.0.0-2405.g820a2f8c.el9.x86_64 12/152 2024-04-21T23:06:39.206 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : cephfs-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 13/152 2024-04-21T23:06:39.206 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : cephfs-shell-2:19.0.0-2405.g820a2f8c.el9.x86_64 14/152 2024-04-21T23:06:39.206 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libcephfs-devel-2:19.0.0-2405.g820a2f8c.el9.x86_ 15/152 2024-04-21T23:06:39.206 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libcephfs2-2:19.0.0-2405.g820a2f8c.el9.x86_64 16/152 2024-04-21T23:06:39.206 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libcephsqlite-2:19.0.0-2405.g820a2f8c.el9.x86_64 17/152 2024-04-21T23:06:39.206 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librados-devel-2:19.0.0-2405.g820a2f8c.el9.x86_6 18/152 2024-04-21T23:06:39.206 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libradosstriper1-2:19.0.0-2405.g820a2f8c.el9.x86 19/152 2024-04-21T23:06:39.206 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librgw2-2:19.0.0-2405.g820a2f8c.el9.x86_64 20/152 2024-04-21T23:06:39.206 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2405.g820a2f8c.el 21/152 2024-04-21T23:06:39.206 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-ceph-common-2:19.0.0-2405.g820a2f8c.el9. 22/152 2024-04-21T23:06:39.206 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cephfs-2:19.0.0-2405.g820a2f8c.el9.x86_6 23/152 2024-04-21T23:06:39.207 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-rados-2:19.0.0-2405.g820a2f8c.el9.x86_64 24/152 2024-04-21T23:06:39.207 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-rbd-2:19.0.0-2405.g820a2f8c.el9.x86_64 25/152 2024-04-21T23:06:39.207 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-rgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 26/152 2024-04-21T23:06:39.207 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : rbd-fuse-2:19.0.0-2405.g820a2f8c.el9.x86_64 27/152 2024-04-21T23:06:39.207 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : rbd-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 28/152 2024-04-21T23:06:39.207 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : rbd-nbd-2:19.0.0-2405.g820a2f8c.el9.x86_64 29/152 2024-04-21T23:06:39.207 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2405.g820a2f8c. 30/152 2024-04-21T23:06:39.218 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2405.g820a2f8c.el9.noa 31/152 2024-04-21T23:06:39.218 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2405.g820a2f8c.el9.n 32/152 2024-04-21T23:06:39.218 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2405.g820 33/152 2024-04-21T23:06:39.218 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2405.g820a2f8c.el 34/152 2024-04-21T23:06:39.218 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2405.g820a2f8c.el9.noarch 35/152 2024-04-21T23:06:39.218 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2405.g820a2f8c.e 36/152 2024-04-21T23:06:39.218 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-volume-2:19.0.0-2405.g820a2f8c.el9.noarch 37/152 2024-04-21T23:06:39.218 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : cephadm-2:19.0.0-2405.g820a2f8c.el9.noarch 38/152 2024-04-21T23:06:39.218 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : cephfs-top-2:19.0.0-2405.g820a2f8c.el9.noarch 39/152 2024-04-21T23:06:39.218 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/152 2024-04-21T23:06:39.219 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/152 2024-04-21T23:06:39.219 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/152 2024-04-21T23:06:39.219 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/152 2024-04-21T23:06:39.219 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/152 2024-04-21T23:06:39.219 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/152 2024-04-21T23:06:39.219 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/152 2024-04-21T23:06:39.219 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/152 2024-04-21T23:06:39.219 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/152 2024-04-21T23:06:39.219 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/152 2024-04-21T23:06:39.219 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/152 2024-04-21T23:06:39.219 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/152 2024-04-21T23:06:39.219 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/152 2024-04-21T23:06:39.219 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/152 2024-04-21T23:06:39.219 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/152 2024-04-21T23:06:39.219 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/152 2024-04-21T23:06:39.220 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/152 2024-04-21T23:06:39.220 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/152 2024-04-21T23:06:39.220 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/152 2024-04-21T23:06:39.220 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/152 2024-04-21T23:06:39.220 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/152 2024-04-21T23:06:39.220 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/152 2024-04-21T23:06:39.220 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/152 2024-04-21T23:06:39.220 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/152 2024-04-21T23:06:39.220 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/152 2024-04-21T23:06:39.220 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/152 2024-04-21T23:06:39.220 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/152 2024-04-21T23:06:39.220 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/152 2024-04-21T23:06:39.220 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 68/152 2024-04-21T23:06:39.220 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/152 2024-04-21T23:06:39.220 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/152 2024-04-21T23:06:39.220 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/152 2024-04-21T23:06:39.221 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/152 2024-04-21T23:06:39.221 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/152 2024-04-21T23:06:39.221 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/152 2024-04-21T23:06:39.221 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/152 2024-04-21T23:06:39.221 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/152 2024-04-21T23:06:39.221 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/152 2024-04-21T23:06:39.221 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/152 2024-04-21T23:06:39.221 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/152 2024-04-21T23:06:39.221 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/152 2024-04-21T23:06:39.221 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/152 2024-04-21T23:06:39.221 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/152 2024-04-21T23:06:39.221 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/152 2024-04-21T23:06:39.221 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/152 2024-04-21T23:06:39.221 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/152 2024-04-21T23:06:39.221 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/152 2024-04-21T23:06:39.222 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/152 2024-04-21T23:06:39.222 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/152 2024-04-21T23:06:39.222 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/152 2024-04-21T23:06:39.222 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/152 2024-04-21T23:06:39.222 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/152 2024-04-21T23:06:39.222 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/152 2024-04-21T23:06:39.222 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/152 2024-04-21T23:06:39.222 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/152 2024-04-21T23:06:39.222 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/152 2024-04-21T23:06:39.222 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/152 2024-04-21T23:06:39.222 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/152 2024-04-21T23:06:39.222 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/152 2024-04-21T23:06:39.222 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/152 2024-04-21T23:06:39.223 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/152 2024-04-21T23:06:39.223 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/152 2024-04-21T23:06:39.223 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/152 2024-04-21T23:06:39.223 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/152 2024-04-21T23:06:39.223 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/152 2024-04-21T23:06:39.223 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/152 2024-04-21T23:06:39.223 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/152 2024-04-21T23:06:39.223 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/152 2024-04-21T23:06:39.223 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/152 2024-04-21T23:06:39.223 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/152 2024-04-21T23:06:39.223 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/152 2024-04-21T23:06:39.223 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/152 2024-04-21T23:06:39.223 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/152 2024-04-21T23:06:39.223 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/152 2024-04-21T23:06:39.223 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/152 2024-04-21T23:06:39.224 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/152 2024-04-21T23:06:39.224 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/152 2024-04-21T23:06:39.224 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/152 2024-04-21T23:06:39.224 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/152 2024-04-21T23:06:39.224 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/152 2024-04-21T23:06:39.224 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/152 2024-04-21T23:06:39.224 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/152 2024-04-21T23:06:39.224 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 122/152 2024-04-21T23:06:39.224 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/152 2024-04-21T23:06:39.224 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/152 2024-04-21T23:06:39.224 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/152 2024-04-21T23:06:39.224 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/152 2024-04-21T23:06:39.224 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/152 2024-04-21T23:06:39.224 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/152 2024-04-21T23:06:39.224 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/152 2024-04-21T23:06:39.224 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/152 2024-04-21T23:06:39.225 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/152 2024-04-21T23:06:39.225 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/152 2024-04-21T23:06:39.225 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/152 2024-04-21T23:06:39.225 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/152 2024-04-21T23:06:39.225 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/152 2024-04-21T23:06:39.225 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/152 2024-04-21T23:06:39.225 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/152 2024-04-21T23:06:39.225 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/152 2024-04-21T23:06:39.225 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/152 2024-04-21T23:06:39.225 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/152 2024-04-21T23:06:39.225 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/152 2024-04-21T23:06:39.225 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/152 2024-04-21T23:06:39.225 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/152 2024-04-21T23:06:39.225 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/152 2024-04-21T23:06:39.225 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/152 2024-04-21T23:06:39.226 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/152 2024-04-21T23:06:39.226 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/152 2024-04-21T23:06:39.226 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/152 2024-04-21T23:06:39.226 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librados2-2:19.0.0-2405.g820a2f8c.el9.x86_64 149/152 2024-04-21T23:06:39.226 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/152 2024-04-21T23:06:39.226 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librbd1-2:19.0.0-2405.g820a2f8c.el9.x86_64 151/152 2024-04-21T23:06:41.259 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/152 2024-04-21T23:06:41.259 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-04-21T23:06:41.260 INFO:teuthology.orchestra.run.smithi174.stdout:Upgraded: 2024-04-21T23:06:41.260 INFO:teuthology.orchestra.run.smithi174.stdout: librados2-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:41.260 INFO:teuthology.orchestra.run.smithi174.stdout: librbd1-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:41.260 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-04-21T23:06:41.260 INFO:teuthology.orchestra.run.smithi174.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-21T23:06:41.260 INFO:teuthology.orchestra.run.smithi174.stdout: automake-1.16.2-8.el9.noarch 2024-04-21T23:06:41.260 INFO:teuthology.orchestra.run.smithi174.stdout: bison-3.7.4-5.el9.x86_64 2024-04-21T23:06:41.260 INFO:teuthology.orchestra.run.smithi174.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-21T23:06:41.260 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:41.260 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-base-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:41.260 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-common-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:41.260 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-fuse-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:41.260 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-grafana-dashboards-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T23:06:41.260 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-immutable-object-cache-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:41.261 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mds-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:41.261 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:41.261 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-cephadm-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T23:06:41.261 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-dashboard-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T23:06:41.261 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T23:06:41.261 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-modules-core-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T23:06:41.261 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-rook-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T23:06:41.261 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mon-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:41.261 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-osd-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:41.261 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-prometheus-alerts-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T23:06:41.261 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-radosgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:41.261 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-selinux-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:41.261 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-test-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:41.262 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-volume-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T23:06:41.262 INFO:teuthology.orchestra.run.smithi174.stdout: cephadm-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T23:06:41.262 INFO:teuthology.orchestra.run.smithi174.stdout: cephfs-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:41.262 INFO:teuthology.orchestra.run.smithi174.stdout: cephfs-shell-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:41.262 INFO:teuthology.orchestra.run.smithi174.stdout: cephfs-top-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T23:06:41.262 INFO:teuthology.orchestra.run.smithi174.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-21T23:06:41.262 INFO:teuthology.orchestra.run.smithi174.stdout: flex-2.6.4-9.el9.x86_64 2024-04-21T23:06:41.262 INFO:teuthology.orchestra.run.smithi174.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-21T23:06:41.262 INFO:teuthology.orchestra.run.smithi174.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-21T23:06:41.262 INFO:teuthology.orchestra.run.smithi174.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-21T23:06:41.262 INFO:teuthology.orchestra.run.smithi174.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-21T23:06:41.262 INFO:teuthology.orchestra.run.smithi174.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-21T23:06:41.263 INFO:teuthology.orchestra.run.smithi174.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-21T23:06:41.263 INFO:teuthology.orchestra.run.smithi174.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-21T23:06:41.263 INFO:teuthology.orchestra.run.smithi174.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-21T23:06:41.263 INFO:teuthology.orchestra.run.smithi174.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-21T23:06:41.263 INFO:teuthology.orchestra.run.smithi174.stdout: libcephfs-devel-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:41.263 INFO:teuthology.orchestra.run.smithi174.stdout: libcephfs2-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:41.263 INFO:teuthology.orchestra.run.smithi174.stdout: libcephsqlite-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:41.263 INFO:teuthology.orchestra.run.smithi174.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-21T23:06:41.263 INFO:teuthology.orchestra.run.smithi174.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-21T23:06:41.263 INFO:teuthology.orchestra.run.smithi174.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-21T23:06:41.263 INFO:teuthology.orchestra.run.smithi174.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-21T23:06:41.263 INFO:teuthology.orchestra.run.smithi174.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-21T23:06:41.263 INFO:teuthology.orchestra.run.smithi174.stdout: librados-devel-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:41.263 INFO:teuthology.orchestra.run.smithi174.stdout: libradosstriper1-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:41.264 INFO:teuthology.orchestra.run.smithi174.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-21T23:06:41.264 INFO:teuthology.orchestra.run.smithi174.stdout: librgw2-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:41.264 INFO:teuthology.orchestra.run.smithi174.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-21T23:06:41.264 INFO:teuthology.orchestra.run.smithi174.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-21T23:06:41.264 INFO:teuthology.orchestra.run.smithi174.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-21T23:06:41.264 INFO:teuthology.orchestra.run.smithi174.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-21T23:06:41.264 INFO:teuthology.orchestra.run.smithi174.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-21T23:06:41.264 INFO:teuthology.orchestra.run.smithi174.stdout: lua-5.4.4-4.el9.x86_64 2024-04-21T23:06:41.264 INFO:teuthology.orchestra.run.smithi174.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-21T23:06:41.264 INFO:teuthology.orchestra.run.smithi174.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-21T23:06:41.264 INFO:teuthology.orchestra.run.smithi174.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-21T23:06:41.264 INFO:teuthology.orchestra.run.smithi174.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-21T23:06:41.265 INFO:teuthology.orchestra.run.smithi174.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-04-21T23:06:41.265 INFO:teuthology.orchestra.run.smithi174.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-21T23:06:41.265 INFO:teuthology.orchestra.run.smithi174.stdout: postgresql-13.14-1.el9.x86_64 2024-04-21T23:06:41.265 INFO:teuthology.orchestra.run.smithi174.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-21T23:06:41.265 INFO:teuthology.orchestra.run.smithi174.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-21T23:06:41.265 INFO:teuthology.orchestra.run.smithi174.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-21T23:06:41.265 INFO:teuthology.orchestra.run.smithi174.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-21T23:06:41.265 INFO:teuthology.orchestra.run.smithi174.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-21T23:06:41.265 INFO:teuthology.orchestra.run.smithi174.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-21T23:06:41.265 INFO:teuthology.orchestra.run.smithi174.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-21T23:06:41.265 INFO:teuthology.orchestra.run.smithi174.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-21T23:06:41.265 INFO:teuthology.orchestra.run.smithi174.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-21T23:06:41.265 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-21T23:06:41.265 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ceph-argparse-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:41.266 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ceph-common-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:41.266 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cephfs-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:41.266 INFO:teuthology.orchestra.run.smithi174.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-21T23:06:41.266 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-21T23:06:41.266 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-21T23:06:41.266 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-21T23:06:41.266 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-21T23:06:41.266 INFO:teuthology.orchestra.run.smithi174.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-21T23:06:41.266 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-21T23:06:41.266 INFO:teuthology.orchestra.run.smithi174.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-21T23:06:41.266 INFO:teuthology.orchestra.run.smithi174.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-21T23:06:41.266 INFO:teuthology.orchestra.run.smithi174.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-21T23:06:41.267 INFO:teuthology.orchestra.run.smithi174.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-21T23:06:41.267 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-21T23:06:41.267 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-21T23:06:41.267 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-21T23:06:41.267 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-21T23:06:41.267 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-21T23:06:41.267 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-21T23:06:41.267 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-21T23:06:41.267 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-21T23:06:41.267 INFO:teuthology.orchestra.run.smithi174.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-21T23:06:41.267 INFO:teuthology.orchestra.run.smithi174.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-21T23:06:41.267 INFO:teuthology.orchestra.run.smithi174.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-21T23:06:41.267 INFO:teuthology.orchestra.run.smithi174.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-21T23:06:41.268 INFO:teuthology.orchestra.run.smithi174.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-21T23:06:41.268 INFO:teuthology.orchestra.run.smithi174.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-21T23:06:41.268 INFO:teuthology.orchestra.run.smithi174.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-21T23:06:41.268 INFO:teuthology.orchestra.run.smithi174.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-21T23:06:41.268 INFO:teuthology.orchestra.run.smithi174.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-21T23:06:41.268 INFO:teuthology.orchestra.run.smithi174.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-21T23:06:41.268 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-21T23:06:41.268 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ply-3.11-14.el9.noarch 2024-04-21T23:06:41.268 INFO:teuthology.orchestra.run.smithi174.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-21T23:06:41.268 INFO:teuthology.orchestra.run.smithi174.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-21T23:06:41.268 INFO:teuthology.orchestra.run.smithi174.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-21T23:06:41.268 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-21T23:06:41.268 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-21T23:06:41.268 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-21T23:06:41.268 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-21T23:06:41.269 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-21T23:06:41.269 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-21T23:06:41.269 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rados-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:41.269 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rbd-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:41.269 INFO:teuthology.orchestra.run.smithi174.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-21T23:06:41.269 INFO:teuthology.orchestra.run.smithi174.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-21T23:06:41.269 INFO:teuthology.orchestra.run.smithi174.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-21T23:06:41.269 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:41.269 INFO:teuthology.orchestra.run.smithi174.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-21T23:06:41.269 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-21T23:06:41.269 INFO:teuthology.orchestra.run.smithi174.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-21T23:06:41.269 INFO:teuthology.orchestra.run.smithi174.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-21T23:06:41.269 INFO:teuthology.orchestra.run.smithi174.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-21T23:06:41.269 INFO:teuthology.orchestra.run.smithi174.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-21T23:06:41.270 INFO:teuthology.orchestra.run.smithi174.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-21T23:06:41.270 INFO:teuthology.orchestra.run.smithi174.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-21T23:06:41.270 INFO:teuthology.orchestra.run.smithi174.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-21T23:06:41.270 INFO:teuthology.orchestra.run.smithi174.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-21T23:06:41.270 INFO:teuthology.orchestra.run.smithi174.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-21T23:06:41.270 INFO:teuthology.orchestra.run.smithi174.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-21T23:06:41.270 INFO:teuthology.orchestra.run.smithi174.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-21T23:06:41.270 INFO:teuthology.orchestra.run.smithi174.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-21T23:06:41.270 INFO:teuthology.orchestra.run.smithi174.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-21T23:06:41.270 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-fuse-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:41.270 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:41.270 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-nbd-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:41.270 INFO:teuthology.orchestra.run.smithi174.stdout: re2-1:20211101-3.el9.x86_64 2024-04-21T23:06:41.271 INFO:teuthology.orchestra.run.smithi174.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-21T23:06:41.271 INFO:teuthology.orchestra.run.smithi174.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-21T23:06:41.271 INFO:teuthology.orchestra.run.smithi174.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-21T23:06:41.271 INFO:teuthology.orchestra.run.smithi174.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-21T23:06:41.271 INFO:teuthology.orchestra.run.smithi174.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-21T23:06:41.271 INFO:teuthology.orchestra.run.smithi174.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-21T23:06:41.271 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-04-21T23:06:41.271 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-04-21T23:06:41.655 DEBUG:teuthology.parallel:result is None 2024-04-21T23:06:42.056 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-test-2:19.0.0-2405.g820a2f8c.el9.x86_64 137/152 2024-04-21T23:06:42.098 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : rbd-fuse-2:19.0.0-2405.g820a2f8c.el9.x86_64 138/152 2024-04-21T23:06:42.138 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : rbd-nbd-2:19.0.0-2405.g820a2f8c.el9.x86_64 139/152 2024-04-21T23:06:42.171 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 140/152 2024-04-21T23:06:42.198 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 141/152 2024-04-21T23:06:42.383 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : cephfs-shell-2:19.0.0-2405.g820a2f8c.el9.x86_64 142/152 2024-04-21T23:06:42.436 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 143/152 2024-04-21T23:06:42.466 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libcephfs-devel-2:19.0.0-2405.g820a2f8c.el9.x86_ 144/152 2024-04-21T23:06:43.037 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-21T23:06:43.079 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-21T23:06:43.100 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-21T23:06:43.877 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : cephfs-top-2:19.0.0-2405.g820a2f8c.el9.noarch 146/152 2024-04-21T23:06:43.959 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 147/152 2024-04-21T23:06:44.000 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 148/152 2024-04-21T23:06:44.082 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : flex-2.6.4-9.el9.x86_64 149/152 2024-04-21T23:06:44.113 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : bison-3.7.4-5.el9.x86_64 150/152 2024-04-21T23:06:44.113 INFO:teuthology.orchestra.run.smithi111.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-04-21T23:06:44.205 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-04-21T23:06:44.205 INFO:teuthology.orchestra.run.smithi111.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 152/152 2024-04-21T23:06:49.994 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 152/152 2024-04-21T23:06:49.994 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-2:19.0.0-2405.g820a2f8c.el9.x86_64 1/152 2024-04-21T23:06:49.994 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-base-2:19.0.0-2405.g820a2f8c.el9.x86_64 2/152 2024-04-21T23:06:49.994 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-common-2:19.0.0-2405.g820a2f8c.el9.x86_64 3/152 2024-04-21T23:06:49.994 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-fuse-2:19.0.0-2405.g820a2f8c.el9.x86_64 4/152 2024-04-21T23:06:49.994 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2405.g820a2 5/152 2024-04-21T23:06:49.994 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mds-2:19.0.0-2405.g820a2f8c.el9.x86_64 6/152 2024-04-21T23:06:49.994 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-2:19.0.0-2405.g820a2f8c.el9.x86_64 7/152 2024-04-21T23:06:49.995 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mon-2:19.0.0-2405.g820a2f8c.el9.x86_64 8/152 2024-04-21T23:06:49.995 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-osd-2:19.0.0-2405.g820a2f8c.el9.x86_64 9/152 2024-04-21T23:06:49.995 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-radosgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 10/152 2024-04-21T23:06:49.995 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-selinux-2:19.0.0-2405.g820a2f8c.el9.x86_64 11/152 2024-04-21T23:06:49.995 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-test-2:19.0.0-2405.g820a2f8c.el9.x86_64 12/152 2024-04-21T23:06:49.995 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : cephfs-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 13/152 2024-04-21T23:06:49.995 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : cephfs-shell-2:19.0.0-2405.g820a2f8c.el9.x86_64 14/152 2024-04-21T23:06:49.995 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libcephfs-devel-2:19.0.0-2405.g820a2f8c.el9.x86_ 15/152 2024-04-21T23:06:49.995 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libcephfs2-2:19.0.0-2405.g820a2f8c.el9.x86_64 16/152 2024-04-21T23:06:49.995 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libcephsqlite-2:19.0.0-2405.g820a2f8c.el9.x86_64 17/152 2024-04-21T23:06:49.995 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librados-devel-2:19.0.0-2405.g820a2f8c.el9.x86_6 18/152 2024-04-21T23:06:49.995 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libradosstriper1-2:19.0.0-2405.g820a2f8c.el9.x86 19/152 2024-04-21T23:06:49.995 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librgw2-2:19.0.0-2405.g820a2f8c.el9.x86_64 20/152 2024-04-21T23:06:49.996 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2405.g820a2f8c.el 21/152 2024-04-21T23:06:49.996 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-ceph-common-2:19.0.0-2405.g820a2f8c.el9. 22/152 2024-04-21T23:06:49.996 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cephfs-2:19.0.0-2405.g820a2f8c.el9.x86_6 23/152 2024-04-21T23:06:49.996 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-rados-2:19.0.0-2405.g820a2f8c.el9.x86_64 24/152 2024-04-21T23:06:49.996 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-rbd-2:19.0.0-2405.g820a2f8c.el9.x86_64 25/152 2024-04-21T23:06:49.996 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-rgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 26/152 2024-04-21T23:06:49.996 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : rbd-fuse-2:19.0.0-2405.g820a2f8c.el9.x86_64 27/152 2024-04-21T23:06:50.018 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : rbd-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 28/152 2024-04-21T23:06:50.019 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : rbd-nbd-2:19.0.0-2405.g820a2f8c.el9.x86_64 29/152 2024-04-21T23:06:50.019 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2405.g820a2f8c. 30/152 2024-04-21T23:06:50.019 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2405.g820a2f8c.el9.noa 31/152 2024-04-21T23:06:50.019 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2405.g820a2f8c.el9.n 32/152 2024-04-21T23:06:50.019 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2405.g820 33/152 2024-04-21T23:06:50.020 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2405.g820a2f8c.el 34/152 2024-04-21T23:06:50.020 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2405.g820a2f8c.el9.noarch 35/152 2024-04-21T23:06:50.020 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2405.g820a2f8c.e 36/152 2024-04-21T23:06:50.020 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-volume-2:19.0.0-2405.g820a2f8c.el9.noarch 37/152 2024-04-21T23:06:50.020 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : cephadm-2:19.0.0-2405.g820a2f8c.el9.noarch 38/152 2024-04-21T23:06:50.021 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : cephfs-top-2:19.0.0-2405.g820a2f8c.el9.noarch 39/152 2024-04-21T23:06:50.021 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/152 2024-04-21T23:06:50.021 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/152 2024-04-21T23:06:50.021 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/152 2024-04-21T23:06:50.021 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/152 2024-04-21T23:06:50.021 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/152 2024-04-21T23:06:50.022 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/152 2024-04-21T23:06:50.022 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/152 2024-04-21T23:06:50.022 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/152 2024-04-21T23:06:50.022 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/152 2024-04-21T23:06:50.022 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/152 2024-04-21T23:06:50.022 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/152 2024-04-21T23:06:50.023 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/152 2024-04-21T23:06:50.023 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/152 2024-04-21T23:06:50.023 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/152 2024-04-21T23:06:50.023 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/152 2024-04-21T23:06:50.023 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/152 2024-04-21T23:06:50.023 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/152 2024-04-21T23:06:50.024 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/152 2024-04-21T23:06:50.024 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/152 2024-04-21T23:06:50.024 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/152 2024-04-21T23:06:50.024 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/152 2024-04-21T23:06:50.024 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/152 2024-04-21T23:06:50.024 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/152 2024-04-21T23:06:50.025 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/152 2024-04-21T23:06:50.025 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/152 2024-04-21T23:06:50.025 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/152 2024-04-21T23:06:50.025 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/152 2024-04-21T23:06:50.025 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/152 2024-04-21T23:06:50.026 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 68/152 2024-04-21T23:06:50.026 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/152 2024-04-21T23:06:50.026 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/152 2024-04-21T23:06:50.026 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/152 2024-04-21T23:06:50.026 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/152 2024-04-21T23:06:50.026 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/152 2024-04-21T23:06:50.027 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/152 2024-04-21T23:06:50.027 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/152 2024-04-21T23:06:50.027 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/152 2024-04-21T23:06:50.027 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/152 2024-04-21T23:06:50.027 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/152 2024-04-21T23:06:50.027 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/152 2024-04-21T23:06:50.028 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/152 2024-04-21T23:06:50.028 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/152 2024-04-21T23:06:50.028 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/152 2024-04-21T23:06:50.028 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/152 2024-04-21T23:06:50.028 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/152 2024-04-21T23:06:50.029 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/152 2024-04-21T23:06:50.029 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/152 2024-04-21T23:06:50.029 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/152 2024-04-21T23:06:50.029 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/152 2024-04-21T23:06:50.029 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/152 2024-04-21T23:06:50.029 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/152 2024-04-21T23:06:50.030 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/152 2024-04-21T23:06:50.030 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/152 2024-04-21T23:06:50.030 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/152 2024-04-21T23:06:50.030 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/152 2024-04-21T23:06:50.030 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/152 2024-04-21T23:06:50.030 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/152 2024-04-21T23:06:50.031 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/152 2024-04-21T23:06:50.031 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/152 2024-04-21T23:06:50.031 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/152 2024-04-21T23:06:50.031 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/152 2024-04-21T23:06:50.031 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/152 2024-04-21T23:06:50.031 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/152 2024-04-21T23:06:50.032 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/152 2024-04-21T23:06:50.032 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/152 2024-04-21T23:06:50.032 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/152 2024-04-21T23:06:50.032 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/152 2024-04-21T23:06:50.032 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/152 2024-04-21T23:06:50.033 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/152 2024-04-21T23:06:50.033 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/152 2024-04-21T23:06:50.033 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/152 2024-04-21T23:06:50.033 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/152 2024-04-21T23:06:50.033 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/152 2024-04-21T23:06:50.033 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/152 2024-04-21T23:06:50.034 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/152 2024-04-21T23:06:50.034 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/152 2024-04-21T23:06:50.034 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/152 2024-04-21T23:06:50.034 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/152 2024-04-21T23:06:50.034 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/152 2024-04-21T23:06:50.034 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/152 2024-04-21T23:06:50.035 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/152 2024-04-21T23:06:50.035 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/152 2024-04-21T23:06:50.035 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 122/152 2024-04-21T23:06:50.035 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/152 2024-04-21T23:06:50.035 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/152 2024-04-21T23:06:50.036 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/152 2024-04-21T23:06:50.036 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/152 2024-04-21T23:06:50.036 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/152 2024-04-21T23:06:50.036 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/152 2024-04-21T23:06:50.036 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/152 2024-04-21T23:06:50.036 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/152 2024-04-21T23:06:50.037 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/152 2024-04-21T23:06:50.037 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/152 2024-04-21T23:06:50.037 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/152 2024-04-21T23:06:50.037 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/152 2024-04-21T23:06:50.037 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/152 2024-04-21T23:06:50.037 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/152 2024-04-21T23:06:50.037 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/152 2024-04-21T23:06:50.038 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/152 2024-04-21T23:06:50.038 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/152 2024-04-21T23:06:50.038 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/152 2024-04-21T23:06:50.038 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/152 2024-04-21T23:06:50.038 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/152 2024-04-21T23:06:50.038 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/152 2024-04-21T23:06:50.038 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/152 2024-04-21T23:06:50.039 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/152 2024-04-21T23:06:50.039 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/152 2024-04-21T23:06:50.039 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/152 2024-04-21T23:06:50.039 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/152 2024-04-21T23:06:50.039 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librados2-2:19.0.0-2405.g820a2f8c.el9.x86_64 149/152 2024-04-21T23:06:50.039 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/152 2024-04-21T23:06:50.039 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librbd1-2:19.0.0-2405.g820a2f8c.el9.x86_64 151/152 2024-04-21T23:06:52.079 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/152 2024-04-21T23:06:52.079 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-21T23:06:52.079 INFO:teuthology.orchestra.run.smithi111.stdout:Upgraded: 2024-04-21T23:06:52.079 INFO:teuthology.orchestra.run.smithi111.stdout: librados2-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:52.079 INFO:teuthology.orchestra.run.smithi111.stdout: librbd1-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:52.079 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-04-21T23:06:52.079 INFO:teuthology.orchestra.run.smithi111.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-21T23:06:52.080 INFO:teuthology.orchestra.run.smithi111.stdout: automake-1.16.2-8.el9.noarch 2024-04-21T23:06:52.080 INFO:teuthology.orchestra.run.smithi111.stdout: bison-3.7.4-5.el9.x86_64 2024-04-21T23:06:52.080 INFO:teuthology.orchestra.run.smithi111.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-21T23:06:52.080 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:52.080 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-base-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:52.080 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-common-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:52.080 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-fuse-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:52.080 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-grafana-dashboards-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T23:06:52.080 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-immutable-object-cache-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:52.080 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mds-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:52.080 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:52.081 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-cephadm-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T23:06:52.081 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-dashboard-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T23:06:52.081 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T23:06:52.081 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-modules-core-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T23:06:52.081 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-rook-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T23:06:52.081 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mon-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:52.081 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-osd-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:52.081 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-prometheus-alerts-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T23:06:52.081 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-radosgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:52.081 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-selinux-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:52.081 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-test-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:52.081 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-volume-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T23:06:52.081 INFO:teuthology.orchestra.run.smithi111.stdout: cephadm-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T23:06:52.081 INFO:teuthology.orchestra.run.smithi111.stdout: cephfs-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:52.082 INFO:teuthology.orchestra.run.smithi111.stdout: cephfs-shell-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:52.082 INFO:teuthology.orchestra.run.smithi111.stdout: cephfs-top-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T23:06:52.082 INFO:teuthology.orchestra.run.smithi111.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-21T23:06:52.082 INFO:teuthology.orchestra.run.smithi111.stdout: flex-2.6.4-9.el9.x86_64 2024-04-21T23:06:52.082 INFO:teuthology.orchestra.run.smithi111.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-21T23:06:52.082 INFO:teuthology.orchestra.run.smithi111.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-21T23:06:52.082 INFO:teuthology.orchestra.run.smithi111.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-21T23:06:52.082 INFO:teuthology.orchestra.run.smithi111.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-21T23:06:52.082 INFO:teuthology.orchestra.run.smithi111.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-21T23:06:52.082 INFO:teuthology.orchestra.run.smithi111.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-21T23:06:52.082 INFO:teuthology.orchestra.run.smithi111.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-21T23:06:52.082 INFO:teuthology.orchestra.run.smithi111.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-21T23:06:52.083 INFO:teuthology.orchestra.run.smithi111.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-21T23:06:52.083 INFO:teuthology.orchestra.run.smithi111.stdout: libcephfs-devel-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:52.083 INFO:teuthology.orchestra.run.smithi111.stdout: libcephfs2-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:52.083 INFO:teuthology.orchestra.run.smithi111.stdout: libcephsqlite-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:52.083 INFO:teuthology.orchestra.run.smithi111.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-21T23:06:52.083 INFO:teuthology.orchestra.run.smithi111.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-21T23:06:52.083 INFO:teuthology.orchestra.run.smithi111.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-21T23:06:52.083 INFO:teuthology.orchestra.run.smithi111.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-21T23:06:52.083 INFO:teuthology.orchestra.run.smithi111.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-21T23:06:52.083 INFO:teuthology.orchestra.run.smithi111.stdout: librados-devel-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:52.083 INFO:teuthology.orchestra.run.smithi111.stdout: libradosstriper1-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:52.083 INFO:teuthology.orchestra.run.smithi111.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-21T23:06:52.083 INFO:teuthology.orchestra.run.smithi111.stdout: librgw2-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:52.084 INFO:teuthology.orchestra.run.smithi111.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-21T23:06:52.084 INFO:teuthology.orchestra.run.smithi111.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-21T23:06:52.084 INFO:teuthology.orchestra.run.smithi111.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-21T23:06:52.084 INFO:teuthology.orchestra.run.smithi111.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-21T23:06:52.084 INFO:teuthology.orchestra.run.smithi111.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-21T23:06:52.084 INFO:teuthology.orchestra.run.smithi111.stdout: lua-5.4.4-4.el9.x86_64 2024-04-21T23:06:52.084 INFO:teuthology.orchestra.run.smithi111.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-21T23:06:52.084 INFO:teuthology.orchestra.run.smithi111.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-21T23:06:52.084 INFO:teuthology.orchestra.run.smithi111.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-21T23:06:52.084 INFO:teuthology.orchestra.run.smithi111.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-21T23:06:52.084 INFO:teuthology.orchestra.run.smithi111.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-04-21T23:06:52.084 INFO:teuthology.orchestra.run.smithi111.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-21T23:06:52.084 INFO:teuthology.orchestra.run.smithi111.stdout: postgresql-13.14-1.el9.x86_64 2024-04-21T23:06:52.085 INFO:teuthology.orchestra.run.smithi111.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-21T23:06:52.085 INFO:teuthology.orchestra.run.smithi111.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-21T23:06:52.085 INFO:teuthology.orchestra.run.smithi111.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-21T23:06:52.085 INFO:teuthology.orchestra.run.smithi111.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-21T23:06:52.085 INFO:teuthology.orchestra.run.smithi111.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-21T23:06:52.085 INFO:teuthology.orchestra.run.smithi111.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-21T23:06:52.085 INFO:teuthology.orchestra.run.smithi111.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-21T23:06:52.085 INFO:teuthology.orchestra.run.smithi111.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-21T23:06:52.085 INFO:teuthology.orchestra.run.smithi111.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-21T23:06:52.085 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-21T23:06:52.085 INFO:teuthology.orchestra.run.smithi111.stdout: python3-ceph-argparse-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:52.085 INFO:teuthology.orchestra.run.smithi111.stdout: python3-ceph-common-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:52.085 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cephfs-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:52.086 INFO:teuthology.orchestra.run.smithi111.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-21T23:06:52.086 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-21T23:06:52.086 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-21T23:06:52.086 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-21T23:06:52.086 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-21T23:06:52.086 INFO:teuthology.orchestra.run.smithi111.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-21T23:06:52.086 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-21T23:06:52.086 INFO:teuthology.orchestra.run.smithi111.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-21T23:06:52.086 INFO:teuthology.orchestra.run.smithi111.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-21T23:06:52.086 INFO:teuthology.orchestra.run.smithi111.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-21T23:06:52.086 INFO:teuthology.orchestra.run.smithi111.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-21T23:06:52.086 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-21T23:06:52.086 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-21T23:06:52.087 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-21T23:06:52.087 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-21T23:06:52.087 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-21T23:06:52.087 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-21T23:06:52.087 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-21T23:06:52.087 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-21T23:06:52.087 INFO:teuthology.orchestra.run.smithi111.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-21T23:06:52.087 INFO:teuthology.orchestra.run.smithi111.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-21T23:06:52.087 INFO:teuthology.orchestra.run.smithi111.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-21T23:06:52.087 INFO:teuthology.orchestra.run.smithi111.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-21T23:06:52.087 INFO:teuthology.orchestra.run.smithi111.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-21T23:06:52.087 INFO:teuthology.orchestra.run.smithi111.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-21T23:06:52.087 INFO:teuthology.orchestra.run.smithi111.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-21T23:06:52.087 INFO:teuthology.orchestra.run.smithi111.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-21T23:06:52.088 INFO:teuthology.orchestra.run.smithi111.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-21T23:06:52.088 INFO:teuthology.orchestra.run.smithi111.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-21T23:06:52.088 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-21T23:06:52.088 INFO:teuthology.orchestra.run.smithi111.stdout: python3-ply-3.11-14.el9.noarch 2024-04-21T23:06:52.088 INFO:teuthology.orchestra.run.smithi111.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-21T23:06:52.088 INFO:teuthology.orchestra.run.smithi111.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-21T23:06:52.088 INFO:teuthology.orchestra.run.smithi111.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-21T23:06:52.088 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-21T23:06:52.088 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-21T23:06:52.088 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-21T23:06:52.088 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-21T23:06:52.088 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-21T23:06:52.088 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-21T23:06:52.089 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rados-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:52.089 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rbd-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:52.089 INFO:teuthology.orchestra.run.smithi111.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-21T23:06:52.089 INFO:teuthology.orchestra.run.smithi111.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-21T23:06:52.089 INFO:teuthology.orchestra.run.smithi111.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-21T23:06:52.089 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:52.089 INFO:teuthology.orchestra.run.smithi111.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-21T23:06:52.089 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-21T23:06:52.089 INFO:teuthology.orchestra.run.smithi111.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-21T23:06:52.089 INFO:teuthology.orchestra.run.smithi111.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-21T23:06:52.089 INFO:teuthology.orchestra.run.smithi111.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-21T23:06:52.089 INFO:teuthology.orchestra.run.smithi111.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-21T23:06:52.089 INFO:teuthology.orchestra.run.smithi111.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-21T23:06:52.089 INFO:teuthology.orchestra.run.smithi111.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-21T23:06:52.090 INFO:teuthology.orchestra.run.smithi111.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-21T23:06:52.090 INFO:teuthology.orchestra.run.smithi111.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-21T23:06:52.090 INFO:teuthology.orchestra.run.smithi111.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-21T23:06:52.090 INFO:teuthology.orchestra.run.smithi111.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-21T23:06:52.090 INFO:teuthology.orchestra.run.smithi111.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-21T23:06:52.090 INFO:teuthology.orchestra.run.smithi111.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-21T23:06:52.090 INFO:teuthology.orchestra.run.smithi111.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-21T23:06:52.090 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-fuse-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:52.090 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:52.090 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-nbd-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:06:52.090 INFO:teuthology.orchestra.run.smithi111.stdout: re2-1:20211101-3.el9.x86_64 2024-04-21T23:06:52.090 INFO:teuthology.orchestra.run.smithi111.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-21T23:06:52.090 INFO:teuthology.orchestra.run.smithi111.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-21T23:06:52.090 INFO:teuthology.orchestra.run.smithi111.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-21T23:06:52.091 INFO:teuthology.orchestra.run.smithi111.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-21T23:06:52.091 INFO:teuthology.orchestra.run.smithi111.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-21T23:06:52.091 INFO:teuthology.orchestra.run.smithi111.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-21T23:06:52.091 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-21T23:06:52.091 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-04-21T23:06:52.434 DEBUG:teuthology.parallel:result is None 2024-04-21T23:06:52.435 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=820a2f8ced66152ed3d765c31444b8076ea28c6b 2024-04-21T23:06:52.584 DEBUG:teuthology.orchestra.run.smithi040:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-21T23:06:52.610 INFO:teuthology.orchestra.run.smithi040.stdout:19.0.0-2405.g820a2f8c.el9 2024-04-21T23:06:52.611 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2405.g820a2f8c.el9 2024-04-21T23:06:52.611 INFO:teuthology.task.install:The correct ceph version 19.0.0-2405.g820a2f8c is installed. 2024-04-21T23:06:52.613 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=820a2f8ced66152ed3d765c31444b8076ea28c6b 2024-04-21T23:06:52.755 DEBUG:teuthology.orchestra.run.smithi111:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-21T23:06:52.776 INFO:teuthology.orchestra.run.smithi111.stdout:19.0.0-2405.g820a2f8c.el9 2024-04-21T23:06:52.776 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2405.g820a2f8c.el9 2024-04-21T23:06:52.776 INFO:teuthology.task.install:The correct ceph version 19.0.0-2405.g820a2f8c is installed. 2024-04-21T23:06:52.778 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=820a2f8ced66152ed3d765c31444b8076ea28c6b 2024-04-21T23:06:52.993 DEBUG:teuthology.orchestra.run.smithi174:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-21T23:06:53.019 INFO:teuthology.orchestra.run.smithi174.stdout:19.0.0-2405.g820a2f8c.el9 2024-04-21T23:06:53.019 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2405.g820a2f8c.el9 2024-04-21T23:06:53.019 INFO:teuthology.task.install:The correct ceph version 19.0.0-2405.g820a2f8c is installed. 2024-04-21T23:06:53.021 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-21T23:06:53.021 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-21T23:06:53.021 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-21T23:06:53.053 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-04-21T23:06:53.053 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-21T23:06:53.078 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-04-21T23:06:53.079 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-21T23:06:53.110 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-21T23:06:53.111 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-21T23:06:53.111 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/usr/bin/daemon-helper 2024-04-21T23:06:53.140 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-21T23:06:53.209 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-04-21T23:06:53.209 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/usr/bin/daemon-helper 2024-04-21T23:06:53.233 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-21T23:06:53.297 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-04-21T23:06:53.297 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/usr/bin/daemon-helper 2024-04-21T23:06:53.326 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-21T23:06:53.396 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-21T23:06:53.396 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-21T23:06:53.396 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-21T23:06:53.426 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-21T23:06:53.492 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-04-21T23:06:53.493 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-21T23:06:53.516 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-21T23:06:53.580 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-04-21T23:06:53.580 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-21T23:06:53.604 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-21T23:06:53.668 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-21T23:06:53.668 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-21T23:06:53.668 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/usr/bin/stdin-killer 2024-04-21T23:06:53.693 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-21T23:06:53.757 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-04-21T23:06:53.757 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/usr/bin/stdin-killer 2024-04-21T23:06:53.780 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-21T23:06:53.843 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-04-21T23:06:53.843 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/usr/bin/stdin-killer 2024-04-21T23:06:53.868 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-21T23:06:53.932 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-21T23:06:54.034 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_export_ephemeral_random_max': 0.1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': '820a2f8ced66152ed3d765c31444b8076ea28c6b', 'subvols': {'create': 2}} 2024-04-21T23:06:54.035 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b 2024-04-21T23:06:54.035 INFO:tasks.cephadm:Cluster fsid is d814e4dc-0033-11ef-bc93-c7b262605968 2024-04-21T23:06:54.035 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-21T23:06:54.035 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.40', 'mon.b': '172.21.15.111', 'mon.c': '172.21.15.174'} 2024-04-21T23:06:54.036 INFO:tasks.cephadm:First mon is mon.a on smithi040 2024-04-21T23:06:54.036 INFO:tasks.cephadm:First mgr is x 2024-04-21T23:06:54.036 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-21T23:06:54.036 DEBUG:teuthology.orchestra.run.smithi040:> sudo hostname $(hostname -s) 2024-04-21T23:06:54.061 DEBUG:teuthology.orchestra.run.smithi111:> sudo hostname $(hostname -s) 2024-04-21T23:06:54.084 DEBUG:teuthology.orchestra.run.smithi174:> sudo hostname $(hostname -s) 2024-04-21T23:06:54.108 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-04-21T23:06:54.108 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=820a2f8ced66152ed3d765c31444b8076ea28c6b 2024-04-21T23:06:54.323 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '820a2f8ced66152ed3d765c31444b8076ea28c6b', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/54945/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2405-g820a2f8c', 'node_name': '172.21.4.69+confusa07', 'job_name': 'ceph-dev-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2405.g820a2f8c'}, 'url': 'https://2.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-04-20 17:52:04.986240', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://2.chacra.ceph.com/repos/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/centos/9/flavors/default/', 'archs': ['x86_64', 'source', 'arm64'], 'distro': 'centos'}] 2024-04-21T23:06:54.543 INFO:tasks.util.chacra:got chacra host 2.chacra.ceph.com, ref squid, sha1 820a2f8ced66152ed3d765c31444b8076ea28c6b from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=820a2f8ced66152ed3d765c31444b8076ea28c6b 2024-04-21T23:06:54.545 INFO:tasks.cephadm:Discovered cachra url: https://2.chacra.ceph.com/binaries/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/centos/9/x86_64/flavors/default/cephadm 2024-04-21T23:06:54.545 DEBUG:teuthology.orchestra.run.smithi040:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-21T23:06:54.983 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r--. 1 ubuntu ubuntu 781418 Apr 21 23:06 /home/ubuntu/cephtest/cephadm 2024-04-21T23:06:54.983 DEBUG:teuthology.orchestra.run.smithi111:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-21T23:06:55.446 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r--. 1 ubuntu ubuntu 781418 Apr 21 23:06 /home/ubuntu/cephtest/cephadm 2024-04-21T23:06:55.446 DEBUG:teuthology.orchestra.run.smithi174:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-21T23:06:56.059 INFO:teuthology.orchestra.run.smithi174.stdout:-rw-r--r--. 1 ubuntu ubuntu 781418 Apr 21 23:06 /home/ubuntu/cephtest/cephadm 2024-04-21T23:06:56.060 DEBUG:teuthology.orchestra.run.smithi040:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-21T23:06:56.079 DEBUG:teuthology.orchestra.run.smithi111:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-21T23:06:56.098 DEBUG:teuthology.orchestra.run.smithi174:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-21T23:06:56.132 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b on all hosts... 2024-04-21T23:06:56.132 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b pull 2024-04-21T23:06:56.135 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b pull 2024-04-21T23:06:56.140 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b pull 2024-04-21T23:06:57.658 INFO:teuthology.orchestra.run.smithi040.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b... 2024-04-21T23:06:57.773 INFO:teuthology.orchestra.run.smithi111.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b... 2024-04-21T23:06:57.919 INFO:teuthology.orchestra.run.smithi174.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b... 2024-04-21T23:07:37.143 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-21T23:07:37.143 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_version": "ceph version 19.0.0-2405-g820a2f8c (820a2f8ced66152ed3d765c31444b8076ea28c6b) squid (dev)", 2024-04-21T23:07:37.143 INFO:teuthology.orchestra.run.smithi040.stdout: "image_id": "6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32", 2024-04-21T23:07:37.143 INFO:teuthology.orchestra.run.smithi040.stdout: "repo_digests": [ 2024-04-21T23:07:37.143 INFO:teuthology.orchestra.run.smithi040.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9" 2024-04-21T23:07:37.144 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-21T23:07:37.144 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-21T23:07:41.870 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-21T23:07:41.870 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_version": "ceph version 19.0.0-2405-g820a2f8c (820a2f8ced66152ed3d765c31444b8076ea28c6b) squid (dev)", 2024-04-21T23:07:41.870 INFO:teuthology.orchestra.run.smithi111.stdout: "image_id": "6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32", 2024-04-21T23:07:41.870 INFO:teuthology.orchestra.run.smithi111.stdout: "repo_digests": [ 2024-04-21T23:07:41.870 INFO:teuthology.orchestra.run.smithi111.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9" 2024-04-21T23:07:41.870 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-21T23:07:41.870 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-21T23:07:43.700 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-04-21T23:07:43.700 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_version": "ceph version 19.0.0-2405-g820a2f8c (820a2f8ced66152ed3d765c31444b8076ea28c6b) squid (dev)", 2024-04-21T23:07:43.700 INFO:teuthology.orchestra.run.smithi174.stdout: "image_id": "6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32", 2024-04-21T23:07:43.700 INFO:teuthology.orchestra.run.smithi174.stdout: "repo_digests": [ 2024-04-21T23:07:43.700 INFO:teuthology.orchestra.run.smithi174.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9" 2024-04-21T23:07:43.700 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-04-21T23:07:43.701 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-04-21T23:07:43.727 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /etc/ceph 2024-04-21T23:07:43.760 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /etc/ceph 2024-04-21T23:07:43.792 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /etc/ceph 2024-04-21T23:07:43.820 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 777 /etc/ceph 2024-04-21T23:07:43.848 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 777 /etc/ceph 2024-04-21T23:07:43.876 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod 777 /etc/ceph 2024-04-21T23:07:43.900 INFO:tasks.cephadm:Writing seed config... 2024-04-21T23:07:43.901 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-04-21T23:07:43.901 INFO:tasks.cephadm: override: [client] debug client = 20 2024-04-21T23:07:43.901 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-21T23:07:43.901 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-04-21T23:07:43.902 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-04-21T23:07:43.902 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-04-21T23:07:43.902 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-04-21T23:07:43.902 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-04-21T23:07:43.902 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-04-21T23:07:43.902 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-04-21T23:07:43.902 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-04-21T23:07:43.902 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-04-21T23:07:43.902 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-04-21T23:07:43.902 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-04-21T23:07:43.902 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-04-21T23:07:43.902 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-04-21T23:07:43.902 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-04-21T23:07:43.902 INFO:tasks.cephadm: override: [mds] mds_export_ephemeral_random_max = 0.1 2024-04-21T23:07:43.903 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-04-21T23:07:43.903 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-04-21T23:07:43.903 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-04-21T23:07:43.903 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-04-21T23:07:43.903 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-04-21T23:07:43.903 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-04-21T23:07:43.903 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-21T23:07:43.903 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-21T23:07:43.903 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-21T23:07:43.903 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-21T23:07:43.903 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-21T23:07:43.903 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-04-21T23:07:43.903 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-04-21T23:07:43.904 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-04-21T23:07:43.904 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-04-21T23:07:43.904 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-04-21T23:07:43.904 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-04-21T23:07:43.904 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-21T23:07:43.904 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-21T23:07:43.904 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-04-21T23:07:43.904 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-04-21T23:07:43.904 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-04-21T23:07:43.904 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-04-21T23:07:43.904 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-04-21T23:07:43.904 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-04-21T23:07:43.904 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-04-21T23:07:43.904 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-04-21T23:07:43.905 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-04-21T23:07:43.905 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-21T23:07:43.905 DEBUG:teuthology.orchestra.run.smithi040:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-21T23:07:43.923 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = d814e4dc-0033-11ef-bc93-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_export_ephemeral_random_max = 0.1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-04-21T23:07:43.923 DEBUG:teuthology.orchestra.run.smithi040:mon.a> sudo journalctl -f -n 0 -u ceph-d814e4dc-0033-11ef-bc93-c7b262605968@mon.a.service 2024-04-21T23:07:43.966 DEBUG:teuthology.orchestra.run.smithi040:mgr.x> sudo journalctl -f -n 0 -u ceph-d814e4dc-0033-11ef-bc93-c7b262605968@mgr.x.service 2024-04-21T23:07:44.009 INFO:tasks.cephadm:Bootstrapping... 2024-04-21T23:07:44.009 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b -v bootstrap --fsid d814e4dc-0033-11ef-bc93-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.40 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-21T23:07:44.208 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-04-21T23:07:44.208 INFO:teuthology.orchestra.run.smithi040.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b', '-v', 'bootstrap', '--fsid', 'd814e4dc-0033-11ef-bc93-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.40', '--skip-admin-label'] 2024-04-21T23:07:44.209 INFO:teuthology.orchestra.run.smithi040.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-21T23:07:44.209 INFO:teuthology.orchestra.run.smithi040.stdout:Verifying podman|docker is present... 2024-04-21T23:07:44.234 INFO:teuthology.orchestra.run.smithi040.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-21T23:07:44.235 INFO:teuthology.orchestra.run.smithi040.stdout:Verifying lvm2 is present... 2024-04-21T23:07:44.235 INFO:teuthology.orchestra.run.smithi040.stdout:Verifying time synchronization is in place... 2024-04-21T23:07:44.242 INFO:teuthology.orchestra.run.smithi040.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-21T23:07:44.242 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-21T23:07:44.248 INFO:teuthology.orchestra.run.smithi040.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-21T23:07:44.248 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stdout inactive 2024-04-21T23:07:44.255 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stdout enabled 2024-04-21T23:07:44.261 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stdout active 2024-04-21T23:07:44.261 INFO:teuthology.orchestra.run.smithi040.stdout:Unit chronyd.service is enabled and running 2024-04-21T23:07:44.261 INFO:teuthology.orchestra.run.smithi040.stdout:Repeating the final host check... 2024-04-21T23:07:44.284 INFO:teuthology.orchestra.run.smithi040.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-21T23:07:44.284 INFO:teuthology.orchestra.run.smithi040.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-21T23:07:44.284 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl is present 2024-04-21T23:07:44.284 INFO:teuthology.orchestra.run.smithi040.stdout:lvcreate is present 2024-04-21T23:07:44.291 INFO:teuthology.orchestra.run.smithi040.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-21T23:07:44.291 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-21T23:07:44.297 INFO:teuthology.orchestra.run.smithi040.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-21T23:07:44.297 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stdout inactive 2024-04-21T23:07:44.303 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stdout enabled 2024-04-21T23:07:44.310 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stdout active 2024-04-21T23:07:44.310 INFO:teuthology.orchestra.run.smithi040.stdout:Unit chronyd.service is enabled and running 2024-04-21T23:07:44.310 INFO:teuthology.orchestra.run.smithi040.stdout:Host looks OK 2024-04-21T23:07:44.310 INFO:teuthology.orchestra.run.smithi040.stdout:Cluster fsid: d814e4dc-0033-11ef-bc93-c7b262605968 2024-04-21T23:07:44.310 INFO:teuthology.orchestra.run.smithi040.stdout:Acquiring lock 140080566625184 on /run/cephadm/d814e4dc-0033-11ef-bc93-c7b262605968.lock 2024-04-21T23:07:44.310 INFO:teuthology.orchestra.run.smithi040.stdout:Lock 140080566625184 acquired on /run/cephadm/d814e4dc-0033-11ef-bc93-c7b262605968.lock 2024-04-21T23:07:44.311 INFO:teuthology.orchestra.run.smithi040.stdout:Verifying IP 172.21.15.40 port 3300 ... 2024-04-21T23:07:44.311 INFO:teuthology.orchestra.run.smithi040.stdout:Verifying IP 172.21.15.40 port 6789 ... 2024-04-21T23:07:44.311 INFO:teuthology.orchestra.run.smithi040.stdout:Base mon IP(s) is [172.21.15.40:3300, 172.21.15.40:6789], mon addrv is [v2:172.21.15.40:3300,v1:172.21.15.40:6789] 2024-04-21T23:07:44.314 INFO:teuthology.orchestra.run.smithi040.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.40 metric 100 2024-04-21T23:07:44.314 INFO:teuthology.orchestra.run.smithi040.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.40 metric 100 2024-04-21T23:07:44.317 INFO:teuthology.orchestra.run.smithi040.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-21T23:07:44.317 INFO:teuthology.orchestra.run.smithi040.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-04-21T23:07:44.317 INFO:teuthology.orchestra.run.smithi040.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-04-21T23:07:44.320 INFO:teuthology.orchestra.run.smithi040.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-21T23:07:44.320 INFO:teuthology.orchestra.run.smithi040.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-21T23:07:44.320 INFO:teuthology.orchestra.run.smithi040.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-21T23:07:44.320 INFO:teuthology.orchestra.run.smithi040.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-04-21T23:07:44.320 INFO:teuthology.orchestra.run.smithi040.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:12cc/64 scope link noprefixroute 2024-04-21T23:07:44.320 INFO:teuthology.orchestra.run.smithi040.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-21T23:07:44.321 INFO:teuthology.orchestra.run.smithi040.stdout:Mon IP `172.21.15.40` is in CIDR network `172.21.0.0/20` 2024-04-21T23:07:44.321 INFO:teuthology.orchestra.run.smithi040.stdout:Mon IP `172.21.15.40` is in CIDR network `172.21.0.0/20` 2024-04-21T23:07:44.321 INFO:teuthology.orchestra.run.smithi040.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-21T23:07:44.321 INFO:teuthology.orchestra.run.smithi040.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-21T23:07:44.322 INFO:teuthology.orchestra.run.smithi040.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b... 2024-04-21T23:07:44.802 INFO:teuthology.orchestra.run.smithi040.stdout:/bin/podman: stdout 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 2024-04-21T23:07:44.802 INFO:teuthology.orchestra.run.smithi040.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b... 2024-04-21T23:07:44.802 INFO:teuthology.orchestra.run.smithi040.stdout:/bin/podman: stderr Getting image source signatures 2024-04-21T23:07:44.803 INFO:teuthology.orchestra.run.smithi040.stdout:/bin/podman: stderr Copying blob sha256:4706d97462bc6e0886c6ef3c0b1ac67e3e6a7e63a606e29958996d1fc15dafd8 2024-04-21T23:07:44.803 INFO:teuthology.orchestra.run.smithi040.stdout:/bin/podman: stderr Copying blob sha256:426fede9fffa2c7dfe40a4f9322a34794b72405170134d54d2c30d9cc6c6a5be 2024-04-21T23:07:44.803 INFO:teuthology.orchestra.run.smithi040.stdout:/bin/podman: stderr Copying config sha256:6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 2024-04-21T23:07:44.803 INFO:teuthology.orchestra.run.smithi040.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-21T23:07:45.673 INFO:teuthology.orchestra.run.smithi040.stdout:ceph: stdout ceph version 19.0.0-2405-g820a2f8c (820a2f8ced66152ed3d765c31444b8076ea28c6b) squid (dev) 2024-04-21T23:07:45.674 INFO:teuthology.orchestra.run.smithi040.stdout:Ceph version: ceph version 19.0.0-2405-g820a2f8c (820a2f8ced66152ed3d765c31444b8076ea28c6b) squid (dev) 2024-04-21T23:07:45.674 INFO:teuthology.orchestra.run.smithi040.stdout:Extracting ceph user uid/gid from container image... 2024-04-21T23:07:46.621 INFO:teuthology.orchestra.run.smithi040.stdout:stat: stdout 167 167 2024-04-21T23:07:46.621 INFO:teuthology.orchestra.run.smithi040.stdout:Creating initial keys... 2024-04-21T23:07:47.569 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph-authtool: stdout AQBCnCVmlieBOBAAG2GAVd1Vgn5FWJhhm2EiZw== 2024-04-21T23:07:48.437 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph-authtool: stdout AQBDnCVmQyXnMxAA/zJGBC/H4wxLE3smnHHDCg== 2024-04-21T23:07:49.384 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph-authtool: stdout AQBEnCVmWDWaLBAAfzBLpLPFdKSWBPj/r1QAZQ== 2024-04-21T23:07:49.385 INFO:teuthology.orchestra.run.smithi040.stdout:Creating initial monmap... 2024-04-21T23:07:50.250 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-21T23:07:50.250 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-04-21T23:07:50.250 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to d814e4dc-0033-11ef-bc93-c7b262605968 2024-04-21T23:07:50.250 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-21T23:07:50.250 INFO:teuthology.orchestra.run.smithi040.stdout:monmaptool for a [v2:172.21.15.40:3300,v1:172.21.15.40:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-21T23:07:50.250 INFO:teuthology.orchestra.run.smithi040.stdout:setting min_mon_release = quincy 2024-04-21T23:07:50.250 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/monmaptool: set fsid to d814e4dc-0033-11ef-bc93-c7b262605968 2024-04-21T23:07:50.250 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-21T23:07:50.251 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:07:50.251 INFO:teuthology.orchestra.run.smithi040.stdout:Creating mon... 2024-04-21T23:07:51.191 INFO:teuthology.orchestra.run.smithi040.stdout:create mon.a on 2024-04-21T23:07:51.404 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-04-21T23:07:51.558 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-21T23:07:51.747 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-d814e4dc-0033-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-d814e4dc-0033-11ef-bc93-c7b262605968.target. 2024-04-21T23:07:51.747 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-d814e4dc-0033-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-d814e4dc-0033-11ef-bc93-c7b262605968.target. 2024-04-21T23:07:52.003 INFO:teuthology.orchestra.run.smithi040.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-d814e4dc-0033-11ef-bc93-c7b262605968@mon.a 2024-04-21T23:07:52.003 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stderr Failed to reset failed state of unit ceph-d814e4dc-0033-11ef-bc93-c7b262605968@mon.a.service: Unit ceph-d814e4dc-0033-11ef-bc93-c7b262605968@mon.a.service not loaded. 2024-04-21T23:07:52.166 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-d814e4dc-0033-11ef-bc93-c7b262605968.target.wants/ceph-d814e4dc-0033-11ef-bc93-c7b262605968@mon.a.service → /etc/systemd/system/ceph-d814e4dc-0033-11ef-bc93-c7b262605968@.service. 2024-04-21T23:07:52.717 INFO:teuthology.orchestra.run.smithi040.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-21T23:07:52.717 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stdout disabled 2024-04-21T23:07:52.726 INFO:teuthology.orchestra.run.smithi040.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-21T23:07:52.726 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stdout inactive 2024-04-21T23:07:52.726 INFO:teuthology.orchestra.run.smithi040.stdout:firewalld.service is not enabled 2024-04-21T23:07:52.726 INFO:teuthology.orchestra.run.smithi040.stdout:Not possible to enable service . firewalld.service is not available 2024-04-21T23:07:52.726 INFO:teuthology.orchestra.run.smithi040.stdout:Waiting for mon to start... 2024-04-21T23:07:52.726 INFO:teuthology.orchestra.run.smithi040.stdout:Waiting for mon... 2024-04-21T23:07:53.826 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:53 smithi040 ceph-mon[27609]: from='client.? 172.21.15.40:0/3768298924' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-21T23:07:53.834 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout cluster: 2024-04-21T23:07:53.834 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout id: d814e4dc-0033-11ef-bc93-c7b262605968 2024-04-21T23:07:53.834 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-21T23:07:53.835 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 2024-04-21T23:07:53.835 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout services: 2024-04-21T23:07:53.835 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.57387s) 2024-04-21T23:07:53.835 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-21T23:07:53.835 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-21T23:07:53.835 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 2024-04-21T23:07:53.835 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout data: 2024-04-21T23:07:53.835 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-21T23:07:53.835 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-21T23:07:53.835 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-21T23:07:53.835 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout pgs: 2024-04-21T23:07:53.835 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 2024-04-21T23:07:53.835 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:53.150+0000 7f9630e41700 1 Processor -- start 2024-04-21T23:07:53.835 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:53.150+0000 7f9630e41700 1 -- start start 2024-04-21T23:07:53.835 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:53.150+0000 7f9630e41700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f962c107c90 0x7f962c1080b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:53.835 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:53.150+0000 7f9630e41700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f962c108680 con 0x7f962c107c90 2024-04-21T23:07:53.836 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:53.151+0000 7f962a59c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f962c107c90 0x7f962c1080b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:53.836 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:53.151+0000 7f962a59c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f962c107c90 0x7f962c1080b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44708/0 (socket says 172.21.15.40:44708) 2024-04-21T23:07:53.836 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:53.151+0000 7f962a59c700 1 -- 172.21.15.40:0/4257796918 learned_addr learned my addr 172.21.15.40:0/4257796918 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:07:53.836 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:53.151+0000 7f962a59c700 1 -- 172.21.15.40:0/4257796918 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f962c108df0 con 0x7f962c107c90 2024-04-21T23:07:53.836 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:53.151+0000 7f962a59c700 1 --2- 172.21.15.40:0/4257796918 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f962c107c90 0x7f962c1080b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f961c0098a0 tx=0x7f961c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52c62c69d6047025 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:07:53.836 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:53.151+0000 7f962959a700 1 -- 172.21.15.40:0/4257796918 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f961c004160 con 0x7f962c107c90 2024-04-21T23:07:53.836 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:53.151+0000 7f962959a700 1 -- 172.21.15.40:0/4257796918 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f961c004300 con 0x7f962c107c90 2024-04-21T23:07:53.836 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:53.151+0000 7f962959a700 1 -- 172.21.15.40:0/4257796918 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f961c0036a0 con 0x7f962c107c90 2024-04-21T23:07:53.836 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:53.152+0000 7f9630e41700 1 -- 172.21.15.40:0/4257796918 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f962c107c90 msgr2=0x7f962c1080b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:53.836 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:53.152+0000 7f9630e41700 1 --2- 172.21.15.40:0/4257796918 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f962c107c90 0x7f962c1080b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f961c0098a0 tx=0x7f961c009bb0 comp rx=0 tx=0).stop 2024-04-21T23:07:53.836 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:53.152+0000 7f9630e41700 1 -- 172.21.15.40:0/4257796918 shutdown_connections 2024-04-21T23:07:53.836 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:53.152+0000 7f9630e41700 1 --2- 172.21.15.40:0/4257796918 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f962c107c90 0x7f962c1080b0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:53.836 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:53.152+0000 7f9630e41700 1 -- 172.21.15.40:0/4257796918 >> 172.21.15.40:0/4257796918 conn(0x7f962c103210 msgr2=0x7f962c105670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:07:53.836 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:53.152+0000 7f9630e41700 1 -- 172.21.15.40:0/4257796918 shutdown_connections 2024-04-21T23:07:53.837 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:53.152+0000 7f9630e41700 1 -- 172.21.15.40:0/4257796918 wait complete. 2024-04-21T23:07:53.837 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:53.153+0000 7f9630e41700 1 Processor -- start 2024-04-21T23:07:53.837 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:53.153+0000 7f9630e41700 1 -- start start 2024-04-21T23:07:53.837 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:53.153+0000 7f9630e41700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f962c107c90 0x7f962c07db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:53.837 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:53.153+0000 7f9630e41700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f962c109320 con 0x7f962c107c90 2024-04-21T23:07:53.837 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:53.153+0000 7f962a59c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f962c107c90 0x7f962c07db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:53.837 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:53.153+0000 7f962a59c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f962c107c90 0x7f962c07db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44712/0 (socket says 172.21.15.40:44712) 2024-04-21T23:07:53.837 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:53.153+0000 7f962a59c700 1 -- 172.21.15.40:0/3768298924 learned_addr learned my addr 172.21.15.40:0/3768298924 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:07:53.837 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:53.153+0000 7f962a59c700 1 -- 172.21.15.40:0/3768298924 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f962c07e0b0 con 0x7f962c107c90 2024-04-21T23:07:53.837 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:53.154+0000 7f962a59c700 1 --2- 172.21.15.40:0/3768298924 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f962c107c90 0x7f962c07db70 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f961c000c00 tx=0x7f961c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:07:53.837 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:53.154+0000 7f961b7fe700 1 -- 172.21.15.40:0/3768298924 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f961c003db0 con 0x7f962c107c90 2024-04-21T23:07:53.837 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:53.154+0000 7f961b7fe700 1 -- 172.21.15.40:0/3768298924 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f961c011630 con 0x7f962c107c90 2024-04-21T23:07:53.837 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:53.154+0000 7f9630e41700 1 -- 172.21.15.40:0/3768298924 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f962c07c2a0 con 0x7f962c107c90 2024-04-21T23:07:53.837 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:53.154+0000 7f9630e41700 1 -- 172.21.15.40:0/3768298924 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f962c07c7e0 con 0x7f962c107c90 2024-04-21T23:07:53.837 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:53.154+0000 7f961b7fe700 1 -- 172.21.15.40:0/3768298924 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f961c011950 con 0x7f962c107c90 2024-04-21T23:07:53.838 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:53.155+0000 7f961b7fe700 1 -- 172.21.15.40:0/3768298924 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f961c011af0 con 0x7f962c107c90 2024-04-21T23:07:53.838 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:53.155+0000 7f961b7fe700 1 -- 172.21.15.40:0/3768298924 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f961c01c450 con 0x7f962c107c90 2024-04-21T23:07:53.838 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:53.155+0000 7f9630e41700 1 -- 172.21.15.40:0/3768298924 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f962c063890 con 0x7f962c107c90 2024-04-21T23:07:53.838 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:53.158+0000 7f961b7fe700 1 -- 172.21.15.40:0/3768298924 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f961c032020 con 0x7f962c107c90 2024-04-21T23:07:53.838 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:53.231+0000 7f9630e41700 1 -- 172.21.15.40:0/3768298924 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f962c194bb0 con 0x7f962c107c90 2024-04-21T23:07:53.838 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:53.231+0000 7f961b7fe700 1 -- 172.21.15.40:0/3768298924 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+316 (secure 0 0 0) 0x7f961c004260 con 0x7f962c107c90 2024-04-21T23:07:53.838 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:53.232+0000 7f9630e41700 1 -- 172.21.15.40:0/3768298924 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f962c107c90 msgr2=0x7f962c07db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:53.838 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:53.232+0000 7f9630e41700 1 --2- 172.21.15.40:0/3768298924 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f962c107c90 0x7f962c07db70 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f961c000c00 tx=0x7f961c004750 comp rx=0 tx=0).stop 2024-04-21T23:07:53.838 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:53.232+0000 7f9630e41700 1 -- 172.21.15.40:0/3768298924 shutdown_connections 2024-04-21T23:07:53.838 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:53.232+0000 7f9630e41700 1 --2- 172.21.15.40:0/3768298924 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f962c107c90 0x7f962c07db70 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:53.838 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:53.232+0000 7f9630e41700 1 -- 172.21.15.40:0/3768298924 >> 172.21.15.40:0/3768298924 conn(0x7f962c103210 msgr2=0x7f962c191ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:07:53.838 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:53.232+0000 7f9630e41700 1 -- 172.21.15.40:0/3768298924 shutdown_connections 2024-04-21T23:07:53.839 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:53.232+0000 7f9630e41700 1 -- 172.21.15.40:0/3768298924 wait complete. 2024-04-21T23:07:53.839 INFO:teuthology.orchestra.run.smithi040.stdout:mon is available 2024-04-21T23:07:53.839 INFO:teuthology.orchestra.run.smithi040.stdout:Assimilating anything we can from ceph.conf... 2024-04-21T23:07:54.884 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 2024-04-21T23:07:54.885 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout [global] 2024-04-21T23:07:54.885 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout fsid = d814e4dc-0033-11ef-bc93-c7b262605968 2024-04-21T23:07:54.885 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.40:3300,v1:172.21.15.40:6789] 2024-04-21T23:07:54.885 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-21T23:07:54.885 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-21T23:07:54.885 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-21T23:07:54.885 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-21T23:07:54.885 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 2024-04-21T23:07:54.885 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-21T23:07:54.885 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-21T23:07:54.885 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 2024-04-21T23:07:54.885 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout [osd] 2024-04-21T23:07:54.885 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-21T23:07:54.886 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-21T23:07:54.886 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-21T23:07:54.886 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-21T23:07:54.886 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-21T23:07:54.886 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:54.247+0000 7f1f2a2e5700 1 Processor -- start 2024-04-21T23:07:54.886 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:54.247+0000 7f1f2a2e5700 1 -- start start 2024-04-21T23:07:54.886 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:54.247+0000 7f1f2a2e5700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1f241062c0 0x7f1f241066e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:54.886 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:54.247+0000 7f1f2a2e5700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1f24106cb0 con 0x7f1f241062c0 2024-04-21T23:07:54.886 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:54.247+0000 7f1f23fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1f241062c0 0x7f1f241066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:54.886 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:54.247+0000 7f1f23fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1f241062c0 0x7f1f241066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44720/0 (socket says 172.21.15.40:44720) 2024-04-21T23:07:54.886 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:54.247+0000 7f1f23fff700 1 -- 172.21.15.40:0/3502394510 learned_addr learned my addr 172.21.15.40:0/3502394510 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:07:54.886 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:54.248+0000 7f1f23fff700 1 -- 172.21.15.40:0/3502394510 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f241074c0 con 0x7f1f241062c0 2024-04-21T23:07:54.886 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:54.248+0000 7f1f23fff700 1 --2- 172.21.15.40:0/3502394510 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1f241062c0 0x7f1f241066e0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f1f1800aa40 tx=0x7f1f1800ad50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93599deef3bb56c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:07:54.887 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:54.248+0000 7f1f22ffd700 1 -- 172.21.15.40:0/3502394510 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1f1800b040 con 0x7f1f241062c0 2024-04-21T23:07:54.887 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:54.248+0000 7f1f22ffd700 1 -- 172.21.15.40:0/3502394510 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f18004160 con 0x7f1f241062c0 2024-04-21T23:07:54.887 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:54.248+0000 7f1f22ffd700 1 -- 172.21.15.40:0/3502394510 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1f18004460 con 0x7f1f241062c0 2024-04-21T23:07:54.887 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:54.248+0000 7f1f2a2e5700 1 -- 172.21.15.40:0/3502394510 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1f241062c0 msgr2=0x7f1f241066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:54.887 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:54.248+0000 7f1f2a2e5700 1 --2- 172.21.15.40:0/3502394510 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1f241062c0 0x7f1f241066e0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f1f1800aa40 tx=0x7f1f1800ad50 comp rx=0 tx=0).stop 2024-04-21T23:07:54.887 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:54.248+0000 7f1f2a2e5700 1 -- 172.21.15.40:0/3502394510 shutdown_connections 2024-04-21T23:07:54.887 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:54.248+0000 7f1f2a2e5700 1 --2- 172.21.15.40:0/3502394510 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1f241062c0 0x7f1f241066e0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:54.887 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:54.248+0000 7f1f2a2e5700 1 -- 172.21.15.40:0/3502394510 >> 172.21.15.40:0/3502394510 conn(0x7f1f241018a0 msgr2=0x7f1f24103cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:07:54.887 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:54.249+0000 7f1f2a2e5700 1 -- 172.21.15.40:0/3502394510 shutdown_connections 2024-04-21T23:07:54.887 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:54.249+0000 7f1f2a2e5700 1 -- 172.21.15.40:0/3502394510 wait complete. 2024-04-21T23:07:54.887 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:54.249+0000 7f1f2a2e5700 1 Processor -- start 2024-04-21T23:07:54.887 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:54.249+0000 7f1f2a2e5700 1 -- start start 2024-04-21T23:07:54.887 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:54.249+0000 7f1f2a2e5700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1f241062c0 0x7f1f241991b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:54.888 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:54.249+0000 7f1f2a2e5700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1f24074bc0 con 0x7f1f241062c0 2024-04-21T23:07:54.888 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:54.250+0000 7f1f23fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1f241062c0 0x7f1f241991b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:54.888 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:54.250+0000 7f1f23fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1f241062c0 0x7f1f241991b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44722/0 (socket says 172.21.15.40:44722) 2024-04-21T23:07:54.888 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:54.250+0000 7f1f23fff700 1 -- 172.21.15.40:0/2346288479 learned_addr learned my addr 172.21.15.40:0/2346288479 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:07:54.888 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:54.250+0000 7f1f23fff700 1 -- 172.21.15.40:0/2346288479 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f241996f0 con 0x7f1f241062c0 2024-04-21T23:07:54.888 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:54.250+0000 7f1f23fff700 1 --2- 172.21.15.40:0/2346288479 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1f241062c0 0x7f1f241991b0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f1f18004000 tx=0x7f1f180047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:07:54.888 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:54.250+0000 7f1f217fa700 1 -- 172.21.15.40:0/2346288479 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1f18028070 con 0x7f1f241062c0 2024-04-21T23:07:54.888 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:54.250+0000 7f1f217fa700 1 -- 172.21.15.40:0/2346288479 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f180038b0 con 0x7f1f241062c0 2024-04-21T23:07:54.888 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:54.250+0000 7f1f2a2e5700 1 -- 172.21.15.40:0/2346288479 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f24199980 con 0x7f1f241062c0 2024-04-21T23:07:54.888 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:54.250+0000 7f1f217fa700 1 -- 172.21.15.40:0/2346288479 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1f18003bb0 con 0x7f1f241062c0 2024-04-21T23:07:54.888 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:54.250+0000 7f1f2a2e5700 1 -- 172.21.15.40:0/2346288479 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f2419c670 con 0x7f1f241062c0 2024-04-21T23:07:54.888 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:54.250+0000 7f1f217fa700 1 -- 172.21.15.40:0/2346288479 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f1f1802b070 con 0x7f1f241062c0 2024-04-21T23:07:54.888 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:54.250+0000 7f1f217fa700 1 -- 172.21.15.40:0/2346288479 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f1f18020070 con 0x7f1f241062c0 2024-04-21T23:07:54.889 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:54.251+0000 7f1f2a2e5700 1 -- 172.21.15.40:0/2346288479 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f24063890 con 0x7f1f241062c0 2024-04-21T23:07:54.889 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:54.253+0000 7f1f217fa700 1 -- 172.21.15.40:0/2346288479 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f1f18032020 con 0x7f1f241062c0 2024-04-21T23:07:54.889 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:54.324+0000 7f1f2a2e5700 1 -- 172.21.15.40:0/2346288479 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f1f2402dc80 con 0x7f1f241062c0 2024-04-21T23:07:54.889 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:54.327+0000 7f1f217fa700 1 -- 172.21.15.40:0/2346288479 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+429 (secure 0 0 0) 0x7f1f18014070 con 0x7f1f241062c0 2024-04-21T23:07:54.889 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:54.328+0000 7f1f217fa700 1 -- 172.21.15.40:0/2346288479 <== mon.0 v2:172.21.15.40:3300/0 8 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f1f18003d50 con 0x7f1f241062c0 2024-04-21T23:07:54.889 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:54.328+0000 7f1f2a2e5700 1 -- 172.21.15.40:0/2346288479 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1f241062c0 msgr2=0x7f1f241991b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:54.889 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:54.328+0000 7f1f2a2e5700 1 --2- 172.21.15.40:0/2346288479 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1f241062c0 0x7f1f241991b0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f1f18004000 tx=0x7f1f180047c0 comp rx=0 tx=0).stop 2024-04-21T23:07:54.889 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:54.329+0000 7f1f2a2e5700 1 -- 172.21.15.40:0/2346288479 shutdown_connections 2024-04-21T23:07:54.889 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:54.329+0000 7f1f2a2e5700 1 --2- 172.21.15.40:0/2346288479 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1f241062c0 0x7f1f241991b0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:54.889 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:54.329+0000 7f1f2a2e5700 1 -- 172.21.15.40:0/2346288479 >> 172.21.15.40:0/2346288479 conn(0x7f1f241018a0 msgr2=0x7f1f2418fc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:07:54.889 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:54.329+0000 7f1f2a2e5700 1 -- 172.21.15.40:0/2346288479 shutdown_connections 2024-04-21T23:07:54.889 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:54.329+0000 7f1f2a2e5700 1 -- 172.21.15.40:0/2346288479 wait complete. 2024-04-21T23:07:54.889 INFO:teuthology.orchestra.run.smithi040.stdout:Generating new minimal ceph.conf... 2024-04-21T23:07:54.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:54 smithi040 ceph-mon[27609]: from='client.? 172.21.15.40:0/2346288479' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-21T23:07:54.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:54 smithi040 ceph-mon[27609]: from='client.? 172.21.15.40:0/2346288479' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-21T23:07:55.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:55 smithi040 ceph-mon[27609]: from='client.? 172.21.15.40:0/406396564' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:07:55.949 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:55.308+0000 7f072c9b6700 1 Processor -- start 2024-04-21T23:07:55.949 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:55.308+0000 7f072c9b6700 1 -- start start 2024-04-21T23:07:55.950 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:55.308+0000 7f072c9b6700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0728107c40 0x7f0728108060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:55.950 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:55.308+0000 7f072c9b6700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0728108630 con 0x7f0728107c40 2024-04-21T23:07:55.950 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:55.309+0000 7f072659c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0728107c40 0x7f0728108060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:55.950 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:55.309+0000 7f072659c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0728107c40 0x7f0728108060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44734/0 (socket says 172.21.15.40:44734) 2024-04-21T23:07:55.950 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:55.309+0000 7f072659c700 1 -- 172.21.15.40:0/563765227 learned_addr learned my addr 172.21.15.40:0/563765227 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:07:55.950 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:55.309+0000 7f072659c700 1 -- 172.21.15.40:0/563765227 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0728108e40 con 0x7f0728107c40 2024-04-21T23:07:55.950 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:55.309+0000 7f072659c700 1 --2- 172.21.15.40:0/563765227 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0728107c40 0x7f0728108060 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f0718009e80 tx=0x7f071800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca702d21eb89555 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:07:55.950 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:55.309+0000 7f072559a700 1 -- 172.21.15.40:0/563765227 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0718017070 con 0x7f0728107c40 2024-04-21T23:07:55.950 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:55.309+0000 7f072559a700 1 -- 172.21.15.40:0/563765227 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f071800b8c0 con 0x7f0728107c40 2024-04-21T23:07:55.950 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:55.309+0000 7f072559a700 1 -- 172.21.15.40:0/563765227 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f07180045c0 con 0x7f0728107c40 2024-04-21T23:07:55.950 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:55.310+0000 7f072c9b6700 1 -- 172.21.15.40:0/563765227 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0728107c40 msgr2=0x7f0728108060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:55.950 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:55.310+0000 7f072c9b6700 1 --2- 172.21.15.40:0/563765227 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0728107c40 0x7f0728108060 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f0718009e80 tx=0x7f071800b330 comp rx=0 tx=0).stop 2024-04-21T23:07:55.951 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:55.310+0000 7f072c9b6700 1 -- 172.21.15.40:0/563765227 shutdown_connections 2024-04-21T23:07:55.951 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:55.310+0000 7f072c9b6700 1 --2- 172.21.15.40:0/563765227 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0728107c40 0x7f0728108060 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:55.951 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:55.310+0000 7f072c9b6700 1 -- 172.21.15.40:0/563765227 >> 172.21.15.40:0/563765227 conn(0x7f07281031e0 msgr2=0x7f0728105620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:07:55.951 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:55.310+0000 7f072c9b6700 1 -- 172.21.15.40:0/563765227 shutdown_connections 2024-04-21T23:07:55.951 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:55.310+0000 7f072c9b6700 1 -- 172.21.15.40:0/563765227 wait complete. 2024-04-21T23:07:55.951 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:55.311+0000 7f072c9b6700 1 Processor -- start 2024-04-21T23:07:55.951 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:55.311+0000 7f072c9b6700 1 -- start start 2024-04-21T23:07:55.951 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:55.311+0000 7f072c9b6700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0728107c40 0x7f072819bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:55.951 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:55.311+0000 7f072c9b6700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f072810c1f0 con 0x7f0728107c40 2024-04-21T23:07:55.951 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:55.311+0000 7f072659c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0728107c40 0x7f072819bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:55.951 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:55.311+0000 7f072659c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0728107c40 0x7f072819bbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44748/0 (socket says 172.21.15.40:44748) 2024-04-21T23:07:55.951 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:55.311+0000 7f072659c700 1 -- 172.21.15.40:0/406396564 learned_addr learned my addr 172.21.15.40:0/406396564 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:07:55.951 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:55.311+0000 7f072659c700 1 -- 172.21.15.40:0/406396564 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f072819c130 con 0x7f0728107c40 2024-04-21T23:07:55.951 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:55.312+0000 7f072659c700 1 --2- 172.21.15.40:0/406396564 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0728107c40 0x7f072819bbf0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0718004210 tx=0x7f071800bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:07:55.951 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:55.312+0000 7f071f7fe700 1 -- 172.21.15.40:0/406396564 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0718020020 con 0x7f0728107c40 2024-04-21T23:07:55.952 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:55.312+0000 7f072c9b6700 1 -- 172.21.15.40:0/406396564 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f072819c3c0 con 0x7f0728107c40 2024-04-21T23:07:55.952 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:55.312+0000 7f071f7fe700 1 -- 172.21.15.40:0/406396564 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f0718003b50 con 0x7f0728107c40 2024-04-21T23:07:55.952 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:55.312+0000 7f071f7fe700 1 -- 172.21.15.40:0/406396564 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0718017040 con 0x7f0728107c40 2024-04-21T23:07:55.952 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:55.312+0000 7f072c9b6700 1 -- 172.21.15.40:0/406396564 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f072819f0b0 con 0x7f0728107c40 2024-04-21T23:07:55.952 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:55.312+0000 7f071f7fe700 1 -- 172.21.15.40:0/406396564 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f0718023020 con 0x7f0728107c40 2024-04-21T23:07:55.952 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:55.312+0000 7f071f7fe700 1 -- 172.21.15.40:0/406396564 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f071800f050 con 0x7f0728107c40 2024-04-21T23:07:55.952 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:55.313+0000 7f072c9b6700 1 -- 172.21.15.40:0/406396564 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0728063890 con 0x7f0728107c40 2024-04-21T23:07:55.952 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:55.315+0000 7f071f7fe700 1 -- 172.21.15.40:0/406396564 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f071802a020 con 0x7f0728107c40 2024-04-21T23:07:55.952 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:55.385+0000 7f072c9b6700 1 -- 172.21.15.40:0/406396564 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f072819f280 con 0x7f0728107c40 2024-04-21T23:07:55.952 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:55.385+0000 7f071f7fe700 1 -- 172.21.15.40:0/406396564 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7f071800b860 con 0x7f0728107c40 2024-04-21T23:07:55.952 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:55.386+0000 7f072c9b6700 1 -- 172.21.15.40:0/406396564 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0728107c40 msgr2=0x7f072819bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:55.952 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:55.386+0000 7f072c9b6700 1 --2- 172.21.15.40:0/406396564 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0728107c40 0x7f072819bbf0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0718004210 tx=0x7f071800bd50 comp rx=0 tx=0).stop 2024-04-21T23:07:55.952 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:55.386+0000 7f072c9b6700 1 -- 172.21.15.40:0/406396564 shutdown_connections 2024-04-21T23:07:55.952 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:55.386+0000 7f072c9b6700 1 --2- 172.21.15.40:0/406396564 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0728107c40 0x7f072819bbf0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:55.952 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:55.386+0000 7f072c9b6700 1 -- 172.21.15.40:0/406396564 >> 172.21.15.40:0/406396564 conn(0x7f07281031e0 msgr2=0x7f0728103cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:07:55.952 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:55.386+0000 7f072c9b6700 1 -- 172.21.15.40:0/406396564 shutdown_connections 2024-04-21T23:07:55.953 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:55.386+0000 7f072c9b6700 1 -- 172.21.15.40:0/406396564 wait complete. 2024-04-21T23:07:55.953 INFO:teuthology.orchestra.run.smithi040.stdout:Restarting the monitor... 2024-04-21T23:07:56.346 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:55 smithi040 systemd[1]: Stopping Ceph mon.a for d814e4dc-0033-11ef-bc93-c7b262605968... 2024-04-21T23:07:56.602 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:56 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mon-a[27601]: 2024-04-21T23:07:56.346+0000 7fbb4afc4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T23:07:56.602 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:56 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mon-a[27601]: 2024-04-21T23:07:56.346+0000 7fbb4afc4700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-21T23:07:56.602 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:56 smithi040 podman[27863]: 2024-04-21 23:07:56.603510798 +0000 UTC m=+0.486512286 container died 250c367b60a96fc9e5019b8a1487d79223c13fd3a35decf934b9dad2ea3ed12c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-820a2f8, org.label-schema.build-date=20240416, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T23:07:57.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:56 smithi040 podman[27863]: 2024-04-21 23:07:56.974184694 +0000 UTC m=+0.857186184 container cleanup 250c367b60a96fc9e5019b8a1487d79223c13fd3a35decf934b9dad2ea3ed12c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mon-a, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=squid-820a2f8, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T23:07:57.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:56 smithi040 bash[27863]: ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mon-a 2024-04-21T23:07:57.549 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 podman[27877]: 2024-04-21 23:07:57.20846007 +0000 UTC m=+0.601341248 container remove 250c367b60a96fc9e5019b8a1487d79223c13fd3a35decf934b9dad2ea3ed12c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mon-a, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, RELEASE=squid-820a2f8, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-21T23:07:57.549 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 systemd[1]: ceph-d814e4dc-0033-11ef-bc93-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-21T23:07:57.549 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 systemd[1]: Stopped Ceph mon.a for d814e4dc-0033-11ef-bc93-c7b262605968. 2024-04-21T23:07:57.549 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 systemd[1]: Starting Ceph mon.a for d814e4dc-0033-11ef-bc93-c7b262605968... 2024-04-21T23:07:57.783 INFO:teuthology.orchestra.run.smithi040.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-21T23:07:57.820 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 podman[27965]: 2024-04-21 23:07:57.550782057 +0000 UTC m=+0.106892492 container create 5fb73014130c79a2b3ca1e60ee21c825a0f1f4d38399689ce2bc910dfe6881aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-820a2f8, org.label-schema.build-date=20240416, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-21T23:07:57.820 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 podman[27965]: 2024-04-21 23:07:57.461754145 +0000 UTC m=+0.017864592 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b 2024-04-21T23:07:57.820 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 podman[27965]: 2024-04-21 23:07:57.728158664 +0000 UTC m=+0.284269094 container init 5fb73014130c79a2b3ca1e60ee21c825a0f1f4d38399689ce2bc910dfe6881aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-820a2f8, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-21T23:07:57.820 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 podman[27965]: 2024-04-21 23:07:57.73121075 +0000 UTC m=+0.287321180 container start 5fb73014130c79a2b3ca1e60ee21c825a0f1f4d38399689ce2bc910dfe6881aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-820a2f8) 2024-04-21T23:07:57.820 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: set uid:gid to 167:167 (ceph:ceph) 2024-04-21T23:07:57.821 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: ceph version 19.0.0-2405-g820a2f8c (820a2f8ced66152ed3d765c31444b8076ea28c6b) squid (dev), process ceph-mon, pid 2 2024-04-21T23:07:57.821 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: pidfile_write: ignore empty --pid-file 2024-04-21T23:07:57.821 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: load: jerasure load: lrc 2024-04-21T23:07:57.821 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: RocksDB version: 7.9.2 2024-04-21T23:07:57.821 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Git sha 0 2024-04-21T23:07:57.821 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Compile date 2024-04-20 15:04:30 2024-04-21T23:07:57.821 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: DB SUMMARY 2024-04-21T23:07:57.821 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: DB Session ID: LT2AMW25P3A6BK69FQ2Z 2024-04-21T23:07:57.821 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: CURRENT file: CURRENT 2024-04-21T23:07:57.821 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: IDENTITY file: IDENTITY 2024-04-21T23:07:57.821 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-21T23:07:57.821 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-21T23:07:57.821 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 104770 ; 2024-04-21T23:07:57.821 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.error_if_exists: 0 2024-04-21T23:07:57.823 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.create_if_missing: 0 2024-04-21T23:07:57.823 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.paranoid_checks: 1 2024-04-21T23:07:57.823 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-21T23:07:57.823 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-21T23:07:57.823 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-21T23:07:57.823 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.env: 0x55fe50cffc60 2024-04-21T23:07:57.823 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.fs: PosixFileSystem 2024-04-21T23:07:57.824 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.info_log: 0x55fe51fd7aa0 2024-04-21T23:07:57.824 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.max_file_opening_threads: 16 2024-04-21T23:07:57.824 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.statistics: (nil) 2024-04-21T23:07:57.824 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.use_fsync: 0 2024-04-21T23:07:57.824 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.max_log_file_size: 0 2024-04-21T23:07:57.824 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-21T23:07:57.824 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-21T23:07:57.824 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.keep_log_file_num: 1000 2024-04-21T23:07:57.824 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.recycle_log_file_num: 0 2024-04-21T23:07:57.824 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.allow_fallocate: 1 2024-04-21T23:07:57.824 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.allow_mmap_reads: 0 2024-04-21T23:07:57.824 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.allow_mmap_writes: 0 2024-04-21T23:07:57.824 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.use_direct_reads: 0 2024-04-21T23:07:57.825 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-21T23:07:57.825 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.create_missing_column_families: 0 2024-04-21T23:07:57.825 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.db_log_dir: 2024-04-21T23:07:57.825 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.wal_dir: 2024-04-21T23:07:57.825 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-21T23:07:57.825 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-21T23:07:57.825 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-21T23:07:57.825 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-21T23:07:57.825 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-21T23:07:57.825 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-21T23:07:57.825 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.advise_random_on_open: 1 2024-04-21T23:07:57.825 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.db_write_buffer_size: 0 2024-04-21T23:07:57.825 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.write_buffer_manager: 0x55fe51faa1e0 2024-04-21T23:07:57.825 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-21T23:07:57.826 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-21T23:07:57.826 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-21T23:07:57.826 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.rate_limiter: (nil) 2024-04-21T23:07:57.826 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-21T23:07:57.826 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.wal_recovery_mode: 2 2024-04-21T23:07:57.826 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.enable_thread_tracking: 0 2024-04-21T23:07:57.826 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.enable_pipelined_write: 0 2024-04-21T23:07:57.826 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.unordered_write: 0 2024-04-21T23:07:57.826 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-21T23:07:57.826 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-21T23:07:57.826 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-21T23:07:57.826 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-21T23:07:57.826 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.row_cache: None 2024-04-21T23:07:57.827 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.wal_filter: None 2024-04-21T23:07:57.827 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-21T23:07:57.827 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.allow_ingest_behind: 0 2024-04-21T23:07:57.827 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.two_write_queues: 0 2024-04-21T23:07:57.827 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.manual_wal_flush: 0 2024-04-21T23:07:57.827 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.wal_compression: 0 2024-04-21T23:07:57.827 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.atomic_flush: 0 2024-04-21T23:07:57.827 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-21T23:07:57.827 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-21T23:07:57.827 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-21T23:07:57.827 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.log_readahead_size: 0 2024-04-21T23:07:57.827 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-21T23:07:57.827 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.best_efforts_recovery: 0 2024-04-21T23:07:57.827 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-21T23:07:57.828 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-21T23:07:57.828 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.allow_data_in_errors: 0 2024-04-21T23:07:57.828 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.db_host_id: __hostname__ 2024-04-21T23:07:57.828 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-21T23:07:57.828 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.max_background_jobs: 2 2024-04-21T23:07:57.828 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.max_background_compactions: -1 2024-04-21T23:07:57.828 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.max_subcompactions: 1 2024-04-21T23:07:57.828 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-21T23:07:57.828 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-21T23:07:57.828 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-21T23:07:57.828 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.max_total_wal_size: 0 2024-04-21T23:07:57.828 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-21T23:07:57.828 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-21T23:07:57.828 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-21T23:07:57.828 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-21T23:07:57.829 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.max_open_files: -1 2024-04-21T23:07:57.829 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.bytes_per_sync: 0 2024-04-21T23:07:57.829 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-21T23:07:57.829 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-21T23:07:57.829 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.compaction_readahead_size: 0 2024-04-21T23:07:57.829 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.max_background_flushes: -1 2024-04-21T23:07:57.829 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Compression algorithms supported: 2024-04-21T23:07:57.829 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-21T23:07:57.829 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: kZSTD supported: 0 2024-04-21T23:07:57.829 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: kXpressCompression supported: 0 2024-04-21T23:07:57.829 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: kLZ4HCCompression supported: 1 2024-04-21T23:07:57.829 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: kZlibCompression supported: 1 2024-04-21T23:07:57.829 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: kSnappyCompression supported: 1 2024-04-21T23:07:57.830 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: kLZ4Compression supported: 1 2024-04-21T23:07:57.830 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: kBZip2Compression supported: 0 2024-04-21T23:07:57.830 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-21T23:07:57.830 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-21T23:07:57.830 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-04-21T23:07:57.830 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-21T23:07:57.830 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-21T23:07:57.830 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.merge_operator: 2024-04-21T23:07:57.830 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.compaction_filter: None 2024-04-21T23:07:57.830 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.compaction_filter_factory: None 2024-04-21T23:07:57.830 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.sst_partitioner_factory: None 2024-04-21T23:07:57.830 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-21T23:07:57.830 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-21T23:07:57.830 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55fe51fd7ba0) 2024-04-21T23:07:57.831 INFO:journalctl@ceph.mon.a.smithi040.stdout: cache_index_and_filter_blocks: 1 2024-04-21T23:07:57.831 INFO:journalctl@ceph.mon.a.smithi040.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-21T23:07:57.831 INFO:journalctl@ceph.mon.a.smithi040.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-21T23:07:57.831 INFO:journalctl@ceph.mon.a.smithi040.stdout: pin_top_level_index_and_filter: 1 2024-04-21T23:07:57.831 INFO:journalctl@ceph.mon.a.smithi040.stdout: index_type: 0 2024-04-21T23:07:57.831 INFO:journalctl@ceph.mon.a.smithi040.stdout: data_block_index_type: 0 2024-04-21T23:07:57.831 INFO:journalctl@ceph.mon.a.smithi040.stdout: index_shortening: 1 2024-04-21T23:07:57.831 INFO:journalctl@ceph.mon.a.smithi040.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-21T23:07:57.831 INFO:journalctl@ceph.mon.a.smithi040.stdout: checksum: 4 2024-04-21T23:07:57.831 INFO:journalctl@ceph.mon.a.smithi040.stdout: no_block_cache: 0 2024-04-21T23:07:57.831 INFO:journalctl@ceph.mon.a.smithi040.stdout: block_cache: 0x55fe51fdb090 2024-04-21T23:07:57.831 INFO:journalctl@ceph.mon.a.smithi040.stdout: block_cache_name: BinnedLRUCache 2024-04-21T23:07:57.831 INFO:journalctl@ceph.mon.a.smithi040.stdout: block_cache_options: 2024-04-21T23:07:57.831 INFO:journalctl@ceph.mon.a.smithi040.stdout: capacity : 536870912 2024-04-21T23:07:57.832 INFO:journalctl@ceph.mon.a.smithi040.stdout: num_shard_bits : 4 2024-04-21T23:07:57.832 INFO:journalctl@ceph.mon.a.smithi040.stdout: strict_capacity_limit : 0 2024-04-21T23:07:57.832 INFO:journalctl@ceph.mon.a.smithi040.stdout: high_pri_pool_ratio: 0.000 2024-04-21T23:07:57.832 INFO:journalctl@ceph.mon.a.smithi040.stdout: block_cache_compressed: (nil) 2024-04-21T23:07:57.832 INFO:journalctl@ceph.mon.a.smithi040.stdout: persistent_cache: (nil) 2024-04-21T23:07:57.832 INFO:journalctl@ceph.mon.a.smithi040.stdout: block_size: 4096 2024-04-21T23:07:57.832 INFO:journalctl@ceph.mon.a.smithi040.stdout: block_size_deviation: 10 2024-04-21T23:07:57.832 INFO:journalctl@ceph.mon.a.smithi040.stdout: block_restart_interval: 16 2024-04-21T23:07:57.832 INFO:journalctl@ceph.mon.a.smithi040.stdout: index_block_restart_interval: 1 2024-04-21T23:07:57.832 INFO:journalctl@ceph.mon.a.smithi040.stdout: metadata_block_size: 4096 2024-04-21T23:07:57.832 INFO:journalctl@ceph.mon.a.smithi040.stdout: partition_filters: 0 2024-04-21T23:07:57.832 INFO:journalctl@ceph.mon.a.smithi040.stdout: use_delta_encoding: 1 2024-04-21T23:07:57.832 INFO:journalctl@ceph.mon.a.smithi040.stdout: filter_policy: bloomfilter 2024-04-21T23:07:57.833 INFO:journalctl@ceph.mon.a.smithi040.stdout: whole_key_filtering: 1 2024-04-21T23:07:57.833 INFO:journalctl@ceph.mon.a.smithi040.stdout: verify_compression: 0 2024-04-21T23:07:57.833 INFO:journalctl@ceph.mon.a.smithi040.stdout: read_amp_bytes_per_bit: 0 2024-04-21T23:07:57.833 INFO:journalctl@ceph.mon.a.smithi040.stdout: format_version: 5 2024-04-21T23:07:57.833 INFO:journalctl@ceph.mon.a.smithi040.stdout: enable_index_compression: 1 2024-04-21T23:07:57.833 INFO:journalctl@ceph.mon.a.smithi040.stdout: block_align: 0 2024-04-21T23:07:57.833 INFO:journalctl@ceph.mon.a.smithi040.stdout: max_auto_readahead_size: 262144 2024-04-21T23:07:57.833 INFO:journalctl@ceph.mon.a.smithi040.stdout: prepopulate_block_cache: 0 2024-04-21T23:07:57.833 INFO:journalctl@ceph.mon.a.smithi040.stdout: initial_auto_readahead_size: 8192 2024-04-21T23:07:57.833 INFO:journalctl@ceph.mon.a.smithi040.stdout: num_file_reads_for_auto_readahead: 2 2024-04-21T23:07:57.833 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.write_buffer_size: 33554432 2024-04-21T23:07:57.833 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.max_write_buffer_number: 2 2024-04-21T23:07:57.833 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.compression: NoCompression 2024-04-21T23:07:57.833 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.bottommost_compression: Disabled 2024-04-21T23:07:57.834 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.prefix_extractor: nullptr 2024-04-21T23:07:57.834 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-21T23:07:57.834 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.num_levels: 7 2024-04-21T23:07:57.834 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-21T23:07:57.834 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-21T23:07:57.834 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-21T23:07:57.834 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-21T23:07:57.834 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-21T23:07:57.834 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-21T23:07:57.834 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-21T23:07:57.834 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-21T23:07:57.834 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-21T23:07:57.834 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-21T23:07:57.835 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-21T23:07:57.835 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-21T23:07:57.835 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-21T23:07:57.835 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.compression_opts.level: 32767 2024-04-21T23:07:57.835 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.compression_opts.strategy: 0 2024-04-21T23:07:57.835 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-21T23:07:57.835 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-21T23:07:57.835 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-21T23:07:57.835 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-21T23:07:57.835 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.compression_opts.enabled: false 2024-04-21T23:07:57.835 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-21T23:07:57.835 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-21T23:07:57.835 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-21T23:07:57.835 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-21T23:07:57.836 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.target_file_size_base: 67108864 2024-04-21T23:07:57.836 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-21T23:07:57.836 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-21T23:07:57.836 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-21T23:07:57.836 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-21T23:07:57.836 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-21T23:07:57.836 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-21T23:07:57.836 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-21T23:07:57.836 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-21T23:07:57.836 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-21T23:07:57.836 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-21T23:07:57.836 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-21T23:07:57.836 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-21T23:07:57.836 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-21T23:07:57.837 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-21T23:07:57.837 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.arena_block_size: 1048576 2024-04-21T23:07:57.837 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-21T23:07:57.837 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-21T23:07:57.837 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.disable_auto_compactions: 0 2024-04-21T23:07:57.837 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-21T23:07:57.837 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-21T23:07:57.837 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-21T23:07:57.837 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-21T23:07:57.837 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-21T23:07:57.837 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-21T23:07:57.837 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-21T23:07:57.837 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-21T23:07:57.837 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-21T23:07:57.838 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-21T23:07:57.838 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.table_properties_collectors: 2024-04-21T23:07:57.838 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.inplace_update_support: 0 2024-04-21T23:07:57.838 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-21T23:07:57.838 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-21T23:07:57.838 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-21T23:07:57.838 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-21T23:07:57.838 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.bloom_locality: 0 2024-04-21T23:07:57.838 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.max_successive_merges: 0 2024-04-21T23:07:57.838 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-21T23:07:57.838 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.paranoid_file_checks: 0 2024-04-21T23:07:57.838 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.force_consistency_checks: 1 2024-04-21T23:07:57.838 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.report_bg_io_stats: 0 2024-04-21T23:07:57.838 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.ttl: 2592000 2024-04-21T23:07:57.839 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-21T23:07:57.839 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-21T23:07:57.839 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-21T23:07:57.839 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.enable_blob_files: false 2024-04-21T23:07:57.839 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.min_blob_size: 0 2024-04-21T23:07:57.839 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.blob_file_size: 268435456 2024-04-21T23:07:57.839 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-21T23:07:57.839 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-21T23:07:57.839 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-21T23:07:57.839 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-21T23:07:57.839 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-21T23:07:57.839 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.blob_file_starting_level: 0 2024-04-21T23:07:57.839 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-21T23:07:57.839 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-21T23:07:57.840 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-21T23:07:57.840 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: c4dded4c-9397-4fe4-a202-515581106d6c 2024-04-21T23:07:57.840 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713740877760015, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-21T23:07:57.840 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-21T23:07:57.840 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713740877761061, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 99889, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 326, "table_properties": {"data_size": 97799, "index_size": 308, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14175, "raw_average_key_size": 49, "raw_value_size": 90100, "raw_average_value_size": 316, "num_data_blocks": 12, "num_entries": 285, "num_filter_entries": 285, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713740877, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "c4dded4c-9397-4fe4-a202-515581106d6c", "db_session_id": "LT2AMW25P3A6BK69FQ2Z", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-21T23:07:57.840 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713740877761146, "job": 1, "event": "recovery_finished"} 2024-04-21T23:07:57.840 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-21T23:07:57.840 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-21T23:07:57.840 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55fe520cc000 2024-04-21T23:07:57.840 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: DB pointer 0x55fe520b6000 2024-04-21T23:07:57.840 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-21T23:07:57.840 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-21T23:07:57.840 INFO:journalctl@ceph.mon.a.smithi040.stdout: ** DB Stats ** 2024-04-21T23:07:57.841 INFO:journalctl@ceph.mon.a.smithi040.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-21T23:07:57.841 INFO:journalctl@ceph.mon.a.smithi040.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-21T23:07:57.841 INFO:journalctl@ceph.mon.a.smithi040.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-21T23:07:57.841 INFO:journalctl@ceph.mon.a.smithi040.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-21T23:07:57.841 INFO:journalctl@ceph.mon.a.smithi040.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-21T23:07:57.841 INFO:journalctl@ceph.mon.a.smithi040.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-21T23:07:57.841 INFO:journalctl@ceph.mon.a.smithi040.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-21T23:07:57.841 INFO:journalctl@ceph.mon.a.smithi040.stdout: 2024-04-21T23:07:57.841 INFO:journalctl@ceph.mon.a.smithi040.stdout: ** Compaction Stats [default] ** 2024-04-21T23:07:57.841 INFO:journalctl@ceph.mon.a.smithi040.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-21T23:07:57.841 INFO:journalctl@ceph.mon.a.smithi040.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-21T23:07:57.841 INFO:journalctl@ceph.mon.a.smithi040.stdout: L0 2/0 99.38 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 149.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-21T23:07:57.841 INFO:journalctl@ceph.mon.a.smithi040.stdout: Sum 2/0 99.38 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 149.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-21T23:07:57.841 INFO:journalctl@ceph.mon.a.smithi040.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 149.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-21T23:07:57.842 INFO:journalctl@ceph.mon.a.smithi040.stdout: 2024-04-21T23:07:57.842 INFO:journalctl@ceph.mon.a.smithi040.stdout: ** Compaction Stats [default] ** 2024-04-21T23:07:57.842 INFO:journalctl@ceph.mon.a.smithi040.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-21T23:07:57.842 INFO:journalctl@ceph.mon.a.smithi040.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-21T23:07:57.842 INFO:journalctl@ceph.mon.a.smithi040.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 149.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-21T23:07:57.842 INFO:journalctl@ceph.mon.a.smithi040.stdout: 2024-04-21T23:07:57.842 INFO:journalctl@ceph.mon.a.smithi040.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-21T23:07:57.842 INFO:journalctl@ceph.mon.a.smithi040.stdout: 2024-04-21T23:07:57.842 INFO:journalctl@ceph.mon.a.smithi040.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-21T23:07:57.842 INFO:journalctl@ceph.mon.a.smithi040.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-21T23:07:57.842 INFO:journalctl@ceph.mon.a.smithi040.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-21T23:07:57.842 INFO:journalctl@ceph.mon.a.smithi040.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-21T23:07:57.842 INFO:journalctl@ceph.mon.a.smithi040.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-21T23:07:57.843 INFO:journalctl@ceph.mon.a.smithi040.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-21T23:07:57.843 INFO:journalctl@ceph.mon.a.smithi040.stdout: Cumulative compaction: 0.00 GB write, 30.76 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-21T23:07:57.843 INFO:journalctl@ceph.mon.a.smithi040.stdout: Interval compaction: 0.00 GB write, 30.76 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-21T23:07:57.843 INFO:journalctl@ceph.mon.a.smithi040.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-21T23:07:57.843 INFO:journalctl@ceph.mon.a.smithi040.stdout: Block cache BinnedLRUCache@0x55fe51fdb090#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-04-21T23:07:57.843 INFO:journalctl@ceph.mon.a.smithi040.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-04-21T23:07:57.843 INFO:journalctl@ceph.mon.a.smithi040.stdout: 2024-04-21T23:07:57.843 INFO:journalctl@ceph.mon.a.smithi040.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-21T23:07:57.843 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: starting mon.a rank 0 at public addrs [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] at bind addrs [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid d814e4dc-0033-11ef-bc93-c7b262605968 2024-04-21T23:07:57.843 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: mon.a@-1(???) e1 preinit fsid d814e4dc-0033-11ef-bc93-c7b262605968 2024-04-21T23:07:57.843 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: mon.a@-1(???).mds e1 new map 2024-04-21T23:07:57.843 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: mon.a@-1(???).mds e1 print_map 2024-04-21T23:07:57.843 INFO:journalctl@ceph.mon.a.smithi040.stdout: e1 2024-04-21T23:07:57.843 INFO:journalctl@ceph.mon.a.smithi040.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-21T23:07:57.844 INFO:journalctl@ceph.mon.a.smithi040.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-21T23:07:57.844 INFO:journalctl@ceph.mon.a.smithi040.stdout: legacy client fscid: -1 2024-04-21T23:07:57.844 INFO:journalctl@ceph.mon.a.smithi040.stdout: 2024-04-21T23:07:57.844 INFO:journalctl@ceph.mon.a.smithi040.stdout: No filesystems configured 2024-04-21T23:07:57.844 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-21T23:07:57.844 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-21T23:07:57.844 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-21T23:07:57.844 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-21T23:07:57.844 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-21T23:07:57.844 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 bash[27965]: 5fb73014130c79a2b3ca1e60ee21c825a0f1f4d38399689ce2bc910dfe6881aa 2024-04-21T23:07:57.844 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 systemd[1]: Started Ceph mon.a for d814e4dc-0033-11ef-bc93-c7b262605968. 2024-04-21T23:07:57.844 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-21T23:07:57.844 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: monmap epoch 1 2024-04-21T23:07:57.844 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: fsid d814e4dc-0033-11ef-bc93-c7b262605968 2024-04-21T23:07:57.845 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: last_changed 2024-04-21T23:07:49.696814+0000 2024-04-21T23:07:57.845 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: created 2024-04-21T23:07:49.696814+0000 2024-04-21T23:07:57.845 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: min_mon_release 19 (squid) 2024-04-21T23:07:57.845 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: election_strategy: 1 2024-04-21T23:07:57.845 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: 0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-21T23:07:57.845 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: fsmap 2024-04-21T23:07:57.845 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: osdmap e1: 0 total, 0 up, 0 in 2024-04-21T23:07:57.845 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:57 smithi040 ceph-mon[27984]: mgrmap e1: no daemons active 2024-04-21T23:07:58.885 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:58.305+0000 7f887ddb7700 1 Processor -- start 2024-04-21T23:07:58.885 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:58.305+0000 7f887ddb7700 1 -- start start 2024-04-21T23:07:58.885 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:58.306+0000 7f887ddb7700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8878106b70 0x7f8878106f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:58.885 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:58.306+0000 7f887ddb7700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8878107560 con 0x7f8878106b70 2024-04-21T23:07:58.885 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:58.306+0000 7f88777fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8878106b70 0x7f8878106f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:58.885 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:58.306+0000 7f88777fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8878106b70 0x7f8878106f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44754/0 (socket says 172.21.15.40:44754) 2024-04-21T23:07:58.885 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:58.306+0000 7f88777fe700 1 -- 172.21.15.40:0/2438708097 learned_addr learned my addr 172.21.15.40:0/2438708097 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:07:58.885 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:58.306+0000 7f88777fe700 1 -- 172.21.15.40:0/2438708097 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8878107d70 con 0x7f8878106b70 2024-04-21T23:07:58.885 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:58.306+0000 7f88777fe700 1 --2- 172.21.15.40:0/2438708097 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8878106b70 0x7f8878106f90 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f88680098a0 tx=0x7f8868009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9aba671c1ce661 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:07:58.885 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:58.306+0000 7f8876ffd700 1 -- 172.21.15.40:0/2438708097 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8868004160 con 0x7f8878106b70 2024-04-21T23:07:58.885 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:58.306+0000 7f8876ffd700 1 -- 172.21.15.40:0/2438708097 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f8868004300 con 0x7f8878106b70 2024-04-21T23:07:58.885 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:58.307+0000 7f8876ffd700 1 -- 172.21.15.40:0/2438708097 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8868011b50 con 0x7f8878106b70 2024-04-21T23:07:58.886 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:58.307+0000 7f887ddb7700 1 -- 172.21.15.40:0/2438708097 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8878106b70 msgr2=0x7f8878106f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:58.886 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:58.307+0000 7f887ddb7700 1 --2- 172.21.15.40:0/2438708097 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8878106b70 0x7f8878106f90 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f88680098a0 tx=0x7f8868009bb0 comp rx=0 tx=0).stop 2024-04-21T23:07:58.886 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:58.307+0000 7f887ddb7700 1 -- 172.21.15.40:0/2438708097 shutdown_connections 2024-04-21T23:07:58.886 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:58.307+0000 7f887ddb7700 1 --2- 172.21.15.40:0/2438708097 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8878106b70 0x7f8878106f90 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:58.886 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:58.307+0000 7f887ddb7700 1 -- 172.21.15.40:0/2438708097 >> 172.21.15.40:0/2438708097 conn(0x7f8878102130 msgr2=0x7f8878104550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:07:58.886 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:58.307+0000 7f887ddb7700 1 -- 172.21.15.40:0/2438708097 shutdown_connections 2024-04-21T23:07:58.886 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:58.307+0000 7f887ddb7700 1 -- 172.21.15.40:0/2438708097 wait complete. 2024-04-21T23:07:58.886 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:58.308+0000 7f887ddb7700 1 Processor -- start 2024-04-21T23:07:58.886 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:58.308+0000 7f887ddb7700 1 -- start start 2024-04-21T23:07:58.886 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:58.308+0000 7f887ddb7700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8878106b70 0x7f8878191d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:58.886 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:58.308+0000 7f887ddb7700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f887810b120 con 0x7f8878106b70 2024-04-21T23:07:58.886 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:58.308+0000 7f88777fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8878106b70 0x7f8878191d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:58.886 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:58.308+0000 7f88777fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8878106b70 0x7f8878191d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44768/0 (socket says 172.21.15.40:44768) 2024-04-21T23:07:58.887 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:58.308+0000 7f88777fe700 1 -- 172.21.15.40:0/3758127487 learned_addr learned my addr 172.21.15.40:0/3758127487 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:07:58.887 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:58.309+0000 7f88777fe700 1 -- 172.21.15.40:0/3758127487 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8878192260 con 0x7f8878106b70 2024-04-21T23:07:58.887 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:58.309+0000 7f88777fe700 1 --2- 172.21.15.40:0/3758127487 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8878106b70 0x7f8878191d20 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f8868011850 tx=0x7f8868011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:07:58.887 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:58.309+0000 7f88757fa700 1 -- 172.21.15.40:0/3758127487 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8868003af0 con 0x7f8878106b70 2024-04-21T23:07:58.887 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:58.309+0000 7f887ddb7700 1 -- 172.21.15.40:0/3758127487 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8878190450 con 0x7f8878106b70 2024-04-21T23:07:58.887 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:58.309+0000 7f88757fa700 1 -- 172.21.15.40:0/3758127487 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f8868020070 con 0x7f8878106b70 2024-04-21T23:07:58.887 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:58.309+0000 7f88757fa700 1 -- 172.21.15.40:0/3758127487 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f88680195a0 con 0x7f8878106b70 2024-04-21T23:07:58.887 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:58.309+0000 7f887ddb7700 1 -- 172.21.15.40:0/3758127487 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8878190990 con 0x7f8878106b70 2024-04-21T23:07:58.887 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:58.309+0000 7f88757fa700 1 -- 172.21.15.40:0/3758127487 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f8868019c30 con 0x7f8878106b70 2024-04-21T23:07:58.887 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:58.309+0000 7f88757fa700 1 -- 172.21.15.40:0/3758127487 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f8868011e00 con 0x7f8878106b70 2024-04-21T23:07:58.887 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:58.310+0000 7f887ddb7700 1 -- 172.21.15.40:0/3758127487 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8878063890 con 0x7f8878106b70 2024-04-21T23:07:58.887 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:58.312+0000 7f88757fa700 1 -- 172.21.15.40:0/3758127487 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f8868003c90 con 0x7f8878106b70 2024-04-21T23:07:58.887 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:58.382+0000 7f887ddb7700 1 -- 172.21.15.40:0/3758127487 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7f8878195630 con 0x7f8878106b70 2024-04-21T23:07:58.888 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:58.384+0000 7f88757fa700 1 -- 172.21.15.40:0/3758127487 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7f886801a430 con 0x7f8878106b70 2024-04-21T23:07:58.888 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:58.385+0000 7f887ddb7700 1 -- 172.21.15.40:0/3758127487 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8878106b70 msgr2=0x7f8878191d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:58.888 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:58.385+0000 7f887ddb7700 1 --2- 172.21.15.40:0/3758127487 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8878106b70 0x7f8878191d20 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f8868011850 tx=0x7f8868011c10 comp rx=0 tx=0).stop 2024-04-21T23:07:58.888 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:58.385+0000 7f887ddb7700 1 -- 172.21.15.40:0/3758127487 shutdown_connections 2024-04-21T23:07:58.888 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:58.385+0000 7f887ddb7700 1 --2- 172.21.15.40:0/3758127487 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8878106b70 0x7f8878191d20 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:58.888 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:58.385+0000 7f887ddb7700 1 -- 172.21.15.40:0/3758127487 >> 172.21.15.40:0/3758127487 conn(0x7f8878102130 msgr2=0x7f8878103030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:07:58.888 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:58.385+0000 7f887ddb7700 1 -- 172.21.15.40:0/3758127487 shutdown_connections 2024-04-21T23:07:58.888 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:07:58.386+0000 7f887ddb7700 1 -- 172.21.15.40:0/3758127487 wait complete. 2024-04-21T23:07:58.888 INFO:teuthology.orchestra.run.smithi040.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-21T23:07:58.902 INFO:teuthology.orchestra.run.smithi040.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-21T23:07:58.902 INFO:teuthology.orchestra.run.smithi040.stdout:Creating mgr... 2024-04-21T23:07:58.902 INFO:teuthology.orchestra.run.smithi040.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-21T23:07:58.903 INFO:teuthology.orchestra.run.smithi040.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-21T23:07:59.154 INFO:teuthology.orchestra.run.smithi040.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-d814e4dc-0033-11ef-bc93-c7b262605968@mgr.x 2024-04-21T23:07:59.154 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stderr Failed to reset failed state of unit ceph-d814e4dc-0033-11ef-bc93-c7b262605968@mgr.x.service: Unit ceph-d814e4dc-0033-11ef-bc93-c7b262605968@mgr.x.service not loaded. 2024-04-21T23:07:59.311 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-d814e4dc-0033-11ef-bc93-c7b262605968.target.wants/ceph-d814e4dc-0033-11ef-bc93-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-d814e4dc-0033-11ef-bc93-c7b262605968@.service. 2024-04-21T23:07:59.335 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:07:59 smithi040 systemd[1]: Starting Ceph mgr.x for d814e4dc-0033-11ef-bc93-c7b262605968... 2024-04-21T23:07:59.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:07:59 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/3758127487' entity='client.admin' 2024-04-21T23:07:59.682 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:07:59 smithi040 podman[28200]: 2024-04-21 23:07:59.458231843 +0000 UTC m=+0.018883575 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b 2024-04-21T23:08:00.932 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:00 smithi040 podman[28200]: 2024-04-21 23:08:00.648688129 +0000 UTC m=+1.209339849 container create 7b8d28516472063ffc63f57bbf7e41b8b6030acb159c557b367597f51489171c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=squid-820a2f8, org.label-schema.build-date=20240416, GIT_CLEAN=True) 2024-04-21T23:08:02.893 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:02 smithi040 podman[28200]: 2024-04-21 23:08:02.522807295 +0000 UTC m=+3.083459016 container init 7b8d28516472063ffc63f57bbf7e41b8b6030acb159c557b367597f51489171c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, RELEASE=squid-820a2f8, org.label-schema.schema-version=1.0, ceph=True) 2024-04-21T23:08:02.893 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:02 smithi040 podman[28200]: 2024-04-21 23:08:02.526575066 +0000 UTC m=+3.087226787 container start 7b8d28516472063ffc63f57bbf7e41b8b6030acb159c557b367597f51489171c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=squid-820a2f8, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T23:08:02.893 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:02 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:08:02.697+0000 7fe8787bb200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-21T23:08:02.893 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:02 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:08:02.893+0000 7fe8787bb200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-21T23:08:03.182 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:03 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:08:03.008+0000 7fe8787bb200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-21T23:08:03.415 INFO:teuthology.orchestra.run.smithi040.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-21T23:08:03.416 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stdout disabled 2024-04-21T23:08:03.423 INFO:teuthology.orchestra.run.smithi040.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-21T23:08:03.423 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stdout inactive 2024-04-21T23:08:03.423 INFO:teuthology.orchestra.run.smithi040.stdout:firewalld.service is not enabled 2024-04-21T23:08:03.423 INFO:teuthology.orchestra.run.smithi040.stdout:Not possible to enable service . firewalld.service is not available 2024-04-21T23:08:03.432 INFO:teuthology.orchestra.run.smithi040.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-21T23:08:03.432 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stdout disabled 2024-04-21T23:08:03.439 INFO:teuthology.orchestra.run.smithi040.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-21T23:08:03.439 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stdout inactive 2024-04-21T23:08:03.439 INFO:teuthology.orchestra.run.smithi040.stdout:firewalld.service is not enabled 2024-04-21T23:08:03.440 INFO:teuthology.orchestra.run.smithi040.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-04-21T23:08:03.440 INFO:teuthology.orchestra.run.smithi040.stdout:Waiting for mgr to start... 2024-04-21T23:08:03.440 INFO:teuthology.orchestra.run.smithi040.stdout:Waiting for mgr... 2024-04-21T23:08:03.682 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:03 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:08:03.337+0000 7fe8787bb200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-21T23:08:03.682 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:03 smithi040 bash[28200]: 7b8d28516472063ffc63f57bbf7e41b8b6030acb159c557b367597f51489171c 2024-04-21T23:08:03.682 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:03 smithi040 systemd[1]: Started Ceph mgr.x for d814e4dc-0033-11ef-bc93-c7b262605968. 2024-04-21T23:08:03.682 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:03 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:08:03.428+0000 7fe8787bb200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-21T23:08:04.876 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:04 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:08:04.700+0000 7fe8787bb200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-21T23:08:04.876 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:04 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:08:04.876+0000 7fe8787bb200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-21T23:08:05.182 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:05 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:08:05.052+0000 7fe8787bb200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-21T23:08:05.661 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:05 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:08:05.352+0000 7fe8787bb200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-21T23:08:05.661 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:05 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:08:05.448+0000 7fe8787bb200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-21T23:08:05.661 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:05 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:08:05.661+0000 7fe8787bb200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-21T23:08:05.932 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:05 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:08:05.861+0000 7fe8787bb200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-21T23:08:06.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:06 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/3090476788' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-21T23:08:06.682 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:06 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:08:06.492+0000 7fe8787bb200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-21T23:08:07.229 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:06 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:08:06.937+0000 7fe8787bb200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-21T23:08:07.229 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:07 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:08:07.037+0000 7fe8787bb200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-21T23:08:07.682 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:07 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:08:07.246+0000 7fe8787bb200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-21T23:08:08.141 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:07 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:08:07.794+0000 7fe8787bb200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-21T23:08:08.169 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:07 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:08:07.889+0000 7fe8787bb200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-21T23:08:08.432 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:08 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:08:08.141+0000 7fe8787bb200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-21T23:08:08.432 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:08 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:08:08.281+0000 7fe8787bb200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-21T23:08:08.432 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:08 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:08:08.380+0000 7fe8787bb200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-21T23:08:08.932 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:08 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:08:08.498+0000 7fe8787bb200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-21T23:08:08.932 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:08 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:08:08.594+0000 7fe8787bb200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-21T23:08:09.932 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:09 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:08:09.457+0000 7fe8787bb200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-21T23:08:09.932 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:09 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:08:09.680+0000 7fe8787bb200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-21T23:08:09.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:09 smithi040 ceph-mon[27984]: Activating manager daemon x 2024-04-21T23:08:09.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:09 smithi040 ceph-mon[27984]: mgrmap e2: x(active, starting, since 0.0031915s) 2024-04-21T23:08:09.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:09 smithi040 ceph-mon[27984]: from='mgr.14100 172.21.15.40:0/3976242814' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-21T23:08:09.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:09 smithi040 ceph-mon[27984]: from='mgr.14100 172.21.15.40:0/3976242814' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-21T23:08:09.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:09 smithi040 ceph-mon[27984]: from='mgr.14100 172.21.15.40:0/3976242814' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-21T23:08:09.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:09 smithi040 ceph-mon[27984]: from='mgr.14100 172.21.15.40:0/3976242814' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T23:08:09.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:09 smithi040 ceph-mon[27984]: from='mgr.14100 172.21.15.40:0/3976242814' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-21T23:08:09.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:09 smithi040 ceph-mon[27984]: Manager daemon x is now available 2024-04-21T23:08:09.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:09 smithi040 ceph-mon[27984]: from='mgr.14100 172.21.15.40:0/3976242814' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-21T23:08:09.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:09 smithi040 ceph-mon[27984]: from='mgr.14100 172.21.15.40:0/3976242814' entity='mgr.x' 2024-04-21T23:08:09.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:09 smithi040 ceph-mon[27984]: from='mgr.14100 172.21.15.40:0/3976242814' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-21T23:08:09.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:09 smithi040 ceph-mon[27984]: from='mgr.14100 172.21.15.40:0/3976242814' entity='mgr.x' 2024-04-21T23:08:09.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:09 smithi040 ceph-mon[27984]: from='mgr.14100 172.21.15.40:0/3976242814' entity='mgr.x' 2024-04-21T23:08:11.351 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 2024-04-21T23:08:11.361 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout { 2024-04-21T23:08:11.361 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "fsid": "d814e4dc-0033-11ef-bc93-c7b262605968", 2024-04-21T23:08:11.362 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "health": { 2024-04-21T23:08:11.362 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-21T23:08:11.362 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-21T23:08:11.362 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-21T23:08:11.362 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-04-21T23:08:11.362 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-21T23:08:11.362 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-21T23:08:11.362 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 0 2024-04-21T23:08:11.362 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout ], 2024-04-21T23:08:11.362 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-21T23:08:11.362 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "a" 2024-04-21T23:08:11.362 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout ], 2024-04-21T23:08:11.363 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-04-21T23:08:11.363 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-21T23:08:11.363 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T23:08:11.363 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-21T23:08:11.363 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-21T23:08:11.363 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-04-21T23:08:11.363 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-21T23:08:11.363 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T23:08:11.363 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-21T23:08:11.363 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-21T23:08:11.363 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-21T23:08:11.363 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-21T23:08:11.363 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-21T23:08:11.363 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-21T23:08:11.363 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-04-21T23:08:11.364 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-21T23:08:11.364 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-21T23:08:11.364 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-21T23:08:11.364 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-21T23:08:11.364 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-21T23:08:11.364 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-21T23:08:11.364 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-21T23:08:11.364 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-21T23:08:11.364 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-21T23:08:11.364 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-04-21T23:08:11.364 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-21T23:08:11.364 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T23:08:11.364 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-21T23:08:11.364 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-21T23:08:11.364 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-04-21T23:08:11.365 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-21T23:08:11.365 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-21T23:08:11.365 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-21T23:08:11.365 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-21T23:08:11.365 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-21T23:08:11.365 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-21T23:08:11.365 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "restful" 2024-04-21T23:08:11.365 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout ], 2024-04-21T23:08:11.365 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-21T23:08:11.365 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-04-21T23:08:11.365 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-21T23:08:11.365 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T23:08:11.365 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "modified": "2024-04-21T23:07:52.659683+0000", 2024-04-21T23:08:11.365 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-21T23:08:11.365 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-04-21T23:08:11.366 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-21T23:08:11.366 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout } 2024-04-21T23:08:11.366 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:06.499+0000 7fa168043700 1 Processor -- start 2024-04-21T23:08:11.366 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:06.499+0000 7fa168043700 1 -- start start 2024-04-21T23:08:11.366 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:06.499+0000 7fa168043700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa160108710 0x7fa16010ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:11.366 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:06.499+0000 7fa168043700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa16007be30 con 0x7fa160108710 2024-04-21T23:08:11.366 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:06.499+0000 7fa165ddf700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa160108710 0x7fa16010ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:11.366 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:06.500+0000 7fa165ddf700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa160108710 0x7fa16010ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60300/0 (socket says 172.21.15.40:60300) 2024-04-21T23:08:11.366 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:06.500+0000 7fa165ddf700 1 -- 172.21.15.40:0/21373882 learned_addr learned my addr 172.21.15.40:0/21373882 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:08:11.367 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:06.500+0000 7fa165ddf700 1 -- 172.21.15.40:0/21373882 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa16010b040 con 0x7fa160108710 2024-04-21T23:08:11.367 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:06.500+0000 7fa165ddf700 1 --2- 172.21.15.40:0/21373882 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa160108710 0x7fa16010ab00 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fa15c0098a0 tx=0x7fa15c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d8d6cb889858f73 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:11.367 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:06.500+0000 7fa164ddd700 1 -- 172.21.15.40:0/21373882 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa15c004160 con 0x7fa160108710 2024-04-21T23:08:11.367 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:06.500+0000 7fa164ddd700 1 -- 172.21.15.40:0/21373882 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fa15c004300 con 0x7fa160108710 2024-04-21T23:08:11.367 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:06.500+0000 7fa164ddd700 1 -- 172.21.15.40:0/21373882 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa15c011b50 con 0x7fa160108710 2024-04-21T23:08:11.367 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:06.501+0000 7fa168043700 1 -- 172.21.15.40:0/21373882 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa160108710 msgr2=0x7fa16010ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:11.367 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:06.501+0000 7fa168043700 1 --2- 172.21.15.40:0/21373882 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa160108710 0x7fa16010ab00 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fa15c0098a0 tx=0x7fa15c009bb0 comp rx=0 tx=0).stop 2024-04-21T23:08:11.367 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:06.501+0000 7fa168043700 1 -- 172.21.15.40:0/21373882 shutdown_connections 2024-04-21T23:08:11.367 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:06.501+0000 7fa168043700 1 --2- 172.21.15.40:0/21373882 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa160108710 0x7fa16010ab00 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:11.367 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:06.501+0000 7fa168043700 1 -- 172.21.15.40:0/21373882 >> 172.21.15.40:0/21373882 conn(0x7fa1601020a0 msgr2=0x7fa160104500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:11.367 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:06.501+0000 7fa168043700 1 -- 172.21.15.40:0/21373882 shutdown_connections 2024-04-21T23:08:11.367 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:06.501+0000 7fa168043700 1 -- 172.21.15.40:0/21373882 wait complete. 2024-04-21T23:08:11.367 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:06.502+0000 7fa168043700 1 Processor -- start 2024-04-21T23:08:11.367 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:06.502+0000 7fa168043700 1 -- start start 2024-04-21T23:08:11.367 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:06.502+0000 7fa168043700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa160108710 0x7fa16019bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:11.368 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:06.502+0000 7fa168043700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa16010c000 con 0x7fa160108710 2024-04-21T23:08:11.368 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:06.502+0000 7fa165ddf700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa160108710 0x7fa16019bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:11.368 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:06.502+0000 7fa165ddf700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa160108710 0x7fa16019bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60308/0 (socket says 172.21.15.40:60308) 2024-04-21T23:08:11.368 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:06.502+0000 7fa165ddf700 1 -- 172.21.15.40:0/3090476788 learned_addr learned my addr 172.21.15.40:0/3090476788 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:08:11.368 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:06.502+0000 7fa165ddf700 1 -- 172.21.15.40:0/3090476788 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa16019c120 con 0x7fa160108710 2024-04-21T23:08:11.368 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:06.502+0000 7fa165ddf700 1 --2- 172.21.15.40:0/3090476788 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa160108710 0x7fa16019bbe0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa15c011670 tx=0x7fa15c011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:11.368 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:06.503+0000 7fa156ffd700 1 -- 172.21.15.40:0/3090476788 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa15c0039c0 con 0x7fa160108710 2024-04-21T23:08:11.368 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:06.503+0000 7fa156ffd700 1 -- 172.21.15.40:0/3090476788 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fa15c011e50 con 0x7fa160108710 2024-04-21T23:08:11.368 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:06.503+0000 7fa168043700 1 -- 172.21.15.40:0/3090476788 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa16019c3b0 con 0x7fa160108710 2024-04-21T23:08:11.368 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:06.503+0000 7fa156ffd700 1 -- 172.21.15.40:0/3090476788 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa15c01a5c0 con 0x7fa160108710 2024-04-21T23:08:11.368 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:06.503+0000 7fa168043700 1 -- 172.21.15.40:0/3090476788 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa16019f0a0 con 0x7fa160108710 2024-04-21T23:08:11.368 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:06.503+0000 7fa156ffd700 1 -- 172.21.15.40:0/3090476788 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fa15c01ac50 con 0x7fa160108710 2024-04-21T23:08:11.368 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:06.503+0000 7fa156ffd700 1 -- 172.21.15.40:0/3090476788 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fa15c019700 con 0x7fa160108710 2024-04-21T23:08:11.368 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:06.503+0000 7fa168043700 1 -- 172.21.15.40:0/3090476788 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa160063890 con 0x7fa160108710 2024-04-21T23:08:11.368 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:06.506+0000 7fa156ffd700 1 -- 172.21.15.40:0/3090476788 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fa15c003b60 con 0x7fa160108710 2024-04-21T23:08:11.369 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:06.580+0000 7fa168043700 1 -- 172.21.15.40:0/3090476788 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fa16019f270 con 0x7fa160108710 2024-04-21T23:08:11.369 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:06.580+0000 7fa156ffd700 1 -- 172.21.15.40:0/3090476788 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fa15c0199a0 con 0x7fa160108710 2024-04-21T23:08:11.369 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:06.581+0000 7fa168043700 1 -- 172.21.15.40:0/3090476788 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa160108710 msgr2=0x7fa16019bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:11.369 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:06.581+0000 7fa168043700 1 --2- 172.21.15.40:0/3090476788 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa160108710 0x7fa16019bbe0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa15c011670 tx=0x7fa15c011b60 comp rx=0 tx=0).stop 2024-04-21T23:08:11.369 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:06.581+0000 7fa168043700 1 -- 172.21.15.40:0/3090476788 shutdown_connections 2024-04-21T23:08:11.369 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:06.581+0000 7fa168043700 1 --2- 172.21.15.40:0/3090476788 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa160108710 0x7fa16019bbe0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:11.369 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:06.581+0000 7fa168043700 1 -- 172.21.15.40:0/3090476788 >> 172.21.15.40:0/3090476788 conn(0x7fa1601020a0 msgr2=0x7fa160102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:11.369 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:06.581+0000 7fa168043700 1 -- 172.21.15.40:0/3090476788 shutdown_connections 2024-04-21T23:08:11.369 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:06.581+0000 7fa168043700 1 -- 172.21.15.40:0/3090476788 wait complete. 2024-04-21T23:08:11.369 INFO:teuthology.orchestra.run.smithi040.stdout:mgr not available, waiting (1/15)... 2024-04-21T23:08:12.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:11 smithi040 ceph-mon[27984]: mgrmap e3: x(active, since 1.00715s) 2024-04-21T23:08:13.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:12 smithi040 ceph-mon[27984]: mgrmap e4: x(active, since 2s) 2024-04-21T23:08:14.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:14 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/3240172961' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-21T23:08:14.660 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 2024-04-21T23:08:14.660 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout { 2024-04-21T23:08:14.660 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "fsid": "d814e4dc-0033-11ef-bc93-c7b262605968", 2024-04-21T23:08:14.661 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "health": { 2024-04-21T23:08:14.661 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-21T23:08:14.661 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-21T23:08:14.661 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-21T23:08:14.661 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-04-21T23:08:14.661 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-21T23:08:14.661 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-21T23:08:14.661 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 0 2024-04-21T23:08:14.661 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout ], 2024-04-21T23:08:14.661 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-21T23:08:14.661 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "a" 2024-04-21T23:08:14.661 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout ], 2024-04-21T23:08:14.661 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "quorum_age": 16, 2024-04-21T23:08:14.661 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-21T23:08:14.661 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T23:08:14.662 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-21T23:08:14.662 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-21T23:08:14.662 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-04-21T23:08:14.662 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-21T23:08:14.662 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T23:08:14.662 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-21T23:08:14.662 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-21T23:08:14.662 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-21T23:08:14.672 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-21T23:08:14.672 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-21T23:08:14.672 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-21T23:08:14.672 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-04-21T23:08:14.672 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-21T23:08:14.672 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-21T23:08:14.672 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-21T23:08:14.672 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-21T23:08:14.672 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-21T23:08:14.672 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-21T23:08:14.672 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-21T23:08:14.673 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-21T23:08:14.673 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-21T23:08:14.673 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-04-21T23:08:14.673 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-21T23:08:14.673 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T23:08:14.673 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-21T23:08:14.673 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-21T23:08:14.673 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-04-21T23:08:14.674 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-21T23:08:14.674 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-21T23:08:14.674 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-21T23:08:14.674 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-21T23:08:14.674 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-21T23:08:14.674 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-21T23:08:14.675 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "restful" 2024-04-21T23:08:14.675 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout ], 2024-04-21T23:08:14.675 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-21T23:08:14.675 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-04-21T23:08:14.675 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-21T23:08:14.675 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T23:08:14.675 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "modified": "2024-04-21T23:07:52.659683+0000", 2024-04-21T23:08:14.676 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-21T23:08:14.676 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-04-21T23:08:14.676 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-21T23:08:14.676 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout } 2024-04-21T23:08:14.676 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:13.796+0000 7f3f95f42700 1 Processor -- start 2024-04-21T23:08:14.676 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:13.796+0000 7f3f95f42700 1 -- start start 2024-04-21T23:08:14.676 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:13.796+0000 7f3f95f42700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f90107c60 0x7f3f90108080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:14.676 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:13.796+0000 7f3f95f42700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3f901085c0 con 0x7f3f90107c60 2024-04-21T23:08:14.677 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:13.796+0000 7f3f8f7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f90107c60 0x7f3f90108080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:14.677 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:13.796+0000 7f3f8f7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f90107c60 0x7f3f90108080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36712/0 (socket says 172.21.15.40:36712) 2024-04-21T23:08:14.677 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:13.796+0000 7f3f8f7fe700 1 -- 172.21.15.40:0/542719175 learned_addr learned my addr 172.21.15.40:0/542719175 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:08:14.677 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:13.797+0000 7f3f8f7fe700 1 -- 172.21.15.40:0/542719175 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f90108740 con 0x7f3f90107c60 2024-04-21T23:08:14.677 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:13.797+0000 7f3f8f7fe700 1 --2- 172.21.15.40:0/542719175 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f90107c60 0x7f3f90108080 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3f800098a0 tx=0x7f3f80009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64b6057241689880 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:14.677 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:13.797+0000 7f3f8e7fc700 1 -- 172.21.15.40:0/542719175 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3f80004160 con 0x7f3f90107c60 2024-04-21T23:08:14.678 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:13.797+0000 7f3f8e7fc700 1 -- 172.21.15.40:0/542719175 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f3f80004300 con 0x7f3f90107c60 2024-04-21T23:08:14.678 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:13.797+0000 7f3f8e7fc700 1 -- 172.21.15.40:0/542719175 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3f80011b50 con 0x7f3f90107c60 2024-04-21T23:08:14.678 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:13.798+0000 7f3f95f42700 1 -- 172.21.15.40:0/542719175 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f90107c60 msgr2=0x7f3f90108080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:14.678 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:13.798+0000 7f3f95f42700 1 --2- 172.21.15.40:0/542719175 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f90107c60 0x7f3f90108080 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3f800098a0 tx=0x7f3f80009bb0 comp rx=0 tx=0).stop 2024-04-21T23:08:14.678 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:13.798+0000 7f3f95f42700 1 -- 172.21.15.40:0/542719175 shutdown_connections 2024-04-21T23:08:14.681 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:13.798+0000 7f3f95f42700 1 --2- 172.21.15.40:0/542719175 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f90107c60 0x7f3f90108080 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:14.681 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:13.798+0000 7f3f95f42700 1 -- 172.21.15.40:0/542719175 >> 172.21.15.40:0/542719175 conn(0x7f3f901032a0 msgr2=0x7f3f901056e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:14.681 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:13.798+0000 7f3f95f42700 1 -- 172.21.15.40:0/542719175 shutdown_connections 2024-04-21T23:08:14.682 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:13.798+0000 7f3f95f42700 1 -- 172.21.15.40:0/542719175 wait complete. 2024-04-21T23:08:14.682 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:13.798+0000 7f3f95f42700 1 Processor -- start 2024-04-21T23:08:14.682 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:13.799+0000 7f3f95f42700 1 -- start start 2024-04-21T23:08:14.682 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:13.799+0000 7f3f95f42700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f90107c60 0x7f3f9007db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:14.682 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:13.799+0000 7f3f95f42700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3f9010c0d0 con 0x7f3f90107c60 2024-04-21T23:08:14.682 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:13.799+0000 7f3f8f7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f90107c60 0x7f3f9007db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:14.683 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:13.799+0000 7f3f8f7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f90107c60 0x7f3f9007db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36728/0 (socket says 172.21.15.40:36728) 2024-04-21T23:08:14.683 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:13.799+0000 7f3f8f7fe700 1 -- 172.21.15.40:0/3240172961 learned_addr learned my addr 172.21.15.40:0/3240172961 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:08:14.683 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:13.799+0000 7f3f8f7fe700 1 -- 172.21.15.40:0/3240172961 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f9007e0b0 con 0x7f3f90107c60 2024-04-21T23:08:14.683 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:13.799+0000 7f3f8f7fe700 1 --2- 172.21.15.40:0/3240172961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f90107c60 0x7f3f9007db70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3f80011670 tx=0x7f3f80011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:14.683 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:13.800+0000 7f3f8cff9700 1 -- 172.21.15.40:0/3240172961 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3f800039c0 con 0x7f3f90107c60 2024-04-21T23:08:14.683 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:13.800+0000 7f3f8cff9700 1 -- 172.21.15.40:0/3240172961 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f3f80011e50 con 0x7f3f90107c60 2024-04-21T23:08:14.684 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:13.800+0000 7f3f95f42700 1 -- 172.21.15.40:0/3240172961 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f9007c2a0 con 0x7f3f90107c60 2024-04-21T23:08:14.684 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:13.800+0000 7f3f8cff9700 1 -- 172.21.15.40:0/3240172961 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3f8001a5c0 con 0x7f3f90107c60 2024-04-21T23:08:14.684 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:13.800+0000 7f3f95f42700 1 -- 172.21.15.40:0/3240172961 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f9007c7e0 con 0x7f3f90107c60 2024-04-21T23:08:14.684 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:13.801+0000 7f3f8cff9700 1 -- 172.21.15.40:0/3240172961 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 4) ==== 46276+0+0 (secure 0 0 0) 0x7f3f8001a870 con 0x7f3f90107c60 2024-04-21T23:08:14.684 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:13.801+0000 7f3f95f42700 1 -- 172.21.15.40:0/3240172961 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f90063890 con 0x7f3f90107c60 2024-04-21T23:08:14.684 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:13.801+0000 7f3f8cff9700 1 --2- 172.21.15.40:0/3240172961 >> [v2:172.21.15.40:6800/1479040326,v1:172.21.15.40:6801/1479040326] conn(0x7f3f78039000 0x7f3f7803b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:14.685 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:13.801+0000 7f3f8cff9700 1 -- 172.21.15.40:0/3240172961 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f3f8004d5a0 con 0x7f3f90107c60 2024-04-21T23:08:14.685 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:13.801+0000 7f3f8effd700 1 --2- 172.21.15.40:0/3240172961 >> [v2:172.21.15.40:6800/1479040326,v1:172.21.15.40:6801/1479040326] conn(0x7f3f78039000 0x7f3f7803b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:14.685 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:13.801+0000 7f3f8effd700 1 --2- 172.21.15.40:0/3240172961 >> [v2:172.21.15.40:6800/1479040326,v1:172.21.15.40:6801/1479040326] conn(0x7f3f78039000 0x7f3f7803b4c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3f84006fd0 tx=0x7f3f84006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:14.685 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:13.806+0000 7f3f8cff9700 1 -- 172.21.15.40:0/3240172961 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f3f8002cc00 con 0x7f3f90107c60 2024-04-21T23:08:14.685 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:14.106+0000 7f3f95f42700 1 -- 172.21.15.40:0/3240172961 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f3f9007ca60 con 0x7f3f90107c60 2024-04-21T23:08:14.685 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:14.107+0000 7f3f8cff9700 1 -- 172.21.15.40:0/3240172961 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f3f8004cfc0 con 0x7f3f90107c60 2024-04-21T23:08:14.686 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:14.109+0000 7f3f95f42700 1 -- 172.21.15.40:0/3240172961 >> [v2:172.21.15.40:6800/1479040326,v1:172.21.15.40:6801/1479040326] conn(0x7f3f78039000 msgr2=0x7f3f7803b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:14.686 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:14.109+0000 7f3f95f42700 1 --2- 172.21.15.40:0/3240172961 >> [v2:172.21.15.40:6800/1479040326,v1:172.21.15.40:6801/1479040326] conn(0x7f3f78039000 0x7f3f7803b4c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3f84006fd0 tx=0x7f3f84006e40 comp rx=0 tx=0).stop 2024-04-21T23:08:14.686 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:14.109+0000 7f3f95f42700 1 -- 172.21.15.40:0/3240172961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f90107c60 msgr2=0x7f3f9007db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:14.686 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:14.109+0000 7f3f95f42700 1 --2- 172.21.15.40:0/3240172961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f90107c60 0x7f3f9007db70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3f80011670 tx=0x7f3f80011b60 comp rx=0 tx=0).stop 2024-04-21T23:08:14.686 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:14.109+0000 7f3f95f42700 1 -- 172.21.15.40:0/3240172961 shutdown_connections 2024-04-21T23:08:14.687 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:14.109+0000 7f3f95f42700 1 --2- 172.21.15.40:0/3240172961 >> [v2:172.21.15.40:6800/1479040326,v1:172.21.15.40:6801/1479040326] conn(0x7f3f78039000 0x7f3f7803b4c0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:14.687 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:14.109+0000 7f3f95f42700 1 --2- 172.21.15.40:0/3240172961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f90107c60 0x7f3f9007db70 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:14.687 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:14.109+0000 7f3f95f42700 1 -- 172.21.15.40:0/3240172961 >> 172.21.15.40:0/3240172961 conn(0x7f3f901032a0 msgr2=0x7f3f90103d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:14.687 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:14.109+0000 7f3f95f42700 1 -- 172.21.15.40:0/3240172961 shutdown_connections 2024-04-21T23:08:14.687 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:14.109+0000 7f3f95f42700 1 -- 172.21.15.40:0/3240172961 wait complete. 2024-04-21T23:08:14.687 INFO:teuthology.orchestra.run.smithi040.stdout:mgr is available 2024-04-21T23:08:15.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:15 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/2453595055' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-21T23:08:15.887 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 2024-04-21T23:08:15.887 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout [global] 2024-04-21T23:08:15.887 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout fsid = d814e4dc-0033-11ef-bc93-c7b262605968 2024-04-21T23:08:15.887 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-21T23:08:15.887 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-21T23:08:15.887 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-21T23:08:15.887 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-21T23:08:15.887 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 2024-04-21T23:08:15.887 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-21T23:08:15.888 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-21T23:08:15.888 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 2024-04-21T23:08:15.888 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout [osd] 2024-04-21T23:08:15.888 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-21T23:08:15.888 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-21T23:08:15.888 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-21T23:08:15.888 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-21T23:08:15.888 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-21T23:08:15.888 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:15.107+0000 7f2f9e1f8700 1 Processor -- start 2024-04-21T23:08:15.888 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:15.107+0000 7f2f9e1f8700 1 -- start start 2024-04-21T23:08:15.888 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:15.107+0000 7f2f9e1f8700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f98108720 0x7f2f9810ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:15.888 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:15.107+0000 7f2f9e1f8700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2f9807be30 con 0x7f2f98108720 2024-04-21T23:08:15.888 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:15.108+0000 7f2f977fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f98108720 0x7f2f9810ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:15.888 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:15.108+0000 7f2f977fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f98108720 0x7f2f9810ab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36736/0 (socket says 172.21.15.40:36736) 2024-04-21T23:08:15.889 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:15.108+0000 7f2f977fe700 1 -- 172.21.15.40:0/3012219296 learned_addr learned my addr 172.21.15.40:0/3012219296 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:08:15.889 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:15.108+0000 7f2f977fe700 1 -- 172.21.15.40:0/3012219296 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f9810b050 con 0x7f2f98108720 2024-04-21T23:08:15.889 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:15.108+0000 7f2f977fe700 1 --2- 172.21.15.40:0/3012219296 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f98108720 0x7f2f9810ab10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f2f900098a0 tx=0x7f2f90009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de81a492a8b44038 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:15.889 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:15.108+0000 7f2f967fc700 1 -- 172.21.15.40:0/3012219296 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2f90004160 con 0x7f2f98108720 2024-04-21T23:08:15.889 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:15.108+0000 7f2f967fc700 1 -- 172.21.15.40:0/3012219296 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f2f90004300 con 0x7f2f98108720 2024-04-21T23:08:15.889 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:15.108+0000 7f2f967fc700 1 -- 172.21.15.40:0/3012219296 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2f90011b50 con 0x7f2f98108720 2024-04-21T23:08:15.889 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:15.109+0000 7f2f9e1f8700 1 -- 172.21.15.40:0/3012219296 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f98108720 msgr2=0x7f2f9810ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:15.889 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:15.109+0000 7f2f9e1f8700 1 --2- 172.21.15.40:0/3012219296 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f98108720 0x7f2f9810ab10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f2f900098a0 tx=0x7f2f90009bb0 comp rx=0 tx=0).stop 2024-04-21T23:08:15.890 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:15.109+0000 7f2f9e1f8700 1 -- 172.21.15.40:0/3012219296 shutdown_connections 2024-04-21T23:08:15.890 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:15.109+0000 7f2f9e1f8700 1 --2- 172.21.15.40:0/3012219296 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f98108720 0x7f2f9810ab10 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:15.890 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:15.109+0000 7f2f9e1f8700 1 -- 172.21.15.40:0/3012219296 >> 172.21.15.40:0/3012219296 conn(0x7f2f98102090 msgr2=0x7f2f98104510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:15.890 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:15.109+0000 7f2f9e1f8700 1 -- 172.21.15.40:0/3012219296 shutdown_connections 2024-04-21T23:08:15.890 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:15.109+0000 7f2f9e1f8700 1 -- 172.21.15.40:0/3012219296 wait complete. 2024-04-21T23:08:15.890 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:15.110+0000 7f2f9e1f8700 1 Processor -- start 2024-04-21T23:08:15.890 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:15.110+0000 7f2f9e1f8700 1 -- start start 2024-04-21T23:08:15.890 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:15.110+0000 7f2f9e1f8700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f98108720 0x7f2f9819bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:15.890 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:15.110+0000 7f2f9e1f8700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2f9810c010 con 0x7f2f98108720 2024-04-21T23:08:15.891 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:15.110+0000 7f2f977fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f98108720 0x7f2f9819bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:15.891 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:15.110+0000 7f2f977fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f98108720 0x7f2f9819bbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36748/0 (socket says 172.21.15.40:36748) 2024-04-21T23:08:15.891 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:15.110+0000 7f2f977fe700 1 -- 172.21.15.40:0/2453595055 learned_addr learned my addr 172.21.15.40:0/2453595055 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:08:15.891 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:15.111+0000 7f2f977fe700 1 -- 172.21.15.40:0/2453595055 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f9819c110 con 0x7f2f98108720 2024-04-21T23:08:15.891 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:15.111+0000 7f2f977fe700 1 --2- 172.21.15.40:0/2453595055 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f98108720 0x7f2f9819bbd0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2f90011b60 tx=0x7f2f90011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:15.891 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:15.111+0000 7f2f94ff9700 1 -- 172.21.15.40:0/2453595055 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2f90003b20 con 0x7f2f98108720 2024-04-21T23:08:15.891 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:15.111+0000 7f2f94ff9700 1 -- 172.21.15.40:0/2453595055 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f2f90003cc0 con 0x7f2f98108720 2024-04-21T23:08:15.891 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:15.111+0000 7f2f9e1f8700 1 -- 172.21.15.40:0/2453595055 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f9819c3a0 con 0x7f2f98108720 2024-04-21T23:08:15.891 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:15.111+0000 7f2f94ff9700 1 -- 172.21.15.40:0/2453595055 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2f90024840 con 0x7f2f98108720 2024-04-21T23:08:15.891 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:15.111+0000 7f2f9e1f8700 1 -- 172.21.15.40:0/2453595055 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f9819f090 con 0x7f2f98108720 2024-04-21T23:08:15.891 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:15.112+0000 7f2f94ff9700 1 -- 172.21.15.40:0/2453595055 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 4) ==== 46276+0+0 (secure 0 0 0) 0x7f2f90022020 con 0x7f2f98108720 2024-04-21T23:08:15.891 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:15.112+0000 7f2f94ff9700 1 --2- 172.21.15.40:0/2453595055 >> [v2:172.21.15.40:6800/1479040326,v1:172.21.15.40:6801/1479040326] conn(0x7f2f84039050 0x7f2f8403b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:15.891 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:15.112+0000 7f2f9e1f8700 1 -- 172.21.15.40:0/2453595055 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f98063890 con 0x7f2f98108720 2024-04-21T23:08:15.891 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:15.112+0000 7f2f94ff9700 1 -- 172.21.15.40:0/2453595055 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f2f90011e00 con 0x7f2f98108720 2024-04-21T23:08:15.891 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:15.112+0000 7f2f96ffd700 1 --2- 172.21.15.40:0/2453595055 >> [v2:172.21.15.40:6800/1479040326,v1:172.21.15.40:6801/1479040326] conn(0x7f2f84039050 0x7f2f8403b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:15.892 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:15.113+0000 7f2f96ffd700 1 --2- 172.21.15.40:0/2453595055 >> [v2:172.21.15.40:6800/1479040326,v1:172.21.15.40:6801/1479040326] conn(0x7f2f84039050 0x7f2f8403b510 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2f80006fd0 tx=0x7f2f80006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:15.892 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:15.117+0000 7f2f94ff9700 1 -- 172.21.15.40:0/2453595055 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f2f9000b9e0 con 0x7f2f98108720 2024-04-21T23:08:15.892 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:15.324+0000 7f2f9e1f8700 1 -- 172.21.15.40:0/2453595055 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f2f9819f260 con 0x7f2f98108720 2024-04-21T23:08:15.892 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:15.325+0000 7f2f94ff9700 1 -- 172.21.15.40:0/2453595055 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+373 (secure 0 0 0) 0x7f2f9819f260 con 0x7f2f98108720 2024-04-21T23:08:15.892 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:15.328+0000 7f2f9e1f8700 1 -- 172.21.15.40:0/2453595055 >> [v2:172.21.15.40:6800/1479040326,v1:172.21.15.40:6801/1479040326] conn(0x7f2f84039050 msgr2=0x7f2f8403b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:15.892 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:15.328+0000 7f2f9e1f8700 1 --2- 172.21.15.40:0/2453595055 >> [v2:172.21.15.40:6800/1479040326,v1:172.21.15.40:6801/1479040326] conn(0x7f2f84039050 0x7f2f8403b510 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2f80006fd0 tx=0x7f2f80006e40 comp rx=0 tx=0).stop 2024-04-21T23:08:15.892 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:15.328+0000 7f2f9e1f8700 1 -- 172.21.15.40:0/2453595055 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f98108720 msgr2=0x7f2f9819bbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:15.892 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:15.328+0000 7f2f9e1f8700 1 --2- 172.21.15.40:0/2453595055 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f98108720 0x7f2f9819bbd0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2f90011b60 tx=0x7f2f90011c40 comp rx=0 tx=0).stop 2024-04-21T23:08:15.892 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:15.328+0000 7f2f9e1f8700 1 -- 172.21.15.40:0/2453595055 shutdown_connections 2024-04-21T23:08:15.893 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:15.328+0000 7f2f9e1f8700 1 --2- 172.21.15.40:0/2453595055 >> [v2:172.21.15.40:6800/1479040326,v1:172.21.15.40:6801/1479040326] conn(0x7f2f84039050 0x7f2f8403b510 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:15.893 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:15.328+0000 7f2f9e1f8700 1 --2- 172.21.15.40:0/2453595055 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f98108720 0x7f2f9819bbd0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:15.893 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:15.328+0000 7f2f9e1f8700 1 -- 172.21.15.40:0/2453595055 >> 172.21.15.40:0/2453595055 conn(0x7f2f98102090 msgr2=0x7f2f98102be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:15.893 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:15.328+0000 7f2f9e1f8700 1 -- 172.21.15.40:0/2453595055 shutdown_connections 2024-04-21T23:08:15.893 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:15.328+0000 7f2f9e1f8700 1 -- 172.21.15.40:0/2453595055 wait complete. 2024-04-21T23:08:15.893 INFO:teuthology.orchestra.run.smithi040.stdout:Enabling cephadm module... 2024-04-21T23:08:16.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:16 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/259486078' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-21T23:08:16.932 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:16 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: ignoring --setuser ceph since I am not root 2024-04-21T23:08:16.932 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:16 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: ignoring --setgroup ceph since I am not root 2024-04-21T23:08:16.932 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:16 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:08:16.822+0000 7ff5b8903200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-21T23:08:17.216 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:17 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:08:17.013+0000 7ff5b8903200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-21T23:08:17.216 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:17 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:08:17.119+0000 7ff5b8903200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-21T23:08:17.223 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:16.319+0000 7fcc6c73f700 1 Processor -- start 2024-04-21T23:08:17.223 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:16.320+0000 7fcc6c73f700 1 -- start start 2024-04-21T23:08:17.224 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:16.320+0000 7fcc6c73f700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc64107c40 0x7fcc64108060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:17.224 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:16.320+0000 7fcc6c73f700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fcc64108630 con 0x7fcc64107c40 2024-04-21T23:08:17.224 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:16.320+0000 7fcc6a4db700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc64107c40 0x7fcc64108060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:17.224 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:16.320+0000 7fcc6a4db700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc64107c40 0x7fcc64108060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36752/0 (socket says 172.21.15.40:36752) 2024-04-21T23:08:17.224 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:16.320+0000 7fcc6a4db700 1 -- 172.21.15.40:0/1140758590 learned_addr learned my addr 172.21.15.40:0/1140758590 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:08:17.224 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:16.320+0000 7fcc6a4db700 1 -- 172.21.15.40:0/1140758590 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc64108e40 con 0x7fcc64107c40 2024-04-21T23:08:17.224 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:16.321+0000 7fcc6a4db700 1 --2- 172.21.15.40:0/1140758590 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc64107c40 0x7fcc64108060 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fcc600098a0 tx=0x7fcc60009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b5e249cd5f3ac94 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:17.224 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:16.321+0000 7fcc694d9700 1 -- 172.21.15.40:0/1140758590 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcc60004160 con 0x7fcc64107c40 2024-04-21T23:08:17.224 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:16.321+0000 7fcc694d9700 1 -- 172.21.15.40:0/1140758590 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fcc60004300 con 0x7fcc64107c40 2024-04-21T23:08:17.224 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:16.321+0000 7fcc694d9700 1 -- 172.21.15.40:0/1140758590 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcc60011b50 con 0x7fcc64107c40 2024-04-21T23:08:17.224 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:16.321+0000 7fcc6c73f700 1 -- 172.21.15.40:0/1140758590 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc64107c40 msgr2=0x7fcc64108060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:17.224 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:16.321+0000 7fcc6c73f700 1 --2- 172.21.15.40:0/1140758590 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc64107c40 0x7fcc64108060 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fcc600098a0 tx=0x7fcc60009bb0 comp rx=0 tx=0).stop 2024-04-21T23:08:17.224 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:16.322+0000 7fcc6c73f700 1 -- 172.21.15.40:0/1140758590 shutdown_connections 2024-04-21T23:08:17.225 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:16.322+0000 7fcc6c73f700 1 --2- 172.21.15.40:0/1140758590 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc64107c40 0x7fcc64108060 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:17.225 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:16.322+0000 7fcc6c73f700 1 -- 172.21.15.40:0/1140758590 >> 172.21.15.40:0/1140758590 conn(0x7fcc641031e0 msgr2=0x7fcc64105620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:17.225 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:16.322+0000 7fcc6c73f700 1 -- 172.21.15.40:0/1140758590 shutdown_connections 2024-04-21T23:08:17.225 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:16.322+0000 7fcc6c73f700 1 -- 172.21.15.40:0/1140758590 wait complete. 2024-04-21T23:08:17.225 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:16.322+0000 7fcc6c73f700 1 Processor -- start 2024-04-21T23:08:17.225 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:16.322+0000 7fcc6c73f700 1 -- start start 2024-04-21T23:08:17.225 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:16.322+0000 7fcc6c73f700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc64107c40 0x7fcc6407db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:17.225 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:16.322+0000 7fcc6c73f700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fcc6410c1f0 con 0x7fcc64107c40 2024-04-21T23:08:17.225 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:16.322+0000 7fcc6a4db700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc64107c40 0x7fcc6407db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:17.225 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:16.323+0000 7fcc6a4db700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc64107c40 0x7fcc6407db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36754/0 (socket says 172.21.15.40:36754) 2024-04-21T23:08:17.225 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:16.323+0000 7fcc6a4db700 1 -- 172.21.15.40:0/259486078 learned_addr learned my addr 172.21.15.40:0/259486078 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:08:17.225 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:16.323+0000 7fcc6a4db700 1 -- 172.21.15.40:0/259486078 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc6407e0b0 con 0x7fcc64107c40 2024-04-21T23:08:17.226 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:16.323+0000 7fcc6a4db700 1 --2- 172.21.15.40:0/259486078 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc64107c40 0x7fcc6407db70 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fcc60011850 tx=0x7fcc60011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:17.226 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:16.323+0000 7fcc577fe700 1 -- 172.21.15.40:0/259486078 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcc60003af0 con 0x7fcc64107c40 2024-04-21T23:08:17.226 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:16.323+0000 7fcc577fe700 1 -- 172.21.15.40:0/259486078 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fcc60020070 con 0x7fcc64107c40 2024-04-21T23:08:17.226 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:16.323+0000 7fcc6c73f700 1 -- 172.21.15.40:0/259486078 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc6407c2a0 con 0x7fcc64107c40 2024-04-21T23:08:17.226 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:16.323+0000 7fcc577fe700 1 -- 172.21.15.40:0/259486078 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcc600195a0 con 0x7fcc64107c40 2024-04-21T23:08:17.226 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:16.323+0000 7fcc6c73f700 1 -- 172.21.15.40:0/259486078 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc6407c7e0 con 0x7fcc64107c40 2024-04-21T23:08:17.226 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:16.324+0000 7fcc577fe700 1 -- 172.21.15.40:0/259486078 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 4) ==== 46276+0+0 (secure 0 0 0) 0x7fcc6001a430 con 0x7fcc64107c40 2024-04-21T23:08:17.226 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:16.324+0000 7fcc6c73f700 1 -- 172.21.15.40:0/259486078 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc64063890 con 0x7fcc64107c40 2024-04-21T23:08:17.226 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:16.324+0000 7fcc577fe700 1 --2- 172.21.15.40:0/259486078 >> [v2:172.21.15.40:6800/1479040326,v1:172.21.15.40:6801/1479040326] conn(0x7fcc50039050 0x7fcc5003b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:17.226 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:16.324+0000 7fcc577fe700 1 -- 172.21.15.40:0/259486078 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fcc60011e00 con 0x7fcc64107c40 2024-04-21T23:08:17.226 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:16.324+0000 7fcc69cda700 1 --2- 172.21.15.40:0/259486078 >> [v2:172.21.15.40:6800/1479040326,v1:172.21.15.40:6801/1479040326] conn(0x7fcc50039050 0x7fcc5003b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:17.226 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:16.325+0000 7fcc69cda700 1 --2- 172.21.15.40:0/259486078 >> [v2:172.21.15.40:6800/1479040326,v1:172.21.15.40:6801/1479040326] conn(0x7fcc50039050 0x7fcc5003b510 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fcc58006fd0 tx=0x7fcc58006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:17.226 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:16.329+0000 7fcc577fe700 1 -- 172.21.15.40:0/259486078 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fcc6000e260 con 0x7fcc64107c40 2024-04-21T23:08:17.227 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:16.595+0000 7fcc6c73f700 1 -- 172.21.15.40:0/259486078 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7fcc6407ca60 con 0x7fcc64107c40 2024-04-21T23:08:17.227 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:16.647+0000 7fcc577fe700 1 -- 172.21.15.40:0/259486078 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7fcc6004cdf0 con 0x7fcc64107c40 2024-04-21T23:08:17.227 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:16.649+0000 7fcc577fe700 1 -- 172.21.15.40:0/259486078 <== mon.0 v2:172.21.15.40:3300/0 8 ==== mgrmap(e 5) ==== 46287+0+0 (secure 0 0 0) 0x7fcc6001a720 con 0x7fcc64107c40 2024-04-21T23:08:17.227 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:16.650+0000 7fcc6c73f700 1 -- 172.21.15.40:0/259486078 >> [v2:172.21.15.40:6800/1479040326,v1:172.21.15.40:6801/1479040326] conn(0x7fcc50039050 msgr2=0x7fcc5003b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:17.227 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:16.650+0000 7fcc6c73f700 1 --2- 172.21.15.40:0/259486078 >> [v2:172.21.15.40:6800/1479040326,v1:172.21.15.40:6801/1479040326] conn(0x7fcc50039050 0x7fcc5003b510 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fcc58006fd0 tx=0x7fcc58006e40 comp rx=0 tx=0).stop 2024-04-21T23:08:17.227 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:16.650+0000 7fcc6c73f700 1 -- 172.21.15.40:0/259486078 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc64107c40 msgr2=0x7fcc6407db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:17.227 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:16.650+0000 7fcc6c73f700 1 --2- 172.21.15.40:0/259486078 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc64107c40 0x7fcc6407db70 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fcc60011850 tx=0x7fcc60011c10 comp rx=0 tx=0).stop 2024-04-21T23:08:17.227 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:16.650+0000 7fcc6c73f700 1 -- 172.21.15.40:0/259486078 shutdown_connections 2024-04-21T23:08:17.227 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:16.650+0000 7fcc6c73f700 1 --2- 172.21.15.40:0/259486078 >> [v2:172.21.15.40:6800/1479040326,v1:172.21.15.40:6801/1479040326] conn(0x7fcc50039050 0x7fcc5003b510 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:17.227 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:16.650+0000 7fcc6c73f700 1 --2- 172.21.15.40:0/259486078 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc64107c40 0x7fcc6407db70 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:17.227 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:16.650+0000 7fcc6c73f700 1 -- 172.21.15.40:0/259486078 >> 172.21.15.40:0/259486078 conn(0x7fcc641031e0 msgr2=0x7fcc64103db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:17.227 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:16.650+0000 7fcc6c73f700 1 -- 172.21.15.40:0/259486078 shutdown_connections 2024-04-21T23:08:17.227 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:16.650+0000 7fcc6c73f700 1 -- 172.21.15.40:0/259486078 wait complete. 2024-04-21T23:08:17.471 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:17 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:08:17.442+0000 7ff5b8903200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-21T23:08:17.877 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:17 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/259486078' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-21T23:08:17.877 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:17 smithi040 ceph-mon[27984]: mgrmap e5: x(active, since 6s) 2024-04-21T23:08:17.877 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:17 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:08:17.532+0000 7ff5b8903200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-21T23:08:18.556 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout { 2024-04-21T23:08:18.556 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-21T23:08:18.556 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-21T23:08:18.556 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-21T23:08:18.557 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-21T23:08:18.557 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout } 2024-04-21T23:08:18.557 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:17.671+0000 7f976cd0b700 1 Processor -- start 2024-04-21T23:08:18.557 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:17.671+0000 7f976cd0b700 1 -- start start 2024-04-21T23:08:18.557 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:17.671+0000 7f976cd0b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9768107c90 0x7f97681080b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:18.557 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:17.671+0000 7f976cd0b700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9768108680 con 0x7f9768107c90 2024-04-21T23:08:18.557 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:17.672+0000 7f976659c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9768107c90 0x7f97681080b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:18.557 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:17.672+0000 7f976659c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9768107c90 0x7f97681080b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36780/0 (socket says 172.21.15.40:36780) 2024-04-21T23:08:18.557 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:17.672+0000 7f976659c700 1 -- 172.21.15.40:0/2164369433 learned_addr learned my addr 172.21.15.40:0/2164369433 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:08:18.557 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:17.672+0000 7f976659c700 1 -- 172.21.15.40:0/2164369433 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9768108df0 con 0x7f9768107c90 2024-04-21T23:08:18.557 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:17.672+0000 7f976659c700 1 --2- 172.21.15.40:0/2164369433 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9768107c90 0x7f97681080b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f97500098a0 tx=0x7f9750009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a604127d92a64842 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:18.557 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:17.672+0000 7f976559a700 1 -- 172.21.15.40:0/2164369433 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9750004160 con 0x7f9768107c90 2024-04-21T23:08:18.557 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:17.672+0000 7f976559a700 1 -- 172.21.15.40:0/2164369433 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f9750004300 con 0x7f9768107c90 2024-04-21T23:08:18.558 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:17.672+0000 7f976559a700 1 -- 172.21.15.40:0/2164369433 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9750011b50 con 0x7f9768107c90 2024-04-21T23:08:18.558 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:17.673+0000 7f976cd0b700 1 -- 172.21.15.40:0/2164369433 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9768107c90 msgr2=0x7f97681080b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:18.558 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:17.673+0000 7f976cd0b700 1 --2- 172.21.15.40:0/2164369433 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9768107c90 0x7f97681080b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f97500098a0 tx=0x7f9750009bb0 comp rx=0 tx=0).stop 2024-04-21T23:08:18.558 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:17.673+0000 7f976cd0b700 1 -- 172.21.15.40:0/2164369433 shutdown_connections 2024-04-21T23:08:18.558 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:17.673+0000 7f976cd0b700 1 --2- 172.21.15.40:0/2164369433 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9768107c90 0x7f97681080b0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:18.558 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:17.673+0000 7f976cd0b700 1 -- 172.21.15.40:0/2164369433 >> 172.21.15.40:0/2164369433 conn(0x7f9768103210 msgr2=0x7f9768105670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:18.558 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:17.673+0000 7f976cd0b700 1 -- 172.21.15.40:0/2164369433 shutdown_connections 2024-04-21T23:08:18.558 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:17.673+0000 7f976cd0b700 1 -- 172.21.15.40:0/2164369433 wait complete. 2024-04-21T23:08:18.558 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:17.674+0000 7f976cd0b700 1 Processor -- start 2024-04-21T23:08:18.558 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:17.674+0000 7f976cd0b700 1 -- start start 2024-04-21T23:08:18.558 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:17.674+0000 7f976cd0b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9768107c90 0x7f976819bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:18.558 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:17.674+0000 7f976cd0b700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f976810c1a0 con 0x7f9768107c90 2024-04-21T23:08:18.558 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:17.674+0000 7f976659c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9768107c90 0x7f976819bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:18.558 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:17.674+0000 7f976659c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9768107c90 0x7f976819bc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36794/0 (socket says 172.21.15.40:36794) 2024-04-21T23:08:18.559 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:17.674+0000 7f976659c700 1 -- 172.21.15.40:0/986527883 learned_addr learned my addr 172.21.15.40:0/986527883 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:08:18.559 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:17.675+0000 7f976659c700 1 -- 172.21.15.40:0/986527883 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f976819c140 con 0x7f9768107c90 2024-04-21T23:08:18.559 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:17.675+0000 7f976659c700 1 --2- 172.21.15.40:0/986527883 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9768107c90 0x7f976819bc00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9750000c00 tx=0x7f9750011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:18.559 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:17.675+0000 7f975f7fe700 1 -- 172.21.15.40:0/986527883 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9750003b20 con 0x7f9768107c90 2024-04-21T23:08:18.559 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:17.675+0000 7f975f7fe700 1 -- 172.21.15.40:0/986527883 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f9750003cc0 con 0x7f9768107c90 2024-04-21T23:08:18.559 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:17.675+0000 7f976cd0b700 1 -- 172.21.15.40:0/986527883 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f976819c3d0 con 0x7f9768107c90 2024-04-21T23:08:18.559 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:17.675+0000 7f975f7fe700 1 -- 172.21.15.40:0/986527883 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9750024910 con 0x7f9768107c90 2024-04-21T23:08:18.559 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:17.675+0000 7f976cd0b700 1 -- 172.21.15.40:0/986527883 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f976819f0c0 con 0x7f9768107c90 2024-04-21T23:08:18.559 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:17.676+0000 7f976cd0b700 1 -- 172.21.15.40:0/986527883 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9768063890 con 0x7f9768107c90 2024-04-21T23:08:18.559 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:17.676+0000 7f975f7fe700 1 -- 172.21.15.40:0/986527883 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 46287+0+0 (secure 0 0 0) 0x7f9750022020 con 0x7f9768107c90 2024-04-21T23:08:18.559 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:17.676+0000 7f975f7fe700 1 --2- 172.21.15.40:0/986527883 >> [v2:172.21.15.40:6800/1479040326,v1:172.21.15.40:6801/1479040326] conn(0x7f9754039050 0x7f975403b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:18.561 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:17.676+0000 7f975f7fe700 1 -- 172.21.15.40:0/986527883 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f9750011e00 con 0x7f9768107c90 2024-04-21T23:08:18.562 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:17.677+0000 7f9765d9b700 1 -- 172.21.15.40:0/986527883 >> [v2:172.21.15.40:6800/1479040326,v1:172.21.15.40:6801/1479040326] conn(0x7f9754039050 msgr2=0x7f975403b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:6800/1479040326 2024-04-21T23:08:18.562 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:17.677+0000 7f9765d9b700 1 --2- 172.21.15.40:0/986527883 >> [v2:172.21.15.40:6800/1479040326,v1:172.21.15.40:6801/1479040326] conn(0x7f9754039050 0x7f975403b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T23:08:18.562 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:17.681+0000 7f975f7fe700 1 -- 172.21.15.40:0/986527883 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f975000f2e0 con 0x7f9768107c90 2024-04-21T23:08:18.562 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:17.878+0000 7f9765d9b700 1 -- 172.21.15.40:0/986527883 >> [v2:172.21.15.40:6800/1479040326,v1:172.21.15.40:6801/1479040326] conn(0x7f9754039050 msgr2=0x7f975403b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:6800/1479040326 2024-04-21T23:08:18.562 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:17.878+0000 7f9765d9b700 1 --2- 172.21.15.40:0/986527883 >> [v2:172.21.15.40:6800/1479040326,v1:172.21.15.40:6801/1479040326] conn(0x7f9754039050 0x7f975403b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-21T23:08:18.562 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:17.989+0000 7f976cd0b700 1 -- 172.21.15.40:0/986527883 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f976819f290 con 0x7f9768107c90 2024-04-21T23:08:18.562 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:17.991+0000 7f975f7fe700 1 -- 172.21.15.40:0/986527883 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7f976819f290 con 0x7f9768107c90 2024-04-21T23:08:18.562 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:17.993+0000 7f976cd0b700 1 -- 172.21.15.40:0/986527883 >> [v2:172.21.15.40:6800/1479040326,v1:172.21.15.40:6801/1479040326] conn(0x7f9754039050 msgr2=0x7f975403b510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T23:08:18.562 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:17.993+0000 7f976cd0b700 1 --2- 172.21.15.40:0/986527883 >> [v2:172.21.15.40:6800/1479040326,v1:172.21.15.40:6801/1479040326] conn(0x7f9754039050 0x7f975403b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:18.562 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:17.993+0000 7f976cd0b700 1 -- 172.21.15.40:0/986527883 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9768107c90 msgr2=0x7f976819bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:18.562 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:17.993+0000 7f976cd0b700 1 --2- 172.21.15.40:0/986527883 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9768107c90 0x7f976819bc00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9750000c00 tx=0x7f9750011c10 comp rx=0 tx=0).stop 2024-04-21T23:08:18.562 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:17.993+0000 7f976cd0b700 1 -- 172.21.15.40:0/986527883 shutdown_connections 2024-04-21T23:08:18.563 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:17.993+0000 7f976cd0b700 1 --2- 172.21.15.40:0/986527883 >> [v2:172.21.15.40:6800/1479040326,v1:172.21.15.40:6801/1479040326] conn(0x7f9754039050 0x7f975403b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:18.563 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:17.993+0000 7f976cd0b700 1 --2- 172.21.15.40:0/986527883 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9768107c90 0x7f976819bc00 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:18.563 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:17.993+0000 7f976cd0b700 1 -- 172.21.15.40:0/986527883 >> 172.21.15.40:0/986527883 conn(0x7f9768103210 msgr2=0x7f9768103df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:18.563 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:17.993+0000 7f976cd0b700 1 -- 172.21.15.40:0/986527883 shutdown_connections 2024-04-21T23:08:18.563 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:17.993+0000 7f976cd0b700 1 -- 172.21.15.40:0/986527883 wait complete. 2024-04-21T23:08:18.563 INFO:teuthology.orchestra.run.smithi040.stdout:Waiting for the mgr to restart... 2024-04-21T23:08:18.563 INFO:teuthology.orchestra.run.smithi040.stdout:Waiting for mgr epoch 5... 2024-04-21T23:08:18.925 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:18 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/986527883' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-21T23:08:18.925 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:18 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:08:18.796+0000 7ff5b8903200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-21T23:08:19.182 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:18 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:08:18.977+0000 7ff5b8903200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-21T23:08:19.182 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:19 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:08:19.155+0000 7ff5b8903200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-21T23:08:19.551 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:19 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:08:19.456+0000 7ff5b8903200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-21T23:08:19.551 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:19 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:08:19.552+0000 7ff5b8903200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-21T23:08:19.932 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:19 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:08:19.761+0000 7ff5b8903200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-21T23:08:20.432 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:19 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:08:19.961+0000 7ff5b8903200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-21T23:08:20.932 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:20 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:08:20.590+0000 7ff5b8903200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-21T23:08:21.333 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:21 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:08:21.027+0000 7ff5b8903200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-21T23:08:21.333 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:21 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:08:21.126+0000 7ff5b8903200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-21T23:08:21.334 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:21 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:08:21.334+0000 7ff5b8903200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-21T23:08:22.182 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:21 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:08:21.877+0000 7ff5b8903200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-21T23:08:22.182 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:21 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:08:21.973+0000 7ff5b8903200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-21T23:08:22.580 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:22 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:08:22.224+0000 7ff5b8903200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-21T23:08:22.580 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:22 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:08:22.363+0000 7ff5b8903200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-21T23:08:22.580 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:22 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:08:22.461+0000 7ff5b8903200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-21T23:08:22.580 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:22 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:08:22.580+0000 7ff5b8903200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-21T23:08:22.932 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:22 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:08:22.675+0000 7ff5b8903200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-21T23:08:23.793 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:23 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:08:23.517+0000 7ff5b8903200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-21T23:08:23.793 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:23 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:08:23.740+0000 7ff5b8903200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-21T23:08:24.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:23 smithi040 ceph-mon[27984]: Active manager daemon x restarted 2024-04-21T23:08:24.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:23 smithi040 ceph-mon[27984]: Activating manager daemon x 2024-04-21T23:08:24.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:23 smithi040 ceph-mon[27984]: osdmap e2: 0 total, 0 up, 0 in 2024-04-21T23:08:24.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:23 smithi040 ceph-mon[27984]: mgrmap e6: x(active, starting, since 0.00359831s) 2024-04-21T23:08:24.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:23 smithi040 ceph-mon[27984]: from='mgr.14116 172.21.15.40:0/550446883' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T23:08:24.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:23 smithi040 ceph-mon[27984]: from='mgr.14116 172.21.15.40:0/550446883' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-21T23:08:24.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:23 smithi040 ceph-mon[27984]: from='mgr.14116 172.21.15.40:0/550446883' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-21T23:08:24.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:23 smithi040 ceph-mon[27984]: from='mgr.14116 172.21.15.40:0/550446883' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-21T23:08:24.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:23 smithi040 ceph-mon[27984]: from='mgr.14116 172.21.15.40:0/550446883' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-21T23:08:24.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:23 smithi040 ceph-mon[27984]: Manager daemon x is now available 2024-04-21T23:08:24.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:23 smithi040 ceph-mon[27984]: from='mgr.14116 172.21.15.40:0/550446883' entity='mgr.x' 2024-04-21T23:08:24.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:23 smithi040 ceph-mon[27984]: from='mgr.14116 172.21.15.40:0/550446883' entity='mgr.x' 2024-04-21T23:08:24.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:23 smithi040 ceph-mon[27984]: from='mgr.14116 172.21.15.40:0/550446883' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:08:24.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:23 smithi040 ceph-mon[27984]: from='mgr.14116 172.21.15.40:0/550446883' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:08:24.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:23 smithi040 ceph-mon[27984]: from='mgr.14116 172.21.15.40:0/550446883' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-21T23:08:25.061 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:24 smithi040 ceph-mon[27984]: Found migration_current of "None". Setting to last migration. 2024-04-21T23:08:25.061 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:24 smithi040 ceph-mon[27984]: from='mgr.14116 172.21.15.40:0/550446883' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-21T23:08:25.061 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:24 smithi040 ceph-mon[27984]: mgrmap e7: x(active, since 1.00573s) 2024-04-21T23:08:25.372 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout { 2024-04-21T23:08:25.372 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-21T23:08:25.372 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-21T23:08:25.372 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout } 2024-04-21T23:08:25.373 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:19.030+0000 7effd6eee700 1 Processor -- start 2024-04-21T23:08:25.373 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:19.030+0000 7effd6eee700 1 -- start start 2024-04-21T23:08:25.373 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:19.031+0000 7effd6eee700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7effd0107c70 0x7effd0108090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:25.373 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:19.031+0000 7effd6eee700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7effd0108660 con 0x7effd0107c70 2024-04-21T23:08:25.373 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:19.031+0000 7effd4c8a700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7effd0107c70 0x7effd0108090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:25.373 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:19.031+0000 7effd4c8a700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7effd0107c70 0x7effd0108090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36796/0 (socket says 172.21.15.40:36796) 2024-04-21T23:08:25.373 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:19.031+0000 7effd4c8a700 1 -- 172.21.15.40:0/3868427631 learned_addr learned my addr 172.21.15.40:0/3868427631 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:08:25.373 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:19.031+0000 7effd4c8a700 1 -- 172.21.15.40:0/3868427631 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effd0108e70 con 0x7effd0107c70 2024-04-21T23:08:25.373 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:19.031+0000 7effd4c8a700 1 --2- 172.21.15.40:0/3868427631 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7effd0107c70 0x7effd0108090 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7effc40098a0 tx=0x7effc4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffd6197302ed4e73 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:25.373 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:19.031+0000 7effcf7fe700 1 -- 172.21.15.40:0/3868427631 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7effc4004160 con 0x7effd0107c70 2024-04-21T23:08:25.373 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:19.031+0000 7effcf7fe700 1 -- 172.21.15.40:0/3868427631 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7effc4004300 con 0x7effd0107c70 2024-04-21T23:08:25.373 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:19.031+0000 7effcf7fe700 1 -- 172.21.15.40:0/3868427631 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7effc4011b50 con 0x7effd0107c70 2024-04-21T23:08:25.373 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:19.032+0000 7effd6eee700 1 -- 172.21.15.40:0/3868427631 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7effd0107c70 msgr2=0x7effd0108090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:25.373 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:19.032+0000 7effd6eee700 1 --2- 172.21.15.40:0/3868427631 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7effd0107c70 0x7effd0108090 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7effc40098a0 tx=0x7effc4009bb0 comp rx=0 tx=0).stop 2024-04-21T23:08:25.373 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:19.032+0000 7effd6eee700 1 -- 172.21.15.40:0/3868427631 shutdown_connections 2024-04-21T23:08:25.374 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:19.032+0000 7effd6eee700 1 --2- 172.21.15.40:0/3868427631 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7effd0107c70 0x7effd0108090 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:25.374 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:19.032+0000 7effd6eee700 1 -- 172.21.15.40:0/3868427631 >> 172.21.15.40:0/3868427631 conn(0x7effd0103210 msgr2=0x7effd0105650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:25.374 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:19.032+0000 7effd6eee700 1 -- 172.21.15.40:0/3868427631 shutdown_connections 2024-04-21T23:08:25.374 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:19.032+0000 7effd6eee700 1 -- 172.21.15.40:0/3868427631 wait complete. 2024-04-21T23:08:25.374 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:19.033+0000 7effd6eee700 1 Processor -- start 2024-04-21T23:08:25.374 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:19.033+0000 7effd6eee700 1 -- start start 2024-04-21T23:08:25.374 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:19.033+0000 7effd6eee700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7effd0107c70 0x7effd019bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:25.374 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:19.033+0000 7effd6eee700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7effd010c220 con 0x7effd0107c70 2024-04-21T23:08:25.374 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:19.033+0000 7effd4c8a700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7effd0107c70 0x7effd019bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:25.374 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:19.033+0000 7effd4c8a700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7effd0107c70 0x7effd019bc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36800/0 (socket says 172.21.15.40:36800) 2024-04-21T23:08:25.374 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:19.033+0000 7effd4c8a700 1 -- 172.21.15.40:0/3194387984 learned_addr learned my addr 172.21.15.40:0/3194387984 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:08:25.374 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:19.034+0000 7effd4c8a700 1 -- 172.21.15.40:0/3194387984 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effd019c140 con 0x7effd0107c70 2024-04-21T23:08:25.374 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:19.034+0000 7effd4c8a700 1 --2- 172.21.15.40:0/3194387984 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7effd0107c70 0x7effd019bc00 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7effc4011670 tx=0x7effc4011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:25.374 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:19.034+0000 7effcdffb700 1 -- 172.21.15.40:0/3194387984 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7effc40039c0 con 0x7effd0107c70 2024-04-21T23:08:25.375 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:19.034+0000 7effcdffb700 1 -- 172.21.15.40:0/3194387984 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7effc4011e50 con 0x7effd0107c70 2024-04-21T23:08:25.375 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:19.034+0000 7effcdffb700 1 -- 172.21.15.40:0/3194387984 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7effc401a5c0 con 0x7effd0107c70 2024-04-21T23:08:25.375 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:19.034+0000 7effd6eee700 1 -- 172.21.15.40:0/3194387984 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effd019c3d0 con 0x7effd0107c70 2024-04-21T23:08:25.375 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:19.034+0000 7effd6eee700 1 -- 172.21.15.40:0/3194387984 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effd019f0c0 con 0x7effd0107c70 2024-04-21T23:08:25.375 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:19.035+0000 7effcdffb700 1 -- 172.21.15.40:0/3194387984 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 46287+0+0 (secure 0 0 0) 0x7effc401a8c0 con 0x7effd0107c70 2024-04-21T23:08:25.375 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:19.035+0000 7effcdffb700 1 --2- 172.21.15.40:0/3194387984 >> [v2:172.21.15.40:6800/1479040326,v1:172.21.15.40:6801/1479040326] conn(0x7effc0039000 0x7effc003b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:25.375 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:19.035+0000 7effcdffb700 1 -- 172.21.15.40:0/3194387984 --> [v2:172.21.15.40:6800/1479040326,v1:172.21.15.40:6801/1479040326] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7effc003bbd0 con 0x7effc0039000 2024-04-21T23:08:25.375 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:19.036+0000 7effcdffb700 1 -- 172.21.15.40:0/3194387984 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7effc404d460 con 0x7effd0107c70 2024-04-21T23:08:25.375 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:19.036+0000 7effcffff700 1 -- 172.21.15.40:0/3194387984 >> [v2:172.21.15.40:6800/1479040326,v1:172.21.15.40:6801/1479040326] conn(0x7effc0039000 msgr2=0x7effc003b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:6800/1479040326 2024-04-21T23:08:25.375 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:19.036+0000 7effcffff700 1 --2- 172.21.15.40:0/3194387984 >> [v2:172.21.15.40:6800/1479040326,v1:172.21.15.40:6801/1479040326] conn(0x7effc0039000 0x7effc003b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T23:08:25.375 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:19.237+0000 7effcffff700 1 -- 172.21.15.40:0/3194387984 >> [v2:172.21.15.40:6800/1479040326,v1:172.21.15.40:6801/1479040326] conn(0x7effc0039000 msgr2=0x7effc003b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:6800/1479040326 2024-04-21T23:08:25.375 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:19.237+0000 7effcffff700 1 --2- 172.21.15.40:0/3194387984 >> [v2:172.21.15.40:6800/1479040326,v1:172.21.15.40:6801/1479040326] conn(0x7effc0039000 0x7effc003b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-21T23:08:25.375 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:19.638+0000 7effcffff700 1 -- 172.21.15.40:0/3194387984 >> [v2:172.21.15.40:6800/1479040326,v1:172.21.15.40:6801/1479040326] conn(0x7effc0039000 msgr2=0x7effc003b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:6800/1479040326 2024-04-21T23:08:25.375 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:19.638+0000 7effcffff700 1 --2- 172.21.15.40:0/3194387984 >> [v2:172.21.15.40:6800/1479040326,v1:172.21.15.40:6801/1479040326] conn(0x7effc0039000 0x7effc003b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-21T23:08:25.375 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:20.439+0000 7effcffff700 1 -- 172.21.15.40:0/3194387984 >> [v2:172.21.15.40:6800/1479040326,v1:172.21.15.40:6801/1479040326] conn(0x7effc0039000 msgr2=0x7effc003b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:6800/1479040326 2024-04-21T23:08:25.376 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:20.439+0000 7effcffff700 1 --2- 172.21.15.40:0/3194387984 >> [v2:172.21.15.40:6800/1479040326,v1:172.21.15.40:6801/1479040326] conn(0x7effc0039000 0x7effc003b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-21T23:08:25.376 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:22.040+0000 7effcffff700 1 -- 172.21.15.40:0/3194387984 >> [v2:172.21.15.40:6800/1479040326,v1:172.21.15.40:6801/1479040326] conn(0x7effc0039000 msgr2=0x7effc003b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:6800/1479040326 2024-04-21T23:08:25.376 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:22.040+0000 7effcffff700 1 --2- 172.21.15.40:0/3194387984 >> [v2:172.21.15.40:6800/1479040326,v1:172.21.15.40:6801/1479040326] conn(0x7effc0039000 0x7effc003b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-21T23:08:25.376 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:23.744+0000 7effcdffb700 1 -- 172.21.15.40:0/3194387984 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mgrmap(e 6) ==== 46054+0+0 (secure 0 0 0) 0x7effc4024430 con 0x7effd0107c70 2024-04-21T23:08:25.376 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:23.744+0000 7effcdffb700 1 -- 172.21.15.40:0/3194387984 >> [v2:172.21.15.40:6800/1479040326,v1:172.21.15.40:6801/1479040326] conn(0x7effc0039000 msgr2=0x7effc003b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T23:08:25.376 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:23.744+0000 7effcdffb700 1 --2- 172.21.15.40:0/3194387984 >> [v2:172.21.15.40:6800/1479040326,v1:172.21.15.40:6801/1479040326] conn(0x7effc0039000 0x7effc003b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:25.376 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:24.746+0000 7effcdffb700 1 -- 172.21.15.40:0/3194387984 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mgrmap(e 7) ==== 46181+0+0 (secure 0 0 0) 0x7effc400bce0 con 0x7effd0107c70 2024-04-21T23:08:25.376 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:24.746+0000 7effcdffb700 1 --2- 172.21.15.40:0/3194387984 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7effc003c930 0x7effc003ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:25.376 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:24.746+0000 7effcdffb700 1 -- 172.21.15.40:0/3194387984 --> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7effc40169b0 con 0x7effc003c930 2024-04-21T23:08:25.376 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:24.746+0000 7effcffff700 1 --2- 172.21.15.40:0/3194387984 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7effc003c930 0x7effc003ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:25.376 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:24.747+0000 7effcffff700 1 --2- 172.21.15.40:0/3194387984 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7effc003c930 0x7effc003ed20 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7effbc003a10 tx=0x7effbc009290 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:25.376 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:24.748+0000 7effcdffb700 1 -- 172.21.15.40:0/3194387984 <== mgr.14116 v2:172.21.15.40:6800/526554319 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7effc40169b0 con 0x7effc003c930 2024-04-21T23:08:25.376 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:24.754+0000 7effd6eee700 1 -- 172.21.15.40:0/3194387984 --> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7effd0063890 con 0x7effc003c930 2024-04-21T23:08:25.377 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:24.755+0000 7effcdffb700 1 -- 172.21.15.40:0/3194387984 <== mgr.14116 v2:172.21.15.40:6800/526554319 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7effd0063890 con 0x7effc003c930 2024-04-21T23:08:25.377 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:24.755+0000 7effd6eee700 1 -- 172.21.15.40:0/3194387984 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7effc003c930 msgr2=0x7effc003ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:25.377 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:24.755+0000 7effd6eee700 1 --2- 172.21.15.40:0/3194387984 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7effc003c930 0x7effc003ed20 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7effbc003a10 tx=0x7effbc009290 comp rx=0 tx=0).stop 2024-04-21T23:08:25.377 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:24.755+0000 7effd6eee700 1 -- 172.21.15.40:0/3194387984 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7effd0107c70 msgr2=0x7effd019bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:25.377 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:24.755+0000 7effd6eee700 1 --2- 172.21.15.40:0/3194387984 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7effd0107c70 0x7effd019bc00 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7effc4011670 tx=0x7effc4011b60 comp rx=0 tx=0).stop 2024-04-21T23:08:25.377 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:24.755+0000 7effd6eee700 1 -- 172.21.15.40:0/3194387984 shutdown_connections 2024-04-21T23:08:25.377 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:24.755+0000 7effd6eee700 1 --2- 172.21.15.40:0/3194387984 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7effc003c930 0x7effc003ed20 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:25.377 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:24.755+0000 7effd6eee700 1 --2- 172.21.15.40:0/3194387984 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7effd0107c70 0x7effd019bc00 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:25.377 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:24.755+0000 7effd6eee700 1 --2- 172.21.15.40:0/3194387984 >> [v2:172.21.15.40:6800/1479040326,v1:172.21.15.40:6801/1479040326] conn(0x7effc0039000 0x7effc003b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:25.377 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:24.755+0000 7effd6eee700 1 -- 172.21.15.40:0/3194387984 >> 172.21.15.40:0/3194387984 conn(0x7effd0103210 msgr2=0x7effd0103dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:25.377 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:24.755+0000 7effd6eee700 1 -- 172.21.15.40:0/3194387984 shutdown_connections 2024-04-21T23:08:25.377 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:24.755+0000 7effd6eee700 1 -- 172.21.15.40:0/3194387984 wait complete. 2024-04-21T23:08:25.377 INFO:teuthology.orchestra.run.smithi040.stdout:mgr epoch 5 is available 2024-04-21T23:08:25.378 INFO:teuthology.orchestra.run.smithi040.stdout:Setting orchestrator backend to cephadm... 2024-04-21T23:08:26.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:25 smithi040 ceph-mon[27984]: from='client.14120 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-21T23:08:26.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:25 smithi040 ceph-mon[27984]: from='client.14120 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-21T23:08:26.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:25 smithi040 ceph-mon[27984]: from='mgr.14116 172.21.15.40:0/550446883' entity='mgr.x' 2024-04-21T23:08:26.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:25 smithi040 ceph-mon[27984]: from='mgr.14116 172.21.15.40:0/550446883' entity='mgr.x' 2024-04-21T23:08:26.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:25 smithi040 ceph-mon[27984]: [21/Apr/2024:23:08:25] ENGINE Bus STARTING 2024-04-21T23:08:26.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:25 smithi040 ceph-mon[27984]: [21/Apr/2024:23:08:25] ENGINE Serving on http://172.21.15.40:8765 2024-04-21T23:08:26.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:25 smithi040 ceph-mon[27984]: [21/Apr/2024:23:08:25] ENGINE Serving on https://172.21.15.40:7150 2024-04-21T23:08:26.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:25 smithi040 ceph-mon[27984]: [21/Apr/2024:23:08:25] ENGINE Bus STARTED 2024-04-21T23:08:26.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:25 smithi040 ceph-mon[27984]: from='mgr.14116 172.21.15.40:0/550446883' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:08:26.568 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:25.836+0000 7fc49ff06700 1 Processor -- start 2024-04-21T23:08:26.568 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:25.836+0000 7fc49ff06700 1 -- start start 2024-04-21T23:08:26.568 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:25.836+0000 7fc49ff06700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc498106500 0x7fc498106920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:26.568 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:25.836+0000 7fc49ff06700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc49807be30 con 0x7fc498106500 2024-04-21T23:08:26.568 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:25.837+0000 7fc49dca2700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc498106500 0x7fc498106920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:26.568 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:25.837+0000 7fc49dca2700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc498106500 0x7fc498106920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40658/0 (socket says 172.21.15.40:40658) 2024-04-21T23:08:26.568 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:25.837+0000 7fc49dca2700 1 -- 172.21.15.40:0/2465987684 learned_addr learned my addr 172.21.15.40:0/2465987684 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:08:26.569 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:25.837+0000 7fc49dca2700 1 -- 172.21.15.40:0/2465987684 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc498106e60 con 0x7fc498106500 2024-04-21T23:08:26.569 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:25.837+0000 7fc49dca2700 1 --2- 172.21.15.40:0/2465987684 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc498106500 0x7fc498106920 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc488009e80 tx=0x7fc48800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=839e787fa37a65e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:26.569 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:25.837+0000 7fc49cca0700 1 -- 172.21.15.40:0/2465987684 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc488017070 con 0x7fc498106500 2024-04-21T23:08:26.569 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:25.837+0000 7fc49cca0700 1 -- 172.21.15.40:0/2465987684 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fc48800b8c0 con 0x7fc498106500 2024-04-21T23:08:26.569 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:25.837+0000 7fc49cca0700 1 -- 172.21.15.40:0/2465987684 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc4880045c0 con 0x7fc498106500 2024-04-21T23:08:26.569 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:25.838+0000 7fc49ff06700 1 -- 172.21.15.40:0/2465987684 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc498106500 msgr2=0x7fc498106920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:26.569 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:25.838+0000 7fc49ff06700 1 --2- 172.21.15.40:0/2465987684 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc498106500 0x7fc498106920 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc488009e80 tx=0x7fc48800b330 comp rx=0 tx=0).stop 2024-04-21T23:08:26.569 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:25.838+0000 7fc49ff06700 1 -- 172.21.15.40:0/2465987684 shutdown_connections 2024-04-21T23:08:26.569 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:25.838+0000 7fc49ff06700 1 --2- 172.21.15.40:0/2465987684 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc498106500 0x7fc498106920 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:26.569 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:25.838+0000 7fc49ff06700 1 -- 172.21.15.40:0/2465987684 >> 172.21.15.40:0/2465987684 conn(0x7fc4981020a0 msgr2=0x7fc498104500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:26.569 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:25.838+0000 7fc49ff06700 1 -- 172.21.15.40:0/2465987684 shutdown_connections 2024-04-21T23:08:26.569 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:25.838+0000 7fc49ff06700 1 -- 172.21.15.40:0/2465987684 wait complete. 2024-04-21T23:08:26.569 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:25.838+0000 7fc49ff06700 1 Processor -- start 2024-04-21T23:08:26.569 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:25.839+0000 7fc49ff06700 1 -- start start 2024-04-21T23:08:26.570 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:25.839+0000 7fc49ff06700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc498106500 0x7fc49819ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:26.570 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:25.839+0000 7fc49ff06700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc498107ef0 con 0x7fc498106500 2024-04-21T23:08:26.570 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:25.839+0000 7fc49dca2700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc498106500 0x7fc49819ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:26.570 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:25.839+0000 7fc49dca2700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc498106500 0x7fc49819ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40672/0 (socket says 172.21.15.40:40672) 2024-04-21T23:08:26.570 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:25.839+0000 7fc49dca2700 1 -- 172.21.15.40:0/4121728240 learned_addr learned my addr 172.21.15.40:0/4121728240 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:08:26.570 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:25.839+0000 7fc49dca2700 1 -- 172.21.15.40:0/4121728240 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc49819e2f0 con 0x7fc498106500 2024-04-21T23:08:26.570 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:25.839+0000 7fc49dca2700 1 --2- 172.21.15.40:0/4121728240 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc498106500 0x7fc49819ddb0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc488004350 tx=0x7fc48800bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:26.570 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:25.839+0000 7fc496ffd700 1 -- 172.21.15.40:0/4121728240 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc488020020 con 0x7fc498106500 2024-04-21T23:08:26.570 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:25.839+0000 7fc496ffd700 1 -- 172.21.15.40:0/4121728240 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fc488003c10 con 0x7fc498106500 2024-04-21T23:08:26.570 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:25.839+0000 7fc49ff06700 1 -- 172.21.15.40:0/4121728240 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc49819e580 con 0x7fc498106500 2024-04-21T23:08:26.570 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:25.839+0000 7fc496ffd700 1 -- 172.21.15.40:0/4121728240 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc488017040 con 0x7fc498106500 2024-04-21T23:08:26.570 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:25.839+0000 7fc49ff06700 1 -- 172.21.15.40:0/4121728240 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4981a1230 con 0x7fc498106500 2024-04-21T23:08:26.570 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:25.840+0000 7fc496ffd700 1 -- 172.21.15.40:0/4121728240 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 7) ==== 46181+0+0 (secure 0 0 0) 0x7fc488023020 con 0x7fc498106500 2024-04-21T23:08:26.570 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:25.840+0000 7fc49ff06700 1 -- 172.21.15.40:0/4121728240 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc498063890 con 0x7fc498106500 2024-04-21T23:08:26.571 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:25.840+0000 7fc496ffd700 1 --2- 172.21.15.40:0/4121728240 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7fc480038f80 0x7fc48003b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:26.571 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:25.841+0000 7fc496ffd700 1 -- 172.21.15.40:0/4121728240 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fc48800f050 con 0x7fc498106500 2024-04-21T23:08:26.571 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:25.841+0000 7fc49d4a1700 1 --2- 172.21.15.40:0/4121728240 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7fc480038f80 0x7fc48003b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:26.571 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:25.841+0000 7fc49d4a1700 1 --2- 172.21.15.40:0/4121728240 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7fc480038f80 0x7fc48003b440 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc48c006fd0 tx=0x7fc48c006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:26.571 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:25.846+0000 7fc496ffd700 1 -- 172.21.15.40:0/4121728240 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fc48802a020 con 0x7fc498106500 2024-04-21T23:08:26.571 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:25.935+0000 7fc496ffd700 1 -- 172.21.15.40:0/4121728240 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fc488025400 con 0x7fc498106500 2024-04-21T23:08:26.571 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:26.069+0000 7fc49ff06700 1 -- 172.21.15.40:0/4121728240 --> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7fc49802ded0 con 0x7fc480038f80 2024-04-21T23:08:26.571 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:26.076+0000 7fc496ffd700 1 -- 172.21.15.40:0/4121728240 <== mgr.14116 v2:172.21.15.40:6800/526554319 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fc49802ded0 con 0x7fc480038f80 2024-04-21T23:08:26.571 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:26.079+0000 7fc49ff06700 1 -- 172.21.15.40:0/4121728240 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7fc480038f80 msgr2=0x7fc48003b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:26.571 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:26.079+0000 7fc49ff06700 1 --2- 172.21.15.40:0/4121728240 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7fc480038f80 0x7fc48003b440 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc48c006fd0 tx=0x7fc48c006e40 comp rx=0 tx=0).stop 2024-04-21T23:08:26.571 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:26.079+0000 7fc49ff06700 1 -- 172.21.15.40:0/4121728240 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc498106500 msgr2=0x7fc49819ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:26.571 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:26.079+0000 7fc49ff06700 1 --2- 172.21.15.40:0/4121728240 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc498106500 0x7fc49819ddb0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc488004350 tx=0x7fc48800bd50 comp rx=0 tx=0).stop 2024-04-21T23:08:26.571 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:26.079+0000 7fc49ff06700 1 -- 172.21.15.40:0/4121728240 shutdown_connections 2024-04-21T23:08:26.571 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:26.079+0000 7fc49ff06700 1 --2- 172.21.15.40:0/4121728240 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7fc480038f80 0x7fc48003b440 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:26.572 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:26.079+0000 7fc49ff06700 1 --2- 172.21.15.40:0/4121728240 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc498106500 0x7fc49819ddb0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:26.572 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:26.079+0000 7fc49ff06700 1 -- 172.21.15.40:0/4121728240 >> 172.21.15.40:0/4121728240 conn(0x7fc4981020a0 msgr2=0x7fc4981949b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:26.572 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:26.079+0000 7fc49ff06700 1 -- 172.21.15.40:0/4121728240 shutdown_connections 2024-04-21T23:08:26.572 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:26.079+0000 7fc49ff06700 1 -- 172.21.15.40:0/4121728240 wait complete. 2024-04-21T23:08:27.012 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:26 smithi040 ceph-mon[27984]: mgrmap e8: x(active, since 2s) 2024-04-21T23:08:27.013 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:26 smithi040 ceph-mon[27984]: from='client.14128 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:08:27.013 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:26 smithi040 ceph-mon[27984]: from='mgr.14116 172.21.15.40:0/550446883' entity='mgr.x' 2024-04-21T23:08:27.013 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:26 smithi040 ceph-mon[27984]: from='mgr.14116 172.21.15.40:0/550446883' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:08:27.850 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-21T23:08:27.850 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:27.024+0000 7f21c20a3700 1 Processor -- start 2024-04-21T23:08:27.850 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:27.024+0000 7f21c20a3700 1 -- start start 2024-04-21T23:08:27.850 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:27.024+0000 7f21c20a3700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21bc107c40 0x7f21bc108060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:27.850 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:27.024+0000 7f21c20a3700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f21bc108630 con 0x7f21bc107c40 2024-04-21T23:08:27.850 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:27.024+0000 7f21bb7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21bc107c40 0x7f21bc108060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:27.851 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:27.024+0000 7f21bb7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21bc107c40 0x7f21bc108060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40680/0 (socket says 172.21.15.40:40680) 2024-04-21T23:08:27.851 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:27.024+0000 7f21bb7fe700 1 -- 172.21.15.40:0/4248837456 learned_addr learned my addr 172.21.15.40:0/4248837456 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:08:27.851 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:27.025+0000 7f21bb7fe700 1 -- 172.21.15.40:0/4248837456 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21bc108e40 con 0x7f21bc107c40 2024-04-21T23:08:27.851 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:27.025+0000 7f21bb7fe700 1 --2- 172.21.15.40:0/4248837456 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21bc107c40 0x7f21bc108060 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f21ac0098a0 tx=0x7f21ac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=286486159dac6f72 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:27.851 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:27.025+0000 7f21ba7fc700 1 -- 172.21.15.40:0/4248837456 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f21ac004160 con 0x7f21bc107c40 2024-04-21T23:08:27.851 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:27.025+0000 7f21ba7fc700 1 -- 172.21.15.40:0/4248837456 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f21ac004300 con 0x7f21bc107c40 2024-04-21T23:08:27.851 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:27.025+0000 7f21ba7fc700 1 -- 172.21.15.40:0/4248837456 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f21ac011b50 con 0x7f21bc107c40 2024-04-21T23:08:27.851 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:27.026+0000 7f21c20a3700 1 -- 172.21.15.40:0/4248837456 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21bc107c40 msgr2=0x7f21bc108060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:27.851 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:27.026+0000 7f21c20a3700 1 --2- 172.21.15.40:0/4248837456 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21bc107c40 0x7f21bc108060 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f21ac0098a0 tx=0x7f21ac009bb0 comp rx=0 tx=0).stop 2024-04-21T23:08:27.851 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:27.026+0000 7f21c20a3700 1 -- 172.21.15.40:0/4248837456 shutdown_connections 2024-04-21T23:08:27.851 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:27.026+0000 7f21c20a3700 1 --2- 172.21.15.40:0/4248837456 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21bc107c40 0x7f21bc108060 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:27.851 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:27.026+0000 7f21c20a3700 1 -- 172.21.15.40:0/4248837456 >> 172.21.15.40:0/4248837456 conn(0x7f21bc1031e0 msgr2=0x7f21bc105620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:27.851 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:27.026+0000 7f21c20a3700 1 -- 172.21.15.40:0/4248837456 shutdown_connections 2024-04-21T23:08:27.851 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:27.026+0000 7f21c20a3700 1 -- 172.21.15.40:0/4248837456 wait complete. 2024-04-21T23:08:27.852 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:27.026+0000 7f21c20a3700 1 Processor -- start 2024-04-21T23:08:27.852 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:27.027+0000 7f21c20a3700 1 -- start start 2024-04-21T23:08:27.852 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:27.027+0000 7f21c20a3700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21bc107c40 0x7f21bc19bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:27.852 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:27.027+0000 7f21c20a3700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f21bc10c1f0 con 0x7f21bc107c40 2024-04-21T23:08:27.852 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:27.027+0000 7f21bb7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21bc107c40 0x7f21bc19bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:27.852 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:27.027+0000 7f21bb7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21bc107c40 0x7f21bc19bc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40696/0 (socket says 172.21.15.40:40696) 2024-04-21T23:08:27.852 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:27.027+0000 7f21bb7fe700 1 -- 172.21.15.40:0/684610758 learned_addr learned my addr 172.21.15.40:0/684610758 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:08:27.852 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:27.027+0000 7f21bb7fe700 1 -- 172.21.15.40:0/684610758 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21bc19c1c0 con 0x7f21bc107c40 2024-04-21T23:08:27.852 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:27.027+0000 7f21bb7fe700 1 --2- 172.21.15.40:0/684610758 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21bc107c40 0x7f21bc19bc80 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f21ac009fb0 tx=0x7f21ac011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:27.852 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:27.027+0000 7f21b8ff9700 1 -- 172.21.15.40:0/684610758 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f21ac0038b0 con 0x7f21bc107c40 2024-04-21T23:08:27.852 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:27.027+0000 7f21b8ff9700 1 -- 172.21.15.40:0/684610758 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f21ac011e50 con 0x7f21bc107c40 2024-04-21T23:08:27.852 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:27.027+0000 7f21c20a3700 1 -- 172.21.15.40:0/684610758 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21bc19c450 con 0x7f21bc107c40 2024-04-21T23:08:27.852 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:27.027+0000 7f21b8ff9700 1 -- 172.21.15.40:0/684610758 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f21ac019550 con 0x7f21bc107c40 2024-04-21T23:08:27.852 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:27.027+0000 7f21c20a3700 1 -- 172.21.15.40:0/684610758 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21bc19f140 con 0x7f21bc107c40 2024-04-21T23:08:27.853 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:27.028+0000 7f21b8ff9700 1 -- 172.21.15.40:0/684610758 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f21ac019800 con 0x7f21bc107c40 2024-04-21T23:08:27.853 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:27.029+0000 7f21c20a3700 1 -- 172.21.15.40:0/684610758 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21bc063890 con 0x7f21bc107c40 2024-04-21T23:08:27.853 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:27.029+0000 7f21b8ff9700 1 --2- 172.21.15.40:0/684610758 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7f21a4039050 0x7f21a403b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:27.853 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:27.029+0000 7f21b8ff9700 1 -- 172.21.15.40:0/684610758 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f21ac04d3f0 con 0x7f21bc107c40 2024-04-21T23:08:27.853 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:27.029+0000 7f21baffd700 1 --2- 172.21.15.40:0/684610758 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7f21a4039050 0x7f21a403b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:27.853 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:27.029+0000 7f21baffd700 1 --2- 172.21.15.40:0/684610758 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7f21a4039050 0x7f21a403b510 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f21b0006fd0 tx=0x7f21b0006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:27.853 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:27.034+0000 7f21b8ff9700 1 -- 172.21.15.40:0/684610758 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f21ac00bc40 con 0x7f21bc107c40 2024-04-21T23:08:27.853 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:27.242+0000 7f21c20a3700 1 -- 172.21.15.40:0/684610758 --> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7f21bc104450 con 0x7f21a4039050 2024-04-21T23:08:27.853 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:27.243+0000 7f21b8ff9700 1 -- 172.21.15.40:0/684610758 <== mgr.14116 v2:172.21.15.40:6800/526554319 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7f21bc104450 con 0x7f21a4039050 2024-04-21T23:08:27.853 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:27.245+0000 7f21c20a3700 1 -- 172.21.15.40:0/684610758 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7f21a4039050 msgr2=0x7f21a403b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:27.853 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:27.245+0000 7f21c20a3700 1 --2- 172.21.15.40:0/684610758 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7f21a4039050 0x7f21a403b510 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f21b0006fd0 tx=0x7f21b0006e40 comp rx=0 tx=0).stop 2024-04-21T23:08:27.853 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:27.245+0000 7f21c20a3700 1 -- 172.21.15.40:0/684610758 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21bc107c40 msgr2=0x7f21bc19bc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:27.853 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:27.245+0000 7f21c20a3700 1 --2- 172.21.15.40:0/684610758 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21bc107c40 0x7f21bc19bc80 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f21ac009fb0 tx=0x7f21ac011bf0 comp rx=0 tx=0).stop 2024-04-21T23:08:27.853 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:27.245+0000 7f21c20a3700 1 -- 172.21.15.40:0/684610758 shutdown_connections 2024-04-21T23:08:27.854 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:27.245+0000 7f21c20a3700 1 --2- 172.21.15.40:0/684610758 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7f21a4039050 0x7f21a403b510 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:27.854 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:27.245+0000 7f21c20a3700 1 --2- 172.21.15.40:0/684610758 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21bc107c40 0x7f21bc19bc80 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:27.854 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:27.245+0000 7f21c20a3700 1 -- 172.21.15.40:0/684610758 >> 172.21.15.40:0/684610758 conn(0x7f21bc1031e0 msgr2=0x7f21bc103d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:27.854 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:27.245+0000 7f21c20a3700 1 -- 172.21.15.40:0/684610758 shutdown_connections 2024-04-21T23:08:27.854 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:27.245+0000 7f21c20a3700 1 -- 172.21.15.40:0/684610758 wait complete. 2024-04-21T23:08:27.854 INFO:teuthology.orchestra.run.smithi040.stdout:Generating ssh key... 2024-04-21T23:08:28.160 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:27 smithi040 ceph-mon[27984]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:08:29.280 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:28 smithi040 ceph-mon[27984]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:08:29.280 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:28 smithi040 ceph-mon[27984]: Generating ssh key... 2024-04-21T23:08:29.280 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:28 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: Generating public/private rsa key pair. 2024-04-21T23:08:29.280 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:28 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: Your identification has been saved in /tmp/tmp_3dp9d6_/key. 2024-04-21T23:08:29.280 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:28 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: Your public key has been saved in /tmp/tmp_3dp9d6_/key.pub. 2024-04-21T23:08:29.281 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:28 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: The key fingerprint is: 2024-04-21T23:08:29.281 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:28 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: SHA256:29iLY0PV3EKdP1nuraKZdkUjAdnk53TNWcEyvDfTr54 ceph-d814e4dc-0033-11ef-bc93-c7b262605968 2024-04-21T23:08:29.281 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:28 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: The key's randomart image is: 2024-04-21T23:08:29.281 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:28 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: +---[RSA 3072]----+ 2024-04-21T23:08:29.281 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:28 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: | .=o..oo| 2024-04-21T23:08:29.281 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:28 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: | ..++oo=| 2024-04-21T23:08:29.281 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:28 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: | +.o*+B| 2024-04-21T23:08:29.281 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:28 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: | . ==+B+| 2024-04-21T23:08:29.281 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:28 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: | S. +oo*| 2024-04-21T23:08:29.281 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:28 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: | .= . +| 2024-04-21T23:08:29.281 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:28 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: | .o o . o | 2024-04-21T23:08:29.281 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:28 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: | +..+o o. | 2024-04-21T23:08:29.281 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:28 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: | ..+=o oE | 2024-04-21T23:08:29.281 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:28 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: +----[SHA256]-----+ 2024-04-21T23:08:29.514 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:28.271+0000 7f370e044700 1 Processor -- start 2024-04-21T23:08:29.514 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:28.271+0000 7f370e044700 1 -- start start 2024-04-21T23:08:29.514 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:28.272+0000 7f370e044700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37081064c0 0x7f37081068e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:29.514 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:28.272+0000 7f370e044700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f370807be70 con 0x7f37081064c0 2024-04-21T23:08:29.515 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:28.272+0000 7f37077fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37081064c0 0x7f37081068e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:29.515 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:28.272+0000 7f37077fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37081064c0 0x7f37081068e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40712/0 (socket says 172.21.15.40:40712) 2024-04-21T23:08:29.515 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:28.272+0000 7f37077fe700 1 -- 172.21.15.40:0/252375318 learned_addr learned my addr 172.21.15.40:0/252375318 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:08:29.515 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:28.272+0000 7f37077fe700 1 -- 172.21.15.40:0/252375318 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3708106e20 con 0x7f37081064c0 2024-04-21T23:08:29.515 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:28.272+0000 7f37077fe700 1 --2- 172.21.15.40:0/252375318 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37081064c0 0x7f37081068e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f36f80098a0 tx=0x7f36f8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f2c0a370bfa184 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:29.515 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:28.273+0000 7f37067fc700 1 -- 172.21.15.40:0/252375318 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f36f8004160 con 0x7f37081064c0 2024-04-21T23:08:29.515 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:28.273+0000 7f37067fc700 1 -- 172.21.15.40:0/252375318 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f36f8004300 con 0x7f37081064c0 2024-04-21T23:08:29.515 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:28.273+0000 7f37067fc700 1 -- 172.21.15.40:0/252375318 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f36f8011b50 con 0x7f37081064c0 2024-04-21T23:08:29.515 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:28.273+0000 7f370e044700 1 -- 172.21.15.40:0/252375318 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37081064c0 msgr2=0x7f37081068e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:29.515 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:28.273+0000 7f370e044700 1 --2- 172.21.15.40:0/252375318 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37081064c0 0x7f37081068e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f36f80098a0 tx=0x7f36f8009bb0 comp rx=0 tx=0).stop 2024-04-21T23:08:29.515 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:28.273+0000 7f370e044700 1 -- 172.21.15.40:0/252375318 shutdown_connections 2024-04-21T23:08:29.515 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:28.273+0000 7f370e044700 1 --2- 172.21.15.40:0/252375318 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37081064c0 0x7f37081068e0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:29.515 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:28.273+0000 7f370e044700 1 -- 172.21.15.40:0/252375318 >> 172.21.15.40:0/252375318 conn(0x7f37081020a0 msgr2=0x7f3708104500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:29.515 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:28.273+0000 7f370e044700 1 -- 172.21.15.40:0/252375318 shutdown_connections 2024-04-21T23:08:29.516 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:28.273+0000 7f370e044700 1 -- 172.21.15.40:0/252375318 wait complete. 2024-04-21T23:08:29.516 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:28.274+0000 7f370e044700 1 Processor -- start 2024-04-21T23:08:29.516 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:28.274+0000 7f370e044700 1 -- start start 2024-04-21T23:08:29.516 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:28.274+0000 7f370e044700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37081064c0 0x7f370819ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:29.516 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:28.274+0000 7f370e044700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3708107eb0 con 0x7f37081064c0 2024-04-21T23:08:29.516 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:28.275+0000 7f37077fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37081064c0 0x7f370819ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:29.516 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:28.275+0000 7f37077fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37081064c0 0x7f370819ddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40728/0 (socket says 172.21.15.40:40728) 2024-04-21T23:08:29.516 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:28.275+0000 7f37077fe700 1 -- 172.21.15.40:0/4209611965 learned_addr learned my addr 172.21.15.40:0/4209611965 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:08:29.516 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:28.275+0000 7f37077fe700 1 -- 172.21.15.40:0/4209611965 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f370819e300 con 0x7f37081064c0 2024-04-21T23:08:29.516 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:28.275+0000 7f37077fe700 1 --2- 172.21.15.40:0/4209611965 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37081064c0 0x7f370819ddc0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f36f8011b60 tx=0x7f36f8011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:29.516 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:28.275+0000 7f3704ff9700 1 -- 172.21.15.40:0/4209611965 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f36f8003b20 con 0x7f37081064c0 2024-04-21T23:08:29.516 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:28.275+0000 7f3704ff9700 1 -- 172.21.15.40:0/4209611965 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f36f8003cc0 con 0x7f37081064c0 2024-04-21T23:08:29.516 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:28.275+0000 7f370e044700 1 -- 172.21.15.40:0/4209611965 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f370819e590 con 0x7f37081064c0 2024-04-21T23:08:29.516 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:28.275+0000 7f3704ff9700 1 -- 172.21.15.40:0/4209611965 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f36f8024840 con 0x7f37081064c0 2024-04-21T23:08:29.517 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:28.275+0000 7f370e044700 1 -- 172.21.15.40:0/4209611965 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37081a1280 con 0x7f37081064c0 2024-04-21T23:08:29.517 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:28.276+0000 7f3704ff9700 1 -- 172.21.15.40:0/4209611965 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f36f8022020 con 0x7f37081064c0 2024-04-21T23:08:29.517 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:28.276+0000 7f370e044700 1 -- 172.21.15.40:0/4209611965 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3708063890 con 0x7f37081064c0 2024-04-21T23:08:29.517 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:28.276+0000 7f3704ff9700 1 --2- 172.21.15.40:0/4209611965 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7f36f00390a0 0x7f36f003b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:29.517 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:28.276+0000 7f3704ff9700 1 -- 172.21.15.40:0/4209611965 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f36f8011e00 con 0x7f37081064c0 2024-04-21T23:08:29.517 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:28.277+0000 7f3706ffd700 1 --2- 172.21.15.40:0/4209611965 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7f36f00390a0 0x7f36f003b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:29.517 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:28.277+0000 7f3706ffd700 1 --2- 172.21.15.40:0/4209611965 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7f36f00390a0 0x7f36f003b560 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f36fc006fd0 tx=0x7f36fc006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:29.517 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:28.282+0000 7f3704ff9700 1 -- 172.21.15.40:0/4209611965 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f36f80172b0 con 0x7f37081064c0 2024-04-21T23:08:29.517 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:28.491+0000 7f370e044700 1 -- 172.21.15.40:0/4209611965 --> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7f37080008d0 con 0x7f36f00390a0 2024-04-21T23:08:29.517 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:28.978+0000 7f3704ff9700 1 -- 172.21.15.40:0/4209611965 <== mgr.14116 v2:172.21.15.40:6800/526554319 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f37080008d0 con 0x7f36f00390a0 2024-04-21T23:08:29.517 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:28.980+0000 7f370e044700 1 -- 172.21.15.40:0/4209611965 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7f36f00390a0 msgr2=0x7f36f003b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:29.517 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:28.980+0000 7f370e044700 1 --2- 172.21.15.40:0/4209611965 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7f36f00390a0 0x7f36f003b560 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f36fc006fd0 tx=0x7f36fc006e40 comp rx=0 tx=0).stop 2024-04-21T23:08:29.517 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:28.980+0000 7f370e044700 1 -- 172.21.15.40:0/4209611965 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37081064c0 msgr2=0x7f370819ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:29.517 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:28.980+0000 7f370e044700 1 --2- 172.21.15.40:0/4209611965 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37081064c0 0x7f370819ddc0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f36f8011b60 tx=0x7f36f8011c40 comp rx=0 tx=0).stop 2024-04-21T23:08:29.517 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:28.980+0000 7f370e044700 1 -- 172.21.15.40:0/4209611965 shutdown_connections 2024-04-21T23:08:29.518 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:28.980+0000 7f370e044700 1 --2- 172.21.15.40:0/4209611965 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7f36f00390a0 0x7f36f003b560 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:29.518 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:28.980+0000 7f370e044700 1 --2- 172.21.15.40:0/4209611965 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37081064c0 0x7f370819ddc0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:29.518 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:28.980+0000 7f370e044700 1 -- 172.21.15.40:0/4209611965 >> 172.21.15.40:0/4209611965 conn(0x7f37081020a0 msgr2=0x7f3708194a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:29.518 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:28.981+0000 7f370e044700 1 -- 172.21.15.40:0/4209611965 shutdown_connections 2024-04-21T23:08:29.518 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:28.981+0000 7f370e044700 1 -- 172.21.15.40:0/4209611965 wait complete. 2024-04-21T23:08:30.169 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:29 smithi040 ceph-mon[27984]: from='mgr.14116 172.21.15.40:0/550446883' entity='mgr.x' 2024-04-21T23:08:30.170 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:29 smithi040 ceph-mon[27984]: from='mgr.14116 172.21.15.40:0/550446883' entity='mgr.x' 2024-04-21T23:08:30.695 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/WodsMK9knduyuDKb9hLCNvwDfLMFRNdb5KKrSmT2QpvbRpl7fzJRrOchfTmuzEAGMqhQU5waMtgcOLYLfnIFUMxQG0Lgv/0djttTzoz6c95cjntw0u7PYNaW3wiPIWqv9TuIqcJLg9UM9JHClUcWkvnNSNkhE7S403S/IAE4OCapAdWuE+tjJuXQgHWqK9vNbiJzo1CRKbjpX4BxQxrdHq3+91fOSmQm3j6vfCzcch/aCefeyo4kMYxcgHpgbVmmQ2hbzcyGfYrF6zsUHdAS5GWYUyQ92cVHIMnUmGLYmjXMXMKVjOZAFqqkSbgZZmGuyu/Cvbf4uLVpTT72ZjBIT7vsA6e15pj3A+zMTLPa/wEdzXjAzVY+GCtjqt1ErgkeK+KQQ/fZSxOzChZG5+9X4b+gcSnc5xMpRa6e6v6N5ywpQK8XcDLFs8CKL4ES2NsQ/3X66UZ8sIYlAivcZyXJojaf4DsX3OuHmbxr+8ZpP24rPuIRzQZDmd3/B9azHk0= ceph-d814e4dc-0033-11ef-bc93-c7b262605968 2024-04-21T23:08:30.695 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:29.952+0000 7f8b8bad6700 1 Processor -- start 2024-04-21T23:08:30.695 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:29.952+0000 7f8b8bad6700 1 -- start start 2024-04-21T23:08:30.695 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:29.953+0000 7f8b8bad6700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b84106b20 0x7f8b84106f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:30.695 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:29.953+0000 7f8b8bad6700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8b84107510 con 0x7f8b84106b20 2024-04-21T23:08:30.696 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:29.953+0000 7f8b89872700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b84106b20 0x7f8b84106f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:30.696 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:29.953+0000 7f8b89872700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b84106b20 0x7f8b84106f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40742/0 (socket says 172.21.15.40:40742) 2024-04-21T23:08:30.696 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:29.953+0000 7f8b89872700 1 -- 172.21.15.40:0/439728847 learned_addr learned my addr 172.21.15.40:0/439728847 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:08:30.696 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:29.953+0000 7f8b89872700 1 -- 172.21.15.40:0/439728847 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b84107d20 con 0x7f8b84106b20 2024-04-21T23:08:30.696 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:29.953+0000 7f8b89872700 1 --2- 172.21.15.40:0/439728847 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b84106b20 0x7f8b84106f40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8b7c009e80 tx=0x7f8b7c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=907c79c1f64bd7d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:30.696 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:29.953+0000 7f8b88870700 1 -- 172.21.15.40:0/439728847 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8b7c017070 con 0x7f8b84106b20 2024-04-21T23:08:30.696 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:29.953+0000 7f8b88870700 1 -- 172.21.15.40:0/439728847 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f8b7c00b8c0 con 0x7f8b84106b20 2024-04-21T23:08:30.696 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:29.953+0000 7f8b88870700 1 -- 172.21.15.40:0/439728847 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8b7c0045c0 con 0x7f8b84106b20 2024-04-21T23:08:30.696 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:29.954+0000 7f8b8bad6700 1 -- 172.21.15.40:0/439728847 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b84106b20 msgr2=0x7f8b84106f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:30.696 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:29.954+0000 7f8b8bad6700 1 --2- 172.21.15.40:0/439728847 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b84106b20 0x7f8b84106f40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8b7c009e80 tx=0x7f8b7c00b330 comp rx=0 tx=0).stop 2024-04-21T23:08:30.696 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:29.954+0000 7f8b8bad6700 1 -- 172.21.15.40:0/439728847 shutdown_connections 2024-04-21T23:08:30.696 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:29.954+0000 7f8b8bad6700 1 --2- 172.21.15.40:0/439728847 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b84106b20 0x7f8b84106f40 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:30.696 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:29.954+0000 7f8b8bad6700 1 -- 172.21.15.40:0/439728847 >> 172.21.15.40:0/439728847 conn(0x7f8b841020a0 msgr2=0x7f8b84104500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:30.696 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:29.954+0000 7f8b8bad6700 1 -- 172.21.15.40:0/439728847 shutdown_connections 2024-04-21T23:08:30.697 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:29.954+0000 7f8b8bad6700 1 -- 172.21.15.40:0/439728847 wait complete. 2024-04-21T23:08:30.697 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:29.955+0000 7f8b8bad6700 1 Processor -- start 2024-04-21T23:08:30.697 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:29.955+0000 7f8b8bad6700 1 -- start start 2024-04-21T23:08:30.697 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:29.955+0000 7f8b8bad6700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b84106b20 0x7f8b8419ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:30.697 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:29.955+0000 7f8b8bad6700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8b8410b0d0 con 0x7f8b84106b20 2024-04-21T23:08:30.697 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:29.955+0000 7f8b89872700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b84106b20 0x7f8b8419ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:30.697 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:29.955+0000 7f8b89872700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b84106b20 0x7f8b8419ab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40754/0 (socket says 172.21.15.40:40754) 2024-04-21T23:08:30.697 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:29.955+0000 7f8b89872700 1 -- 172.21.15.40:0/2776731328 learned_addr learned my addr 172.21.15.40:0/2776731328 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:08:30.697 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:29.955+0000 7f8b89872700 1 -- 172.21.15.40:0/2776731328 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b8419b050 con 0x7f8b84106b20 2024-04-21T23:08:30.697 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:29.956+0000 7f8b89872700 1 --2- 172.21.15.40:0/2776731328 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b84106b20 0x7f8b8419ab10 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8b7c009530 tx=0x7f8b7c00bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:30.697 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:29.956+0000 7f8b82ffd700 1 -- 172.21.15.40:0/2776731328 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8b7c020020 con 0x7f8b84106b20 2024-04-21T23:08:30.697 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:29.956+0000 7f8b82ffd700 1 -- 172.21.15.40:0/2776731328 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f8b7c003b50 con 0x7f8b84106b20 2024-04-21T23:08:30.697 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:29.956+0000 7f8b8bad6700 1 -- 172.21.15.40:0/2776731328 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b8419b2e0 con 0x7f8b84106b20 2024-04-21T23:08:30.697 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:29.956+0000 7f8b82ffd700 1 -- 172.21.15.40:0/2776731328 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8b7c017040 con 0x7f8b84106b20 2024-04-21T23:08:30.698 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:29.956+0000 7f8b8bad6700 1 -- 172.21.15.40:0/2776731328 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b8419dfd0 con 0x7f8b84106b20 2024-04-21T23:08:30.698 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:29.957+0000 7f8b82ffd700 1 -- 172.21.15.40:0/2776731328 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f8b7c023020 con 0x7f8b84106b20 2024-04-21T23:08:30.698 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:29.957+0000 7f8b8bad6700 1 -- 172.21.15.40:0/2776731328 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b84063890 con 0x7f8b84106b20 2024-04-21T23:08:30.698 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:29.957+0000 7f8b82ffd700 1 --2- 172.21.15.40:0/2776731328 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7f8b6c0390a0 0x7f8b6c03b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:30.698 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:29.957+0000 7f8b82ffd700 1 -- 172.21.15.40:0/2776731328 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f8b7c00f050 con 0x7f8b84106b20 2024-04-21T23:08:30.698 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:29.957+0000 7f8b89071700 1 --2- 172.21.15.40:0/2776731328 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7f8b6c0390a0 0x7f8b6c03b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:30.698 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:29.958+0000 7f8b89071700 1 --2- 172.21.15.40:0/2776731328 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7f8b6c0390a0 0x7f8b6c03b560 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8b78006fd0 tx=0x7f8b78006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:30.698 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:29.962+0000 7f8b82ffd700 1 -- 172.21.15.40:0/2776731328 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f8b7c02a020 con 0x7f8b84106b20 2024-04-21T23:08:30.698 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:30.170+0000 7f8b8bad6700 1 -- 172.21.15.40:0/2776731328 --> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7f8b841032c0 con 0x7f8b6c0390a0 2024-04-21T23:08:30.698 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:30.171+0000 7f8b82ffd700 1 -- 172.21.15.40:0/2776731328 <== mgr.14116 v2:172.21.15.40:6800/526554319 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7f8b841032c0 con 0x7f8b6c0390a0 2024-04-21T23:08:30.698 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:30.173+0000 7f8b8bad6700 1 -- 172.21.15.40:0/2776731328 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7f8b6c0390a0 msgr2=0x7f8b6c03b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:30.698 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:30.173+0000 7f8b8bad6700 1 --2- 172.21.15.40:0/2776731328 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7f8b6c0390a0 0x7f8b6c03b560 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8b78006fd0 tx=0x7f8b78006e40 comp rx=0 tx=0).stop 2024-04-21T23:08:30.698 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:30.173+0000 7f8b8bad6700 1 -- 172.21.15.40:0/2776731328 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b84106b20 msgr2=0x7f8b8419ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:30.699 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:30.173+0000 7f8b8bad6700 1 --2- 172.21.15.40:0/2776731328 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b84106b20 0x7f8b8419ab10 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8b7c009530 tx=0x7f8b7c00bd50 comp rx=0 tx=0).stop 2024-04-21T23:08:30.699 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:30.174+0000 7f8b8bad6700 1 -- 172.21.15.40:0/2776731328 shutdown_connections 2024-04-21T23:08:30.699 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:30.174+0000 7f8b8bad6700 1 --2- 172.21.15.40:0/2776731328 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7f8b6c0390a0 0x7f8b6c03b560 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:30.699 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:30.174+0000 7f8b8bad6700 1 --2- 172.21.15.40:0/2776731328 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b84106b20 0x7f8b8419ab10 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:30.699 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:30.174+0000 7f8b8bad6700 1 -- 172.21.15.40:0/2776731328 >> 172.21.15.40:0/2776731328 conn(0x7f8b841020a0 msgr2=0x7f8b84102bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:30.699 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:30.174+0000 7f8b8bad6700 1 -- 172.21.15.40:0/2776731328 shutdown_connections 2024-04-21T23:08:30.699 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:30.174+0000 7f8b8bad6700 1 -- 172.21.15.40:0/2776731328 wait complete. 2024-04-21T23:08:30.699 INFO:teuthology.orchestra.run.smithi040.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-21T23:08:30.699 INFO:teuthology.orchestra.run.smithi040.stdout:Adding key to root@localhost authorized_keys... 2024-04-21T23:08:30.699 INFO:teuthology.orchestra.run.smithi040.stdout:Adding host smithi040... 2024-04-21T23:08:31.089 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:30 smithi040 ceph-mon[27984]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:08:32.214 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:31 smithi040 ceph-mon[27984]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi040", "addr": "172.21.15.40", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:08:33.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:32 smithi040 ceph-mon[27984]: Deploying cephadm binary to smithi040 2024-04-21T23:08:36.136 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout Added host 'smithi040' with addr '172.21.15.40' 2024-04-21T23:08:36.136 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:31.109+0000 7fdf66128700 1 Processor -- start 2024-04-21T23:08:36.136 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:31.109+0000 7fdf66128700 1 -- start start 2024-04-21T23:08:36.136 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:31.109+0000 7fdf66128700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf60108740 0x7fdf6010ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:36.136 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:31.109+0000 7fdf66128700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fdf6007be50 con 0x7fdf60108740 2024-04-21T23:08:36.136 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:31.109+0000 7fdf5f7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf60108740 0x7fdf6010ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:36.136 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:31.109+0000 7fdf5f7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf60108740 0x7fdf6010ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40762/0 (socket says 172.21.15.40:40762) 2024-04-21T23:08:36.136 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:31.109+0000 7fdf5f7fe700 1 -- 172.21.15.40:0/1749801331 learned_addr learned my addr 172.21.15.40:0/1749801331 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:08:36.136 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:31.110+0000 7fdf5f7fe700 1 -- 172.21.15.40:0/1749801331 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf6010b070 con 0x7fdf60108740 2024-04-21T23:08:36.137 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:31.110+0000 7fdf5f7fe700 1 --2- 172.21.15.40:0/1749801331 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf60108740 0x7fdf6010ab30 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fdf500098a0 tx=0x7fdf50009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61ed185021fba3d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:36.137 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:31.110+0000 7fdf5effd700 1 -- 172.21.15.40:0/1749801331 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdf50004160 con 0x7fdf60108740 2024-04-21T23:08:36.137 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:31.110+0000 7fdf5effd700 1 -- 172.21.15.40:0/1749801331 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fdf50004300 con 0x7fdf60108740 2024-04-21T23:08:36.137 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:31.110+0000 7fdf5effd700 1 -- 172.21.15.40:0/1749801331 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdf50011b50 con 0x7fdf60108740 2024-04-21T23:08:36.137 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:31.111+0000 7fdf66128700 1 -- 172.21.15.40:0/1749801331 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf60108740 msgr2=0x7fdf6010ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:36.137 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:31.111+0000 7fdf66128700 1 --2- 172.21.15.40:0/1749801331 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf60108740 0x7fdf6010ab30 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fdf500098a0 tx=0x7fdf50009bb0 comp rx=0 tx=0).stop 2024-04-21T23:08:36.137 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:31.111+0000 7fdf66128700 1 -- 172.21.15.40:0/1749801331 shutdown_connections 2024-04-21T23:08:36.137 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:31.111+0000 7fdf66128700 1 --2- 172.21.15.40:0/1749801331 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf60108740 0x7fdf6010ab30 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:36.137 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:31.111+0000 7fdf66128700 1 -- 172.21.15.40:0/1749801331 >> 172.21.15.40:0/1749801331 conn(0x7fdf60102130 msgr2=0x7fdf60104550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:36.137 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:31.111+0000 7fdf66128700 1 -- 172.21.15.40:0/1749801331 shutdown_connections 2024-04-21T23:08:36.137 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:31.111+0000 7fdf66128700 1 -- 172.21.15.40:0/1749801331 wait complete. 2024-04-21T23:08:36.137 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:31.111+0000 7fdf66128700 1 Processor -- start 2024-04-21T23:08:36.137 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:31.111+0000 7fdf66128700 1 -- start start 2024-04-21T23:08:36.137 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:31.112+0000 7fdf66128700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf60108740 0x7fdf6019bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:36.138 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:31.112+0000 7fdf66128700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fdf6010c040 con 0x7fdf60108740 2024-04-21T23:08:36.138 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:31.112+0000 7fdf5f7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf60108740 0x7fdf6019bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:36.138 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:31.112+0000 7fdf5f7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf60108740 0x7fdf6019bc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40768/0 (socket says 172.21.15.40:40768) 2024-04-21T23:08:36.138 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:31.112+0000 7fdf5f7fe700 1 -- 172.21.15.40:0/2573363440 learned_addr learned my addr 172.21.15.40:0/2573363440 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:08:36.138 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:31.112+0000 7fdf5f7fe700 1 -- 172.21.15.40:0/2573363440 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf6019c160 con 0x7fdf60108740 2024-04-21T23:08:36.138 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:31.112+0000 7fdf5f7fe700 1 --2- 172.21.15.40:0/2573363440 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf60108740 0x7fdf6019bc20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fdf50011b60 tx=0x7fdf50011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:36.138 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:31.112+0000 7fdf5d7fa700 1 -- 172.21.15.40:0/2573363440 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdf50003970 con 0x7fdf60108740 2024-04-21T23:08:36.138 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:31.112+0000 7fdf5d7fa700 1 -- 172.21.15.40:0/2573363440 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fdf50003b10 con 0x7fdf60108740 2024-04-21T23:08:36.138 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:31.112+0000 7fdf66128700 1 -- 172.21.15.40:0/2573363440 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf6019c3f0 con 0x7fdf60108740 2024-04-21T23:08:36.138 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:31.113+0000 7fdf5d7fa700 1 -- 172.21.15.40:0/2573363440 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdf50024610 con 0x7fdf60108740 2024-04-21T23:08:36.138 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:31.113+0000 7fdf66128700 1 -- 172.21.15.40:0/2573363440 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf6019f0e0 con 0x7fdf60108740 2024-04-21T23:08:36.138 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:31.113+0000 7fdf5d7fa700 1 -- 172.21.15.40:0/2573363440 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fdf50024910 con 0x7fdf60108740 2024-04-21T23:08:36.138 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:31.113+0000 7fdf66128700 1 -- 172.21.15.40:0/2573363440 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf60063890 con 0x7fdf60108740 2024-04-21T23:08:36.138 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:31.114+0000 7fdf5d7fa700 1 --2- 172.21.15.40:0/2573363440 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7fdf4c039050 0x7fdf4c03b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:36.139 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:31.114+0000 7fdf5d7fa700 1 -- 172.21.15.40:0/2573363440 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fdf50011e00 con 0x7fdf60108740 2024-04-21T23:08:36.139 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:31.114+0000 7fdf56ffd700 1 --2- 172.21.15.40:0/2573363440 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7fdf4c039050 0x7fdf4c03b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:36.139 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:31.114+0000 7fdf56ffd700 1 --2- 172.21.15.40:0/2573363440 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7fdf4c039050 0x7fdf4c03b510 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fdf48006fd0 tx=0x7fdf48006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:36.139 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:31.119+0000 7fdf5d7fa700 1 -- 172.21.15.40:0/2573363440 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fdf5000bc40 con 0x7fdf60108740 2024-04-21T23:08:36.139 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:31.328+0000 7fdf66128700 1 -- 172.21.15.40:0/2573363440 --> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi040", "addr": "172.21.15.40", "target": ["mon-mgr", ""]}) -- 0x7fdf60103390 con 0x7fdf4c039050 2024-04-21T23:08:36.139 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:35.562+0000 7fdf5d7fa700 1 -- 172.21.15.40:0/2573363440 <== mgr.14116 v2:172.21.15.40:6800/526554319 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7fdf60103390 con 0x7fdf4c039050 2024-04-21T23:08:36.139 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:35.565+0000 7fdf66128700 1 -- 172.21.15.40:0/2573363440 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7fdf4c039050 msgr2=0x7fdf4c03b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:36.139 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:35.565+0000 7fdf66128700 1 --2- 172.21.15.40:0/2573363440 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7fdf4c039050 0x7fdf4c03b510 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fdf48006fd0 tx=0x7fdf48006e40 comp rx=0 tx=0).stop 2024-04-21T23:08:36.139 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:35.565+0000 7fdf66128700 1 -- 172.21.15.40:0/2573363440 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf60108740 msgr2=0x7fdf6019bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:36.139 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:35.565+0000 7fdf66128700 1 --2- 172.21.15.40:0/2573363440 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf60108740 0x7fdf6019bc20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fdf50011b60 tx=0x7fdf50011c40 comp rx=0 tx=0).stop 2024-04-21T23:08:36.139 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:35.565+0000 7fdf66128700 1 -- 172.21.15.40:0/2573363440 shutdown_connections 2024-04-21T23:08:36.139 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:35.565+0000 7fdf66128700 1 --2- 172.21.15.40:0/2573363440 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7fdf4c039050 0x7fdf4c03b510 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:36.139 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:35.565+0000 7fdf66128700 1 --2- 172.21.15.40:0/2573363440 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf60108740 0x7fdf6019bc20 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:36.139 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:35.565+0000 7fdf66128700 1 -- 172.21.15.40:0/2573363440 >> 172.21.15.40:0/2573363440 conn(0x7fdf60102130 msgr2=0x7fdf60102c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:36.140 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:35.565+0000 7fdf66128700 1 -- 172.21.15.40:0/2573363440 shutdown_connections 2024-04-21T23:08:36.140 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:35.565+0000 7fdf66128700 1 -- 172.21.15.40:0/2573363440 wait complete. 2024-04-21T23:08:36.140 INFO:teuthology.orchestra.run.smithi040.stdout:Deploying unmanaged mon service... 2024-04-21T23:08:36.799 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:36 smithi040 ceph-mon[27984]: from='mgr.14116 172.21.15.40:0/550446883' entity='mgr.x' 2024-04-21T23:08:36.799 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:36 smithi040 ceph-mon[27984]: Added host smithi040 2024-04-21T23:08:36.800 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:36 smithi040 ceph-mon[27984]: from='mgr.14116 172.21.15.40:0/550446883' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:08:37.394 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-21T23:08:37.395 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:36.570+0000 7f37289e0700 1 Processor -- start 2024-04-21T23:08:37.395 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:36.570+0000 7f37289e0700 1 -- start start 2024-04-21T23:08:37.395 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:36.570+0000 7f37289e0700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3724107c40 0x7f3724108060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:37.395 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:36.570+0000 7f37289e0700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3724108630 con 0x7f3724107c40 2024-04-21T23:08:37.395 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:36.570+0000 7f372259c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3724107c40 0x7f3724108060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:37.395 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:36.570+0000 7f372259c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3724107c40 0x7f3724108060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33522/0 (socket says 172.21.15.40:33522) 2024-04-21T23:08:37.395 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:36.570+0000 7f372259c700 1 -- 172.21.15.40:0/3876766787 learned_addr learned my addr 172.21.15.40:0/3876766787 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:08:37.395 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:36.570+0000 7f372259c700 1 -- 172.21.15.40:0/3876766787 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3724108e40 con 0x7f3724107c40 2024-04-21T23:08:37.395 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:36.571+0000 7f372259c700 1 --2- 172.21.15.40:0/3876766787 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3724107c40 0x7f3724108060 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f37140098a0 tx=0x7f3714009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cd4584bde91bf60 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:37.395 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:36.571+0000 7f372159a700 1 -- 172.21.15.40:0/3876766787 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3714004160 con 0x7f3724107c40 2024-04-21T23:08:37.395 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:36.571+0000 7f372159a700 1 -- 172.21.15.40:0/3876766787 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f3714004300 con 0x7f3724107c40 2024-04-21T23:08:37.395 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:36.571+0000 7f372159a700 1 -- 172.21.15.40:0/3876766787 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3714011b50 con 0x7f3724107c40 2024-04-21T23:08:37.396 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:36.571+0000 7f37289e0700 1 -- 172.21.15.40:0/3876766787 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3724107c40 msgr2=0x7f3724108060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:37.396 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:36.571+0000 7f37289e0700 1 --2- 172.21.15.40:0/3876766787 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3724107c40 0x7f3724108060 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f37140098a0 tx=0x7f3714009bb0 comp rx=0 tx=0).stop 2024-04-21T23:08:37.396 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:36.571+0000 7f37289e0700 1 -- 172.21.15.40:0/3876766787 shutdown_connections 2024-04-21T23:08:37.396 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:36.572+0000 7f37289e0700 1 --2- 172.21.15.40:0/3876766787 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3724107c40 0x7f3724108060 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:37.396 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:36.572+0000 7f37289e0700 1 -- 172.21.15.40:0/3876766787 >> 172.21.15.40:0/3876766787 conn(0x7f37241031e0 msgr2=0x7f3724105620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:37.396 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:36.572+0000 7f37289e0700 1 -- 172.21.15.40:0/3876766787 shutdown_connections 2024-04-21T23:08:37.396 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:36.572+0000 7f37289e0700 1 -- 172.21.15.40:0/3876766787 wait complete. 2024-04-21T23:08:37.396 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:36.572+0000 7f37289e0700 1 Processor -- start 2024-04-21T23:08:37.396 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:36.572+0000 7f37289e0700 1 -- start start 2024-04-21T23:08:37.396 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:36.572+0000 7f37289e0700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3724107c40 0x7f372419bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:37.396 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:36.572+0000 7f37289e0700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f372410c1f0 con 0x7f3724107c40 2024-04-21T23:08:37.396 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:36.573+0000 7f372259c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3724107c40 0x7f372419bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:37.396 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:36.573+0000 7f372259c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3724107c40 0x7f372419bc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33524/0 (socket says 172.21.15.40:33524) 2024-04-21T23:08:37.396 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:36.573+0000 7f372259c700 1 -- 172.21.15.40:0/4030615273 learned_addr learned my addr 172.21.15.40:0/4030615273 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:08:37.397 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:36.573+0000 7f372259c700 1 -- 172.21.15.40:0/4030615273 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f372419c1c0 con 0x7f3724107c40 2024-04-21T23:08:37.397 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:36.573+0000 7f372259c700 1 --2- 172.21.15.40:0/4030615273 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3724107c40 0x7f372419bc80 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3714011850 tx=0x7f3714011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:37.397 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:36.573+0000 7f37137fe700 1 -- 172.21.15.40:0/4030615273 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3714003af0 con 0x7f3724107c40 2024-04-21T23:08:37.397 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:36.573+0000 7f37137fe700 1 -- 172.21.15.40:0/4030615273 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f3714020070 con 0x7f3724107c40 2024-04-21T23:08:37.397 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:36.573+0000 7f37137fe700 1 -- 172.21.15.40:0/4030615273 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f37140195a0 con 0x7f3724107c40 2024-04-21T23:08:37.397 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:36.573+0000 7f37289e0700 1 -- 172.21.15.40:0/4030615273 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f372419c450 con 0x7f3724107c40 2024-04-21T23:08:37.397 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:36.573+0000 7f37289e0700 1 -- 172.21.15.40:0/4030615273 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f372419f140 con 0x7f3724107c40 2024-04-21T23:08:37.397 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:36.574+0000 7f37137fe700 1 -- 172.21.15.40:0/4030615273 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f3714019850 con 0x7f3724107c40 2024-04-21T23:08:37.397 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:36.574+0000 7f37289e0700 1 -- 172.21.15.40:0/4030615273 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3724063890 con 0x7f3724107c40 2024-04-21T23:08:37.397 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:36.574+0000 7f37137fe700 1 --2- 172.21.15.40:0/4030615273 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7f370c0390a0 0x7f370c03b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:37.397 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:36.574+0000 7f37137fe700 1 -- 172.21.15.40:0/4030615273 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f3714011e00 con 0x7f3724107c40 2024-04-21T23:08:37.397 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:36.574+0000 7f3721d9b700 1 --2- 172.21.15.40:0/4030615273 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7f370c0390a0 0x7f370c03b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:37.397 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:36.575+0000 7f3721d9b700 1 --2- 172.21.15.40:0/4030615273 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7f370c0390a0 0x7f370c03b560 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f3718006fd0 tx=0x7f3718006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:37.397 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:36.579+0000 7f37137fe700 1 -- 172.21.15.40:0/4030615273 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f3714003c90 con 0x7f3724107c40 2024-04-21T23:08:37.397 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:36.800+0000 7f37289e0700 1 -- 172.21.15.40:0/4030615273 --> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f37241044c0 con 0x7f370c0390a0 2024-04-21T23:08:37.397 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:36.804+0000 7f37137fe700 1 -- 172.21.15.40:0/4030615273 <== mgr.14116 v2:172.21.15.40:6800/526554319 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f37241044c0 con 0x7f370c0390a0 2024-04-21T23:08:37.398 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:36.807+0000 7f37289e0700 1 -- 172.21.15.40:0/4030615273 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7f370c0390a0 msgr2=0x7f370c03b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:37.398 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:36.807+0000 7f37289e0700 1 --2- 172.21.15.40:0/4030615273 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7f370c0390a0 0x7f370c03b560 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f3718006fd0 tx=0x7f3718006e40 comp rx=0 tx=0).stop 2024-04-21T23:08:37.398 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:36.807+0000 7f37289e0700 1 -- 172.21.15.40:0/4030615273 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3724107c40 msgr2=0x7f372419bc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:37.398 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:36.807+0000 7f37289e0700 1 --2- 172.21.15.40:0/4030615273 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3724107c40 0x7f372419bc80 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3714011850 tx=0x7f3714011c10 comp rx=0 tx=0).stop 2024-04-21T23:08:37.398 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:36.807+0000 7f37289e0700 1 -- 172.21.15.40:0/4030615273 shutdown_connections 2024-04-21T23:08:37.398 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:36.807+0000 7f37289e0700 1 --2- 172.21.15.40:0/4030615273 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7f370c0390a0 0x7f370c03b560 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:37.398 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:36.807+0000 7f37289e0700 1 --2- 172.21.15.40:0/4030615273 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3724107c40 0x7f372419bc80 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:37.398 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:36.807+0000 7f37289e0700 1 -- 172.21.15.40:0/4030615273 >> 172.21.15.40:0/4030615273 conn(0x7f37241031e0 msgr2=0x7f3724103db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:37.398 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:36.807+0000 7f37289e0700 1 -- 172.21.15.40:0/4030615273 shutdown_connections 2024-04-21T23:08:37.398 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:36.807+0000 7f37289e0700 1 -- 172.21.15.40:0/4030615273 wait complete. 2024-04-21T23:08:37.398 INFO:teuthology.orchestra.run.smithi040.stdout:Deploying unmanaged mgr service... 2024-04-21T23:08:38.070 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:37 smithi040 ceph-mon[27984]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:08:38.070 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:37 smithi040 ceph-mon[27984]: Saving service mon spec with placement count:5 2024-04-21T23:08:38.070 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:37 smithi040 ceph-mon[27984]: from='mgr.14116 172.21.15.40:0/550446883' entity='mgr.x' 2024-04-21T23:08:38.914 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-21T23:08:38.914 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:38.118+0000 7f229e476700 1 Processor -- start 2024-04-21T23:08:38.914 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:38.118+0000 7f229e476700 1 -- start start 2024-04-21T23:08:38.914 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:38.118+0000 7f229e476700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2298108720 0x7f229810ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:38.914 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:38.118+0000 7f229e476700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f229807be70 con 0x7f2298108720 2024-04-21T23:08:38.914 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:38.118+0000 7f2297fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2298108720 0x7f229810ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:38.914 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:38.118+0000 7f2297fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2298108720 0x7f229810ab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33528/0 (socket says 172.21.15.40:33528) 2024-04-21T23:08:38.914 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:38.118+0000 7f2297fff700 1 -- 172.21.15.40:0/1724729074 learned_addr learned my addr 172.21.15.40:0/1724729074 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:08:38.914 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:38.119+0000 7f2297fff700 1 -- 172.21.15.40:0/1724729074 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f229810b050 con 0x7f2298108720 2024-04-21T23:08:38.914 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:38.119+0000 7f2297fff700 1 --2- 172.21.15.40:0/1724729074 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2298108720 0x7f229810ab10 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f22880098a0 tx=0x7f2288009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46aca1dbc4bb6b0a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:38.915 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:38.119+0000 7f2296ffd700 1 -- 172.21.15.40:0/1724729074 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2288004160 con 0x7f2298108720 2024-04-21T23:08:38.915 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:38.119+0000 7f2296ffd700 1 -- 172.21.15.40:0/1724729074 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f2288004300 con 0x7f2298108720 2024-04-21T23:08:38.915 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:38.119+0000 7f2296ffd700 1 -- 172.21.15.40:0/1724729074 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2288011b50 con 0x7f2298108720 2024-04-21T23:08:38.915 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:38.120+0000 7f229e476700 1 -- 172.21.15.40:0/1724729074 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2298108720 msgr2=0x7f229810ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:38.915 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:38.120+0000 7f229e476700 1 --2- 172.21.15.40:0/1724729074 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2298108720 0x7f229810ab10 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f22880098a0 tx=0x7f2288009bb0 comp rx=0 tx=0).stop 2024-04-21T23:08:38.915 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:38.120+0000 7f229e476700 1 -- 172.21.15.40:0/1724729074 shutdown_connections 2024-04-21T23:08:38.915 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:38.120+0000 7f229e476700 1 --2- 172.21.15.40:0/1724729074 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2298108720 0x7f229810ab10 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:38.915 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:38.120+0000 7f229e476700 1 -- 172.21.15.40:0/1724729074 >> 172.21.15.40:0/1724729074 conn(0x7f2298102130 msgr2=0x7f2298104550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:38.915 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:38.120+0000 7f229e476700 1 -- 172.21.15.40:0/1724729074 shutdown_connections 2024-04-21T23:08:38.915 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:38.120+0000 7f229e476700 1 -- 172.21.15.40:0/1724729074 wait complete. 2024-04-21T23:08:38.915 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:38.120+0000 7f229e476700 1 Processor -- start 2024-04-21T23:08:38.915 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:38.121+0000 7f229e476700 1 -- start start 2024-04-21T23:08:38.915 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:38.121+0000 7f229e476700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2298108720 0x7f229819bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:38.915 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:38.121+0000 7f229e476700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f229810c070 con 0x7f2298108720 2024-04-21T23:08:38.915 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:38.121+0000 7f2297fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2298108720 0x7f229819bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:38.915 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:38.121+0000 7f2297fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2298108720 0x7f229819bc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33540/0 (socket says 172.21.15.40:33540) 2024-04-21T23:08:38.916 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:38.121+0000 7f2297fff700 1 -- 172.21.15.40:0/172736711 learned_addr learned my addr 172.21.15.40:0/172736711 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:08:38.916 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:38.121+0000 7f2297fff700 1 -- 172.21.15.40:0/172736711 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f229819c180 con 0x7f2298108720 2024-04-21T23:08:38.916 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:38.121+0000 7f2297fff700 1 --2- 172.21.15.40:0/172736711 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2298108720 0x7f229819bc40 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2288009fb0 tx=0x7f2288011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:38.916 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:38.122+0000 7f22957fa700 1 -- 172.21.15.40:0/172736711 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f22880038b0 con 0x7f2298108720 2024-04-21T23:08:38.916 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:38.122+0000 7f22957fa700 1 -- 172.21.15.40:0/172736711 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f2288011e50 con 0x7f2298108720 2024-04-21T23:08:38.916 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:38.122+0000 7f229e476700 1 -- 172.21.15.40:0/172736711 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f229819c410 con 0x7f2298108720 2024-04-21T23:08:38.916 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:38.122+0000 7f22957fa700 1 -- 172.21.15.40:0/172736711 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2288019420 con 0x7f2298108720 2024-04-21T23:08:38.916 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:38.122+0000 7f229e476700 1 -- 172.21.15.40:0/172736711 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f229819f100 con 0x7f2298108720 2024-04-21T23:08:38.916 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:38.123+0000 7f22957fa700 1 -- 172.21.15.40:0/172736711 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f2288019720 con 0x7f2298108720 2024-04-21T23:08:38.916 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:38.123+0000 7f229e476700 1 -- 172.21.15.40:0/172736711 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2298063890 con 0x7f2298108720 2024-04-21T23:08:38.916 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:38.123+0000 7f22957fa700 1 --2- 172.21.15.40:0/172736711 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7f2280039000 0x7f228003b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:38.916 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:38.123+0000 7f22957fa700 1 -- 172.21.15.40:0/172736711 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f228804d330 con 0x7f2298108720 2024-04-21T23:08:38.916 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:38.123+0000 7f22977fe700 1 --2- 172.21.15.40:0/172736711 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7f2280039000 0x7f228003b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:38.916 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:38.124+0000 7f22977fe700 1 --2- 172.21.15.40:0/172736711 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7f2280039000 0x7f228003b4c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f228c006fd0 tx=0x7f228c006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:38.916 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:38.129+0000 7f22957fa700 1 -- 172.21.15.40:0/172736711 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f228800ba30 con 0x7f2298108720 2024-04-21T23:08:38.917 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:38.350+0000 7f229e476700 1 -- 172.21.15.40:0/172736711 --> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f2298103380 con 0x7f2280039000 2024-04-21T23:08:38.917 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:38.355+0000 7f22957fa700 1 -- 172.21.15.40:0/172736711 <== mgr.14116 v2:172.21.15.40:6800/526554319 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f2298103380 con 0x7f2280039000 2024-04-21T23:08:38.917 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:38.357+0000 7f229e476700 1 -- 172.21.15.40:0/172736711 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7f2280039000 msgr2=0x7f228003b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:38.917 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:38.357+0000 7f229e476700 1 --2- 172.21.15.40:0/172736711 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7f2280039000 0x7f228003b4c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f228c006fd0 tx=0x7f228c006e40 comp rx=0 tx=0).stop 2024-04-21T23:08:38.917 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:38.357+0000 7f229e476700 1 -- 172.21.15.40:0/172736711 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2298108720 msgr2=0x7f229819bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:38.917 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:38.357+0000 7f229e476700 1 --2- 172.21.15.40:0/172736711 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2298108720 0x7f229819bc40 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2288009fb0 tx=0x7f2288011bf0 comp rx=0 tx=0).stop 2024-04-21T23:08:38.917 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:38.357+0000 7f229e476700 1 -- 172.21.15.40:0/172736711 shutdown_connections 2024-04-21T23:08:38.917 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:38.357+0000 7f229e476700 1 --2- 172.21.15.40:0/172736711 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7f2280039000 0x7f228003b4c0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:38.917 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:38.357+0000 7f229e476700 1 --2- 172.21.15.40:0/172736711 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2298108720 0x7f229819bc40 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:38.917 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:38.357+0000 7f229e476700 1 -- 172.21.15.40:0/172736711 >> 172.21.15.40:0/172736711 conn(0x7f2298102130 msgr2=0x7f2298102c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:38.917 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:38.357+0000 7f229e476700 1 -- 172.21.15.40:0/172736711 shutdown_connections 2024-04-21T23:08:38.917 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:38.357+0000 7f229e476700 1 -- 172.21.15.40:0/172736711 wait complete. 2024-04-21T23:08:39.656 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:39 smithi040 ceph-mon[27984]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:08:39.656 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:39 smithi040 ceph-mon[27984]: Saving service mgr spec with placement count:2 2024-04-21T23:08:39.657 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:39 smithi040 ceph-mon[27984]: from='mgr.14116 172.21.15.40:0/550446883' entity='mgr.x' 2024-04-21T23:08:40.447 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:39.441+0000 7f4b9d591700 1 Processor -- start 2024-04-21T23:08:40.447 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:39.441+0000 7f4b9d591700 1 -- start start 2024-04-21T23:08:40.448 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:39.441+0000 7f4b9d591700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b98106b70 0x7f4b98106f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:40.448 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:39.441+0000 7f4b9d591700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4b98107560 con 0x7f4b98106b70 2024-04-21T23:08:40.448 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:39.441+0000 7f4b96ffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b98106b70 0x7f4b98106f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:40.448 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:39.441+0000 7f4b96ffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b98106b70 0x7f4b98106f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33542/0 (socket says 172.21.15.40:33542) 2024-04-21T23:08:40.448 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:39.441+0000 7f4b96ffd700 1 -- 172.21.15.40:0/386468933 learned_addr learned my addr 172.21.15.40:0/386468933 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:08:40.448 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:39.442+0000 7f4b96ffd700 1 -- 172.21.15.40:0/386468933 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b98107d70 con 0x7f4b98106b70 2024-04-21T23:08:40.448 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:39.442+0000 7f4b96ffd700 1 --2- 172.21.15.40:0/386468933 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b98106b70 0x7f4b98106f90 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4b880098a0 tx=0x7f4b88009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cab20c4f33c75c3d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:40.448 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:39.442+0000 7f4b95ffb700 1 -- 172.21.15.40:0/386468933 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4b88004160 con 0x7f4b98106b70 2024-04-21T23:08:40.448 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:39.442+0000 7f4b95ffb700 1 -- 172.21.15.40:0/386468933 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f4b88004300 con 0x7f4b98106b70 2024-04-21T23:08:40.448 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:39.442+0000 7f4b95ffb700 1 -- 172.21.15.40:0/386468933 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4b88011b50 con 0x7f4b98106b70 2024-04-21T23:08:40.449 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:39.442+0000 7f4b9d591700 1 -- 172.21.15.40:0/386468933 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b98106b70 msgr2=0x7f4b98106f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:40.449 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:39.442+0000 7f4b9d591700 1 --2- 172.21.15.40:0/386468933 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b98106b70 0x7f4b98106f90 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4b880098a0 tx=0x7f4b88009bb0 comp rx=0 tx=0).stop 2024-04-21T23:08:40.449 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:39.443+0000 7f4b9d591700 1 -- 172.21.15.40:0/386468933 shutdown_connections 2024-04-21T23:08:40.449 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:39.443+0000 7f4b9d591700 1 --2- 172.21.15.40:0/386468933 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b98106b70 0x7f4b98106f90 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:40.449 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:39.443+0000 7f4b9d591700 1 -- 172.21.15.40:0/386468933 >> 172.21.15.40:0/386468933 conn(0x7f4b98102130 msgr2=0x7f4b98104550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:40.449 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:39.443+0000 7f4b9d591700 1 -- 172.21.15.40:0/386468933 shutdown_connections 2024-04-21T23:08:40.449 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:39.443+0000 7f4b9d591700 1 -- 172.21.15.40:0/386468933 wait complete. 2024-04-21T23:08:40.449 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:39.443+0000 7f4b9d591700 1 Processor -- start 2024-04-21T23:08:40.449 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:39.443+0000 7f4b9d591700 1 -- start start 2024-04-21T23:08:40.449 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:39.444+0000 7f4b9d591700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b98106b70 0x7f4b98191d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:40.449 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:39.444+0000 7f4b9d591700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4b9810b120 con 0x7f4b98106b70 2024-04-21T23:08:40.449 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:39.444+0000 7f4b96ffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b98106b70 0x7f4b98191d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:40.449 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:39.444+0000 7f4b96ffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b98106b70 0x7f4b98191d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33550/0 (socket says 172.21.15.40:33550) 2024-04-21T23:08:40.450 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:39.444+0000 7f4b96ffd700 1 -- 172.21.15.40:0/1252917758 learned_addr learned my addr 172.21.15.40:0/1252917758 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:08:40.450 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:39.444+0000 7f4b96ffd700 1 -- 172.21.15.40:0/1252917758 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b98192260 con 0x7f4b98106b70 2024-04-21T23:08:40.450 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:39.444+0000 7f4b96ffd700 1 --2- 172.21.15.40:0/1252917758 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b98106b70 0x7f4b98191d20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4b88009fb0 tx=0x7f4b88011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:40.450 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:39.444+0000 7f4b7ffff700 1 -- 172.21.15.40:0/1252917758 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4b880038b0 con 0x7f4b98106b70 2024-04-21T23:08:40.450 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:39.444+0000 7f4b7ffff700 1 -- 172.21.15.40:0/1252917758 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f4b88011e50 con 0x7f4b98106b70 2024-04-21T23:08:40.450 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:39.444+0000 7f4b7ffff700 1 -- 172.21.15.40:0/1252917758 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4b88019550 con 0x7f4b98106b70 2024-04-21T23:08:40.450 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:39.444+0000 7f4b9d591700 1 -- 172.21.15.40:0/1252917758 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b98190450 con 0x7f4b98106b70 2024-04-21T23:08:40.450 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:39.444+0000 7f4b9d591700 1 -- 172.21.15.40:0/1252917758 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b98190990 con 0x7f4b98106b70 2024-04-21T23:08:40.450 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:39.445+0000 7f4b7ffff700 1 -- 172.21.15.40:0/1252917758 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f4b88019850 con 0x7f4b98106b70 2024-04-21T23:08:40.450 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:39.445+0000 7f4b9d591700 1 -- 172.21.15.40:0/1252917758 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b98063890 con 0x7f4b98106b70 2024-04-21T23:08:40.450 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:39.445+0000 7f4b7ffff700 1 --2- 172.21.15.40:0/1252917758 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7f4b80039000 0x7f4b8003b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:40.450 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:39.446+0000 7f4b7ffff700 1 -- 172.21.15.40:0/1252917758 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f4b8804d560 con 0x7f4b98106b70 2024-04-21T23:08:40.450 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:39.446+0000 7f4b967fc700 1 --2- 172.21.15.40:0/1252917758 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7f4b80039000 0x7f4b8003b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:40.451 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:39.446+0000 7f4b967fc700 1 --2- 172.21.15.40:0/1252917758 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7f4b80039000 0x7f4b8003b4c0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4b8c006fd0 tx=0x7f4b8c006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:40.451 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:39.450+0000 7f4b7ffff700 1 -- 172.21.15.40:0/1252917758 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f4b8800fda0 con 0x7f4b98106b70 2024-04-21T23:08:40.451 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:39.657+0000 7f4b9d591700 1 -- 172.21.15.40:0/1252917758 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f4b98195aa0 con 0x7f4b98106b70 2024-04-21T23:08:40.451 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:39.662+0000 7f4b7ffff700 1 -- 172.21.15.40:0/1252917758 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v6)=0 v6) ==== 142+0+0 (secure 0 0 0) 0x7f4b88003ac0 con 0x7f4b98106b70 2024-04-21T23:08:40.451 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:39.664+0000 7f4b9d591700 1 -- 172.21.15.40:0/1252917758 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7f4b80039000 msgr2=0x7f4b8003b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:40.451 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:39.664+0000 7f4b9d591700 1 --2- 172.21.15.40:0/1252917758 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7f4b80039000 0x7f4b8003b4c0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4b8c006fd0 tx=0x7f4b8c006e40 comp rx=0 tx=0).stop 2024-04-21T23:08:40.451 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:39.664+0000 7f4b9d591700 1 -- 172.21.15.40:0/1252917758 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b98106b70 msgr2=0x7f4b98191d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:40.451 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:39.664+0000 7f4b9d591700 1 --2- 172.21.15.40:0/1252917758 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b98106b70 0x7f4b98191d20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4b88009fb0 tx=0x7f4b88011bf0 comp rx=0 tx=0).stop 2024-04-21T23:08:40.451 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:39.664+0000 7f4b9d591700 1 -- 172.21.15.40:0/1252917758 shutdown_connections 2024-04-21T23:08:40.451 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:39.664+0000 7f4b9d591700 1 --2- 172.21.15.40:0/1252917758 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7f4b80039000 0x7f4b8003b4c0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:40.451 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:39.664+0000 7f4b9d591700 1 --2- 172.21.15.40:0/1252917758 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b98106b70 0x7f4b98191d20 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:40.451 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:39.664+0000 7f4b9d591700 1 -- 172.21.15.40:0/1252917758 >> 172.21.15.40:0/1252917758 conn(0x7f4b98102130 msgr2=0x7f4b98103030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:40.451 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:39.665+0000 7f4b9d591700 1 -- 172.21.15.40:0/1252917758 shutdown_connections 2024-04-21T23:08:40.452 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:39.665+0000 7f4b9d591700 1 -- 172.21.15.40:0/1252917758 wait complete. 2024-04-21T23:08:40.884 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:40 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/1252917758' entity='client.admin' 2024-04-21T23:08:40.884 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:40 smithi040 ceph-mon[27984]: from='mgr.14116 172.21.15.40:0/550446883' entity='mgr.x' 2024-04-21T23:08:40.884 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:40 smithi040 ceph-mon[27984]: from='mgr.14116 172.21.15.40:0/550446883' entity='mgr.x' 2024-04-21T23:08:42.358 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:42 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/3823746354' entity='client.admin' 2024-04-21T23:08:42.358 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:42 smithi040 ceph-mon[27984]: from='mgr.14116 172.21.15.40:0/550446883' entity='mgr.x' 2024-04-21T23:08:42.366 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:40.907+0000 7f348c3ab700 1 Processor -- start 2024-04-21T23:08:42.366 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:40.907+0000 7f348c3ab700 1 -- start start 2024-04-21T23:08:42.366 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:40.907+0000 7f348c3ab700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3484107c80 0x7f34841080a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:42.366 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:40.908+0000 7f348c3ab700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3484108670 con 0x7f3484107c80 2024-04-21T23:08:42.366 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:40.908+0000 7f348a147700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3484107c80 0x7f34841080a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:42.366 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:40.908+0000 7f348a147700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3484107c80 0x7f34841080a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33560/0 (socket says 172.21.15.40:33560) 2024-04-21T23:08:42.366 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:40.908+0000 7f348a147700 1 -- 172.21.15.40:0/4153256634 learned_addr learned my addr 172.21.15.40:0/4153256634 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:08:42.366 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:40.908+0000 7f348a147700 1 -- 172.21.15.40:0/4153256634 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3484108e30 con 0x7f3484107c80 2024-04-21T23:08:42.366 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:40.908+0000 7f348a147700 1 --2- 172.21.15.40:0/4153256634 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3484107c80 0x7f34841080a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f34800098a0 tx=0x7f3480009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f57a544c8ea7386 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:42.366 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:40.909+0000 7f3489145700 1 -- 172.21.15.40:0/4153256634 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3480004160 con 0x7f3484107c80 2024-04-21T23:08:42.367 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:40.909+0000 7f3489145700 1 -- 172.21.15.40:0/4153256634 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3480004300 con 0x7f3484107c80 2024-04-21T23:08:42.367 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:40.909+0000 7f3489145700 1 -- 172.21.15.40:0/4153256634 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3480011b20 con 0x7f3484107c80 2024-04-21T23:08:42.367 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:40.909+0000 7f348c3ab700 1 -- 172.21.15.40:0/4153256634 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3484107c80 msgr2=0x7f34841080a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:42.367 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:40.909+0000 7f348c3ab700 1 --2- 172.21.15.40:0/4153256634 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3484107c80 0x7f34841080a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f34800098a0 tx=0x7f3480009bb0 comp rx=0 tx=0).stop 2024-04-21T23:08:42.367 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:40.909+0000 7f348c3ab700 1 -- 172.21.15.40:0/4153256634 shutdown_connections 2024-04-21T23:08:42.367 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:40.909+0000 7f348c3ab700 1 --2- 172.21.15.40:0/4153256634 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3484107c80 0x7f34841080a0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:42.367 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:40.909+0000 7f348c3ab700 1 -- 172.21.15.40:0/4153256634 >> 172.21.15.40:0/4153256634 conn(0x7f3484103240 msgr2=0x7f3484105660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:42.367 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:40.909+0000 7f348c3ab700 1 -- 172.21.15.40:0/4153256634 shutdown_connections 2024-04-21T23:08:42.367 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:40.909+0000 7f348c3ab700 1 -- 172.21.15.40:0/4153256634 wait complete. 2024-04-21T23:08:42.367 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:40.910+0000 7f348c3ab700 1 Processor -- start 2024-04-21T23:08:42.367 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:40.910+0000 7f348c3ab700 1 -- start start 2024-04-21T23:08:42.367 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:40.910+0000 7f348c3ab700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3484107c80 0x7f348407db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:42.367 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:40.910+0000 7f348c3ab700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f348410c1e0 con 0x7f3484107c80 2024-04-21T23:08:42.367 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:40.910+0000 7f348a147700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3484107c80 0x7f348407db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:42.368 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:40.910+0000 7f348a147700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3484107c80 0x7f348407db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33574/0 (socket says 172.21.15.40:33574) 2024-04-21T23:08:42.368 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:40.910+0000 7f348a147700 1 -- 172.21.15.40:0/3823746354 learned_addr learned my addr 172.21.15.40:0/3823746354 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:08:42.368 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:40.911+0000 7f348a147700 1 -- 172.21.15.40:0/3823746354 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f348407e0b0 con 0x7f3484107c80 2024-04-21T23:08:42.368 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:40.911+0000 7f348a147700 1 --2- 172.21.15.40:0/3823746354 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3484107c80 0x7f348407db70 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f34800117e0 tx=0x7f3480011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:42.368 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:40.911+0000 7f347b7fe700 1 -- 172.21.15.40:0/3823746354 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3480003bf0 con 0x7f3484107c80 2024-04-21T23:08:42.368 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:40.911+0000 7f347b7fe700 1 -- 172.21.15.40:0/3823746354 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f348001a910 con 0x7f3484107c80 2024-04-21T23:08:42.368 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:40.911+0000 7f348c3ab700 1 -- 172.21.15.40:0/3823746354 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f348407c2a0 con 0x7f3484107c80 2024-04-21T23:08:42.368 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:40.911+0000 7f347b7fe700 1 -- 172.21.15.40:0/3823746354 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3480024970 con 0x7f3484107c80 2024-04-21T23:08:42.368 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:40.911+0000 7f348c3ab700 1 -- 172.21.15.40:0/3823746354 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f348407c760 con 0x7f3484107c80 2024-04-21T23:08:42.368 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:40.912+0000 7f347b7fe700 1 -- 172.21.15.40:0/3823746354 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f3480022020 con 0x7f3484107c80 2024-04-21T23:08:42.368 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:40.912+0000 7f347b7fe700 1 --2- 172.21.15.40:0/3823746354 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7f34700390a0 0x7f347003b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:42.368 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:40.912+0000 7f347b7fe700 1 -- 172.21.15.40:0/3823746354 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f348004d430 con 0x7f3484107c80 2024-04-21T23:08:42.368 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:40.912+0000 7f348c3ab700 1 -- 172.21.15.40:0/3823746354 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3484063890 con 0x7f3484107c80 2024-04-21T23:08:42.369 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:40.912+0000 7f3489946700 1 --2- 172.21.15.40:0/3823746354 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7f34700390a0 0x7f347003b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:42.369 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:40.913+0000 7f3489946700 1 --2- 172.21.15.40:0/3823746354 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7f34700390a0 0x7f347003b560 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3474006fd0 tx=0x7f3474006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:42.369 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:40.917+0000 7f347b7fe700 1 -- 172.21.15.40:0/3823746354 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f348002d910 con 0x7f3484107c80 2024-04-21T23:08:42.369 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:41.137+0000 7f348c3ab700 1 -- 172.21.15.40:0/3823746354 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f348407c9e0 con 0x7f3484107c80 2024-04-21T23:08:42.369 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:41.140+0000 7f347b7fe700 1 -- 172.21.15.40:0/3823746354 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7f3480032030 con 0x7f3484107c80 2024-04-21T23:08:42.369 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:41.142+0000 7f348c3ab700 1 -- 172.21.15.40:0/3823746354 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7f34700390a0 msgr2=0x7f347003b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:42.369 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:41.142+0000 7f348c3ab700 1 --2- 172.21.15.40:0/3823746354 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7f34700390a0 0x7f347003b560 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3474006fd0 tx=0x7f3474006e40 comp rx=0 tx=0).stop 2024-04-21T23:08:42.369 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:41.142+0000 7f348c3ab700 1 -- 172.21.15.40:0/3823746354 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3484107c80 msgr2=0x7f348407db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:42.369 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:41.142+0000 7f348c3ab700 1 --2- 172.21.15.40:0/3823746354 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3484107c80 0x7f348407db70 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f34800117e0 tx=0x7f3480011ba0 comp rx=0 tx=0).stop 2024-04-21T23:08:42.369 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:41.142+0000 7f348c3ab700 1 -- 172.21.15.40:0/3823746354 shutdown_connections 2024-04-21T23:08:42.369 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:41.142+0000 7f348c3ab700 1 --2- 172.21.15.40:0/3823746354 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7f34700390a0 0x7f347003b560 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:42.369 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:41.142+0000 7f348c3ab700 1 --2- 172.21.15.40:0/3823746354 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3484107c80 0x7f348407db70 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:42.369 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:41.142+0000 7f348c3ab700 1 -- 172.21.15.40:0/3823746354 >> 172.21.15.40:0/3823746354 conn(0x7f3484103240 msgr2=0x7f3484103d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:42.369 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:41.142+0000 7f348c3ab700 1 -- 172.21.15.40:0/3823746354 shutdown_connections 2024-04-21T23:08:42.370 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:41.142+0000 7f348c3ab700 1 -- 172.21.15.40:0/3823746354 wait complete. 2024-04-21T23:08:42.370 INFO:teuthology.orchestra.run.smithi040.stdout:Enabling the dashboard module... 2024-04-21T23:08:43.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:43 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/2334150061' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-21T23:08:43.432 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:43 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: ignoring --setuser ceph since I am not root 2024-04-21T23:08:43.432 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:43 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: ignoring --setgroup ceph since I am not root 2024-04-21T23:08:43.432 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:43 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:08:43.328+0000 7f3ad6993200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-21T23:08:43.724 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:43 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:08:43.525+0000 7f3ad6993200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-21T23:08:43.724 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:43 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:08:43.631+0000 7f3ad6993200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-21T23:08:43.731 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:42.852+0000 7f132e4b8700 1 Processor -- start 2024-04-21T23:08:43.731 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:42.852+0000 7f132e4b8700 1 -- start start 2024-04-21T23:08:43.731 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:42.852+0000 7f132e4b8700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1328106290 0x7f13281066b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:43.731 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:42.852+0000 7f132e4b8700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1328106c80 con 0x7f1328106290 2024-04-21T23:08:43.731 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:42.852+0000 7f1327fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1328106290 0x7f13281066b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:43.731 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:42.852+0000 7f1327fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1328106290 0x7f13281066b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34212/0 (socket says 172.21.15.40:34212) 2024-04-21T23:08:43.732 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:42.852+0000 7f1327fff700 1 -- 172.21.15.40:0/1754427810 learned_addr learned my addr 172.21.15.40:0/1754427810 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:08:43.732 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:42.853+0000 7f1327fff700 1 -- 172.21.15.40:0/1754427810 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1328107490 con 0x7f1328106290 2024-04-21T23:08:43.732 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:42.853+0000 7f1327fff700 1 --2- 172.21.15.40:0/1754427810 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1328106290 0x7f13281066b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f131800aa40 tx=0x7f131800ad50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd06235a07e0b938 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:43.732 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:42.853+0000 7f1326ffd700 1 -- 172.21.15.40:0/1754427810 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f131800b040 con 0x7f1328106290 2024-04-21T23:08:43.732 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:42.853+0000 7f1326ffd700 1 -- 172.21.15.40:0/1754427810 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f131800c8f0 con 0x7f1328106290 2024-04-21T23:08:43.732 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:42.853+0000 7f1326ffd700 1 -- 172.21.15.40:0/1754427810 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1318003c10 con 0x7f1328106290 2024-04-21T23:08:43.732 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:42.854+0000 7f132e4b8700 1 -- 172.21.15.40:0/1754427810 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1328106290 msgr2=0x7f13281066b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:43.732 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:42.854+0000 7f132e4b8700 1 --2- 172.21.15.40:0/1754427810 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1328106290 0x7f13281066b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f131800aa40 tx=0x7f131800ad50 comp rx=0 tx=0).stop 2024-04-21T23:08:43.732 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:42.854+0000 7f132e4b8700 1 -- 172.21.15.40:0/1754427810 shutdown_connections 2024-04-21T23:08:43.732 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:42.854+0000 7f132e4b8700 1 --2- 172.21.15.40:0/1754427810 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1328106290 0x7f13281066b0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:43.732 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:42.854+0000 7f132e4b8700 1 -- 172.21.15.40:0/1754427810 >> 172.21.15.40:0/1754427810 conn(0x7f1328101810 msgr2=0x7f1328103c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:43.733 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:42.854+0000 7f132e4b8700 1 -- 172.21.15.40:0/1754427810 shutdown_connections 2024-04-21T23:08:43.733 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:42.854+0000 7f132e4b8700 1 -- 172.21.15.40:0/1754427810 wait complete. 2024-04-21T23:08:43.733 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:42.854+0000 7f132e4b8700 1 Processor -- start 2024-04-21T23:08:43.733 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:42.854+0000 7f132e4b8700 1 -- start start 2024-04-21T23:08:43.733 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:42.855+0000 7f132e4b8700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1328106290 0x7f132818ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:43.733 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:42.855+0000 7f132e4b8700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1328107c60 con 0x7f1328106290 2024-04-21T23:08:43.733 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:42.855+0000 7f1327fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1328106290 0x7f132818ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:43.733 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:42.855+0000 7f1327fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1328106290 0x7f132818ee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34224/0 (socket says 172.21.15.40:34224) 2024-04-21T23:08:43.733 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:42.855+0000 7f1327fff700 1 -- 172.21.15.40:0/2334150061 learned_addr learned my addr 172.21.15.40:0/2334150061 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:08:43.733 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:42.855+0000 7f1327fff700 1 -- 172.21.15.40:0/2334150061 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f132818f390 con 0x7f1328106290 2024-04-21T23:08:43.734 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:42.855+0000 7f1327fff700 1 --2- 172.21.15.40:0/2334150061 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1328106290 0x7f132818ee50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f131800afd0 tx=0x7f1318006c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:43.734 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:42.856+0000 7f13257fa700 1 -- 172.21.15.40:0/2334150061 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1318025070 con 0x7f1328106290 2024-04-21T23:08:43.734 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:42.856+0000 7f13257fa700 1 -- 172.21.15.40:0/2334150061 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1318004340 con 0x7f1328106290 2024-04-21T23:08:43.734 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:42.856+0000 7f132e4b8700 1 -- 172.21.15.40:0/2334150061 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f132818d580 con 0x7f1328106290 2024-04-21T23:08:43.734 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:42.856+0000 7f13257fa700 1 -- 172.21.15.40:0/2334150061 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1318010610 con 0x7f1328106290 2024-04-21T23:08:43.734 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:42.856+0000 7f132e4b8700 1 -- 172.21.15.40:0/2334150061 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f132818dac0 con 0x7f1328106290 2024-04-21T23:08:43.734 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:42.857+0000 7f13257fa700 1 -- 172.21.15.40:0/2334150061 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f1318028070 con 0x7f1328106290 2024-04-21T23:08:43.734 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:42.857+0000 7f13257fa700 1 --2- 172.21.15.40:0/2334150061 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7f1310039050 0x7f131003b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:43.734 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:42.857+0000 7f132e4b8700 1 -- 172.21.15.40:0/2334150061 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1328063890 con 0x7f1328106290 2024-04-21T23:08:43.734 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:42.857+0000 7f13257fa700 1 -- 172.21.15.40:0/2334150061 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f131804d030 con 0x7f1328106290 2024-04-21T23:08:43.734 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:42.857+0000 7f13277fe700 1 --2- 172.21.15.40:0/2334150061 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7f1310039050 0x7f131003b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:43.734 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:42.857+0000 7f13277fe700 1 --2- 172.21.15.40:0/2334150061 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7f1310039050 0x7f131003b510 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f131c006fd0 tx=0x7f131c006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:43.734 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:42.863+0000 7f13257fa700 1 -- 172.21.15.40:0/2334150061 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f1318014070 con 0x7f1328106290 2024-04-21T23:08:43.735 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:43.134+0000 7f132e4b8700 1 -- 172.21.15.40:0/2334150061 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7f13281954a0 con 0x7f1328106290 2024-04-21T23:08:43.735 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:43.142+0000 7f13257fa700 1 -- 172.21.15.40:0/2334150061 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7f1318026260 con 0x7f1328106290 2024-04-21T23:08:43.735 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:43.144+0000 7f13257fa700 1 -- 172.21.15.40:0/2334150061 <== mon.0 v2:172.21.15.40:3300/0 8 ==== mgrmap(e 9) ==== 46300+0+0 (secure 0 0 0) 0x7f1318020a00 con 0x7f1328106290 2024-04-21T23:08:43.735 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:43.145+0000 7f132e4b8700 1 -- 172.21.15.40:0/2334150061 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7f1310039050 msgr2=0x7f131003b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:43.735 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:43.145+0000 7f132e4b8700 1 --2- 172.21.15.40:0/2334150061 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7f1310039050 0x7f131003b510 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f131c006fd0 tx=0x7f131c006e40 comp rx=0 tx=0).stop 2024-04-21T23:08:43.735 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:43.145+0000 7f132e4b8700 1 -- 172.21.15.40:0/2334150061 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1328106290 msgr2=0x7f132818ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:43.735 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:43.145+0000 7f132e4b8700 1 --2- 172.21.15.40:0/2334150061 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1328106290 0x7f132818ee50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f131800afd0 tx=0x7f1318006c80 comp rx=0 tx=0).stop 2024-04-21T23:08:43.735 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:43.146+0000 7f132e4b8700 1 -- 172.21.15.40:0/2334150061 shutdown_connections 2024-04-21T23:08:43.735 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:43.146+0000 7f132e4b8700 1 --2- 172.21.15.40:0/2334150061 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7f1310039050 0x7f131003b510 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:43.735 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:43.146+0000 7f132e4b8700 1 --2- 172.21.15.40:0/2334150061 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1328106290 0x7f132818ee50 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:43.735 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:43.146+0000 7f132e4b8700 1 -- 172.21.15.40:0/2334150061 >> 172.21.15.40:0/2334150061 conn(0x7f1328101810 msgr2=0x7f1328192830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:43.735 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:43.146+0000 7f132e4b8700 1 -- 172.21.15.40:0/2334150061 shutdown_connections 2024-04-21T23:08:43.735 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:43.146+0000 7f132e4b8700 1 -- 172.21.15.40:0/2334150061 wait complete. 2024-04-21T23:08:43.999 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:43 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:08:43.956+0000 7f3ad6993200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-21T23:08:44.429 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:44 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/2334150061' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-21T23:08:44.429 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:44 smithi040 ceph-mon[27984]: mgrmap e9: x(active, since 19s) 2024-04-21T23:08:44.430 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:44 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:08:44.047+0000 7f3ad6993200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-21T23:08:45.163 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:45 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/557826728' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-21T23:08:45.166 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout { 2024-04-21T23:08:45.173 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-21T23:08:45.173 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-21T23:08:45.173 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-21T23:08:45.173 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-21T23:08:45.173 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout } 2024-04-21T23:08:45.173 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:44.223+0000 7f661b2f8700 1 Processor -- start 2024-04-21T23:08:45.173 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:44.223+0000 7f661b2f8700 1 -- start start 2024-04-21T23:08:45.173 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:44.224+0000 7f661b2f8700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6614106ba0 0x7f6614106fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:45.173 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:44.224+0000 7f661b2f8700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6614107590 con 0x7f6614106ba0 2024-04-21T23:08:45.174 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:44.224+0000 7f6619094700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6614106ba0 0x7f6614106fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:45.174 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:44.224+0000 7f6619094700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6614106ba0 0x7f6614106fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34256/0 (socket says 172.21.15.40:34256) 2024-04-21T23:08:45.174 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:44.224+0000 7f6619094700 1 -- 172.21.15.40:0/4072890247 learned_addr learned my addr 172.21.15.40:0/4072890247 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:08:45.174 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:44.224+0000 7f6619094700 1 -- 172.21.15.40:0/4072890247 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6614107da0 con 0x7f6614106ba0 2024-04-21T23:08:45.174 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:44.224+0000 7f6619094700 1 --2- 172.21.15.40:0/4072890247 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6614106ba0 0x7f6614106fc0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f661000bb40 tx=0x7f661000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3b8fe4c3d9dbf38 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:45.174 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:44.225+0000 7f660bfff700 1 -- 172.21.15.40:0/4072890247 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f661000dbe0 con 0x7f6614106ba0 2024-04-21T23:08:45.174 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:44.225+0000 7f660bfff700 1 -- 172.21.15.40:0/4072890247 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6610004510 con 0x7f6614106ba0 2024-04-21T23:08:45.174 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:44.225+0000 7f660bfff700 1 -- 172.21.15.40:0/4072890247 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6610010470 con 0x7f6614106ba0 2024-04-21T23:08:45.174 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:44.225+0000 7f661b2f8700 1 -- 172.21.15.40:0/4072890247 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6614106ba0 msgr2=0x7f6614106fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:45.174 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:44.225+0000 7f661b2f8700 1 --2- 172.21.15.40:0/4072890247 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6614106ba0 0x7f6614106fc0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f661000bb40 tx=0x7f661000be50 comp rx=0 tx=0).stop 2024-04-21T23:08:45.174 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:44.226+0000 7f661b2f8700 1 -- 172.21.15.40:0/4072890247 shutdown_connections 2024-04-21T23:08:45.174 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:44.226+0000 7f661b2f8700 1 --2- 172.21.15.40:0/4072890247 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6614106ba0 0x7f6614106fc0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:45.174 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:44.226+0000 7f661b2f8700 1 -- 172.21.15.40:0/4072890247 >> 172.21.15.40:0/4072890247 conn(0x7f6614102100 msgr2=0x7f6614104580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:45.175 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:44.226+0000 7f661b2f8700 1 -- 172.21.15.40:0/4072890247 shutdown_connections 2024-04-21T23:08:45.175 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:44.226+0000 7f661b2f8700 1 -- 172.21.15.40:0/4072890247 wait complete. 2024-04-21T23:08:45.175 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:44.226+0000 7f661b2f8700 1 Processor -- start 2024-04-21T23:08:45.175 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:44.226+0000 7f661b2f8700 1 -- start start 2024-04-21T23:08:45.175 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:44.227+0000 7f661b2f8700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6614106ba0 0x7f6614199a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:45.175 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:44.227+0000 7f661b2f8700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f661407afd0 con 0x7f6614106ba0 2024-04-21T23:08:45.175 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:44.227+0000 7f6619094700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6614106ba0 0x7f6614199a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:45.175 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:44.227+0000 7f6619094700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6614106ba0 0x7f6614199a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34258/0 (socket says 172.21.15.40:34258) 2024-04-21T23:08:45.175 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:44.227+0000 7f6619094700 1 -- 172.21.15.40:0/557826728 learned_addr learned my addr 172.21.15.40:0/557826728 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:08:45.175 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:44.227+0000 7f6619094700 1 -- 172.21.15.40:0/557826728 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6614199fa0 con 0x7f6614106ba0 2024-04-21T23:08:45.175 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:44.227+0000 7f6619094700 1 --2- 172.21.15.40:0/557826728 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6614106ba0 0x7f6614199a60 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f661000b340 tx=0x7f6610003e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:45.175 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:44.227+0000 7f660a7fc700 1 -- 172.21.15.40:0/557826728 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6610027070 con 0x7f6614106ba0 2024-04-21T23:08:45.175 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:44.227+0000 7f660a7fc700 1 -- 172.21.15.40:0/557826728 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6610010c50 con 0x7f6614106ba0 2024-04-21T23:08:45.176 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:44.227+0000 7f660a7fc700 1 -- 172.21.15.40:0/557826728 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6610022be0 con 0x7f6614106ba0 2024-04-21T23:08:45.176 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:44.227+0000 7f661b2f8700 1 -- 172.21.15.40:0/557826728 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f661419a230 con 0x7f6614106ba0 2024-04-21T23:08:45.176 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:44.228+0000 7f661b2f8700 1 -- 172.21.15.40:0/557826728 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f661419ced0 con 0x7f6614106ba0 2024-04-21T23:08:45.176 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:44.228+0000 7f661b2f8700 1 -- 172.21.15.40:0/557826728 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6614063890 con 0x7f6614106ba0 2024-04-21T23:08:45.176 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:44.229+0000 7f660a7fc700 1 -- 172.21.15.40:0/557826728 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 9) ==== 46300+0+0 (secure 0 0 0) 0x7f661002a070 con 0x7f6614106ba0 2024-04-21T23:08:45.176 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:44.229+0000 7f660a7fc700 1 --2- 172.21.15.40:0/557826728 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7f6600039050 0x7f660003b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:45.176 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:44.229+0000 7f660a7fc700 1 -- 172.21.15.40:0/557826728 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f661001f070 con 0x7f6614106ba0 2024-04-21T23:08:45.176 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:44.229+0000 7f6618893700 1 -- 172.21.15.40:0/557826728 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7f6600039050 msgr2=0x7f660003b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:6800/526554319 2024-04-21T23:08:45.176 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:44.229+0000 7f6618893700 1 --2- 172.21.15.40:0/557826728 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7f6600039050 0x7f660003b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T23:08:45.176 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:44.234+0000 7f660a7fc700 1 -- 172.21.15.40:0/557826728 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f6610031030 con 0x7f6614106ba0 2024-04-21T23:08:45.176 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:44.430+0000 7f6618893700 1 -- 172.21.15.40:0/557826728 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7f6600039050 msgr2=0x7f660003b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:6800/526554319 2024-04-21T23:08:45.176 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:44.430+0000 7f6618893700 1 --2- 172.21.15.40:0/557826728 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7f6600039050 0x7f660003b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-21T23:08:45.176 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:44.550+0000 7f661b2f8700 1 -- 172.21.15.40:0/557826728 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f6614192fc0 con 0x7f6614106ba0 2024-04-21T23:08:45.176 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:44.551+0000 7f660a7fc700 1 -- 172.21.15.40:0/557826728 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7f6614192fc0 con 0x7f6614106ba0 2024-04-21T23:08:45.177 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:44.553+0000 7f661b2f8700 1 -- 172.21.15.40:0/557826728 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7f6600039050 msgr2=0x7f660003b510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T23:08:45.177 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:44.553+0000 7f661b2f8700 1 --2- 172.21.15.40:0/557826728 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7f6600039050 0x7f660003b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:45.177 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:44.553+0000 7f661b2f8700 1 -- 172.21.15.40:0/557826728 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6614106ba0 msgr2=0x7f6614199a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:45.177 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:44.553+0000 7f661b2f8700 1 --2- 172.21.15.40:0/557826728 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6614106ba0 0x7f6614199a60 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f661000b340 tx=0x7f6610003e80 comp rx=0 tx=0).stop 2024-04-21T23:08:45.177 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:44.553+0000 7f661b2f8700 1 -- 172.21.15.40:0/557826728 shutdown_connections 2024-04-21T23:08:45.177 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:44.553+0000 7f661b2f8700 1 --2- 172.21.15.40:0/557826728 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7f6600039050 0x7f660003b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:45.177 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:44.553+0000 7f661b2f8700 1 --2- 172.21.15.40:0/557826728 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6614106ba0 0x7f6614199a60 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:45.177 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:44.553+0000 7f661b2f8700 1 -- 172.21.15.40:0/557826728 >> 172.21.15.40:0/557826728 conn(0x7f6614102100 msgr2=0x7f6614190080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:45.177 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:44.553+0000 7f661b2f8700 1 -- 172.21.15.40:0/557826728 shutdown_connections 2024-04-21T23:08:45.177 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:44.553+0000 7f661b2f8700 1 -- 172.21.15.40:0/557826728 wait complete. 2024-04-21T23:08:45.177 INFO:teuthology.orchestra.run.smithi040.stdout:Waiting for the mgr to restart... 2024-04-21T23:08:45.177 INFO:teuthology.orchestra.run.smithi040.stdout:Waiting for mgr epoch 9... 2024-04-21T23:08:45.420 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:45 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:08:45.313+0000 7f3ad6993200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-21T23:08:45.675 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:45 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:08:45.495+0000 7f3ad6993200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-21T23:08:45.675 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:45 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:08:45.675+0000 7f3ad6993200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-21T23:08:46.244 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:45 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:08:45.976+0000 7f3ad6993200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-21T23:08:46.245 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:46 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:08:46.073+0000 7f3ad6993200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-21T23:08:46.682 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:46 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:08:46.284+0000 7f3ad6993200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-21T23:08:46.682 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:46 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:08:46.484+0000 7f3ad6993200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-21T23:08:47.432 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:47 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:08:47.112+0000 7f3ad6993200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-21T23:08:47.866 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:47 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:08:47.555+0000 7f3ad6993200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-21T23:08:47.866 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:47 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:08:47.656+0000 7f3ad6993200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-21T23:08:47.866 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:47 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:08:47.866+0000 7f3ad6993200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-21T23:08:48.647 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:48 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:08:48.415+0000 7f3ad6993200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-21T23:08:48.647 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:48 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:08:48.511+0000 7f3ad6993200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-21T23:08:48.911 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:48 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:08:48.768+0000 7f3ad6993200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-21T23:08:48.911 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:48 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:08:48.911+0000 7f3ad6993200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-21T23:08:49.182 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:49 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:08:49.013+0000 7f3ad6993200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-21T23:08:49.182 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:49 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:08:49.133+0000 7f3ad6993200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-21T23:08:49.682 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:49 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:08:49.229+0000 7f3ad6993200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-21T23:08:50.355 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:50 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:08:50.075+0000 7f3ad6993200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-21T23:08:50.356 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:08:50 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:08:50.303+0000 7f3ad6993200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-21T23:08:50.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:50 smithi040 ceph-mon[27984]: Active manager daemon x restarted 2024-04-21T23:08:50.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:50 smithi040 ceph-mon[27984]: Activating manager daemon x 2024-04-21T23:08:50.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:50 smithi040 ceph-mon[27984]: osdmap e3: 0 total, 0 up, 0 in 2024-04-21T23:08:50.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:50 smithi040 ceph-mon[27984]: mgrmap e10: x(active, starting, since 0.0031356s) 2024-04-21T23:08:50.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:50 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T23:08:50.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:50 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-21T23:08:50.683 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:50 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-21T23:08:50.683 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:50 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-21T23:08:50.683 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:50 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-21T23:08:50.683 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:50 smithi040 ceph-mon[27984]: Manager daemon x is now available 2024-04-21T23:08:50.683 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:50 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:08:51.637 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:51 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-21T23:08:51.638 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:51 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-21T23:08:51.638 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:51 smithi040 ceph-mon[27984]: [21/Apr/2024:23:08:50] ENGINE Bus STARTING 2024-04-21T23:08:51.638 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:51 smithi040 ceph-mon[27984]: [21/Apr/2024:23:08:50] ENGINE Serving on http://172.21.15.40:8765 2024-04-21T23:08:51.638 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:51 smithi040 ceph-mon[27984]: [21/Apr/2024:23:08:50] ENGINE Serving on https://172.21.15.40:7150 2024-04-21T23:08:51.638 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:51 smithi040 ceph-mon[27984]: [21/Apr/2024:23:08:50] ENGINE Bus STARTED 2024-04-21T23:08:51.638 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:51 smithi040 ceph-mon[27984]: mgrmap e11: x(active, since 1.00635s) 2024-04-21T23:08:51.831 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout { 2024-04-21T23:08:51.831 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-21T23:08:51.831 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-21T23:08:51.832 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout } 2024-04-21T23:08:51.832 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:45.638+0000 7f8ebef59700 1 Processor -- start 2024-04-21T23:08:51.832 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:45.638+0000 7f8ebef59700 1 -- start start 2024-04-21T23:08:51.832 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:45.638+0000 7f8ebef59700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8eb81080e0 0x7f8eb8108500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:51.832 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:45.639+0000 7f8ebef59700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8eb8108ad0 con 0x7f8eb81080e0 2024-04-21T23:08:51.832 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:45.639+0000 7f8ebccf5700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8eb81080e0 0x7f8eb8108500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:51.832 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:45.639+0000 7f8ebccf5700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8eb81080e0 0x7f8eb8108500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34264/0 (socket says 172.21.15.40:34264) 2024-04-21T23:08:51.832 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:45.639+0000 7f8ebccf5700 1 -- 172.21.15.40:0/3069291533 learned_addr learned my addr 172.21.15.40:0/3069291533 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:08:51.832 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:45.639+0000 7f8ebccf5700 1 -- 172.21.15.40:0/3069291533 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8eb8108c50 con 0x7f8eb81080e0 2024-04-21T23:08:51.832 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:45.639+0000 7f8ebccf5700 1 --2- 172.21.15.40:0/3069291533 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8eb81080e0 0x7f8eb8108500 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8eac0098a0 tx=0x7f8eac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b64cf7f01522438f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:51.832 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:45.639+0000 7f8eb77fe700 1 -- 172.21.15.40:0/3069291533 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8eac004160 con 0x7f8eb81080e0 2024-04-21T23:08:51.832 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:45.639+0000 7f8eb77fe700 1 -- 172.21.15.40:0/3069291533 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8eac004300 con 0x7f8eb81080e0 2024-04-21T23:08:51.832 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:45.639+0000 7f8eb77fe700 1 -- 172.21.15.40:0/3069291533 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8eac011b20 con 0x7f8eb81080e0 2024-04-21T23:08:51.832 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:45.640+0000 7f8ebef59700 1 -- 172.21.15.40:0/3069291533 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8eb81080e0 msgr2=0x7f8eb8108500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:51.833 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:45.640+0000 7f8ebef59700 1 --2- 172.21.15.40:0/3069291533 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8eb81080e0 0x7f8eb8108500 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8eac0098a0 tx=0x7f8eac009bb0 comp rx=0 tx=0).stop 2024-04-21T23:08:51.833 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:45.640+0000 7f8ebef59700 1 -- 172.21.15.40:0/3069291533 shutdown_connections 2024-04-21T23:08:51.833 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:45.640+0000 7f8ebef59700 1 --2- 172.21.15.40:0/3069291533 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8eb81080e0 0x7f8eb8108500 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:51.833 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:45.640+0000 7f8ebef59700 1 -- 172.21.15.40:0/3069291533 >> 172.21.15.40:0/3069291533 conn(0x7f8eb8101810 msgr2=0x7f8eb8103c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:51.833 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:45.640+0000 7f8ebef59700 1 -- 172.21.15.40:0/3069291533 shutdown_connections 2024-04-21T23:08:51.833 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:45.640+0000 7f8ebef59700 1 -- 172.21.15.40:0/3069291533 wait complete. 2024-04-21T23:08:51.833 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:45.641+0000 7f8ebef59700 1 Processor -- start 2024-04-21T23:08:51.833 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:45.641+0000 7f8ebef59700 1 -- start start 2024-04-21T23:08:51.833 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:45.641+0000 7f8ebef59700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8eb81080e0 0x7f8eb819dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:51.833 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:45.641+0000 7f8ebef59700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8eb8106b00 con 0x7f8eb81080e0 2024-04-21T23:08:51.833 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:45.641+0000 7f8ebccf5700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8eb81080e0 0x7f8eb819dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:51.833 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:45.641+0000 7f8ebccf5700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8eb81080e0 0x7f8eb819dd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34272/0 (socket says 172.21.15.40:34272) 2024-04-21T23:08:51.833 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:45.641+0000 7f8ebccf5700 1 -- 172.21.15.40:0/2462947363 learned_addr learned my addr 172.21.15.40:0/2462947363 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:08:51.833 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:45.641+0000 7f8ebccf5700 1 -- 172.21.15.40:0/2462947363 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8eb819e280 con 0x7f8eb81080e0 2024-04-21T23:08:51.834 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:45.642+0000 7f8ebccf5700 1 --2- 172.21.15.40:0/2462947363 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8eb81080e0 0x7f8eb819dd40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8eac0117e0 tx=0x7f8eac011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:51.834 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:45.642+0000 7f8eb5ffb700 1 -- 172.21.15.40:0/2462947363 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8eac003bf0 con 0x7f8eb81080e0 2024-04-21T23:08:51.834 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:45.642+0000 7f8eb5ffb700 1 -- 172.21.15.40:0/2462947363 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8eac01a910 con 0x7f8eb81080e0 2024-04-21T23:08:51.834 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:45.642+0000 7f8ebef59700 1 -- 172.21.15.40:0/2462947363 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8eb819e570 con 0x7f8eb81080e0 2024-04-21T23:08:51.834 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:45.642+0000 7f8eb5ffb700 1 -- 172.21.15.40:0/2462947363 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8eac024970 con 0x7f8eb81080e0 2024-04-21T23:08:51.834 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:45.642+0000 7f8ebef59700 1 -- 172.21.15.40:0/2462947363 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8eb81a11a0 con 0x7f8eb81080e0 2024-04-21T23:08:51.834 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:45.643+0000 7f8eb5ffb700 1 -- 172.21.15.40:0/2462947363 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 9) ==== 46300+0+0 (secure 0 0 0) 0x7f8eac022020 con 0x7f8eb81080e0 2024-04-21T23:08:51.834 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:45.643+0000 7f8eb5ffb700 1 --2- 172.21.15.40:0/2462947363 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7f8ea8039050 0x7f8ea803b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:51.834 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:45.643+0000 7f8ebef59700 1 -- 172.21.15.40:0/2462947363 --> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f8eb8063890 con 0x7f8ea8039050 2024-04-21T23:08:51.834 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:45.643+0000 7f8eb5ffb700 1 -- 172.21.15.40:0/2462947363 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f8eac04d5b0 con 0x7f8eb81080e0 2024-04-21T23:08:51.834 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:45.643+0000 7f8eb7fff700 1 -- 172.21.15.40:0/2462947363 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7f8ea8039050 msgr2=0x7f8ea803b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:6800/526554319 2024-04-21T23:08:51.834 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:45.643+0000 7f8eb7fff700 1 --2- 172.21.15.40:0/2462947363 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7f8ea8039050 0x7f8ea803b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T23:08:51.834 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:45.844+0000 7f8eb7fff700 1 -- 172.21.15.40:0/2462947363 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7f8ea8039050 msgr2=0x7f8ea803b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:6800/526554319 2024-04-21T23:08:51.835 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:45.844+0000 7f8eb7fff700 1 --2- 172.21.15.40:0/2462947363 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7f8ea8039050 0x7f8ea803b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-21T23:08:51.835 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:46.245+0000 7f8eb7fff700 1 -- 172.21.15.40:0/2462947363 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7f8ea8039050 msgr2=0x7f8ea803b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:6800/526554319 2024-04-21T23:08:51.835 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:46.245+0000 7f8eb7fff700 1 --2- 172.21.15.40:0/2462947363 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7f8ea8039050 0x7f8ea803b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-21T23:08:51.835 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:47.046+0000 7f8eb7fff700 1 -- 172.21.15.40:0/2462947363 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7f8ea8039050 msgr2=0x7f8ea803b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:6800/526554319 2024-04-21T23:08:51.835 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:47.046+0000 7f8eb7fff700 1 --2- 172.21.15.40:0/2462947363 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7f8ea8039050 0x7f8ea803b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-21T23:08:51.835 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:48.647+0000 7f8eb7fff700 1 -- 172.21.15.40:0/2462947363 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7f8ea8039050 msgr2=0x7f8ea803b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:6800/526554319 2024-04-21T23:08:51.835 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:48.647+0000 7f8eb7fff700 1 --2- 172.21.15.40:0/2462947363 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7f8ea8039050 0x7f8ea803b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-21T23:08:51.835 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:50.307+0000 7f8eb5ffb700 1 -- 172.21.15.40:0/2462947363 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mgrmap(e 10) ==== 46067+0+0 (secure 0 0 0) 0x7f8eac02d910 con 0x7f8eb81080e0 2024-04-21T23:08:51.835 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:50.307+0000 7f8eb5ffb700 1 -- 172.21.15.40:0/2462947363 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7f8ea8039050 msgr2=0x7f8ea803b510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T23:08:51.835 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:50.307+0000 7f8eb5ffb700 1 --2- 172.21.15.40:0/2462947363 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7f8ea8039050 0x7f8ea803b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:51.835 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:51.308+0000 7f8eb5ffb700 1 -- 172.21.15.40:0/2462947363 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mgrmap(e 11) ==== 46194+0+0 (secure 0 0 0) 0x7f8eac04f7d0 con 0x7f8eb81080e0 2024-04-21T23:08:51.835 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:51.309+0000 7f8eb5ffb700 1 --2- 172.21.15.40:0/2462947363 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f8ea803c920 0x7f8ea803ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:51.835 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:51.309+0000 7f8eb5ffb700 1 -- 172.21.15.40:0/2462947363 --> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f8eac008ec0 con 0x7f8ea803c920 2024-04-21T23:08:51.835 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:51.309+0000 7f8eb7fff700 1 --2- 172.21.15.40:0/2462947363 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f8ea803c920 0x7f8ea803ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:51.836 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:51.310+0000 7f8eb7fff700 1 --2- 172.21.15.40:0/2462947363 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f8ea803c920 0x7f8ea803ed10 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f8ea4003a10 tx=0x7f8ea40092b0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:51.836 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:51.311+0000 7f8eb5ffb700 1 -- 172.21.15.40:0/2462947363 <== mgr.14148 v2:172.21.15.40:6800/2500760788 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f8eac008ec0 con 0x7f8ea803c920 2024-04-21T23:08:51.836 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:51.317+0000 7f8ebef59700 1 -- 172.21.15.40:0/2462947363 --> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f8eb8195b00 con 0x7f8ea803c920 2024-04-21T23:08:51.836 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:51.317+0000 7f8eb5ffb700 1 -- 172.21.15.40:0/2462947363 <== mgr.14148 v2:172.21.15.40:6800/2500760788 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7f8eb8195b00 con 0x7f8ea803c920 2024-04-21T23:08:51.836 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:51.318+0000 7f8ebef59700 1 -- 172.21.15.40:0/2462947363 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f8ea803c920 msgr2=0x7f8ea803ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:51.836 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:51.318+0000 7f8ebef59700 1 --2- 172.21.15.40:0/2462947363 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f8ea803c920 0x7f8ea803ed10 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f8ea4003a10 tx=0x7f8ea40092b0 comp rx=0 tx=0).stop 2024-04-21T23:08:51.836 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:51.318+0000 7f8ebef59700 1 -- 172.21.15.40:0/2462947363 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8eb81080e0 msgr2=0x7f8eb819dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:51.836 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:51.318+0000 7f8ebef59700 1 --2- 172.21.15.40:0/2462947363 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8eb81080e0 0x7f8eb819dd40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8eac0117e0 tx=0x7f8eac011c10 comp rx=0 tx=0).stop 2024-04-21T23:08:51.836 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:51.318+0000 7f8ebef59700 1 -- 172.21.15.40:0/2462947363 shutdown_connections 2024-04-21T23:08:51.836 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:51.318+0000 7f8ebef59700 1 --2- 172.21.15.40:0/2462947363 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f8ea803c920 0x7f8ea803ed10 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:51.837 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:51.318+0000 7f8ebef59700 1 --2- 172.21.15.40:0/2462947363 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8eb81080e0 0x7f8eb819dd40 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:51.837 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:51.318+0000 7f8ebef59700 1 --2- 172.21.15.40:0/2462947363 >> [v2:172.21.15.40:6800/526554319,v1:172.21.15.40:6801/526554319] conn(0x7f8ea8039050 0x7f8ea803b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:51.837 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:51.318+0000 7f8ebef59700 1 -- 172.21.15.40:0/2462947363 >> 172.21.15.40:0/2462947363 conn(0x7f8eb8101810 msgr2=0x7f8eb8194360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:51.837 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:51.318+0000 7f8ebef59700 1 -- 172.21.15.40:0/2462947363 shutdown_connections 2024-04-21T23:08:51.837 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:51.318+0000 7f8ebef59700 1 -- 172.21.15.40:0/2462947363 wait complete. 2024-04-21T23:08:51.837 INFO:teuthology.orchestra.run.smithi040.stdout:mgr epoch 9 is available 2024-04-21T23:08:51.837 INFO:teuthology.orchestra.run.smithi040.stdout:Generating a dashboard self-signed certificate... 2024-04-21T23:08:52.483 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:52 smithi040 ceph-mon[27984]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-21T23:08:52.483 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:52 smithi040 ceph-mon[27984]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-21T23:08:53.196 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-21T23:08:53.196 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:52.258+0000 7f3a3834b700 1 Processor -- start 2024-04-21T23:08:53.196 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:52.258+0000 7f3a3834b700 1 -- start start 2024-04-21T23:08:53.197 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:52.258+0000 7f3a3834b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a30107c40 0x7f3a30108060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:53.197 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:52.258+0000 7f3a3834b700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3a30108630 con 0x7f3a30107c40 2024-04-21T23:08:53.197 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:52.258+0000 7f3a360e7700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a30107c40 0x7f3a30108060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:53.197 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:52.258+0000 7f3a360e7700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a30107c40 0x7f3a30108060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54342/0 (socket says 172.21.15.40:54342) 2024-04-21T23:08:53.197 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:52.258+0000 7f3a360e7700 1 -- 172.21.15.40:0/340248060 learned_addr learned my addr 172.21.15.40:0/340248060 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:08:53.197 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:52.259+0000 7f3a360e7700 1 -- 172.21.15.40:0/340248060 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a30108e40 con 0x7f3a30107c40 2024-04-21T23:08:53.197 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:52.259+0000 7f3a360e7700 1 --2- 172.21.15.40:0/340248060 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a30107c40 0x7f3a30108060 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3a2c0098a0 tx=0x7f3a2c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a9c32dc1f2eecc8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:53.197 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:52.259+0000 7f3a350e5700 1 -- 172.21.15.40:0/340248060 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3a2c004160 con 0x7f3a30107c40 2024-04-21T23:08:53.197 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:52.259+0000 7f3a350e5700 1 -- 172.21.15.40:0/340248060 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a2c004300 con 0x7f3a30107c40 2024-04-21T23:08:53.197 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:52.259+0000 7f3a350e5700 1 -- 172.21.15.40:0/340248060 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3a2c011b20 con 0x7f3a30107c40 2024-04-21T23:08:53.197 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:52.260+0000 7f3a3834b700 1 -- 172.21.15.40:0/340248060 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a30107c40 msgr2=0x7f3a30108060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:53.198 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:52.260+0000 7f3a3834b700 1 --2- 172.21.15.40:0/340248060 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a30107c40 0x7f3a30108060 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3a2c0098a0 tx=0x7f3a2c009bb0 comp rx=0 tx=0).stop 2024-04-21T23:08:53.198 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:52.260+0000 7f3a3834b700 1 -- 172.21.15.40:0/340248060 shutdown_connections 2024-04-21T23:08:53.198 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:52.260+0000 7f3a3834b700 1 --2- 172.21.15.40:0/340248060 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a30107c40 0x7f3a30108060 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:53.198 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:52.260+0000 7f3a3834b700 1 -- 172.21.15.40:0/340248060 >> 172.21.15.40:0/340248060 conn(0x7f3a301031e0 msgr2=0x7f3a30105620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:53.198 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:52.260+0000 7f3a3834b700 1 -- 172.21.15.40:0/340248060 shutdown_connections 2024-04-21T23:08:53.198 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:52.260+0000 7f3a3834b700 1 -- 172.21.15.40:0/340248060 wait complete. 2024-04-21T23:08:53.198 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:52.260+0000 7f3a3834b700 1 Processor -- start 2024-04-21T23:08:53.198 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:52.260+0000 7f3a3834b700 1 -- start start 2024-04-21T23:08:53.198 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:52.261+0000 7f3a3834b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a30107c40 0x7f3a3007db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:53.198 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:52.261+0000 7f3a3834b700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3a3010c1f0 con 0x7f3a30107c40 2024-04-21T23:08:53.198 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:52.261+0000 7f3a360e7700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a30107c40 0x7f3a3007db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:53.198 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:52.261+0000 7f3a360e7700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a30107c40 0x7f3a3007db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54352/0 (socket says 172.21.15.40:54352) 2024-04-21T23:08:53.199 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:52.261+0000 7f3a360e7700 1 -- 172.21.15.40:0/3106295960 learned_addr learned my addr 172.21.15.40:0/3106295960 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:08:53.199 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:52.261+0000 7f3a360e7700 1 -- 172.21.15.40:0/3106295960 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a3007e060 con 0x7f3a30107c40 2024-04-21T23:08:53.199 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:52.261+0000 7f3a360e7700 1 --2- 172.21.15.40:0/3106295960 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a30107c40 0x7f3a3007db20 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f3a2c0116a0 tx=0x7f3a2c011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:53.199 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:52.261+0000 7f3a277fe700 1 -- 172.21.15.40:0/3106295960 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3a2c003a40 con 0x7f3a30107c40 2024-04-21T23:08:53.199 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:52.261+0000 7f3a277fe700 1 -- 172.21.15.40:0/3106295960 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a2c003be0 con 0x7f3a30107c40 2024-04-21T23:08:53.199 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:52.261+0000 7f3a3834b700 1 -- 172.21.15.40:0/3106295960 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a3007c2a0 con 0x7f3a30107c40 2024-04-21T23:08:53.199 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:52.261+0000 7f3a277fe700 1 -- 172.21.15.40:0/3106295960 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3a2c01a720 con 0x7f3a30107c40 2024-04-21T23:08:53.199 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:52.261+0000 7f3a3834b700 1 -- 172.21.15.40:0/3106295960 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a3007c700 con 0x7f3a30107c40 2024-04-21T23:08:53.199 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:52.262+0000 7f3a277fe700 1 -- 172.21.15.40:0/3106295960 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 11) ==== 46194+0+0 (secure 0 0 0) 0x7f3a2c01a9d0 con 0x7f3a30107c40 2024-04-21T23:08:53.199 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:52.262+0000 7f3a3834b700 1 -- 172.21.15.40:0/3106295960 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a30063890 con 0x7f3a30107c40 2024-04-21T23:08:53.199 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:52.263+0000 7f3a277fe700 1 --2- 172.21.15.40:0/3106295960 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f3a1c038f90 0x7f3a1c03b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:53.200 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:52.263+0000 7f3a277fe700 1 -- 172.21.15.40:0/3106295960 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f3a2c04d4f0 con 0x7f3a30107c40 2024-04-21T23:08:53.200 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:52.263+0000 7f3a358e6700 1 --2- 172.21.15.40:0/3106295960 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f3a1c038f90 0x7f3a1c03b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:53.200 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:52.263+0000 7f3a358e6700 1 --2- 172.21.15.40:0/3106295960 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f3a1c038f90 0x7f3a1c03b450 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3a20006fd0 tx=0x7f3a20006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:53.200 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:52.268+0000 7f3a277fe700 1 -- 172.21.15.40:0/3106295960 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f3a2c011e20 con 0x7f3a30107c40 2024-04-21T23:08:53.200 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:52.361+0000 7f3a277fe700 1 -- 172.21.15.40:0/3106295960 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mgrmap(e 12) ==== 46300+0+0 (secure 0 0 0) 0x7f3a2c02d3f0 con 0x7f3a30107c40 2024-04-21T23:08:53.200 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:52.484+0000 7f3a3834b700 1 -- 172.21.15.40:0/3106295960 --> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7f3a3007ced0 con 0x7f3a1c038f90 2024-04-21T23:08:53.200 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:52.594+0000 7f3a277fe700 1 -- 172.21.15.40:0/3106295960 <== mgr.14148 v2:172.21.15.40:6800/2500760788 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7f3a3007ced0 con 0x7f3a1c038f90 2024-04-21T23:08:53.200 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:52.596+0000 7f3a3834b700 1 -- 172.21.15.40:0/3106295960 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f3a1c038f90 msgr2=0x7f3a1c03b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:53.200 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:52.596+0000 7f3a3834b700 1 --2- 172.21.15.40:0/3106295960 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f3a1c038f90 0x7f3a1c03b450 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3a20006fd0 tx=0x7f3a20006e40 comp rx=0 tx=0).stop 2024-04-21T23:08:53.200 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:52.596+0000 7f3a3834b700 1 -- 172.21.15.40:0/3106295960 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a30107c40 msgr2=0x7f3a3007db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:53.200 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:52.596+0000 7f3a3834b700 1 --2- 172.21.15.40:0/3106295960 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a30107c40 0x7f3a3007db20 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f3a2c0116a0 tx=0x7f3a2c011ba0 comp rx=0 tx=0).stop 2024-04-21T23:08:53.200 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:52.596+0000 7f3a3834b700 1 -- 172.21.15.40:0/3106295960 shutdown_connections 2024-04-21T23:08:53.201 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:52.596+0000 7f3a3834b700 1 --2- 172.21.15.40:0/3106295960 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f3a1c038f90 0x7f3a1c03b450 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:53.201 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:52.596+0000 7f3a3834b700 1 --2- 172.21.15.40:0/3106295960 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a30107c40 0x7f3a3007db20 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:53.201 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:52.596+0000 7f3a3834b700 1 -- 172.21.15.40:0/3106295960 >> 172.21.15.40:0/3106295960 conn(0x7f3a301031e0 msgr2=0x7f3a30103d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:53.201 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:52.596+0000 7f3a3834b700 1 -- 172.21.15.40:0/3106295960 shutdown_connections 2024-04-21T23:08:53.201 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:52.596+0000 7f3a3834b700 1 -- 172.21.15.40:0/3106295960 wait complete. 2024-04-21T23:08:53.201 INFO:teuthology.orchestra.run.smithi040.stdout:Creating initial admin user... 2024-04-21T23:08:53.574 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:53 smithi040 ceph-mon[27984]: mgrmap e12: x(active, since 2s) 2024-04-21T23:08:53.574 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:53 smithi040 ceph-mon[27984]: from='client.14160 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:08:53.574 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:53 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:08:53.574 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:53 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:08:54.685 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$n5qXfpeBv5BrUzYJn8mUQeTlNz1W4f8sR0r30dtQXYNtmdMCbr7iK", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1713740934, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-21T23:08:54.685 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:53.585+0000 7efd55b5f700 1 Processor -- start 2024-04-21T23:08:54.685 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:53.585+0000 7efd55b5f700 1 -- start start 2024-04-21T23:08:54.686 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:53.585+0000 7efd55b5f700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd501086d0 0x7efd5010aac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:54.686 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:53.585+0000 7efd55b5f700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7efd5007be70 con 0x7efd501086d0 2024-04-21T23:08:54.686 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:53.585+0000 7efd4f7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd501086d0 0x7efd5010aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:54.686 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:53.585+0000 7efd4f7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd501086d0 0x7efd5010aac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54354/0 (socket says 172.21.15.40:54354) 2024-04-21T23:08:54.686 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:53.585+0000 7efd4f7fe700 1 -- 172.21.15.40:0/534748278 learned_addr learned my addr 172.21.15.40:0/534748278 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:08:54.686 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:53.586+0000 7efd4f7fe700 1 -- 172.21.15.40:0/534748278 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd5010b000 con 0x7efd501086d0 2024-04-21T23:08:54.686 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:53.586+0000 7efd4f7fe700 1 --2- 172.21.15.40:0/534748278 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd501086d0 0x7efd5010aac0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7efd400098a0 tx=0x7efd40009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbb2d82d66552716 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:54.686 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:53.586+0000 7efd4e7fc700 1 -- 172.21.15.40:0/534748278 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efd40004160 con 0x7efd501086d0 2024-04-21T23:08:54.686 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:53.586+0000 7efd4e7fc700 1 -- 172.21.15.40:0/534748278 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd40004300 con 0x7efd501086d0 2024-04-21T23:08:54.686 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:53.586+0000 7efd4e7fc700 1 -- 172.21.15.40:0/534748278 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efd40011b20 con 0x7efd501086d0 2024-04-21T23:08:54.686 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:53.587+0000 7efd55b5f700 1 -- 172.21.15.40:0/534748278 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd501086d0 msgr2=0x7efd5010aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:54.686 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:53.587+0000 7efd55b5f700 1 --2- 172.21.15.40:0/534748278 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd501086d0 0x7efd5010aac0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7efd400098a0 tx=0x7efd40009bb0 comp rx=0 tx=0).stop 2024-04-21T23:08:54.686 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:53.587+0000 7efd55b5f700 1 -- 172.21.15.40:0/534748278 shutdown_connections 2024-04-21T23:08:54.687 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:53.587+0000 7efd55b5f700 1 --2- 172.21.15.40:0/534748278 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd501086d0 0x7efd5010aac0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:54.687 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:53.587+0000 7efd55b5f700 1 -- 172.21.15.40:0/534748278 >> 172.21.15.40:0/534748278 conn(0x7efd501020a0 msgr2=0x7efd50104500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:54.687 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:53.587+0000 7efd55b5f700 1 -- 172.21.15.40:0/534748278 shutdown_connections 2024-04-21T23:08:54.687 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:53.587+0000 7efd55b5f700 1 -- 172.21.15.40:0/534748278 wait complete. 2024-04-21T23:08:54.687 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:53.587+0000 7efd55b5f700 1 Processor -- start 2024-04-21T23:08:54.687 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:53.588+0000 7efd55b5f700 1 -- start start 2024-04-21T23:08:54.687 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:53.588+0000 7efd55b5f700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd501086d0 0x7efd5019bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:54.687 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:53.588+0000 7efd55b5f700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7efd5010c060 con 0x7efd501086d0 2024-04-21T23:08:54.687 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:53.588+0000 7efd4f7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd501086d0 0x7efd5019bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:54.687 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:53.588+0000 7efd4f7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd501086d0 0x7efd5019bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54368/0 (socket says 172.21.15.40:54368) 2024-04-21T23:08:54.687 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:53.588+0000 7efd4f7fe700 1 -- 172.21.15.40:0/1836099767 learned_addr learned my addr 172.21.15.40:0/1836099767 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:08:54.687 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:53.588+0000 7efd4f7fe700 1 -- 172.21.15.40:0/1836099767 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd5019c090 con 0x7efd501086d0 2024-04-21T23:08:54.687 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:53.588+0000 7efd4f7fe700 1 --2- 172.21.15.40:0/1836099767 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd501086d0 0x7efd5019bb50 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7efd400117e0 tx=0x7efd40011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:54.687 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:53.589+0000 7efd4cff9700 1 -- 172.21.15.40:0/1836099767 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efd40003bf0 con 0x7efd501086d0 2024-04-21T23:08:54.688 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:53.589+0000 7efd55b5f700 1 -- 172.21.15.40:0/1836099767 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd5019c320 con 0x7efd501086d0 2024-04-21T23:08:54.688 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:53.589+0000 7efd55b5f700 1 -- 172.21.15.40:0/1836099767 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd5019c780 con 0x7efd501086d0 2024-04-21T23:08:54.688 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:53.589+0000 7efd4cff9700 1 -- 172.21.15.40:0/1836099767 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd4001a910 con 0x7efd501086d0 2024-04-21T23:08:54.688 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:53.589+0000 7efd4cff9700 1 -- 172.21.15.40:0/1836099767 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efd40024970 con 0x7efd501086d0 2024-04-21T23:08:54.688 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:53.590+0000 7efd4cff9700 1 -- 172.21.15.40:0/1836099767 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 12) ==== 46300+0+0 (secure 0 0 0) 0x7efd40022020 con 0x7efd501086d0 2024-04-21T23:08:54.688 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:53.590+0000 7efd4cff9700 1 --2- 172.21.15.40:0/1836099767 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7efd380390b0 0x7efd3803b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:54.688 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:53.590+0000 7efd4cff9700 1 -- 172.21.15.40:0/1836099767 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7efd4004d630 con 0x7efd501086d0 2024-04-21T23:08:54.688 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:53.590+0000 7efd55b5f700 1 -- 172.21.15.40:0/1836099767 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd3c005360 con 0x7efd501086d0 2024-04-21T23:08:54.688 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:53.590+0000 7efd4effd700 1 --2- 172.21.15.40:0/1836099767 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7efd380390b0 0x7efd3803b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:54.688 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:53.591+0000 7efd4effd700 1 --2- 172.21.15.40:0/1836099767 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7efd380390b0 0x7efd3803b570 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7efd44006fd0 tx=0x7efd44006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:54.688 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:53.595+0000 7efd4cff9700 1 -- 172.21.15.40:0/1836099767 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7efd40011e20 con 0x7efd501086d0 2024-04-21T23:08:54.688 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:53.832+0000 7efd55b5f700 1 -- 172.21.15.40:0/1836099767 --> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7efd3c002430 con 0x7efd380390b0 2024-04-21T23:08:54.688 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:54.079+0000 7efd4cff9700 1 -- 172.21.15.40:0/1836099767 <== mgr.14148 v2:172.21.15.40:6800/2500760788 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7efd3c002430 con 0x7efd380390b0 2024-04-21T23:08:54.688 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:54.081+0000 7efd55b5f700 1 -- 172.21.15.40:0/1836099767 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7efd380390b0 msgr2=0x7efd3803b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:54.689 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:54.081+0000 7efd55b5f700 1 --2- 172.21.15.40:0/1836099767 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7efd380390b0 0x7efd3803b570 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7efd44006fd0 tx=0x7efd44006e40 comp rx=0 tx=0).stop 2024-04-21T23:08:54.689 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:54.081+0000 7efd55b5f700 1 -- 172.21.15.40:0/1836099767 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd501086d0 msgr2=0x7efd5019bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:54.689 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:54.081+0000 7efd55b5f700 1 --2- 172.21.15.40:0/1836099767 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd501086d0 0x7efd5019bb50 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7efd400117e0 tx=0x7efd40011c10 comp rx=0 tx=0).stop 2024-04-21T23:08:54.689 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:54.081+0000 7efd55b5f700 1 -- 172.21.15.40:0/1836099767 shutdown_connections 2024-04-21T23:08:54.689 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:54.081+0000 7efd55b5f700 1 --2- 172.21.15.40:0/1836099767 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7efd380390b0 0x7efd3803b570 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:54.689 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:54.081+0000 7efd55b5f700 1 --2- 172.21.15.40:0/1836099767 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd501086d0 0x7efd5019bb50 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:54.689 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:54.081+0000 7efd55b5f700 1 -- 172.21.15.40:0/1836099767 >> 172.21.15.40:0/1836099767 conn(0x7efd501020a0 msgr2=0x7efd50102b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:54.689 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:54.081+0000 7efd55b5f700 1 -- 172.21.15.40:0/1836099767 shutdown_connections 2024-04-21T23:08:54.689 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:54.081+0000 7efd55b5f700 1 -- 172.21.15.40:0/1836099767 wait complete. 2024-04-21T23:08:54.689 INFO:teuthology.orchestra.run.smithi040.stdout:Fetching dashboard port number... 2024-04-21T23:08:54.804 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:54 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:08:54.804 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:54 smithi040 ceph-mon[27984]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:08:54.804 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:54 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:08:56.043 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:55 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:08:56.043 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:55 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/577975543' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-21T23:08:56.588 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 8443 2024-04-21T23:08:56.588 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:55.127+0000 7fe18faa1700 1 Processor -- start 2024-04-21T23:08:56.588 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:55.127+0000 7fe18faa1700 1 -- start start 2024-04-21T23:08:56.588 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:55.128+0000 7fe18faa1700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe188108710 0x7fe18810ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:56.588 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:55.128+0000 7fe18faa1700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe18807be30 con 0x7fe188108710 2024-04-21T23:08:56.588 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:55.128+0000 7fe18d83d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe188108710 0x7fe18810ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:56.588 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:55.128+0000 7fe18d83d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe188108710 0x7fe18810ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54376/0 (socket says 172.21.15.40:54376) 2024-04-21T23:08:56.589 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:55.128+0000 7fe18d83d700 1 -- 172.21.15.40:0/1686734638 learned_addr learned my addr 172.21.15.40:0/1686734638 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:08:56.589 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:55.128+0000 7fe18d83d700 1 -- 172.21.15.40:0/1686734638 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe18810b040 con 0x7fe188108710 2024-04-21T23:08:56.589 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:55.128+0000 7fe18d83d700 1 --2- 172.21.15.40:0/1686734638 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe188108710 0x7fe18810ab00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe1840098a0 tx=0x7fe184009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3c3ca26870cd7cd server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:56.589 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:55.128+0000 7fe18c83b700 1 -- 172.21.15.40:0/1686734638 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe184004160 con 0x7fe188108710 2024-04-21T23:08:56.589 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:55.128+0000 7fe18c83b700 1 -- 172.21.15.40:0/1686734638 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe184004300 con 0x7fe188108710 2024-04-21T23:08:56.589 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:55.128+0000 7fe18c83b700 1 -- 172.21.15.40:0/1686734638 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe184011b20 con 0x7fe188108710 2024-04-21T23:08:56.589 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:55.129+0000 7fe18faa1700 1 -- 172.21.15.40:0/1686734638 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe188108710 msgr2=0x7fe18810ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:56.589 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:55.129+0000 7fe18faa1700 1 --2- 172.21.15.40:0/1686734638 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe188108710 0x7fe18810ab00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe1840098a0 tx=0x7fe184009bb0 comp rx=0 tx=0).stop 2024-04-21T23:08:56.589 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:55.129+0000 7fe18faa1700 1 -- 172.21.15.40:0/1686734638 shutdown_connections 2024-04-21T23:08:56.589 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:55.129+0000 7fe18faa1700 1 --2- 172.21.15.40:0/1686734638 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe188108710 0x7fe18810ab00 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:56.589 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:55.129+0000 7fe18faa1700 1 -- 172.21.15.40:0/1686734638 >> 172.21.15.40:0/1686734638 conn(0x7fe1881020a0 msgr2=0x7fe188104500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:56.589 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:55.129+0000 7fe18faa1700 1 -- 172.21.15.40:0/1686734638 shutdown_connections 2024-04-21T23:08:56.589 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:55.129+0000 7fe18faa1700 1 -- 172.21.15.40:0/1686734638 wait complete. 2024-04-21T23:08:56.590 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:55.130+0000 7fe18faa1700 1 Processor -- start 2024-04-21T23:08:56.590 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:55.130+0000 7fe18faa1700 1 -- start start 2024-04-21T23:08:56.590 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:55.130+0000 7fe18faa1700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe188108710 0x7fe18819bba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:56.590 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:55.130+0000 7fe18faa1700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe18810c040 con 0x7fe188108710 2024-04-21T23:08:56.590 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:55.130+0000 7fe18d83d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe188108710 0x7fe18819bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:56.590 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:55.130+0000 7fe18d83d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe188108710 0x7fe18819bba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54392/0 (socket says 172.21.15.40:54392) 2024-04-21T23:08:56.590 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:55.130+0000 7fe18d83d700 1 -- 172.21.15.40:0/577975543 learned_addr learned my addr 172.21.15.40:0/577975543 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:08:56.590 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:55.131+0000 7fe18d83d700 1 -- 172.21.15.40:0/577975543 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe18819c0e0 con 0x7fe188108710 2024-04-21T23:08:56.590 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:55.131+0000 7fe18d83d700 1 --2- 172.21.15.40:0/577975543 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe188108710 0x7fe18819bba0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe1840117e0 tx=0x7fe184011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:56.590 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:55.131+0000 7fe17affd700 1 -- 172.21.15.40:0/577975543 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe184003bf0 con 0x7fe188108710 2024-04-21T23:08:56.590 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:55.131+0000 7fe17affd700 1 -- 172.21.15.40:0/577975543 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe18401a910 con 0x7fe188108710 2024-04-21T23:08:56.590 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:55.131+0000 7fe18faa1700 1 -- 172.21.15.40:0/577975543 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe18819c370 con 0x7fe188108710 2024-04-21T23:08:56.590 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:55.131+0000 7fe17affd700 1 -- 172.21.15.40:0/577975543 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe184024970 con 0x7fe188108710 2024-04-21T23:08:56.591 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:55.131+0000 7fe18faa1700 1 -- 172.21.15.40:0/577975543 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe18819c7d0 con 0x7fe188108710 2024-04-21T23:08:56.591 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:55.132+0000 7fe17affd700 1 -- 172.21.15.40:0/577975543 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 12) ==== 46300+0+0 (secure 0 0 0) 0x7fe184022020 con 0x7fe188108710 2024-04-21T23:08:56.591 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:55.132+0000 7fe18faa1700 1 -- 172.21.15.40:0/577975543 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe188063890 con 0x7fe188108710 2024-04-21T23:08:56.591 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:55.132+0000 7fe17affd700 1 --2- 172.21.15.40:0/577975543 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fe174039050 0x7fe17403b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:56.591 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:55.132+0000 7fe17affd700 1 -- 172.21.15.40:0/577975543 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fe18404d520 con 0x7fe188108710 2024-04-21T23:08:56.591 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:55.132+0000 7fe18d03c700 1 --2- 172.21.15.40:0/577975543 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fe174039050 0x7fe17403b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:56.591 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:55.133+0000 7fe18d03c700 1 --2- 172.21.15.40:0/577975543 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fe174039050 0x7fe17403b510 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fe17c006fd0 tx=0x7fe17c006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:56.591 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:55.137+0000 7fe17affd700 1 -- 172.21.15.40:0/577975543 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fe184011e20 con 0x7fe188108710 2024-04-21T23:08:56.591 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:55.364+0000 7fe18faa1700 1 -- 172.21.15.40:0/577975543 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7fe18819f260 con 0x7fe188108710 2024-04-21T23:08:56.591 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:55.364+0000 7fe17affd700 1 -- 172.21.15.40:0/577975543 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7fe184019c70 con 0x7fe188108710 2024-04-21T23:08:56.591 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:55.367+0000 7fe18faa1700 1 -- 172.21.15.40:0/577975543 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fe174039050 msgr2=0x7fe17403b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:56.591 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:55.367+0000 7fe18faa1700 1 --2- 172.21.15.40:0/577975543 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fe174039050 0x7fe17403b510 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fe17c006fd0 tx=0x7fe17c006e40 comp rx=0 tx=0).stop 2024-04-21T23:08:56.592 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:55.367+0000 7fe18faa1700 1 -- 172.21.15.40:0/577975543 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe188108710 msgr2=0x7fe18819bba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:56.592 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:55.367+0000 7fe18faa1700 1 --2- 172.21.15.40:0/577975543 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe188108710 0x7fe18819bba0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe1840117e0 tx=0x7fe184011c10 comp rx=0 tx=0).stop 2024-04-21T23:08:56.592 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:55.368+0000 7fe18faa1700 1 -- 172.21.15.40:0/577975543 shutdown_connections 2024-04-21T23:08:56.592 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:55.368+0000 7fe18faa1700 1 --2- 172.21.15.40:0/577975543 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fe174039050 0x7fe17403b510 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:56.592 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:55.368+0000 7fe18faa1700 1 --2- 172.21.15.40:0/577975543 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe188108710 0x7fe18819bba0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:56.592 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:55.368+0000 7fe18faa1700 1 -- 172.21.15.40:0/577975543 >> 172.21.15.40:0/577975543 conn(0x7fe1881020a0 msgr2=0x7fe188102b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:56.592 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:55.368+0000 7fe18faa1700 1 -- 172.21.15.40:0/577975543 shutdown_connections 2024-04-21T23:08:56.592 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:55.368+0000 7fe18faa1700 1 -- 172.21.15.40:0/577975543 wait complete. 2024-04-21T23:08:56.596 INFO:teuthology.orchestra.run.smithi040.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-21T23:08:56.597 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stdout disabled 2024-04-21T23:08:56.605 INFO:teuthology.orchestra.run.smithi040.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-21T23:08:56.605 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stdout inactive 2024-04-21T23:08:56.605 INFO:teuthology.orchestra.run.smithi040.stdout:firewalld.service is not enabled 2024-04-21T23:08:56.605 INFO:teuthology.orchestra.run.smithi040.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-21T23:08:56.607 INFO:teuthology.orchestra.run.smithi040.stdout:Ceph Dashboard is now available at: 2024-04-21T23:08:56.607 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:08:56.607 INFO:teuthology.orchestra.run.smithi040.stdout: URL: https://smithi040:8443/ 2024-04-21T23:08:56.607 INFO:teuthology.orchestra.run.smithi040.stdout: User: admin 2024-04-21T23:08:56.607 INFO:teuthology.orchestra.run.smithi040.stdout: Password: 0znj9ql2on 2024-04-21T23:08:56.607 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:08:56.608 INFO:teuthology.orchestra.run.smithi040.stdout:Saving cluster configuration to /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/config directory 2024-04-21T23:08:56.608 INFO:teuthology.orchestra.run.smithi040.stdout:Enabling autotune for osd_memory_target 2024-04-21T23:08:58.129 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:57 smithi040 ceph-mon[27984]: mgrmap e13: x(active, since 6s) 2024-04-21T23:08:58.137 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:57.481+0000 7fd410ce4700 1 Processor -- start 2024-04-21T23:08:58.137 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:57.481+0000 7fd410ce4700 1 -- start start 2024-04-21T23:08:58.137 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:57.481+0000 7fd410ce4700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd40c07dec0 0x7fd40c07c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:58.137 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:57.481+0000 7fd410ce4700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd40c07c8f0 con 0x7fd40c07dec0 2024-04-21T23:08:58.137 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:57.481+0000 7fd40a59c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd40c07dec0 0x7fd40c07c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:58.137 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:57.481+0000 7fd40a59c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd40c07dec0 0x7fd40c07c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54398/0 (socket says 172.21.15.40:54398) 2024-04-21T23:08:58.137 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:57.481+0000 7fd40a59c700 1 -- 172.21.15.40:0/4114524030 learned_addr learned my addr 172.21.15.40:0/4114524030 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:08:58.137 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:57.482+0000 7fd40a59c700 1 -- 172.21.15.40:0/4114524030 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd40c07ca70 con 0x7fd40c07dec0 2024-04-21T23:08:58.137 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:57.482+0000 7fd40a59c700 1 --2- 172.21.15.40:0/4114524030 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd40c07dec0 0x7fd40c07c320 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc0098a0 tx=0x7fd3fc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41b44c23c11576c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:58.137 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:57.482+0000 7fd40959a700 1 -- 172.21.15.40:0/4114524030 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd3fc004160 con 0x7fd40c07dec0 2024-04-21T23:08:58.138 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:57.482+0000 7fd40959a700 1 -- 172.21.15.40:0/4114524030 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3fc004300 con 0x7fd40c07dec0 2024-04-21T23:08:58.138 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:57.482+0000 7fd40959a700 1 -- 172.21.15.40:0/4114524030 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd3fc011b20 con 0x7fd40c07dec0 2024-04-21T23:08:58.138 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:57.482+0000 7fd410ce4700 1 -- 172.21.15.40:0/4114524030 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd40c07dec0 msgr2=0x7fd40c07c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:58.138 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:57.482+0000 7fd410ce4700 1 --2- 172.21.15.40:0/4114524030 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd40c07dec0 0x7fd40c07c320 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc0098a0 tx=0x7fd3fc009bb0 comp rx=0 tx=0).stop 2024-04-21T23:08:58.138 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:57.483+0000 7fd410ce4700 1 -- 172.21.15.40:0/4114524030 shutdown_connections 2024-04-21T23:08:58.138 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:57.483+0000 7fd410ce4700 1 --2- 172.21.15.40:0/4114524030 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd40c07dec0 0x7fd40c07c320 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:58.138 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:57.483+0000 7fd410ce4700 1 -- 172.21.15.40:0/4114524030 >> 172.21.15.40:0/4114524030 conn(0x7fd40c1031e0 msgr2=0x7fd40c105620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:58.138 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:57.483+0000 7fd410ce4700 1 -- 172.21.15.40:0/4114524030 shutdown_connections 2024-04-21T23:08:58.138 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:57.483+0000 7fd410ce4700 1 -- 172.21.15.40:0/4114524030 wait complete. 2024-04-21T23:08:58.138 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:57.483+0000 7fd410ce4700 1 Processor -- start 2024-04-21T23:08:58.138 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:57.483+0000 7fd410ce4700 1 -- start start 2024-04-21T23:08:58.138 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:57.484+0000 7fd410ce4700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd40c07dec0 0x7fd40c19ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:58.139 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:57.484+0000 7fd410ce4700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd40c10e280 con 0x7fd40c07dec0 2024-04-21T23:08:58.139 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:57.484+0000 7fd40a59c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd40c07dec0 0x7fd40c19ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:58.139 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:57.484+0000 7fd40a59c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd40c07dec0 0x7fd40c19ddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54400/0 (socket says 172.21.15.40:54400) 2024-04-21T23:08:58.139 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:57.484+0000 7fd40a59c700 1 -- 172.21.15.40:0/215909281 learned_addr learned my addr 172.21.15.40:0/215909281 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:08:58.139 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:57.484+0000 7fd40a59c700 1 -- 172.21.15.40:0/215909281 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd40c19e310 con 0x7fd40c07dec0 2024-04-21T23:08:58.139 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:57.484+0000 7fd40a59c700 1 --2- 172.21.15.40:0/215909281 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd40c07dec0 0x7fd40c19ddd0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc000c00 tx=0x7fd3fc011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:58.139 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:57.484+0000 7fd3fb7fe700 1 -- 172.21.15.40:0/215909281 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd3fc003b60 con 0x7fd40c07dec0 2024-04-21T23:08:58.139 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:57.484+0000 7fd3fb7fe700 1 -- 172.21.15.40:0/215909281 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3fc01a910 con 0x7fd40c07dec0 2024-04-21T23:08:58.139 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:57.484+0000 7fd410ce4700 1 -- 172.21.15.40:0/215909281 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd40c19e5a0 con 0x7fd40c07dec0 2024-04-21T23:08:58.139 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:57.484+0000 7fd3fb7fe700 1 -- 172.21.15.40:0/215909281 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd3fc024930 con 0x7fd40c07dec0 2024-04-21T23:08:58.139 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:57.485+0000 7fd410ce4700 1 -- 172.21.15.40:0/215909281 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd40c19ea00 con 0x7fd40c07dec0 2024-04-21T23:08:58.139 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:57.485+0000 7fd3fb7fe700 1 -- 172.21.15.40:0/215909281 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7fd3fc022020 con 0x7fd40c07dec0 2024-04-21T23:08:58.139 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:57.485+0000 7fd410ce4700 1 -- 172.21.15.40:0/215909281 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd40c063890 con 0x7fd40c07dec0 2024-04-21T23:08:58.139 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:57.486+0000 7fd3fb7fe700 1 --2- 172.21.15.40:0/215909281 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fd3f40391a0 0x7fd3f403b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:58.140 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:57.486+0000 7fd3fb7fe700 1 -- 172.21.15.40:0/215909281 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fd3fc04d3a0 con 0x7fd40c07dec0 2024-04-21T23:08:58.140 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:57.486+0000 7fd409d9b700 1 --2- 172.21.15.40:0/215909281 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fd3f40391a0 0x7fd3f403b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:58.140 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:57.486+0000 7fd409d9b700 1 --2- 172.21.15.40:0/215909281 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fd3f40391a0 0x7fd3f403b660 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd400006fd0 tx=0x7fd400006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:58.140 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:57.491+0000 7fd3fb7fe700 1 -- 172.21.15.40:0/215909281 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fd3fc02a2a0 con 0x7fd40c07dec0 2024-04-21T23:08:58.140 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:57.699+0000 7fd410ce4700 1 -- 172.21.15.40:0/215909281 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7fd40c1a13d0 con 0x7fd40c07dec0 2024-04-21T23:08:58.140 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:57.699+0000 7fd3fb7fe700 1 -- 172.21.15.40:0/215909281 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7fd3fc02d3f0 con 0x7fd40c07dec0 2024-04-21T23:08:58.140 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:57.701+0000 7fd410ce4700 1 -- 172.21.15.40:0/215909281 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fd3f40391a0 msgr2=0x7fd3f403b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:58.140 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:57.701+0000 7fd410ce4700 1 --2- 172.21.15.40:0/215909281 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fd3f40391a0 0x7fd3f403b660 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd400006fd0 tx=0x7fd400006e40 comp rx=0 tx=0).stop 2024-04-21T23:08:58.140 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:57.701+0000 7fd410ce4700 1 -- 172.21.15.40:0/215909281 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd40c07dec0 msgr2=0x7fd40c19ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:58.140 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:57.701+0000 7fd410ce4700 1 --2- 172.21.15.40:0/215909281 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd40c07dec0 0x7fd40c19ddd0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc000c00 tx=0x7fd3fc011c10 comp rx=0 tx=0).stop 2024-04-21T23:08:58.140 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:57.701+0000 7fd410ce4700 1 -- 172.21.15.40:0/215909281 shutdown_connections 2024-04-21T23:08:58.141 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:57.701+0000 7fd410ce4700 1 --2- 172.21.15.40:0/215909281 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fd3f40391a0 0x7fd3f403b660 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:58.141 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:57.701+0000 7fd410ce4700 1 --2- 172.21.15.40:0/215909281 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd40c07dec0 0x7fd40c19ddd0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:58.141 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:57.701+0000 7fd410ce4700 1 -- 172.21.15.40:0/215909281 >> 172.21.15.40:0/215909281 conn(0x7fd40c1031e0 msgr2=0x7fd40c103d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:58.141 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:57.701+0000 7fd410ce4700 1 -- 172.21.15.40:0/215909281 shutdown_connections 2024-04-21T23:08:58.141 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:57.701+0000 7fd410ce4700 1 -- 172.21.15.40:0/215909281 wait complete. 2024-04-21T23:08:59.485 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:58.584+0000 7f590078e700 1 Processor -- start 2024-04-21T23:08:59.485 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:58.584+0000 7f590078e700 1 -- start start 2024-04-21T23:08:59.486 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:58.584+0000 7f590078e700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58f8107c40 0x7f58f8108060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:59.486 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:58.584+0000 7f590078e700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f58f8108630 con 0x7f58f8107c40 2024-04-21T23:08:59.486 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:58.585+0000 7f58fe52a700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58f8107c40 0x7f58f8108060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:59.486 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:58.585+0000 7f58fe52a700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58f8107c40 0x7f58f8108060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54416/0 (socket says 172.21.15.40:54416) 2024-04-21T23:08:59.486 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:58.585+0000 7f58fe52a700 1 -- 172.21.15.40:0/2257575385 learned_addr learned my addr 172.21.15.40:0/2257575385 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:08:59.486 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:58.585+0000 7f58fe52a700 1 -- 172.21.15.40:0/2257575385 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58f8108e40 con 0x7f58f8107c40 2024-04-21T23:08:59.486 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:58.585+0000 7f58fe52a700 1 --2- 172.21.15.40:0/2257575385 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58f8107c40 0x7f58f8108060 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f58f4009e80 tx=0x7f58f400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc7a1f833970d928 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:59.486 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:58.585+0000 7f58fd528700 1 -- 172.21.15.40:0/2257575385 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f58f4017070 con 0x7f58f8107c40 2024-04-21T23:08:59.486 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:58.585+0000 7f58fd528700 1 -- 172.21.15.40:0/2257575385 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58f400b8c0 con 0x7f58f8107c40 2024-04-21T23:08:59.486 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:58.585+0000 7f58fd528700 1 -- 172.21.15.40:0/2257575385 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f58f40045c0 con 0x7f58f8107c40 2024-04-21T23:08:59.486 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:58.586+0000 7f590078e700 1 -- 172.21.15.40:0/2257575385 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58f8107c40 msgr2=0x7f58f8108060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:59.486 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:58.586+0000 7f590078e700 1 --2- 172.21.15.40:0/2257575385 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58f8107c40 0x7f58f8108060 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f58f4009e80 tx=0x7f58f400b330 comp rx=0 tx=0).stop 2024-04-21T23:08:59.487 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:58.586+0000 7f590078e700 1 -- 172.21.15.40:0/2257575385 shutdown_connections 2024-04-21T23:08:59.487 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:58.586+0000 7f590078e700 1 --2- 172.21.15.40:0/2257575385 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58f8107c40 0x7f58f8108060 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:59.487 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:58.586+0000 7f590078e700 1 -- 172.21.15.40:0/2257575385 >> 172.21.15.40:0/2257575385 conn(0x7f58f81031e0 msgr2=0x7f58f8105620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:59.487 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:58.586+0000 7f590078e700 1 -- 172.21.15.40:0/2257575385 shutdown_connections 2024-04-21T23:08:59.487 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:58.586+0000 7f590078e700 1 -- 172.21.15.40:0/2257575385 wait complete. 2024-04-21T23:08:59.487 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:58.586+0000 7f590078e700 1 Processor -- start 2024-04-21T23:08:59.487 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:58.587+0000 7f590078e700 1 -- start start 2024-04-21T23:08:59.487 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:58.587+0000 7f590078e700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58f8107c40 0x7f58f819bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:59.487 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:58.587+0000 7f590078e700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f58f810c1f0 con 0x7f58f8107c40 2024-04-21T23:08:59.487 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:58.587+0000 7f58fe52a700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58f8107c40 0x7f58f819bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:59.487 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:58.587+0000 7f58fe52a700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58f8107c40 0x7f58f819bc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54432/0 (socket says 172.21.15.40:54432) 2024-04-21T23:08:59.487 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:58.587+0000 7f58fe52a700 1 -- 172.21.15.40:0/1875320010 learned_addr learned my addr 172.21.15.40:0/1875320010 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:08:59.487 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:58.587+0000 7f58fe52a700 1 -- 172.21.15.40:0/1875320010 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58f819c1a0 con 0x7f58f8107c40 2024-04-21T23:08:59.487 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:58.587+0000 7f58fe52a700 1 --2- 172.21.15.40:0/1875320010 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58f8107c40 0x7f58f819bc60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f58f4009530 tx=0x7f58f4003d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:59.487 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:58.587+0000 7f58ef7fe700 1 -- 172.21.15.40:0/1875320010 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f58f4020020 con 0x7f58f8107c40 2024-04-21T23:08:59.488 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:58.587+0000 7f58ef7fe700 1 -- 172.21.15.40:0/1875320010 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58f400bba0 con 0x7f58f8107c40 2024-04-21T23:08:59.488 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:58.587+0000 7f590078e700 1 -- 172.21.15.40:0/1875320010 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58f819c430 con 0x7f58f8107c40 2024-04-21T23:08:59.488 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:58.587+0000 7f58ef7fe700 1 -- 172.21.15.40:0/1875320010 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f58f4017040 con 0x7f58f8107c40 2024-04-21T23:08:59.488 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:58.587+0000 7f590078e700 1 -- 172.21.15.40:0/1875320010 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58f819f120 con 0x7f58f8107c40 2024-04-21T23:08:59.488 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:58.588+0000 7f58ef7fe700 1 -- 172.21.15.40:0/1875320010 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f58f4023020 con 0x7f58f8107c40 2024-04-21T23:08:59.488 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:58.588+0000 7f58ef7fe700 1 --2- 172.21.15.40:0/1875320010 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f58e40391a0 0x7f58e403b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:59.488 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:58.588+0000 7f58ef7fe700 1 -- 172.21.15.40:0/1875320010 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f58f400f050 con 0x7f58f8107c40 2024-04-21T23:08:59.488 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:58.588+0000 7f590078e700 1 -- 172.21.15.40:0/1875320010 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58f8063890 con 0x7f58f8107c40 2024-04-21T23:08:59.488 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:58.589+0000 7f58fdd29700 1 --2- 172.21.15.40:0/1875320010 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f58e40391a0 0x7f58e403b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:59.488 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:58.589+0000 7f58fdd29700 1 --2- 172.21.15.40:0/1875320010 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f58e40391a0 0x7f58e403b660 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f58e8006fd0 tx=0x7f58e8006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:59.488 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:58.594+0000 7f58ef7fe700 1 -- 172.21.15.40:0/1875320010 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f58f402a020 con 0x7f58f8107c40 2024-04-21T23:08:59.488 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:58.918+0000 7f590078e700 1 -- 172.21.15.40:0/1875320010 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f58f819f7c0 con 0x7f58f8107c40 2024-04-21T23:08:59.488 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:58.921+0000 7f58ef7fe700 1 -- 172.21.15.40:0/1875320010 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v27)=0 set mgr/dashboard/cluster/status v27) ==== 153+0+0 (secure 0 0 0) 0x7f58f401b530 con 0x7f58f8107c40 2024-04-21T23:08:59.488 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:58.923+0000 7f590078e700 1 -- 172.21.15.40:0/1875320010 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f58e40391a0 msgr2=0x7f58e403b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:59.489 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:58.923+0000 7f590078e700 1 --2- 172.21.15.40:0/1875320010 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f58e40391a0 0x7f58e403b660 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f58e8006fd0 tx=0x7f58e8006e40 comp rx=0 tx=0).stop 2024-04-21T23:08:59.489 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:58.923+0000 7f590078e700 1 -- 172.21.15.40:0/1875320010 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58f8107c40 msgr2=0x7f58f819bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:59.489 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:58.923+0000 7f590078e700 1 --2- 172.21.15.40:0/1875320010 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58f8107c40 0x7f58f819bc60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f58f4009530 tx=0x7f58f4003d00 comp rx=0 tx=0).stop 2024-04-21T23:08:59.489 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:58.923+0000 7f590078e700 1 -- 172.21.15.40:0/1875320010 shutdown_connections 2024-04-21T23:08:59.489 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:58.923+0000 7f590078e700 1 --2- 172.21.15.40:0/1875320010 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f58e40391a0 0x7f58e403b660 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:59.489 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:58.923+0000 7f590078e700 1 --2- 172.21.15.40:0/1875320010 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58f8107c40 0x7f58f819bc60 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:59.489 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:58.923+0000 7f590078e700 1 -- 172.21.15.40:0/1875320010 >> 172.21.15.40:0/1875320010 conn(0x7f58f81031e0 msgr2=0x7f58f8103d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:59.489 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:58.923+0000 7f590078e700 1 -- 172.21.15.40:0/1875320010 shutdown_connections 2024-04-21T23:08:59.489 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-21T23:08:58.923+0000 7f590078e700 1 -- 172.21.15.40:0/1875320010 wait complete. 2024-04-21T23:08:59.489 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-21T23:08:59.489 INFO:teuthology.orchestra.run.smithi040.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-21T23:08:59.489 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:08:59.489 INFO:teuthology.orchestra.run.smithi040.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-21T23:08:59.489 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:08:59.490 INFO:teuthology.orchestra.run.smithi040.stdout:Or, if you are only running a single cluster on this host: 2024-04-21T23:08:59.490 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:08:59.490 INFO:teuthology.orchestra.run.smithi040.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-21T23:08:59.490 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:08:59.490 INFO:teuthology.orchestra.run.smithi040.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-21T23:08:59.490 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:08:59.490 INFO:teuthology.orchestra.run.smithi040.stdout: ceph telemetry on 2024-04-21T23:08:59.490 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:08:59.490 INFO:teuthology.orchestra.run.smithi040.stdout:For more information see: 2024-04-21T23:08:59.490 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:08:59.490 INFO:teuthology.orchestra.run.smithi040.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-21T23:08:59.490 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:08:59.490 INFO:teuthology.orchestra.run.smithi040.stdout:Bootstrap complete. 2024-04-21T23:08:59.521 INFO:tasks.cephadm:Fetching config... 2024-04-21T23:08:59.521 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-21T23:08:59.521 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-21T23:08:59.537 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-21T23:08:59.537 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-21T23:08:59.537 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-21T23:08:59.593 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-21T23:08:59.594 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-21T23:08:59.594 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.a/keyring of=/dev/stdout 2024-04-21T23:08:59.660 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-21T23:08:59.660 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-21T23:08:59.661 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-21T23:08:59.715 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-21T23:08:59.716 DEBUG:teuthology.orchestra.run.smithi040:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/WodsMK9knduyuDKb9hLCNvwDfLMFRNdb5KKrSmT2QpvbRpl7fzJRrOchfTmuzEAGMqhQU5waMtgcOLYLfnIFUMxQG0Lgv/0djttTzoz6c95cjntw0u7PYNaW3wiPIWqv9TuIqcJLg9UM9JHClUcWkvnNSNkhE7S403S/IAE4OCapAdWuE+tjJuXQgHWqK9vNbiJzo1CRKbjpX4BxQxrdHq3+91fOSmQm3j6vfCzcch/aCefeyo4kMYxcgHpgbVmmQ2hbzcyGfYrF6zsUHdAS5GWYUyQ92cVHIMnUmGLYmjXMXMKVjOZAFqqkSbgZZmGuyu/Cvbf4uLVpTT72ZjBIT7vsA6e15pj3A+zMTLPa/wEdzXjAzVY+GCtjqt1ErgkeK+KQQ/fZSxOzChZG5+9X4b+gcSnc5xMpRa6e6v6N5ywpQK8XcDLFs8CKL4ES2NsQ/3X66UZ8sIYlAivcZyXJojaf4DsX3OuHmbxr+8ZpP24rPuIRzQZDmd3/B9azHk0= ceph-d814e4dc-0033-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-21T23:08:59.802 INFO:teuthology.orchestra.run.smithi040.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/WodsMK9knduyuDKb9hLCNvwDfLMFRNdb5KKrSmT2QpvbRpl7fzJRrOchfTmuzEAGMqhQU5waMtgcOLYLfnIFUMxQG0Lgv/0djttTzoz6c95cjntw0u7PYNaW3wiPIWqv9TuIqcJLg9UM9JHClUcWkvnNSNkhE7S403S/IAE4OCapAdWuE+tjJuXQgHWqK9vNbiJzo1CRKbjpX4BxQxrdHq3+91fOSmQm3j6vfCzcch/aCefeyo4kMYxcgHpgbVmmQ2hbzcyGfYrF6zsUHdAS5GWYUyQ92cVHIMnUmGLYmjXMXMKVjOZAFqqkSbgZZmGuyu/Cvbf4uLVpTT72ZjBIT7vsA6e15pj3A+zMTLPa/wEdzXjAzVY+GCtjqt1ErgkeK+KQQ/fZSxOzChZG5+9X4b+gcSnc5xMpRa6e6v6N5ywpQK8XcDLFs8CKL4ES2NsQ/3X66UZ8sIYlAivcZyXJojaf4DsX3OuHmbxr+8ZpP24rPuIRzQZDmd3/B9azHk0= ceph-d814e4dc-0033-11ef-bc93-c7b262605968 2024-04-21T23:08:59.816 DEBUG:teuthology.orchestra.run.smithi111:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/WodsMK9knduyuDKb9hLCNvwDfLMFRNdb5KKrSmT2QpvbRpl7fzJRrOchfTmuzEAGMqhQU5waMtgcOLYLfnIFUMxQG0Lgv/0djttTzoz6c95cjntw0u7PYNaW3wiPIWqv9TuIqcJLg9UM9JHClUcWkvnNSNkhE7S403S/IAE4OCapAdWuE+tjJuXQgHWqK9vNbiJzo1CRKbjpX4BxQxrdHq3+91fOSmQm3j6vfCzcch/aCefeyo4kMYxcgHpgbVmmQ2hbzcyGfYrF6zsUHdAS5GWYUyQ92cVHIMnUmGLYmjXMXMKVjOZAFqqkSbgZZmGuyu/Cvbf4uLVpTT72ZjBIT7vsA6e15pj3A+zMTLPa/wEdzXjAzVY+GCtjqt1ErgkeK+KQQ/fZSxOzChZG5+9X4b+gcSnc5xMpRa6e6v6N5ywpQK8XcDLFs8CKL4ES2NsQ/3X66UZ8sIYlAivcZyXJojaf4DsX3OuHmbxr+8ZpP24rPuIRzQZDmd3/B9azHk0= ceph-d814e4dc-0033-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-21T23:08:59.855 INFO:teuthology.orchestra.run.smithi111.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/WodsMK9knduyuDKb9hLCNvwDfLMFRNdb5KKrSmT2QpvbRpl7fzJRrOchfTmuzEAGMqhQU5waMtgcOLYLfnIFUMxQG0Lgv/0djttTzoz6c95cjntw0u7PYNaW3wiPIWqv9TuIqcJLg9UM9JHClUcWkvnNSNkhE7S403S/IAE4OCapAdWuE+tjJuXQgHWqK9vNbiJzo1CRKbjpX4BxQxrdHq3+91fOSmQm3j6vfCzcch/aCefeyo4kMYxcgHpgbVmmQ2hbzcyGfYrF6zsUHdAS5GWYUyQ92cVHIMnUmGLYmjXMXMKVjOZAFqqkSbgZZmGuyu/Cvbf4uLVpTT72ZjBIT7vsA6e15pj3A+zMTLPa/wEdzXjAzVY+GCtjqt1ErgkeK+KQQ/fZSxOzChZG5+9X4b+gcSnc5xMpRa6e6v6N5ywpQK8XcDLFs8CKL4ES2NsQ/3X66UZ8sIYlAivcZyXJojaf4DsX3OuHmbxr+8ZpP24rPuIRzQZDmd3/B9azHk0= ceph-d814e4dc-0033-11ef-bc93-c7b262605968 2024-04-21T23:08:59.867 DEBUG:teuthology.orchestra.run.smithi174:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/WodsMK9knduyuDKb9hLCNvwDfLMFRNdb5KKrSmT2QpvbRpl7fzJRrOchfTmuzEAGMqhQU5waMtgcOLYLfnIFUMxQG0Lgv/0djttTzoz6c95cjntw0u7PYNaW3wiPIWqv9TuIqcJLg9UM9JHClUcWkvnNSNkhE7S403S/IAE4OCapAdWuE+tjJuXQgHWqK9vNbiJzo1CRKbjpX4BxQxrdHq3+91fOSmQm3j6vfCzcch/aCefeyo4kMYxcgHpgbVmmQ2hbzcyGfYrF6zsUHdAS5GWYUyQ92cVHIMnUmGLYmjXMXMKVjOZAFqqkSbgZZmGuyu/Cvbf4uLVpTT72ZjBIT7vsA6e15pj3A+zMTLPa/wEdzXjAzVY+GCtjqt1ErgkeK+KQQ/fZSxOzChZG5+9X4b+gcSnc5xMpRa6e6v6N5ywpQK8XcDLFs8CKL4ES2NsQ/3X66UZ8sIYlAivcZyXJojaf4DsX3OuHmbxr+8ZpP24rPuIRzQZDmd3/B9azHk0= ceph-d814e4dc-0033-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-21T23:08:59.906 INFO:teuthology.orchestra.run.smithi174.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/WodsMK9knduyuDKb9hLCNvwDfLMFRNdb5KKrSmT2QpvbRpl7fzJRrOchfTmuzEAGMqhQU5waMtgcOLYLfnIFUMxQG0Lgv/0djttTzoz6c95cjntw0u7PYNaW3wiPIWqv9TuIqcJLg9UM9JHClUcWkvnNSNkhE7S403S/IAE4OCapAdWuE+tjJuXQgHWqK9vNbiJzo1CRKbjpX4BxQxrdHq3+91fOSmQm3j6vfCzcch/aCefeyo4kMYxcgHpgbVmmQ2hbzcyGfYrF6zsUHdAS5GWYUyQ92cVHIMnUmGLYmjXMXMKVjOZAFqqkSbgZZmGuyu/Cvbf4uLVpTT72ZjBIT7vsA6e15pj3A+zMTLPa/wEdzXjAzVY+GCtjqt1ErgkeK+KQQ/fZSxOzChZG5+9X4b+gcSnc5xMpRa6e6v6N5ywpQK8XcDLFs8CKL4ES2NsQ/3X66UZ8sIYlAivcZyXJojaf4DsX3OuHmbxr+8ZpP24rPuIRzQZDmd3/B9azHk0= ceph-d814e4dc-0033-11ef-bc93-c7b262605968 2024-04-21T23:08:59.917 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-21T23:09:00.064 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:08:59 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/1875320010' entity='client.admin' 2024-04-21T23:09:00.167 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:09:02.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:02.417+0000 7f8e5c1a7700 1 -- 172.21.15.40:0/1228820276 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e540fff60 msgr2=0x7f8e54100380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:02.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:02.417+0000 7f8e5c1a7700 1 --2- 172.21.15.40:0/1228820276 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e540fff60 0x7f8e54100380 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f8e50009910 tx=0x7f8e50009c20 comp rx=0 tx=0).stop 2024-04-21T23:09:02.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:02.417+0000 7f8e5c1a7700 1 -- 172.21.15.40:0/1228820276 shutdown_connections 2024-04-21T23:09:02.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:02.417+0000 7f8e5c1a7700 1 --2- 172.21.15.40:0/1228820276 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e540fff60 0x7f8e54100380 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:02.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:02.417+0000 7f8e5c1a7700 1 -- 172.21.15.40:0/1228820276 >> 172.21.15.40:0/1228820276 conn(0x7f8e540fb8e0 msgr2=0x7f8e540fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:02.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:02.417+0000 7f8e5c1a7700 1 -- 172.21.15.40:0/1228820276 shutdown_connections 2024-04-21T23:09:02.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:02.417+0000 7f8e5c1a7700 1 -- 172.21.15.40:0/1228820276 wait complete. 2024-04-21T23:09:02.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:02.418+0000 7f8e5c1a7700 1 Processor -- start 2024-04-21T23:09:02.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:02.418+0000 7f8e5c1a7700 1 -- start start 2024-04-21T23:09:02.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:02.418+0000 7f8e5c1a7700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e540fff60 0x7f8e54199590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:02.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:02.418+0000 7f8e5c1a7700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8e54109f40 con 0x7f8e540fff60 2024-04-21T23:09:02.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:02.418+0000 7f8e59f43700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e540fff60 0x7f8e54199590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:02.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:02.418+0000 7f8e59f43700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e540fff60 0x7f8e54199590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56920/0 (socket says 172.21.15.40:56920) 2024-04-21T23:09:02.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:02.418+0000 7f8e59f43700 1 -- 172.21.15.40:0/1693812801 learned_addr learned my addr 172.21.15.40:0/1693812801 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:09:02.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:02.418+0000 7f8e59f43700 1 -- 172.21.15.40:0/1693812801 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e54199ad0 con 0x7f8e540fff60 2024-04-21T23:09:02.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:02.418+0000 7f8e59f43700 1 --2- 172.21.15.40:0/1693812801 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e540fff60 0x7f8e54199590 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8e50005950 tx=0x7f8e50005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:02.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:02.419+0000 7f8e4affd700 1 -- 172.21.15.40:0/1693812801 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8e50026070 con 0x7f8e540fff60 2024-04-21T23:09:02.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:02.419+0000 7f8e4affd700 1 -- 172.21.15.40:0/1693812801 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e5000bc30 con 0x7f8e540fff60 2024-04-21T23:09:02.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:02.419+0000 7f8e5c1a7700 1 -- 172.21.15.40:0/1693812801 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e54199d60 con 0x7f8e540fff60 2024-04-21T23:09:02.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:02.419+0000 7f8e4affd700 1 -- 172.21.15.40:0/1693812801 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8e50003b50 con 0x7f8e540fff60 2024-04-21T23:09:02.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:02.419+0000 7f8e5c1a7700 1 -- 172.21.15.40:0/1693812801 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e5419ca50 con 0x7f8e540fff60 2024-04-21T23:09:02.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:02.419+0000 7f8e4affd700 1 -- 172.21.15.40:0/1693812801 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f8e50029070 con 0x7f8e540fff60 2024-04-21T23:09:02.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:02.420+0000 7f8e5c1a7700 1 -- 172.21.15.40:0/1693812801 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e5410a6c0 con 0x7f8e540fff60 2024-04-21T23:09:02.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:02.420+0000 7f8e4affd700 1 --2- 172.21.15.40:0/1693812801 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f8e40039150 0x7f8e4003b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:02.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:02.420+0000 7f8e4affd700 1 -- 172.21.15.40:0/1693812801 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f8e5001e070 con 0x7f8e540fff60 2024-04-21T23:09:02.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:02.420+0000 7f8e59742700 1 --2- 172.21.15.40:0/1693812801 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f8e40039150 0x7f8e4003b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:02.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:02.420+0000 7f8e59742700 1 --2- 172.21.15.40:0/1693812801 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f8e40039150 0x7f8e4003b610 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f8e44006fd0 tx=0x7f8e44006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:02.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:02.425+0000 7f8e4affd700 1 -- 172.21.15.40:0/1693812801 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f8e50013070 con 0x7f8e540fff60 2024-04-21T23:09:02.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:02.628+0000 7f8e5c1a7700 1 -- 172.21.15.40:0/1693812801 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7f8e5402dc40 con 0x7f8e540fff60 2024-04-21T23:09:02.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:02.631+0000 7f8e4affd700 1 -- 172.21.15.40:0/1693812801 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7f8e50030020 con 0x7f8e540fff60 2024-04-21T23:09:02.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:02.633+0000 7f8e5c1a7700 1 -- 172.21.15.40:0/1693812801 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f8e40039150 msgr2=0x7f8e4003b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:02.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:02.633+0000 7f8e5c1a7700 1 --2- 172.21.15.40:0/1693812801 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f8e40039150 0x7f8e4003b610 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f8e44006fd0 tx=0x7f8e44006e40 comp rx=0 tx=0).stop 2024-04-21T23:09:02.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:02.633+0000 7f8e5c1a7700 1 -- 172.21.15.40:0/1693812801 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e540fff60 msgr2=0x7f8e54199590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:02.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:02.633+0000 7f8e5c1a7700 1 --2- 172.21.15.40:0/1693812801 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e540fff60 0x7f8e54199590 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8e50005950 tx=0x7f8e50005030 comp rx=0 tx=0).stop 2024-04-21T23:09:02.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:02.633+0000 7f8e5c1a7700 1 -- 172.21.15.40:0/1693812801 shutdown_connections 2024-04-21T23:09:02.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:02.633+0000 7f8e5c1a7700 1 --2- 172.21.15.40:0/1693812801 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f8e40039150 0x7f8e4003b610 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:02.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:02.633+0000 7f8e5c1a7700 1 --2- 172.21.15.40:0/1693812801 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e540fff60 0x7f8e54199590 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:02.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:02.633+0000 7f8e5c1a7700 1 -- 172.21.15.40:0/1693812801 >> 172.21.15.40:0/1693812801 conn(0x7f8e540fb8e0 msgr2=0x7f8e540fc360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:02.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:02.633+0000 7f8e5c1a7700 1 -- 172.21.15.40:0/1693812801 shutdown_connections 2024-04-21T23:09:02.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:02.633+0000 7f8e5c1a7700 1 -- 172.21.15.40:0/1693812801 wait complete. 2024-04-21T23:09:02.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:02 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:02.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:02 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:02.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:02 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi040", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:09:02.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:02 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:02.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:02 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:09:02.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:02 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:03.270 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-21T23:09:03.271 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-21T23:09:03.524 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:09:03.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:03 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/1693812801' entity='client.admin' 2024-04-21T23:09:03.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:03 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:09:03.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:03 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:04.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:04.980+0000 7f783bdf5700 1 -- 172.21.15.40:0/4243354444 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78341033a0 msgr2=0x7f78341037c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:04.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:04.980+0000 7f783bdf5700 1 --2- 172.21.15.40:0/4243354444 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78341033a0 0x7f78341037c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7824009910 tx=0x7f7824009c20 comp rx=0 tx=0).stop 2024-04-21T23:09:04.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:04.980+0000 7f783bdf5700 1 -- 172.21.15.40:0/4243354444 shutdown_connections 2024-04-21T23:09:04.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:04.980+0000 7f783bdf5700 1 --2- 172.21.15.40:0/4243354444 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78341033a0 0x7f78341037c0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:04.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:04.980+0000 7f783bdf5700 1 -- 172.21.15.40:0/4243354444 >> 172.21.15.40:0/4243354444 conn(0x7f78340fd280 msgr2=0x7f78340ff6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:04.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:04.980+0000 7f783bdf5700 1 -- 172.21.15.40:0/4243354444 shutdown_connections 2024-04-21T23:09:04.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:04.980+0000 7f783bdf5700 1 -- 172.21.15.40:0/4243354444 wait complete. 2024-04-21T23:09:04.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:04.980+0000 7f783bdf5700 1 Processor -- start 2024-04-21T23:09:04.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:04.980+0000 7f783bdf5700 1 -- start start 2024-04-21T23:09:04.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:04.981+0000 7f783bdf5700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78341033a0 0x7f78341974e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:04.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:04.981+0000 7f783bdf5700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7834107e60 con 0x7f78341033a0 2024-04-21T23:09:04.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:04.981+0000 7f7839b91700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78341033a0 0x7f78341974e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:04.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:04.981+0000 7f7839b91700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78341033a0 0x7f78341974e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56950/0 (socket says 172.21.15.40:56950) 2024-04-21T23:09:04.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:04.981+0000 7f7839b91700 1 -- 172.21.15.40:0/1148232748 learned_addr learned my addr 172.21.15.40:0/1148232748 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:09:04.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:04.981+0000 7f7839b91700 1 -- 172.21.15.40:0/1148232748 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7834197a20 con 0x7f78341033a0 2024-04-21T23:09:04.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:04.981+0000 7f7839b91700 1 --2- 172.21.15.40:0/1148232748 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78341033a0 0x7f78341974e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f7824006010 tx=0x7f7824005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:04.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:04.981+0000 7f782affd700 1 -- 172.21.15.40:0/1148232748 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7824027070 con 0x7f78341033a0 2024-04-21T23:09:04.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:04.981+0000 7f783bdf5700 1 -- 172.21.15.40:0/1148232748 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7834197cb0 con 0x7f78341033a0 2024-04-21T23:09:04.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:04.981+0000 7f782affd700 1 -- 172.21.15.40:0/1148232748 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f782400be40 con 0x7f78341033a0 2024-04-21T23:09:04.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:04.981+0000 7f782affd700 1 -- 172.21.15.40:0/1148232748 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f782400f460 con 0x7f78341033a0 2024-04-21T23:09:04.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:04.981+0000 7f783bdf5700 1 -- 172.21.15.40:0/1148232748 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f783419a9a0 con 0x7f78341033a0 2024-04-21T23:09:04.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:04.982+0000 7f782affd700 1 -- 172.21.15.40:0/1148232748 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f782402a070 con 0x7f78341033a0 2024-04-21T23:09:04.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:04.982+0000 7f783bdf5700 1 -- 172.21.15.40:0/1148232748 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78341085e0 con 0x7f78341033a0 2024-04-21T23:09:04.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:04.982+0000 7f782affd700 1 --2- 172.21.15.40:0/1148232748 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f78200390f0 0x7f782003b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:04.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:04.983+0000 7f782affd700 1 -- 172.21.15.40:0/1148232748 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f782401e030 con 0x7f78341033a0 2024-04-21T23:09:04.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:04.983+0000 7f7839390700 1 --2- 172.21.15.40:0/1148232748 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f78200390f0 0x7f782003b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:04.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:04.983+0000 7f7839390700 1 --2- 172.21.15.40:0/1148232748 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f78200390f0 0x7f782003b5b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f7830006fd0 tx=0x7f7830006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:04.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:04.988+0000 7f782affd700 1 -- 172.21.15.40:0/1148232748 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f7824013070 con 0x7f78341033a0 2024-04-21T23:09:05.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:05.203+0000 7f783bdf5700 1 -- 172.21.15.40:0/1148232748 --> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f7834062670 con 0x7f78200390f0 2024-04-21T23:09:05.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:05.206+0000 7f782affd700 1 -- 172.21.15.40:0/1148232748 <== mgr.14148 v2:172.21.15.40:6800/2500760788 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f7834062670 con 0x7f78200390f0 2024-04-21T23:09:05.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:05.208+0000 7f783bdf5700 1 -- 172.21.15.40:0/1148232748 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f78200390f0 msgr2=0x7f782003b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:05.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:05.208+0000 7f783bdf5700 1 --2- 172.21.15.40:0/1148232748 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f78200390f0 0x7f782003b5b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f7830006fd0 tx=0x7f7830006e40 comp rx=0 tx=0).stop 2024-04-21T23:09:05.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:05.208+0000 7f783bdf5700 1 -- 172.21.15.40:0/1148232748 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78341033a0 msgr2=0x7f78341974e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:05.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:05.208+0000 7f783bdf5700 1 --2- 172.21.15.40:0/1148232748 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78341033a0 0x7f78341974e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f7824006010 tx=0x7f7824005050 comp rx=0 tx=0).stop 2024-04-21T23:09:05.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:05.209+0000 7f783bdf5700 1 -- 172.21.15.40:0/1148232748 shutdown_connections 2024-04-21T23:09:05.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:05.209+0000 7f783bdf5700 1 --2- 172.21.15.40:0/1148232748 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f78200390f0 0x7f782003b5b0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:05.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:05.209+0000 7f783bdf5700 1 --2- 172.21.15.40:0/1148232748 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78341033a0 0x7f78341974e0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:05.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:05.209+0000 7f783bdf5700 1 -- 172.21.15.40:0/1148232748 >> 172.21.15.40:0/1148232748 conn(0x7f78340fd280 msgr2=0x7f78340fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:05.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:05.209+0000 7f783bdf5700 1 -- 172.21.15.40:0/1148232748 shutdown_connections 2024-04-21T23:09:05.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:05.209+0000 7f783bdf5700 1 -- 172.21.15.40:0/1148232748 wait complete. 2024-04-21T23:09:05.976 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi111 2024-04-21T23:09:05.976 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-04-21T23:09:05.976 DEBUG:teuthology.orchestra.run.smithi111:> dd of=/etc/ceph/ceph.conf 2024-04-21T23:09:05.995 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-04-21T23:09:05.996 DEBUG:teuthology.orchestra.run.smithi111:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-21T23:09:06.053 INFO:tasks.cephadm:Adding host smithi111 to orchestrator... 2024-04-21T23:09:06.054 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph orch host add smithi111 2024-04-21T23:09:06.293 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:09:06.523 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:06 smithi040 ceph-mon[27984]: from='client.14172 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:09:06.524 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:06 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:06.524 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:06 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:09:06.524 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:06 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:09:06.524 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:06 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:09:06.524 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:06 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:06.524 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:06 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:06.524 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:06 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:07.454 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:07 smithi040 ceph-mon[27984]: Updating smithi040:/etc/ceph/ceph.conf 2024-04-21T23:09:07.454 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:07 smithi040 ceph-mon[27984]: Updating smithi040:/var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-21T23:09:07.454 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:07 smithi040 ceph-mon[27984]: Updating smithi040:/etc/ceph/ceph.client.admin.keyring 2024-04-21T23:09:07.454 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:07 smithi040 ceph-mon[27984]: Updating smithi040:/var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T23:09:07.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:07.728+0000 7f6492422700 1 -- 172.21.15.40:0/3822345773 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f648c074550 msgr2=0x7f648c074930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:07.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:07.728+0000 7f6492422700 1 --2- 172.21.15.40:0/3822345773 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f648c074550 0x7f648c074930 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f647c009910 tx=0x7f647c009c20 comp rx=0 tx=0).stop 2024-04-21T23:09:07.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:07.729+0000 7f6492422700 1 -- 172.21.15.40:0/3822345773 shutdown_connections 2024-04-21T23:09:07.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:07.729+0000 7f6492422700 1 --2- 172.21.15.40:0/3822345773 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f648c074550 0x7f648c074930 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:07.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:07.729+0000 7f6492422700 1 -- 172.21.15.40:0/3822345773 >> 172.21.15.40:0/3822345773 conn(0x7f648c0fd770 msgr2=0x7f648c0ffb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:07.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:07.729+0000 7f6492422700 1 -- 172.21.15.40:0/3822345773 shutdown_connections 2024-04-21T23:09:07.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:07.729+0000 7f6492422700 1 -- 172.21.15.40:0/3822345773 wait complete. 2024-04-21T23:09:07.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:07.729+0000 7f6492422700 1 Processor -- start 2024-04-21T23:09:07.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:07.729+0000 7f6492422700 1 -- start start 2024-04-21T23:09:07.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:07.730+0000 7f6492422700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f648c074550 0x7f648c101c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:07.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:07.730+0000 7f6492422700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f648c10e4d0 con 0x7f648c074550 2024-04-21T23:09:07.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:07.730+0000 7f648bfff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f648c074550 0x7f648c101c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:07.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:07.730+0000 7f648bfff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f648c074550 0x7f648c101c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56966/0 (socket says 172.21.15.40:56966) 2024-04-21T23:09:07.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:07.730+0000 7f648bfff700 1 -- 172.21.15.40:0/3409388224 learned_addr learned my addr 172.21.15.40:0/3409388224 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:09:07.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:07.730+0000 7f648bfff700 1 -- 172.21.15.40:0/3409388224 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f648c105b80 con 0x7f648c074550 2024-04-21T23:09:07.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:07.730+0000 7f648bfff700 1 --2- 172.21.15.40:0/3409388224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f648c074550 0x7f648c101c20 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f647c006010 tx=0x7f647c005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:07.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:07.730+0000 7f64897fa700 1 -- 172.21.15.40:0/3409388224 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f647c026070 con 0x7f648c074550 2024-04-21T23:09:07.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:07.731+0000 7f64897fa700 1 -- 172.21.15.40:0/3409388224 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f647c00bc30 con 0x7f648c074550 2024-04-21T23:09:07.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:07.731+0000 7f6492422700 1 -- 172.21.15.40:0/3409388224 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f648c102160 con 0x7f648c074550 2024-04-21T23:09:07.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:07.731+0000 7f64897fa700 1 -- 172.21.15.40:0/3409388224 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f647c003ab0 con 0x7f648c074550 2024-04-21T23:09:07.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:07.731+0000 7f6492422700 1 -- 172.21.15.40:0/3409388224 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f648c1025c0 con 0x7f648c074550 2024-04-21T23:09:07.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:07.731+0000 7f64897fa700 1 -- 172.21.15.40:0/3409388224 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f647c029070 con 0x7f648c074550 2024-04-21T23:09:07.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:07.732+0000 7f6492422700 1 -- 172.21.15.40:0/3409388224 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f648c075c90 con 0x7f648c074550 2024-04-21T23:09:07.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:07.732+0000 7f64897fa700 1 --2- 172.21.15.40:0/3409388224 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f64740390f0 0x7f647403b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:07.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:07.732+0000 7f64897fa700 1 -- 172.21.15.40:0/3409388224 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f647c01e070 con 0x7f648c074550 2024-04-21T23:09:07.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:07.732+0000 7f648b7fe700 1 --2- 172.21.15.40:0/3409388224 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f64740390f0 0x7f647403b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:07.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:07.732+0000 7f648b7fe700 1 --2- 172.21.15.40:0/3409388224 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f64740390f0 0x7f647403b5b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6480006fd0 tx=0x7f6480006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:07.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:07.737+0000 7f64897fa700 1 -- 172.21.15.40:0/3409388224 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f647c013070 con 0x7f648c074550 2024-04-21T23:09:07.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:07.946+0000 7f6492422700 1 -- 172.21.15.40:0/3409388224 --> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi111", "target": ["mon-mgr", ""]}) -- 0x7f648c103150 con 0x7f64740390f0 2024-04-21T23:09:08.681 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:08 smithi040 ceph-mon[27984]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi111", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:09:09.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:09 smithi040 ceph-mon[27984]: Deploying cephadm binary to smithi111 2024-04-21T23:09:11.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:11 smithi040 ceph-mon[27984]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:09:12.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:12.157+0000 7f64897fa700 1 -- 172.21.15.40:0/3409388224 <== mgr.14148 v2:172.21.15.40:6800/2500760788 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f648c103150 con 0x7f64740390f0 2024-04-21T23:09:12.157 INFO:teuthology.orchestra.run.smithi040.stdout:Added host 'smithi111' with addr '172.21.15.111' 2024-04-21T23:09:12.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:12.159+0000 7f6492422700 1 -- 172.21.15.40:0/3409388224 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f64740390f0 msgr2=0x7f647403b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:12.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:12.159+0000 7f6492422700 1 --2- 172.21.15.40:0/3409388224 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f64740390f0 0x7f647403b5b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6480006fd0 tx=0x7f6480006e40 comp rx=0 tx=0).stop 2024-04-21T23:09:12.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:12.159+0000 7f6492422700 1 -- 172.21.15.40:0/3409388224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f648c074550 msgr2=0x7f648c101c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:12.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:12.159+0000 7f6492422700 1 --2- 172.21.15.40:0/3409388224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f648c074550 0x7f648c101c20 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f647c006010 tx=0x7f647c005050 comp rx=0 tx=0).stop 2024-04-21T23:09:12.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:12.159+0000 7f6492422700 1 -- 172.21.15.40:0/3409388224 shutdown_connections 2024-04-21T23:09:12.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:12.159+0000 7f6492422700 1 --2- 172.21.15.40:0/3409388224 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f64740390f0 0x7f647403b5b0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:12.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:12.159+0000 7f6492422700 1 --2- 172.21.15.40:0/3409388224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f648c074550 0x7f648c101c20 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:12.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:12.159+0000 7f6492422700 1 -- 172.21.15.40:0/3409388224 >> 172.21.15.40:0/3409388224 conn(0x7f648c0fd770 msgr2=0x7f648c105f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:12.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:12.159+0000 7f6492422700 1 -- 172.21.15.40:0/3409388224 shutdown_connections 2024-04-21T23:09:12.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:12.159+0000 7f6492422700 1 -- 172.21.15.40:0/3409388224 wait complete. 2024-04-21T23:09:12.706 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-21T23:09:12.949 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:09:13.355 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:13 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:13.355 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:13 smithi040 ceph-mon[27984]: Added host smithi111 2024-04-21T23:09:13.356 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:13 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:09:13.356 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:13 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:14.402 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:14 smithi040 ceph-mon[27984]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:09:14.403 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:14 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:14.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:14.439+0000 7fb2cf1d0700 1 -- 172.21.15.40:0/3624096534 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2c8101690 msgr2=0x7fb2c8101a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:14.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:14.439+0000 7fb2cf1d0700 1 --2- 172.21.15.40:0/3624096534 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2c8101690 0x7fb2c8101a70 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc009910 tx=0x7fb2bc009c20 comp rx=0 tx=0).stop 2024-04-21T23:09:14.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:14.439+0000 7fb2cf1d0700 1 -- 172.21.15.40:0/3624096534 shutdown_connections 2024-04-21T23:09:14.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:14.439+0000 7fb2cf1d0700 1 --2- 172.21.15.40:0/3624096534 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2c8101690 0x7fb2c8101a70 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:14.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:14.439+0000 7fb2cf1d0700 1 -- 172.21.15.40:0/3624096534 >> 172.21.15.40:0/3624096534 conn(0x7fb2c8078160 msgr2=0x7fb2c8078570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:14.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:14.439+0000 7fb2cf1d0700 1 -- 172.21.15.40:0/3624096534 shutdown_connections 2024-04-21T23:09:14.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:14.439+0000 7fb2cf1d0700 1 -- 172.21.15.40:0/3624096534 wait complete. 2024-04-21T23:09:14.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:14.439+0000 7fb2cf1d0700 1 Processor -- start 2024-04-21T23:09:14.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:14.439+0000 7fb2cf1d0700 1 -- start start 2024-04-21T23:09:14.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:14.440+0000 7fb2cf1d0700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2c8101690 0x7fb2c81985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:14.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:14.440+0000 7fb2cf1d0700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c810b1f0 con 0x7fb2c8101690 2024-04-21T23:09:14.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:14.440+0000 7fb2ccf6c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2c8101690 0x7fb2c81985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:14.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:14.440+0000 7fb2ccf6c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2c8101690 0x7fb2c81985c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50200/0 (socket says 172.21.15.40:50200) 2024-04-21T23:09:14.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:14.440+0000 7fb2ccf6c700 1 -- 172.21.15.40:0/2468077264 learned_addr learned my addr 172.21.15.40:0/2468077264 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:09:14.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:14.440+0000 7fb2ccf6c700 1 -- 172.21.15.40:0/2468077264 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2c8198b00 con 0x7fb2c8101690 2024-04-21T23:09:14.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:14.440+0000 7fb2ccf6c700 1 --2- 172.21.15.40:0/2468077264 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2c8101690 0x7fb2c81985c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc005850 tx=0x7fb2bc005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:14.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:14.440+0000 7fb2c5ffb700 1 -- 172.21.15.40:0/2468077264 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb2bc026070 con 0x7fb2c8101690 2024-04-21T23:09:14.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:14.440+0000 7fb2c5ffb700 1 -- 172.21.15.40:0/2468077264 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2bc00bbd0 con 0x7fb2c8101690 2024-04-21T23:09:14.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:14.440+0000 7fb2c5ffb700 1 -- 172.21.15.40:0/2468077264 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb2bc003970 con 0x7fb2c8101690 2024-04-21T23:09:14.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:14.440+0000 7fb2cf1d0700 1 -- 172.21.15.40:0/2468077264 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2c8198df0 con 0x7fb2c8101690 2024-04-21T23:09:14.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:14.441+0000 7fb2cf1d0700 1 -- 172.21.15.40:0/2468077264 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2c81992b0 con 0x7fb2c8101690 2024-04-21T23:09:14.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:14.441+0000 7fb2c5ffb700 1 -- 172.21.15.40:0/2468077264 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7fb2bc029070 con 0x7fb2c8101690 2024-04-21T23:09:14.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:14.442+0000 7fb2cf1d0700 1 -- 172.21.15.40:0/2468077264 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2c8102440 con 0x7fb2c8101690 2024-04-21T23:09:14.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:14.442+0000 7fb2c5ffb700 1 --2- 172.21.15.40:0/2468077264 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fb2b80390f0 0x7fb2b803b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:14.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:14.442+0000 7fb2c5ffb700 1 -- 172.21.15.40:0/2468077264 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fb2bc01e070 con 0x7fb2c8101690 2024-04-21T23:09:14.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:14.442+0000 7fb2c7fff700 1 --2- 172.21.15.40:0/2468077264 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fb2b80390f0 0x7fb2b803b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:14.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:14.442+0000 7fb2c7fff700 1 --2- 172.21.15.40:0/2468077264 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fb2b80390f0 0x7fb2b803b5b0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fb2b4006fd0 tx=0x7fb2b4006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:14.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:14.447+0000 7fb2c5ffb700 1 -- 172.21.15.40:0/2468077264 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fb2bc013070 con 0x7fb2c8101690 2024-04-21T23:09:14.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:14.658+0000 7fb2cf1d0700 1 -- 172.21.15.40:0/2468077264 --> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb2c81008c0 con 0x7fb2b80390f0 2024-04-21T23:09:14.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:14.659+0000 7fb2c5ffb700 1 -- 172.21.15.40:0/2468077264 <== mgr.14148 v2:172.21.15.40:6800/2500760788 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+160 (secure 0 0 0) 0x7fb2c81008c0 con 0x7fb2b80390f0 2024-04-21T23:09:14.659 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:09:14.659 INFO:teuthology.orchestra.run.smithi040.stdout:[{"addr": "172.21.15.40", "hostname": "smithi040", "labels": [], "status": ""}, {"addr": "172.21.15.111", "hostname": "smithi111", "labels": [], "status": ""}] 2024-04-21T23:09:14.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:14.662+0000 7fb2cf1d0700 1 -- 172.21.15.40:0/2468077264 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fb2b80390f0 msgr2=0x7fb2b803b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:14.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:14.662+0000 7fb2cf1d0700 1 --2- 172.21.15.40:0/2468077264 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fb2b80390f0 0x7fb2b803b5b0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fb2b4006fd0 tx=0x7fb2b4006e40 comp rx=0 tx=0).stop 2024-04-21T23:09:14.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:14.662+0000 7fb2cf1d0700 1 -- 172.21.15.40:0/2468077264 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2c8101690 msgr2=0x7fb2c81985c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:14.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:14.662+0000 7fb2cf1d0700 1 --2- 172.21.15.40:0/2468077264 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2c8101690 0x7fb2c81985c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc005850 tx=0x7fb2bc005030 comp rx=0 tx=0).stop 2024-04-21T23:09:14.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:14.662+0000 7fb2cf1d0700 1 -- 172.21.15.40:0/2468077264 shutdown_connections 2024-04-21T23:09:14.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:14.662+0000 7fb2cf1d0700 1 --2- 172.21.15.40:0/2468077264 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fb2b80390f0 0x7fb2b803b5b0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:14.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:14.662+0000 7fb2cf1d0700 1 --2- 172.21.15.40:0/2468077264 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2c8101690 0x7fb2c81985c0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:14.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:14.662+0000 7fb2cf1d0700 1 -- 172.21.15.40:0/2468077264 >> 172.21.15.40:0/2468077264 conn(0x7fb2c8078160 msgr2=0x7fb2c81027d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:14.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:14.662+0000 7fb2cf1d0700 1 -- 172.21.15.40:0/2468077264 shutdown_connections 2024-04-21T23:09:14.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:14.662+0000 7fb2cf1d0700 1 -- 172.21.15.40:0/2468077264 wait complete. 2024-04-21T23:09:15.326 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi174 2024-04-21T23:09:15.326 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-04-21T23:09:15.326 DEBUG:teuthology.orchestra.run.smithi174:> dd of=/etc/ceph/ceph.conf 2024-04-21T23:09:15.345 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-04-21T23:09:15.345 DEBUG:teuthology.orchestra.run.smithi174:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-21T23:09:15.403 INFO:tasks.cephadm:Adding host smithi174 to orchestrator... 2024-04-21T23:09:15.403 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph orch host add smithi174 2024-04-21T23:09:15.643 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:09:16.059 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:15 smithi040 ceph-mon[27984]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:09:16.059 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:15 smithi040 ceph-mon[27984]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T23:09:16.059 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:15 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:17.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:17.105+0000 7fe1df6e4700 1 -- 172.21.15.40:0/2262348771 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe1d8101ed0 msgr2=0x7fe1d81022b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:17.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:17.105+0000 7fe1df6e4700 1 --2- 172.21.15.40:0/2262348771 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe1d8101ed0 0x7fe1d81022b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe1d4009910 tx=0x7fe1d4009c20 comp rx=0 tx=0).stop 2024-04-21T23:09:17.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:17.105+0000 7fe1df6e4700 1 -- 172.21.15.40:0/2262348771 shutdown_connections 2024-04-21T23:09:17.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:17.105+0000 7fe1df6e4700 1 --2- 172.21.15.40:0/2262348771 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe1d8101ed0 0x7fe1d81022b0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:17.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:17.105+0000 7fe1df6e4700 1 -- 172.21.15.40:0/2262348771 >> 172.21.15.40:0/2262348771 conn(0x7fe1d80fd760 msgr2=0x7fe1d80ffb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:17.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:17.106+0000 7fe1df6e4700 1 -- 172.21.15.40:0/2262348771 shutdown_connections 2024-04-21T23:09:17.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:17.106+0000 7fe1df6e4700 1 -- 172.21.15.40:0/2262348771 wait complete. 2024-04-21T23:09:17.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:17.106+0000 7fe1df6e4700 1 Processor -- start 2024-04-21T23:09:17.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:17.106+0000 7fe1df6e4700 1 -- start start 2024-04-21T23:09:17.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:17.106+0000 7fe1df6e4700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe1d8101ed0 0x7fe1d819d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:17.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:17.106+0000 7fe1df6e4700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe1d810c2a0 con 0x7fe1d8101ed0 2024-04-21T23:09:17.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:17.107+0000 7fe1dd480700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe1d8101ed0 0x7fe1d819d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:17.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:17.107+0000 7fe1dd480700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe1d8101ed0 0x7fe1d819d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50218/0 (socket says 172.21.15.40:50218) 2024-04-21T23:09:17.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:17.107+0000 7fe1dd480700 1 -- 172.21.15.40:0/1636761532 learned_addr learned my addr 172.21.15.40:0/1636761532 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:09:17.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:17.107+0000 7fe1dd480700 1 -- 172.21.15.40:0/1636761532 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1d819d950 con 0x7fe1d8101ed0 2024-04-21T23:09:17.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:17.107+0000 7fe1dd480700 1 --2- 172.21.15.40:0/1636761532 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe1d8101ed0 0x7fe1d819d410 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe1d4005850 tx=0x7fe1d4005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:17.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:17.107+0000 7fe1ce7fc700 1 -- 172.21.15.40:0/1636761532 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe1d4026070 con 0x7fe1d8101ed0 2024-04-21T23:09:17.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:17.107+0000 7fe1ce7fc700 1 -- 172.21.15.40:0/1636761532 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1d400bbd0 con 0x7fe1d8101ed0 2024-04-21T23:09:17.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:17.107+0000 7fe1df6e4700 1 -- 172.21.15.40:0/1636761532 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1d819dc40 con 0x7fe1d8101ed0 2024-04-21T23:09:17.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:17.107+0000 7fe1ce7fc700 1 -- 172.21.15.40:0/1636761532 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe1d4003970 con 0x7fe1d8101ed0 2024-04-21T23:09:17.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:17.107+0000 7fe1df6e4700 1 -- 172.21.15.40:0/1636761532 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1d81975d0 con 0x7fe1d8101ed0 2024-04-21T23:09:17.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:17.108+0000 7fe1ce7fc700 1 -- 172.21.15.40:0/1636761532 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7fe1d4029070 con 0x7fe1d8101ed0 2024-04-21T23:09:17.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:17.108+0000 7fe1df6e4700 1 -- 172.21.15.40:0/1636761532 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1d81033b0 con 0x7fe1d8101ed0 2024-04-21T23:09:17.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:17.108+0000 7fe1ce7fc700 1 --2- 172.21.15.40:0/1636761532 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fe1c4039150 0x7fe1c403b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:17.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:17.108+0000 7fe1ce7fc700 1 -- 172.21.15.40:0/1636761532 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fe1d401e070 con 0x7fe1d8101ed0 2024-04-21T23:09:17.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:17.108+0000 7fe1dcc7f700 1 --2- 172.21.15.40:0/1636761532 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fe1c4039150 0x7fe1c403b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:17.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:17.109+0000 7fe1dcc7f700 1 --2- 172.21.15.40:0/1636761532 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fe1c4039150 0x7fe1c403b610 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fe1c8006fd0 tx=0x7fe1c8006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:17.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:17.113+0000 7fe1ce7fc700 1 -- 172.21.15.40:0/1636761532 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fe1d4050030 con 0x7fe1d8101ed0 2024-04-21T23:09:17.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:17.326+0000 7fe1df6e4700 1 -- 172.21.15.40:0/1636761532 --> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi174", "target": ["mon-mgr", ""]}) -- 0x7fe1d8198b00 con 0x7fe1c4039150 2024-04-21T23:09:18.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:17 smithi040 ceph-mon[27984]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:09:18.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:17 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:18.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:17 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:18.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:17 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:18.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:17 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:18.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:17 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi111", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:09:18.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:17 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:09:18.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:17 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:09:19.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:18 smithi040 ceph-mon[27984]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi174", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:09:19.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:18 smithi040 ceph-mon[27984]: Updating smithi111:/etc/ceph/ceph.conf 2024-04-21T23:09:19.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:18 smithi040 ceph-mon[27984]: Deploying cephadm binary to smithi174 2024-04-21T23:09:19.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:18 smithi040 ceph-mon[27984]: Updating smithi111:/var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-21T23:09:20.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:19 smithi040 ceph-mon[27984]: Updating smithi111:/etc/ceph/ceph.client.admin.keyring 2024-04-21T23:09:20.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:19 smithi040 ceph-mon[27984]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:09:20.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:19 smithi040 ceph-mon[27984]: Updating smithi111:/var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T23:09:21.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:21.537+0000 7fe1ce7fc700 1 -- 172.21.15.40:0/1636761532 <== mgr.14148 v2:172.21.15.40:6800/2500760788 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7fe1d8198b00 con 0x7fe1c4039150 2024-04-21T23:09:21.538 INFO:teuthology.orchestra.run.smithi040.stdout:Added host 'smithi174' with addr '172.21.15.174' 2024-04-21T23:09:21.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:21.540+0000 7fe1df6e4700 1 -- 172.21.15.40:0/1636761532 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fe1c4039150 msgr2=0x7fe1c403b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:21.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:21.540+0000 7fe1df6e4700 1 --2- 172.21.15.40:0/1636761532 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fe1c4039150 0x7fe1c403b610 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fe1c8006fd0 tx=0x7fe1c8006e40 comp rx=0 tx=0).stop 2024-04-21T23:09:21.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:21.540+0000 7fe1df6e4700 1 -- 172.21.15.40:0/1636761532 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe1d8101ed0 msgr2=0x7fe1d819d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:21.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:21.540+0000 7fe1df6e4700 1 --2- 172.21.15.40:0/1636761532 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe1d8101ed0 0x7fe1d819d410 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe1d4005850 tx=0x7fe1d4005030 comp rx=0 tx=0).stop 2024-04-21T23:09:21.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:21.540+0000 7fe1df6e4700 1 -- 172.21.15.40:0/1636761532 shutdown_connections 2024-04-21T23:09:21.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:21.540+0000 7fe1df6e4700 1 --2- 172.21.15.40:0/1636761532 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fe1c4039150 0x7fe1c403b610 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:21.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:21.540+0000 7fe1df6e4700 1 --2- 172.21.15.40:0/1636761532 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe1d8101ed0 0x7fe1d819d410 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:21.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:21.540+0000 7fe1df6e4700 1 -- 172.21.15.40:0/1636761532 >> 172.21.15.40:0/1636761532 conn(0x7fe1d80fd760 msgr2=0x7fe1d8104c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:21.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:21.540+0000 7fe1df6e4700 1 -- 172.21.15.40:0/1636761532 shutdown_connections 2024-04-21T23:09:21.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:21.540+0000 7fe1df6e4700 1 -- 172.21.15.40:0/1636761532 wait complete. 2024-04-21T23:09:22.051 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:22 smithi040 ceph-mon[27984]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:09:22.052 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:22 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:22.052 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:22 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:22.052 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:22 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:22.052 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:22 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:22.052 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:22 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:09:22.087 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-21T23:09:22.334 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:09:23.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:23 smithi040 ceph-mon[27984]: Added host smithi174 2024-04-21T23:09:23.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:23 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:23.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:23 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:23.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:23.909+0000 7ffa388f7700 1 -- 172.21.15.40:0/1407776986 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa34100da0 msgr2=0x7ffa34101180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:23.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:23.909+0000 7ffa388f7700 1 --2- 172.21.15.40:0/1407776986 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa34100da0 0x7ffa34101180 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ffa24009910 tx=0x7ffa24009c20 comp rx=0 tx=0).stop 2024-04-21T23:09:23.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:23.909+0000 7ffa388f7700 1 -- 172.21.15.40:0/1407776986 shutdown_connections 2024-04-21T23:09:23.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:23.909+0000 7ffa388f7700 1 --2- 172.21.15.40:0/1407776986 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa34100da0 0x7ffa34101180 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:23.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:23.909+0000 7ffa388f7700 1 -- 172.21.15.40:0/1407776986 >> 172.21.15.40:0/1407776986 conn(0x7ffa340fc630 msgr2=0x7ffa340fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:23.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:23.909+0000 7ffa388f7700 1 -- 172.21.15.40:0/1407776986 shutdown_connections 2024-04-21T23:09:23.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:23.909+0000 7ffa388f7700 1 -- 172.21.15.40:0/1407776986 wait complete. 2024-04-21T23:09:23.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:23.910+0000 7ffa388f7700 1 Processor -- start 2024-04-21T23:09:23.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:23.910+0000 7ffa388f7700 1 -- start start 2024-04-21T23:09:23.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:23.910+0000 7ffa388f7700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa34100da0 0x7ffa341985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:23.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:23.910+0000 7ffa388f7700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ffa3410b170 con 0x7ffa34100da0 2024-04-21T23:09:23.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:23.910+0000 7ffa3259c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa34100da0 0x7ffa341985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:23.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:23.910+0000 7ffa3259c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa34100da0 0x7ffa341985d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41206/0 (socket says 172.21.15.40:41206) 2024-04-21T23:09:23.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:23.910+0000 7ffa3259c700 1 -- 172.21.15.40:0/1757644105 learned_addr learned my addr 172.21.15.40:0/1757644105 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:09:23.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:23.911+0000 7ffa3259c700 1 -- 172.21.15.40:0/1757644105 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa34198b10 con 0x7ffa34100da0 2024-04-21T23:09:23.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:23.911+0000 7ffa3259c700 1 --2- 172.21.15.40:0/1757644105 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa34100da0 0x7ffa341985d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ffa24006010 tx=0x7ffa24005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:23.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:23.911+0000 7ffa237fe700 1 -- 172.21.15.40:0/1757644105 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffa24026070 con 0x7ffa34100da0 2024-04-21T23:09:23.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:23.911+0000 7ffa237fe700 1 -- 172.21.15.40:0/1757644105 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa2400bbd0 con 0x7ffa34100da0 2024-04-21T23:09:23.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:23.911+0000 7ffa388f7700 1 -- 172.21.15.40:0/1757644105 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa34198e00 con 0x7ffa34100da0 2024-04-21T23:09:23.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:23.911+0000 7ffa388f7700 1 -- 172.21.15.40:0/1757644105 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa341992c0 con 0x7ffa34100da0 2024-04-21T23:09:23.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:23.911+0000 7ffa237fe700 1 -- 172.21.15.40:0/1757644105 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffa24003910 con 0x7ffa34100da0 2024-04-21T23:09:23.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:23.912+0000 7ffa237fe700 1 -- 172.21.15.40:0/1757644105 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7ffa24029070 con 0x7ffa34100da0 2024-04-21T23:09:23.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:23.912+0000 7ffa388f7700 1 -- 172.21.15.40:0/1757644105 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa34102280 con 0x7ffa34100da0 2024-04-21T23:09:23.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:23.912+0000 7ffa237fe700 1 --2- 172.21.15.40:0/1757644105 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7ffa1c0390f0 0x7ffa1c03b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:23.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:23.912+0000 7ffa237fe700 1 -- 172.21.15.40:0/1757644105 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7ffa2401e070 con 0x7ffa34100da0 2024-04-21T23:09:23.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:23.912+0000 7ffa31d9b700 1 --2- 172.21.15.40:0/1757644105 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7ffa1c0390f0 0x7ffa1c03b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:23.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:23.913+0000 7ffa31d9b700 1 --2- 172.21.15.40:0/1757644105 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7ffa1c0390f0 0x7ffa1c03b5b0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7ffa28006fd0 tx=0x7ffa28006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:23.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:23.917+0000 7ffa237fe700 1 -- 172.21.15.40:0/1757644105 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7ffa24013070 con 0x7ffa34100da0 2024-04-21T23:09:24.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:24.132+0000 7ffa388f7700 1 -- 172.21.15.40:0/1757644105 --> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ffa3406a6b0 con 0x7ffa1c0390f0 2024-04-21T23:09:24.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:24.133+0000 7ffa237fe700 1 -- 172.21.15.40:0/1757644105 <== mgr.14148 v2:172.21.15.40:6800/2500760788 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+240 (secure 0 0 0) 0x7ffa3406a6b0 con 0x7ffa1c0390f0 2024-04-21T23:09:24.133 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:09:24.133 INFO:teuthology.orchestra.run.smithi040.stdout:[{"addr": "172.21.15.40", "hostname": "smithi040", "labels": [], "status": ""}, {"addr": "172.21.15.111", "hostname": "smithi111", "labels": [], "status": ""}, {"addr": "172.21.15.174", "hostname": "smithi174", "labels": [], "status": ""}] 2024-04-21T23:09:24.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:24.136+0000 7ffa388f7700 1 -- 172.21.15.40:0/1757644105 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7ffa1c0390f0 msgr2=0x7ffa1c03b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:24.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:24.136+0000 7ffa388f7700 1 --2- 172.21.15.40:0/1757644105 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7ffa1c0390f0 0x7ffa1c03b5b0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7ffa28006fd0 tx=0x7ffa28006e40 comp rx=0 tx=0).stop 2024-04-21T23:09:24.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:24.136+0000 7ffa388f7700 1 -- 172.21.15.40:0/1757644105 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa34100da0 msgr2=0x7ffa341985d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:24.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:24.136+0000 7ffa388f7700 1 --2- 172.21.15.40:0/1757644105 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa34100da0 0x7ffa341985d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ffa24006010 tx=0x7ffa24005030 comp rx=0 tx=0).stop 2024-04-21T23:09:24.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:24.136+0000 7ffa388f7700 1 -- 172.21.15.40:0/1757644105 shutdown_connections 2024-04-21T23:09:24.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:24.136+0000 7ffa388f7700 1 --2- 172.21.15.40:0/1757644105 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7ffa1c0390f0 0x7ffa1c03b5b0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:24.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:24.136+0000 7ffa388f7700 1 --2- 172.21.15.40:0/1757644105 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa34100da0 0x7ffa341985d0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:24.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:24.136+0000 7ffa388f7700 1 -- 172.21.15.40:0/1757644105 >> 172.21.15.40:0/1757644105 conn(0x7ffa340fc630 msgr2=0x7ffa34102770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:24.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:24.136+0000 7ffa388f7700 1 -- 172.21.15.40:0/1757644105 shutdown_connections 2024-04-21T23:09:24.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:24.136+0000 7ffa388f7700 1 -- 172.21.15.40:0/1757644105 wait complete. 2024-04-21T23:09:24.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:24 smithi040 ceph-mon[27984]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:09:24.711 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-21T23:09:24.712 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph osd crush tunables default 2024-04-21T23:09:24.966 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:09:25.385 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:25 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:25.385 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:25 smithi040 ceph-mon[27984]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T23:09:26.390 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:26 smithi040 ceph-mon[27984]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:09:26.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:26.416+0000 7f998aa2e700 1 -- 172.21.15.40:0/3356262480 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99840745d0 msgr2=0x7f99840749b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:26.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:26.416+0000 7f998aa2e700 1 --2- 172.21.15.40:0/3356262480 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99840745d0 0x7f99840749b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9974009910 tx=0x7f9974009c20 comp rx=0 tx=0).stop 2024-04-21T23:09:26.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:26.416+0000 7f998aa2e700 1 -- 172.21.15.40:0/3356262480 shutdown_connections 2024-04-21T23:09:26.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:26.416+0000 7f998aa2e700 1 --2- 172.21.15.40:0/3356262480 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99840745d0 0x7f99840749b0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:26.416+0000 7f998aa2e700 1 -- 172.21.15.40:0/3356262480 >> 172.21.15.40:0/3356262480 conn(0x7f99840fd750 msgr2=0x7f99840ffb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:26.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:26.416+0000 7f998aa2e700 1 -- 172.21.15.40:0/3356262480 shutdown_connections 2024-04-21T23:09:26.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:26.416+0000 7f998aa2e700 1 -- 172.21.15.40:0/3356262480 wait complete. 2024-04-21T23:09:26.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:26.416+0000 7f998aa2e700 1 Processor -- start 2024-04-21T23:09:26.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:26.416+0000 7f998aa2e700 1 -- start start 2024-04-21T23:09:26.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:26.417+0000 7f998aa2e700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99840745d0 0x7f998419dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:26.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:26.417+0000 7f998aa2e700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f99841128e0 con 0x7f99840745d0 2024-04-21T23:09:26.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:26.417+0000 7f9983fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99840745d0 0x7f998419dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:26.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:26.417+0000 7f9983fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99840745d0 0x7f998419dd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41230/0 (socket says 172.21.15.40:41230) 2024-04-21T23:09:26.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:26.417+0000 7f9983fff700 1 -- 172.21.15.40:0/2094741563 learned_addr learned my addr 172.21.15.40:0/2094741563 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:09:26.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:26.417+0000 7f9983fff700 1 -- 172.21.15.40:0/2094741563 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f998419e2c0 con 0x7f99840745d0 2024-04-21T23:09:26.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:26.417+0000 7f9983fff700 1 --2- 172.21.15.40:0/2094741563 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99840745d0 0x7f998419dd80 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9974006010 tx=0x7f9974005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:26.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:26.418+0000 7f99817fa700 1 -- 172.21.15.40:0/2094741563 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9974026070 con 0x7f99840745d0 2024-04-21T23:09:26.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:26.418+0000 7f998aa2e700 1 -- 172.21.15.40:0/2094741563 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f998419e550 con 0x7f99840745d0 2024-04-21T23:09:26.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:26.418+0000 7f99817fa700 1 -- 172.21.15.40:0/2094741563 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f997400bc30 con 0x7f99840745d0 2024-04-21T23:09:26.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:26.418+0000 7f99817fa700 1 -- 172.21.15.40:0/2094741563 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9974003ab0 con 0x7f99840745d0 2024-04-21T23:09:26.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:26.418+0000 7f998aa2e700 1 -- 172.21.15.40:0/2094741563 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f998419e9b0 con 0x7f99840745d0 2024-04-21T23:09:26.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:26.419+0000 7f99817fa700 1 -- 172.21.15.40:0/2094741563 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f9974029070 con 0x7f99840745d0 2024-04-21T23:09:26.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:26.419+0000 7f99817fa700 1 --2- 172.21.15.40:0/2094741563 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f996c0390f0 0x7f996c03b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:26.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:26.419+0000 7f99817fa700 1 -- 172.21.15.40:0/2094741563 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f997401e070 con 0x7f99840745d0 2024-04-21T23:09:26.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:26.419+0000 7f998aa2e700 1 -- 172.21.15.40:0/2094741563 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9984075d10 con 0x7f99840745d0 2024-04-21T23:09:26.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:26.419+0000 7f99837fe700 1 --2- 172.21.15.40:0/2094741563 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f996c0390f0 0x7f996c03b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:26.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:26.419+0000 7f99837fe700 1 --2- 172.21.15.40:0/2094741563 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f996c0390f0 0x7f996c03b5b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f9978006fd0 tx=0x7f9978006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:26.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:26.424+0000 7f99817fa700 1 -- 172.21.15.40:0/2094741563 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f9974030020 con 0x7f99840745d0 2024-04-21T23:09:26.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:26.635+0000 7f998aa2e700 1 -- 172.21.15.40:0/2094741563 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f998419f0e0 con 0x7f99840745d0 2024-04-21T23:09:27.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:27.133+0000 7f99817fa700 1 -- 172.21.15.40:0/2094741563 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f9974013070 con 0x7f99840745d0 2024-04-21T23:09:27.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:27.135+0000 7f998aa2e700 1 -- 172.21.15.40:0/2094741563 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f996c0390f0 msgr2=0x7f996c03b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:27.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:27.135+0000 7f998aa2e700 1 --2- 172.21.15.40:0/2094741563 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f996c0390f0 0x7f996c03b5b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f9978006fd0 tx=0x7f9978006e40 comp rx=0 tx=0).stop 2024-04-21T23:09:27.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:27.135+0000 7f998aa2e700 1 -- 172.21.15.40:0/2094741563 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99840745d0 msgr2=0x7f998419dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:27.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:27.135+0000 7f998aa2e700 1 --2- 172.21.15.40:0/2094741563 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99840745d0 0x7f998419dd80 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9974006010 tx=0x7f9974005050 comp rx=0 tx=0).stop 2024-04-21T23:09:27.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:27.136+0000 7f998aa2e700 1 -- 172.21.15.40:0/2094741563 shutdown_connections 2024-04-21T23:09:27.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:27.136+0000 7f998aa2e700 1 --2- 172.21.15.40:0/2094741563 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f996c0390f0 0x7f996c03b5b0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:27.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:27.136+0000 7f998aa2e700 1 --2- 172.21.15.40:0/2094741563 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99840745d0 0x7f998419dd80 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:27.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:27.136+0000 7f998aa2e700 1 -- 172.21.15.40:0/2094741563 >> 172.21.15.40:0/2094741563 conn(0x7f99840fd750 msgr2=0x7f99840fdb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:27.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:27.136+0000 7f998aa2e700 1 -- 172.21.15.40:0/2094741563 shutdown_connections 2024-04-21T23:09:27.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:27.136+0000 7f998aa2e700 1 -- 172.21.15.40:0/2094741563 wait complete. 2024-04-21T23:09:27.136 INFO:teuthology.orchestra.run.smithi040.stderr:adjusted tunables profile to default 2024-04-21T23:09:27.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:27 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/2094741563' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-21T23:09:27.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:27 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:27.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:27 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:27.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:27 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:27.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:27 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:27.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:27 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi174", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:09:27.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:27 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:09:27.433 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:27 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:09:27.823 INFO:tasks.cephadm:Adding mon.a on smithi040 2024-04-21T23:09:27.824 INFO:tasks.cephadm:Adding mon.b on smithi111 2024-04-21T23:09:27.824 INFO:tasks.cephadm:Adding mon.c on smithi174 2024-04-21T23:09:27.824 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph orch apply mon '3;smithi040:172.21.15.40=a;smithi111:172.21.15.111=b;smithi174:172.21.15.174=c' 2024-04-21T23:09:28.053 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-21T23:09:28.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:28 smithi040 ceph-mon[27984]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:09:28.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:28 smithi040 ceph-mon[27984]: Updating smithi174:/etc/ceph/ceph.conf 2024-04-21T23:09:28.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:28 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/2094741563' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-21T23:09:28.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:28 smithi040 ceph-mon[27984]: osdmap e4: 0 total, 0 up, 0 in 2024-04-21T23:09:28.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:28 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:28.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:28 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:28.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:28 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:29.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:29 smithi040 ceph-mon[27984]: Updating smithi174:/var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-21T23:09:29.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:29 smithi040 ceph-mon[27984]: Updating smithi174:/etc/ceph/ceph.client.admin.keyring 2024-04-21T23:09:29.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:29 smithi040 ceph-mon[27984]: Updating smithi174:/var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T23:09:29.619 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:29.618+0000 7f0c22af9700 1 -- 172.21.15.174:0/2378124797 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c1c102ee0 msgr2=0x7f0c1c1032c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:29.619 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:29.618+0000 7f0c22af9700 1 --2- 172.21.15.174:0/2378124797 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c1c102ee0 0x7f0c1c1032c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c009910 tx=0x7f0c0c009c20 comp rx=0 tx=0).stop 2024-04-21T23:09:29.619 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:29.618+0000 7f0c22af9700 1 -- 172.21.15.174:0/2378124797 shutdown_connections 2024-04-21T23:09:29.619 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:29.618+0000 7f0c22af9700 1 --2- 172.21.15.174:0/2378124797 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c1c102ee0 0x7f0c1c1032c0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:29.619 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:29.618+0000 7f0c22af9700 1 -- 172.21.15.174:0/2378124797 >> 172.21.15.174:0/2378124797 conn(0x7f0c1c0fc690 msgr2=0x7f0c1c0feab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:29.619 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:29.618+0000 7f0c22af9700 1 -- 172.21.15.174:0/2378124797 shutdown_connections 2024-04-21T23:09:29.619 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:29.618+0000 7f0c22af9700 1 -- 172.21.15.174:0/2378124797 wait complete. 2024-04-21T23:09:29.619 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:29.619+0000 7f0c22af9700 1 Processor -- start 2024-04-21T23:09:29.620 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:29.619+0000 7f0c22af9700 1 -- start start 2024-04-21T23:09:29.620 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:29.619+0000 7f0c22af9700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c1c102ee0 0x7f0c1c19d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:29.620 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:29.619+0000 7f0c22af9700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0c1c10c330 con 0x7f0c1c102ee0 2024-04-21T23:09:29.620 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:29.619+0000 7f0c20895700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c1c102ee0 0x7f0c1c19d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:29.620 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:29.619+0000 7f0c20895700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c1c102ee0 0x7f0c1c19d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.174:50904/0 (socket says 172.21.15.174:50904) 2024-04-21T23:09:29.620 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:29.619+0000 7f0c20895700 1 -- 172.21.15.174:0/4082645086 learned_addr learned my addr 172.21.15.174:0/4082645086 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-04-21T23:09:29.621 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:29.620+0000 7f0c20895700 1 -- 172.21.15.174:0/4082645086 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c1c19d9a0 con 0x7f0c1c102ee0 2024-04-21T23:09:29.621 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:29.620+0000 7f0c20895700 1 --2- 172.21.15.174:0/4082645086 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c1c102ee0 0x7f0c1c19d460 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c006010 tx=0x7f0c0c005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:29.621 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:29.620+0000 7f0c19ffb700 1 -- 172.21.15.174:0/4082645086 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0c0c026070 con 0x7f0c1c102ee0 2024-04-21T23:09:29.621 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:29.620+0000 7f0c19ffb700 1 -- 172.21.15.174:0/4082645086 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0c0c00bc30 con 0x7f0c1c102ee0 2024-04-21T23:09:29.622 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:29.620+0000 7f0c22af9700 1 -- 172.21.15.174:0/4082645086 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c1c19dc30 con 0x7f0c1c102ee0 2024-04-21T23:09:29.622 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:29.620+0000 7f0c19ffb700 1 -- 172.21.15.174:0/4082645086 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0c0c003ab0 con 0x7f0c1c102ee0 2024-04-21T23:09:29.622 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:29.620+0000 7f0c22af9700 1 -- 172.21.15.174:0/4082645086 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c1c197620 con 0x7f0c1c102ee0 2024-04-21T23:09:29.622 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:29.621+0000 7f0c19ffb700 1 -- 172.21.15.174:0/4082645086 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f0c0c029070 con 0x7f0c1c102ee0 2024-04-21T23:09:29.622 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:29.621+0000 7f0c22af9700 1 -- 172.21.15.174:0/4082645086 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c1c06a8e0 con 0x7f0c1c102ee0 2024-04-21T23:09:29.622 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:29.621+0000 7f0c19ffb700 1 --2- 172.21.15.174:0/4082645086 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f0c040390f0 0x7f0c0403b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:29.622 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:29.621+0000 7f0c19ffb700 1 -- 172.21.15.174:0/4082645086 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f0c0c01e070 con 0x7f0c1c102ee0 2024-04-21T23:09:29.622 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:29.622+0000 7f0c1bfff700 1 --2- 172.21.15.174:0/4082645086 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f0c040390f0 0x7f0c0403b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:29.622 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:29.622+0000 7f0c1bfff700 1 --2- 172.21.15.174:0/4082645086 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f0c040390f0 0x7f0c0403b5b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0c10006fd0 tx=0x7f0c10006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:29.626 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:29.627+0000 7f0c19ffb700 1 -- 172.21.15.174:0/4082645086 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f0c0c013070 con 0x7f0c1c102ee0 2024-04-21T23:09:29.834 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:29.834+0000 7f0c22af9700 1 -- 172.21.15.174:0/4082645086 --> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi040:172.21.15.40=a;smithi111:172.21.15.111=b;smithi174:172.21.15.174=c", "target": ["mon-mgr", ""]}) -- 0x7f0c1c198b50 con 0x7f0c040390f0 2024-04-21T23:09:29.840 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:29.841+0000 7f0c19ffb700 1 -- 172.21.15.174:0/4082645086 <== mgr.14148 v2:172.21.15.40:6800/2500760788 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f0c1c198b50 con 0x7f0c040390f0 2024-04-21T23:09:29.841 INFO:teuthology.orchestra.run.smithi174.stdout:Scheduled mon update... 2024-04-21T23:09:29.843 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:29.843+0000 7f0c22af9700 1 -- 172.21.15.174:0/4082645086 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f0c040390f0 msgr2=0x7f0c0403b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:29.843 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:29.843+0000 7f0c22af9700 1 --2- 172.21.15.174:0/4082645086 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f0c040390f0 0x7f0c0403b5b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0c10006fd0 tx=0x7f0c10006e40 comp rx=0 tx=0).stop 2024-04-21T23:09:29.843 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:29.843+0000 7f0c22af9700 1 -- 172.21.15.174:0/4082645086 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c1c102ee0 msgr2=0x7f0c1c19d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:29.843 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:29.843+0000 7f0c22af9700 1 --2- 172.21.15.174:0/4082645086 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c1c102ee0 0x7f0c1c19d460 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c006010 tx=0x7f0c0c005050 comp rx=0 tx=0).stop 2024-04-21T23:09:29.843 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:29.843+0000 7f0c22af9700 1 -- 172.21.15.174:0/4082645086 shutdown_connections 2024-04-21T23:09:29.843 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:29.843+0000 7f0c22af9700 1 --2- 172.21.15.174:0/4082645086 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f0c040390f0 0x7f0c0403b5b0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:29.843 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:29.843+0000 7f0c22af9700 1 --2- 172.21.15.174:0/4082645086 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c1c102ee0 0x7f0c1c19d460 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:29.843 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:29.843+0000 7f0c22af9700 1 -- 172.21.15.174:0/4082645086 >> 172.21.15.174:0/4082645086 conn(0x7f0c1c0fc690 msgr2=0x7f0c1c103c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:29.843 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:29.843+0000 7f0c22af9700 1 -- 172.21.15.174:0/4082645086 shutdown_connections 2024-04-21T23:09:29.843 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:29.843+0000 7f0c22af9700 1 -- 172.21.15.174:0/4082645086 wait complete. 2024-04-21T23:09:30.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:30 smithi040 ceph-mon[27984]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:09:30.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:30 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:30.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:30 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:09:30.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:30 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:09:30.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:30 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:09:30.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:30 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:30.433 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:30 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T23:09:30.433 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:30 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:09:30.441 DEBUG:teuthology.orchestra.run.smithi111:mon.b> sudo journalctl -f -n 0 -u ceph-d814e4dc-0033-11ef-bc93-c7b262605968@mon.b.service 2024-04-21T23:09:30.444 DEBUG:teuthology.orchestra.run.smithi174:mon.c> sudo journalctl -f -n 0 -u ceph-d814e4dc-0033-11ef-bc93-c7b262605968@mon.c.service 2024-04-21T23:09:30.446 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-21T23:09:30.446 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-21T23:09:31.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:31 smithi040 ceph-mon[27984]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi040:172.21.15.40=a;smithi111:172.21.15.111=b;smithi174:172.21.15.174=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:09:31.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:31 smithi040 ceph-mon[27984]: Saving service mon spec with placement smithi040:172.21.15.40=a;smithi111:172.21.15.111=b;smithi174:172.21.15.174=c;count:3 2024-04-21T23:09:31.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:31 smithi040 ceph-mon[27984]: Deploying daemon mon.c on smithi174 2024-04-21T23:09:32.158 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-21T23:09:32.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:32 smithi040 ceph-mon[27984]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:09:33.857 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:33 smithi174 systemd[1]: Starting Ceph mon.c for d814e4dc-0033-11ef-bc93-c7b262605968... 2024-04-21T23:09:34.141 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:34.140+0000 7f977933c700 1 -- 172.21.15.174:0/2815392783 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9774100510 msgr2=0x7f97741008f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:34.141 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:34.140+0000 7f977933c700 1 --2- 172.21.15.174:0/2815392783 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9774100510 0x7f97741008f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9764009910 tx=0x7f9764009c20 comp rx=0 tx=0).stop 2024-04-21T23:09:34.142 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:34.140+0000 7f977933c700 1 -- 172.21.15.174:0/2815392783 shutdown_connections 2024-04-21T23:09:34.142 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:34.140+0000 7f977933c700 1 --2- 172.21.15.174:0/2815392783 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9774100510 0x7f97741008f0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:34.142 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:34.140+0000 7f977933c700 1 -- 172.21.15.174:0/2815392783 >> 172.21.15.174:0/2815392783 conn(0x7f97740772b0 msgr2=0x7f97740776c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:34.142 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:34.140+0000 7f977933c700 1 -- 172.21.15.174:0/2815392783 shutdown_connections 2024-04-21T23:09:34.142 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:34.140+0000 7f977933c700 1 -- 172.21.15.174:0/2815392783 wait complete. 2024-04-21T23:09:34.142 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:34.140+0000 7f977933c700 1 Processor -- start 2024-04-21T23:09:34.142 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:34.141+0000 7f977933c700 1 -- start start 2024-04-21T23:09:34.142 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:34.141+0000 7f977933c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9774100510 0x7f9774198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:34.142 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:34.141+0000 7f977933c700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f977410a8e0 con 0x7f9774100510 2024-04-21T23:09:34.142 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:34.141+0000 7f9772ffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9774100510 0x7f9774198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:34.142 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:34.141+0000 7f9772ffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9774100510 0x7f9774198600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.174:47044/0 (socket says 172.21.15.174:47044) 2024-04-21T23:09:34.142 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:34.141+0000 7f9772ffd700 1 -- 172.21.15.174:0/3371992644 learned_addr learned my addr 172.21.15.174:0/3371992644 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-04-21T23:09:34.142 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:34.142+0000 7f9772ffd700 1 -- 172.21.15.174:0/3371992644 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9774198b40 con 0x7f9774100510 2024-04-21T23:09:34.143 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:34.142+0000 7f9772ffd700 1 --2- 172.21.15.174:0/3371992644 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9774100510 0x7f9774198600 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f9764005850 tx=0x7f9764005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:34.144 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:34.142+0000 7f975bfff700 1 -- 172.21.15.174:0/3371992644 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9764026070 con 0x7f9774100510 2024-04-21T23:09:34.144 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:34.142+0000 7f975bfff700 1 -- 172.21.15.174:0/3371992644 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f976400bbd0 con 0x7f9774100510 2024-04-21T23:09:34.144 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:34.142+0000 7f977933c700 1 -- 172.21.15.174:0/3371992644 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9774198e30 con 0x7f9774100510 2024-04-21T23:09:34.144 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:34.142+0000 7f975bfff700 1 -- 172.21.15.174:0/3371992644 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9764003970 con 0x7f9774100510 2024-04-21T23:09:34.144 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:34.142+0000 7f977933c700 1 -- 172.21.15.174:0/3371992644 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97741992f0 con 0x7f9774100510 2024-04-21T23:09:34.144 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:34.143+0000 7f977933c700 1 -- 172.21.15.174:0/3371992644 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9760005330 con 0x7f9774100510 2024-04-21T23:09:34.144 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:34.143+0000 7f975bfff700 1 -- 172.21.15.174:0/3371992644 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f9764029070 con 0x7f9774100510 2024-04-21T23:09:34.145 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:34.144+0000 7f975bfff700 1 --2- 172.21.15.174:0/3371992644 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f975c0390f0 0x7f975c03b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:34.145 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:34.144+0000 7f975bfff700 1 -- 172.21.15.174:0/3371992644 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f976401e070 con 0x7f9774100510 2024-04-21T23:09:34.145 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:34.144+0000 7f97727fc700 1 --2- 172.21.15.174:0/3371992644 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f975c0390f0 0x7f975c03b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:34.145 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:34.144+0000 7f97727fc700 1 --2- 172.21.15.174:0/3371992644 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f975c0390f0 0x7f975c03b5b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f9768006fd0 tx=0x7f9768006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:34.149 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:34.149+0000 7f975bfff700 1 -- 172.21.15.174:0/3371992644 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f9764013070 con 0x7f9774100510 2024-04-21T23:09:34.391 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 podman[31374]: 2024-04-21 23:09:34.175995868 +0000 UTC m=+0.017723857 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-21T23:09:34.391 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 podman[31374]: 2024-04-21 23:09:34.27800717 +0000 UTC m=+0.119735156 container create 2500b95b86b5aa92e16def89b40df69f8ef4f99398a445434e904402392d284d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mon-c, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-820a2f8, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416) 2024-04-21T23:09:34.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:34 smithi040 ceph-mon[27984]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:09:34.440 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:34.440+0000 7f977933c700 1 -- 172.21.15.174:0/3371992644 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f9760005620 con 0x7f9774100510 2024-04-21T23:09:34.440 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:34.441+0000 7f975bfff700 1 -- 172.21.15.174:0/3371992644 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7f9764030020 con 0x7f9774100510 2024-04-21T23:09:34.441 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-04-21T23:09:34.441 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"d814e4dc-0033-11ef-bc93-c7b262605968","modified":"2024-04-21T23:07:49.696814Z","created":"2024-04-21T23:07:49.696814Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-21T23:09:34.443 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:34.443+0000 7f977933c700 1 -- 172.21.15.174:0/3371992644 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f975c0390f0 msgr2=0x7f975c03b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:34.443 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:34.443+0000 7f977933c700 1 --2- 172.21.15.174:0/3371992644 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f975c0390f0 0x7f975c03b5b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f9768006fd0 tx=0x7f9768006e40 comp rx=0 tx=0).stop 2024-04-21T23:09:34.443 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:34.443+0000 7f977933c700 1 -- 172.21.15.174:0/3371992644 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9774100510 msgr2=0x7f9774198600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:34.443 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:34.443+0000 7f977933c700 1 --2- 172.21.15.174:0/3371992644 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9774100510 0x7f9774198600 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f9764005850 tx=0x7f9764005030 comp rx=0 tx=0).stop 2024-04-21T23:09:34.443 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:34.443+0000 7f977933c700 1 -- 172.21.15.174:0/3371992644 shutdown_connections 2024-04-21T23:09:34.443 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:34.443+0000 7f977933c700 1 --2- 172.21.15.174:0/3371992644 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f975c0390f0 0x7f975c03b5b0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:34.443 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:34.443+0000 7f977933c700 1 --2- 172.21.15.174:0/3371992644 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9774100510 0x7f9774198600 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:34.443 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:34.443+0000 7f977933c700 1 -- 172.21.15.174:0/3371992644 >> 172.21.15.174:0/3371992644 conn(0x7f97740772b0 msgr2=0x7f9774108080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:34.443 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:34.443+0000 7f977933c700 1 -- 172.21.15.174:0/3371992644 shutdown_connections 2024-04-21T23:09:34.443 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:34.443+0000 7f977933c700 1 -- 172.21.15.174:0/3371992644 wait complete. 2024-04-21T23:09:34.444 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2024-04-21T23:09:34.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 podman[31374]: 2024-04-21 23:09:34.489365574 +0000 UTC m=+0.331093554 container init 2500b95b86b5aa92e16def89b40df69f8ef4f99398a445434e904402392d284d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mon-c, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=squid-820a2f8, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-21T23:09:34.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 podman[31374]: 2024-04-21 23:09:34.492689998 +0000 UTC m=+0.334417976 container start 2500b95b86b5aa92e16def89b40df69f8ef4f99398a445434e904402392d284d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mon-c, RELEASE=squid-820a2f8, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2) 2024-04-21T23:09:34.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: set uid:gid to 167:167 (ceph:ceph) 2024-04-21T23:09:34.695 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: ceph version 19.0.0-2405-g820a2f8c (820a2f8ced66152ed3d765c31444b8076ea28c6b) squid (dev), process ceph-mon, pid 2 2024-04-21T23:09:34.695 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: pidfile_write: ignore empty --pid-file 2024-04-21T23:09:34.695 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: load: jerasure load: lrc 2024-04-21T23:09:34.695 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: RocksDB version: 7.9.2 2024-04-21T23:09:34.695 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Git sha 0 2024-04-21T23:09:34.695 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Compile date 2024-04-20 15:04:30 2024-04-21T23:09:34.695 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: DB SUMMARY 2024-04-21T23:09:34.695 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: DB Session ID: PSBN70Z7WDHCSE4Z68GQ 2024-04-21T23:09:34.695 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: CURRENT file: CURRENT 2024-04-21T23:09:34.695 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: IDENTITY file: IDENTITY 2024-04-21T23:09:34.695 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-04-21T23:09:34.695 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-04-21T23:09:34.695 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-04-21T23:09:34.695 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.error_if_exists: 0 2024-04-21T23:09:34.696 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.create_if_missing: 0 2024-04-21T23:09:34.696 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.paranoid_checks: 1 2024-04-21T23:09:34.696 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-21T23:09:34.696 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-21T23:09:34.696 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-21T23:09:34.696 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.env: 0x55614a58dc60 2024-04-21T23:09:34.696 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.fs: PosixFileSystem 2024-04-21T23:09:34.696 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.info_log: 0x55614b741aa0 2024-04-21T23:09:34.696 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.max_file_opening_threads: 16 2024-04-21T23:09:34.696 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.statistics: (nil) 2024-04-21T23:09:34.696 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.use_fsync: 0 2024-04-21T23:09:34.696 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.max_log_file_size: 0 2024-04-21T23:09:34.696 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-21T23:09:34.696 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-21T23:09:34.697 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.keep_log_file_num: 1000 2024-04-21T23:09:34.697 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.recycle_log_file_num: 0 2024-04-21T23:09:34.697 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.allow_fallocate: 1 2024-04-21T23:09:34.697 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.allow_mmap_reads: 0 2024-04-21T23:09:34.697 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.allow_mmap_writes: 0 2024-04-21T23:09:34.697 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.use_direct_reads: 0 2024-04-21T23:09:34.697 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-21T23:09:34.697 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.create_missing_column_families: 0 2024-04-21T23:09:34.697 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.db_log_dir: 2024-04-21T23:09:34.697 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.wal_dir: 2024-04-21T23:09:34.697 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-21T23:09:34.697 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-21T23:09:34.697 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-21T23:09:34.697 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-21T23:09:34.698 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-21T23:09:34.698 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-21T23:09:34.698 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.advise_random_on_open: 1 2024-04-21T23:09:34.698 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.db_write_buffer_size: 0 2024-04-21T23:09:34.698 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.write_buffer_manager: 0x55614b7141e0 2024-04-21T23:09:34.698 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-21T23:09:34.698 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-21T23:09:34.698 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-21T23:09:34.698 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.rate_limiter: (nil) 2024-04-21T23:09:34.698 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-21T23:09:34.698 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.wal_recovery_mode: 2 2024-04-21T23:09:34.698 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.enable_thread_tracking: 0 2024-04-21T23:09:34.698 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.enable_pipelined_write: 0 2024-04-21T23:09:34.698 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.unordered_write: 0 2024-04-21T23:09:34.699 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-21T23:09:34.699 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-21T23:09:34.699 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-21T23:09:34.699 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-21T23:09:34.699 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.row_cache: None 2024-04-21T23:09:34.699 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.wal_filter: None 2024-04-21T23:09:34.700 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-21T23:09:34.700 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.allow_ingest_behind: 0 2024-04-21T23:09:34.700 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.two_write_queues: 0 2024-04-21T23:09:34.701 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.manual_wal_flush: 0 2024-04-21T23:09:34.701 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.wal_compression: 0 2024-04-21T23:09:34.701 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.atomic_flush: 0 2024-04-21T23:09:34.701 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-21T23:09:34.701 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-21T23:09:34.701 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-21T23:09:34.701 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.log_readahead_size: 0 2024-04-21T23:09:34.701 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-21T23:09:34.701 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.best_efforts_recovery: 0 2024-04-21T23:09:34.701 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-21T23:09:34.701 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-21T23:09:34.701 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.allow_data_in_errors: 0 2024-04-21T23:09:34.701 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.db_host_id: __hostname__ 2024-04-21T23:09:34.701 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-21T23:09:34.702 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.max_background_jobs: 2 2024-04-21T23:09:34.702 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.max_background_compactions: -1 2024-04-21T23:09:34.702 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.max_subcompactions: 1 2024-04-21T23:09:34.702 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-21T23:09:34.702 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-21T23:09:34.702 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-21T23:09:34.702 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.max_total_wal_size: 0 2024-04-21T23:09:34.702 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-21T23:09:34.702 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-21T23:09:34.702 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-21T23:09:34.702 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-21T23:09:34.702 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.max_open_files: -1 2024-04-21T23:09:34.702 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.bytes_per_sync: 0 2024-04-21T23:09:34.702 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-21T23:09:34.703 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-21T23:09:34.703 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.compaction_readahead_size: 0 2024-04-21T23:09:34.703 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.max_background_flushes: -1 2024-04-21T23:09:34.703 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Compression algorithms supported: 2024-04-21T23:09:34.703 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-21T23:09:34.703 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: kZSTD supported: 0 2024-04-21T23:09:34.703 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: kXpressCompression supported: 0 2024-04-21T23:09:34.703 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: kLZ4HCCompression supported: 1 2024-04-21T23:09:34.703 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: kZlibCompression supported: 1 2024-04-21T23:09:34.703 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: kSnappyCompression supported: 1 2024-04-21T23:09:34.703 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: kLZ4Compression supported: 1 2024-04-21T23:09:34.703 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: kBZip2Compression supported: 0 2024-04-21T23:09:34.703 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-21T23:09:34.703 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-21T23:09:34.704 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-04-21T23:09:34.704 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-21T23:09:34.704 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-21T23:09:34.704 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.merge_operator: 2024-04-21T23:09:34.704 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.compaction_filter: None 2024-04-21T23:09:34.704 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.compaction_filter_factory: None 2024-04-21T23:09:34.704 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.sst_partitioner_factory: None 2024-04-21T23:09:34.704 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-21T23:09:34.704 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-21T23:09:34.704 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55614b741ba0) 2024-04-21T23:09:34.704 INFO:journalctl@ceph.mon.c.smithi174.stdout: cache_index_and_filter_blocks: 1 2024-04-21T23:09:34.704 INFO:journalctl@ceph.mon.c.smithi174.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-21T23:09:34.704 INFO:journalctl@ceph.mon.c.smithi174.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-21T23:09:34.704 INFO:journalctl@ceph.mon.c.smithi174.stdout: pin_top_level_index_and_filter: 1 2024-04-21T23:09:34.705 INFO:journalctl@ceph.mon.c.smithi174.stdout: index_type: 0 2024-04-21T23:09:34.705 INFO:journalctl@ceph.mon.c.smithi174.stdout: data_block_index_type: 0 2024-04-21T23:09:34.705 INFO:journalctl@ceph.mon.c.smithi174.stdout: index_shortening: 1 2024-04-21T23:09:34.705 INFO:journalctl@ceph.mon.c.smithi174.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-21T23:09:34.705 INFO:journalctl@ceph.mon.c.smithi174.stdout: checksum: 4 2024-04-21T23:09:34.705 INFO:journalctl@ceph.mon.c.smithi174.stdout: no_block_cache: 0 2024-04-21T23:09:34.705 INFO:journalctl@ceph.mon.c.smithi174.stdout: block_cache: 0x55614b745090 2024-04-21T23:09:34.705 INFO:journalctl@ceph.mon.c.smithi174.stdout: block_cache_name: BinnedLRUCache 2024-04-21T23:09:34.705 INFO:journalctl@ceph.mon.c.smithi174.stdout: block_cache_options: 2024-04-21T23:09:34.705 INFO:journalctl@ceph.mon.c.smithi174.stdout: capacity : 536870912 2024-04-21T23:09:34.705 INFO:journalctl@ceph.mon.c.smithi174.stdout: num_shard_bits : 4 2024-04-21T23:09:34.705 INFO:journalctl@ceph.mon.c.smithi174.stdout: strict_capacity_limit : 0 2024-04-21T23:09:34.705 INFO:journalctl@ceph.mon.c.smithi174.stdout: high_pri_pool_ratio: 0.000 2024-04-21T23:09:34.705 INFO:journalctl@ceph.mon.c.smithi174.stdout: block_cache_compressed: (nil) 2024-04-21T23:09:34.705 INFO:journalctl@ceph.mon.c.smithi174.stdout: persistent_cache: (nil) 2024-04-21T23:09:34.706 INFO:journalctl@ceph.mon.c.smithi174.stdout: block_size: 4096 2024-04-21T23:09:34.706 INFO:journalctl@ceph.mon.c.smithi174.stdout: block_size_deviation: 10 2024-04-21T23:09:34.706 INFO:journalctl@ceph.mon.c.smithi174.stdout: block_restart_interval: 16 2024-04-21T23:09:34.706 INFO:journalctl@ceph.mon.c.smithi174.stdout: index_block_restart_interval: 1 2024-04-21T23:09:34.706 INFO:journalctl@ceph.mon.c.smithi174.stdout: metadata_block_size: 4096 2024-04-21T23:09:34.706 INFO:journalctl@ceph.mon.c.smithi174.stdout: partition_filters: 0 2024-04-21T23:09:34.706 INFO:journalctl@ceph.mon.c.smithi174.stdout: use_delta_encoding: 1 2024-04-21T23:09:34.706 INFO:journalctl@ceph.mon.c.smithi174.stdout: filter_policy: bloomfilter 2024-04-21T23:09:34.706 INFO:journalctl@ceph.mon.c.smithi174.stdout: whole_key_filtering: 1 2024-04-21T23:09:34.706 INFO:journalctl@ceph.mon.c.smithi174.stdout: verify_compression: 0 2024-04-21T23:09:34.706 INFO:journalctl@ceph.mon.c.smithi174.stdout: read_amp_bytes_per_bit: 0 2024-04-21T23:09:34.706 INFO:journalctl@ceph.mon.c.smithi174.stdout: format_version: 5 2024-04-21T23:09:34.706 INFO:journalctl@ceph.mon.c.smithi174.stdout: enable_index_compression: 1 2024-04-21T23:09:34.706 INFO:journalctl@ceph.mon.c.smithi174.stdout: block_align: 0 2024-04-21T23:09:34.707 INFO:journalctl@ceph.mon.c.smithi174.stdout: max_auto_readahead_size: 262144 2024-04-21T23:09:34.707 INFO:journalctl@ceph.mon.c.smithi174.stdout: prepopulate_block_cache: 0 2024-04-21T23:09:34.707 INFO:journalctl@ceph.mon.c.smithi174.stdout: initial_auto_readahead_size: 8192 2024-04-21T23:09:34.707 INFO:journalctl@ceph.mon.c.smithi174.stdout: num_file_reads_for_auto_readahead: 2 2024-04-21T23:09:34.707 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.write_buffer_size: 33554432 2024-04-21T23:09:34.707 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.max_write_buffer_number: 2 2024-04-21T23:09:34.707 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.compression: NoCompression 2024-04-21T23:09:34.707 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.bottommost_compression: Disabled 2024-04-21T23:09:34.707 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.prefix_extractor: nullptr 2024-04-21T23:09:34.707 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-21T23:09:34.707 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.num_levels: 7 2024-04-21T23:09:34.707 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-21T23:09:34.707 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-21T23:09:34.707 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-21T23:09:34.708 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-21T23:09:34.708 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-21T23:09:34.708 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-21T23:09:34.708 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-21T23:09:34.708 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-21T23:09:34.708 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-21T23:09:34.708 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-21T23:09:34.708 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-21T23:09:34.708 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-21T23:09:34.708 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-21T23:09:34.708 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.compression_opts.level: 32767 2024-04-21T23:09:34.708 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.compression_opts.strategy: 0 2024-04-21T23:09:34.708 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-21T23:09:34.708 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-21T23:09:34.709 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-21T23:09:34.709 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-21T23:09:34.709 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.compression_opts.enabled: false 2024-04-21T23:09:34.709 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-21T23:09:34.709 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-21T23:09:34.709 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-21T23:09:34.709 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-21T23:09:34.709 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.target_file_size_base: 67108864 2024-04-21T23:09:34.709 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-21T23:09:34.709 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-21T23:09:34.709 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-21T23:09:34.709 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-21T23:09:34.709 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-21T23:09:34.709 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-21T23:09:34.710 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-21T23:09:34.710 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-21T23:09:34.710 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-21T23:09:34.710 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-21T23:09:34.710 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-21T23:09:34.710 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-21T23:09:34.710 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-21T23:09:34.710 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-21T23:09:34.710 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.arena_block_size: 1048576 2024-04-21T23:09:34.710 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-21T23:09:34.710 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-21T23:09:34.710 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.disable_auto_compactions: 0 2024-04-21T23:09:34.710 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-21T23:09:34.710 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-21T23:09:34.711 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-21T23:09:34.711 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-21T23:09:34.711 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-21T23:09:34.711 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-21T23:09:34.711 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-21T23:09:34.711 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-21T23:09:34.711 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-21T23:09:34.711 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-21T23:09:34.711 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.table_properties_collectors: 2024-04-21T23:09:34.711 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.inplace_update_support: 0 2024-04-21T23:09:34.711 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-21T23:09:34.711 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-21T23:09:34.711 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-21T23:09:34.711 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-21T23:09:34.711 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.bloom_locality: 0 2024-04-21T23:09:34.712 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.max_successive_merges: 0 2024-04-21T23:09:34.712 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-21T23:09:34.712 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.paranoid_file_checks: 0 2024-04-21T23:09:34.712 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.force_consistency_checks: 1 2024-04-21T23:09:34.712 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.report_bg_io_stats: 0 2024-04-21T23:09:34.712 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.ttl: 2592000 2024-04-21T23:09:34.712 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-21T23:09:34.712 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-21T23:09:34.712 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-21T23:09:34.712 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.enable_blob_files: false 2024-04-21T23:09:34.712 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.min_blob_size: 0 2024-04-21T23:09:34.712 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.blob_file_size: 268435456 2024-04-21T23:09:34.712 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-21T23:09:34.712 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-21T23:09:34.712 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-21T23:09:34.713 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-21T23:09:34.713 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-21T23:09:34.713 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.blob_file_starting_level: 0 2024-04-21T23:09:34.713 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-21T23:09:34.713 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-21T23:09:34.713 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-04-21T23:09:34.713 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: e66904a5-fe1e-498e-9fce-4b86f451d9c3 2024-04-21T23:09:34.713 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713740974522426, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-21T23:09:34.713 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-04-21T23:09:34.713 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713740974522837, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713740974, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "e66904a5-fe1e-498e-9fce-4b86f451d9c3", "db_session_id": "PSBN70Z7WDHCSE4Z68GQ", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-04-21T23:09:34.713 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713740974522915, "job": 1, "event": "recovery_finished"} 2024-04-21T23:09:34.713 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-04-21T23:09:34.713 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-21T23:09:34.713 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55614b836000 2024-04-21T23:09:34.714 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: DB pointer 0x55614b820000 2024-04-21T23:09:34.714 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-21T23:09:34.714 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-21T23:09:34.714 INFO:journalctl@ceph.mon.c.smithi174.stdout: ** DB Stats ** 2024-04-21T23:09:34.714 INFO:journalctl@ceph.mon.c.smithi174.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-21T23:09:34.714 INFO:journalctl@ceph.mon.c.smithi174.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-21T23:09:34.714 INFO:journalctl@ceph.mon.c.smithi174.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-21T23:09:34.714 INFO:journalctl@ceph.mon.c.smithi174.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-21T23:09:34.714 INFO:journalctl@ceph.mon.c.smithi174.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-21T23:09:34.714 INFO:journalctl@ceph.mon.c.smithi174.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-21T23:09:34.714 INFO:journalctl@ceph.mon.c.smithi174.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-21T23:09:34.714 INFO:journalctl@ceph.mon.c.smithi174.stdout: 2024-04-21T23:09:34.714 INFO:journalctl@ceph.mon.c.smithi174.stdout: ** Compaction Stats [default] ** 2024-04-21T23:09:34.714 INFO:journalctl@ceph.mon.c.smithi174.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-21T23:09:34.715 INFO:journalctl@ceph.mon.c.smithi174.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-21T23:09:34.715 INFO:journalctl@ceph.mon.c.smithi174.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-21T23:09:34.715 INFO:journalctl@ceph.mon.c.smithi174.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-21T23:09:34.715 INFO:journalctl@ceph.mon.c.smithi174.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-21T23:09:34.715 INFO:journalctl@ceph.mon.c.smithi174.stdout: 2024-04-21T23:09:34.715 INFO:journalctl@ceph.mon.c.smithi174.stdout: ** Compaction Stats [default] ** 2024-04-21T23:09:34.715 INFO:journalctl@ceph.mon.c.smithi174.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-21T23:09:34.715 INFO:journalctl@ceph.mon.c.smithi174.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-21T23:09:34.715 INFO:journalctl@ceph.mon.c.smithi174.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-21T23:09:34.715 INFO:journalctl@ceph.mon.c.smithi174.stdout: 2024-04-21T23:09:34.715 INFO:journalctl@ceph.mon.c.smithi174.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-21T23:09:34.715 INFO:journalctl@ceph.mon.c.smithi174.stdout: 2024-04-21T23:09:34.715 INFO:journalctl@ceph.mon.c.smithi174.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-21T23:09:34.715 INFO:journalctl@ceph.mon.c.smithi174.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-21T23:09:34.716 INFO:journalctl@ceph.mon.c.smithi174.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-21T23:09:34.716 INFO:journalctl@ceph.mon.c.smithi174.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-21T23:09:34.716 INFO:journalctl@ceph.mon.c.smithi174.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-21T23:09:34.716 INFO:journalctl@ceph.mon.c.smithi174.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-21T23:09:34.716 INFO:journalctl@ceph.mon.c.smithi174.stdout: Cumulative compaction: 0.00 GB write, 0.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-21T23:09:34.716 INFO:journalctl@ceph.mon.c.smithi174.stdout: Interval compaction: 0.00 GB write, 0.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-21T23:09:34.716 INFO:journalctl@ceph.mon.c.smithi174.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-21T23:09:34.716 INFO:journalctl@ceph.mon.c.smithi174.stdout: Block cache BinnedLRUCache@0x55614b745090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-04-21T23:09:34.716 INFO:journalctl@ceph.mon.c.smithi174.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-04-21T23:09:34.716 INFO:journalctl@ceph.mon.c.smithi174.stdout: 2024-04-21T23:09:34.716 INFO:journalctl@ceph.mon.c.smithi174.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-21T23:09:34.716 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-04-21T23:09:34.716 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: using public_addr v2:172.21.15.174:0/0 -> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-04-21T23:09:34.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: starting mon.c rank -1 at public addrs [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] at bind addrs [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid d814e4dc-0033-11ef-bc93-c7b262605968 2024-04-21T23:09:34.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: mon.c@-1(???) e0 preinit fsid d814e4dc-0033-11ef-bc93-c7b262605968 2024-04-21T23:09:34.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: mon.c@-1(synchronizing).mds e1 new map 2024-04-21T23:09:34.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: mon.c@-1(synchronizing).mds e1 print_map 2024-04-21T23:09:34.717 INFO:journalctl@ceph.mon.c.smithi174.stdout: e1 2024-04-21T23:09:34.717 INFO:journalctl@ceph.mon.c.smithi174.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-21T23:09:34.717 INFO:journalctl@ceph.mon.c.smithi174.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-21T23:09:34.717 INFO:journalctl@ceph.mon.c.smithi174.stdout: legacy client fscid: -1 2024-04-21T23:09:34.717 INFO:journalctl@ceph.mon.c.smithi174.stdout: 2024-04-21T23:09:34.717 INFO:journalctl@ceph.mon.c.smithi174.stdout: No filesystems configured 2024-04-21T23:09:34.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-21T23:09:34.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-21T23:09:34.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-21T23:09:34.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-21T23:09:34.717 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-21T23:09:34.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-21T23:09:34.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-21T23:09:34.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-21T23:09:34.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-21T23:09:34.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-21T23:09:34.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:09:34.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T23:09:34.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:34.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:09:34.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:34.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:34.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:34.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:34.718 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi111", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:09:34.719 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:09:34.719 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:09:34.719 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi174", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:09:34.719 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: Updating smithi111:/etc/ceph/ceph.conf 2024-04-21T23:09:34.719 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: Deploying cephadm binary to smithi174 2024-04-21T23:09:34.719 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: Updating smithi111:/var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-21T23:09:34.719 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: Updating smithi111:/etc/ceph/ceph.client.admin.keyring 2024-04-21T23:09:34.719 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:09:34.719 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: Updating smithi111:/var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T23:09:34.719 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:09:34.719 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:34.719 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:34.719 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:34.719 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:34.719 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:09:34.720 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: Added host smithi174 2024-04-21T23:09:34.720 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:34.720 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:34.720 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:09:34.720 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:34.720 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T23:09:34.720 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:09:34.720 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/2094741563' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-21T23:09:34.720 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:34.720 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:34.720 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:34.720 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:34.720 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi174", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:09:34.720 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:09:34.721 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:09:34.721 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:09:34.721 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: Updating smithi174:/etc/ceph/ceph.conf 2024-04-21T23:09:34.721 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/2094741563' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-21T23:09:34.721 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: osdmap e4: 0 total, 0 up, 0 in 2024-04-21T23:09:34.721 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:34.721 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:34.721 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:34.721 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: Updating smithi174:/var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-21T23:09:34.721 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: Updating smithi174:/etc/ceph/ceph.client.admin.keyring 2024-04-21T23:09:34.721 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: Updating smithi174:/var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T23:09:34.721 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:09:34.721 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:34.721 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:09:34.721 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:09:34.722 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:09:34.722 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:34.722 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T23:09:34.722 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:09:34.722 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi040:172.21.15.40=a;smithi111:172.21.15.111=b;smithi174:172.21.15.174=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:09:34.722 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: Saving service mon spec with placement smithi040:172.21.15.40=a;smithi111:172.21.15.111=b;smithi174:172.21.15.174=c;count:3 2024-04-21T23:09:34.722 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: Deploying daemon mon.c on smithi174 2024-04-21T23:09:34.722 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:09:34.722 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:09:34.722 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:34 smithi174 ceph-mon[31409]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-21T23:09:35.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:35 smithi174 bash[31374]: 2500b95b86b5aa92e16def89b40df69f8ef4f99398a445434e904402392d284d 2024-04-21T23:09:35.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:35 smithi174 systemd[1]: Started Ceph mon.c for d814e4dc-0033-11ef-bc93-c7b262605968. 2024-04-21T23:09:36.144 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-21T23:09:36.144 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-21T23:09:36.388 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.c/config 2024-04-21T23:09:39.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:39 smithi040 ceph-mon[27984]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:09:39.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:39 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T23:09:39.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:39 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:09:39.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:39 smithi040 ceph-mon[27984]: mon.a calling monitor election 2024-04-21T23:09:39.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:39 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:09:39.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:39 smithi040 ceph-mon[27984]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:09:39.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:39 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:09:39.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:39 smithi040 ceph-mon[27984]: mon.c calling monitor election 2024-04-21T23:09:39.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:39 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:09:39.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:39 smithi040 ceph-mon[27984]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:09:39.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:39 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:09:39.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:39 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:09:39.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:39 smithi040 ceph-mon[27984]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-21T23:09:39.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:39 smithi040 ceph-mon[27984]: monmap epoch 2 2024-04-21T23:09:39.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:39 smithi040 ceph-mon[27984]: fsid d814e4dc-0033-11ef-bc93-c7b262605968 2024-04-21T23:09:39.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:39 smithi040 ceph-mon[27984]: last_changed 2024-04-21T23:09:34.547977+0000 2024-04-21T23:09:39.934 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:39 smithi040 ceph-mon[27984]: created 2024-04-21T23:07:49.696814+0000 2024-04-21T23:09:39.934 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:39 smithi040 ceph-mon[27984]: min_mon_release 19 (squid) 2024-04-21T23:09:39.934 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:39 smithi040 ceph-mon[27984]: election_strategy: 1 2024-04-21T23:09:39.934 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:39 smithi040 ceph-mon[27984]: 0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-21T23:09:39.934 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:39 smithi040 ceph-mon[27984]: 1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.c 2024-04-21T23:09:39.934 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:39 smithi040 ceph-mon[27984]: fsmap 2024-04-21T23:09:39.934 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:39 smithi040 ceph-mon[27984]: osdmap e4: 0 total, 0 up, 0 in 2024-04-21T23:09:39.934 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:39 smithi040 ceph-mon[27984]: mgrmap e13: x(active, since 49s) 2024-04-21T23:09:39.934 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:39 smithi040 ceph-mon[27984]: overall HEALTH_OK 2024-04-21T23:09:39.934 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:39 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:39.934 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:39 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:39.934 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:39 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:39.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:39 smithi174 ceph-mon[31409]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:09:39.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:39 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T23:09:39.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:39 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:09:39.945 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:39 smithi174 ceph-mon[31409]: mon.a calling monitor election 2024-04-21T23:09:39.945 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:39 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:09:39.945 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:39 smithi174 ceph-mon[31409]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:09:39.945 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:39 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:09:39.945 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:39 smithi174 ceph-mon[31409]: mon.c calling monitor election 2024-04-21T23:09:39.945 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:39 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:09:39.945 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:39 smithi174 ceph-mon[31409]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:09:39.945 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:39 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:09:39.945 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:39 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:09:39.945 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:39 smithi174 ceph-mon[31409]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-21T23:09:39.945 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:39 smithi174 ceph-mon[31409]: monmap epoch 2 2024-04-21T23:09:39.945 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:39 smithi174 ceph-mon[31409]: fsid d814e4dc-0033-11ef-bc93-c7b262605968 2024-04-21T23:09:39.945 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:39 smithi174 ceph-mon[31409]: last_changed 2024-04-21T23:09:34.547977+0000 2024-04-21T23:09:39.945 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:39 smithi174 ceph-mon[31409]: created 2024-04-21T23:07:49.696814+0000 2024-04-21T23:09:39.946 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:39 smithi174 ceph-mon[31409]: min_mon_release 19 (squid) 2024-04-21T23:09:39.946 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:39 smithi174 ceph-mon[31409]: election_strategy: 1 2024-04-21T23:09:39.946 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:39 smithi174 ceph-mon[31409]: 0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-21T23:09:39.946 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:39 smithi174 ceph-mon[31409]: 1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.c 2024-04-21T23:09:39.946 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:39 smithi174 ceph-mon[31409]: fsmap 2024-04-21T23:09:39.946 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:39 smithi174 ceph-mon[31409]: osdmap e4: 0 total, 0 up, 0 in 2024-04-21T23:09:39.946 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:39 smithi174 ceph-mon[31409]: mgrmap e13: x(active, since 49s) 2024-04-21T23:09:39.946 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:39 smithi174 ceph-mon[31409]: overall HEALTH_OK 2024-04-21T23:09:39.946 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:39 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:39.946 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:39 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:39.946 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:39 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:40.800 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:40.800+0000 7f274bf0a700 1 -- 172.21.15.174:0/966838658 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27300035c0 msgr2=0x7f2730005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:40.801 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:40.800+0000 7f274bf0a700 1 --2- 172.21.15.174:0/966838658 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27300035c0 0x7f2730005a50 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f2744100b90 tx=0x7f273400feb0 comp rx=0 tx=0).stop 2024-04-21T23:09:40.801 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:40.800+0000 7f274bf0a700 1 -- 172.21.15.174:0/966838658 shutdown_connections 2024-04-21T23:09:40.801 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:40.800+0000 7f274bf0a700 1 --2- 172.21.15.174:0/966838658 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27300035c0 0x7f2730005a50 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:40.801 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:40.800+0000 7f274bf0a700 1 -- 172.21.15.174:0/966838658 >> 172.21.15.174:0/966838658 conn(0x7f27440fc790 msgr2=0x7f27440febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:40.801 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:40.800+0000 7f274bf0a700 1 -- 172.21.15.174:0/966838658 shutdown_connections 2024-04-21T23:09:40.801 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:40.800+0000 7f274bf0a700 1 -- 172.21.15.174:0/966838658 wait complete. 2024-04-21T23:09:40.805 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:40.801+0000 7f274bf0a700 1 Processor -- start 2024-04-21T23:09:40.805 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:40.801+0000 7f274bf0a700 1 -- start start 2024-04-21T23:09:40.805 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:40.802+0000 7f274bf0a700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27300035c0 0x7f274419d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:40.805 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:40.802+0000 7f274bf0a700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f274419db30 0x7f27441976c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:40.805 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:40.802+0000 7f274bf0a700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2744108cf0 con 0x7f27300035c0 2024-04-21T23:09:40.805 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:40.802+0000 7f274bf0a700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2744108b70 con 0x7f274419db30 2024-04-21T23:09:40.806 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:40.802+0000 7f27494a5700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f274419db30 0x7f27441976c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:40.806 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:40.802+0000 7f2749ca6700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27300035c0 0x7f274419d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:40.806 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:40.802+0000 7f27494a5700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f274419db30 0x7f27441976c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.174:56394/0 (socket says 172.21.15.174:56394) 2024-04-21T23:09:40.806 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:40.802+0000 7f2749ca6700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27300035c0 0x7f274419d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.174:47108/0 (socket says 172.21.15.174:47108) 2024-04-21T23:09:40.806 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:40.802+0000 7f27494a5700 1 -- 172.21.15.174:0/381163099 learned_addr learned my addr 172.21.15.174:0/381163099 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-04-21T23:09:40.806 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:40.802+0000 7f27494a5700 1 -- 172.21.15.174:0/381163099 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f274419db30 msgr2=0x7f27441976c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-21T23:09:40.806 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:40.802+0000 7f27494a5700 1 -- 172.21.15.174:0/381163099 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f274419db30 msgr2=0x7f27441976c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-21T23:09:40.806 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:40.802+0000 7f27494a5700 1 --2- 172.21.15.174:0/381163099 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f274419db30 0x7f27441976c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-21T23:09:40.806 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:40.802+0000 7f27494a5700 1 --2- 172.21.15.174:0/381163099 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f274419db30 0x7f27441976c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T23:09:40.807 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:40.802+0000 7f2749ca6700 1 -- 172.21.15.174:0/381163099 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f274419db30 msgr2=0x7f27441976c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T23:09:40.807 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:40.802+0000 7f2749ca6700 1 --2- 172.21.15.174:0/381163099 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f274419db30 0x7f27441976c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:40.807 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:40.802+0000 7f2749ca6700 1 -- 172.21.15.174:0/381163099 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f274419e0b0 con 0x7f27300035c0 2024-04-21T23:09:40.807 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:40.802+0000 7f2749ca6700 1 --2- 172.21.15.174:0/381163099 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27300035c0 0x7f274419d5f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f2734010010 tx=0x7f27340055a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:40.807 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:40.802+0000 7f273affd700 1 -- 172.21.15.174:0/381163099 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f2734004b20 con 0x7f27300035c0 2024-04-21T23:09:40.807 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:40.803+0000 7f273affd700 1 -- 172.21.15.174:0/381163099 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2734013e40 con 0x7f27300035c0 2024-04-21T23:09:40.807 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:40.803+0000 7f274bf0a700 1 -- 172.21.15.174:0/381163099 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2744197c60 con 0x7f27300035c0 2024-04-21T23:09:40.807 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:40.803+0000 7f273affd700 1 -- 172.21.15.174:0/381163099 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f2734011b60 con 0x7f27300035c0 2024-04-21T23:09:40.807 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:40.803+0000 7f274bf0a700 1 -- 172.21.15.174:0/381163099 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27441981f0 con 0x7f27300035c0 2024-04-21T23:09:40.807 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:40.803+0000 7f273affd700 1 -- 172.21.15.174:0/381163099 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f2734023070 con 0x7f27300035c0 2024-04-21T23:09:40.808 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:40.804+0000 7f273affd700 1 --2- 172.21.15.174:0/381163099 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f2728038ee0 0x7f272803b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:40.808 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:40.804+0000 7f273affd700 1 -- 172.21.15.174:0/381163099 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f2734020050 con 0x7f27300035c0 2024-04-21T23:09:40.808 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:40.804+0000 7f274bf0a700 1 -- 172.21.15.174:0/381163099 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27441023a0 con 0x7f27300035c0 2024-04-21T23:09:40.808 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:40.804+0000 7f27494a5700 1 --2- 172.21.15.174:0/381163099 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f2728038ee0 0x7f272803b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:40.808 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:40.804+0000 7f27494a5700 1 --2- 172.21.15.174:0/381163099 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f2728038ee0 0x7f272803b3a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2740009180 tx=0x7f2740008040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:40.808 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:40.809+0000 7f273affd700 1 -- 172.21.15.174:0/381163099 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f2734028020 con 0x7f27300035c0 2024-04-21T23:09:40.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:40 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T23:09:40.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:40 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:09:40.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:40 smithi040 ceph-mon[27984]: Deploying daemon mon.b on smithi111 2024-04-21T23:09:40.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:40 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:09:40.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:40 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T23:09:40.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:40 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:09:40.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:40 smithi174 ceph-mon[31409]: Deploying daemon mon.b on smithi111 2024-04-21T23:09:40.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:40 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:09:41.093 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:41.093+0000 7f274bf0a700 1 -- 172.21.15.174:0/381163099 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f2744198bf0 con 0x7f27300035c0 2024-04-21T23:09:41.093 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:41.093+0000 7f273affd700 1 -- 172.21.15.174:0/381163099 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7f273402b350 con 0x7f27300035c0 2024-04-21T23:09:41.094 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-04-21T23:09:41.094 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":2,"fsid":"d814e4dc-0033-11ef-bc93-c7b262605968","modified":"2024-04-21T23:09:34.547977Z","created":"2024-04-21T23:07:49.696814Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:3300","nonce":0},{"type":"v1","addr":"172.21.15.174:6789","nonce":0}]},"addr":"172.21.15.174:6789/0","public_addr":"172.21.15.174:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-21T23:09:41.095 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:41.096+0000 7f274bf0a700 1 -- 172.21.15.174:0/381163099 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f2728038ee0 msgr2=0x7f272803b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:41.096 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:41.096+0000 7f274bf0a700 1 --2- 172.21.15.174:0/381163099 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f2728038ee0 0x7f272803b3a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2740009180 tx=0x7f2740008040 comp rx=0 tx=0).stop 2024-04-21T23:09:41.096 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:41.096+0000 7f274bf0a700 1 -- 172.21.15.174:0/381163099 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27300035c0 msgr2=0x7f274419d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:41.096 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:41.096+0000 7f274bf0a700 1 --2- 172.21.15.174:0/381163099 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27300035c0 0x7f274419d5f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f2734010010 tx=0x7f27340055a0 comp rx=0 tx=0).stop 2024-04-21T23:09:41.096 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:41.096+0000 7f274bf0a700 1 -- 172.21.15.174:0/381163099 shutdown_connections 2024-04-21T23:09:41.096 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:41.096+0000 7f274bf0a700 1 --2- 172.21.15.174:0/381163099 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f2728038ee0 0x7f272803b3a0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:41.096 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:41.096+0000 7f274bf0a700 1 --2- 172.21.15.174:0/381163099 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27300035c0 0x7f274419d5f0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:41.096 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:41.096+0000 7f274bf0a700 1 --2- 172.21.15.174:0/381163099 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f274419db30 0x7f27441976c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:41.096 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:41.096+0000 7f274bf0a700 1 -- 172.21.15.174:0/381163099 >> 172.21.15.174:0/381163099 conn(0x7f27440fc790 msgr2=0x7f274406ac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:41.096 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:41.096+0000 7f274bf0a700 1 -- 172.21.15.174:0/381163099 shutdown_connections 2024-04-21T23:09:41.096 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:41.096+0000 7f274bf0a700 1 -- 172.21.15.174:0/381163099 wait complete. 2024-04-21T23:09:41.097 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 2 2024-04-21T23:09:41.671 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:41 smithi174 ceph-mon[31409]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:09:41.672 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:41 smithi174 ceph-mon[31409]: from='client.? 172.21.15.174:0/381163099' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-21T23:09:41.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:41 smithi040 ceph-mon[27984]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:09:41.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:41 smithi040 ceph-mon[27984]: from='client.? 172.21.15.174:0/381163099' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-21T23:09:42.699 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-21T23:09:42.700 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-21T23:09:42.940 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.c/config 2024-04-21T23:09:43.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:43 smithi111 systemd[1]: Started Ceph mon.b for d814e4dc-0033-11ef-bc93-c7b262605968. 2024-04-21T23:09:48.840 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:48.839+0000 7f6afa99a700 1 -- 172.21.15.174:0/2173998752 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6adc0035c0 msgr2=0x7f6adc005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:48.841 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:48.839+0000 7f6afa99a700 1 --2- 172.21.15.174:0/2173998752 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6adc0035c0 0x7f6adc005a50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6af4101c80 tx=0x7f6ae400fe60 comp rx=0 tx=0).stop 2024-04-21T23:09:48.841 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:48.839+0000 7f6afa99a700 1 -- 172.21.15.174:0/2173998752 shutdown_connections 2024-04-21T23:09:48.841 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:48.839+0000 7f6afa99a700 1 --2- 172.21.15.174:0/2173998752 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6adc0035c0 0x7f6adc005a50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:48.841 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:48.839+0000 7f6afa99a700 1 -- 172.21.15.174:0/2173998752 >> 172.21.15.174:0/2173998752 conn(0x7f6af40fd840 msgr2=0x7f6af40ffc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:48.841 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:48.839+0000 7f6afa99a700 1 -- 172.21.15.174:0/2173998752 shutdown_connections 2024-04-21T23:09:48.841 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:48.839+0000 7f6afa99a700 1 -- 172.21.15.174:0/2173998752 wait complete. 2024-04-21T23:09:48.841 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:48.839+0000 7f6afa99a700 1 Processor -- start 2024-04-21T23:09:48.841 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:48.840+0000 7f6afa99a700 1 -- start start 2024-04-21T23:09:48.841 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:48.840+0000 7f6afa99a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6adc0035c0 0x7f6af419d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:48.841 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:48.840+0000 7f6afa99a700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6af419db90 0x7f6af41977d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:48.841 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:48.840+0000 7f6afa99a700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6af4197d10 0x7f6af41981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:48.841 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:48.840+0000 7f6afa99a700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6af4109d90 con 0x7f6af419db90 2024-04-21T23:09:48.842 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:48.840+0000 7f6afa99a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f6af4109c10 con 0x7f6adc0035c0 2024-04-21T23:09:48.842 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:48.840+0000 7f6afa99a700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6af4109f10 con 0x7f6af4197d10 2024-04-21T23:09:48.842 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:48.840+0000 7f6af3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6adc0035c0 0x7f6af419d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:48.842 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:48.840+0000 7f6af8f37700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6af4197d10 0x7f6af41981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:48.842 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:48.840+0000 7f6af3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6adc0035c0 0x7f6af419d650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.174:32948/0 (socket says 172.21.15.174:32948) 2024-04-21T23:09:48.842 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:48.840+0000 7f6af3fff700 1 -- 172.21.15.174:0/2554956151 learned_addr learned my addr 172.21.15.174:0/2554956151 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-04-21T23:09:48.842 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:48.840+0000 7f6af37fe700 1 --2- 172.21.15.174:0/2554956151 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6af419db90 0x7f6af41977d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:48.842 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:48.841+0000 7f6af3fff700 1 -- 172.21.15.174:0/2554956151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6adc0035c0 msgr2=0x7f6af419d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-21T23:09:48.842 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:48.841+0000 7f6af3fff700 1 -- 172.21.15.174:0/2554956151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6adc0035c0 msgr2=0x7f6af419d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-21T23:09:48.842 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:48.841+0000 7f6af3fff700 1 --2- 172.21.15.174:0/2554956151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6adc0035c0 0x7f6af419d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-21T23:09:48.842 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:48.841+0000 7f6af8f37700 1 -- 172.21.15.174:0/2554956151 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6af4197d10 msgr2=0x7f6af41981c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-04-21T23:09:48.842 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:48.841+0000 7f6af8f37700 1 -- 172.21.15.174:0/2554956151 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6af4197d10 msgr2=0x7f6af41981c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-21T23:09:48.842 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:48.841+0000 7f6af3fff700 1 --2- 172.21.15.174:0/2554956151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6adc0035c0 0x7f6af419d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T23:09:48.843 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:48.841+0000 7f6af8f37700 1 --2- 172.21.15.174:0/2554956151 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6af4197d10 0x7f6af41981c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-21T23:09:48.843 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:48.841+0000 7f6af8f37700 1 --2- 172.21.15.174:0/2554956151 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6af4197d10 0x7f6af41981c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T23:09:48.843 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:48.841+0000 7f6af37fe700 1 -- 172.21.15.174:0/2554956151 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6af4197d10 msgr2=0x7f6af41981c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T23:09:48.843 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:48.841+0000 7f6af37fe700 1 --2- 172.21.15.174:0/2554956151 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6af4197d10 0x7f6af41981c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:48.843 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:48.841+0000 7f6af37fe700 1 -- 172.21.15.174:0/2554956151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6adc0035c0 msgr2=0x7f6af419d650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T23:09:48.843 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:48.841+0000 7f6af37fe700 1 --2- 172.21.15.174:0/2554956151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6adc0035c0 0x7f6af419d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:48.843 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:48.841+0000 7f6af37fe700 1 -- 172.21.15.174:0/2554956151 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6af41a44b0 con 0x7f6af419db90 2024-04-21T23:09:48.843 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:48.841+0000 7f6af37fe700 1 --2- 172.21.15.174:0/2554956151 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6af419db90 0x7f6af41977d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6ae800b550 tx=0x7f6ae800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:48.844 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:48.841+0000 7f6af17fa700 1 -- 172.21.15.174:0/2554956151 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ae8004230 con 0x7f6af419db90 2024-04-21T23:09:48.844 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:48.841+0000 7f6af17fa700 1 -- 172.21.15.174:0/2554956151 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ae80043d0 con 0x7f6af419db90 2024-04-21T23:09:48.844 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:48.841+0000 7f6af17fa700 1 -- 172.21.15.174:0/2554956151 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ae8011c20 con 0x7f6af419db90 2024-04-21T23:09:48.844 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:48.841+0000 7f6afa99a700 1 -- 172.21.15.174:0/2554956151 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6af41a4740 con 0x7f6af419db90 2024-04-21T23:09:48.845 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:48.842+0000 7f6afa99a700 1 -- 172.21.15.174:0/2554956151 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6af41a4ca0 con 0x7f6af419db90 2024-04-21T23:09:48.845 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:48.842+0000 7f6af17fa700 1 -- 172.21.15.174:0/2554956151 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f6ae801a400 con 0x7f6af419db90 2024-04-21T23:09:48.845 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:48.843+0000 7f6af17fa700 1 --2- 172.21.15.174:0/2554956151 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f6ae0039580 0x7f6ae003ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:48.845 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:48.843+0000 7f6afa99a700 1 -- 172.21.15.174:0/2554956151 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6af4103490 con 0x7f6af419db90 2024-04-21T23:09:48.845 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:48.843+0000 7f6af17fa700 1 -- 172.21.15.174:0/2554956151 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f6ae804c170 con 0x7f6af419db90 2024-04-21T23:09:48.845 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:48.843+0000 7f6af3fff700 1 --2- 172.21.15.174:0/2554956151 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f6ae0039580 0x7f6ae003ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:48.845 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:48.843+0000 7f6af3fff700 1 --2- 172.21.15.174:0/2554956151 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f6ae0039580 0x7f6ae003ba40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6ae4001eb0 tx=0x7f6ae4005de0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:48.848 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:48.848+0000 7f6af17fa700 1 -- 172.21.15.174:0/2554956151 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f6ae8003c80 con 0x7f6af419db90 2024-04-21T23:09:48.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:48 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T23:09:48.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:48 smithi040 ceph-mon[27984]: mon.a calling monitor election 2024-04-21T23:09:48.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:48 smithi040 ceph-mon[27984]: mon.c calling monitor election 2024-04-21T23:09:48.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:48 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:09:48.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:48 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:09:48.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:48 smithi040 ceph-mon[27984]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:09:48.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:48 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:09:48.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:48 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:09:48.934 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:48 smithi040 ceph-mon[27984]: mon.b calling monitor election 2024-04-21T23:09:48.934 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:48 smithi040 ceph-mon[27984]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:09:48.934 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:48 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:09:48.934 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:48 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:09:48.934 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:48 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:09:48.934 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:48 smithi040 ceph-mon[27984]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-21T23:09:48.934 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:48 smithi040 ceph-mon[27984]: monmap epoch 3 2024-04-21T23:09:48.934 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:48 smithi040 ceph-mon[27984]: fsid d814e4dc-0033-11ef-bc93-c7b262605968 2024-04-21T23:09:48.935 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:48 smithi040 ceph-mon[27984]: last_changed 2024-04-21T23:09:43.585393+0000 2024-04-21T23:09:48.935 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:48 smithi040 ceph-mon[27984]: created 2024-04-21T23:07:49.696814+0000 2024-04-21T23:09:48.935 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:48 smithi040 ceph-mon[27984]: min_mon_release 19 (squid) 2024-04-21T23:09:48.935 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:48 smithi040 ceph-mon[27984]: election_strategy: 1 2024-04-21T23:09:48.935 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:48 smithi040 ceph-mon[27984]: 0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-21T23:09:48.935 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:48 smithi040 ceph-mon[27984]: 1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.c 2024-04-21T23:09:48.935 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:48 smithi040 ceph-mon[27984]: 2: [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] mon.b 2024-04-21T23:09:48.935 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:48 smithi040 ceph-mon[27984]: fsmap 2024-04-21T23:09:48.936 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:48 smithi040 ceph-mon[27984]: osdmap e4: 0 total, 0 up, 0 in 2024-04-21T23:09:48.936 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:48 smithi040 ceph-mon[27984]: mgrmap e13: x(active, since 58s) 2024-04-21T23:09:48.936 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:48 smithi040 ceph-mon[27984]: overall HEALTH_OK 2024-04-21T23:09:48.936 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:48 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:48.936 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:48 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:48.936 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:48 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:48.936 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:48 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:48.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:48 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T23:09:48.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:48 smithi174 ceph-mon[31409]: mon.a calling monitor election 2024-04-21T23:09:48.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:48 smithi174 ceph-mon[31409]: mon.c calling monitor election 2024-04-21T23:09:48.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:48 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:09:48.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:48 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:09:48.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:48 smithi174 ceph-mon[31409]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:09:48.945 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:48 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:09:48.945 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:48 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:09:48.945 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:48 smithi174 ceph-mon[31409]: mon.b calling monitor election 2024-04-21T23:09:48.945 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:48 smithi174 ceph-mon[31409]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:09:48.945 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:48 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:09:48.945 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:48 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:09:48.945 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:48 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:09:48.945 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:48 smithi174 ceph-mon[31409]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-21T23:09:48.945 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:48 smithi174 ceph-mon[31409]: monmap epoch 3 2024-04-21T23:09:48.945 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:48 smithi174 ceph-mon[31409]: fsid d814e4dc-0033-11ef-bc93-c7b262605968 2024-04-21T23:09:48.945 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:48 smithi174 ceph-mon[31409]: last_changed 2024-04-21T23:09:43.585393+0000 2024-04-21T23:09:48.945 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:48 smithi174 ceph-mon[31409]: created 2024-04-21T23:07:49.696814+0000 2024-04-21T23:09:48.945 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:48 smithi174 ceph-mon[31409]: min_mon_release 19 (squid) 2024-04-21T23:09:48.945 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:48 smithi174 ceph-mon[31409]: election_strategy: 1 2024-04-21T23:09:48.946 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:48 smithi174 ceph-mon[31409]: 0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-21T23:09:48.946 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:48 smithi174 ceph-mon[31409]: 1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.c 2024-04-21T23:09:48.946 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:48 smithi174 ceph-mon[31409]: 2: [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] mon.b 2024-04-21T23:09:48.946 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:48 smithi174 ceph-mon[31409]: fsmap 2024-04-21T23:09:48.946 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:48 smithi174 ceph-mon[31409]: osdmap e4: 0 total, 0 up, 0 in 2024-04-21T23:09:48.946 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:48 smithi174 ceph-mon[31409]: mgrmap e13: x(active, since 58s) 2024-04-21T23:09:48.946 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:48 smithi174 ceph-mon[31409]: overall HEALTH_OK 2024-04-21T23:09:48.946 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:48 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:48.947 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:48 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:48.947 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:48 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:48.947 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:48 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:48.947 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:48 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T23:09:48.947 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:48 smithi111 ceph-mon[31085]: mon.a calling monitor election 2024-04-21T23:09:48.947 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:48 smithi111 ceph-mon[31085]: mon.c calling monitor election 2024-04-21T23:09:48.947 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:48 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:09:48.948 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:48 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:09:48.948 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:48 smithi111 ceph-mon[31085]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:09:48.948 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:48 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:09:48.948 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:48 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:09:48.948 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:48 smithi111 ceph-mon[31085]: mon.b calling monitor election 2024-04-21T23:09:48.948 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:48 smithi111 ceph-mon[31085]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:09:48.948 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:48 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:09:48.948 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:48 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:09:48.948 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:48 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:09:48.948 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:48 smithi111 ceph-mon[31085]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-21T23:09:48.948 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:48 smithi111 ceph-mon[31085]: monmap epoch 3 2024-04-21T23:09:48.948 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:48 smithi111 ceph-mon[31085]: fsid d814e4dc-0033-11ef-bc93-c7b262605968 2024-04-21T23:09:48.948 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:48 smithi111 ceph-mon[31085]: last_changed 2024-04-21T23:09:43.585393+0000 2024-04-21T23:09:48.948 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:48 smithi111 ceph-mon[31085]: created 2024-04-21T23:07:49.696814+0000 2024-04-21T23:09:48.948 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:48 smithi111 ceph-mon[31085]: min_mon_release 19 (squid) 2024-04-21T23:09:48.949 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:48 smithi111 ceph-mon[31085]: election_strategy: 1 2024-04-21T23:09:48.949 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:48 smithi111 ceph-mon[31085]: 0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-21T23:09:48.949 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:48 smithi111 ceph-mon[31085]: 1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.c 2024-04-21T23:09:48.949 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:48 smithi111 ceph-mon[31085]: 2: [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] mon.b 2024-04-21T23:09:48.949 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:48 smithi111 ceph-mon[31085]: fsmap 2024-04-21T23:09:48.949 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:48 smithi111 ceph-mon[31085]: osdmap e4: 0 total, 0 up, 0 in 2024-04-21T23:09:48.949 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:48 smithi111 ceph-mon[31085]: mgrmap e13: x(active, since 58s) 2024-04-21T23:09:48.949 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:48 smithi111 ceph-mon[31085]: overall HEALTH_OK 2024-04-21T23:09:48.949 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:48 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:48.949 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:48 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:48.949 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:48 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:48.949 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:48 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:49.138 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:49.138+0000 7f6afa99a700 1 -- 172.21.15.174:0/2554956151 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f6af41a4f60 con 0x7f6af419db90 2024-04-21T23:09:49.138 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:49.139+0000 7f6af17fa700 1 -- 172.21.15.174:0/2554956151 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1285 (secure 0 0 0) 0x7f6ae801a790 con 0x7f6af419db90 2024-04-21T23:09:49.138 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-04-21T23:09:49.139 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":3,"fsid":"d814e4dc-0033-11ef-bc93-c7b262605968","modified":"2024-04-21T23:09:43.585393Z","created":"2024-04-21T23:07:49.696814Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:3300","nonce":0},{"type":"v1","addr":"172.21.15.174:6789","nonce":0}]},"addr":"172.21.15.174:6789/0","public_addr":"172.21.15.174:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-04-21T23:09:49.141 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:49.141+0000 7f6afa99a700 1 -- 172.21.15.174:0/2554956151 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f6ae0039580 msgr2=0x7f6ae003ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:49.141 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:49.141+0000 7f6afa99a700 1 --2- 172.21.15.174:0/2554956151 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f6ae0039580 0x7f6ae003ba40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6ae4001eb0 tx=0x7f6ae4005de0 comp rx=0 tx=0).stop 2024-04-21T23:09:49.141 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:49.141+0000 7f6afa99a700 1 -- 172.21.15.174:0/2554956151 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6af419db90 msgr2=0x7f6af41977d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:49.141 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:49.141+0000 7f6afa99a700 1 --2- 172.21.15.174:0/2554956151 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6af419db90 0x7f6af41977d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6ae800b550 tx=0x7f6ae800b910 comp rx=0 tx=0).stop 2024-04-21T23:09:49.141 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:49.142+0000 7f6afa99a700 1 -- 172.21.15.174:0/2554956151 shutdown_connections 2024-04-21T23:09:49.142 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:49.142+0000 7f6afa99a700 1 --2- 172.21.15.174:0/2554956151 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6af4197d10 0x7f6af41981c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:49.142 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:49.142+0000 7f6afa99a700 1 --2- 172.21.15.174:0/2554956151 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f6ae0039580 0x7f6ae003ba40 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:49.142 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:49.142+0000 7f6afa99a700 1 --2- 172.21.15.174:0/2554956151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6adc0035c0 0x7f6af419d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:49.142 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:49.142+0000 7f6afa99a700 1 --2- 172.21.15.174:0/2554956151 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6af419db90 0x7f6af41977d0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:49.142 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:49.142+0000 7f6afa99a700 1 -- 172.21.15.174:0/2554956151 >> 172.21.15.174:0/2554956151 conn(0x7f6af40fd840 msgr2=0x7f6af4105990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:49.142 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:49.142+0000 7f6afa99a700 1 -- 172.21.15.174:0/2554956151 shutdown_connections 2024-04-21T23:09:49.142 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:49.142+0000 7f6afa99a700 1 -- 172.21.15.174:0/2554956151 wait complete. 2024-04-21T23:09:49.142 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 3 2024-04-21T23:09:49.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:49 smithi174 ceph-mon[31409]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:09:49.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:49 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:09:49.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:49 smithi174 ceph-mon[31409]: from='client.? 172.21.15.174:0/2554956151' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-21T23:09:49.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:49 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:09:49.761 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-21T23:09:49.761 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph config generate-minimal-conf 2024-04-21T23:09:49.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:49 smithi040 ceph-mon[27984]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:09:49.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:49 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:09:49.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:49 smithi040 ceph-mon[27984]: from='client.? 172.21.15.174:0/2554956151' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-21T23:09:49.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:49 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:09:49.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:49 smithi111 ceph-mon[31085]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:09:49.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:49 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:09:49.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:49 smithi111 ceph-mon[31085]: from='client.? 172.21.15.174:0/2554956151' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-21T23:09:49.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:49 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:09:50.016 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:09:51.328 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:51 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:51.328 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:51 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:51.328 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:51 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:51.328 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:51 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:51.328 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:51 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:51.328 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:51 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:09:51.328 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:51 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:09:51.329 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:51 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:51.329 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:51 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:51.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:51 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:51.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:51 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:51.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:51 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:51.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:51 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:51.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:51 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:51.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:51 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:09:51.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:51 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:09:51.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:51 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:51.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:51 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:51.446 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:51 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:51.446 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:51 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:51.446 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:51 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:51.446 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:51 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:51.446 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:51 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:51.446 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:51 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:09:51.447 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:51 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:09:51.447 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:51 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:51.447 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:51 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:51.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:51.666+0000 7fefe279c700 1 -- 172.21.15.40:0/1535697082 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fefdc106bf0 msgr2=0x7fefdc106fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:51.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:51.666+0000 7fefe279c700 1 --2- 172.21.15.40:0/1535697082 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fefdc106bf0 0x7fefdc106fd0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fefcc009910 tx=0x7fefcc009c20 comp rx=0 tx=0).stop 2024-04-21T23:09:51.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:51.666+0000 7fefe279c700 1 -- 172.21.15.40:0/1535697082 shutdown_connections 2024-04-21T23:09:51.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:51.666+0000 7fefe279c700 1 --2- 172.21.15.40:0/1535697082 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fefdc106bf0 0x7fefdc106fd0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:51.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:51.666+0000 7fefe279c700 1 -- 172.21.15.40:0/1535697082 >> 172.21.15.40:0/1535697082 conn(0x7fefdc0fc630 msgr2=0x7fefdc0fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:51.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:51.666+0000 7fefe279c700 1 -- 172.21.15.40:0/1535697082 shutdown_connections 2024-04-21T23:09:51.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:51.666+0000 7fefe279c700 1 -- 172.21.15.40:0/1535697082 wait complete. 2024-04-21T23:09:51.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:51.666+0000 7fefe279c700 1 Processor -- start 2024-04-21T23:09:51.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:51.667+0000 7fefe279c700 1 -- start start 2024-04-21T23:09:51.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:51.667+0000 7fefe279c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fefdc106bf0 0x7fefdc1987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:51.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:51.667+0000 7fefe279c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fefdc198ce0 0x7fefdc199160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:51.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:51.667+0000 7fefe279c700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fefdc1a1810 0x7fefdc1a3c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:51.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:51.667+0000 7fefe279c700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fefdc10b160 con 0x7fefdc198ce0 2024-04-21T23:09:51.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:51.667+0000 7fefe279c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fefdc10afe0 con 0x7fefdc106bf0 2024-04-21T23:09:51.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:51.667+0000 7fefe279c700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fefdc10b2e0 con 0x7fefdc1a1810 2024-04-21T23:09:51.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:51.667+0000 7fefdb7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fefdc198ce0 0x7fefdc199160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:51.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:51.667+0000 7fefdb7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fefdc198ce0 0x7fefdc199160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50178/0 (socket says 172.21.15.40:50178) 2024-04-21T23:09:51.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:51.667+0000 7fefdb7fe700 1 -- 172.21.15.40:0/1370383332 learned_addr learned my addr 172.21.15.40:0/1370383332 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:09:51.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:51.667+0000 7fefdbfff700 1 --2- 172.21.15.40:0/1370383332 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fefdc106bf0 0x7fefdc1987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:51.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:51.667+0000 7fefe0d39700 1 --2- 172.21.15.40:0/1370383332 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fefdc1a1810 0x7fefdc1a3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:51.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:51.667+0000 7fefdb7fe700 1 -- 172.21.15.40:0/1370383332 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fefdc1a1810 msgr2=0x7fefdc1a3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:51.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:51.667+0000 7fefdb7fe700 1 --2- 172.21.15.40:0/1370383332 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fefdc1a1810 0x7fefdc1a3c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:51.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:51.667+0000 7fefdb7fe700 1 -- 172.21.15.40:0/1370383332 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fefdc106bf0 msgr2=0x7fefdc1987a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:51.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:51.667+0000 7fefdb7fe700 1 --2- 172.21.15.40:0/1370383332 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fefdc106bf0 0x7fefdc1987a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:51.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:51.667+0000 7fefdb7fe700 1 -- 172.21.15.40:0/1370383332 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefdc1a4470 con 0x7fefdc198ce0 2024-04-21T23:09:51.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:51.667+0000 7fefe0d39700 1 --2- 172.21.15.40:0/1370383332 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fefdc1a1810 0x7fefdc1a3c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:09:51.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:51.668+0000 7fefdb7fe700 1 --2- 172.21.15.40:0/1370383332 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fefdc198ce0 0x7fefdc199160 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fefd000b780 tx=0x7fefd000ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:51.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:51.668+0000 7fefd97fa700 1 -- 172.21.15.40:0/1370383332 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefd0004230 con 0x7fefdc198ce0 2024-04-21T23:09:51.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:51.668+0000 7fefd97fa700 1 -- 172.21.15.40:0/1370383332 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fefd00043d0 con 0x7fefdc198ce0 2024-04-21T23:09:51.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:51.668+0000 7fefd97fa700 1 -- 172.21.15.40:0/1370383332 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefd0011c90 con 0x7fefdc198ce0 2024-04-21T23:09:51.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:51.668+0000 7fefe279c700 1 -- 172.21.15.40:0/1370383332 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefdc1a4760 con 0x7fefdc198ce0 2024-04-21T23:09:51.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:51.668+0000 7fefe279c700 1 -- 172.21.15.40:0/1370383332 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefdc1a4c50 con 0x7fefdc198ce0 2024-04-21T23:09:51.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:51.669+0000 7fefd97fa700 1 -- 172.21.15.40:0/1370383332 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7fefd001a400 con 0x7fefdc198ce0 2024-04-21T23:09:51.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:51.669+0000 7fefd97fa700 1 --2- 172.21.15.40:0/1370383332 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fefc4039580 0x7fefc403ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:51.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:51.669+0000 7fefd97fa700 1 -- 172.21.15.40:0/1370383332 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fefd004bc90 con 0x7fefdc198ce0 2024-04-21T23:09:51.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:51.669+0000 7fefe279c700 1 -- 172.21.15.40:0/1370383332 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fefdc063890 con 0x7fefdc198ce0 2024-04-21T23:09:51.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:51.670+0000 7fefdbfff700 1 --2- 172.21.15.40:0/1370383332 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fefc4039580 0x7fefc403ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:51.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:51.670+0000 7fefdbfff700 1 --2- 172.21.15.40:0/1370383332 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fefc4039580 0x7fefc403ba40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fefcc009750 tx=0x7fefcc00b540 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:51.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:51.675+0000 7fefd97fa700 1 -- 172.21.15.40:0/1370383332 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fefd00151c0 con 0x7fefdc198ce0 2024-04-21T23:09:51.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:51.883+0000 7fefe279c700 1 -- 172.21.15.40:0/1370383332 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7fefdc1a5130 con 0x7fefdc198ce0 2024-04-21T23:09:51.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:51.883+0000 7fefd97fa700 1 -- 172.21.15.40:0/1370383332 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+275 (secure 0 0 0) 0x7fefd0018750 con 0x7fefdc198ce0 2024-04-21T23:09:51.882 INFO:teuthology.orchestra.run.smithi040.stdout:# minimal ceph.conf for d814e4dc-0033-11ef-bc93-c7b262605968 2024-04-21T23:09:51.883 INFO:teuthology.orchestra.run.smithi040.stdout:[global] 2024-04-21T23:09:51.883 INFO:teuthology.orchestra.run.smithi040.stdout: fsid = d814e4dc-0033-11ef-bc93-c7b262605968 2024-04-21T23:09:51.883 INFO:teuthology.orchestra.run.smithi040.stdout: mon_host = [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-04-21T23:09:51.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:51.885+0000 7fefe279c700 1 -- 172.21.15.40:0/1370383332 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fefc4039580 msgr2=0x7fefc403ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:51.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:51.885+0000 7fefe279c700 1 --2- 172.21.15.40:0/1370383332 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fefc4039580 0x7fefc403ba40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fefcc009750 tx=0x7fefcc00b540 comp rx=0 tx=0).stop 2024-04-21T23:09:51.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:51.885+0000 7fefe279c700 1 -- 172.21.15.40:0/1370383332 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fefdc198ce0 msgr2=0x7fefdc199160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:51.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:51.885+0000 7fefe279c700 1 --2- 172.21.15.40:0/1370383332 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fefdc198ce0 0x7fefdc199160 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fefd000b780 tx=0x7fefd000ba90 comp rx=0 tx=0).stop 2024-04-21T23:09:51.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:51.885+0000 7fefe279c700 1 -- 172.21.15.40:0/1370383332 shutdown_connections 2024-04-21T23:09:51.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:51.885+0000 7fefe279c700 1 --2- 172.21.15.40:0/1370383332 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fefdc1a1810 0x7fefdc1a3c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:51.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:51.885+0000 7fefe279c700 1 --2- 172.21.15.40:0/1370383332 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fefc4039580 0x7fefc403ba40 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:51.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:51.885+0000 7fefe279c700 1 --2- 172.21.15.40:0/1370383332 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fefdc106bf0 0x7fefdc1987a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:51.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:51.885+0000 7fefe279c700 1 --2- 172.21.15.40:0/1370383332 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fefdc198ce0 0x7fefdc199160 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:51.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:51.885+0000 7fefe279c700 1 -- 172.21.15.40:0/1370383332 >> 172.21.15.40:0/1370383332 conn(0x7fefdc0fc630 msgr2=0x7fefdc0fca10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:51.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:51.885+0000 7fefe279c700 1 -- 172.21.15.40:0/1370383332 shutdown_connections 2024-04-21T23:09:51.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:09:51.885+0000 7fefe279c700 1 -- 172.21.15.40:0/1370383332 wait complete. 2024-04-21T23:09:52.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:52 smithi040 ceph-mon[27984]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:09:52.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:52 smithi040 ceph-mon[27984]: Updating smithi040:/etc/ceph/ceph.conf 2024-04-21T23:09:52.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:52 smithi040 ceph-mon[27984]: Updating smithi111:/etc/ceph/ceph.conf 2024-04-21T23:09:52.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:52 smithi040 ceph-mon[27984]: Updating smithi174:/etc/ceph/ceph.conf 2024-04-21T23:09:52.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:52 smithi040 ceph-mon[27984]: Updating smithi040:/var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-21T23:09:52.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:52 smithi040 ceph-mon[27984]: Updating smithi174:/var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-21T23:09:52.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:52 smithi040 ceph-mon[27984]: Updating smithi111:/var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-21T23:09:52.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:52 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:52.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:52 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:52.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:52 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:52.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:52 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:52.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:52 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:52.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:52 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:52.185 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:52 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:52.185 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:52 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:52.185 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:52 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:52.185 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:52 smithi040 ceph-mon[27984]: Reconfiguring mon.a (unknown last config time)... 2024-04-21T23:09:52.185 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:52 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T23:09:52.185 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:52 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-21T23:09:52.185 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:52 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:09:52.185 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:52 smithi040 ceph-mon[27984]: Reconfiguring daemon mon.a on smithi040 2024-04-21T23:09:52.186 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:52 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/1370383332' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:09:52.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:52 smithi174 ceph-mon[31409]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:09:52.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:52 smithi174 ceph-mon[31409]: Updating smithi040:/etc/ceph/ceph.conf 2024-04-21T23:09:52.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:52 smithi174 ceph-mon[31409]: Updating smithi111:/etc/ceph/ceph.conf 2024-04-21T23:09:52.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:52 smithi174 ceph-mon[31409]: Updating smithi174:/etc/ceph/ceph.conf 2024-04-21T23:09:52.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:52 smithi174 ceph-mon[31409]: Updating smithi040:/var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-21T23:09:52.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:52 smithi174 ceph-mon[31409]: Updating smithi174:/var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-21T23:09:52.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:52 smithi174 ceph-mon[31409]: Updating smithi111:/var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-21T23:09:52.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:52 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:52.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:52 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:52.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:52 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:52.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:52 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:52.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:52 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:52.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:52 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:52.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:52 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:52.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:52 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:52.446 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:52 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:52.446 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:52 smithi174 ceph-mon[31409]: Reconfiguring mon.a (unknown last config time)... 2024-04-21T23:09:52.446 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:52 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T23:09:52.446 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:52 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-21T23:09:52.446 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:52 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:09:52.446 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:52 smithi174 ceph-mon[31409]: Reconfiguring daemon mon.a on smithi040 2024-04-21T23:09:52.446 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:52 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/1370383332' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:09:52.446 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:52 smithi111 ceph-mon[31085]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:09:52.446 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:52 smithi111 ceph-mon[31085]: Updating smithi040:/etc/ceph/ceph.conf 2024-04-21T23:09:52.446 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:52 smithi111 ceph-mon[31085]: Updating smithi111:/etc/ceph/ceph.conf 2024-04-21T23:09:52.446 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:52 smithi111 ceph-mon[31085]: Updating smithi174:/etc/ceph/ceph.conf 2024-04-21T23:09:52.446 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:52 smithi111 ceph-mon[31085]: Updating smithi040:/var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-21T23:09:52.446 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:52 smithi111 ceph-mon[31085]: Updating smithi174:/var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-21T23:09:52.446 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:52 smithi111 ceph-mon[31085]: Updating smithi111:/var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-21T23:09:52.447 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:52 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:52.447 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:52 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:52.447 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:52 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:52.447 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:52 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:52.447 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:52 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:52.447 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:52 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:52.447 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:52 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:52.447 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:52 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:52.447 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:52 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:52.447 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:52 smithi111 ceph-mon[31085]: Reconfiguring mon.a (unknown last config time)... 2024-04-21T23:09:52.447 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:52 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T23:09:52.447 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:52 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-21T23:09:52.447 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:52 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:09:52.447 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:52 smithi111 ceph-mon[31085]: Reconfiguring daemon mon.a on smithi040 2024-04-21T23:09:52.447 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:52 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/1370383332' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:09:53.220 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-21T23:09:53.220 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-21T23:09:53.220 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/ceph/ceph.conf 2024-04-21T23:09:53.247 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-21T23:09:53.247 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-21T23:09:53.311 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-04-21T23:09:53.311 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/ceph/ceph.conf 2024-04-21T23:09:53.337 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-04-21T23:09:53.337 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-21T23:09:53.400 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-04-21T23:09:53.400 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/etc/ceph/ceph.conf 2024-04-21T23:09:53.431 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-04-21T23:09:53.432 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-21T23:09:53.499 INFO:tasks.cephadm:Adding mgr.x on smithi040 2024-04-21T23:09:53.499 INFO:tasks.cephadm:Adding mgr.y on smithi111 2024-04-21T23:09:53.499 INFO:tasks.cephadm:Adding mgr.z on smithi174 2024-04-21T23:09:53.500 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph orch apply mgr '3;smithi040=x;smithi111=y;smithi174=z' 2024-04-21T23:09:53.774 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.c/config 2024-04-21T23:09:54.004 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:53 smithi174 ceph-mon[31409]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:09:54.005 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:53 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:54.005 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:53 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:54.005 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:53 smithi174 ceph-mon[31409]: Reconfiguring mon.b (monmap changed)... 2024-04-21T23:09:54.005 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:53 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T23:09:54.005 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:53 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-21T23:09:54.005 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:53 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:09:54.005 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:53 smithi174 ceph-mon[31409]: Reconfiguring daemon mon.b on smithi111 2024-04-21T23:09:54.084 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:53 smithi111 ceph-mon[31085]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:09:54.084 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:53 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:54.084 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:53 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:54.084 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:53 smithi111 ceph-mon[31085]: Reconfiguring mon.b (monmap changed)... 2024-04-21T23:09:54.084 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:53 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T23:09:54.084 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:53 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-21T23:09:54.084 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:53 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:09:54.084 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:53 smithi111 ceph-mon[31085]: Reconfiguring daemon mon.b on smithi111 2024-04-21T23:09:54.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:53 smithi040 ceph-mon[27984]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:09:54.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:53 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:54.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:53 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:54.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:53 smithi040 ceph-mon[27984]: Reconfiguring mon.b (monmap changed)... 2024-04-21T23:09:54.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:53 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T23:09:54.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:53 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-21T23:09:54.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:53 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:09:54.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:53 smithi040 ceph-mon[27984]: Reconfiguring daemon mon.b on smithi111 2024-04-21T23:09:55.423 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:55 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:55.423 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:55 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:55.424 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:55 smithi174 ceph-mon[31409]: Reconfiguring mon.c (monmap changed)... 2024-04-21T23:09:55.424 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:55 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T23:09:55.424 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:55 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-21T23:09:55.424 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:55 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:09:55.424 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:55 smithi174 ceph-mon[31409]: Reconfiguring daemon mon.c on smithi174 2024-04-21T23:09:55.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:55 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:55.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:55 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:55.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:55 smithi040 ceph-mon[27984]: Reconfiguring mon.c (monmap changed)... 2024-04-21T23:09:55.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:55 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T23:09:55.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:55 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-21T23:09:55.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:55 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:09:55.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:55 smithi040 ceph-mon[27984]: Reconfiguring daemon mon.c on smithi174 2024-04-21T23:09:55.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:55 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:55.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:55 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:55.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:55 smithi111 ceph-mon[31085]: Reconfiguring mon.c (monmap changed)... 2024-04-21T23:09:55.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:55 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T23:09:55.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:55 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-21T23:09:55.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:55 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:09:55.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:55 smithi111 ceph-mon[31085]: Reconfiguring daemon mon.c on smithi174 2024-04-21T23:09:55.917 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:55.525+0000 7f5807875700 1 -- 172.21.15.174:0/3377175925 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5800101dc0 msgr2=0x7f58001021e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:55.917 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:55.525+0000 7f5807875700 1 --2- 172.21.15.174:0/3377175925 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5800101dc0 0x7f58001021e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f57fc009910 tx=0x7f57fc009c20 comp rx=0 tx=0).stop 2024-04-21T23:09:55.917 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:55.525+0000 7f5807875700 1 -- 172.21.15.174:0/3377175925 shutdown_connections 2024-04-21T23:09:55.917 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:55.525+0000 7f5807875700 1 --2- 172.21.15.174:0/3377175925 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5800101dc0 0x7f58001021e0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:55.917 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:55.525+0000 7f5807875700 1 -- 172.21.15.174:0/3377175925 >> 172.21.15.174:0/3377175925 conn(0x7f58000fd360 msgr2=0x7f58000ff7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:55.918 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:55.525+0000 7f5807875700 1 -- 172.21.15.174:0/3377175925 shutdown_connections 2024-04-21T23:09:55.918 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:55.525+0000 7f5807875700 1 -- 172.21.15.174:0/3377175925 wait complete. 2024-04-21T23:09:55.918 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:55.526+0000 7f5807875700 1 Processor -- start 2024-04-21T23:09:55.918 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:55.526+0000 7f5807875700 1 -- start start 2024-04-21T23:09:55.918 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:55.526+0000 7f5807875700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5800101dc0 0x7f5800197670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:55.918 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:55.526+0000 7f5807875700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5800197bb0 0x7f580019cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:55.918 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:55.526+0000 7f5807875700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5800198030 0x7f580019f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:55.918 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:55.526+0000 7f5807875700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5800107e00 con 0x7f5800101dc0 2024-04-21T23:09:55.918 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:55.526+0000 7f5807875700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f5800107c80 con 0x7f5800197bb0 2024-04-21T23:09:55.918 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:55.526+0000 7f5807875700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5800107f80 con 0x7f5800198030 2024-04-21T23:09:55.918 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:55.526+0000 7f5805e12700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5800198030 0x7f580019f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:55.918 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:55.526+0000 7f5805611700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5800101dc0 0x7f5800197670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:55.918 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:55.526+0000 7f5805e12700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5800198030 0x7f580019f1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.174:54818/0 (socket says 172.21.15.174:54818) 2024-04-21T23:09:55.919 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:55.526+0000 7f5805611700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5800101dc0 0x7f5800197670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.174:49702/0 (socket says 172.21.15.174:49702) 2024-04-21T23:09:55.919 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:55.526+0000 7f5805e12700 1 -- 172.21.15.174:0/1828948489 learned_addr learned my addr 172.21.15.174:0/1828948489 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-04-21T23:09:55.919 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:55.527+0000 7f5804e10700 1 --2- 172.21.15.174:0/1828948489 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5800197bb0 0x7f580019cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:55.919 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:55.527+0000 7f5805e12700 1 -- 172.21.15.174:0/1828948489 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5800197bb0 msgr2=0x7f580019cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:55.919 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:55.527+0000 7f5805e12700 1 --2- 172.21.15.174:0/1828948489 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5800197bb0 0x7f580019cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:55.919 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:55.527+0000 7f5805e12700 1 -- 172.21.15.174:0/1828948489 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5800101dc0 msgr2=0x7f5800197670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:55.919 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:55.527+0000 7f5805e12700 1 --2- 172.21.15.174:0/1828948489 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5800101dc0 0x7f5800197670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:55.919 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:55.527+0000 7f5805e12700 1 -- 172.21.15.174:0/1828948489 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f580019fa80 con 0x7f5800198030 2024-04-21T23:09:55.919 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:55.527+0000 7f5805611700 1 --2- 172.21.15.174:0/1828948489 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5800101dc0 0x7f5800197670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:09:55.919 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:55.527+0000 7f5805e12700 1 --2- 172.21.15.174:0/1828948489 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5800198030 0x7f580019f1b0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f57f800ba50 tx=0x7f57f800bd60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:55.919 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:55.527+0000 7f57f67fc700 1 -- 172.21.15.174:0/1828948489 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57f8019070 con 0x7f5800198030 2024-04-21T23:09:55.919 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:55.527+0000 7f57f67fc700 1 -- 172.21.15.174:0/1828948489 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f57f8014440 con 0x7f5800198030 2024-04-21T23:09:55.919 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:55.527+0000 7f5807875700 1 -- 172.21.15.174:0/1828948489 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f580019fd70 con 0x7f5800198030 2024-04-21T23:09:55.920 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:55.527+0000 7f57f67fc700 1 -- 172.21.15.174:0/1828948489 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57f80075b0 con 0x7f5800198030 2024-04-21T23:09:55.920 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:55.527+0000 7f5807875700 1 -- 172.21.15.174:0/1828948489 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58001a0200 con 0x7f5800198030 2024-04-21T23:09:55.920 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:55.528+0000 7f57f67fc700 1 -- 172.21.15.174:0/1828948489 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f57f801c070 con 0x7f5800198030 2024-04-21T23:09:55.920 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:55.528+0000 7f5807875700 1 -- 172.21.15.174:0/1828948489 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5800063890 con 0x7f5800198030 2024-04-21T23:09:55.920 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:55.528+0000 7f57f67fc700 1 --2- 172.21.15.174:0/1828948489 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f57ec039580 0x7f57ec03ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:55.920 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:55.528+0000 7f57f67fc700 1 -- 172.21.15.174:0/1828948489 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f57f804b8d0 con 0x7f5800198030 2024-04-21T23:09:55.920 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:55.528+0000 7f5805611700 1 --2- 172.21.15.174:0/1828948489 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f57ec039580 0x7f57ec03ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:55.920 INFO:teuthology.orchestra.run.smithi174.stdout:Scheduled mgr update... 2024-04-21T23:09:55.920 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:55.529+0000 7f5805611700 1 --2- 172.21.15.174:0/1828948489 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f57ec039580 0x7f57ec03ba40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f57fc005200 tx=0x7f57fc01a040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:55.921 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:55.533+0000 7f57f67fc700 1 -- 172.21.15.174:0/1828948489 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f57f8012500 con 0x7f5800198030 2024-04-21T23:09:55.921 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:55.746+0000 7f5807875700 1 -- 172.21.15.174:0/1828948489 --> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi040=x;smithi111=y;smithi174=z", "target": ["mon-mgr", ""]}) -- 0x7f5800062670 con 0x7f57ec039580 2024-04-21T23:09:55.921 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:55.753+0000 7f57f67fc700 1 -- 172.21.15.174:0/1828948489 <== mgr.14148 v2:172.21.15.40:6800/2500760788 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f5800062670 con 0x7f57ec039580 2024-04-21T23:09:55.921 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:55.755+0000 7f5807875700 1 -- 172.21.15.174:0/1828948489 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f57ec039580 msgr2=0x7f57ec03ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:55.921 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:55.755+0000 7f5807875700 1 --2- 172.21.15.174:0/1828948489 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f57ec039580 0x7f57ec03ba40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f57fc005200 tx=0x7f57fc01a040 comp rx=0 tx=0).stop 2024-04-21T23:09:55.921 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:55.755+0000 7f5807875700 1 -- 172.21.15.174:0/1828948489 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5800198030 msgr2=0x7f580019f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:55.921 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:55.755+0000 7f5807875700 1 --2- 172.21.15.174:0/1828948489 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5800198030 0x7f580019f1b0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f57f800ba50 tx=0x7f57f800bd60 comp rx=0 tx=0).stop 2024-04-21T23:09:55.921 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:55.755+0000 7f5807875700 1 -- 172.21.15.174:0/1828948489 shutdown_connections 2024-04-21T23:09:55.921 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:55.755+0000 7f5807875700 1 --2- 172.21.15.174:0/1828948489 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5800198030 0x7f580019f1b0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:55.921 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:55.755+0000 7f5807875700 1 --2- 172.21.15.174:0/1828948489 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f57ec039580 0x7f57ec03ba40 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:55.921 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:55.755+0000 7f5807875700 1 --2- 172.21.15.174:0/1828948489 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5800101dc0 0x7f5800197670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:55.921 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:55.755+0000 7f5807875700 1 --2- 172.21.15.174:0/1828948489 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5800197bb0 0x7f580019cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:55.921 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:55.755+0000 7f5807875700 1 -- 172.21.15.174:0/1828948489 >> 172.21.15.174:0/1828948489 conn(0x7f58000fd360 msgr2=0x7f58000fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:55.922 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:55.755+0000 7f5807875700 1 -- 172.21.15.174:0/1828948489 shutdown_connections 2024-04-21T23:09:55.922 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:09:55.755+0000 7f5807875700 1 -- 172.21.15.174:0/1828948489 wait complete. 2024-04-21T23:09:56.428 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:56 smithi174 ceph-mon[31409]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:09:56.428 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:56 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:56.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:56 smithi040 ceph-mon[27984]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:09:56.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:56 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:56.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:56 smithi111 ceph-mon[31085]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:09:56.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:56 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:56.459 DEBUG:teuthology.orchestra.run.smithi111:mgr.y> sudo journalctl -f -n 0 -u ceph-d814e4dc-0033-11ef-bc93-c7b262605968@mgr.y.service 2024-04-21T23:09:56.461 DEBUG:teuthology.orchestra.run.smithi174:mgr.z> sudo journalctl -f -n 0 -u ceph-d814e4dc-0033-11ef-bc93-c7b262605968@mgr.z.service 2024-04-21T23:09:56.463 INFO:tasks.cephadm:Deploying OSDs... 2024-04-21T23:09:56.464 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-21T23:09:56.464 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/scratch_devs of=/dev/stdout 2024-04-21T23:09:56.484 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-21T23:09:56.484 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_1 2024-04-21T23:09:56.543 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-21T23:09:56.543 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T23:09:56.543 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-04-21T23:09:56.543 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T23:09:56.543 INFO:teuthology.orchestra.run.smithi040.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T23:09:56.543 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2024-04-21 23:09:00.571960347 +0000 2024-04-21T23:09:56.543 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2024-04-21 23:04:29.419376505 +0000 2024-04-21T23:09:56.543 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2024-04-21 23:04:29.419376505 +0000 2024-04-21T23:09:56.543 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: 2024-04-21 23:04:29.419376505 +0000 2024-04-21T23:09:56.543 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-21T23:09:56.611 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2024-04-21T23:09:56.611 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2024-04-21T23:09:56.611 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000236319 s, 2.2 MB/s 2024-04-21T23:09:56.613 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-21T23:09:56.674 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_2 2024-04-21T23:09:56.732 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-21T23:09:56.732 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T23:09:56.732 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-04-21T23:09:56.732 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T23:09:56.733 INFO:teuthology.orchestra.run.smithi040.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T23:09:56.733 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2024-04-21 23:09:00.602959919 +0000 2024-04-21T23:09:56.733 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2024-04-21 23:04:29.773371448 +0000 2024-04-21T23:09:56.733 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2024-04-21 23:04:29.773371448 +0000 2024-04-21T23:09:56.733 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: 2024-04-21 23:04:29.773371448 +0000 2024-04-21T23:09:56.733 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-21T23:09:56.799 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2024-04-21T23:09:56.799 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2024-04-21T23:09:56.799 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000196681 s, 2.6 MB/s 2024-04-21T23:09:56.801 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-21T23:09:56.859 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_3 2024-04-21T23:09:56.917 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-21T23:09:56.917 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T23:09:56.917 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-04-21T23:09:56.917 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T23:09:56.917 INFO:teuthology.orchestra.run.smithi040.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T23:09:56.917 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2024-04-21 23:09:00.632959504 +0000 2024-04-21T23:09:56.918 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2024-04-21 23:04:30.122366462 +0000 2024-04-21T23:09:56.918 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2024-04-21 23:04:30.122366462 +0000 2024-04-21T23:09:56.918 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: 2024-04-21 23:04:30.122366462 +0000 2024-04-21T23:09:56.918 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-21T23:09:56.982 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2024-04-21T23:09:56.982 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2024-04-21T23:09:56.982 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000200257 s, 2.6 MB/s 2024-04-21T23:09:56.983 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-21T23:09:57.039 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_4 2024-04-21T23:09:57.095 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-21T23:09:57.095 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T23:09:57.095 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-04-21T23:09:57.095 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T23:09:57.096 INFO:teuthology.orchestra.run.smithi040.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T23:09:57.096 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2024-04-21 23:09:00.664959061 +0000 2024-04-21T23:09:57.096 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2024-04-21 23:04:30.465361561 +0000 2024-04-21T23:09:57.096 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2024-04-21 23:04:30.465361561 +0000 2024-04-21T23:09:57.096 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: 2024-04-21 23:04:30.465361561 +0000 2024-04-21T23:09:57.096 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-21T23:09:57.160 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2024-04-21T23:09:57.160 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2024-04-21T23:09:57.160 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000191007 s, 2.7 MB/s 2024-04-21T23:09:57.161 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-21T23:09:57.218 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-04-21T23:09:57.219 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/scratch_devs of=/dev/stdout 2024-04-21T23:09:57.238 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-21T23:09:57.238 DEBUG:teuthology.orchestra.run.smithi111:> stat /dev/vg_nvme/lv_1 2024-04-21T23:09:57.296 INFO:teuthology.orchestra.run.smithi111.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-21T23:09:57.296 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T23:09:57.296 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-04-21T23:09:57.296 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T23:09:57.296 INFO:teuthology.orchestra.run.smithi111.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T23:09:57.296 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2024-04-21 23:09:42.989350525 +0000 2024-04-21T23:09:57.296 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2024-04-21 23:04:41.745933239 +0000 2024-04-21T23:09:57.296 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2024-04-21 23:04:41.745933239 +0000 2024-04-21T23:09:57.296 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: 2024-04-21 23:04:41.745933239 +0000 2024-04-21T23:09:57.297 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-21T23:09:57.361 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records in 2024-04-21T23:09:57.362 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records out 2024-04-21T23:09:57.362 INFO:teuthology.orchestra.run.smithi111.stderr:512 bytes copied, 0.000258778 s, 2.0 MB/s 2024-04-21T23:09:57.363 DEBUG:teuthology.orchestra.run.smithi111:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-21T23:09:57.420 DEBUG:teuthology.orchestra.run.smithi111:> stat /dev/vg_nvme/lv_2 2024-04-21T23:09:57.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:57 smithi040 ceph-mon[27984]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi040=x;smithi111=y;smithi174=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:09:57.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:57 smithi040 ceph-mon[27984]: Saving service mgr spec with placement smithi040=x;smithi111=y;smithi174=z;count:3 2024-04-21T23:09:57.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:57 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:57.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:57 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:57.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:57 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:09:57.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:57 smithi174 ceph-mon[31409]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi040=x;smithi111=y;smithi174=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:09:57.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:57 smithi174 ceph-mon[31409]: Saving service mgr spec with placement smithi040=x;smithi111=y;smithi174=z;count:3 2024-04-21T23:09:57.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:57 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:57.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:57 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:57.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:57 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:09:57.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:57 smithi111 ceph-mon[31085]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi040=x;smithi111=y;smithi174=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:09:57.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:57 smithi111 ceph-mon[31085]: Saving service mgr spec with placement smithi040=x;smithi111=y;smithi174=z;count:3 2024-04-21T23:09:57.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:57 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:57.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:57 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:57.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:57 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:09:57.460 INFO:teuthology.orchestra.run.smithi111.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-21T23:09:57.460 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T23:09:57.460 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-04-21T23:09:57.460 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T23:09:57.460 INFO:teuthology.orchestra.run.smithi111.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T23:09:57.460 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2024-04-21 23:09:42.989350525 +0000 2024-04-21T23:09:57.460 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2024-04-21 23:04:42.071927582 +0000 2024-04-21T23:09:57.460 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2024-04-21 23:04:42.071927582 +0000 2024-04-21T23:09:57.460 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: 2024-04-21 23:04:42.071927582 +0000 2024-04-21T23:09:57.461 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-21T23:09:57.524 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records in 2024-04-21T23:09:57.524 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records out 2024-04-21T23:09:57.524 INFO:teuthology.orchestra.run.smithi111.stderr:512 bytes copied, 0.000229732 s, 2.2 MB/s 2024-04-21T23:09:57.525 DEBUG:teuthology.orchestra.run.smithi111:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-21T23:09:57.581 DEBUG:teuthology.orchestra.run.smithi111:> stat /dev/vg_nvme/lv_3 2024-04-21T23:09:57.637 INFO:teuthology.orchestra.run.smithi111.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-21T23:09:57.637 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T23:09:57.637 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-04-21T23:09:57.637 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T23:09:57.638 INFO:teuthology.orchestra.run.smithi111.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T23:09:57.638 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2024-04-21 23:09:42.990350508 +0000 2024-04-21T23:09:57.638 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2024-04-21 23:04:42.401921856 +0000 2024-04-21T23:09:57.638 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2024-04-21 23:04:42.401921856 +0000 2024-04-21T23:09:57.638 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: 2024-04-21 23:04:42.401921856 +0000 2024-04-21T23:09:57.638 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-21T23:09:57.701 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records in 2024-04-21T23:09:57.701 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records out 2024-04-21T23:09:57.701 INFO:teuthology.orchestra.run.smithi111.stderr:512 bytes copied, 0.000180246 s, 2.8 MB/s 2024-04-21T23:09:57.702 DEBUG:teuthology.orchestra.run.smithi111:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-21T23:09:57.758 DEBUG:teuthology.orchestra.run.smithi111:> stat /dev/vg_nvme/lv_4 2024-04-21T23:09:57.812 INFO:teuthology.orchestra.run.smithi111.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-21T23:09:57.813 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T23:09:57.813 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-04-21T23:09:57.813 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T23:09:57.813 INFO:teuthology.orchestra.run.smithi111.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T23:09:57.813 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2024-04-21 23:09:42.990350508 +0000 2024-04-21T23:09:57.813 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2024-04-21 23:04:42.740915973 +0000 2024-04-21T23:09:57.813 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2024-04-21 23:04:42.740915973 +0000 2024-04-21T23:09:57.813 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: 2024-04-21 23:04:42.740915973 +0000 2024-04-21T23:09:57.813 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-21T23:09:57.876 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records in 2024-04-21T23:09:57.876 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records out 2024-04-21T23:09:57.876 INFO:teuthology.orchestra.run.smithi111.stderr:512 bytes copied, 0.000193876 s, 2.6 MB/s 2024-04-21T23:09:57.877 DEBUG:teuthology.orchestra.run.smithi111:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-21T23:09:57.933 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-04-21T23:09:57.933 DEBUG:teuthology.orchestra.run.smithi174:> dd if=/scratch_devs of=/dev/stdout 2024-04-21T23:09:57.949 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-21T23:09:57.949 DEBUG:teuthology.orchestra.run.smithi174:> stat /dev/vg_nvme/lv_1 2024-04-21T23:09:58.004 INFO:teuthology.orchestra.run.smithi174.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-21T23:09:58.004 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T23:09:58.004 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-04-21T23:09:58.005 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T23:09:58.005 INFO:teuthology.orchestra.run.smithi174.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T23:09:58.005 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 2024-04-21 23:09:33.674850260 +0000 2024-04-21T23:09:58.005 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2024-04-21 23:04:35.594773424 +0000 2024-04-21T23:09:58.005 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2024-04-21 23:04:35.594773424 +0000 2024-04-21T23:09:58.005 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: 2024-04-21 23:04:35.594773424 +0000 2024-04-21T23:09:58.005 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-21T23:09:58.069 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records in 2024-04-21T23:09:58.069 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records out 2024-04-21T23:09:58.069 INFO:teuthology.orchestra.run.smithi174.stderr:512 bytes copied, 0.000251335 s, 2.0 MB/s 2024-04-21T23:09:58.070 DEBUG:teuthology.orchestra.run.smithi174:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-21T23:09:58.126 DEBUG:teuthology.orchestra.run.smithi174:> stat /dev/vg_nvme/lv_2 2024-04-21T23:09:58.181 INFO:teuthology.orchestra.run.smithi174.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-21T23:09:58.181 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T23:09:58.181 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-04-21T23:09:58.181 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T23:09:58.181 INFO:teuthology.orchestra.run.smithi174.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T23:09:58.181 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 2024-04-21 23:09:33.675850246 +0000 2024-04-21T23:09:58.182 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2024-04-21 23:04:35.954768337 +0000 2024-04-21T23:09:58.182 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2024-04-21 23:04:35.954768337 +0000 2024-04-21T23:09:58.182 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: 2024-04-21 23:04:35.954768337 +0000 2024-04-21T23:09:58.182 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-21T23:09:58.243 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records in 2024-04-21T23:09:58.243 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records out 2024-04-21T23:09:58.243 INFO:teuthology.orchestra.run.smithi174.stderr:512 bytes copied, 0.000180006 s, 2.8 MB/s 2024-04-21T23:09:58.244 DEBUG:teuthology.orchestra.run.smithi174:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-21T23:09:58.300 DEBUG:teuthology.orchestra.run.smithi174:> stat /dev/vg_nvme/lv_3 2024-04-21T23:09:58.355 INFO:teuthology.orchestra.run.smithi174.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-21T23:09:58.355 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T23:09:58.355 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-04-21T23:09:58.355 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T23:09:58.355 INFO:teuthology.orchestra.run.smithi174.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T23:09:58.355 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 2024-04-21 23:09:33.675850246 +0000 2024-04-21T23:09:58.356 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2024-04-21 23:04:36.292763560 +0000 2024-04-21T23:09:58.356 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2024-04-21 23:04:36.292763560 +0000 2024-04-21T23:09:58.356 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: 2024-04-21 23:04:36.292763560 +0000 2024-04-21T23:09:58.356 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-21T23:09:58.408 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:58 smithi174 ceph-mon[31409]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:09:58.408 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:58 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:09:58.408 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:58 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:09:58.408 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:58 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:58.408 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:58 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-21T23:09:58.408 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:58 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-21T23:09:58.408 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:58 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-21T23:09:58.408 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:58 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:09:58.418 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records in 2024-04-21T23:09:58.418 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records out 2024-04-21T23:09:58.418 INFO:teuthology.orchestra.run.smithi174.stderr:512 bytes copied, 0.000204866 s, 2.5 MB/s 2024-04-21T23:09:58.420 DEBUG:teuthology.orchestra.run.smithi174:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-21T23:09:58.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:58 smithi040 ceph-mon[27984]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:09:58.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:58 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:09:58.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:58 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:09:58.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:58 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:58.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:58 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-21T23:09:58.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:58 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-21T23:09:58.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:58 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-21T23:09:58.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:58 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:09:58.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:58 smithi111 ceph-mon[31085]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:09:58.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:58 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:09:58.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:58 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:09:58.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:58 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:09:58.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:58 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-21T23:09:58.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:58 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-21T23:09:58.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:58 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-21T23:09:58.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:58 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:09:58.476 DEBUG:teuthology.orchestra.run.smithi174:> stat /dev/vg_nvme/lv_4 2024-04-21T23:09:58.531 INFO:teuthology.orchestra.run.smithi174.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-21T23:09:58.531 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T23:09:58.531 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-04-21T23:09:58.531 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T23:09:58.531 INFO:teuthology.orchestra.run.smithi174.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T23:09:58.531 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 2024-04-21 23:09:33.675850246 +0000 2024-04-21T23:09:58.531 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2024-04-21 23:04:36.634758727 +0000 2024-04-21T23:09:58.531 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2024-04-21 23:04:36.634758727 +0000 2024-04-21T23:09:58.532 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: 2024-04-21 23:04:36.634758727 +0000 2024-04-21T23:09:58.532 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-21T23:09:58.593 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records in 2024-04-21T23:09:58.593 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records out 2024-04-21T23:09:58.593 INFO:teuthology.orchestra.run.smithi174.stderr:512 bytes copied, 0.000198635 s, 2.6 MB/s 2024-04-21T23:09:58.594 DEBUG:teuthology.orchestra.run.smithi174:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-21T23:09:58.650 INFO:tasks.cephadm:Deploying osd.0 on smithi040 with /dev/vg_nvme/lv_4... 2024-04-21T23:09:58.650 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-21T23:09:58.900 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:09:59.326 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:09:59 smithi040 ceph-mon[27984]: Deploying daemon mgr.z on smithi174 2024-04-21T23:09:59.380 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:09:59 smithi174 ceph-mon[31409]: Deploying daemon mgr.z on smithi174 2024-04-21T23:09:59.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:09:59 smithi111 ceph-mon[31085]: Deploying daemon mgr.z on smithi174 2024-04-21T23:09:59.694 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Apr 21 23:09:59 smithi174 systemd[1]: Starting Ceph mgr.z for d814e4dc-0033-11ef-bc93-c7b262605968... 2024-04-21T23:10:00.014 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Apr 21 23:09:59 smithi174 podman[33242]: 2024-04-21 23:09:59.667294307 +0000 UTC m=+0.017691270 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-21T23:10:00.015 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Apr 21 23:09:59 smithi174 podman[33242]: 2024-04-21 23:09:59.781582625 +0000 UTC m=+0.131979561 container create a7eda681942db0f41a0b1ada86e558a5271dff1767c39119e57f24cf1c8b11b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-z, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, RELEASE=squid-820a2f8) 2024-04-21T23:10:00.015 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Apr 21 23:10:00 smithi174 podman[33242]: 2024-04-21 23:10:00.000581139 +0000 UTC m=+0.350978087 container init a7eda681942db0f41a0b1ada86e558a5271dff1767c39119e57f24cf1c8b11b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-z, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=squid-820a2f8, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0) 2024-04-21T23:10:00.015 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Apr 21 23:10:00 smithi174 podman[33242]: 2024-04-21 23:10:00.003159781 +0000 UTC m=+0.353556720 container start a7eda681942db0f41a0b1ada86e558a5271dff1767c39119e57f24cf1c8b11b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-z, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, RELEASE=squid-820a2f8, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-21T23:10:00.335 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:00 smithi040 ceph-mon[27984]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:10:00.335 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:00 smithi040 ceph-mon[27984]: overall HEALTH_OK 2024-04-21T23:10:00.337 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:00 smithi174 ceph-mon[31409]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:10:00.337 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:00 smithi174 ceph-mon[31409]: overall HEALTH_OK 2024-04-21T23:10:00.337 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Apr 21 23:10:00 smithi174 ceph-mgr[33261]: -- 172.21.15.174:0/1721864643 <== mon.1 v2:172.21.15.174:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5596429dc5a0 con 0x5596429b0800 2024-04-21T23:10:00.338 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Apr 21 23:10:00 smithi174 bash[33242]: a7eda681942db0f41a0b1ada86e558a5271dff1767c39119e57f24cf1c8b11b3 2024-04-21T23:10:00.338 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Apr 21 23:10:00 smithi174 systemd[1]: Started Ceph mgr.z for d814e4dc-0033-11ef-bc93-c7b262605968. 2024-04-21T23:10:00.338 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Apr 21 23:10:00 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-z[33257]: 2024-04-21T23:10:00.153+0000 7f3ed138a200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-21T23:10:00.338 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Apr 21 23:10:00 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-z[33257]: 2024-04-21T23:10:00.337+0000 7f3ed138a200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-21T23:10:00.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:00 smithi111 ceph-mon[31085]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:10:00.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:00 smithi111 ceph-mon[31085]: overall HEALTH_OK 2024-04-21T23:10:00.693 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Apr 21 23:10:00 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-z[33257]: 2024-04-21T23:10:00.439+0000 7f3ed138a200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-21T23:10:00.921 INFO:teuthology.orchestra.run.smithi040.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-21T23:10:00.921 INFO:teuthology.orchestra.run.smithi040.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-21T23:10:00.921 INFO:teuthology.orchestra.run.smithi040.stderr: stderr: 10+0 records in 2024-04-21T23:10:00.921 INFO:teuthology.orchestra.run.smithi040.stderr:10+0 records out 2024-04-21T23:10:00.921 INFO:teuthology.orchestra.run.smithi040.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0111608 s, 940 MB/s 2024-04-21T23:10:00.921 INFO:teuthology.orchestra.run.smithi040.stderr: stderr: 2024-04-21T23:10:00.921 INFO:teuthology.orchestra.run.smithi040.stderr:--> Zapping successful for: 2024-04-21T23:10:01.169 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Apr 21 23:10:00 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-z[33257]: 2024-04-21T23:10:00.744+0000 7f3ed138a200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-21T23:10:01.169 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Apr 21 23:10:00 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-z[33257]: 2024-04-21T23:10:00.827+0000 7f3ed138a200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-21T23:10:01.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:01 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:01.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:01 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:01.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:01 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:01.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:01 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-21T23:10:01.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:01 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-21T23:10:01.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:01 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-21T23:10:01.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:01 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:10:01.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:01 smithi040 ceph-mon[27984]: Deploying daemon mgr.y on smithi111 2024-04-21T23:10:01.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:01 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:01.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:01 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:01.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:01 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:01.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:01 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-21T23:10:01.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:01 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-21T23:10:01.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:01 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-21T23:10:01.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:01 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:10:01.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:01 smithi111 ceph-mon[31085]: Deploying daemon mgr.y on smithi111 2024-04-21T23:10:01.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:01 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:01.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:01 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:01.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:01 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:01.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:01 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-21T23:10:01.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:01 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-21T23:10:01.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:01 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-21T23:10:01.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:01 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:10:01.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:01 smithi174 ceph-mon[31409]: Deploying daemon mgr.y on smithi111 2024-04-21T23:10:01.499 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi040:vg_nvme/lv_4 2024-04-21T23:10:01.743 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:10:02.186 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:10:02 smithi111 systemd[1]: Starting Ceph mgr.y for d814e4dc-0033-11ef-bc93-c7b262605968... 2024-04-21T23:10:02.388 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:02 smithi174 ceph-mon[31409]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:10:02.388 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Apr 21 23:10:02 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-z[33257]: 2024-04-21T23:10:02.047+0000 7f3ed138a200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-21T23:10:02.388 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Apr 21 23:10:02 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-z[33257]: 2024-04-21T23:10:02.216+0000 7f3ed138a200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-21T23:10:02.388 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Apr 21 23:10:02 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-z[33257]: 2024-04-21T23:10:02.387+0000 7f3ed138a200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-21T23:10:02.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:02 smithi040 ceph-mon[27984]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:10:02.451 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:02 smithi111 ceph-mon[31085]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:10:02.451 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:10:02 smithi111 podman[32464]: 2024-04-21 23:10:02.334745388 +0000 UTC m=+0.104466956 container create 8f392e362b15b39584278d2989fb8cecadf326bbfad9fe4d4bc8fb1b4a2823a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-y, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=squid-820a2f8, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-21T23:10:02.452 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:10:02 smithi111 podman[32464]: 2024-04-21 23:10:02.248717351 +0000 UTC m=+0.018438933 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-21T23:10:02.887 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:10:02 smithi111 podman[32464]: 2024-04-21 23:10:02.536876988 +0000 UTC m=+0.306598561 container init 8f392e362b15b39584278d2989fb8cecadf326bbfad9fe4d4bc8fb1b4a2823a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-y, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=squid-820a2f8, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T23:10:02.887 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:10:02 smithi111 podman[32464]: 2024-04-21 23:10:02.539308184 +0000 UTC m=+0.309029754 container start 8f392e362b15b39584278d2989fb8cecadf326bbfad9fe4d4bc8fb1b4a2823a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-y, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-820a2f8, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-21T23:10:02.887 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:10:02 smithi111 ceph-mgr[32483]: -- 172.21.15.111:0/990632617 <== mon.2 v2:172.21.15.111:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55dce0a0e5a0 con 0x55dce09e2800 2024-04-21T23:10:02.888 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:10:02 smithi111 bash[32464]: 8f392e362b15b39584278d2989fb8cecadf326bbfad9fe4d4bc8fb1b4a2823a7 2024-04-21T23:10:02.888 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:10:02 smithi111 systemd[1]: Started Ceph mgr.y for d814e4dc-0033-11ef-bc93-c7b262605968. 2024-04-21T23:10:02.888 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:10:02 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-y[32479]: 2024-04-21T23:10:02.701+0000 7f4ccd61e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-21T23:10:02.888 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:10:02 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-y[32479]: 2024-04-21T23:10:02.888+0000 7f4ccd61e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-21T23:10:02.943 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Apr 21 23:10:02 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-z[33257]: 2024-04-21T23:10:02.676+0000 7f3ed138a200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-21T23:10:02.944 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Apr 21 23:10:02 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-z[33257]: 2024-04-21T23:10:02.769+0000 7f3ed138a200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-21T23:10:03.194 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:10:02 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-y[32479]: 2024-04-21T23:10:02.994+0000 7f4ccd61e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-21T23:10:03.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:10:03.279+0000 7f82bcb26700 1 -- 172.21.15.40:0/4039014212 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82b810a190 msgr2=0x7f82b810c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:03.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:10:03.279+0000 7f82bcb26700 1 --2- 172.21.15.40:0/4039014212 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82b810a190 0x7f82b810c600 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f82b0009990 tx=0x7f82b0009ca0 comp rx=0 tx=0).stop 2024-04-21T23:10:03.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:10:03.279+0000 7f82bcb26700 1 -- 172.21.15.40:0/4039014212 shutdown_connections 2024-04-21T23:10:03.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:10:03.279+0000 7f82bcb26700 1 --2- 172.21.15.40:0/4039014212 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82b810a190 0x7f82b810c600 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:03.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:10:03.279+0000 7f82bcb26700 1 --2- 172.21.15.40:0/4039014212 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82b80761e0 0x7f82b8074640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:03.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:10:03.279+0000 7f82bcb26700 1 --2- 172.21.15.40:0/4039014212 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82b8074c10 0x7f82b8075090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:03.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:10:03.279+0000 7f82bcb26700 1 -- 172.21.15.40:0/4039014212 >> 172.21.15.40:0/4039014212 conn(0x7f82b80fd830 msgr2=0x7f82b80ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:03.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:10:03.279+0000 7f82bcb26700 1 -- 172.21.15.40:0/4039014212 shutdown_connections 2024-04-21T23:10:03.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:10:03.279+0000 7f82bcb26700 1 -- 172.21.15.40:0/4039014212 wait complete. 2024-04-21T23:10:03.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:10:03.280+0000 7f82bcb26700 1 Processor -- start 2024-04-21T23:10:03.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:10:03.280+0000 7f82bcb26700 1 -- start start 2024-04-21T23:10:03.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:10:03.280+0000 7f82bcb26700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82b8074c10 0x7f82b819a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:03.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:10:03.280+0000 7f82bcb26700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82b80761e0 0x7f82b819abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:03.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:10:03.280+0000 7f82bcb26700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82b810a190 0x7f82b81a1c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:03.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:10:03.280+0000 7f82bcb26700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f82b81100b0 con 0x7f82b810a190 2024-04-21T23:10:03.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:10:03.280+0000 7f82bcb26700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f82b810ff30 con 0x7f82b80761e0 2024-04-21T23:10:03.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:10:03.280+0000 7f82bcb26700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f82b8110230 con 0x7f82b8074c10 2024-04-21T23:10:03.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:10:03.280+0000 7f82b6d9d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82b810a190 0x7f82b81a1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:03.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:10:03.280+0000 7f82b6d9d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82b810a190 0x7f82b81a1c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59104/0 (socket says 172.21.15.40:59104) 2024-04-21T23:10:03.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:10:03.280+0000 7f82b6d9d700 1 -- 172.21.15.40:0/3595957954 learned_addr learned my addr 172.21.15.40:0/3595957954 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:10:03.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:10:03.280+0000 7f82b659c700 1 --2- 172.21.15.40:0/3595957954 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82b8074c10 0x7f82b819a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:03.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:10:03.280+0000 7f82b5d9b700 1 --2- 172.21.15.40:0/3595957954 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82b80761e0 0x7f82b819abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:03.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:10:03.280+0000 7f82b6d9d700 1 -- 172.21.15.40:0/3595957954 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82b8074c10 msgr2=0x7f82b819a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:03.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:10:03.280+0000 7f82b6d9d700 1 --2- 172.21.15.40:0/3595957954 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82b8074c10 0x7f82b819a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:03.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:10:03.280+0000 7f82b6d9d700 1 -- 172.21.15.40:0/3595957954 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82b80761e0 msgr2=0x7f82b819abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:03.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:10:03.280+0000 7f82b6d9d700 1 --2- 172.21.15.40:0/3595957954 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82b80761e0 0x7f82b819abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:03.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:10:03.280+0000 7f82b6d9d700 1 -- 172.21.15.40:0/3595957954 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82b81a22a0 con 0x7f82b810a190 2024-04-21T23:10:03.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:10:03.281+0000 7f82b6d9d700 1 --2- 172.21.15.40:0/3595957954 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82b810a190 0x7f82b81a1c00 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f82b000bbb0 tx=0x7f82b000bbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:03.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:10:03.281+0000 7f82af7fe700 1 -- 172.21.15.40:0/3595957954 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82b0027070 con 0x7f82b810a190 2024-04-21T23:10:03.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:10:03.281+0000 7f82bcb26700 1 -- 172.21.15.40:0/3595957954 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82b81a2530 con 0x7f82b810a190 2024-04-21T23:10:03.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:10:03.281+0000 7f82af7fe700 1 -- 172.21.15.40:0/3595957954 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f82b0022470 con 0x7f82b810a190 2024-04-21T23:10:03.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:10:03.281+0000 7f82af7fe700 1 -- 172.21.15.40:0/3595957954 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82b000f6c0 con 0x7f82b810a190 2024-04-21T23:10:03.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:10:03.281+0000 7f82bcb26700 1 -- 172.21.15.40:0/3595957954 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82b81a2980 con 0x7f82b810a190 2024-04-21T23:10:03.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:10:03.282+0000 7f82af7fe700 1 -- 172.21.15.40:0/3595957954 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f82b002a070 con 0x7f82b810a190 2024-04-21T23:10:03.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:10:03.282+0000 7f82bcb26700 1 -- 172.21.15.40:0/3595957954 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82b8068f20 con 0x7f82b810a190 2024-04-21T23:10:03.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:10:03.282+0000 7f82af7fe700 1 --2- 172.21.15.40:0/3595957954 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f82a4039330 0x7f82a403b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:03.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:10:03.282+0000 7f82af7fe700 1 -- 172.21.15.40:0/3595957954 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f82b004d2e0 con 0x7f82b810a190 2024-04-21T23:10:03.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:10:03.282+0000 7f82b659c700 1 --2- 172.21.15.40:0/3595957954 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f82a4039330 0x7f82a403b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:03.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:10:03.282+0000 7f82b659c700 1 --2- 172.21.15.40:0/3595957954 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f82a4039330 0x7f82a403b7f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f82a800f540 tx=0x7f82a800b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:03.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:10:03.287+0000 7f82af7fe700 1 -- 172.21.15.40:0/3595957954 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f82b0013070 con 0x7f82b810a190 2024-04-21T23:10:03.444 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Apr 21 23:10:02 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-z[33257]: 2024-04-21T23:10:02.979+0000 7f3ed138a200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-21T23:10:03.444 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Apr 21 23:10:03 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-z[33257]: 2024-04-21T23:10:03.179+0000 7f3ed138a200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-21T23:10:03.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:10:03.504+0000 7f82bcb26700 1 -- 172.21.15.40:0/3595957954 --> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f82b8062670 con 0x7f82a4039330 2024-04-21T23:10:03.688 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:10:03 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-y[32479]: 2024-04-21T23:10:03.317+0000 7f4ccd61e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-21T23:10:03.688 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:10:03 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-y[32479]: 2024-04-21T23:10:03.411+0000 7f4ccd61e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-21T23:10:03.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:03 smithi040 ceph-mon[27984]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:10:03.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:03 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:03.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:03 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:03.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:03 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:03.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:03 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:03.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:03 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:10:03.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:03 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:10:03.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:03 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:10:03.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:03 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:10:03.944 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Apr 21 23:10:03 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-z[33257]: 2024-04-21T23:10:03.825+0000 7f3ed138a200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-21T23:10:03.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:03 smithi174 ceph-mon[31409]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:10:03.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:03 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:03.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:03 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:03.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:03 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:03.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:03 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:03.945 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:03 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:10:03.945 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:03 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:10:03.945 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:03 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:10:03.945 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:03 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:10:03.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:03 smithi111 ceph-mon[31085]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:10:03.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:03 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:03.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:03 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:03.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:03 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:03.946 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:03 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:03.946 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:03 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:10:03.946 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:03 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:10:03.946 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:03 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:10:03.946 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:03 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:10:04.563 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Apr 21 23:10:04 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-z[33257]: 2024-04-21T23:10:04.266+0000 7f3ed138a200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-21T23:10:04.564 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Apr 21 23:10:04 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-z[33257]: 2024-04-21T23:10:04.364+0000 7f3ed138a200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-21T23:10:04.564 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Apr 21 23:10:04 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-z[33257]: 2024-04-21T23:10:04.564+0000 7f3ed138a200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-21T23:10:04.944 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:10:04 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-y[32479]: 2024-04-21T23:10:04.637+0000 7f4ccd61e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-21T23:10:04.944 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:10:04 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-y[32479]: 2024-04-21T23:10:04.804+0000 7f4ccd61e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-21T23:10:05.258 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:10:04 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-y[32479]: 2024-04-21T23:10:04.972+0000 7f4ccd61e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-21T23:10:05.258 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:10:05 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-y[32479]: 2024-04-21T23:10:05.258+0000 7f4ccd61e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-21T23:10:05.258 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:05 smithi111 ceph-mon[31085]: from='client.14220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:10:05.259 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:05 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:05.259 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:05 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:05.259 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:05 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:05.259 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:05 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:05.259 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:05 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:10:05.259 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:05 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:10:05.259 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:05 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:05.259 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:05 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-21T23:10:05.259 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:05 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-21T23:10:05.259 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:05 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:10:05.360 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:05 smithi040 ceph-mon[27984]: from='client.14220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:10:05.360 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:05 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:05.361 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:05 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:05.361 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:05 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:05.361 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:05 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:05.361 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:05 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:10:05.361 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:05 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:10:05.361 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:05 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:05.361 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:05 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-21T23:10:05.361 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:05 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-21T23:10:05.361 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:05 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:10:05.413 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Apr 21 23:10:05 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-z[33257]: 2024-04-21T23:10:05.082+0000 7f3ed138a200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-21T23:10:05.413 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Apr 21 23:10:05 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-z[33257]: 2024-04-21T23:10:05.174+0000 7f3ed138a200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-21T23:10:05.413 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:05 smithi174 ceph-mon[31409]: from='client.14220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:10:05.413 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:05 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:05.414 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:05 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:05.414 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:05 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:05.414 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:05 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:05.414 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:05 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:10:05.414 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:05 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:10:05.414 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:05 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:05.414 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:05 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-21T23:10:05.414 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:05 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-21T23:10:05.414 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:05 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:10:05.550 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:10:05 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-y[32479]: 2024-04-21T23:10:05.348+0000 7f4ccd61e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-21T23:10:05.551 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:10:05 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-y[32479]: 2024-04-21T23:10:05.550+0000 7f4ccd61e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-21T23:10:05.694 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Apr 21 23:10:05 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-z[33257]: 2024-04-21T23:10:05.413+0000 7f3ed138a200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-21T23:10:05.694 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Apr 21 23:10:05 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-z[33257]: 2024-04-21T23:10:05.546+0000 7f3ed138a200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-21T23:10:05.694 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Apr 21 23:10:05 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-z[33257]: 2024-04-21T23:10:05.640+0000 7f3ed138a200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-21T23:10:05.944 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:10:05 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-y[32479]: 2024-04-21T23:10:05.739+0000 7f4ccd61e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-21T23:10:06.190 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Apr 21 23:10:05 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-z[33257]: 2024-04-21T23:10:05.755+0000 7f3ed138a200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-21T23:10:06.191 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Apr 21 23:10:05 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-z[33257]: 2024-04-21T23:10:05.845+0000 7f3ed138a200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-21T23:10:06.435 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:06 smithi040 ceph-mon[27984]: Reconfiguring mgr.x (unknown last config time)... 2024-04-21T23:10:06.435 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:06 smithi040 ceph-mon[27984]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:10:06.435 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:06 smithi040 ceph-mon[27984]: Reconfiguring daemon mgr.x on smithi040 2024-04-21T23:10:06.435 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:06 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:06.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:06 smithi174 ceph-mon[31409]: Reconfiguring mgr.x (unknown last config time)... 2024-04-21T23:10:06.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:06 smithi174 ceph-mon[31409]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:10:06.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:06 smithi174 ceph-mon[31409]: Reconfiguring daemon mgr.x on smithi040 2024-04-21T23:10:06.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:06 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:06.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:06 smithi111 ceph-mon[31085]: Reconfiguring mgr.x (unknown last config time)... 2024-04-21T23:10:06.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:06 smithi111 ceph-mon[31085]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:10:06.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:06 smithi111 ceph-mon[31085]: Reconfiguring daemon mgr.x on smithi040 2024-04-21T23:10:06.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:06 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:06.444 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:10:06 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-y[32479]: 2024-04-21T23:10:06.335+0000 7f4ccd61e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-21T23:10:06.944 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Apr 21 23:10:06 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-z[33257]: 2024-04-21T23:10:06.661+0000 7f3ed138a200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-21T23:10:06.944 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Apr 21 23:10:06 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-z[33257]: 2024-04-21T23:10:06.873+0000 7f3ed138a200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-21T23:10:07.046 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:10:06 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-y[32479]: 2024-04-21T23:10:06.753+0000 7f4ccd61e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-21T23:10:07.046 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:10:06 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-y[32479]: 2024-04-21T23:10:06.848+0000 7f4ccd61e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-21T23:10:07.046 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:10:07 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-y[32479]: 2024-04-21T23:10:07.045+0000 7f4ccd61e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-21T23:10:07.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:10:07.300+0000 7f82af7fe700 1 -- 172.21.15.40:0/3595957954 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mgrmap(e 14) ==== 92060+0+0 (secure 0 0 0) 0x7f82b0009d10 con 0x7f82b810a190 2024-04-21T23:10:07.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:07 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:07.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:07 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:07.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:07 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:10:07.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:07 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:10:07.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:07 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:10:07.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:07 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:07.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:07 smithi111 ceph-mon[31085]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:10:07.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:07 smithi111 ceph-mon[31085]: Standby manager daemon z started 2024-04-21T23:10:07.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:07 smithi111 ceph-mon[31085]: from='mgr.? 172.21.15.174:0/3901159454' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-21T23:10:07.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:07 smithi111 ceph-mon[31085]: from='mgr.? 172.21.15.174:0/3901159454' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T23:10:07.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:07 smithi111 ceph-mon[31085]: from='mgr.? 172.21.15.174:0/3901159454' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-21T23:10:07.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:07 smithi111 ceph-mon[31085]: from='mgr.? 172.21.15.174:0/3901159454' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T23:10:07.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:07 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:07.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:07 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:07.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:07 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:10:07.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:07 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:10:07.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:07 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:10:07.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:07 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:07.683 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:07 smithi040 ceph-mon[27984]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:10:07.683 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:07 smithi040 ceph-mon[27984]: Standby manager daemon z started 2024-04-21T23:10:07.683 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:07 smithi040 ceph-mon[27984]: from='mgr.? 172.21.15.174:0/3901159454' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-21T23:10:07.683 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:07 smithi040 ceph-mon[27984]: from='mgr.? 172.21.15.174:0/3901159454' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T23:10:07.683 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:07 smithi040 ceph-mon[27984]: from='mgr.? 172.21.15.174:0/3901159454' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-21T23:10:07.683 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:07 smithi040 ceph-mon[27984]: from='mgr.? 172.21.15.174:0/3901159454' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T23:10:07.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:07 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:07.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:07 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:07.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:07 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:10:07.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:07 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:10:07.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:07 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:10:07.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:07 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:07.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:07 smithi174 ceph-mon[31409]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:10:07.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:07 smithi174 ceph-mon[31409]: Standby manager daemon z started 2024-04-21T23:10:07.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:07 smithi174 ceph-mon[31409]: from='mgr.? 172.21.15.174:0/3901159454' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-21T23:10:07.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:07 smithi174 ceph-mon[31409]: from='mgr.? 172.21.15.174:0/3901159454' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T23:10:07.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:07 smithi174 ceph-mon[31409]: from='mgr.? 172.21.15.174:0/3901159454' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-21T23:10:07.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:07 smithi174 ceph-mon[31409]: from='mgr.? 172.21.15.174:0/3901159454' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T23:10:07.892 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:10:07 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-y[32479]: 2024-04-21T23:10:07.566+0000 7f4ccd61e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-21T23:10:07.893 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:10:07 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-y[32479]: 2024-04-21T23:10:07.655+0000 7f4ccd61e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-21T23:10:08.194 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:10:07 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-y[32479]: 2024-04-21T23:10:07.892+0000 7f4ccd61e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-21T23:10:08.194 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:10:08 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-y[32479]: 2024-04-21T23:10:08.023+0000 7f4ccd61e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-21T23:10:08.194 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:10:08 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-y[32479]: 2024-04-21T23:10:08.115+0000 7f4ccd61e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-21T23:10:08.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:08 smithi040 ceph-mon[27984]: mgrmap e14: x(active, since 76s), standbys: z 2024-04-21T23:10:08.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:08 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-21T23:10:08.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:08 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/1790523577' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0e8d692a-22fe-4a87-8d99-9151f96cbb02"}]: dispatch 2024-04-21T23:10:08.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:08 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/1790523577' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0e8d692a-22fe-4a87-8d99-9151f96cbb02"}]': finished 2024-04-21T23:10:08.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:08 smithi040 ceph-mon[27984]: osdmap e5: 1 total, 0 up, 1 in 2024-04-21T23:10:08.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:08 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T23:10:08.683 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:08 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/1595988811' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:10:08.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:08 smithi174 ceph-mon[31409]: mgrmap e14: x(active, since 76s), standbys: z 2024-04-21T23:10:08.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:08 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-21T23:10:08.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:08 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/1790523577' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0e8d692a-22fe-4a87-8d99-9151f96cbb02"}]: dispatch 2024-04-21T23:10:08.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:08 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/1790523577' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0e8d692a-22fe-4a87-8d99-9151f96cbb02"}]': finished 2024-04-21T23:10:08.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:08 smithi174 ceph-mon[31409]: osdmap e5: 1 total, 0 up, 1 in 2024-04-21T23:10:08.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:08 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T23:10:08.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:08 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/1595988811' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:10:08.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:08 smithi111 ceph-mon[31085]: mgrmap e14: x(active, since 76s), standbys: z 2024-04-21T23:10:08.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:08 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-21T23:10:08.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:08 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/1790523577' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0e8d692a-22fe-4a87-8d99-9151f96cbb02"}]: dispatch 2024-04-21T23:10:08.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:08 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/1790523577' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0e8d692a-22fe-4a87-8d99-9151f96cbb02"}]': finished 2024-04-21T23:10:08.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:08 smithi111 ceph-mon[31085]: osdmap e5: 1 total, 0 up, 1 in 2024-04-21T23:10:08.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:08 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T23:10:08.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:08 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/1595988811' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:10:08.695 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:10:08 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-y[32479]: 2024-04-21T23:10:08.229+0000 7f4ccd61e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-21T23:10:08.695 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:10:08 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-y[32479]: 2024-04-21T23:10:08.317+0000 7f4ccd61e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-21T23:10:09.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:09 smithi111 ceph-mon[31085]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:10:09.444 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:10:09 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-y[32479]: 2024-04-21T23:10:09.118+0000 7f4ccd61e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-21T23:10:09.444 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:10:09 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-y[32479]: 2024-04-21T23:10:09.330+0000 7f4ccd61e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-21T23:10:09.584 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:09 smithi040 ceph-mon[27984]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:10:09.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:09 smithi174 ceph-mon[31409]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:10:10.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:10:10.314+0000 7f82af7fe700 1 -- 172.21.15.40:0/3595957954 <== mon.0 v2:172.21.15.40:3300/0 8 ==== mgrmap(e 15) ==== 137777+0+0 (secure 0 0 0) 0x7f82b0003910 con 0x7f82b810a190 2024-04-21T23:10:10.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:10 smithi040 ceph-mon[27984]: Standby manager daemon y started 2024-04-21T23:10:10.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:10 smithi040 ceph-mon[27984]: from='mgr.? 172.21.15.111:0/358027275' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-21T23:10:10.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:10 smithi040 ceph-mon[27984]: from='mgr.? 172.21.15.111:0/358027275' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T23:10:10.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:10 smithi040 ceph-mon[27984]: from='mgr.? 172.21.15.111:0/358027275' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-21T23:10:10.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:10 smithi040 ceph-mon[27984]: from='mgr.? 172.21.15.111:0/358027275' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T23:10:10.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:10 smithi174 ceph-mon[31409]: Standby manager daemon y started 2024-04-21T23:10:10.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:10 smithi174 ceph-mon[31409]: from='mgr.? 172.21.15.111:0/358027275' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-21T23:10:10.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:10 smithi174 ceph-mon[31409]: from='mgr.? 172.21.15.111:0/358027275' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T23:10:10.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:10 smithi174 ceph-mon[31409]: from='mgr.? 172.21.15.111:0/358027275' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-21T23:10:10.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:10 smithi174 ceph-mon[31409]: from='mgr.? 172.21.15.111:0/358027275' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T23:10:10.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:10 smithi111 ceph-mon[31085]: Standby manager daemon y started 2024-04-21T23:10:10.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:10 smithi111 ceph-mon[31085]: from='mgr.? 172.21.15.111:0/358027275' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-21T23:10:10.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:10 smithi111 ceph-mon[31085]: from='mgr.? 172.21.15.111:0/358027275' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T23:10:10.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:10 smithi111 ceph-mon[31085]: from='mgr.? 172.21.15.111:0/358027275' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-21T23:10:10.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:10 smithi111 ceph-mon[31085]: from='mgr.? 172.21.15.111:0/358027275' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T23:10:11.638 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:11 smithi111 ceph-mon[31085]: mgrmap e15: x(active, since 80s), standbys: z, y 2024-04-21T23:10:11.639 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:11 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-21T23:10:11.639 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:11 smithi111 ceph-mon[31085]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:10:11.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:11 smithi040 ceph-mon[27984]: mgrmap e15: x(active, since 80s), standbys: z, y 2024-04-21T23:10:11.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:11 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-21T23:10:11.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:11 smithi040 ceph-mon[27984]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:10:11.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:11 smithi174 ceph-mon[31409]: mgrmap e15: x(active, since 80s), standbys: z, y 2024-04-21T23:10:11.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:11 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-21T23:10:11.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:11 smithi174 ceph-mon[31409]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:10:13.510 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:13 smithi040 ceph-mon[27984]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:10:13.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:13 smithi174 ceph-mon[31409]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:10:13.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:13 smithi111 ceph-mon[31085]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:10:15.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:15 smithi040 ceph-mon[27984]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:10:15.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:15 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-21T23:10:15.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:15 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:10:15.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:15 smithi040 ceph-mon[27984]: Deploying daemon osd.0 on smithi040 2024-04-21T23:10:15.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:15 smithi174 ceph-mon[31409]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:10:15.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:15 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-21T23:10:15.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:15 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:10:15.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:15 smithi174 ceph-mon[31409]: Deploying daemon osd.0 on smithi040 2024-04-21T23:10:15.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:15 smithi111 ceph-mon[31085]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:10:15.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:15 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-21T23:10:15.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:15 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:10:15.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:15 smithi111 ceph-mon[31085]: Deploying daemon osd.0 on smithi040 2024-04-21T23:10:17.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:17 smithi040 ceph-mon[27984]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:10:17.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:17 smithi174 ceph-mon[31409]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:10:17.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:17 smithi111 ceph-mon[31085]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:10:19.679 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:19 smithi040 ceph-mon[27984]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:10:19.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:19 smithi174 ceph-mon[31409]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:10:19.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:19 smithi111 ceph-mon[31085]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:10:21.129 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:20 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:21.129 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:20 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:21.129 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:20 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:10:21.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:20 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:21.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:20 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:21.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:20 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:10:21.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:20 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:21.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:20 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:21.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:20 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:10:21.931 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:21 smithi040 ceph-mon[27984]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:10:22.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:21 smithi174 ceph-mon[31409]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:10:22.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:21 smithi111 ceph-mon[31085]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:10:23.368 INFO:teuthology.orchestra.run.smithi040.stdout:Created osd(s) 0 on host 'smithi040' 2024-04-21T23:10:23.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:10:23.368+0000 7f82af7fe700 1 -- 172.21.15.40:0/3595957954 <== mgr.14148 v2:172.21.15.40:6800/2500760788 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f82b8062670 con 0x7f82a4039330 2024-04-21T23:10:23.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:10:23.371+0000 7f82bcb26700 1 -- 172.21.15.40:0/3595957954 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f82a4039330 msgr2=0x7f82a403b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:23.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:10:23.371+0000 7f82bcb26700 1 --2- 172.21.15.40:0/3595957954 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f82a4039330 0x7f82a403b7f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f82a800f540 tx=0x7f82a800b040 comp rx=0 tx=0).stop 2024-04-21T23:10:23.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:10:23.371+0000 7f82bcb26700 1 -- 172.21.15.40:0/3595957954 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82b810a190 msgr2=0x7f82b81a1c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:23.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:10:23.371+0000 7f82bcb26700 1 --2- 172.21.15.40:0/3595957954 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82b810a190 0x7f82b81a1c00 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f82b000bbb0 tx=0x7f82b000bbe0 comp rx=0 tx=0).stop 2024-04-21T23:10:23.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:10:23.371+0000 7f82bcb26700 1 -- 172.21.15.40:0/3595957954 shutdown_connections 2024-04-21T23:10:23.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:10:23.371+0000 7f82bcb26700 1 --2- 172.21.15.40:0/3595957954 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82b8074c10 0x7f82b819a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:23.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:10:23.371+0000 7f82bcb26700 1 --2- 172.21.15.40:0/3595957954 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f82a4039330 0x7f82a403b7f0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:23.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:10:23.371+0000 7f82bcb26700 1 --2- 172.21.15.40:0/3595957954 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82b810a190 0x7f82b81a1c00 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:23.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:10:23.371+0000 7f82bcb26700 1 --2- 172.21.15.40:0/3595957954 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82b80761e0 0x7f82b819abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:23.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:10:23.371+0000 7f82bcb26700 1 -- 172.21.15.40:0/3595957954 >> 172.21.15.40:0/3595957954 conn(0x7f82b80fd830 msgr2=0x7f82b81060e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:23.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:10:23.371+0000 7f82bcb26700 1 -- 172.21.15.40:0/3595957954 shutdown_connections 2024-04-21T23:10:23.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:10:23.371+0000 7f82bcb26700 1 -- 172.21.15.40:0/3595957954 wait complete. 2024-04-21T23:10:23.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:23 smithi040 ceph-mon[27984]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:10:23.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:23 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:23.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:23 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:23.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:23 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:10:23.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:23 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:10:23.683 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:23 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:23.683 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:23 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:23.683 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:23 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:23.683 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:23 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:10:23.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:23 smithi174 ceph-mon[31409]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:10:23.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:23 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:23.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:23 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:23.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:23 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:10:23.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:23 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:10:23.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:23 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:23.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:23 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:23.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:23 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:23.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:23 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:10:23.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:23 smithi111 ceph-mon[31085]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:10:23.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:23 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:23.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:23 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:23.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:23 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:10:23.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:23 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:10:23.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:23 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:23.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:23 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:23.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:23 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:23.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:23 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:10:24.073 DEBUG:teuthology.orchestra.run.smithi040:osd.0> sudo journalctl -f -n 0 -u ceph-d814e4dc-0033-11ef-bc93-c7b262605968@osd.0.service 2024-04-21T23:10:24.076 INFO:tasks.cephadm:Deploying osd.1 on smithi111 with /dev/vg_nvme/lv_4... 2024-04-21T23:10:24.076 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-21T23:10:24.319 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.b/config 2024-04-21T23:10:24.673 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:24 smithi111 ceph-mon[31085]: from='osd.0 [v2:172.21.15.40:6802/2009758784,v1:172.21.15.40:6803/2009758784]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-21T23:10:24.681 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:24 smithi040 ceph-mon[27984]: from='osd.0 [v2:172.21.15.40:6802/2009758784,v1:172.21.15.40:6803/2009758784]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-21T23:10:24.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:24 smithi174 ceph-mon[31409]: from='osd.0 [v2:172.21.15.40:6802/2009758784,v1:172.21.15.40:6803/2009758784]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-21T23:10:25.548 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:25 smithi111 ceph-mon[31085]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:10:25.548 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:25 smithi111 ceph-mon[31085]: from='osd.0 [v2:172.21.15.40:6802/2009758784,v1:172.21.15.40:6803/2009758784]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-21T23:10:25.549 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:25 smithi111 ceph-mon[31085]: osdmap e6: 1 total, 0 up, 1 in 2024-04-21T23:10:25.549 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:25 smithi111 ceph-mon[31085]: from='osd.0 [v2:172.21.15.40:6802/2009758784,v1:172.21.15.40:6803/2009758784]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-21T23:10:25.549 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:25 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T23:10:25.761 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:25 smithi040 ceph-mon[27984]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:10:25.761 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:25 smithi040 ceph-mon[27984]: from='osd.0 [v2:172.21.15.40:6802/2009758784,v1:172.21.15.40:6803/2009758784]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-21T23:10:25.761 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:25 smithi040 ceph-mon[27984]: osdmap e6: 1 total, 0 up, 1 in 2024-04-21T23:10:25.761 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:25 smithi040 ceph-mon[27984]: from='osd.0 [v2:172.21.15.40:6802/2009758784,v1:172.21.15.40:6803/2009758784]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-21T23:10:25.761 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:25 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T23:10:25.761 INFO:journalctl@ceph.osd.0.smithi040.stdout:Apr 21 23:10:25 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-0[40139]: 2024-04-21T23:10:25.509+0000 7f29a69a0700 -1 osd.0 0 waiting for initial osdmap 2024-04-21T23:10:25.761 INFO:journalctl@ceph.osd.0.smithi040.stdout:Apr 21 23:10:25 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-0[40139]: 2024-04-21T23:10:25.512+0000 7f29a2793700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T23:10:25.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:25 smithi174 ceph-mon[31409]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:10:25.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:25 smithi174 ceph-mon[31409]: from='osd.0 [v2:172.21.15.40:6802/2009758784,v1:172.21.15.40:6803/2009758784]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-21T23:10:25.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:25 smithi174 ceph-mon[31409]: osdmap e6: 1 total, 0 up, 1 in 2024-04-21T23:10:25.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:25 smithi174 ceph-mon[31409]: from='osd.0 [v2:172.21.15.40:6802/2009758784,v1:172.21.15.40:6803/2009758784]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-21T23:10:25.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:25 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T23:10:26.110 INFO:teuthology.orchestra.run.smithi111.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-21T23:10:26.110 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-21T23:10:26.110 INFO:teuthology.orchestra.run.smithi111.stderr: stderr: 10+0 records in 2024-04-21T23:10:26.111 INFO:teuthology.orchestra.run.smithi111.stderr:10+0 records out 2024-04-21T23:10:26.111 INFO:teuthology.orchestra.run.smithi111.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0127341 s, 823 MB/s 2024-04-21T23:10:26.111 INFO:teuthology.orchestra.run.smithi111.stderr:--> Zapping successful for: 2024-04-21T23:10:26.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:26 smithi040 ceph-mon[27984]: from='osd.0 [v2:172.21.15.40:6802/2009758784,v1:172.21.15.40:6803/2009758784]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-04-21T23:10:26.683 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:26 smithi040 ceph-mon[27984]: osdmap e7: 1 total, 0 up, 1 in 2024-04-21T23:10:26.683 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:26 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T23:10:26.683 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:26 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T23:10:26.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:26 smithi111 ceph-mon[31085]: from='osd.0 [v2:172.21.15.40:6802/2009758784,v1:172.21.15.40:6803/2009758784]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-04-21T23:10:26.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:26 smithi111 ceph-mon[31085]: osdmap e7: 1 total, 0 up, 1 in 2024-04-21T23:10:26.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:26 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T23:10:26.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:26 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T23:10:26.757 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi111:vg_nvme/lv_4 2024-04-21T23:10:26.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:26 smithi174 ceph-mon[31409]: from='osd.0 [v2:172.21.15.40:6802/2009758784,v1:172.21.15.40:6803/2009758784]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-04-21T23:10:26.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:26 smithi174 ceph-mon[31409]: osdmap e7: 1 total, 0 up, 1 in 2024-04-21T23:10:26.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:26 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T23:10:26.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:26 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T23:10:26.993 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.b/config 2024-04-21T23:10:27.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:27 smithi111 ceph-mon[31085]: purged_snaps scrub starts 2024-04-21T23:10:27.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:27 smithi111 ceph-mon[31085]: purged_snaps scrub ok 2024-04-21T23:10:27.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:27 smithi111 ceph-mon[31085]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:10:27.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:27 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T23:10:27.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:27 smithi111 ceph-mon[31085]: osd.0 [v2:172.21.15.40:6802/2009758784,v1:172.21.15.40:6803/2009758784] boot 2024-04-21T23:10:27.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:27 smithi111 ceph-mon[31085]: osdmap e8: 1 total, 1 up, 1 in 2024-04-21T23:10:27.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:27 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T23:10:27.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:27 smithi111 ceph-mon[31085]: Detected new or changed devices on smithi040 2024-04-21T23:10:27.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:27 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:27.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:27 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:27.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:27 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:10:27.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:27 smithi111 ceph-mon[31085]: Adjusting osd_memory_target on smithi040 to 17093M 2024-04-21T23:10:27.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:27 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:27.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:27 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:10:27.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:27 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:10:27.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:27 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:27.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:27 smithi040 ceph-mon[27984]: purged_snaps scrub starts 2024-04-21T23:10:27.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:27 smithi040 ceph-mon[27984]: purged_snaps scrub ok 2024-04-21T23:10:27.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:27 smithi040 ceph-mon[27984]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:10:27.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:27 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T23:10:27.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:27 smithi040 ceph-mon[27984]: osd.0 [v2:172.21.15.40:6802/2009758784,v1:172.21.15.40:6803/2009758784] boot 2024-04-21T23:10:27.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:27 smithi040 ceph-mon[27984]: osdmap e8: 1 total, 1 up, 1 in 2024-04-21T23:10:27.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:27 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T23:10:27.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:27 smithi040 ceph-mon[27984]: Detected new or changed devices on smithi040 2024-04-21T23:10:27.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:27 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:27.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:27 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:27.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:27 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:10:27.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:27 smithi040 ceph-mon[27984]: Adjusting osd_memory_target on smithi040 to 17093M 2024-04-21T23:10:27.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:27 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:27.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:27 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:10:27.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:27 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:10:27.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:27 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:27.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:27 smithi174 ceph-mon[31409]: purged_snaps scrub starts 2024-04-21T23:10:27.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:27 smithi174 ceph-mon[31409]: purged_snaps scrub ok 2024-04-21T23:10:27.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:27 smithi174 ceph-mon[31409]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:10:27.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:27 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T23:10:27.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:27 smithi174 ceph-mon[31409]: osd.0 [v2:172.21.15.40:6802/2009758784,v1:172.21.15.40:6803/2009758784] boot 2024-04-21T23:10:27.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:27 smithi174 ceph-mon[31409]: osdmap e8: 1 total, 1 up, 1 in 2024-04-21T23:10:27.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:27 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T23:10:27.945 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:27 smithi174 ceph-mon[31409]: Detected new or changed devices on smithi040 2024-04-21T23:10:27.945 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:27 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:27.945 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:27 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:27.945 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:27 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:10:27.945 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:27 smithi174 ceph-mon[31409]: Adjusting osd_memory_target on smithi040 to 17093M 2024-04-21T23:10:27.945 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:27 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:27.945 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:27 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:10:27.945 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:27 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:10:27.945 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:27 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:28.544 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:10:28.543+0000 7f06db1ee700 1 -- 172.21.15.111:0/2779264165 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06d4102380 msgr2=0x7f06d41027e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:28.544 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:10:28.543+0000 7f06db1ee700 1 --2- 172.21.15.111:0/2779264165 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06d4102380 0x7f06d41027e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f06c40099c0 tx=0x7f06c4009cd0 comp rx=0 tx=0).stop 2024-04-21T23:10:28.545 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:10:28.543+0000 7f06db1ee700 1 -- 172.21.15.111:0/2779264165 shutdown_connections 2024-04-21T23:10:28.545 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:10:28.543+0000 7f06db1ee700 1 --2- 172.21.15.111:0/2779264165 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06d4102d20 0x7f06d41095d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:28.545 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:10:28.543+0000 7f06db1ee700 1 --2- 172.21.15.111:0/2779264165 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06d4101180 0x7f06d41015a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:28.545 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:10:28.543+0000 7f06db1ee700 1 --2- 172.21.15.111:0/2779264165 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06d4102380 0x7f06d41027e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:28.545 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:10:28.543+0000 7f06db1ee700 1 -- 172.21.15.111:0/2779264165 >> 172.21.15.111:0/2779264165 conn(0x7f06d40fc720 msgr2=0x7f06d40feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:28.545 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:10:28.543+0000 7f06db1ee700 1 -- 172.21.15.111:0/2779264165 shutdown_connections 2024-04-21T23:10:28.545 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:10:28.543+0000 7f06db1ee700 1 -- 172.21.15.111:0/2779264165 wait complete. 2024-04-21T23:10:28.545 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:10:28.544+0000 7f06db1ee700 1 Processor -- start 2024-04-21T23:10:28.545 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:10:28.544+0000 7f06db1ee700 1 -- start start 2024-04-21T23:10:28.545 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:10:28.544+0000 7f06db1ee700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06d4101180 0x7f06d4196290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:28.545 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:10:28.544+0000 7f06db1ee700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06d4102380 0x7f06d41967d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:28.545 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:10:28.544+0000 7f06db1ee700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06d4102d20 0x7f06d419d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:28.545 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:10:28.544+0000 7f06db1ee700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f06d410bcf0 con 0x7f06d4102d20 2024-04-21T23:10:28.546 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:10:28.544+0000 7f06db1ee700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f06d410bb70 con 0x7f06d4101180 2024-04-21T23:10:28.546 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:10:28.544+0000 7f06db1ee700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f06d410be70 con 0x7f06d4102380 2024-04-21T23:10:28.546 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:10:28.544+0000 7f06d8f8a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06d4101180 0x7f06d4196290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:28.546 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:10:28.544+0000 7f06d8f8a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06d4101180 0x7f06d4196290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60704/0 (socket says 172.21.15.111:60704) 2024-04-21T23:10:28.546 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:10:28.544+0000 7f06d8f8a700 1 -- 172.21.15.111:0/652581141 learned_addr learned my addr 172.21.15.111:0/652581141 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-04-21T23:10:28.546 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:10:28.544+0000 7f06d978b700 1 --2- 172.21.15.111:0/652581141 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06d4102d20 0x7f06d419d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:28.546 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:10:28.544+0000 7f06cbfff700 1 --2- 172.21.15.111:0/652581141 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06d4102380 0x7f06d41967d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:28.546 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:10:28.544+0000 7f06d8f8a700 1 -- 172.21.15.111:0/652581141 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06d4102380 msgr2=0x7f06d41967d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:28.546 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:10:28.544+0000 7f06d8f8a700 1 --2- 172.21.15.111:0/652581141 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06d4102380 0x7f06d41967d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:28.546 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:10:28.544+0000 7f06d8f8a700 1 -- 172.21.15.111:0/652581141 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06d4102d20 msgr2=0x7f06d419d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:28.546 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:10:28.544+0000 7f06d8f8a700 1 --2- 172.21.15.111:0/652581141 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06d4102d20 0x7f06d419d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:28.546 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:10:28.544+0000 7f06d8f8a700 1 -- 172.21.15.111:0/652581141 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06d419def0 con 0x7f06d4101180 2024-04-21T23:10:28.546 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:10:28.544+0000 7f06d978b700 1 --2- 172.21.15.111:0/652581141 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06d4102d20 0x7f06d419d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:10:28.547 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:10:28.544+0000 7f06d8f8a700 1 --2- 172.21.15.111:0/652581141 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06d4101180 0x7f06d4196290 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f06d000c970 tx=0x7f06d000cd30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:28.547 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:10:28.545+0000 7f06c9ffb700 1 -- 172.21.15.111:0/652581141 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d0004220 con 0x7f06d4101180 2024-04-21T23:10:28.547 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:10:28.545+0000 7f06c9ffb700 1 -- 172.21.15.111:0/652581141 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06d0007d80 con 0x7f06d4101180 2024-04-21T23:10:28.547 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:10:28.545+0000 7f06db1ee700 1 -- 172.21.15.111:0/652581141 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06d419e1e0 con 0x7f06d4101180 2024-04-21T23:10:28.547 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:10:28.545+0000 7f06c9ffb700 1 -- 172.21.15.111:0/652581141 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d000f450 con 0x7f06d4101180 2024-04-21T23:10:28.547 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:10:28.545+0000 7f06db1ee700 1 -- 172.21.15.111:0/652581141 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06d419e7a0 con 0x7f06d4101180 2024-04-21T23:10:28.548 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:10:28.546+0000 7f06db1ee700 1 -- 172.21.15.111:0/652581141 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06d4068f20 con 0x7f06d4101180 2024-04-21T23:10:28.548 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:10:28.547+0000 7f06c9ffb700 1 -- 172.21.15.111:0/652581141 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 15) ==== 137777+0+0 (secure 0 0 0) 0x7f06d00078a0 con 0x7f06d4101180 2024-04-21T23:10:28.548 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:10:28.548+0000 7f06c9ffb700 1 --2- 172.21.15.111:0/652581141 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f06bc0a4da0 0x7f06bc0a7260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:28.548 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:10:28.548+0000 7f06c9ffb700 1 -- 172.21.15.111:0/652581141 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7f06d00cd5a0 con 0x7f06d4101180 2024-04-21T23:10:28.548 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:10:28.548+0000 7f06cbfff700 1 --2- 172.21.15.111:0/652581141 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f06bc0a4da0 0x7f06bc0a7260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:28.548 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:10:28.548+0000 7f06cbfff700 1 --2- 172.21.15.111:0/652581141 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f06bc0a4da0 0x7f06bc0a7260 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f06c4000c00 tx=0x7f06c4005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:28.550 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:10:28.551+0000 7f06c9ffb700 1 -- 172.21.15.111:0/652581141 <== mon.2 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f06d0014030 con 0x7f06d4101180 2024-04-21T23:10:28.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:28 smithi111 ceph-mon[31085]: osdmap e9: 1 total, 1 up, 1 in 2024-04-21T23:10:28.754 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:10:28.755+0000 7f06db1ee700 1 -- 172.21.15.111:0/652581141 --> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f06d4062670 con 0x7f06bc0a4da0 2024-04-21T23:10:28.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:28 smithi040 ceph-mon[27984]: osdmap e9: 1 total, 1 up, 1 in 2024-04-21T23:10:28.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:28 smithi174 ceph-mon[31409]: osdmap e9: 1 total, 1 up, 1 in 2024-04-21T23:10:29.626 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:29 smithi111 ceph-mon[31085]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:10:29.626 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:29 smithi111 ceph-mon[31085]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:10:29.626 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:29 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:10:29.626 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:29 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:10:29.627 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:29 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:10:29.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:29 smithi040 ceph-mon[27984]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:10:29.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:29 smithi040 ceph-mon[27984]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:10:29.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:29 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:10:29.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:29 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:10:29.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:29 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:10:29.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:29 smithi174 ceph-mon[31409]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:10:29.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:29 smithi174 ceph-mon[31409]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:10:29.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:29 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:10:29.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:29 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:10:29.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:29 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:10:31.934 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:31 smithi040 ceph-mon[27984]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:10:31.934 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:31 smithi040 ceph-mon[27984]: from='client.? 172.21.15.111:0/4043674836' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fe5fe11b-0e84-4359-83a7-1ba45105755a"}]: dispatch 2024-04-21T23:10:31.934 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:31 smithi040 ceph-mon[27984]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fe5fe11b-0e84-4359-83a7-1ba45105755a"}]: dispatch 2024-04-21T23:10:31.934 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:31 smithi040 ceph-mon[27984]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fe5fe11b-0e84-4359-83a7-1ba45105755a"}]': finished 2024-04-21T23:10:31.934 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:31 smithi040 ceph-mon[27984]: osdmap e10: 2 total, 1 up, 2 in 2024-04-21T23:10:31.934 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:31 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T23:10:31.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:31 smithi174 ceph-mon[31409]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:10:31.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:31 smithi174 ceph-mon[31409]: from='client.? 172.21.15.111:0/4043674836' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fe5fe11b-0e84-4359-83a7-1ba45105755a"}]: dispatch 2024-04-21T23:10:31.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:31 smithi174 ceph-mon[31409]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fe5fe11b-0e84-4359-83a7-1ba45105755a"}]: dispatch 2024-04-21T23:10:31.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:31 smithi174 ceph-mon[31409]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fe5fe11b-0e84-4359-83a7-1ba45105755a"}]': finished 2024-04-21T23:10:31.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:31 smithi174 ceph-mon[31409]: osdmap e10: 2 total, 1 up, 2 in 2024-04-21T23:10:31.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:31 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T23:10:31.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:31 smithi111 ceph-mon[31085]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:10:31.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:31 smithi111 ceph-mon[31085]: from='client.? 172.21.15.111:0/4043674836' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fe5fe11b-0e84-4359-83a7-1ba45105755a"}]: dispatch 2024-04-21T23:10:31.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:31 smithi111 ceph-mon[31085]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fe5fe11b-0e84-4359-83a7-1ba45105755a"}]: dispatch 2024-04-21T23:10:31.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:31 smithi111 ceph-mon[31085]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fe5fe11b-0e84-4359-83a7-1ba45105755a"}]': finished 2024-04-21T23:10:31.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:31 smithi111 ceph-mon[31085]: osdmap e10: 2 total, 1 up, 2 in 2024-04-21T23:10:31.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:31 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T23:10:32.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:32 smithi040 ceph-mon[27984]: from='client.? 172.21.15.111:0/2138388382' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:10:32.938 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:32 smithi174 ceph-mon[31409]: from='client.? 172.21.15.111:0/2138388382' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:10:32.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:32 smithi111 ceph-mon[31085]: from='client.? 172.21.15.111:0/2138388382' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:10:33.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:33 smithi040 ceph-mon[27984]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:10:33.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:33 smithi174 ceph-mon[31409]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:10:33.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:33 smithi111 ceph-mon[31085]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:10:35.563 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:35 smithi111 ceph-mon[31085]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:10:35.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:35 smithi040 ceph-mon[27984]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:10:35.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:35 smithi174 ceph-mon[31409]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:10:37.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:37 smithi111 ceph-mon[31085]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:10:37.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:37 smithi040 ceph-mon[27984]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:10:37.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:37 smithi174 ceph-mon[31409]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:10:38.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:38 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-21T23:10:38.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:38 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:10:38.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:38 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-21T23:10:38.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:38 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:10:38.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:38 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-21T23:10:38.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:38 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:10:39.833 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:39 smithi040 ceph-mon[27984]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:10:39.834 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:39 smithi040 ceph-mon[27984]: Deploying daemon osd.1 on smithi111 2024-04-21T23:10:39.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:39 smithi174 ceph-mon[31409]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:10:39.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:39 smithi174 ceph-mon[31409]: Deploying daemon osd.1 on smithi111 2024-04-21T23:10:39.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:39 smithi111 ceph-mon[31085]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:10:39.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:39 smithi111 ceph-mon[31085]: Deploying daemon osd.1 on smithi111 2024-04-21T23:10:41.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:41 smithi111 ceph-mon[31085]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:10:41.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:41 smithi040 ceph-mon[27984]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:10:41.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:41 smithi174 ceph-mon[31409]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:10:43.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:43 smithi040 ceph-mon[27984]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:10:43.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:43 smithi174 ceph-mon[31409]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:10:43.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:43 smithi111 ceph-mon[31085]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:10:44.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:44 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:44.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:44 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:44.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:44 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:10:44.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:44 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:44.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:44 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:44.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:44 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:10:44.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:44 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:44.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:44 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:44.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:44 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:10:45.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:45 smithi174 ceph-mon[31409]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:10:45.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:45 smithi111 ceph-mon[31085]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:10:46.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:45 smithi040 ceph-mon[27984]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:10:46.772 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:10:46.773+0000 7f06c9ffb700 1 -- 172.21.15.111:0/652581141 <== mgr.14148 v2:172.21.15.40:6800/2500760788 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f06d4062670 con 0x7f06bc0a4da0 2024-04-21T23:10:46.772 INFO:teuthology.orchestra.run.smithi111.stdout:Created osd(s) 1 on host 'smithi111' 2024-04-21T23:10:46.774 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:10:46.775+0000 7f06db1ee700 1 -- 172.21.15.111:0/652581141 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f06bc0a4da0 msgr2=0x7f06bc0a7260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:46.774 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:10:46.775+0000 7f06db1ee700 1 --2- 172.21.15.111:0/652581141 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f06bc0a4da0 0x7f06bc0a7260 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f06c4000c00 tx=0x7f06c4005fd0 comp rx=0 tx=0).stop 2024-04-21T23:10:46.774 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:10:46.775+0000 7f06db1ee700 1 -- 172.21.15.111:0/652581141 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06d4101180 msgr2=0x7f06d4196290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:46.774 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:10:46.775+0000 7f06db1ee700 1 --2- 172.21.15.111:0/652581141 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06d4101180 0x7f06d4196290 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f06d000c970 tx=0x7f06d000cd30 comp rx=0 tx=0).stop 2024-04-21T23:10:46.774 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:10:46.775+0000 7f06db1ee700 1 -- 172.21.15.111:0/652581141 shutdown_connections 2024-04-21T23:10:46.775 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:10:46.775+0000 7f06db1ee700 1 --2- 172.21.15.111:0/652581141 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f06bc0a4da0 0x7f06bc0a7260 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:46.775 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:10:46.775+0000 7f06db1ee700 1 --2- 172.21.15.111:0/652581141 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06d4102d20 0x7f06d419d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:46.775 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:10:46.775+0000 7f06db1ee700 1 --2- 172.21.15.111:0/652581141 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06d4101180 0x7f06d4196290 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:46.775 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:10:46.775+0000 7f06db1ee700 1 --2- 172.21.15.111:0/652581141 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06d4102380 0x7f06d41967d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:46.775 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:10:46.775+0000 7f06db1ee700 1 -- 172.21.15.111:0/652581141 >> 172.21.15.111:0/652581141 conn(0x7f06d40fc720 msgr2=0x7f06d4105680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:46.775 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:10:46.775+0000 7f06db1ee700 1 -- 172.21.15.111:0/652581141 shutdown_connections 2024-04-21T23:10:46.775 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:10:46.775+0000 7f06db1ee700 1 -- 172.21.15.111:0/652581141 wait complete. 2024-04-21T23:10:47.625 DEBUG:teuthology.orchestra.run.smithi111:osd.1> sudo journalctl -f -n 0 -u ceph-d814e4dc-0033-11ef-bc93-c7b262605968@osd.1.service 2024-04-21T23:10:47.627 INFO:tasks.cephadm:Deploying osd.2 on smithi174 with /dev/vg_nvme/lv_4... 2024-04-21T23:10:47.627 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-21T23:10:47.883 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.c/config 2024-04-21T23:10:47.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:47 smithi174 ceph-mon[31409]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:10:47.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:47 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:47.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:47 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:47.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:47 smithi174 ceph-mon[31409]: from='osd.1 [v2:172.21.15.111:6800/1763903836,v1:172.21.15.111:6801/1763903836]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-21T23:10:47.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:47 smithi174 ceph-mon[31409]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-21T23:10:47.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:47 smithi111 ceph-mon[31085]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:10:47.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:47 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:47.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:47 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:47.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:47 smithi111 ceph-mon[31085]: from='osd.1 [v2:172.21.15.111:6800/1763903836,v1:172.21.15.111:6801/1763903836]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-21T23:10:47.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:47 smithi111 ceph-mon[31085]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-21T23:10:48.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:47 smithi040 ceph-mon[27984]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:10:48.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:47 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:48.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:47 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:48.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:47 smithi040 ceph-mon[27984]: from='osd.1 [v2:172.21.15.111:6800/1763903836,v1:172.21.15.111:6801/1763903836]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-21T23:10:48.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:47 smithi040 ceph-mon[27984]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-21T23:10:48.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:48 smithi111 ceph-mon[31085]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-21T23:10:48.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:48 smithi111 ceph-mon[31085]: from='osd.1 [v2:172.21.15.111:6800/1763903836,v1:172.21.15.111:6801/1763903836]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-21T23:10:48.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:48 smithi111 ceph-mon[31085]: osdmap e11: 2 total, 1 up, 2 in 2024-04-21T23:10:48.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:48 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T23:10:48.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:48 smithi111 ceph-mon[31085]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-21T23:10:48.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:48 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:48.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:48 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:48.945 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 21 23:10:48 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-1[35676]: 2024-04-21T23:10:48.742+0000 7f3778bfd700 -1 osd.1 0 waiting for initial osdmap 2024-04-21T23:10:48.945 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 21 23:10:48 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-1[35676]: 2024-04-21T23:10:48.745+0000 7f37749f0700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T23:10:49.019 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:48 smithi174 ceph-mon[31409]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-21T23:10:49.019 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:48 smithi174 ceph-mon[31409]: from='osd.1 [v2:172.21.15.111:6800/1763903836,v1:172.21.15.111:6801/1763903836]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-21T23:10:49.019 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:48 smithi174 ceph-mon[31409]: osdmap e11: 2 total, 1 up, 2 in 2024-04-21T23:10:49.019 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:48 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T23:10:49.019 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:48 smithi174 ceph-mon[31409]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-21T23:10:49.019 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:48 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:49.019 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:48 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:49.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:48 smithi040 ceph-mon[27984]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-21T23:10:49.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:48 smithi040 ceph-mon[27984]: from='osd.1 [v2:172.21.15.111:6800/1763903836,v1:172.21.15.111:6801/1763903836]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-21T23:10:49.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:48 smithi040 ceph-mon[27984]: osdmap e11: 2 total, 1 up, 2 in 2024-04-21T23:10:49.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:48 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T23:10:49.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:48 smithi040 ceph-mon[27984]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-21T23:10:49.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:48 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:49.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:48 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:49.739 INFO:teuthology.orchestra.run.smithi174.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-21T23:10:49.740 INFO:teuthology.orchestra.run.smithi174.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-21T23:10:49.740 INFO:teuthology.orchestra.run.smithi174.stderr: stderr: 10+0 records in 2024-04-21T23:10:49.740 INFO:teuthology.orchestra.run.smithi174.stderr:10+0 records out 2024-04-21T23:10:49.740 INFO:teuthology.orchestra.run.smithi174.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112531 s, 932 MB/s 2024-04-21T23:10:49.740 INFO:teuthology.orchestra.run.smithi174.stderr:--> Zapping successful for: 2024-04-21T23:10:49.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:49 smithi111 ceph-mon[31085]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:10:49.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:49 smithi111 ceph-mon[31085]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-04-21T23:10:49.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:49 smithi111 ceph-mon[31085]: osdmap e12: 2 total, 1 up, 2 in 2024-04-21T23:10:49.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:49 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T23:10:49.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:49 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T23:10:49.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:49 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T23:10:50.051 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:49 smithi174 ceph-mon[31409]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:10:50.051 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:49 smithi174 ceph-mon[31409]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-04-21T23:10:50.051 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:49 smithi174 ceph-mon[31409]: osdmap e12: 2 total, 1 up, 2 in 2024-04-21T23:10:50.051 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:49 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T23:10:50.051 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:49 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T23:10:50.052 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:49 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T23:10:50.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:49 smithi040 ceph-mon[27984]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:10:50.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:49 smithi040 ceph-mon[27984]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-04-21T23:10:50.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:49 smithi040 ceph-mon[27984]: osdmap e12: 2 total, 1 up, 2 in 2024-04-21T23:10:50.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:49 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T23:10:50.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:49 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T23:10:50.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:49 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T23:10:50.355 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi174:vg_nvme/lv_4 2024-04-21T23:10:50.604 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.c/config 2024-04-21T23:10:50.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:50 smithi040 ceph-mon[27984]: purged_snaps scrub starts 2024-04-21T23:10:50.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:50 smithi040 ceph-mon[27984]: purged_snaps scrub ok 2024-04-21T23:10:50.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:50 smithi040 ceph-mon[27984]: osd.1 [v2:172.21.15.111:6800/1763903836,v1:172.21.15.111:6801/1763903836] boot 2024-04-21T23:10:50.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:50 smithi040 ceph-mon[27984]: osdmap e13: 2 total, 2 up, 2 in 2024-04-21T23:10:50.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:50 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T23:10:50.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:50 smithi174 ceph-mon[31409]: purged_snaps scrub starts 2024-04-21T23:10:50.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:50 smithi174 ceph-mon[31409]: purged_snaps scrub ok 2024-04-21T23:10:50.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:50 smithi174 ceph-mon[31409]: osd.1 [v2:172.21.15.111:6800/1763903836,v1:172.21.15.111:6801/1763903836] boot 2024-04-21T23:10:50.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:50 smithi174 ceph-mon[31409]: osdmap e13: 2 total, 2 up, 2 in 2024-04-21T23:10:50.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:50 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T23:10:51.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:50 smithi111 ceph-mon[31085]: purged_snaps scrub starts 2024-04-21T23:10:51.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:50 smithi111 ceph-mon[31085]: purged_snaps scrub ok 2024-04-21T23:10:51.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:50 smithi111 ceph-mon[31085]: osd.1 [v2:172.21.15.111:6800/1763903836,v1:172.21.15.111:6801/1763903836] boot 2024-04-21T23:10:51.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:50 smithi111 ceph-mon[31085]: osdmap e13: 2 total, 2 up, 2 in 2024-04-21T23:10:51.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:50 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T23:10:51.897 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:51 smithi174 ceph-mon[31409]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:10:51.897 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:51 smithi174 ceph-mon[31409]: osdmap e14: 2 total, 2 up, 2 in 2024-04-21T23:10:51.897 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:51 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:51.897 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:51 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:51.897 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:51 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:10:51.897 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:51 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:51.897 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:51 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:10:51.897 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:51 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:10:51.897 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:51 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:51.897 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:51 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:10:51.898 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:51 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:10:51.898 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:51 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:10:51.898 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:51 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:52.124 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:10:52.123+0000 7fc2018f0700 1 -- 172.21.15.174:0/458553541 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1fc102380 msgr2=0x7fc1fc1027e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:52.124 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:10:52.123+0000 7fc2018f0700 1 --2- 172.21.15.174:0/458553541 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1fc102380 0x7fc1fc1027e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc1e4009990 tx=0x7fc1e4009ca0 comp rx=0 tx=0).stop 2024-04-21T23:10:52.124 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:10:52.123+0000 7fc2018f0700 1 -- 172.21.15.174:0/458553541 shutdown_connections 2024-04-21T23:10:52.124 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:10:52.123+0000 7fc2018f0700 1 --2- 172.21.15.174:0/458553541 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1fc102d20 0x7fc1fc1095d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:52.125 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:10:52.123+0000 7fc2018f0700 1 --2- 172.21.15.174:0/458553541 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1fc101180 0x7fc1fc1015a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:52.125 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:10:52.123+0000 7fc2018f0700 1 --2- 172.21.15.174:0/458553541 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1fc102380 0x7fc1fc1027e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:52.125 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:10:52.123+0000 7fc2018f0700 1 -- 172.21.15.174:0/458553541 >> 172.21.15.174:0/458553541 conn(0x7fc1fc0fc720 msgr2=0x7fc1fc0feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:52.125 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:10:52.123+0000 7fc2018f0700 1 -- 172.21.15.174:0/458553541 shutdown_connections 2024-04-21T23:10:52.125 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:10:52.123+0000 7fc2018f0700 1 -- 172.21.15.174:0/458553541 wait complete. 2024-04-21T23:10:52.125 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:10:52.123+0000 7fc2018f0700 1 Processor -- start 2024-04-21T23:10:52.125 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:10:52.123+0000 7fc2018f0700 1 -- start start 2024-04-21T23:10:52.125 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:10:52.124+0000 7fc2018f0700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1fc101180 0x7fc1fc06b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:52.125 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:10:52.124+0000 7fc2018f0700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1fc102380 0x7fc1fc19f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:52.125 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:10:52.124+0000 7fc2018f0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1fc102d20 0x7fc1fc1a1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:52.125 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:10:52.124+0000 7fc2018f0700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc1fc10ce60 con 0x7fc1fc102380 2024-04-21T23:10:52.125 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:10:52.124+0000 7fc2018f0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc1fc10cce0 con 0x7fc1fc102d20 2024-04-21T23:10:52.125 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:10:52.124+0000 7fc2018f0700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc1fc10cfe0 con 0x7fc1fc101180 2024-04-21T23:10:52.125 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:10:52.124+0000 7fc1faffd700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1fc101180 0x7fc1fc06b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:52.126 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:10:52.124+0000 7fc1faffd700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1fc101180 0x7fc1fc06b540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.174:36224/0 (socket says 172.21.15.174:36224) 2024-04-21T23:10:52.126 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:10:52.124+0000 7fc1faffd700 1 -- 172.21.15.174:0/2819542453 learned_addr learned my addr 172.21.15.174:0/2819542453 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-04-21T23:10:52.126 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:10:52.124+0000 7fc1fb7fe700 1 --2- 172.21.15.174:0/2819542453 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1fc102d20 0x7fc1fc1a1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:52.126 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:10:52.124+0000 7fc1f27fc700 1 --2- 172.21.15.174:0/2819542453 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1fc102380 0x7fc1fc19f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:52.126 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:10:52.124+0000 7fc1faffd700 1 -- 172.21.15.174:0/2819542453 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1fc102d20 msgr2=0x7fc1fc1a1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:52.126 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:10:52.124+0000 7fc1faffd700 1 --2- 172.21.15.174:0/2819542453 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1fc102d20 0x7fc1fc1a1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:52.126 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:10:52.124+0000 7fc1faffd700 1 -- 172.21.15.174:0/2819542453 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1fc102380 msgr2=0x7fc1fc19f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:52.126 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:10:52.124+0000 7fc1faffd700 1 --2- 172.21.15.174:0/2819542453 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1fc102380 0x7fc1fc19f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:52.126 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:10:52.124+0000 7fc1faffd700 1 -- 172.21.15.174:0/2819542453 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1fc1a24a0 con 0x7fc1fc101180 2024-04-21T23:10:52.126 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:10:52.124+0000 7fc1f27fc700 1 --2- 172.21.15.174:0/2819542453 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1fc102380 0x7fc1fc19f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:10:52.126 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:10:52.124+0000 7fc1faffd700 1 --2- 172.21.15.174:0/2819542453 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1fc101180 0x7fc1fc06b540 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fc1ec00b500 tx=0x7fc1ec00b810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:52.126 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:10:52.124+0000 7fc2008ee700 1 -- 172.21.15.174:0/2819542453 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1ec015020 con 0x7fc1fc101180 2024-04-21T23:10:52.126 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:10:52.125+0000 7fc2008ee700 1 -- 172.21.15.174:0/2819542453 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1ec011d10 con 0x7fc1fc101180 2024-04-21T23:10:52.127 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:10:52.125+0000 7fc2018f0700 1 -- 172.21.15.174:0/2819542453 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1fc1a2790 con 0x7fc1fc101180 2024-04-21T23:10:52.127 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:10:52.125+0000 7fc2008ee700 1 -- 172.21.15.174:0/2819542453 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1ec010e50 con 0x7fc1fc101180 2024-04-21T23:10:52.127 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:10:52.125+0000 7fc2018f0700 1 -- 172.21.15.174:0/2819542453 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1fc1a2d50 con 0x7fc1fc101180 2024-04-21T23:10:52.127 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:10:52.126+0000 7fc2018f0700 1 -- 172.21.15.174:0/2819542453 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1fc068f20 con 0x7fc1fc101180 2024-04-21T23:10:52.128 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:10:52.127+0000 7fc2008ee700 1 -- 172.21.15.174:0/2819542453 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 15) ==== 137777+0+0 (secure 0 0 0) 0x7fc1ec018020 con 0x7fc1fc101180 2024-04-21T23:10:52.128 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:10:52.128+0000 7fc2008ee700 1 --2- 172.21.15.174:0/2819542453 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fc1e80a2d30 0x7fc1e80a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:52.128 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:10:52.128+0000 7fc2008ee700 1 -- 172.21.15.174:0/2819542453 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7fc1ec0cd3b0 con 0x7fc1fc101180 2024-04-21T23:10:52.128 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:10:52.128+0000 7fc1f27fc700 1 --2- 172.21.15.174:0/2819542453 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fc1e80a2d30 0x7fc1e80a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:52.128 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:10:52.129+0000 7fc1f27fc700 1 --2- 172.21.15.174:0/2819542453 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fc1e80a2d30 0x7fc1e80a51f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc1e40039c0 tx=0x7fc1e4005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:52.131 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:10:52.132+0000 7fc2008ee700 1 -- 172.21.15.174:0/2819542453 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fc1ec01f020 con 0x7fc1fc101180 2024-04-21T23:10:52.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:51 smithi040 ceph-mon[27984]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:10:52.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:51 smithi040 ceph-mon[27984]: osdmap e14: 2 total, 2 up, 2 in 2024-04-21T23:10:52.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:51 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:52.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:51 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:52.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:51 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:10:52.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:51 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:52.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:51 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:10:52.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:51 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:10:52.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:51 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:52.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:51 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:10:52.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:51 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:10:52.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:51 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:10:52.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:51 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:52.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:51 smithi111 ceph-mon[31085]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:10:52.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:51 smithi111 ceph-mon[31085]: osdmap e14: 2 total, 2 up, 2 in 2024-04-21T23:10:52.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:51 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:52.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:51 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:52.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:51 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:10:52.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:51 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:52.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:51 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:10:52.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:51 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:10:52.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:51 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:52.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:51 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:10:52.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:51 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:10:52.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:51 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:10:52.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:51 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:10:52.332 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:10:52.332+0000 7fc2018f0700 1 -- 172.21.15.174:0/2819542453 --> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fc1fc062670 con 0x7fc1e80a2d30 2024-04-21T23:10:53.025 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:52 smithi174 ceph-mon[31409]: Detected new or changed devices on smithi111 2024-04-21T23:10:53.026 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:52 smithi174 ceph-mon[31409]: Adjusting osd_memory_target on smithi111 to 17088M 2024-04-21T23:10:53.026 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:52 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:10:53.026 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:52 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:10:53.026 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:52 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:10:53.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:52 smithi040 ceph-mon[27984]: Detected new or changed devices on smithi111 2024-04-21T23:10:53.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:52 smithi040 ceph-mon[27984]: Adjusting osd_memory_target on smithi111 to 17088M 2024-04-21T23:10:53.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:52 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:10:53.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:52 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:10:53.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:52 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:10:53.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:52 smithi111 ceph-mon[31085]: Detected new or changed devices on smithi111 2024-04-21T23:10:53.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:52 smithi111 ceph-mon[31085]: Adjusting osd_memory_target on smithi111 to 17088M 2024-04-21T23:10:53.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:52 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:10:53.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:52 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:10:53.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:52 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:10:54.030 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:53 smithi174 ceph-mon[31409]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:10:54.030 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:53 smithi174 ceph-mon[31409]: from='client.24157 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:10:54.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:53 smithi040 ceph-mon[27984]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:10:54.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:53 smithi040 ceph-mon[27984]: from='client.24157 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:10:54.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:53 smithi111 ceph-mon[31085]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:10:54.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:53 smithi111 ceph-mon[31085]: from='client.24157 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:10:56.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:55 smithi040 ceph-mon[27984]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:10:56.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:55 smithi040 ceph-mon[27984]: from='client.? 172.21.15.174:0/854484792' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "813ad735-5cc2-4da0-ae16-6db238395fbe"}]: dispatch 2024-04-21T23:10:56.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:55 smithi040 ceph-mon[27984]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "813ad735-5cc2-4da0-ae16-6db238395fbe"}]: dispatch 2024-04-21T23:10:56.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:55 smithi040 ceph-mon[27984]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "813ad735-5cc2-4da0-ae16-6db238395fbe"}]': finished 2024-04-21T23:10:56.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:55 smithi040 ceph-mon[27984]: osdmap e15: 3 total, 2 up, 3 in 2024-04-21T23:10:56.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:55 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T23:10:56.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:55 smithi040 ceph-mon[27984]: from='client.? 172.21.15.174:0/3201185558' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:10:56.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:55 smithi174 ceph-mon[31409]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:10:56.195 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:55 smithi174 ceph-mon[31409]: from='client.? 172.21.15.174:0/854484792' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "813ad735-5cc2-4da0-ae16-6db238395fbe"}]: dispatch 2024-04-21T23:10:56.195 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:55 smithi174 ceph-mon[31409]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "813ad735-5cc2-4da0-ae16-6db238395fbe"}]: dispatch 2024-04-21T23:10:56.195 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:55 smithi174 ceph-mon[31409]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "813ad735-5cc2-4da0-ae16-6db238395fbe"}]': finished 2024-04-21T23:10:56.195 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:55 smithi174 ceph-mon[31409]: osdmap e15: 3 total, 2 up, 3 in 2024-04-21T23:10:56.195 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:55 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T23:10:56.195 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:55 smithi174 ceph-mon[31409]: from='client.? 172.21.15.174:0/3201185558' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:10:56.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:55 smithi111 ceph-mon[31085]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:10:56.196 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:55 smithi111 ceph-mon[31085]: from='client.? 172.21.15.174:0/854484792' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "813ad735-5cc2-4da0-ae16-6db238395fbe"}]: dispatch 2024-04-21T23:10:56.196 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:55 smithi111 ceph-mon[31085]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "813ad735-5cc2-4da0-ae16-6db238395fbe"}]: dispatch 2024-04-21T23:10:56.196 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:55 smithi111 ceph-mon[31085]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "813ad735-5cc2-4da0-ae16-6db238395fbe"}]': finished 2024-04-21T23:10:56.196 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:55 smithi111 ceph-mon[31085]: osdmap e15: 3 total, 2 up, 3 in 2024-04-21T23:10:56.196 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:55 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T23:10:56.196 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:55 smithi111 ceph-mon[31085]: from='client.? 172.21.15.174:0/3201185558' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:10:58.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:57 smithi040 ceph-mon[27984]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:10:58.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:57 smithi174 ceph-mon[31409]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:10:58.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:57 smithi111 ceph-mon[31085]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:11:00.059 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:10:59 smithi174 ceph-mon[31409]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:11:00.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:10:59 smithi040 ceph-mon[27984]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:11:00.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:10:59 smithi111 ceph-mon[31085]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:11:02.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:01 smithi040 ceph-mon[27984]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:11:02.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:01 smithi174 ceph-mon[31409]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:11:02.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:01 smithi111 ceph-mon[31085]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:11:02.913 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:02 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-21T23:11:02.913 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:02 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:11:02.913 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:02 smithi174 ceph-mon[31409]: Deploying daemon osd.2 on smithi174 2024-04-21T23:11:03.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:02 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-21T23:11:03.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:02 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:11:03.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:02 smithi040 ceph-mon[27984]: Deploying daemon osd.2 on smithi174 2024-04-21T23:11:03.193 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:02 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-21T23:11:03.193 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:02 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:11:03.193 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:02 smithi111 ceph-mon[31085]: Deploying daemon osd.2 on smithi174 2024-04-21T23:11:04.093 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:03 smithi174 ceph-mon[31409]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:11:04.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:03 smithi040 ceph-mon[27984]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:11:04.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:03 smithi111 ceph-mon[31085]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:11:06.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:05 smithi040 ceph-mon[27984]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:11:06.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:05 smithi111 ceph-mon[31085]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:11:06.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:05 smithi174 ceph-mon[31409]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:11:08.080 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:07 smithi174 ceph-mon[31409]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:11:08.080 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:07 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:08.080 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:07 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:08.080 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:07 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:11:08.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:07 smithi040 ceph-mon[27984]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:11:08.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:07 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:08.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:07 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:08.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:07 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:11:08.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:07 smithi111 ceph-mon[31085]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:11:08.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:07 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:08.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:07 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:08.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:07 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:11:09.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:09 smithi174 ceph-mon[31409]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:11:09.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:09 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:09.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:09 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:09.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:09 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:11:09.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:09 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:11:09.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:09 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:10.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:09 smithi040 ceph-mon[27984]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:11:10.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:09 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:10.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:09 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:10.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:09 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:11:10.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:09 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:11:10.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:09 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:10.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:09 smithi111 ceph-mon[31085]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:11:10.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:09 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:10.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:09 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:10.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:09 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:11:10.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:09 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:11:10.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:09 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:11.003 INFO:teuthology.orchestra.run.smithi174.stdout:Created osd(s) 2 on host 'smithi174' 2024-04-21T23:11:11.004 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:11:11.003+0000 7fc2008ee700 1 -- 172.21.15.174:0/2819542453 <== mgr.14148 v2:172.21.15.40:6800/2500760788 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fc1fc062670 con 0x7fc1e80a2d30 2024-04-21T23:11:11.005 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:11:11.006+0000 7fc2018f0700 1 -- 172.21.15.174:0/2819542453 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fc1e80a2d30 msgr2=0x7fc1e80a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:11.005 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:11:11.006+0000 7fc2018f0700 1 --2- 172.21.15.174:0/2819542453 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fc1e80a2d30 0x7fc1e80a51f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc1e40039c0 tx=0x7fc1e4005fb0 comp rx=0 tx=0).stop 2024-04-21T23:11:11.005 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:11:11.006+0000 7fc2018f0700 1 -- 172.21.15.174:0/2819542453 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1fc101180 msgr2=0x7fc1fc06b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:11.005 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:11:11.006+0000 7fc2018f0700 1 --2- 172.21.15.174:0/2819542453 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1fc101180 0x7fc1fc06b540 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fc1ec00b500 tx=0x7fc1ec00b810 comp rx=0 tx=0).stop 2024-04-21T23:11:11.006 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:11:11.006+0000 7fc2018f0700 1 -- 172.21.15.174:0/2819542453 shutdown_connections 2024-04-21T23:11:11.006 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:11:11.006+0000 7fc2018f0700 1 --2- 172.21.15.174:0/2819542453 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1fc101180 0x7fc1fc06b540 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:11.006 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:11:11.006+0000 7fc2018f0700 1 --2- 172.21.15.174:0/2819542453 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fc1e80a2d30 0x7fc1e80a51f0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:11.006 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:11:11.006+0000 7fc2018f0700 1 --2- 172.21.15.174:0/2819542453 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1fc102d20 0x7fc1fc1a1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:11.006 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:11:11.006+0000 7fc2018f0700 1 --2- 172.21.15.174:0/2819542453 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1fc102380 0x7fc1fc19f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:11.006 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:11:11.006+0000 7fc2018f0700 1 -- 172.21.15.174:0/2819542453 >> 172.21.15.174:0/2819542453 conn(0x7fc1fc0fc720 msgr2=0x7fc1fc1057a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:11.006 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:11:11.006+0000 7fc2018f0700 1 -- 172.21.15.174:0/2819542453 shutdown_connections 2024-04-21T23:11:11.006 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:11:11.006+0000 7fc2018f0700 1 -- 172.21.15.174:0/2819542453 wait complete. 2024-04-21T23:11:11.668 DEBUG:teuthology.orchestra.run.smithi174:osd.2> sudo journalctl -f -n 0 -u ceph-d814e4dc-0033-11ef-bc93-c7b262605968@osd.2.service 2024-04-21T23:11:11.670 INFO:tasks.cephadm:Deploying osd.3 on smithi040 with /dev/vg_nvme/lv_3... 2024-04-21T23:11:11.670 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-21T23:11:11.919 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:11:12.272 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:11 smithi174 ceph-mon[31409]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:11:12.272 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:11 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:12.272 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:11 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:12.272 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:11 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:11:12.272 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:11 smithi174 ceph-mon[31409]: from='osd.2 [v2:172.21.15.174:6800/439695825,v1:172.21.15.174:6801/439695825]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-21T23:11:12.289 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:11 smithi040 ceph-mon[27984]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:11:12.289 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:11 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:12.289 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:11 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:12.290 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:11 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:11:12.290 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:11 smithi040 ceph-mon[27984]: from='osd.2 [v2:172.21.15.174:6800/439695825,v1:172.21.15.174:6801/439695825]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-21T23:11:12.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:11 smithi111 ceph-mon[31085]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:11:12.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:11 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:12.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:11 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:12.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:11 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:11:12.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:11 smithi111 ceph-mon[31085]: from='osd.2 [v2:172.21.15.174:6800/439695825,v1:172.21.15.174:6801/439695825]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-21T23:11:13.125 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:13 smithi174 ceph-mon[31409]: from='osd.2 [v2:172.21.15.174:6800/439695825,v1:172.21.15.174:6801/439695825]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-21T23:11:13.125 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:13 smithi174 ceph-mon[31409]: osdmap e16: 3 total, 2 up, 3 in 2024-04-21T23:11:13.125 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:13 smithi174 ceph-mon[31409]: from='osd.2 [v2:172.21.15.174:6800/439695825,v1:172.21.15.174:6801/439695825]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-04-21T23:11:13.125 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:13 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T23:11:13.125 INFO:journalctl@ceph.osd.2.smithi174.stdout:Apr 21 23:11:13 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-2[36550]: 2024-04-21T23:11:13.002+0000 7f0f44c19700 -1 osd.2 0 waiting for initial osdmap 2024-04-21T23:11:13.126 INFO:journalctl@ceph.osd.2.smithi174.stdout:Apr 21 23:11:13 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-2[36550]: 2024-04-21T23:11:13.008+0000 7f0f40a0c700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T23:11:13.269 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:13 smithi111 ceph-mon[31085]: from='osd.2 [v2:172.21.15.174:6800/439695825,v1:172.21.15.174:6801/439695825]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-21T23:11:13.270 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:13 smithi111 ceph-mon[31085]: osdmap e16: 3 total, 2 up, 3 in 2024-04-21T23:11:13.270 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:13 smithi111 ceph-mon[31085]: from='osd.2 [v2:172.21.15.174:6800/439695825,v1:172.21.15.174:6801/439695825]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-04-21T23:11:13.270 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:13 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T23:11:13.324 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:13 smithi040 ceph-mon[27984]: from='osd.2 [v2:172.21.15.174:6800/439695825,v1:172.21.15.174:6801/439695825]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-21T23:11:13.324 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:13 smithi040 ceph-mon[27984]: osdmap e16: 3 total, 2 up, 3 in 2024-04-21T23:11:13.324 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:13 smithi040 ceph-mon[27984]: from='osd.2 [v2:172.21.15.174:6800/439695825,v1:172.21.15.174:6801/439695825]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-04-21T23:11:13.324 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:13 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T23:11:14.081 INFO:teuthology.orchestra.run.smithi040.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-21T23:11:14.081 INFO:teuthology.orchestra.run.smithi040.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-21T23:11:14.081 INFO:teuthology.orchestra.run.smithi040.stderr: stderr: 10+0 records in 2024-04-21T23:11:14.081 INFO:teuthology.orchestra.run.smithi040.stderr:10+0 records out 2024-04-21T23:11:14.081 INFO:teuthology.orchestra.run.smithi040.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111054 s, 944 MB/s 2024-04-21T23:11:14.082 INFO:teuthology.orchestra.run.smithi040.stderr:--> Zapping successful for: 2024-04-21T23:11:14.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:14 smithi174 ceph-mon[31409]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:11:14.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:14 smithi174 ceph-mon[31409]: from='osd.2 [v2:172.21.15.174:6800/439695825,v1:172.21.15.174:6801/439695825]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-04-21T23:11:14.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:14 smithi174 ceph-mon[31409]: osdmap e17: 3 total, 2 up, 3 in 2024-04-21T23:11:14.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:14 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T23:11:14.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:14 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T23:11:14.418 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:14 smithi040 ceph-mon[27984]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:11:14.418 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:14 smithi040 ceph-mon[27984]: from='osd.2 [v2:172.21.15.174:6800/439695825,v1:172.21.15.174:6801/439695825]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-04-21T23:11:14.418 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:14 smithi040 ceph-mon[27984]: osdmap e17: 3 total, 2 up, 3 in 2024-04-21T23:11:14.418 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:14 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T23:11:14.418 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:14 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T23:11:14.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:14 smithi111 ceph-mon[31085]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:11:14.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:14 smithi111 ceph-mon[31085]: from='osd.2 [v2:172.21.15.174:6800/439695825,v1:172.21.15.174:6801/439695825]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-04-21T23:11:14.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:14 smithi111 ceph-mon[31085]: osdmap e17: 3 total, 2 up, 3 in 2024-04-21T23:11:14.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:14 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T23:11:14.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:14 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T23:11:14.864 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi040:vg_nvme/lv_3 2024-04-21T23:11:15.106 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:11:15.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:15 smithi040 ceph-mon[27984]: purged_snaps scrub starts 2024-04-21T23:11:15.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:15 smithi040 ceph-mon[27984]: purged_snaps scrub ok 2024-04-21T23:11:15.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:15 smithi040 ceph-mon[27984]: osd.2 [v2:172.21.15.174:6800/439695825,v1:172.21.15.174:6801/439695825] boot 2024-04-21T23:11:15.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:15 smithi040 ceph-mon[27984]: osdmap e18: 3 total, 3 up, 3 in 2024-04-21T23:11:15.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:15 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T23:11:15.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:15 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-21T23:11:15.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:15 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:15.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:15 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:15.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:15 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:11:15.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:15 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:15.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:15 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:11:15.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:15 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:11:15.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:15 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:15.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:15 smithi111 ceph-mon[31085]: purged_snaps scrub starts 2024-04-21T23:11:15.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:15 smithi111 ceph-mon[31085]: purged_snaps scrub ok 2024-04-21T23:11:15.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:15 smithi111 ceph-mon[31085]: osd.2 [v2:172.21.15.174:6800/439695825,v1:172.21.15.174:6801/439695825] boot 2024-04-21T23:11:15.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:15 smithi111 ceph-mon[31085]: osdmap e18: 3 total, 3 up, 3 in 2024-04-21T23:11:15.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:15 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T23:11:15.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:15 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-21T23:11:15.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:15 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:15.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:15 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:15.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:15 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:11:15.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:15 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:15.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:15 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:11:15.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:15 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:11:15.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:15 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:15.446 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:15 smithi174 ceph-mon[31409]: purged_snaps scrub starts 2024-04-21T23:11:15.446 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:15 smithi174 ceph-mon[31409]: purged_snaps scrub ok 2024-04-21T23:11:15.446 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:15 smithi174 ceph-mon[31409]: osd.2 [v2:172.21.15.174:6800/439695825,v1:172.21.15.174:6801/439695825] boot 2024-04-21T23:11:15.446 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:15 smithi174 ceph-mon[31409]: osdmap e18: 3 total, 3 up, 3 in 2024-04-21T23:11:15.446 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:15 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T23:11:15.446 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:15 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-21T23:11:15.446 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:15 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:15.446 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:15 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:15.446 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:15 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:11:15.446 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:15 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:15.446 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:15 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:11:15.447 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:15 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:11:15.447 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:15 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:16.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:16 smithi040 ceph-mon[27984]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:11:16.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:16 smithi040 ceph-mon[27984]: Detected new or changed devices on smithi174 2024-04-21T23:11:16.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:16 smithi040 ceph-mon[27984]: Adjusting osd_memory_target on smithi174 to 17088M 2024-04-21T23:11:16.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:16 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-21T23:11:16.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:16 smithi040 ceph-mon[27984]: osdmap e19: 3 total, 3 up, 3 in 2024-04-21T23:11:16.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:16 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-21T23:11:16.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:16 smithi174 ceph-mon[31409]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:11:16.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:16 smithi174 ceph-mon[31409]: Detected new or changed devices on smithi174 2024-04-21T23:11:16.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:16 smithi174 ceph-mon[31409]: Adjusting osd_memory_target on smithi174 to 17088M 2024-04-21T23:11:16.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:16 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-21T23:11:16.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:16 smithi174 ceph-mon[31409]: osdmap e19: 3 total, 3 up, 3 in 2024-04-21T23:11:16.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:16 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-21T23:11:16.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:16 smithi111 ceph-mon[31085]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:11:16.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:16 smithi111 ceph-mon[31085]: Detected new or changed devices on smithi174 2024-04-21T23:11:16.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:16 smithi111 ceph-mon[31085]: Adjusting osd_memory_target on smithi174 to 17088M 2024-04-21T23:11:16.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:16 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-21T23:11:16.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:16 smithi111 ceph-mon[31085]: osdmap e19: 3 total, 3 up, 3 in 2024-04-21T23:11:16.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:16 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-21T23:11:16.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:11:16.877+0000 7fe64d1be700 1 -- 172.21.15.40:0/3681819525 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6481034b0 msgr2=0x7fe648103910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:16.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:11:16.877+0000 7fe64d1be700 1 --2- 172.21.15.40:0/3681819525 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6481034b0 0x7fe648103910 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fe6300099c0 tx=0x7fe630009cd0 comp rx=0 tx=0).stop 2024-04-21T23:11:16.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:11:16.877+0000 7fe64d1be700 1 -- 172.21.15.40:0/3681819525 shutdown_connections 2024-04-21T23:11:16.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:11:16.877+0000 7fe64d1be700 1 --2- 172.21.15.40:0/3681819525 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe6481022b0 0x7fe6481026d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:16.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:11:16.877+0000 7fe64d1be700 1 --2- 172.21.15.40:0/3681819525 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe648103e50 0x7fe64810a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:16.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:11:16.877+0000 7fe64d1be700 1 --2- 172.21.15.40:0/3681819525 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6481034b0 0x7fe648103910 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:16.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:11:16.877+0000 7fe64d1be700 1 -- 172.21.15.40:0/3681819525 >> 172.21.15.40:0/3681819525 conn(0x7fe6480fd850 msgr2=0x7fe6480ffc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:16.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:11:16.877+0000 7fe64d1be700 1 -- 172.21.15.40:0/3681819525 shutdown_connections 2024-04-21T23:11:16.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:11:16.877+0000 7fe64d1be700 1 -- 172.21.15.40:0/3681819525 wait complete. 2024-04-21T23:11:16.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:11:16.878+0000 7fe64d1be700 1 Processor -- start 2024-04-21T23:11:16.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:11:16.878+0000 7fe64d1be700 1 -- start start 2024-04-21T23:11:16.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:11:16.878+0000 7fe64d1be700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6481022b0 0x7fe648198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:16.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:11:16.878+0000 7fe64d1be700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe6481034b0 0x7fe6481989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:16.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:11:16.878+0000 7fe64d1be700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe648103e50 0x7fe64819fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:16.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:11:16.878+0000 7fe64d1be700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe64810de80 con 0x7fe648103e50 2024-04-21T23:11:16.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:11:16.878+0000 7fe64d1be700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe64810dd00 con 0x7fe6481022b0 2024-04-21T23:11:16.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:11:16.878+0000 7fe64d1be700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe64810e000 con 0x7fe6481034b0 2024-04-21T23:11:16.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:11:16.879+0000 7fe64659c700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe6481034b0 0x7fe6481989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:16.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:11:16.879+0000 7fe646d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6481022b0 0x7fe648198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:16.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:11:16.879+0000 7fe64759e700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe648103e50 0x7fe64819fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:16.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:11:16.879+0000 7fe64659c700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe6481034b0 0x7fe6481989a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.40:42612/0 (socket says 172.21.15.40:42612) 2024-04-21T23:11:16.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:11:16.879+0000 7fe64659c700 1 -- 172.21.15.40:0/825332362 learned_addr learned my addr 172.21.15.40:0/825332362 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:11:16.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:11:16.879+0000 7fe64659c700 1 -- 172.21.15.40:0/825332362 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6481022b0 msgr2=0x7fe648198460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:16.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:11:16.879+0000 7fe64659c700 1 --2- 172.21.15.40:0/825332362 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6481022b0 0x7fe648198460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:16.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:11:16.879+0000 7fe64659c700 1 -- 172.21.15.40:0/825332362 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe648103e50 msgr2=0x7fe64819fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:16.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:11:16.879+0000 7fe64659c700 1 --2- 172.21.15.40:0/825332362 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe648103e50 0x7fe64819fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:16.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:11:16.879+0000 7fe64659c700 1 -- 172.21.15.40:0/825332362 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6481a00c0 con 0x7fe6481034b0 2024-04-21T23:11:16.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:11:16.879+0000 7fe64759e700 1 --2- 172.21.15.40:0/825332362 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe648103e50 0x7fe64819fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:11:16.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:11:16.879+0000 7fe64659c700 1 --2- 172.21.15.40:0/825332362 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe6481034b0 0x7fe6481989a0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fe63000b580 tx=0x7fe63000fa00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:16.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:11:16.879+0000 7fe63ffff700 1 -- 172.21.15.40:0/825332362 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe630027070 con 0x7fe6481034b0 2024-04-21T23:11:16.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:11:16.879+0000 7fe63ffff700 1 -- 172.21.15.40:0/825332362 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe63000fe30 con 0x7fe6481034b0 2024-04-21T23:11:16.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:11:16.879+0000 7fe64d1be700 1 -- 172.21.15.40:0/825332362 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6481a0350 con 0x7fe6481034b0 2024-04-21T23:11:16.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:11:16.880+0000 7fe63ffff700 1 -- 172.21.15.40:0/825332362 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe630017650 con 0x7fe6481034b0 2024-04-21T23:11:16.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:11:16.880+0000 7fe64d1be700 1 -- 172.21.15.40:0/825332362 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6481a07a0 con 0x7fe6481034b0 2024-04-21T23:11:16.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:11:16.880+0000 7fe64d1be700 1 -- 172.21.15.40:0/825332362 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe648068f20 con 0x7fe6481034b0 2024-04-21T23:11:16.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:11:16.882+0000 7fe63ffff700 1 -- 172.21.15.40:0/825332362 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 15) ==== 137777+0+0 (secure 0 0 0) 0x7fe63002a070 con 0x7fe6481034b0 2024-04-21T23:11:16.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:11:16.883+0000 7fe63ffff700 1 --2- 172.21.15.40:0/825332362 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fe6340a2c10 0x7fe6340a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:16.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:11:16.883+0000 7fe646d9d700 1 --2- 172.21.15.40:0/825332362 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fe6340a2c10 0x7fe6340a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:16.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:11:16.883+0000 7fe63ffff700 1 -- 172.21.15.40:0/825332362 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 3237+0+0 (secure 0 0 0) 0x7fe6300cec70 con 0x7fe6481034b0 2024-04-21T23:11:16.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:11:16.883+0000 7fe646d9d700 1 --2- 172.21.15.40:0/825332362 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fe6340a2c10 0x7fe6340a50d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe648103310 tx=0x7fe638009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:16.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:11:16.886+0000 7fe63ffff700 1 -- 172.21.15.40:0/825332362 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fe630031080 con 0x7fe6481034b0 2024-04-21T23:11:17.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:11:17.105+0000 7fe64d1be700 1 -- 172.21.15.40:0/825332362 --> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fe648062670 con 0x7fe6340a2c10 2024-04-21T23:11:17.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:17 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-21T23:11:17.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:17 smithi040 ceph-mon[27984]: osdmap e20: 3 total, 3 up, 3 in 2024-04-21T23:11:17.432 INFO:journalctl@ceph.osd.0.smithi040.stdout:Apr 21 23:11:17 smithi040 sudo[44847]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-21T23:11:17.433 INFO:journalctl@ceph.osd.0.smithi040.stdout:Apr 21 23:11:17 smithi040 sudo[44847]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T23:11:17.433 INFO:journalctl@ceph.osd.0.smithi040.stdout:Apr 21 23:11:17 smithi040 sudo[44847]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T23:11:17.433 INFO:journalctl@ceph.osd.0.smithi040.stdout:Apr 21 23:11:17 smithi040 sudo[44847]: pam_unix(sudo:session): session closed for user root 2024-04-21T23:11:17.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:17 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-21T23:11:17.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:17 smithi174 ceph-mon[31409]: osdmap e20: 3 total, 3 up, 3 in 2024-04-21T23:11:17.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:17 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-21T23:11:17.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:17 smithi111 ceph-mon[31085]: osdmap e20: 3 total, 3 up, 3 in 2024-04-21T23:11:17.899 INFO:journalctl@ceph.osd.0.smithi040.stdout:Apr 21 23:11:17 smithi040 sudo[44936]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-21T23:11:17.900 INFO:journalctl@ceph.osd.0.smithi040.stdout:Apr 21 23:11:17 smithi040 sudo[44936]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T23:11:17.900 INFO:journalctl@ceph.osd.0.smithi040.stdout:Apr 21 23:11:17 smithi040 sudo[44936]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T23:11:17.900 INFO:journalctl@ceph.osd.0.smithi040.stdout:Apr 21 23:11:17 smithi040 sudo[44936]: pam_unix(sudo:session): session closed for user root 2024-04-21T23:11:18.095 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 21 23:11:17 smithi111 sudo[39783]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-21T23:11:18.095 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 21 23:11:17 smithi111 sudo[39783]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T23:11:18.095 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 21 23:11:17 smithi111 sudo[39783]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T23:11:18.095 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 21 23:11:17 smithi111 sudo[39783]: pam_unix(sudo:session): session closed for user root 2024-04-21T23:11:18.096 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:18 smithi111 ceph-mon[31085]: pgmap v83: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:11:18.096 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:18 smithi111 ceph-mon[31085]: osdmap e21: 3 total, 3 up, 3 in 2024-04-21T23:11:18.096 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:18 smithi111 ceph-mon[31085]: from='client.24190 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:11:18.096 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:18 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:11:18.096 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:18 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:11:18.096 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:18 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:11:18.096 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:18 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:18.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:18 smithi040 ceph-mon[27984]: pgmap v83: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:11:18.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:18 smithi040 ceph-mon[27984]: osdmap e21: 3 total, 3 up, 3 in 2024-04-21T23:11:18.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:18 smithi040 ceph-mon[27984]: from='client.24190 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:11:18.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:18 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:11:18.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:18 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:11:18.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:18 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:11:18.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:18 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:18.364 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:18 smithi174 ceph-mon[31409]: pgmap v83: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:11:18.364 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:18 smithi174 ceph-mon[31409]: osdmap e21: 3 total, 3 up, 3 in 2024-04-21T23:11:18.365 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:18 smithi174 ceph-mon[31409]: from='client.24190 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:11:18.365 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:18 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:11:18.365 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:18 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:11:18.365 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:18 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:11:18.365 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:18 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:18.444 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 21 23:11:18 smithi111 sudo[39786]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-21T23:11:18.444 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 21 23:11:18 smithi111 sudo[39786]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T23:11:18.444 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 21 23:11:18 smithi111 sudo[39786]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T23:11:18.444 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 21 23:11:18 smithi111 sudo[39786]: pam_unix(sudo:session): session closed for user root 2024-04-21T23:11:18.619 INFO:journalctl@ceph.osd.2.smithi174.stdout:Apr 21 23:11:18 smithi174 sudo[40439]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-21T23:11:18.619 INFO:journalctl@ceph.osd.2.smithi174.stdout:Apr 21 23:11:18 smithi174 sudo[40439]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T23:11:18.619 INFO:journalctl@ceph.osd.2.smithi174.stdout:Apr 21 23:11:18 smithi174 sudo[40439]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T23:11:18.619 INFO:journalctl@ceph.osd.2.smithi174.stdout:Apr 21 23:11:18 smithi174 sudo[40439]: pam_unix(sudo:session): session closed for user root 2024-04-21T23:11:18.943 INFO:journalctl@ceph.osd.2.smithi174.stdout:Apr 21 23:11:18 smithi174 sudo[40442]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-21T23:11:18.944 INFO:journalctl@ceph.osd.2.smithi174.stdout:Apr 21 23:11:18 smithi174 sudo[40442]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T23:11:18.944 INFO:journalctl@ceph.osd.2.smithi174.stdout:Apr 21 23:11:18 smithi174 sudo[40442]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T23:11:18.944 INFO:journalctl@ceph.osd.2.smithi174.stdout:Apr 21 23:11:18 smithi174 sudo[40442]: pam_unix(sudo:session): session closed for user root 2024-04-21T23:11:19.355 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:19 smithi040 sudo[45017]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-21T23:11:19.355 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:19 smithi040 sudo[45017]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T23:11:19.356 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:19 smithi040 sudo[45017]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T23:11:19.356 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:19 smithi040 sudo[45017]: pam_unix(sudo:session): session closed for user root 2024-04-21T23:11:19.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:19 smithi040 sudo[45037]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-21T23:11:19.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:19 smithi040 sudo[45037]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T23:11:19.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:19 smithi040 sudo[45037]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T23:11:19.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:19 smithi040 sudo[45037]: pam_unix(sudo:session): session closed for user root 2024-04-21T23:11:19.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:19 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:19.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:19 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:19.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:19 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:19.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:19 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:20.068 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:19 smithi174 sudo[40445]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-21T23:11:20.069 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:19 smithi174 sudo[40445]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T23:11:20.069 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:19 smithi174 sudo[40445]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T23:11:20.069 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:19 smithi174 sudo[40445]: pam_unix(sudo:session): session closed for user root 2024-04-21T23:11:20.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:11:20.077+0000 7fe63ffff700 1 -- 172.21.15.40:0/825332362 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fe630099100 con 0x7fe6481034b0 2024-04-21T23:11:20.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:20 smithi174 sudo[40448]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-21T23:11:20.446 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:20 smithi174 sudo[40448]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T23:11:20.446 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:20 smithi174 sudo[40448]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T23:11:20.446 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:20 smithi174 sudo[40448]: pam_unix(sudo:session): session closed for user root 2024-04-21T23:11:20.446 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:20 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:20.446 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:20 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:20.563 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:20 smithi111 sudo[39789]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-21T23:11:20.563 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:20 smithi111 sudo[39789]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T23:11:20.563 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:20 smithi111 sudo[39789]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T23:11:20.563 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:20 smithi111 sudo[39789]: pam_unix(sudo:session): session closed for user root 2024-04-21T23:11:20.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:20 smithi040 ceph-mon[27984]: pgmap v85: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:11:20.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:20 smithi040 ceph-mon[27984]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-21T23:11:20.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:20 smithi040 ceph-mon[27984]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-21T23:11:20.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:20 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T23:11:20.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:20 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:11:20.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:20 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:11:20.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:20 smithi040 ceph-mon[27984]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-21T23:11:20.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:20 smithi040 ceph-mon[27984]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-21T23:11:20.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:20 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T23:11:20.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:20 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:11:20.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:20 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:11:20.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:20 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/2324161516' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "09becb03-b7ce-4ffc-917d-b7900ec37e9c"}]: dispatch 2024-04-21T23:11:20.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:20 smithi040 ceph-mon[27984]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-21T23:11:20.934 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:20 smithi040 ceph-mon[27984]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-21T23:11:20.934 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:20 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T23:11:20.934 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:20 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:11:20.934 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:20 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:11:20.934 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:20 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/2324161516' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "09becb03-b7ce-4ffc-917d-b7900ec37e9c"}]': finished 2024-04-21T23:11:20.934 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:20 smithi040 ceph-mon[27984]: osdmap e22: 4 total, 3 up, 4 in 2024-04-21T23:11:20.934 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:20 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T23:11:20.934 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:20 smithi040 ceph-mon[27984]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-21T23:11:20.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:20 smithi174 ceph-mon[31409]: pgmap v85: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:11:20.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:20 smithi174 ceph-mon[31409]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-21T23:11:20.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:20 smithi174 ceph-mon[31409]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-21T23:11:20.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:20 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T23:11:20.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:20 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:11:20.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:20 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:11:20.945 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:20 smithi174 ceph-mon[31409]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-21T23:11:20.945 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:20 smithi174 ceph-mon[31409]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-21T23:11:20.945 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:20 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T23:11:20.945 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:20 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:11:20.945 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:20 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:11:20.945 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:20 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/2324161516' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "09becb03-b7ce-4ffc-917d-b7900ec37e9c"}]: dispatch 2024-04-21T23:11:20.945 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:20 smithi174 ceph-mon[31409]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-21T23:11:20.945 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:20 smithi174 ceph-mon[31409]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-21T23:11:20.945 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:20 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T23:11:20.945 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:20 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:11:20.945 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:20 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:11:20.945 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:20 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/2324161516' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "09becb03-b7ce-4ffc-917d-b7900ec37e9c"}]': finished 2024-04-21T23:11:20.945 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:20 smithi174 ceph-mon[31409]: osdmap e22: 4 total, 3 up, 4 in 2024-04-21T23:11:20.946 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:20 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T23:11:20.946 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:20 smithi174 ceph-mon[31409]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-21T23:11:20.946 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:20 smithi111 sudo[39792]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-21T23:11:20.946 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:20 smithi111 sudo[39792]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T23:11:20.946 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:20 smithi111 sudo[39792]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T23:11:20.946 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:20 smithi111 sudo[39792]: pam_unix(sudo:session): session closed for user root 2024-04-21T23:11:20.946 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:20 smithi111 ceph-mon[31085]: pgmap v85: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:11:20.946 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:20 smithi111 ceph-mon[31085]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-21T23:11:20.946 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:20 smithi111 ceph-mon[31085]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-21T23:11:20.947 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:20 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T23:11:20.947 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:20 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:11:20.947 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:20 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:11:20.947 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:20 smithi111 ceph-mon[31085]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-21T23:11:20.947 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:20 smithi111 ceph-mon[31085]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-21T23:11:20.947 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:20 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T23:11:20.947 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:20 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:11:20.947 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:20 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:11:20.947 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:20 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/2324161516' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "09becb03-b7ce-4ffc-917d-b7900ec37e9c"}]: dispatch 2024-04-21T23:11:20.947 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:20 smithi111 ceph-mon[31085]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-21T23:11:20.947 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:20 smithi111 ceph-mon[31085]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-21T23:11:20.947 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:20 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T23:11:20.947 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:20 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:11:20.947 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:20 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:11:20.948 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:20 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/2324161516' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "09becb03-b7ce-4ffc-917d-b7900ec37e9c"}]': finished 2024-04-21T23:11:20.948 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:20 smithi111 ceph-mon[31085]: osdmap e22: 4 total, 3 up, 4 in 2024-04-21T23:11:20.948 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:20 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T23:11:20.948 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:20 smithi111 ceph-mon[31085]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-21T23:11:21.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:21 smithi040 ceph-mon[27984]: pgmap v87: 1 pgs: 1 creating+peering; 0 B data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:11:21.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:21 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/688091650' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:11:21.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:21 smithi174 ceph-mon[31409]: pgmap v87: 1 pgs: 1 creating+peering; 0 B data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:11:21.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:21 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/688091650' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:11:21.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:21 smithi111 ceph-mon[31085]: pgmap v87: 1 pgs: 1 creating+peering; 0 B data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:11:21.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:21 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/688091650' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:11:23.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:23 smithi040 ceph-mon[27984]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:11:23.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:23 smithi174 ceph-mon[31409]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:11:23.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:23 smithi111 ceph-mon[31085]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:11:25.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:25 smithi040 ceph-mon[27984]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:11:25.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:25 smithi174 ceph-mon[31409]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:11:25.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:25 smithi111 ceph-mon[31085]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:11:27.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:27 smithi040 ceph-mon[27984]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:11:27.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:27 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-21T23:11:27.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:27 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:11:27.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:27 smithi174 ceph-mon[31409]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:11:27.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:27 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-21T23:11:27.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:27 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:11:27.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:27 smithi111 ceph-mon[31085]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:11:27.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:27 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-21T23:11:27.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:27 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:11:28.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:28 smithi040 ceph-mon[27984]: Deploying daemon osd.3 on smithi040 2024-04-21T23:11:28.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:28 smithi174 ceph-mon[31409]: Deploying daemon osd.3 on smithi040 2024-04-21T23:11:28.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:28 smithi111 ceph-mon[31085]: Deploying daemon osd.3 on smithi040 2024-04-21T23:11:29.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:29 smithi040 ceph-mon[27984]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:11:29.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:29 smithi174 ceph-mon[31409]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:11:29.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:29 smithi111 ceph-mon[31085]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:11:31.911 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:31 smithi040 ceph-mon[27984]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:11:31.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:31 smithi174 ceph-mon[31409]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:11:31.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:31 smithi111 ceph-mon[31085]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:11:34.342 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:33 smithi040 ceph-mon[27984]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:11:34.343 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:33 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:34.343 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:33 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:11:34.343 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:33 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:34.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:33 smithi174 ceph-mon[31409]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:11:34.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:33 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:34.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:33 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:11:34.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:33 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:34.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:33 smithi111 ceph-mon[31085]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:11:34.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:33 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:34.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:33 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:11:34.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:33 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:36.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:36 smithi040 ceph-mon[27984]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:11:36.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:36 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:36.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:36 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:36.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:36 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:11:36.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:36 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:11:36.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:36 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:36.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:36 smithi174 ceph-mon[31409]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:11:36.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:36 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:36.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:36 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:36.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:36 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:11:36.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:36 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:11:36.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:36 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:36.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:36 smithi111 ceph-mon[31085]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:11:36.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:36 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:36.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:36 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:36.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:36 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:11:36.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:36 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:11:36.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:36 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:36.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:11:36.728+0000 7fe63ffff700 1 -- 172.21.15.40:0/825332362 <== mgr.14148 v2:172.21.15.40:6800/2500760788 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fe648062670 con 0x7fe6340a2c10 2024-04-21T23:11:36.728 INFO:teuthology.orchestra.run.smithi040.stdout:Created osd(s) 3 on host 'smithi040' 2024-04-21T23:11:36.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:11:36.731+0000 7fe64d1be700 1 -- 172.21.15.40:0/825332362 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fe6340a2c10 msgr2=0x7fe6340a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:36.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:11:36.731+0000 7fe64d1be700 1 --2- 172.21.15.40:0/825332362 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fe6340a2c10 0x7fe6340a50d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe648103310 tx=0x7fe638009260 comp rx=0 tx=0).stop 2024-04-21T23:11:36.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:11:36.731+0000 7fe64d1be700 1 -- 172.21.15.40:0/825332362 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe6481034b0 msgr2=0x7fe6481989a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:36.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:11:36.731+0000 7fe64d1be700 1 --2- 172.21.15.40:0/825332362 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe6481034b0 0x7fe6481989a0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fe63000b580 tx=0x7fe63000fa00 comp rx=0 tx=0).stop 2024-04-21T23:11:36.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:11:36.731+0000 7fe64d1be700 1 -- 172.21.15.40:0/825332362 shutdown_connections 2024-04-21T23:11:36.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:11:36.731+0000 7fe64d1be700 1 --2- 172.21.15.40:0/825332362 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fe6340a2c10 0x7fe6340a50d0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:11:36.731+0000 7fe64d1be700 1 --2- 172.21.15.40:0/825332362 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe648103e50 0x7fe64819fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:11:36.731+0000 7fe64d1be700 1 --2- 172.21.15.40:0/825332362 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6481022b0 0x7fe648198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:11:36.731+0000 7fe64d1be700 1 --2- 172.21.15.40:0/825332362 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe6481034b0 0x7fe6481989a0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:11:36.731+0000 7fe64d1be700 1 -- 172.21.15.40:0/825332362 >> 172.21.15.40:0/825332362 conn(0x7fe6480fd850 msgr2=0x7fe6481067b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:36.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:11:36.731+0000 7fe64d1be700 1 -- 172.21.15.40:0/825332362 shutdown_connections 2024-04-21T23:11:36.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:11:36.731+0000 7fe64d1be700 1 -- 172.21.15.40:0/825332362 wait complete. 2024-04-21T23:11:37.434 DEBUG:teuthology.orchestra.run.smithi040:osd.3> sudo journalctl -f -n 0 -u ceph-d814e4dc-0033-11ef-bc93-c7b262605968@osd.3.service 2024-04-21T23:11:37.437 INFO:tasks.cephadm:Deploying osd.4 on smithi111 with /dev/vg_nvme/lv_3... 2024-04-21T23:11:37.437 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-21T23:11:37.691 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.b/config 2024-04-21T23:11:38.054 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:37 smithi111 ceph-mon[31085]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:11:38.055 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:37 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:38.055 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:37 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:38.055 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:37 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:11:38.055 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:37 smithi111 ceph-mon[31085]: from='osd.3 [v2:172.21.15.40:6810/3454312436,v1:172.21.15.40:6811/3454312436]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-21T23:11:38.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:37 smithi040 ceph-mon[27984]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:11:38.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:37 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:38.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:37 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:38.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:37 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:11:38.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:37 smithi040 ceph-mon[27984]: from='osd.3 [v2:172.21.15.40:6810/3454312436,v1:172.21.15.40:6811/3454312436]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-21T23:11:38.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:37 smithi174 ceph-mon[31409]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:11:38.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:37 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:38.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:37 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:38.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:37 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:11:38.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:37 smithi174 ceph-mon[31409]: from='osd.3 [v2:172.21.15.40:6810/3454312436,v1:172.21.15.40:6811/3454312436]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-21T23:11:38.753 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:38 smithi040 ceph-mon[27984]: from='osd.3 [v2:172.21.15.40:6810/3454312436,v1:172.21.15.40:6811/3454312436]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-21T23:11:38.753 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:38 smithi040 ceph-mon[27984]: osdmap e23: 4 total, 3 up, 4 in 2024-04-21T23:11:38.753 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:38 smithi040 ceph-mon[27984]: from='osd.3 [v2:172.21.15.40:6810/3454312436,v1:172.21.15.40:6811/3454312436]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-21T23:11:38.754 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:38 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T23:11:38.991 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:38 smithi111 ceph-mon[31085]: from='osd.3 [v2:172.21.15.40:6810/3454312436,v1:172.21.15.40:6811/3454312436]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-21T23:11:38.991 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:38 smithi111 ceph-mon[31085]: osdmap e23: 4 total, 3 up, 4 in 2024-04-21T23:11:38.991 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:38 smithi111 ceph-mon[31085]: from='osd.3 [v2:172.21.15.40:6810/3454312436,v1:172.21.15.40:6811/3454312436]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-21T23:11:38.991 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:38 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T23:11:39.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:38 smithi174 ceph-mon[31409]: from='osd.3 [v2:172.21.15.40:6810/3454312436,v1:172.21.15.40:6811/3454312436]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-21T23:11:39.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:38 smithi174 ceph-mon[31409]: osdmap e23: 4 total, 3 up, 4 in 2024-04-21T23:11:39.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:38 smithi174 ceph-mon[31409]: from='osd.3 [v2:172.21.15.40:6810/3454312436,v1:172.21.15.40:6811/3454312436]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-21T23:11:39.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:38 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T23:11:39.645 INFO:journalctl@ceph.osd.3.smithi040.stdout:Apr 21 23:11:39 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-3[47087]: 2024-04-21T23:11:39.421+0000 7fe7a6c43700 -1 osd.3 0 waiting for initial osdmap 2024-04-21T23:11:39.646 INFO:journalctl@ceph.osd.3.smithi040.stdout:Apr 21 23:11:39 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-3[47087]: 2024-04-21T23:11:39.427+0000 7fe79e22d700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T23:11:39.681 INFO:teuthology.orchestra.run.smithi111.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-21T23:11:39.681 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-21T23:11:39.681 INFO:teuthology.orchestra.run.smithi111.stderr: stderr: 10+0 records in 2024-04-21T23:11:39.681 INFO:teuthology.orchestra.run.smithi111.stderr:10+0 records out 2024-04-21T23:11:39.681 INFO:teuthology.orchestra.run.smithi111.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111598 s, 940 MB/s 2024-04-21T23:11:39.681 INFO:teuthology.orchestra.run.smithi111.stderr:--> Zapping successful for: 2024-04-21T23:11:39.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:39 smithi040 ceph-mon[27984]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:11:39.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:39 smithi040 ceph-mon[27984]: from='osd.3 [v2:172.21.15.40:6810/3454312436,v1:172.21.15.40:6811/3454312436]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-04-21T23:11:39.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:39 smithi040 ceph-mon[27984]: osdmap e24: 4 total, 3 up, 4 in 2024-04-21T23:11:39.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:39 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T23:11:39.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:39 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T23:11:39.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:39 smithi040 ceph-mon[27984]: from='osd.3 [v2:172.21.15.40:6810/3454312436,v1:172.21.15.40:6811/3454312436]' entity='osd.3' 2024-04-21T23:11:39.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:39 smithi040 ceph-mon[27984]: osd.3 [v2:172.21.15.40:6810/3454312436,v1:172.21.15.40:6811/3454312436] boot 2024-04-21T23:11:39.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:39 smithi040 ceph-mon[27984]: osdmap e25: 4 total, 4 up, 4 in 2024-04-21T23:11:39.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:39 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T23:11:39.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:39 smithi111 ceph-mon[31085]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:11:39.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:39 smithi111 ceph-mon[31085]: from='osd.3 [v2:172.21.15.40:6810/3454312436,v1:172.21.15.40:6811/3454312436]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-04-21T23:11:39.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:39 smithi111 ceph-mon[31085]: osdmap e24: 4 total, 3 up, 4 in 2024-04-21T23:11:39.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:39 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T23:11:39.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:39 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T23:11:39.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:39 smithi111 ceph-mon[31085]: from='osd.3 [v2:172.21.15.40:6810/3454312436,v1:172.21.15.40:6811/3454312436]' entity='osd.3' 2024-04-21T23:11:39.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:39 smithi111 ceph-mon[31085]: osd.3 [v2:172.21.15.40:6810/3454312436,v1:172.21.15.40:6811/3454312436] boot 2024-04-21T23:11:39.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:39 smithi111 ceph-mon[31085]: osdmap e25: 4 total, 4 up, 4 in 2024-04-21T23:11:39.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:39 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T23:11:40.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:39 smithi174 ceph-mon[31409]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:11:40.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:39 smithi174 ceph-mon[31409]: from='osd.3 [v2:172.21.15.40:6810/3454312436,v1:172.21.15.40:6811/3454312436]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-04-21T23:11:40.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:39 smithi174 ceph-mon[31409]: osdmap e24: 4 total, 3 up, 4 in 2024-04-21T23:11:40.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:39 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T23:11:40.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:39 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T23:11:40.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:39 smithi174 ceph-mon[31409]: from='osd.3 [v2:172.21.15.40:6810/3454312436,v1:172.21.15.40:6811/3454312436]' entity='osd.3' 2024-04-21T23:11:40.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:39 smithi174 ceph-mon[31409]: osd.3 [v2:172.21.15.40:6810/3454312436,v1:172.21.15.40:6811/3454312436] boot 2024-04-21T23:11:40.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:39 smithi174 ceph-mon[31409]: osdmap e25: 4 total, 4 up, 4 in 2024-04-21T23:11:40.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:39 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T23:11:40.328 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi111:vg_nvme/lv_3 2024-04-21T23:11:40.585 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.b/config 2024-04-21T23:11:40.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:40 smithi111 ceph-mon[31085]: purged_snaps scrub starts 2024-04-21T23:11:40.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:40 smithi111 ceph-mon[31085]: purged_snaps scrub ok 2024-04-21T23:11:40.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:40 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:40.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:40 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:40.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:40 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:11:40.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:40 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:11:40.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:40 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:40.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:40 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:11:40.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:40 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:11:40.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:40 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:40.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:40 smithi111 ceph-mon[31085]: osdmap e26: 4 total, 4 up, 4 in 2024-04-21T23:11:41.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:40 smithi040 ceph-mon[27984]: purged_snaps scrub starts 2024-04-21T23:11:41.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:40 smithi040 ceph-mon[27984]: purged_snaps scrub ok 2024-04-21T23:11:41.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:40 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:41.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:40 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:41.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:40 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:11:41.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:40 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:11:41.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:40 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:41.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:40 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:11:41.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:40 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:11:41.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:40 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:41.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:40 smithi040 ceph-mon[27984]: osdmap e26: 4 total, 4 up, 4 in 2024-04-21T23:11:41.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:40 smithi174 ceph-mon[31409]: purged_snaps scrub starts 2024-04-21T23:11:41.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:40 smithi174 ceph-mon[31409]: purged_snaps scrub ok 2024-04-21T23:11:41.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:40 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:41.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:40 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:41.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:40 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:11:41.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:40 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:11:41.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:40 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:41.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:40 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:11:41.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:40 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:11:41.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:40 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:41.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:40 smithi174 ceph-mon[31409]: osdmap e26: 4 total, 4 up, 4 in 2024-04-21T23:11:41.934 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:41 smithi111 ceph-mon[31085]: Detected new or changed devices on smithi040 2024-04-21T23:11:41.934 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:41 smithi111 ceph-mon[31085]: Adjusting osd_memory_target on smithi040 to 8546M 2024-04-21T23:11:41.934 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:41 smithi111 ceph-mon[31085]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 396 MiB used, 357 GiB / 358 GiB avail 2024-04-21T23:11:41.934 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:41 smithi111 ceph-mon[31085]: osdmap e27: 4 total, 4 up, 4 in 2024-04-21T23:11:42.172 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:11:42.170+0000 7ff20a37f700 1 -- 172.21.15.111:0/1733834025 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff204105740 msgr2=0x7ff204107b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:42.172 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:11:42.170+0000 7ff20a37f700 1 --2- 172.21.15.111:0/1733834025 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff204105740 0x7ff204107b30 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc00b0a0 tx=0x7ff1fc00b3b0 comp rx=0 tx=0).stop 2024-04-21T23:11:42.172 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:11:42.170+0000 7ff20a37f700 1 -- 172.21.15.111:0/1733834025 shutdown_connections 2024-04-21T23:11:42.172 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:11:42.170+0000 7ff20a37f700 1 --2- 172.21.15.111:0/1733834025 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff204102e10 0x7ff204105200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:42.172 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:11:42.170+0000 7ff20a37f700 1 --2- 172.21.15.111:0/1733834025 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff204108070 0x7ff20410a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:42.172 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:11:42.170+0000 7ff20a37f700 1 --2- 172.21.15.111:0/1733834025 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff204105740 0x7ff204107b30 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:42.172 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:11:42.170+0000 7ff20a37f700 1 -- 172.21.15.111:0/1733834025 >> 172.21.15.111:0/1733834025 conn(0x7ff2040fc780 msgr2=0x7ff2040feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:42.172 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:11:42.170+0000 7ff20a37f700 1 -- 172.21.15.111:0/1733834025 shutdown_connections 2024-04-21T23:11:42.172 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:11:42.170+0000 7ff20a37f700 1 -- 172.21.15.111:0/1733834025 wait complete. 2024-04-21T23:11:42.172 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:11:42.170+0000 7ff20a37f700 1 Processor -- start 2024-04-21T23:11:42.173 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:11:42.170+0000 7ff20a37f700 1 -- start start 2024-04-21T23:11:42.173 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:11:42.171+0000 7ff20a37f700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff204102e10 0x7ff204102720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:42.173 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:11:42.171+0000 7ff20a37f700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff204105740 0x7ff204100dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:42.173 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:11:42.171+0000 7ff20a37f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff204108070 0x7ff204101300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:42.173 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:11:42.171+0000 7ff20a37f700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff20410dec0 con 0x7ff204105740 2024-04-21T23:11:42.173 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:11:42.171+0000 7ff20a37f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff20410dd40 con 0x7ff204108070 2024-04-21T23:11:42.173 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:11:42.171+0000 7ff20a37f700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff20410e040 con 0x7ff204102e10 2024-04-21T23:11:42.173 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:11:42.171+0000 7ff20891c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff204108070 0x7ff204101300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:42.173 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:11:42.171+0000 7ff20891c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff204108070 0x7ff204101300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:51144/0 (socket says 172.21.15.111:51144) 2024-04-21T23:11:42.173 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:11:42.171+0000 7ff20891c700 1 -- 172.21.15.111:0/1200882234 learned_addr learned my addr 172.21.15.111:0/1200882234 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-04-21T23:11:42.173 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:11:42.171+0000 7ff2037fe700 1 --2- 172.21.15.111:0/1200882234 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff204105740 0x7ff204100dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:42.174 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:11:42.171+0000 7ff203fff700 1 --2- 172.21.15.111:0/1200882234 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff204102e10 0x7ff204102720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:42.174 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:11:42.171+0000 7ff20891c700 1 -- 172.21.15.111:0/1200882234 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff204102e10 msgr2=0x7ff204102720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:42.174 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:11:42.171+0000 7ff20891c700 1 --2- 172.21.15.111:0/1200882234 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff204102e10 0x7ff204102720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:42.174 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:11:42.171+0000 7ff20891c700 1 -- 172.21.15.111:0/1200882234 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff204105740 msgr2=0x7ff204100dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:42.174 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:11:42.171+0000 7ff20891c700 1 --2- 172.21.15.111:0/1200882234 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff204105740 0x7ff204100dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:42.174 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:11:42.171+0000 7ff20891c700 1 -- 172.21.15.111:0/1200882234 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff204101bd0 con 0x7ff204108070 2024-04-21T23:11:42.174 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:11:42.171+0000 7ff20891c700 1 --2- 172.21.15.111:0/1200882234 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff204108070 0x7ff204101300 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff1ec00d980 tx=0x7ff1ec00dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:42.174 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:11:42.171+0000 7ff2017fa700 1 -- 172.21.15.111:0/1200882234 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1ec017070 con 0x7ff204108070 2024-04-21T23:11:42.174 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:11:42.172+0000 7ff20a37f700 1 -- 172.21.15.111:0/1200882234 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2041a25a0 con 0x7ff204108070 2024-04-21T23:11:42.174 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:11:42.172+0000 7ff2017fa700 1 -- 172.21.15.111:0/1200882234 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1ec005020 con 0x7ff204108070 2024-04-21T23:11:42.174 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:11:42.172+0000 7ff2017fa700 1 -- 172.21.15.111:0/1200882234 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1ec009b50 con 0x7ff204108070 2024-04-21T23:11:42.175 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:11:42.172+0000 7ff20a37f700 1 -- 172.21.15.111:0/1200882234 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2041a2a60 con 0x7ff204108070 2024-04-21T23:11:42.175 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:11:42.173+0000 7ff20a37f700 1 -- 172.21.15.111:0/1200882234 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff204068f20 con 0x7ff204108070 2024-04-21T23:11:42.176 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:11:42.174+0000 7ff2017fa700 1 -- 172.21.15.111:0/1200882234 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7ff1ec009cf0 con 0x7ff204108070 2024-04-21T23:11:42.176 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:11:42.174+0000 7ff2017fa700 1 --2- 172.21.15.111:0/1200882234 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7ff1f00a2c20 0x7ff1f00a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:42.176 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:11:42.174+0000 7ff2017fa700 1 -- 172.21.15.111:0/1200882234 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 3698+0+0 (secure 0 0 0) 0x7ff1ec0cdd80 con 0x7ff204108070 2024-04-21T23:11:42.176 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:11:42.174+0000 7ff203fff700 1 --2- 172.21.15.111:0/1200882234 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7ff1f00a2c20 0x7ff1f00a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:42.176 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:11:42.175+0000 7ff203fff700 1 --2- 172.21.15.111:0/1200882234 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7ff1f00a2c20 0x7ff1f00a50e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff1f4005950 tx=0x7ff1f40058e0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:42.178 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:11:42.179+0000 7ff2017fa700 1 -- 172.21.15.111:0/1200882234 <== mon.2 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7ff1ec020020 con 0x7ff204108070 2024-04-21T23:11:42.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:41 smithi040 ceph-mon[27984]: Detected new or changed devices on smithi040 2024-04-21T23:11:42.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:41 smithi040 ceph-mon[27984]: Adjusting osd_memory_target on smithi040 to 8546M 2024-04-21T23:11:42.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:41 smithi040 ceph-mon[27984]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 396 MiB used, 357 GiB / 358 GiB avail 2024-04-21T23:11:42.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:41 smithi040 ceph-mon[27984]: osdmap e27: 4 total, 4 up, 4 in 2024-04-21T23:11:42.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:41 smithi174 ceph-mon[31409]: Detected new or changed devices on smithi040 2024-04-21T23:11:42.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:41 smithi174 ceph-mon[31409]: Adjusting osd_memory_target on smithi040 to 8546M 2024-04-21T23:11:42.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:41 smithi174 ceph-mon[31409]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 396 MiB used, 357 GiB / 358 GiB avail 2024-04-21T23:11:42.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:41 smithi174 ceph-mon[31409]: osdmap e27: 4 total, 4 up, 4 in 2024-04-21T23:11:42.381 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:11:42.381+0000 7ff20a37f700 1 -- 172.21.15.111:0/1200882234 --> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7ff204062670 con 0x7ff1f00a2c20 2024-04-21T23:11:42.995 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:42 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:11:42.996 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:42 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:11:42.996 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:42 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:11:43.011 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:42 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:11:43.011 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:42 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:11:43.011 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:42 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:11:43.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:42 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:11:43.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:42 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:11:43.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:42 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:11:44.023 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:43 smithi111 ceph-mon[31085]: pgmap v103: 1 pgs: 1 remapped+peering; 577 KiB data, 396 MiB used, 357 GiB / 358 GiB avail 2024-04-21T23:11:44.024 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:43 smithi111 ceph-mon[31085]: from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:11:44.024 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:43 smithi111 ceph-mon[31085]: osdmap e28: 4 total, 4 up, 4 in 2024-04-21T23:11:44.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:43 smithi040 ceph-mon[27984]: pgmap v103: 1 pgs: 1 remapped+peering; 577 KiB data, 396 MiB used, 357 GiB / 358 GiB avail 2024-04-21T23:11:44.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:43 smithi040 ceph-mon[27984]: from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:11:44.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:43 smithi040 ceph-mon[27984]: osdmap e28: 4 total, 4 up, 4 in 2024-04-21T23:11:44.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:43 smithi174 ceph-mon[31409]: pgmap v103: 1 pgs: 1 remapped+peering; 577 KiB data, 396 MiB used, 357 GiB / 358 GiB avail 2024-04-21T23:11:44.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:43 smithi174 ceph-mon[31409]: from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:11:44.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:43 smithi174 ceph-mon[31409]: osdmap e28: 4 total, 4 up, 4 in 2024-04-21T23:11:45.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:44 smithi040 ceph-mon[27984]: osdmap e29: 4 total, 4 up, 4 in 2024-04-21T23:11:45.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:44 smithi174 ceph-mon[31409]: osdmap e29: 4 total, 4 up, 4 in 2024-04-21T23:11:45.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:44 smithi111 ceph-mon[31085]: osdmap e29: 4 total, 4 up, 4 in 2024-04-21T23:11:45.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:45 smithi111 ceph-mon[31085]: pgmap v106: 1 pgs: 1 remapped+peering; 577 KiB data, 396 MiB used, 357 GiB / 358 GiB avail 2024-04-21T23:11:45.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:45 smithi111 ceph-mon[31085]: from='client.? 172.21.15.111:0/2237007560' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "46eabfd7-d88a-4efd-94eb-04354d3a9d7e"}]: dispatch 2024-04-21T23:11:45.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:45 smithi111 ceph-mon[31085]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "46eabfd7-d88a-4efd-94eb-04354d3a9d7e"}]: dispatch 2024-04-21T23:11:45.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:45 smithi111 ceph-mon[31085]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "46eabfd7-d88a-4efd-94eb-04354d3a9d7e"}]': finished 2024-04-21T23:11:45.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:45 smithi111 ceph-mon[31085]: osdmap e30: 5 total, 4 up, 5 in 2024-04-21T23:11:45.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:45 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T23:11:46.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:45 smithi040 ceph-mon[27984]: pgmap v106: 1 pgs: 1 remapped+peering; 577 KiB data, 396 MiB used, 357 GiB / 358 GiB avail 2024-04-21T23:11:46.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:45 smithi040 ceph-mon[27984]: from='client.? 172.21.15.111:0/2237007560' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "46eabfd7-d88a-4efd-94eb-04354d3a9d7e"}]: dispatch 2024-04-21T23:11:46.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:45 smithi040 ceph-mon[27984]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "46eabfd7-d88a-4efd-94eb-04354d3a9d7e"}]: dispatch 2024-04-21T23:11:46.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:45 smithi040 ceph-mon[27984]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "46eabfd7-d88a-4efd-94eb-04354d3a9d7e"}]': finished 2024-04-21T23:11:46.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:45 smithi040 ceph-mon[27984]: osdmap e30: 5 total, 4 up, 5 in 2024-04-21T23:11:46.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:45 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T23:11:46.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:45 smithi174 ceph-mon[31409]: pgmap v106: 1 pgs: 1 remapped+peering; 577 KiB data, 396 MiB used, 357 GiB / 358 GiB avail 2024-04-21T23:11:46.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:45 smithi174 ceph-mon[31409]: from='client.? 172.21.15.111:0/2237007560' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "46eabfd7-d88a-4efd-94eb-04354d3a9d7e"}]: dispatch 2024-04-21T23:11:46.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:45 smithi174 ceph-mon[31409]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "46eabfd7-d88a-4efd-94eb-04354d3a9d7e"}]: dispatch 2024-04-21T23:11:46.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:45 smithi174 ceph-mon[31409]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "46eabfd7-d88a-4efd-94eb-04354d3a9d7e"}]': finished 2024-04-21T23:11:46.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:45 smithi174 ceph-mon[31409]: osdmap e30: 5 total, 4 up, 5 in 2024-04-21T23:11:46.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:45 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T23:11:47.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:46 smithi040 ceph-mon[27984]: from='client.? 172.21.15.111:0/968554733' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:11:47.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:46 smithi174 ceph-mon[31409]: from='client.? 172.21.15.111:0/968554733' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:11:47.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:46 smithi111 ceph-mon[31085]: from='client.? 172.21.15.111:0/968554733' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:11:47.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:47 smithi174 ceph-mon[31409]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 113 MiB used, 357 GiB / 358 GiB avail; 125 KiB/s, 0 objects/s recovering 2024-04-21T23:11:48.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:47 smithi040 ceph-mon[27984]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 113 MiB used, 357 GiB / 358 GiB avail; 125 KiB/s, 0 objects/s recovering 2024-04-21T23:11:48.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:47 smithi111 ceph-mon[31085]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 113 MiB used, 357 GiB / 358 GiB avail; 125 KiB/s, 0 objects/s recovering 2024-04-21T23:11:50.121 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:49 smithi111 ceph-mon[31085]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-21T23:11:50.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:49 smithi040 ceph-mon[27984]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-21T23:11:50.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:49 smithi174 ceph-mon[31409]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-21T23:11:51.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:51 smithi111 ceph-mon[31085]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-04-21T23:11:52.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:51 smithi040 ceph-mon[27984]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-04-21T23:11:52.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:51 smithi174 ceph-mon[31409]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-04-21T23:11:53.037 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:52 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-21T23:11:53.037 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:52 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:11:53.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:52 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-21T23:11:53.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:52 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:11:53.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:52 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-21T23:11:53.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:52 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:11:53.903 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:53 smithi111 ceph-mon[31085]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-04-21T23:11:53.903 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:53 smithi111 ceph-mon[31085]: Deploying daemon osd.4 on smithi111 2024-04-21T23:11:54.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:53 smithi040 ceph-mon[27984]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-04-21T23:11:54.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:53 smithi040 ceph-mon[27984]: Deploying daemon osd.4 on smithi111 2024-04-21T23:11:54.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:53 smithi174 ceph-mon[31409]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-04-21T23:11:54.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:53 smithi174 ceph-mon[31409]: Deploying daemon osd.4 on smithi111 2024-04-21T23:11:55.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:55 smithi111 ceph-mon[31085]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-21T23:11:56.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:55 smithi040 ceph-mon[27984]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-21T23:11:56.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:55 smithi174 ceph-mon[31409]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-21T23:11:58.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:57 smithi040 ceph-mon[27984]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-21T23:11:58.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:57 smithi174 ceph-mon[31409]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-21T23:11:58.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:57 smithi111 ceph-mon[31085]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-21T23:11:59.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:58 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:59.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:58 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:59.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:58 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:11:59.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:58 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:59.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:58 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:59.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:58 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:11:59.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:58 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:59.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:58 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:11:59.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:58 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:12:00.142 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:11:59 smithi111 ceph-mon[31085]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-21T23:12:00.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:11:59 smithi040 ceph-mon[27984]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-21T23:12:00.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:11:59 smithi174 ceph-mon[31409]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-21T23:12:01.398 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:12:01.398+0000 7ff2017fa700 1 -- 172.21.15.111:0/1200882234 <== mgr.14148 v2:172.21.15.40:6800/2500760788 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7ff204062670 con 0x7ff1f00a2c20 2024-04-21T23:12:01.399 INFO:teuthology.orchestra.run.smithi111.stdout:Created osd(s) 4 on host 'smithi111' 2024-04-21T23:12:01.400 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:12:01.400+0000 7ff20a37f700 1 -- 172.21.15.111:0/1200882234 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7ff1f00a2c20 msgr2=0x7ff1f00a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:01.401 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:12:01.400+0000 7ff20a37f700 1 --2- 172.21.15.111:0/1200882234 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7ff1f00a2c20 0x7ff1f00a50e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff1f4005950 tx=0x7ff1f40058e0 comp rx=0 tx=0).stop 2024-04-21T23:12:01.401 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:12:01.400+0000 7ff20a37f700 1 -- 172.21.15.111:0/1200882234 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff204108070 msgr2=0x7ff204101300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:01.401 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:12:01.400+0000 7ff20a37f700 1 --2- 172.21.15.111:0/1200882234 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff204108070 0x7ff204101300 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff1ec00d980 tx=0x7ff1ec00dc90 comp rx=0 tx=0).stop 2024-04-21T23:12:01.401 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:12:01.400+0000 7ff20a37f700 1 -- 172.21.15.111:0/1200882234 shutdown_connections 2024-04-21T23:12:01.401 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:12:01.400+0000 7ff20a37f700 1 --2- 172.21.15.111:0/1200882234 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff204102e10 0x7ff204102720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:01.401 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:12:01.400+0000 7ff20a37f700 1 --2- 172.21.15.111:0/1200882234 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7ff1f00a2c20 0x7ff1f00a50e0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:01.401 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:12:01.400+0000 7ff20a37f700 1 --2- 172.21.15.111:0/1200882234 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff204108070 0x7ff204101300 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:01.401 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:12:01.400+0000 7ff20a37f700 1 --2- 172.21.15.111:0/1200882234 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff204105740 0x7ff204100dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:01.401 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:12:01.401+0000 7ff20a37f700 1 -- 172.21.15.111:0/1200882234 >> 172.21.15.111:0/1200882234 conn(0x7ff2040fc780 msgr2=0x7ff2041036d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:01.401 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:12:01.401+0000 7ff20a37f700 1 -- 172.21.15.111:0/1200882234 shutdown_connections 2024-04-21T23:12:01.401 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:12:01.401+0000 7ff20a37f700 1 -- 172.21.15.111:0/1200882234 wait complete. 2024-04-21T23:12:02.015 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:01 smithi111 ceph-mon[31085]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-21T23:12:02.015 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:01 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:02.015 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:01 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:02.015 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:01 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:02.015 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:01 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:02.015 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:01 smithi111 ceph-mon[31085]: from='osd.4 [v2:172.21.15.111:6808/1810866070,v1:172.21.15.111:6809/1810866070]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-21T23:12:02.015 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:01 smithi111 ceph-mon[31085]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-21T23:12:02.043 DEBUG:teuthology.orchestra.run.smithi111:osd.4> sudo journalctl -f -n 0 -u ceph-d814e4dc-0033-11ef-bc93-c7b262605968@osd.4.service 2024-04-21T23:12:02.046 INFO:tasks.cephadm:Deploying osd.5 on smithi174 with /dev/vg_nvme/lv_3... 2024-04-21T23:12:02.046 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-21T23:12:02.245 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:01 smithi174 ceph-mon[31409]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-21T23:12:02.246 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:01 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:02.246 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:01 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:02.246 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:01 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:02.246 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:01 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:02.246 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:01 smithi174 ceph-mon[31409]: from='osd.4 [v2:172.21.15.111:6808/1810866070,v1:172.21.15.111:6809/1810866070]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-21T23:12:02.246 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:01 smithi174 ceph-mon[31409]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-21T23:12:02.289 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.c/config 2024-04-21T23:12:02.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:01 smithi040 ceph-mon[27984]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-21T23:12:02.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:01 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:02.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:01 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:02.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:01 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:02.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:01 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:02.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:01 smithi040 ceph-mon[27984]: from='osd.4 [v2:172.21.15.111:6808/1810866070,v1:172.21.15.111:6809/1810866070]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-21T23:12:02.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:01 smithi040 ceph-mon[27984]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-21T23:12:03.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:03 smithi174 ceph-mon[31409]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-21T23:12:03.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:03 smithi174 ceph-mon[31409]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-21T23:12:03.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:03 smithi174 ceph-mon[31409]: from='osd.4 [v2:172.21.15.111:6808/1810866070,v1:172.21.15.111:6809/1810866070]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-21T23:12:03.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:03 smithi174 ceph-mon[31409]: osdmap e31: 5 total, 4 up, 5 in 2024-04-21T23:12:03.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:03 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T23:12:03.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:03 smithi174 ceph-mon[31409]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-21T23:12:03.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:03 smithi111 ceph-mon[31085]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-21T23:12:03.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:03 smithi111 ceph-mon[31085]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-21T23:12:03.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:03 smithi111 ceph-mon[31085]: from='osd.4 [v2:172.21.15.111:6808/1810866070,v1:172.21.15.111:6809/1810866070]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-21T23:12:03.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:03 smithi111 ceph-mon[31085]: osdmap e31: 5 total, 4 up, 5 in 2024-04-21T23:12:03.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:03 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T23:12:03.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:03 smithi111 ceph-mon[31085]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-21T23:12:03.945 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 21 23:12:03 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-4[42582]: 2024-04-21T23:12:03.701+0000 7fa4a64e6700 -1 osd.4 0 waiting for initial osdmap 2024-04-21T23:12:03.946 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 21 23:12:03 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-4[42582]: 2024-04-21T23:12:03.706+0000 7fa4a22d9700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T23:12:04.181 INFO:teuthology.orchestra.run.smithi174.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-21T23:12:04.181 INFO:teuthology.orchestra.run.smithi174.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-21T23:12:04.181 INFO:teuthology.orchestra.run.smithi174.stderr: stderr: 10+0 records in 2024-04-21T23:12:04.181 INFO:teuthology.orchestra.run.smithi174.stderr:10+0 records out 2024-04-21T23:12:04.182 INFO:teuthology.orchestra.run.smithi174.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0139711 s, 751 MB/s 2024-04-21T23:12:04.182 INFO:teuthology.orchestra.run.smithi174.stderr:--> Zapping successful for: 2024-04-21T23:12:04.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:03 smithi040 ceph-mon[27984]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-21T23:12:04.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:03 smithi040 ceph-mon[27984]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-21T23:12:04.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:03 smithi040 ceph-mon[27984]: from='osd.4 [v2:172.21.15.111:6808/1810866070,v1:172.21.15.111:6809/1810866070]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-21T23:12:04.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:03 smithi040 ceph-mon[27984]: osdmap e31: 5 total, 4 up, 5 in 2024-04-21T23:12:04.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:03 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T23:12:04.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:03 smithi040 ceph-mon[27984]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-21T23:12:04.755 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:04 smithi174 ceph-mon[31409]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-04-21T23:12:04.756 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:04 smithi174 ceph-mon[31409]: osdmap e32: 5 total, 4 up, 5 in 2024-04-21T23:12:04.756 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:04 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T23:12:04.756 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:04 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T23:12:04.756 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:04 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T23:12:04.783 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi174:vg_nvme/lv_3 2024-04-21T23:12:05.029 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.c/config 2024-04-21T23:12:05.105 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:04 smithi111 ceph-mon[31085]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-04-21T23:12:05.106 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:04 smithi111 ceph-mon[31085]: osdmap e32: 5 total, 4 up, 5 in 2024-04-21T23:12:05.106 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:04 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T23:12:05.106 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:04 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T23:12:05.106 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:04 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T23:12:05.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:04 smithi040 ceph-mon[27984]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-04-21T23:12:05.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:04 smithi040 ceph-mon[27984]: osdmap e32: 5 total, 4 up, 5 in 2024-04-21T23:12:05.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:04 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T23:12:05.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:04 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T23:12:05.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:04 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T23:12:06.139 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:05 smithi174 ceph-mon[31409]: purged_snaps scrub starts 2024-04-21T23:12:06.139 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:05 smithi174 ceph-mon[31409]: purged_snaps scrub ok 2024-04-21T23:12:06.139 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:05 smithi174 ceph-mon[31409]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-21T23:12:06.139 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:05 smithi174 ceph-mon[31409]: osd.4 [v2:172.21.15.111:6808/1810866070,v1:172.21.15.111:6809/1810866070] boot 2024-04-21T23:12:06.139 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:05 smithi174 ceph-mon[31409]: osdmap e33: 5 total, 5 up, 5 in 2024-04-21T23:12:06.139 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:05 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T23:12:06.139 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:05 smithi174 ceph-mon[31409]: Detected new or changed devices on smithi111 2024-04-21T23:12:06.140 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:05 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:06.140 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:05 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:06.140 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:05 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:12:06.140 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:05 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:12:06.140 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:05 smithi174 ceph-mon[31409]: Adjusting osd_memory_target on smithi111 to 8544M 2024-04-21T23:12:06.140 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:05 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:06.140 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:05 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:12:06.140 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:05 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:12:06.140 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:05 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:06.140 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:05 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:12:06.140 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:05 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:12:06.140 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:05 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:12:06.140 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:05 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:06.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:05 smithi040 ceph-mon[27984]: purged_snaps scrub starts 2024-04-21T23:12:06.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:05 smithi040 ceph-mon[27984]: purged_snaps scrub ok 2024-04-21T23:12:06.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:05 smithi040 ceph-mon[27984]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-21T23:12:06.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:05 smithi040 ceph-mon[27984]: osd.4 [v2:172.21.15.111:6808/1810866070,v1:172.21.15.111:6809/1810866070] boot 2024-04-21T23:12:06.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:05 smithi040 ceph-mon[27984]: osdmap e33: 5 total, 5 up, 5 in 2024-04-21T23:12:06.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:05 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T23:12:06.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:05 smithi040 ceph-mon[27984]: Detected new or changed devices on smithi111 2024-04-21T23:12:06.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:05 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:06.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:05 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:06.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:05 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:12:06.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:05 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:12:06.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:05 smithi040 ceph-mon[27984]: Adjusting osd_memory_target on smithi111 to 8544M 2024-04-21T23:12:06.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:05 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:06.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:05 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:12:06.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:05 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:12:06.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:05 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:06.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:05 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:12:06.184 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:05 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:12:06.184 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:05 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:12:06.184 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:05 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:06.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:05 smithi111 ceph-mon[31085]: purged_snaps scrub starts 2024-04-21T23:12:06.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:05 smithi111 ceph-mon[31085]: purged_snaps scrub ok 2024-04-21T23:12:06.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:05 smithi111 ceph-mon[31085]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-21T23:12:06.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:05 smithi111 ceph-mon[31085]: osd.4 [v2:172.21.15.111:6808/1810866070,v1:172.21.15.111:6809/1810866070] boot 2024-04-21T23:12:06.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:05 smithi111 ceph-mon[31085]: osdmap e33: 5 total, 5 up, 5 in 2024-04-21T23:12:06.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:05 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T23:12:06.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:05 smithi111 ceph-mon[31085]: Detected new or changed devices on smithi111 2024-04-21T23:12:06.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:05 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:06.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:05 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:06.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:05 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:12:06.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:05 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:12:06.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:05 smithi111 ceph-mon[31085]: Adjusting osd_memory_target on smithi111 to 8544M 2024-04-21T23:12:06.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:05 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:06.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:05 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:12:06.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:05 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:12:06.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:05 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:06.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:05 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:12:06.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:05 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:12:06.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:05 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:12:06.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:05 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:06.762 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:12:06.762+0000 7f1ab8512700 1 -- 172.21.15.174:0/3922315729 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1ab0102280 msgr2=0x7f1ab01026a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:06.762 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:12:06.762+0000 7f1ab8512700 1 --2- 172.21.15.174:0/3922315729 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1ab0102280 0x7f1ab01026a0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f1aac009990 tx=0x7f1aac009ca0 comp rx=0 tx=0).stop 2024-04-21T23:12:06.762 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:06 smithi174 ceph-mon[31409]: osdmap e34: 5 total, 5 up, 5 in 2024-04-21T23:12:06.764 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:12:06.762+0000 7f1ab8512700 1 -- 172.21.15.174:0/3922315729 shutdown_connections 2024-04-21T23:12:06.764 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:12:06.762+0000 7f1ab8512700 1 --2- 172.21.15.174:0/3922315729 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1ab0102280 0x7f1ab01026a0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:06.764 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:12:06.762+0000 7f1ab8512700 1 --2- 172.21.15.174:0/3922315729 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ab0103e20 0x7f1ab010a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:06.764 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:12:06.762+0000 7f1ab8512700 1 --2- 172.21.15.174:0/3922315729 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ab0103480 0x7f1ab01038e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:06.764 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:12:06.762+0000 7f1ab8512700 1 -- 172.21.15.174:0/3922315729 >> 172.21.15.174:0/3922315729 conn(0x7f1ab00fd820 msgr2=0x7f1ab00ffc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:06.765 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:12:06.762+0000 7f1ab8512700 1 -- 172.21.15.174:0/3922315729 shutdown_connections 2024-04-21T23:12:06.765 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:12:06.762+0000 7f1ab8512700 1 -- 172.21.15.174:0/3922315729 wait complete. 2024-04-21T23:12:06.765 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:12:06.762+0000 7f1ab8512700 1 Processor -- start 2024-04-21T23:12:06.765 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:12:06.762+0000 7f1ab8512700 1 -- start start 2024-04-21T23:12:06.765 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:12:06.763+0000 7f1ab8512700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ab0102280 0x7f1ab0075fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:06.765 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:12:06.763+0000 7f1ab8512700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ab0103480 0x7f1ab0074620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:06.765 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:12:06.763+0000 7f1ab8512700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1ab0103e20 0x7f1ab0074b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:06.765 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:12:06.763+0000 7f1ab8512700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1ab010de60 con 0x7f1ab0103480 2024-04-21T23:12:06.765 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:12:06.763+0000 7f1ab8512700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f1ab010dce0 con 0x7f1ab0102280 2024-04-21T23:12:06.765 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:12:06.763+0000 7f1ab8512700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1ab010dfe0 con 0x7f1ab0103e20 2024-04-21T23:12:06.765 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:12:06.763+0000 7f1ab6aaf700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1ab0103e20 0x7f1ab0074b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:06.765 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:12:06.763+0000 7f1ab6aaf700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1ab0103e20 0x7f1ab0074b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.174:40516/0 (socket says 172.21.15.174:40516) 2024-04-21T23:12:06.765 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:12:06.763+0000 7f1ab6aaf700 1 -- 172.21.15.174:0/1446014118 learned_addr learned my addr 172.21.15.174:0/1446014118 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-04-21T23:12:06.766 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:12:06.763+0000 7f1ab5aad700 1 --2- 172.21.15.174:0/1446014118 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ab0103480 0x7f1ab0074620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:06.766 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:12:06.763+0000 7f1ab62ae700 1 --2- 172.21.15.174:0/1446014118 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ab0102280 0x7f1ab0075fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:06.766 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:12:06.763+0000 7f1ab6aaf700 1 -- 172.21.15.174:0/1446014118 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ab0102280 msgr2=0x7f1ab0075fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:06.766 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:12:06.763+0000 7f1ab6aaf700 1 --2- 172.21.15.174:0/1446014118 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ab0102280 0x7f1ab0075fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:06.766 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:12:06.763+0000 7f1ab6aaf700 1 -- 172.21.15.174:0/1446014118 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ab0103480 msgr2=0x7f1ab0074620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:06.766 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:12:06.763+0000 7f1ab6aaf700 1 --2- 172.21.15.174:0/1446014118 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ab0103480 0x7f1ab0074620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:06.766 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:12:06.763+0000 7f1ab6aaf700 1 -- 172.21.15.174:0/1446014118 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ab0075410 con 0x7f1ab0103e20 2024-04-21T23:12:06.766 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:12:06.763+0000 7f1ab5aad700 1 --2- 172.21.15.174:0/1446014118 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ab0103480 0x7f1ab0074620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:12:06.766 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:12:06.763+0000 7f1ab6aaf700 1 --2- 172.21.15.174:0/1446014118 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1ab0103e20 0x7f1ab0074b60 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f1aa8007ee0 tx=0x7f1aa800d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:06.766 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:12:06.763+0000 7f1aa77fe700 1 -- 172.21.15.174:0/1446014118 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aa800dbd0 con 0x7f1ab0103e20 2024-04-21T23:12:06.766 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:12:06.763+0000 7f1aa77fe700 1 -- 172.21.15.174:0/1446014118 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1aa800dd70 con 0x7f1ab0103e20 2024-04-21T23:12:06.766 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:12:06.763+0000 7f1ab8512700 1 -- 172.21.15.174:0/1446014118 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ab01a2690 con 0x7f1ab0103e20 2024-04-21T23:12:06.766 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:12:06.763+0000 7f1aa77fe700 1 -- 172.21.15.174:0/1446014118 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aa80133f0 con 0x7f1ab0103e20 2024-04-21T23:12:06.766 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:12:06.764+0000 7f1ab8512700 1 -- 172.21.15.174:0/1446014118 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ab01a2bd0 con 0x7f1ab0103e20 2024-04-21T23:12:06.767 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:12:06.765+0000 7f1ab8512700 1 -- 172.21.15.174:0/1446014118 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ab0068f20 con 0x7f1ab0103e20 2024-04-21T23:12:06.767 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:12:06.765+0000 7f1aa77fe700 1 -- 172.21.15.174:0/1446014118 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f1aa801a020 con 0x7f1ab0103e20 2024-04-21T23:12:06.767 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:12:06.766+0000 7f1aa77fe700 1 --2- 172.21.15.174:0/1446014118 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f1a9c0a2ca0 0x7f1a9c0a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:06.768 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:12:06.766+0000 7f1aa77fe700 1 -- 172.21.15.174:0/1446014118 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 4101+0+0 (secure 0 0 0) 0x7f1aa80cd750 con 0x7f1ab0103e20 2024-04-21T23:12:06.768 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:12:06.766+0000 7f1ab62ae700 1 --2- 172.21.15.174:0/1446014118 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f1a9c0a2ca0 0x7f1a9c0a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:06.768 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:12:06.767+0000 7f1ab62ae700 1 --2- 172.21.15.174:0/1446014118 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f1a9c0a2ca0 0x7f1a9c0a5160 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1aac006010 tx=0x7f1aac005e80 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:06.770 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:12:06.771+0000 7f1aa77fe700 1 -- 172.21.15.174:0/1446014118 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f1aa8010070 con 0x7f1ab0103e20 2024-04-21T23:12:06.975 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:12:06.975+0000 7f1ab8512700 1 -- 172.21.15.174:0/1446014118 --> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f1ab0062670 con 0x7f1a9c0a2ca0 2024-04-21T23:12:07.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:06 smithi040 ceph-mon[27984]: osdmap e34: 5 total, 5 up, 5 in 2024-04-21T23:12:07.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:06 smithi111 ceph-mon[31085]: osdmap e34: 5 total, 5 up, 5 in 2024-04-21T23:12:07.894 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:07 smithi174 ceph-mon[31409]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:12:07.894 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:07 smithi174 ceph-mon[31409]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:12:07.894 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:07 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:12:07.894 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:07 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:12:07.894 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:07 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:12:08.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:07 smithi040 ceph-mon[27984]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:12:08.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:07 smithi040 ceph-mon[27984]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:12:08.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:07 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:12:08.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:07 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:12:08.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:07 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:12:08.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:07 smithi111 ceph-mon[31085]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:12:08.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:07 smithi111 ceph-mon[31085]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:12:08.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:07 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:12:08.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:07 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:12:08.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:07 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:12:09.984 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:09 smithi040 ceph-mon[27984]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:12:10.124 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:09 smithi174 ceph-mon[31409]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:12:10.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:09 smithi111 ceph-mon[31085]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:12:11.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:10 smithi040 ceph-mon[27984]: from='client.? 172.21.15.174:0/820496865' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f73b582-6c65-41db-aab0-9b360f83d5c6"}]: dispatch 2024-04-21T23:12:11.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:10 smithi040 ceph-mon[27984]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f73b582-6c65-41db-aab0-9b360f83d5c6"}]: dispatch 2024-04-21T23:12:11.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:10 smithi040 ceph-mon[27984]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6f73b582-6c65-41db-aab0-9b360f83d5c6"}]': finished 2024-04-21T23:12:11.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:10 smithi040 ceph-mon[27984]: osdmap e35: 6 total, 5 up, 6 in 2024-04-21T23:12:11.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:10 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T23:12:11.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:10 smithi040 ceph-mon[27984]: from='client.? 172.21.15.174:0/2836962506' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:12:11.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:10 smithi174 ceph-mon[31409]: from='client.? 172.21.15.174:0/820496865' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f73b582-6c65-41db-aab0-9b360f83d5c6"}]: dispatch 2024-04-21T23:12:11.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:10 smithi174 ceph-mon[31409]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f73b582-6c65-41db-aab0-9b360f83d5c6"}]: dispatch 2024-04-21T23:12:11.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:10 smithi174 ceph-mon[31409]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6f73b582-6c65-41db-aab0-9b360f83d5c6"}]': finished 2024-04-21T23:12:11.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:10 smithi174 ceph-mon[31409]: osdmap e35: 6 total, 5 up, 6 in 2024-04-21T23:12:11.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:10 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T23:12:11.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:10 smithi174 ceph-mon[31409]: from='client.? 172.21.15.174:0/2836962506' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:12:11.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:10 smithi111 ceph-mon[31085]: from='client.? 172.21.15.174:0/820496865' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f73b582-6c65-41db-aab0-9b360f83d5c6"}]: dispatch 2024-04-21T23:12:11.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:10 smithi111 ceph-mon[31085]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f73b582-6c65-41db-aab0-9b360f83d5c6"}]: dispatch 2024-04-21T23:12:11.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:10 smithi111 ceph-mon[31085]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6f73b582-6c65-41db-aab0-9b360f83d5c6"}]': finished 2024-04-21T23:12:11.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:10 smithi111 ceph-mon[31085]: osdmap e35: 6 total, 5 up, 6 in 2024-04-21T23:12:11.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:10 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T23:12:11.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:10 smithi111 ceph-mon[31085]: from='client.? 172.21.15.174:0/2836962506' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:12:12.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:11 smithi040 ceph-mon[27984]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:12:12.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:11 smithi174 ceph-mon[31409]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:12:12.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:11 smithi111 ceph-mon[31085]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:12:13.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:13 smithi174 ceph-mon[31409]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:12:14.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:13 smithi040 ceph-mon[27984]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:12:14.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:13 smithi111 ceph-mon[31085]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:12:15.852 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:15 smithi174 ceph-mon[31409]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:12:16.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:15 smithi040 ceph-mon[27984]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:12:16.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:15 smithi111 ceph-mon[31085]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:12:17.846 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:17 smithi174 ceph-mon[31409]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:12:17.846 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:17 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-21T23:12:17.846 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:17 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:12:17.846 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:17 smithi174 ceph-mon[31409]: Deploying daemon osd.5 on smithi174 2024-04-21T23:12:18.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:17 smithi040 ceph-mon[27984]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:12:18.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:17 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-21T23:12:18.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:17 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:12:18.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:17 smithi040 ceph-mon[27984]: Deploying daemon osd.5 on smithi174 2024-04-21T23:12:18.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:17 smithi111 ceph-mon[31085]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:12:18.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:17 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-21T23:12:18.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:17 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:12:18.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:17 smithi111 ceph-mon[31085]: Deploying daemon osd.5 on smithi174 2024-04-21T23:12:20.071 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:19 smithi174 ceph-mon[31409]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:12:20.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:19 smithi040 ceph-mon[27984]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:12:20.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:19 smithi111 ceph-mon[31085]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:12:21.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:21 smithi174 ceph-mon[31409]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:12:22.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:21 smithi040 ceph-mon[27984]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:12:22.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:21 smithi111 ceph-mon[31085]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:12:23.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:23 smithi040 ceph-mon[27984]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:12:23.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:23 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:23.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:23 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:23.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:23 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:12:23.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:23 smithi174 ceph-mon[31409]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:12:23.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:23 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:23.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:23 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:23.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:23 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:12:23.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:23 smithi111 ceph-mon[31085]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:12:23.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:23 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:23.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:23 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:23.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:23 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:12:25.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:25 smithi040 ceph-mon[27984]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:12:25.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:25 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:25.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:25 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:25.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:25 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:12:25.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:25 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:12:25.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:25 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:25.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:25 smithi174 ceph-mon[31409]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:12:25.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:25 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:25.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:25 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:25.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:25 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:12:25.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:25 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:12:25.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:25 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:25.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:25 smithi111 ceph-mon[31085]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:12:25.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:25 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:25.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:25 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:25.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:25 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:12:25.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:25 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:12:25.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:25 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:26.391 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:12:26.392+0000 7f1aa77fe700 1 -- 172.21.15.174:0/1446014118 <== mgr.14148 v2:172.21.15.40:6800/2500760788 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f1ab0062670 con 0x7f1a9c0a2ca0 2024-04-21T23:12:26.392 INFO:teuthology.orchestra.run.smithi174.stdout:Created osd(s) 5 on host 'smithi174' 2024-04-21T23:12:26.394 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:12:26.394+0000 7f1ab8512700 1 -- 172.21.15.174:0/1446014118 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f1a9c0a2ca0 msgr2=0x7f1a9c0a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:26.394 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:12:26.394+0000 7f1ab8512700 1 --2- 172.21.15.174:0/1446014118 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f1a9c0a2ca0 0x7f1a9c0a5160 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1aac006010 tx=0x7f1aac005e80 comp rx=0 tx=0).stop 2024-04-21T23:12:26.394 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:12:26.394+0000 7f1ab8512700 1 -- 172.21.15.174:0/1446014118 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1ab0103e20 msgr2=0x7f1ab0074b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:26.394 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:12:26.394+0000 7f1ab8512700 1 --2- 172.21.15.174:0/1446014118 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1ab0103e20 0x7f1ab0074b60 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f1aa8007ee0 tx=0x7f1aa800d3b0 comp rx=0 tx=0).stop 2024-04-21T23:12:26.395 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:12:26.394+0000 7f1ab8512700 1 -- 172.21.15.174:0/1446014118 shutdown_connections 2024-04-21T23:12:26.395 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:12:26.394+0000 7f1ab8512700 1 --2- 172.21.15.174:0/1446014118 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1ab0103e20 0x7f1ab0074b60 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:26.395 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:12:26.394+0000 7f1ab8512700 1 --2- 172.21.15.174:0/1446014118 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f1a9c0a2ca0 0x7f1a9c0a5160 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:26.395 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:12:26.394+0000 7f1ab8512700 1 --2- 172.21.15.174:0/1446014118 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ab0102280 0x7f1ab0075fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:26.395 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:12:26.394+0000 7f1ab8512700 1 --2- 172.21.15.174:0/1446014118 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ab0103480 0x7f1ab0074620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:26.395 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:12:26.394+0000 7f1ab8512700 1 -- 172.21.15.174:0/1446014118 >> 172.21.15.174:0/1446014118 conn(0x7f1ab00fd820 msgr2=0x7f1ab01067d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:26.395 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:12:26.395+0000 7f1ab8512700 1 -- 172.21.15.174:0/1446014118 shutdown_connections 2024-04-21T23:12:26.395 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:12:26.395+0000 7f1ab8512700 1 -- 172.21.15.174:0/1446014118 wait complete. 2024-04-21T23:12:27.064 DEBUG:teuthology.orchestra.run.smithi174:osd.5> sudo journalctl -f -n 0 -u ceph-d814e4dc-0033-11ef-bc93-c7b262605968@osd.5.service 2024-04-21T23:12:27.066 INFO:tasks.cephadm:Deploying osd.6 on smithi040 with /dev/vg_nvme/lv_2... 2024-04-21T23:12:27.066 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-21T23:12:27.321 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:12:27.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:27 smithi040 ceph-mon[27984]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:12:27.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:27 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:27.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:27 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:27.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:27 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:12:27.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:27 smithi040 ceph-mon[27984]: from='osd.5 [v2:172.21.15.174:6808/3752986711,v1:172.21.15.174:6809/3752986711]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-21T23:12:27.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:27 smithi040 ceph-mon[27984]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-21T23:12:27.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:27 smithi174 ceph-mon[31409]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:12:27.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:27 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:27.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:27 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:27.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:27 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:12:27.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:27 smithi174 ceph-mon[31409]: from='osd.5 [v2:172.21.15.174:6808/3752986711,v1:172.21.15.174:6809/3752986711]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-21T23:12:27.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:27 smithi174 ceph-mon[31409]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-21T23:12:27.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:27 smithi111 ceph-mon[31085]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:12:27.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:27 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:27.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:27 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:27.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:27 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:12:27.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:27 smithi111 ceph-mon[31085]: from='osd.5 [v2:172.21.15.174:6808/3752986711,v1:172.21.15.174:6809/3752986711]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-21T23:12:27.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:27 smithi111 ceph-mon[31085]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-21T23:12:28.550 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:28 smithi174 ceph-mon[31409]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-21T23:12:28.551 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:28 smithi174 ceph-mon[31409]: osdmap e36: 6 total, 5 up, 6 in 2024-04-21T23:12:28.551 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:28 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T23:12:28.551 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:28 smithi174 ceph-mon[31409]: from='osd.5 [v2:172.21.15.174:6808/3752986711,v1:172.21.15.174:6809/3752986711]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-04-21T23:12:28.551 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:28 smithi174 ceph-mon[31409]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-04-21T23:12:28.649 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:28 smithi040 ceph-mon[27984]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-21T23:12:28.649 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:28 smithi040 ceph-mon[27984]: osdmap e36: 6 total, 5 up, 6 in 2024-04-21T23:12:28.649 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:28 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T23:12:28.649 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:28 smithi040 ceph-mon[27984]: from='osd.5 [v2:172.21.15.174:6808/3752986711,v1:172.21.15.174:6809/3752986711]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-04-21T23:12:28.649 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:28 smithi040 ceph-mon[27984]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-04-21T23:12:28.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:28 smithi111 ceph-mon[31085]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-21T23:12:28.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:28 smithi111 ceph-mon[31085]: osdmap e36: 6 total, 5 up, 6 in 2024-04-21T23:12:28.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:28 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T23:12:28.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:28 smithi111 ceph-mon[31085]: from='osd.5 [v2:172.21.15.174:6808/3752986711,v1:172.21.15.174:6809/3752986711]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-04-21T23:12:28.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:28 smithi111 ceph-mon[31085]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-04-21T23:12:29.334 INFO:teuthology.orchestra.run.smithi040.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-21T23:12:29.334 INFO:teuthology.orchestra.run.smithi040.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-21T23:12:29.334 INFO:teuthology.orchestra.run.smithi040.stderr: stderr: 10+0 records in 2024-04-21T23:12:29.334 INFO:teuthology.orchestra.run.smithi040.stderr:10+0 records out 2024-04-21T23:12:29.334 INFO:teuthology.orchestra.run.smithi040.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0110657 s, 948 MB/s 2024-04-21T23:12:29.334 INFO:teuthology.orchestra.run.smithi040.stderr:--> Zapping successful for: 2024-04-21T23:12:29.377 INFO:journalctl@ceph.osd.5.smithi174.stdout:Apr 21 23:12:29 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-5[43337]: 2024-04-21T23:12:29.043+0000 7f5a7a748700 -1 osd.5 0 waiting for initial osdmap 2024-04-21T23:12:29.377 INFO:journalctl@ceph.osd.5.smithi174.stdout:Apr 21 23:12:29 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-5[43337]: 2024-04-21T23:12:29.051+0000 7f5a71d32700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T23:12:29.649 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:29 smithi040 ceph-mon[27984]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:12:29.649 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:29 smithi040 ceph-mon[27984]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-04-21T23:12:29.649 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:29 smithi040 ceph-mon[27984]: osdmap e37: 6 total, 5 up, 6 in 2024-04-21T23:12:29.649 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:29 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T23:12:29.649 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:29 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T23:12:29.649 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:29 smithi040 ceph-mon[27984]: from='osd.5 ' entity='osd.5' 2024-04-21T23:12:29.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:29 smithi174 ceph-mon[31409]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:12:29.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:29 smithi174 ceph-mon[31409]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-04-21T23:12:29.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:29 smithi174 ceph-mon[31409]: osdmap e37: 6 total, 5 up, 6 in 2024-04-21T23:12:29.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:29 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T23:12:29.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:29 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T23:12:29.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:29 smithi174 ceph-mon[31409]: from='osd.5 ' entity='osd.5' 2024-04-21T23:12:29.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:29 smithi111 ceph-mon[31085]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:12:29.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:29 smithi111 ceph-mon[31085]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-04-21T23:12:29.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:29 smithi111 ceph-mon[31085]: osdmap e37: 6 total, 5 up, 6 in 2024-04-21T23:12:29.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:29 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T23:12:29.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:29 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T23:12:29.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:29 smithi111 ceph-mon[31085]: from='osd.5 ' entity='osd.5' 2024-04-21T23:12:30.000 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi040:vg_nvme/lv_2 2024-04-21T23:12:30.253 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:12:30.640 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:30 smithi040 ceph-mon[27984]: purged_snaps scrub starts 2024-04-21T23:12:30.640 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:30 smithi040 ceph-mon[27984]: purged_snaps scrub ok 2024-04-21T23:12:30.641 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:30 smithi040 ceph-mon[27984]: osd.5 [v2:172.21.15.174:6808/3752986711,v1:172.21.15.174:6809/3752986711] boot 2024-04-21T23:12:30.641 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:30 smithi040 ceph-mon[27984]: osdmap e38: 6 total, 6 up, 6 in 2024-04-21T23:12:30.641 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:30 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T23:12:30.641 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:30 smithi040 ceph-mon[27984]: Detected new or changed devices on smithi174 2024-04-21T23:12:30.641 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:30 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:30.641 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:30 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:30.641 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:30 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:12:30.641 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:30 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:12:30.641 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:30 smithi040 ceph-mon[27984]: Adjusting osd_memory_target on smithi174 to 8544M 2024-04-21T23:12:30.641 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:30 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:30.641 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:30 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:12:30.641 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:30 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:12:30.641 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:30 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:30.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:30 smithi174 ceph-mon[31409]: purged_snaps scrub starts 2024-04-21T23:12:30.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:30 smithi174 ceph-mon[31409]: purged_snaps scrub ok 2024-04-21T23:12:30.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:30 smithi174 ceph-mon[31409]: osd.5 [v2:172.21.15.174:6808/3752986711,v1:172.21.15.174:6809/3752986711] boot 2024-04-21T23:12:30.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:30 smithi174 ceph-mon[31409]: osdmap e38: 6 total, 6 up, 6 in 2024-04-21T23:12:30.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:30 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T23:12:30.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:30 smithi174 ceph-mon[31409]: Detected new or changed devices on smithi174 2024-04-21T23:12:30.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:30 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:30.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:30 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:30.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:30 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:12:30.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:30 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:12:30.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:30 smithi174 ceph-mon[31409]: Adjusting osd_memory_target on smithi174 to 8544M 2024-04-21T23:12:30.695 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:30 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:30.695 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:30 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:12:30.695 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:30 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:12:30.695 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:30 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:30.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:30 smithi111 ceph-mon[31085]: purged_snaps scrub starts 2024-04-21T23:12:30.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:30 smithi111 ceph-mon[31085]: purged_snaps scrub ok 2024-04-21T23:12:30.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:30 smithi111 ceph-mon[31085]: osd.5 [v2:172.21.15.174:6808/3752986711,v1:172.21.15.174:6809/3752986711] boot 2024-04-21T23:12:30.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:30 smithi111 ceph-mon[31085]: osdmap e38: 6 total, 6 up, 6 in 2024-04-21T23:12:30.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:30 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T23:12:30.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:30 smithi111 ceph-mon[31085]: Detected new or changed devices on smithi174 2024-04-21T23:12:30.696 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:30 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:30.696 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:30 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:30.696 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:30 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:12:30.696 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:30 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:12:30.696 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:30 smithi111 ceph-mon[31085]: Adjusting osd_memory_target on smithi174 to 8544M 2024-04-21T23:12:30.696 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:30 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:30.696 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:30 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:12:30.696 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:30 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:12:30.696 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:30 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:31.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:31 smithi040 ceph-mon[27984]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 536 GiB / 536 GiB avail 2024-04-21T23:12:31.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:31 smithi040 ceph-mon[27984]: osdmap e39: 6 total, 6 up, 6 in 2024-04-21T23:12:31.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:31 smithi174 ceph-mon[31409]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 536 GiB / 536 GiB avail 2024-04-21T23:12:31.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:31 smithi174 ceph-mon[31409]: osdmap e39: 6 total, 6 up, 6 in 2024-04-21T23:12:31.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:31 smithi111 ceph-mon[31085]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 536 GiB / 536 GiB avail 2024-04-21T23:12:31.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:31 smithi111 ceph-mon[31085]: osdmap e39: 6 total, 6 up, 6 in 2024-04-21T23:12:31.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:12:31.851+0000 7f1e4eeda700 1 -- 172.21.15.40:0/2414729278 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e48102d30 msgr2=0x7f1e481095e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:31.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:12:31.851+0000 7f1e4eeda700 1 --2- 172.21.15.40:0/2414729278 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e48102d30 0x7f1e481095e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1e4000b0d0 tx=0x7f1e4000b3e0 comp rx=0 tx=0).stop 2024-04-21T23:12:31.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:12:31.851+0000 7f1e4eeda700 1 -- 172.21.15.40:0/2414729278 shutdown_connections 2024-04-21T23:12:31.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:12:31.851+0000 7f1e4eeda700 1 --2- 172.21.15.40:0/2414729278 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1e48101190 0x7f1e481015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:31.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:12:31.851+0000 7f1e4eeda700 1 --2- 172.21.15.40:0/2414729278 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e48102d30 0x7f1e481095e0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:31.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:12:31.851+0000 7f1e4eeda700 1 --2- 172.21.15.40:0/2414729278 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e48102390 0x7f1e481027f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:31.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:12:31.851+0000 7f1e4eeda700 1 -- 172.21.15.40:0/2414729278 >> 172.21.15.40:0/2414729278 conn(0x7f1e480fc730 msgr2=0x7f1e480feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:31.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:12:31.851+0000 7f1e4eeda700 1 -- 172.21.15.40:0/2414729278 shutdown_connections 2024-04-21T23:12:31.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:12:31.851+0000 7f1e4eeda700 1 -- 172.21.15.40:0/2414729278 wait complete. 2024-04-21T23:12:31.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:12:31.851+0000 7f1e4eeda700 1 Processor -- start 2024-04-21T23:12:31.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:12:31.851+0000 7f1e4eeda700 1 -- start start 2024-04-21T23:12:31.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:12:31.852+0000 7f1e4eeda700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1e48101190 0x7f1e4806aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:31.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:12:31.852+0000 7f1e4eeda700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e48102390 0x7f1e4819f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:31.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:12:31.852+0000 7f1e4eeda700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e48102d30 0x7f1e481a1b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:31.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:12:31.852+0000 7f1e4eeda700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1e4810cde0 con 0x7f1e48102390 2024-04-21T23:12:31.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:12:31.852+0000 7f1e4eeda700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f1e4810cc60 con 0x7f1e48102d30 2024-04-21T23:12:31.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:12:31.852+0000 7f1e4eeda700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1e4810cf60 con 0x7f1e48101190 2024-04-21T23:12:31.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:12:31.852+0000 7f1e47fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e48102390 0x7f1e4819f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:31.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:12:31.852+0000 7f1e47fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e48102390 0x7f1e4819f620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:53114/0 (socket says 172.21.15.40:53114) 2024-04-21T23:12:31.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:12:31.852+0000 7f1e47fff700 1 -- 172.21.15.40:0/3876876755 learned_addr learned my addr 172.21.15.40:0/3876876755 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:12:31.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:12:31.852+0000 7f1e4d477700 1 --2- 172.21.15.40:0/3876876755 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e48102d30 0x7f1e481a1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:31.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:12:31.852+0000 7f1e4cc76700 1 --2- 172.21.15.40:0/3876876755 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1e48101190 0x7f1e4806aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:31.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:12:31.852+0000 7f1e47fff700 1 -- 172.21.15.40:0/3876876755 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1e48101190 msgr2=0x7f1e4806aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:31.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:12:31.852+0000 7f1e47fff700 1 --2- 172.21.15.40:0/3876876755 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1e48101190 0x7f1e4806aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:31.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:12:31.852+0000 7f1e47fff700 1 -- 172.21.15.40:0/3876876755 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e48102d30 msgr2=0x7f1e481a1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:31.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:12:31.852+0000 7f1e47fff700 1 --2- 172.21.15.40:0/3876876755 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e48102d30 0x7f1e481a1b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:31.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:12:31.852+0000 7f1e47fff700 1 -- 172.21.15.40:0/3876876755 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e481a2350 con 0x7f1e48102390 2024-04-21T23:12:31.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:12:31.852+0000 7f1e4d477700 1 --2- 172.21.15.40:0/3876876755 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e48102d30 0x7f1e481a1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:12:31.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:12:31.852+0000 7f1e47fff700 1 --2- 172.21.15.40:0/3876876755 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e48102390 0x7f1e4819f620 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1e3400e9a0 tx=0x7f1e3400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:31.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:12:31.852+0000 7f1e45ffb700 1 -- 172.21.15.40:0/3876876755 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e3400ca10 con 0x7f1e48102390 2024-04-21T23:12:31.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:12:31.852+0000 7f1e4eeda700 1 -- 172.21.15.40:0/3876876755 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e481a2640 con 0x7f1e48102390 2024-04-21T23:12:31.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:12:31.852+0000 7f1e45ffb700 1 -- 172.21.15.40:0/3876876755 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e3400cbb0 con 0x7f1e48102390 2024-04-21T23:12:31.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:12:31.852+0000 7f1e45ffb700 1 -- 172.21.15.40:0/3876876755 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e34018680 con 0x7f1e48102390 2024-04-21T23:12:31.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:12:31.853+0000 7f1e4eeda700 1 -- 172.21.15.40:0/3876876755 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e481a2bd0 con 0x7f1e48102390 2024-04-21T23:12:31.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:12:31.854+0000 7f1e4eeda700 1 -- 172.21.15.40:0/3876876755 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e48068f20 con 0x7f1e48102390 2024-04-21T23:12:31.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:12:31.855+0000 7f1e45ffb700 1 -- 172.21.15.40:0/3876876755 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f1e34010430 con 0x7f1e48102390 2024-04-21T23:12:31.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:12:31.856+0000 7f1e45ffb700 1 --2- 172.21.15.40:0/3876876755 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f1e380a2c20 0x7f1e380a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:31.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:12:31.856+0000 7f1e4cc76700 1 --2- 172.21.15.40:0/3876876755 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f1e380a2c20 0x7f1e380a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:31.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:12:31.856+0000 7f1e45ffb700 1 -- 172.21.15.40:0/3876876755 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 4562+0+0 (secure 0 0 0) 0x7f1e340ce3e0 con 0x7f1e48102390 2024-04-21T23:12:31.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:12:31.856+0000 7f1e4cc76700 1 --2- 172.21.15.40:0/3876876755 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f1e380a2c20 0x7f1e380a50e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1e3c005950 tx=0x7f1e3c00a300 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:31.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:12:31.860+0000 7f1e45ffb700 1 -- 172.21.15.40:0/3876876755 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f1e3409c1e0 con 0x7f1e48102390 2024-04-21T23:12:32.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:12:32.076+0000 7f1e4eeda700 1 -- 172.21.15.40:0/3876876755 --> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f1e48062670 con 0x7f1e380a2c20 2024-04-21T23:12:32.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:32 smithi040 ceph-mon[27984]: osdmap e40: 6 total, 6 up, 6 in 2024-04-21T23:12:32.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:32 smithi040 ceph-mon[27984]: from='client.14349 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:12:32.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:32 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:12:32.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:32 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:12:32.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:32 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:12:32.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:32 smithi174 ceph-mon[31409]: osdmap e40: 6 total, 6 up, 6 in 2024-04-21T23:12:32.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:32 smithi174 ceph-mon[31409]: from='client.14349 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:12:32.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:32 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:12:32.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:32 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:12:32.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:32 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:12:32.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:32 smithi111 ceph-mon[31085]: osdmap e40: 6 total, 6 up, 6 in 2024-04-21T23:12:32.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:32 smithi111 ceph-mon[31085]: from='client.14349 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:12:32.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:32 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:12:32.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:32 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:12:32.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:32 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:12:33.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:33 smithi040 ceph-mon[27984]: pgmap v141: 1 pgs: 1 remapped+peering; 577 KiB data, 274 MiB used, 536 GiB / 536 GiB avail 2024-04-21T23:12:33.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:33 smithi040 ceph-mon[27984]: osdmap e41: 6 total, 6 up, 6 in 2024-04-21T23:12:33.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:33 smithi174 ceph-mon[31409]: pgmap v141: 1 pgs: 1 remapped+peering; 577 KiB data, 274 MiB used, 536 GiB / 536 GiB avail 2024-04-21T23:12:33.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:33 smithi174 ceph-mon[31409]: osdmap e41: 6 total, 6 up, 6 in 2024-04-21T23:12:33.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:33 smithi111 ceph-mon[31085]: pgmap v141: 1 pgs: 1 remapped+peering; 577 KiB data, 274 MiB used, 536 GiB / 536 GiB avail 2024-04-21T23:12:33.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:33 smithi111 ceph-mon[31085]: osdmap e41: 6 total, 6 up, 6 in 2024-04-21T23:12:34.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:34 smithi040 ceph-mon[27984]: osdmap e42: 6 total, 6 up, 6 in 2024-04-21T23:12:34.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:34 smithi174 ceph-mon[31409]: osdmap e42: 6 total, 6 up, 6 in 2024-04-21T23:12:34.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:34 smithi111 ceph-mon[31085]: osdmap e42: 6 total, 6 up, 6 in 2024-04-21T23:12:35.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:35 smithi040 ceph-mon[27984]: pgmap v144: 1 pgs: 1 remapped+peering; 577 KiB data, 274 MiB used, 536 GiB / 536 GiB avail 2024-04-21T23:12:35.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:35 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/1531561101' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "39f305f0-12d2-4732-aaa0-cd66700af538"}]: dispatch 2024-04-21T23:12:35.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:35 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/1531561101' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "39f305f0-12d2-4732-aaa0-cd66700af538"}]': finished 2024-04-21T23:12:35.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:35 smithi040 ceph-mon[27984]: osdmap e43: 7 total, 6 up, 7 in 2024-04-21T23:12:35.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:35 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T23:12:35.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:35 smithi174 ceph-mon[31409]: pgmap v144: 1 pgs: 1 remapped+peering; 577 KiB data, 274 MiB used, 536 GiB / 536 GiB avail 2024-04-21T23:12:35.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:35 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/1531561101' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "39f305f0-12d2-4732-aaa0-cd66700af538"}]: dispatch 2024-04-21T23:12:35.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:35 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/1531561101' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "39f305f0-12d2-4732-aaa0-cd66700af538"}]': finished 2024-04-21T23:12:35.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:35 smithi174 ceph-mon[31409]: osdmap e43: 7 total, 6 up, 7 in 2024-04-21T23:12:35.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:35 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T23:12:35.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:35 smithi111 ceph-mon[31085]: pgmap v144: 1 pgs: 1 remapped+peering; 577 KiB data, 274 MiB used, 536 GiB / 536 GiB avail 2024-04-21T23:12:35.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:35 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/1531561101' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "39f305f0-12d2-4732-aaa0-cd66700af538"}]: dispatch 2024-04-21T23:12:35.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:35 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/1531561101' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "39f305f0-12d2-4732-aaa0-cd66700af538"}]': finished 2024-04-21T23:12:35.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:35 smithi111 ceph-mon[31085]: osdmap e43: 7 total, 6 up, 7 in 2024-04-21T23:12:35.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:35 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T23:12:36.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:36 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/4123426691' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:12:36.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:36 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/4123426691' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:12:36.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:36 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/4123426691' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:12:37.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:37 smithi040 ceph-mon[27984]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 116 KiB/s, 0 objects/s recovering 2024-04-21T23:12:37.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:37 smithi174 ceph-mon[31409]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 116 KiB/s, 0 objects/s recovering 2024-04-21T23:12:37.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:37 smithi111 ceph-mon[31085]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 116 KiB/s, 0 objects/s recovering 2024-04-21T23:12:39.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:39 smithi040 ceph-mon[27984]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-21T23:12:39.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:39 smithi174 ceph-mon[31409]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-21T23:12:39.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:39 smithi111 ceph-mon[31085]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-21T23:12:41.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:41 smithi040 ceph-mon[27984]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-04-21T23:12:41.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:41 smithi174 ceph-mon[31409]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-04-21T23:12:41.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:41 smithi111 ceph-mon[31085]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-04-21T23:12:42.681 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:42 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-21T23:12:42.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:42 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:12:42.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:42 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-21T23:12:42.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:42 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:12:42.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:42 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-21T23:12:42.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:42 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:12:43.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:43 smithi040 ceph-mon[27984]: Deploying daemon osd.6 on smithi040 2024-04-21T23:12:43.683 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:43 smithi040 ceph-mon[27984]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-21T23:12:43.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:43 smithi174 ceph-mon[31409]: Deploying daemon osd.6 on smithi040 2024-04-21T23:12:43.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:43 smithi174 ceph-mon[31409]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-21T23:12:43.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:43 smithi111 ceph-mon[31085]: Deploying daemon osd.6 on smithi040 2024-04-21T23:12:43.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:43 smithi111 ceph-mon[31085]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-21T23:12:45.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:45 smithi174 ceph-mon[31409]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-21T23:12:45.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:45 smithi111 ceph-mon[31085]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-21T23:12:45.824 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:45 smithi040 ceph-mon[27984]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-21T23:12:47.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:47 smithi174 ceph-mon[31409]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 50 KiB/s, 0 objects/s recovering 2024-04-21T23:12:47.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:47 smithi111 ceph-mon[31085]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 50 KiB/s, 0 objects/s recovering 2024-04-21T23:12:47.704 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:47 smithi040 ceph-mon[27984]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 50 KiB/s, 0 objects/s recovering 2024-04-21T23:12:49.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:49 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:49.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:49 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:12:49.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:49 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:49.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:49 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:49.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:49 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:12:49.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:49 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:49.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:49 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:49.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:49 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:12:49.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:49 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:50.344 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:50 smithi040 ceph-mon[27984]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-21T23:12:50.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:50 smithi174 ceph-mon[31409]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-21T23:12:50.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:50 smithi111 ceph-mon[31085]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-21T23:12:52.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:12:52.002+0000 7f1e45ffb700 1 -- 172.21.15.40:0/3876876755 <== mgr.14148 v2:172.21.15.40:6800/2500760788 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f1e48062670 con 0x7f1e380a2c20 2024-04-21T23:12:52.002 INFO:teuthology.orchestra.run.smithi040.stdout:Created osd(s) 6 on host 'smithi040' 2024-04-21T23:12:52.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:12:52.004+0000 7f1e4eeda700 1 -- 172.21.15.40:0/3876876755 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f1e380a2c20 msgr2=0x7f1e380a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:52.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:12:52.004+0000 7f1e4eeda700 1 --2- 172.21.15.40:0/3876876755 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f1e380a2c20 0x7f1e380a50e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1e3c005950 tx=0x7f1e3c00a300 comp rx=0 tx=0).stop 2024-04-21T23:12:52.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:12:52.004+0000 7f1e4eeda700 1 -- 172.21.15.40:0/3876876755 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e48102390 msgr2=0x7f1e4819f620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:52.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:12:52.004+0000 7f1e4eeda700 1 --2- 172.21.15.40:0/3876876755 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e48102390 0x7f1e4819f620 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1e3400e9a0 tx=0x7f1e3400ecb0 comp rx=0 tx=0).stop 2024-04-21T23:12:52.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:12:52.004+0000 7f1e4eeda700 1 -- 172.21.15.40:0/3876876755 shutdown_connections 2024-04-21T23:12:52.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:12:52.004+0000 7f1e4eeda700 1 --2- 172.21.15.40:0/3876876755 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1e48101190 0x7f1e4806aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:52.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:12:52.004+0000 7f1e4eeda700 1 --2- 172.21.15.40:0/3876876755 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f1e380a2c20 0x7f1e380a50e0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:52.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:12:52.004+0000 7f1e4eeda700 1 --2- 172.21.15.40:0/3876876755 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e48102d30 0x7f1e481a1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:52.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:12:52.004+0000 7f1e4eeda700 1 --2- 172.21.15.40:0/3876876755 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e48102390 0x7f1e4819f620 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:52.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:12:52.004+0000 7f1e4eeda700 1 -- 172.21.15.40:0/3876876755 >> 172.21.15.40:0/3876876755 conn(0x7f1e480fc730 msgr2=0x7f1e48105630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:52.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:12:52.004+0000 7f1e4eeda700 1 -- 172.21.15.40:0/3876876755 shutdown_connections 2024-04-21T23:12:52.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:12:52.005+0000 7f1e4eeda700 1 -- 172.21.15.40:0/3876876755 wait complete. 2024-04-21T23:12:52.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:52 smithi040 ceph-mon[27984]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-21T23:12:52.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:52 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:52.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:52 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:52.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:52 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:12:52.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:52 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:12:52.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:52 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:52.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:52 smithi040 ceph-mon[27984]: from='osd.6 [v2:172.21.15.40:6818/1618855427,v1:172.21.15.40:6819/1618855427]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-21T23:12:52.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:52 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:52.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:52 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:52.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:52 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:12:52.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:52 smithi174 ceph-mon[31409]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-21T23:12:52.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:52 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:52.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:52 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:52.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:52 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:12:52.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:52 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:12:52.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:52 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:52.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:52 smithi174 ceph-mon[31409]: from='osd.6 [v2:172.21.15.40:6818/1618855427,v1:172.21.15.40:6819/1618855427]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-21T23:12:52.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:52 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:52.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:52 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:52.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:52 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:12:52.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:52 smithi111 ceph-mon[31085]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-21T23:12:52.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:52 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:52.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:52 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:52.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:52 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:12:52.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:52 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:12:52.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:52 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:52.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:52 smithi111 ceph-mon[31085]: from='osd.6 [v2:172.21.15.40:6818/1618855427,v1:172.21.15.40:6819/1618855427]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-21T23:12:52.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:52 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:52.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:52 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:52.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:52 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:12:52.777 DEBUG:teuthology.orchestra.run.smithi040:osd.6> sudo journalctl -f -n 0 -u ceph-d814e4dc-0033-11ef-bc93-c7b262605968@osd.6.service 2024-04-21T23:12:52.780 INFO:tasks.cephadm:Deploying osd.7 on smithi111 with /dev/vg_nvme/lv_2... 2024-04-21T23:12:52.780 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-21T23:12:53.025 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.b/config 2024-04-21T23:12:53.361 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:53 smithi040 ceph-mon[27984]: from='osd.6 [v2:172.21.15.40:6818/1618855427,v1:172.21.15.40:6819/1618855427]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-21T23:12:53.361 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:53 smithi040 ceph-mon[27984]: osdmap e44: 7 total, 6 up, 7 in 2024-04-21T23:12:53.361 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:53 smithi040 ceph-mon[27984]: from='osd.6 [v2:172.21.15.40:6818/1618855427,v1:172.21.15.40:6819/1618855427]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-21T23:12:53.361 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:53 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T23:12:53.361 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:53 smithi040 ceph-mon[27984]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-21T23:12:53.362 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 21 23:12:53 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-6[53641]: 2024-04-21T23:12:53.262+0000 7f4fdd554700 -1 osd.6 0 waiting for initial osdmap 2024-04-21T23:12:53.362 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 21 23:12:53 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-6[53641]: 2024-04-21T23:12:53.270+0000 7f4fd9347700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T23:12:53.404 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:53 smithi111 ceph-mon[31085]: from='osd.6 [v2:172.21.15.40:6818/1618855427,v1:172.21.15.40:6819/1618855427]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-21T23:12:53.405 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:53 smithi111 ceph-mon[31085]: osdmap e44: 7 total, 6 up, 7 in 2024-04-21T23:12:53.405 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:53 smithi111 ceph-mon[31085]: from='osd.6 [v2:172.21.15.40:6818/1618855427,v1:172.21.15.40:6819/1618855427]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-21T23:12:53.405 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:53 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T23:12:53.405 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:53 smithi111 ceph-mon[31085]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-21T23:12:53.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:53 smithi174 ceph-mon[31409]: from='osd.6 [v2:172.21.15.40:6818/1618855427,v1:172.21.15.40:6819/1618855427]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-21T23:12:53.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:53 smithi174 ceph-mon[31409]: osdmap e44: 7 total, 6 up, 7 in 2024-04-21T23:12:53.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:53 smithi174 ceph-mon[31409]: from='osd.6 [v2:172.21.15.40:6818/1618855427,v1:172.21.15.40:6819/1618855427]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-21T23:12:53.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:53 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T23:12:53.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:53 smithi174 ceph-mon[31409]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-21T23:12:54.383 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:54 smithi040 ceph-mon[27984]: from='osd.6 [v2:172.21.15.40:6818/1618855427,v1:172.21.15.40:6819/1618855427]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-04-21T23:12:54.384 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:54 smithi040 ceph-mon[27984]: osdmap e45: 7 total, 6 up, 7 in 2024-04-21T23:12:54.384 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:54 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T23:12:54.384 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:54 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T23:12:54.545 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:54 smithi111 ceph-mon[31085]: from='osd.6 [v2:172.21.15.40:6818/1618855427,v1:172.21.15.40:6819/1618855427]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-04-21T23:12:54.545 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:54 smithi111 ceph-mon[31085]: osdmap e45: 7 total, 6 up, 7 in 2024-04-21T23:12:54.545 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:54 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T23:12:54.545 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:54 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T23:12:54.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:54 smithi174 ceph-mon[31409]: from='osd.6 [v2:172.21.15.40:6818/1618855427,v1:172.21.15.40:6819/1618855427]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-04-21T23:12:54.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:54 smithi174 ceph-mon[31409]: osdmap e45: 7 total, 6 up, 7 in 2024-04-21T23:12:54.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:54 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T23:12:54.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:54 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T23:12:55.260 INFO:teuthology.orchestra.run.smithi111.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-21T23:12:55.260 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-21T23:12:55.260 INFO:teuthology.orchestra.run.smithi111.stderr: stderr: 10+0 records in 2024-04-21T23:12:55.260 INFO:teuthology.orchestra.run.smithi111.stderr:10+0 records out 2024-04-21T23:12:55.261 INFO:teuthology.orchestra.run.smithi111.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112166 s, 935 MB/s 2024-04-21T23:12:55.261 INFO:teuthology.orchestra.run.smithi111.stderr:--> Zapping successful for: 2024-04-21T23:12:55.566 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:55 smithi111 ceph-mon[31085]: purged_snaps scrub starts 2024-04-21T23:12:55.566 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:55 smithi111 ceph-mon[31085]: purged_snaps scrub ok 2024-04-21T23:12:55.566 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:55 smithi111 ceph-mon[31085]: osd.6 [v2:172.21.15.40:6818/1618855427,v1:172.21.15.40:6819/1618855427] boot 2024-04-21T23:12:55.566 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:55 smithi111 ceph-mon[31085]: osdmap e46: 7 total, 7 up, 7 in 2024-04-21T23:12:55.566 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:55 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T23:12:55.566 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:55 smithi111 ceph-mon[31085]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:12:55.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:55 smithi040 ceph-mon[27984]: purged_snaps scrub starts 2024-04-21T23:12:55.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:55 smithi040 ceph-mon[27984]: purged_snaps scrub ok 2024-04-21T23:12:55.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:55 smithi040 ceph-mon[27984]: osd.6 [v2:172.21.15.40:6818/1618855427,v1:172.21.15.40:6819/1618855427] boot 2024-04-21T23:12:55.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:55 smithi040 ceph-mon[27984]: osdmap e46: 7 total, 7 up, 7 in 2024-04-21T23:12:55.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:55 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T23:12:55.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:55 smithi040 ceph-mon[27984]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:12:55.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:55 smithi174 ceph-mon[31409]: purged_snaps scrub starts 2024-04-21T23:12:55.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:55 smithi174 ceph-mon[31409]: purged_snaps scrub ok 2024-04-21T23:12:55.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:55 smithi174 ceph-mon[31409]: osd.6 [v2:172.21.15.40:6818/1618855427,v1:172.21.15.40:6819/1618855427] boot 2024-04-21T23:12:55.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:55 smithi174 ceph-mon[31409]: osdmap e46: 7 total, 7 up, 7 in 2024-04-21T23:12:55.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:55 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T23:12:55.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:55 smithi174 ceph-mon[31409]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:12:55.933 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi111:vg_nvme/lv_2 2024-04-21T23:12:56.172 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.b/config 2024-04-21T23:12:56.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:56 smithi040 ceph-mon[27984]: osdmap e47: 7 total, 7 up, 7 in 2024-04-21T23:12:56.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:56 smithi040 ceph-mon[27984]: Detected new or changed devices on smithi040 2024-04-21T23:12:56.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:56 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:56.433 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:56 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:56.433 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:56 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:12:56.433 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:56 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:12:56.433 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:56 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:12:56.433 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:56 smithi040 ceph-mon[27984]: Adjusting osd_memory_target on smithi040 to 5697M 2024-04-21T23:12:56.433 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:56 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:56.433 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:56 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:12:56.433 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:56 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:12:56.434 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:56 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:56.528 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:56 smithi111 ceph-mon[31085]: osdmap e47: 7 total, 7 up, 7 in 2024-04-21T23:12:56.529 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:56 smithi111 ceph-mon[31085]: Detected new or changed devices on smithi040 2024-04-21T23:12:56.529 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:56 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:56.529 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:56 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:56.529 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:56 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:12:56.529 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:56 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:12:56.529 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:56 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:12:56.529 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:56 smithi111 ceph-mon[31085]: Adjusting osd_memory_target on smithi040 to 5697M 2024-04-21T23:12:56.529 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:56 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:56.529 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:56 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:12:56.529 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:56 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:12:56.529 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:56 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:56.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:56 smithi174 ceph-mon[31409]: osdmap e47: 7 total, 7 up, 7 in 2024-04-21T23:12:56.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:56 smithi174 ceph-mon[31409]: Detected new or changed devices on smithi040 2024-04-21T23:12:56.695 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:56 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:56.695 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:56 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:56.695 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:56 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:12:56.695 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:56 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:12:56.695 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:56 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:12:56.695 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:56 smithi174 ceph-mon[31409]: Adjusting osd_memory_target on smithi040 to 5697M 2024-04-21T23:12:56.696 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:56 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:56.696 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:56 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:12:56.696 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:56 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:12:56.696 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:56 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:12:57.349 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:57 smithi111 ceph-mon[31085]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:12:57.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:57 smithi040 ceph-mon[27984]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:12:57.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:57 smithi174 ceph-mon[31409]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:12:57.855 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:12:57.856+0000 7fb9c8b5d700 1 -- 172.21.15.111:0/2047887876 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9c4102d20 msgr2=0x7fb9c41095d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:57.855 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:12:57.856+0000 7fb9c8b5d700 1 --2- 172.21.15.111:0/2047887876 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9c4102d20 0x7fb9c41095d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc00b0d0 tx=0x7fb9bc00b3e0 comp rx=0 tx=0).stop 2024-04-21T23:12:57.855 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:12:57.856+0000 7fb9c8b5d700 1 -- 172.21.15.111:0/2047887876 shutdown_connections 2024-04-21T23:12:57.855 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:12:57.856+0000 7fb9c8b5d700 1 --2- 172.21.15.111:0/2047887876 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9c4101180 0x7fb9c41015a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:57.855 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:12:57.856+0000 7fb9c8b5d700 1 --2- 172.21.15.111:0/2047887876 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9c4102d20 0x7fb9c41095d0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:57.855 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:12:57.856+0000 7fb9c8b5d700 1 --2- 172.21.15.111:0/2047887876 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb9c4102380 0x7fb9c41027e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:57.855 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:12:57.856+0000 7fb9c8b5d700 1 -- 172.21.15.111:0/2047887876 >> 172.21.15.111:0/2047887876 conn(0x7fb9c40fc720 msgr2=0x7fb9c40feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:57.856 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:12:57.856+0000 7fb9c8b5d700 1 -- 172.21.15.111:0/2047887876 shutdown_connections 2024-04-21T23:12:57.856 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:12:57.856+0000 7fb9c8b5d700 1 -- 172.21.15.111:0/2047887876 wait complete. 2024-04-21T23:12:57.858 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:12:57.856+0000 7fb9c8b5d700 1 Processor -- start 2024-04-21T23:12:57.858 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:12:57.857+0000 7fb9c8b5d700 1 -- start start 2024-04-21T23:12:57.858 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:12:57.857+0000 7fb9c8b5d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9c4101180 0x7fb9c410bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:57.858 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:12:57.857+0000 7fb9c8b5d700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9c4102380 0x7fb9c410a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:57.858 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:12:57.857+0000 7fb9c8b5d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb9c4102d20 0x7fb9c410a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:57.858 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:12:57.857+0000 7fb9c8b5d700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb9c410dec0 con 0x7fb9c4102d20 2024-04-21T23:12:57.858 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:12:57.857+0000 7fb9c8b5d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb9c410dd40 con 0x7fb9c4101180 2024-04-21T23:12:57.858 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:12:57.857+0000 7fb9c8b5d700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb9c410e040 con 0x7fb9c4102380 2024-04-21T23:12:57.859 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:12:57.857+0000 7fb9c259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9c4101180 0x7fb9c410bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:57.859 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:12:57.857+0000 7fb9c259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9c4101180 0x7fb9c410bde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59168/0 (socket says 172.21.15.111:59168) 2024-04-21T23:12:57.859 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:12:57.857+0000 7fb9c259c700 1 -- 172.21.15.111:0/1735658630 learned_addr learned my addr 172.21.15.111:0/1735658630 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-04-21T23:12:57.859 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:12:57.857+0000 7fb9b9d9b700 1 --2- 172.21.15.111:0/1735658630 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9c4102380 0x7fb9c410a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:57.859 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:12:57.857+0000 7fb9c2d9d700 1 --2- 172.21.15.111:0/1735658630 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb9c4102d20 0x7fb9c410a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:57.859 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:12:57.857+0000 7fb9c259c700 1 -- 172.21.15.111:0/1735658630 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9c4102380 msgr2=0x7fb9c410a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:57.859 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:12:57.857+0000 7fb9c259c700 1 --2- 172.21.15.111:0/1735658630 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9c4102380 0x7fb9c410a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:57.859 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:12:57.857+0000 7fb9c259c700 1 -- 172.21.15.111:0/1735658630 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb9c4102d20 msgr2=0x7fb9c410a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:57.859 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:12:57.857+0000 7fb9c259c700 1 --2- 172.21.15.111:0/1735658630 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb9c4102d20 0x7fb9c410a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:57.859 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:12:57.857+0000 7fb9c259c700 1 -- 172.21.15.111:0/1735658630 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9c410b220 con 0x7fb9c4101180 2024-04-21T23:12:57.859 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:12:57.857+0000 7fb9c2d9d700 1 --2- 172.21.15.111:0/1735658630 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb9c4102d20 0x7fb9c410a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:12:57.859 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:12:57.857+0000 7fb9c259c700 1 --2- 172.21.15.111:0/1735658630 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9c4101180 0x7fb9c410bde0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb9b400c970 tx=0x7fb9b400cd30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:57.859 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:12:57.858+0000 7fb9bbfff700 1 -- 172.21.15.111:0/1735658630 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9b4004220 con 0x7fb9c4101180 2024-04-21T23:12:57.860 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:12:57.858+0000 7fb9bbfff700 1 -- 172.21.15.111:0/1735658630 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb9b4007d80 con 0x7fb9c4101180 2024-04-21T23:12:57.860 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:12:57.858+0000 7fb9c8b5d700 1 -- 172.21.15.111:0/1735658630 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9c41a2600 con 0x7fb9c4101180 2024-04-21T23:12:57.860 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:12:57.858+0000 7fb9bbfff700 1 -- 172.21.15.111:0/1735658630 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9b4003e50 con 0x7fb9c4101180 2024-04-21T23:12:57.860 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:12:57.858+0000 7fb9c8b5d700 1 -- 172.21.15.111:0/1735658630 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9c41a2bc0 con 0x7fb9c4101180 2024-04-21T23:12:57.860 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:12:57.859+0000 7fb9c8b5d700 1 -- 172.21.15.111:0/1735658630 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9c4068f20 con 0x7fb9c4101180 2024-04-21T23:12:57.861 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:12:57.860+0000 7fb9bbfff700 1 -- 172.21.15.111:0/1735658630 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fb9b40078a0 con 0x7fb9c4101180 2024-04-21T23:12:57.861 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:12:57.860+0000 7fb9bbfff700 1 --2- 172.21.15.111:0/1735658630 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fb9b00a2e40 0x7fb9b00a5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:57.861 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:12:57.860+0000 7fb9bbfff700 1 -- 172.21.15.111:0/1735658630 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 4965+0+0 (secure 0 0 0) 0x7fb9b40cdf60 con 0x7fb9c4101180 2024-04-21T23:12:57.861 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:12:57.860+0000 7fb9b9d9b700 1 --2- 172.21.15.111:0/1735658630 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fb9b00a2e40 0x7fb9b00a5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:57.861 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:12:57.861+0000 7fb9b9d9b700 1 --2- 172.21.15.111:0/1735658630 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fb9b00a2e40 0x7fb9b00a5300 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb9ac006fd0 tx=0x7fb9ac008040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:57.864 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:12:57.865+0000 7fb9bbfff700 1 -- 172.21.15.111:0/1735658630 <== mon.2 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fb9b4014030 con 0x7fb9c4101180 2024-04-21T23:12:58.070 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:12:58.070+0000 7fb9c8b5d700 1 -- 172.21.15.111:0/1735658630 --> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fb9c4062670 con 0x7fb9b00a2e40 2024-04-21T23:12:58.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:58 smithi111 ceph-mon[31085]: from='client.24275 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:12:58.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:58 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:12:58.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:58 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:12:58.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:58 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:12:58.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:58 smithi040 ceph-mon[27984]: from='client.24275 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:12:58.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:58 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:12:58.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:58 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:12:58.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:58 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:12:58.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:58 smithi174 ceph-mon[31409]: from='client.24275 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:12:58.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:58 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:12:58.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:58 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:12:58.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:58 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:12:59.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:12:59 smithi040 ceph-mon[27984]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:12:59.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:12:59 smithi174 ceph-mon[31409]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:12:59.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:12:59 smithi111 ceph-mon[31085]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:13:01.596 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:01 smithi111 ceph-mon[31085]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:13:01.596 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:01 smithi111 ceph-mon[31085]: from='client.? 172.21.15.111:0/599672177' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "769f2468-5102-490b-89f8-d0444d3169a1"}]: dispatch 2024-04-21T23:13:01.596 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:01 smithi111 ceph-mon[31085]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "769f2468-5102-490b-89f8-d0444d3169a1"}]: dispatch 2024-04-21T23:13:01.596 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:01 smithi111 ceph-mon[31085]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "769f2468-5102-490b-89f8-d0444d3169a1"}]': finished 2024-04-21T23:13:01.596 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:01 smithi111 ceph-mon[31085]: osdmap e48: 8 total, 7 up, 8 in 2024-04-21T23:13:01.596 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:01 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T23:13:01.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:01 smithi040 ceph-mon[27984]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:13:01.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:01 smithi040 ceph-mon[27984]: from='client.? 172.21.15.111:0/599672177' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "769f2468-5102-490b-89f8-d0444d3169a1"}]: dispatch 2024-04-21T23:13:01.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:01 smithi040 ceph-mon[27984]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "769f2468-5102-490b-89f8-d0444d3169a1"}]: dispatch 2024-04-21T23:13:01.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:01 smithi040 ceph-mon[27984]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "769f2468-5102-490b-89f8-d0444d3169a1"}]': finished 2024-04-21T23:13:01.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:01 smithi040 ceph-mon[27984]: osdmap e48: 8 total, 7 up, 8 in 2024-04-21T23:13:01.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:01 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T23:13:01.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:01 smithi174 ceph-mon[31409]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:13:01.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:01 smithi174 ceph-mon[31409]: from='client.? 172.21.15.111:0/599672177' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "769f2468-5102-490b-89f8-d0444d3169a1"}]: dispatch 2024-04-21T23:13:01.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:01 smithi174 ceph-mon[31409]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "769f2468-5102-490b-89f8-d0444d3169a1"}]: dispatch 2024-04-21T23:13:01.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:01 smithi174 ceph-mon[31409]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "769f2468-5102-490b-89f8-d0444d3169a1"}]': finished 2024-04-21T23:13:01.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:01 smithi174 ceph-mon[31409]: osdmap e48: 8 total, 7 up, 8 in 2024-04-21T23:13:01.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:01 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T23:13:02.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:02 smithi040 ceph-mon[27984]: from='client.? 172.21.15.111:0/51164920' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:13:02.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:02 smithi174 ceph-mon[31409]: from='client.? 172.21.15.111:0/51164920' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:13:02.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:02 smithi111 ceph-mon[31085]: from='client.? 172.21.15.111:0/51164920' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:13:03.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:03 smithi040 ceph-mon[27984]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:13:03.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:03 smithi174 ceph-mon[31409]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:13:03.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:03 smithi111 ceph-mon[31085]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:13:05.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:05 smithi040 ceph-mon[27984]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:13:05.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:05 smithi111 ceph-mon[31085]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:13:05.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:05 smithi174 ceph-mon[31409]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:13:07.675 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:07 smithi111 ceph-mon[31085]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:13:07.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:07 smithi040 ceph-mon[27984]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:13:07.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:07 smithi174 ceph-mon[31409]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:13:08.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:08 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-21T23:13:08.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:08 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:13:08.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:08 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-21T23:13:08.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:08 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:13:08.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:08 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-21T23:13:08.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:08 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:13:09.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:09 smithi040 ceph-mon[27984]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:13:09.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:09 smithi040 ceph-mon[27984]: Deploying daemon osd.7 on smithi111 2024-04-21T23:13:09.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:09 smithi174 ceph-mon[31409]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:13:09.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:09 smithi174 ceph-mon[31409]: Deploying daemon osd.7 on smithi111 2024-04-21T23:13:09.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:09 smithi111 ceph-mon[31085]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:13:09.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:09 smithi111 ceph-mon[31085]: Deploying daemon osd.7 on smithi111 2024-04-21T23:13:11.638 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:11 smithi111 ceph-mon[31085]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:13:11.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:11 smithi040 ceph-mon[27984]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:13:11.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:11 smithi174 ceph-mon[31409]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:13:13.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:13 smithi040 ceph-mon[27984]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:13:13.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:13 smithi174 ceph-mon[31409]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:13:13.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:13 smithi111 ceph-mon[31085]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:13:15.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:15 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:13:15.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:15 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:13:15.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:15 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:13:15.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:15 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:13:15.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:15 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:13:15.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:15 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:13:15.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:15 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:13:15.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:15 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:13:15.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:15 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:13:16.258 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:16 smithi111 ceph-mon[31085]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:13:16.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:16 smithi040 ceph-mon[27984]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:13:16.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:16 smithi174 ceph-mon[31409]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:13:18.045 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:13:18.045+0000 7fb9bbfff700 1 -- 172.21.15.111:0/1735658630 <== mgr.14148 v2:172.21.15.40:6800/2500760788 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fb9c4062670 con 0x7fb9b00a2e40 2024-04-21T23:13:18.045 INFO:teuthology.orchestra.run.smithi111.stdout:Created osd(s) 7 on host 'smithi111' 2024-04-21T23:13:18.047 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:13:18.048+0000 7fb9c8b5d700 1 -- 172.21.15.111:0/1735658630 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fb9b00a2e40 msgr2=0x7fb9b00a5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:18.047 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:13:18.048+0000 7fb9c8b5d700 1 --2- 172.21.15.111:0/1735658630 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fb9b00a2e40 0x7fb9b00a5300 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb9ac006fd0 tx=0x7fb9ac008040 comp rx=0 tx=0).stop 2024-04-21T23:13:18.047 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:13:18.048+0000 7fb9c8b5d700 1 -- 172.21.15.111:0/1735658630 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9c4101180 msgr2=0x7fb9c410bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:18.047 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:13:18.048+0000 7fb9c8b5d700 1 --2- 172.21.15.111:0/1735658630 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9c4101180 0x7fb9c410bde0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb9b400c970 tx=0x7fb9b400cd30 comp rx=0 tx=0).stop 2024-04-21T23:13:18.047 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:13:18.048+0000 7fb9c8b5d700 1 -- 172.21.15.111:0/1735658630 shutdown_connections 2024-04-21T23:13:18.047 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:13:18.048+0000 7fb9c8b5d700 1 --2- 172.21.15.111:0/1735658630 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fb9b00a2e40 0x7fb9b00a5300 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:18.047 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:13:18.048+0000 7fb9c8b5d700 1 --2- 172.21.15.111:0/1735658630 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb9c4102d20 0x7fb9c410a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:18.047 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:13:18.048+0000 7fb9c8b5d700 1 --2- 172.21.15.111:0/1735658630 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9c4101180 0x7fb9c410bde0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:18.048 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:13:18.048+0000 7fb9c8b5d700 1 --2- 172.21.15.111:0/1735658630 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9c4102380 0x7fb9c410a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:18.048 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:13:18.048+0000 7fb9c8b5d700 1 -- 172.21.15.111:0/1735658630 >> 172.21.15.111:0/1735658630 conn(0x7fb9c40fc720 msgr2=0x7fb9c4105620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:18.048 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:13:18.048+0000 7fb9c8b5d700 1 -- 172.21.15.111:0/1735658630 shutdown_connections 2024-04-21T23:13:18.048 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:13:18.048+0000 7fb9c8b5d700 1 -- 172.21.15.111:0/1735658630 wait complete. 2024-04-21T23:13:18.370 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:18 smithi111 ceph-mon[31085]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:13:18.370 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:18 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:13:18.370 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:18 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:13:18.370 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:18 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:13:18.370 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:18 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:13:18.371 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:18 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:13:18.371 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:18 smithi111 ceph-mon[31085]: from='osd.7 [v2:172.21.15.111:6816/927054162,v1:172.21.15.111:6817/927054162]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-21T23:13:18.371 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:18 smithi111 ceph-mon[31085]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-21T23:13:18.371 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:18 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:13:18.371 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:18 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:13:18.371 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:18 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:13:18.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:18 smithi040 ceph-mon[27984]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:13:18.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:18 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:13:18.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:18 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:13:18.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:18 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:13:18.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:18 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:13:18.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:18 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:13:18.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:18 smithi040 ceph-mon[27984]: from='osd.7 [v2:172.21.15.111:6816/927054162,v1:172.21.15.111:6817/927054162]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-21T23:13:18.433 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:18 smithi040 ceph-mon[27984]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-21T23:13:18.433 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:18 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:13:18.433 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:18 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:13:18.433 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:18 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:13:18.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:18 smithi174 ceph-mon[31409]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:13:18.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:18 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:13:18.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:18 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:13:18.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:18 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:13:18.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:18 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:13:18.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:18 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:13:18.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:18 smithi174 ceph-mon[31409]: from='osd.7 [v2:172.21.15.111:6816/927054162,v1:172.21.15.111:6817/927054162]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-21T23:13:18.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:18 smithi174 ceph-mon[31409]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-21T23:13:18.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:18 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:13:18.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:18 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:13:18.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:18 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:13:18.690 DEBUG:teuthology.orchestra.run.smithi111:osd.7> sudo journalctl -f -n 0 -u ceph-d814e4dc-0033-11ef-bc93-c7b262605968@osd.7.service 2024-04-21T23:13:18.692 INFO:tasks.cephadm:Deploying osd.8 on smithi174 with /dev/vg_nvme/lv_2... 2024-04-21T23:13:18.693 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-21T23:13:18.938 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.c/config 2024-04-21T23:13:19.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:19 smithi111 ceph-mon[31085]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-21T23:13:19.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:19 smithi111 ceph-mon[31085]: osdmap e49: 8 total, 7 up, 8 in 2024-04-21T23:13:19.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:19 smithi111 ceph-mon[31085]: from='osd.7 [v2:172.21.15.111:6816/927054162,v1:172.21.15.111:6817/927054162]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-21T23:13:19.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:19 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T23:13:19.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:19 smithi111 ceph-mon[31085]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-21T23:13:19.195 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 23:13:19 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-7[49121]: 2024-04-21T23:13:19.118+0000 7ff22895a700 -1 osd.7 0 waiting for initial osdmap 2024-04-21T23:13:19.195 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 23:13:19 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-7[49121]: 2024-04-21T23:13:19.132+0000 7ff22474d700 -1 osd.7 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T23:13:19.408 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:19 smithi174 ceph-mon[31409]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-21T23:13:19.409 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:19 smithi174 ceph-mon[31409]: osdmap e49: 8 total, 7 up, 8 in 2024-04-21T23:13:19.409 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:19 smithi174 ceph-mon[31409]: from='osd.7 [v2:172.21.15.111:6816/927054162,v1:172.21.15.111:6817/927054162]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-21T23:13:19.409 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:19 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T23:13:19.409 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:19 smithi174 ceph-mon[31409]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-21T23:13:19.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:19 smithi040 ceph-mon[27984]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-21T23:13:19.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:19 smithi040 ceph-mon[27984]: osdmap e49: 8 total, 7 up, 8 in 2024-04-21T23:13:19.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:19 smithi040 ceph-mon[27984]: from='osd.7 [v2:172.21.15.111:6816/927054162,v1:172.21.15.111:6817/927054162]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-21T23:13:19.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:19 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T23:13:19.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:19 smithi040 ceph-mon[27984]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-21T23:13:20.400 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:20 smithi111 ceph-mon[31085]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:13:20.400 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:20 smithi111 ceph-mon[31085]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-04-21T23:13:20.400 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:20 smithi111 ceph-mon[31085]: osdmap e50: 8 total, 7 up, 8 in 2024-04-21T23:13:20.400 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:20 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T23:13:20.401 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:20 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T23:13:20.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:20 smithi040 ceph-mon[27984]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:13:20.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:20 smithi040 ceph-mon[27984]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-04-21T23:13:20.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:20 smithi040 ceph-mon[27984]: osdmap e50: 8 total, 7 up, 8 in 2024-04-21T23:13:20.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:20 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T23:13:20.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:20 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T23:13:20.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:20 smithi174 ceph-mon[31409]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:13:20.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:20 smithi174 ceph-mon[31409]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-04-21T23:13:20.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:20 smithi174 ceph-mon[31409]: osdmap e50: 8 total, 7 up, 8 in 2024-04-21T23:13:20.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:20 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T23:13:20.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:20 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T23:13:21.008 INFO:teuthology.orchestra.run.smithi174.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-21T23:13:21.008 INFO:teuthology.orchestra.run.smithi174.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-21T23:13:21.008 INFO:teuthology.orchestra.run.smithi174.stderr: stderr: 10+0 records in 2024-04-21T23:13:21.008 INFO:teuthology.orchestra.run.smithi174.stderr:10+0 records out 2024-04-21T23:13:21.008 INFO:teuthology.orchestra.run.smithi174.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0155569 s, 674 MB/s 2024-04-21T23:13:21.008 INFO:teuthology.orchestra.run.smithi174.stderr:--> Zapping successful for: 2024-04-21T23:13:21.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:21 smithi040 ceph-mon[27984]: purged_snaps scrub starts 2024-04-21T23:13:21.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:21 smithi040 ceph-mon[27984]: purged_snaps scrub ok 2024-04-21T23:13:21.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:21 smithi040 ceph-mon[27984]: osd.7 [v2:172.21.15.111:6816/927054162,v1:172.21.15.111:6817/927054162] boot 2024-04-21T23:13:21.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:21 smithi040 ceph-mon[27984]: osdmap e51: 8 total, 8 up, 8 in 2024-04-21T23:13:21.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:21 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T23:13:21.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:21 smithi040 ceph-mon[27984]: osdmap e52: 8 total, 8 up, 8 in 2024-04-21T23:13:21.301 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:21 smithi174 ceph-mon[31409]: purged_snaps scrub starts 2024-04-21T23:13:21.301 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:21 smithi174 ceph-mon[31409]: purged_snaps scrub ok 2024-04-21T23:13:21.302 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:21 smithi174 ceph-mon[31409]: osd.7 [v2:172.21.15.111:6816/927054162,v1:172.21.15.111:6817/927054162] boot 2024-04-21T23:13:21.302 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:21 smithi174 ceph-mon[31409]: osdmap e51: 8 total, 8 up, 8 in 2024-04-21T23:13:21.302 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:21 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T23:13:21.302 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:21 smithi174 ceph-mon[31409]: osdmap e52: 8 total, 8 up, 8 in 2024-04-21T23:13:21.302 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:21 smithi111 ceph-mon[31085]: purged_snaps scrub starts 2024-04-21T23:13:21.303 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:21 smithi111 ceph-mon[31085]: purged_snaps scrub ok 2024-04-21T23:13:21.303 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:21 smithi111 ceph-mon[31085]: osd.7 [v2:172.21.15.111:6816/927054162,v1:172.21.15.111:6817/927054162] boot 2024-04-21T23:13:21.303 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:21 smithi111 ceph-mon[31085]: osdmap e51: 8 total, 8 up, 8 in 2024-04-21T23:13:21.303 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:21 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T23:13:21.303 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:21 smithi111 ceph-mon[31085]: osdmap e52: 8 total, 8 up, 8 in 2024-04-21T23:13:21.657 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi174:vg_nvme/lv_2 2024-04-21T23:13:21.905 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.c/config 2024-04-21T23:13:22.298 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:22 smithi174 ceph-mon[31409]: pgmap v176: 1 pgs: 1 unknown; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:13:22.299 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:22 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:13:22.299 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:22 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:13:22.299 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:22 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:13:22.299 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:22 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:13:22.299 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:22 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:13:22.299 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:22 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:13:22.299 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:22 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:13:22.299 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:22 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:13:22.299 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:22 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:13:22.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:22 smithi040 ceph-mon[27984]: pgmap v176: 1 pgs: 1 unknown; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:13:22.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:22 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:13:22.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:22 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:13:22.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:22 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:13:22.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:22 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:13:22.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:22 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:13:22.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:22 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:13:22.433 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:22 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:13:22.433 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:22 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:13:22.433 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:22 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:13:22.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:22 smithi111 ceph-mon[31085]: pgmap v176: 1 pgs: 1 unknown; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:13:22.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:22 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:13:22.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:22 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:13:22.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:22 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:13:22.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:22 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:13:22.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:22 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:13:22.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:22 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:13:22.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:22 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:13:22.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:22 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:13:22.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:22 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:13:23.400 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:23 smithi174 ceph-mon[31409]: Detected new or changed devices on smithi111 2024-04-21T23:13:23.400 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:23 smithi174 ceph-mon[31409]: Adjusting osd_memory_target on smithi111 to 5696M 2024-04-21T23:13:23.400 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:23 smithi174 ceph-mon[31409]: osdmap e53: 8 total, 8 up, 8 in 2024-04-21T23:13:23.400 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:23 smithi174 ceph-mon[31409]: osdmap e54: 8 total, 8 up, 8 in 2024-04-21T23:13:23.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:23 smithi040 ceph-mon[27984]: Detected new or changed devices on smithi111 2024-04-21T23:13:23.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:23 smithi040 ceph-mon[27984]: Adjusting osd_memory_target on smithi111 to 5696M 2024-04-21T23:13:23.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:23 smithi040 ceph-mon[27984]: osdmap e53: 8 total, 8 up, 8 in 2024-04-21T23:13:23.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:23 smithi040 ceph-mon[27984]: osdmap e54: 8 total, 8 up, 8 in 2024-04-21T23:13:23.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:23 smithi111 ceph-mon[31085]: Detected new or changed devices on smithi111 2024-04-21T23:13:23.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:23 smithi111 ceph-mon[31085]: Adjusting osd_memory_target on smithi111 to 5696M 2024-04-21T23:13:23.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:23 smithi111 ceph-mon[31085]: osdmap e53: 8 total, 8 up, 8 in 2024-04-21T23:13:23.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:23 smithi111 ceph-mon[31085]: osdmap e54: 8 total, 8 up, 8 in 2024-04-21T23:13:23.638 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:13:23.636+0000 7f43fc2fe700 1 -- 172.21.15.174:0/3487584471 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43f4103e30 msgr2=0x7f43f410a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:23.639 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:13:23.636+0000 7f43fc2fe700 1 --2- 172.21.15.174:0/3487584471 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43f4103e30 0x7f43f410a6e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f43ec00b0a0 tx=0x7f43ec00b3b0 comp rx=0 tx=0).stop 2024-04-21T23:13:23.639 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:13:23.636+0000 7f43fc2fe700 1 -- 172.21.15.174:0/3487584471 shutdown_connections 2024-04-21T23:13:23.639 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:13:23.636+0000 7f43fc2fe700 1 --2- 172.21.15.174:0/3487584471 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43f4103e30 0x7f43f410a6e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:23.639 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:13:23.636+0000 7f43fc2fe700 1 --2- 172.21.15.174:0/3487584471 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43f4102290 0x7f43f41026b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:23.639 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:13:23.636+0000 7f43fc2fe700 1 --2- 172.21.15.174:0/3487584471 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f43f4103490 0x7f43f41038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:23.639 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:13:23.637+0000 7f43fc2fe700 1 -- 172.21.15.174:0/3487584471 >> 172.21.15.174:0/3487584471 conn(0x7f43f40fd830 msgr2=0x7f43f40ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:23.639 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:13:23.637+0000 7f43fc2fe700 1 -- 172.21.15.174:0/3487584471 shutdown_connections 2024-04-21T23:13:23.639 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:13:23.637+0000 7f43fc2fe700 1 -- 172.21.15.174:0/3487584471 wait complete. 2024-04-21T23:13:23.639 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:13:23.637+0000 7f43fc2fe700 1 Processor -- start 2024-04-21T23:13:23.639 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:13:23.637+0000 7f43fc2fe700 1 -- start start 2024-04-21T23:13:23.639 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:13:23.637+0000 7f43fc2fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43f4102290 0x7f43f4075fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:23.639 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:13:23.637+0000 7f43fc2fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f43f4103490 0x7f43f4074620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:23.639 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:13:23.637+0000 7f43fc2fe700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43f4103e30 0x7f43f4074b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:23.640 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:13:23.637+0000 7f43fc2fe700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f43f410df70 con 0x7f43f4103490 2024-04-21T23:13:23.640 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:13:23.637+0000 7f43fc2fe700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f43f410ddf0 con 0x7f43f4102290 2024-04-21T23:13:23.640 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:13:23.637+0000 7f43fc2fe700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f43f410e0f0 con 0x7f43f4103e30 2024-04-21T23:13:23.640 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:13:23.637+0000 7f43fa89b700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43f4103e30 0x7f43f4074b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:23.640 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:13:23.637+0000 7f43fa89b700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43f4103e30 0x7f43f4074b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.174:35930/0 (socket says 172.21.15.174:35930) 2024-04-21T23:13:23.640 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:13:23.637+0000 7f43fa89b700 1 -- 172.21.15.174:0/4015640302 learned_addr learned my addr 172.21.15.174:0/4015640302 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-04-21T23:13:23.640 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:13:23.637+0000 7f43f9899700 1 --2- 172.21.15.174:0/4015640302 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f43f4103490 0x7f43f4074620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:23.640 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:13:23.638+0000 7f43fa09a700 1 --2- 172.21.15.174:0/4015640302 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43f4102290 0x7f43f4075fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:23.640 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:13:23.638+0000 7f43fa89b700 1 -- 172.21.15.174:0/4015640302 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43f4102290 msgr2=0x7f43f4075fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:23.640 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:13:23.638+0000 7f43fa89b700 1 --2- 172.21.15.174:0/4015640302 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43f4102290 0x7f43f4075fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:23.640 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:13:23.638+0000 7f43fa89b700 1 -- 172.21.15.174:0/4015640302 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f43f4103490 msgr2=0x7f43f4074620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:23.640 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:13:23.638+0000 7f43fa89b700 1 --2- 172.21.15.174:0/4015640302 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f43f4103490 0x7f43f4074620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:23.640 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:13:23.638+0000 7f43fa89b700 1 -- 172.21.15.174:0/4015640302 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43f4075410 con 0x7f43f4103e30 2024-04-21T23:13:23.641 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:13:23.638+0000 7f43fa89b700 1 --2- 172.21.15.174:0/4015640302 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43f4103e30 0x7f43f4074b60 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f43ec00b070 tx=0x7f43ec00be60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:23.641 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:13:23.638+0000 7f43eb7fe700 1 -- 172.21.15.174:0/4015640302 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43ec021020 con 0x7f43f4103e30 2024-04-21T23:13:23.641 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:13:23.638+0000 7f43eb7fe700 1 -- 172.21.15.174:0/4015640302 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f43ec005520 con 0x7f43f4103e30 2024-04-21T23:13:23.641 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:13:23.638+0000 7f43fc2fe700 1 -- 172.21.15.174:0/4015640302 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43f41a26e0 con 0x7f43f4103e30 2024-04-21T23:13:23.641 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:13:23.638+0000 7f43eb7fe700 1 -- 172.21.15.174:0/4015640302 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43ec01b900 con 0x7f43f4103e30 2024-04-21T23:13:23.641 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:13:23.638+0000 7f43fc2fe700 1 -- 172.21.15.174:0/4015640302 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43f41a2bc0 con 0x7f43f4103e30 2024-04-21T23:13:23.641 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:13:23.639+0000 7f43fc2fe700 1 -- 172.21.15.174:0/4015640302 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43f4068f20 con 0x7f43f4103e30 2024-04-21T23:13:23.642 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:13:23.640+0000 7f43eb7fe700 1 -- 172.21.15.174:0/4015640302 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f43ec024020 con 0x7f43f4103e30 2024-04-21T23:13:23.642 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:13:23.641+0000 7f43eb7fe700 1 --2- 172.21.15.174:0/4015640302 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f43e00a2c20 0x7f43e00a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:23.642 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:13:23.641+0000 7f43eb7fe700 1 -- 172.21.15.174:0/4015640302 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 5397+0+0 (secure 0 0 0) 0x7f43ec00f020 con 0x7f43f4103e30 2024-04-21T23:13:23.642 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:13:23.641+0000 7f43fa09a700 1 --2- 172.21.15.174:0/4015640302 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f43e00a2c20 0x7f43e00a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:23.642 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:13:23.642+0000 7f43fa09a700 1 --2- 172.21.15.174:0/4015640302 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f43e00a2c20 0x7f43e00a50e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f43f41032f0 tx=0x7f43f0009190 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:23.644 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:13:23.645+0000 7f43eb7fe700 1 -- 172.21.15.174:0/4015640302 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f43ec09d5f0 con 0x7f43f4103e30 2024-04-21T23:13:23.852 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:13:23.852+0000 7f43fc2fe700 1 -- 172.21.15.174:0/4015640302 --> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f43f4062670 con 0x7f43e00a2c20 2024-04-21T23:13:24.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:24 smithi040 ceph-mon[27984]: pgmap v179: 1 pgs: 1 unknown; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:13:24.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:24 smithi040 ceph-mon[27984]: osdmap e55: 8 total, 8 up, 8 in 2024-04-21T23:13:24.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:24 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:13:24.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:24 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:13:24.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:24 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:13:24.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:24 smithi174 ceph-mon[31409]: pgmap v179: 1 pgs: 1 unknown; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:13:24.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:24 smithi174 ceph-mon[31409]: osdmap e55: 8 total, 8 up, 8 in 2024-04-21T23:13:24.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:24 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:13:24.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:24 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:13:24.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:24 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:13:24.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:24 smithi111 ceph-mon[31085]: pgmap v179: 1 pgs: 1 unknown; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:13:24.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:24 smithi111 ceph-mon[31085]: osdmap e55: 8 total, 8 up, 8 in 2024-04-21T23:13:24.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:24 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:13:24.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:24 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:13:24.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:24 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:13:25.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:25 smithi040 ceph-mon[27984]: from='client.24295 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:13:25.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:25 smithi174 ceph-mon[31409]: from='client.24295 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:13:25.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:25 smithi111 ceph-mon[31085]: from='client.24295 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:13:26.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:26 smithi040 ceph-mon[27984]: pgmap v182: 1 pgs: 1 unknown; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:13:26.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:26 smithi174 ceph-mon[31409]: pgmap v182: 1 pgs: 1 unknown; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:13:26.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:26 smithi111 ceph-mon[31085]: pgmap v182: 1 pgs: 1 unknown; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:13:27.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:27 smithi040 ceph-mon[27984]: from='client.? 172.21.15.174:0/856696311' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1a444e80-007d-473b-b972-4c814a813c22"}]: dispatch 2024-04-21T23:13:27.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:27 smithi040 ceph-mon[27984]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1a444e80-007d-473b-b972-4c814a813c22"}]: dispatch 2024-04-21T23:13:27.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:27 smithi040 ceph-mon[27984]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1a444e80-007d-473b-b972-4c814a813c22"}]': finished 2024-04-21T23:13:27.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:27 smithi040 ceph-mon[27984]: osdmap e56: 9 total, 8 up, 9 in 2024-04-21T23:13:27.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:27 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T23:13:27.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:27 smithi174 ceph-mon[31409]: from='client.? 172.21.15.174:0/856696311' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1a444e80-007d-473b-b972-4c814a813c22"}]: dispatch 2024-04-21T23:13:27.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:27 smithi174 ceph-mon[31409]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1a444e80-007d-473b-b972-4c814a813c22"}]: dispatch 2024-04-21T23:13:27.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:27 smithi174 ceph-mon[31409]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1a444e80-007d-473b-b972-4c814a813c22"}]': finished 2024-04-21T23:13:27.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:27 smithi174 ceph-mon[31409]: osdmap e56: 9 total, 8 up, 9 in 2024-04-21T23:13:27.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:27 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T23:13:27.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:27 smithi111 ceph-mon[31085]: from='client.? 172.21.15.174:0/856696311' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1a444e80-007d-473b-b972-4c814a813c22"}]: dispatch 2024-04-21T23:13:27.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:27 smithi111 ceph-mon[31085]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1a444e80-007d-473b-b972-4c814a813c22"}]: dispatch 2024-04-21T23:13:27.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:27 smithi111 ceph-mon[31085]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1a444e80-007d-473b-b972-4c814a813c22"}]': finished 2024-04-21T23:13:27.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:27 smithi111 ceph-mon[31085]: osdmap e56: 9 total, 8 up, 9 in 2024-04-21T23:13:27.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:27 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T23:13:28.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:28 smithi040 ceph-mon[27984]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:13:28.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:28 smithi040 ceph-mon[27984]: from='client.? 172.21.15.174:0/107406622' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:13:28.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:28 smithi174 ceph-mon[31409]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:13:28.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:28 smithi174 ceph-mon[31409]: from='client.? 172.21.15.174:0/107406622' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:13:28.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:28 smithi111 ceph-mon[31085]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:13:28.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:28 smithi111 ceph-mon[31085]: from='client.? 172.21.15.174:0/107406622' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:13:30.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:30 smithi040 ceph-mon[27984]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:13:30.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:30 smithi174 ceph-mon[31409]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:13:30.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:30 smithi111 ceph-mon[31085]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:13:32.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:32 smithi040 ceph-mon[27984]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:13:32.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:32 smithi174 ceph-mon[31409]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:13:32.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:32 smithi111 ceph-mon[31085]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:13:34.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:34 smithi040 ceph-mon[27984]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:13:34.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:34 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-21T23:13:34.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:34 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:13:34.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:34 smithi174 ceph-mon[31409]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:13:34.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:34 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-21T23:13:34.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:34 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:13:34.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:34 smithi111 ceph-mon[31085]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:13:34.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:34 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-21T23:13:34.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:34 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:13:35.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:35 smithi040 ceph-mon[27984]: Deploying daemon osd.8 on smithi174 2024-04-21T23:13:35.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:35 smithi174 ceph-mon[31409]: Deploying daemon osd.8 on smithi174 2024-04-21T23:13:35.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:35 smithi111 ceph-mon[31085]: Deploying daemon osd.8 on smithi174 2024-04-21T23:13:36.285 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:36 smithi174 ceph-mon[31409]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:13:36.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:36 smithi040 ceph-mon[27984]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:13:36.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:36 smithi111 ceph-mon[31085]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:13:38.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:38 smithi040 ceph-mon[27984]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:13:38.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:38 smithi111 ceph-mon[31085]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:13:38.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:38 smithi174 ceph-mon[31409]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:13:40.394 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:40 smithi174 ceph-mon[31409]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:13:40.394 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:40 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:13:40.394 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:40 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:13:40.394 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:40 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:13:40.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:40 smithi040 ceph-mon[27984]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:13:40.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:40 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:13:40.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:40 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:13:40.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:40 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:13:40.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:40 smithi111 ceph-mon[31085]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:13:40.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:40 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:13:40.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:40 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:13:40.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:40 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:13:42.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:42 smithi040 ceph-mon[27984]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:13:42.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:42 smithi111 ceph-mon[31085]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:13:42.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:42 smithi174 ceph-mon[31409]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:13:43.575 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:13:43.575+0000 7f43eb7fe700 1 -- 172.21.15.174:0/4015640302 <== mgr.14148 v2:172.21.15.40:6800/2500760788 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f43f4062670 con 0x7f43e00a2c20 2024-04-21T23:13:43.575 INFO:teuthology.orchestra.run.smithi174.stdout:Created osd(s) 8 on host 'smithi174' 2024-04-21T23:13:43.577 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:13:43.577+0000 7f43fc2fe700 1 -- 172.21.15.174:0/4015640302 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f43e00a2c20 msgr2=0x7f43e00a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:43.577 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:13:43.577+0000 7f43fc2fe700 1 --2- 172.21.15.174:0/4015640302 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f43e00a2c20 0x7f43e00a50e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f43f41032f0 tx=0x7f43f0009190 comp rx=0 tx=0).stop 2024-04-21T23:13:43.577 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:13:43.577+0000 7f43fc2fe700 1 -- 172.21.15.174:0/4015640302 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43f4103e30 msgr2=0x7f43f4074b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:43.577 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:13:43.577+0000 7f43fc2fe700 1 --2- 172.21.15.174:0/4015640302 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43f4103e30 0x7f43f4074b60 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f43ec00b070 tx=0x7f43ec00be60 comp rx=0 tx=0).stop 2024-04-21T23:13:43.577 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:13:43.577+0000 7f43fc2fe700 1 -- 172.21.15.174:0/4015640302 shutdown_connections 2024-04-21T23:13:43.577 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:13:43.577+0000 7f43fc2fe700 1 --2- 172.21.15.174:0/4015640302 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43f4103e30 0x7f43f4074b60 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:43.577 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:13:43.577+0000 7f43fc2fe700 1 --2- 172.21.15.174:0/4015640302 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f43e00a2c20 0x7f43e00a50e0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:43.578 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:13:43.577+0000 7f43fc2fe700 1 --2- 172.21.15.174:0/4015640302 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43f4102290 0x7f43f4075fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:43.578 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:13:43.577+0000 7f43fc2fe700 1 --2- 172.21.15.174:0/4015640302 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f43f4103490 0x7f43f4074620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:43.578 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:13:43.577+0000 7f43fc2fe700 1 -- 172.21.15.174:0/4015640302 >> 172.21.15.174:0/4015640302 conn(0x7f43f40fd830 msgr2=0x7f43f4106790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:43.578 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:13:43.578+0000 7f43fc2fe700 1 -- 172.21.15.174:0/4015640302 shutdown_connections 2024-04-21T23:13:43.578 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:13:43.578+0000 7f43fc2fe700 1 -- 172.21.15.174:0/4015640302 wait complete. 2024-04-21T23:13:43.888 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:43 smithi174 ceph-mon[31409]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:13:43.889 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:43 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:13:43.889 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:43 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:13:43.889 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:43 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:13:43.889 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:43 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:13:43.889 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:43 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:13:43.889 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:43 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:13:43.889 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:43 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:13:43.889 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:43 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:13:43.889 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:43 smithi174 ceph-mon[31409]: from='osd.8 [v2:172.21.15.174:6816/3817848746,v1:172.21.15.174:6817/3817848746]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-21T23:13:43.889 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:43 smithi174 ceph-mon[31409]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-21T23:13:43.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:43 smithi111 ceph-mon[31085]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:13:43.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:43 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:13:43.946 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:43 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:13:43.946 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:43 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:13:43.946 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:43 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:13:43.946 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:43 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:13:43.946 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:43 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:13:43.946 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:43 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:13:43.946 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:43 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:13:43.946 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:43 smithi111 ceph-mon[31085]: from='osd.8 [v2:172.21.15.174:6816/3817848746,v1:172.21.15.174:6817/3817848746]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-21T23:13:43.946 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:43 smithi111 ceph-mon[31085]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-21T23:13:44.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:43 smithi040 ceph-mon[27984]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:13:44.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:43 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:13:44.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:43 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:13:44.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:43 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:13:44.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:43 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:13:44.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:43 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:13:44.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:43 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:13:44.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:43 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:13:44.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:43 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:13:44.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:43 smithi040 ceph-mon[27984]: from='osd.8 [v2:172.21.15.174:6816/3817848746,v1:172.21.15.174:6817/3817848746]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-21T23:13:44.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:43 smithi040 ceph-mon[27984]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-21T23:13:44.198 DEBUG:teuthology.orchestra.run.smithi174:osd.8> sudo journalctl -f -n 0 -u ceph-d814e4dc-0033-11ef-bc93-c7b262605968@osd.8.service 2024-04-21T23:13:44.200 INFO:tasks.cephadm:Deploying osd.9 on smithi040 with /dev/vg_nvme/lv_1... 2024-04-21T23:13:44.200 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-21T23:13:44.455 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:13:45.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:45 smithi174 ceph-mon[31409]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:13:45.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:45 smithi174 ceph-mon[31409]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-21T23:13:45.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:45 smithi174 ceph-mon[31409]: osdmap e57: 9 total, 8 up, 9 in 2024-04-21T23:13:45.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:45 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T23:13:45.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:45 smithi174 ceph-mon[31409]: from='osd.8 [v2:172.21.15.174:6816/3817848746,v1:172.21.15.174:6817/3817848746]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-04-21T23:13:45.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:45 smithi174 ceph-mon[31409]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-04-21T23:13:45.695 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:13:45 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:13:45.557+0000 7fcab8823700 -1 osd.8 0 waiting for initial osdmap 2024-04-21T23:13:45.695 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:13:45 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:13:45.568+0000 7fcab4616700 -1 osd.8 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T23:13:45.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:45 smithi040 ceph-mon[27984]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:13:45.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:45 smithi040 ceph-mon[27984]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-21T23:13:45.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:45 smithi040 ceph-mon[27984]: osdmap e57: 9 total, 8 up, 9 in 2024-04-21T23:13:45.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:45 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T23:13:45.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:45 smithi040 ceph-mon[27984]: from='osd.8 [v2:172.21.15.174:6816/3817848746,v1:172.21.15.174:6817/3817848746]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-04-21T23:13:45.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:45 smithi040 ceph-mon[27984]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-04-21T23:13:45.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:45 smithi111 ceph-mon[31085]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:13:45.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:45 smithi111 ceph-mon[31085]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-21T23:13:45.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:45 smithi111 ceph-mon[31085]: osdmap e57: 9 total, 8 up, 9 in 2024-04-21T23:13:45.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:45 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T23:13:45.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:45 smithi111 ceph-mon[31085]: from='osd.8 [v2:172.21.15.174:6816/3817848746,v1:172.21.15.174:6817/3817848746]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-04-21T23:13:45.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:45 smithi111 ceph-mon[31085]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-04-21T23:13:46.636 INFO:teuthology.orchestra.run.smithi040.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-21T23:13:46.636 INFO:teuthology.orchestra.run.smithi040.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-21T23:13:46.636 INFO:teuthology.orchestra.run.smithi040.stderr: stderr: 10+0 records in 2024-04-21T23:13:46.636 INFO:teuthology.orchestra.run.smithi040.stderr:10+0 records out 2024-04-21T23:13:46.636 INFO:teuthology.orchestra.run.smithi040.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0118427 s, 885 MB/s 2024-04-21T23:13:46.637 INFO:teuthology.orchestra.run.smithi040.stderr:--> Zapping successful for: 2024-04-21T23:13:46.895 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:46 smithi174 ceph-mon[31409]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-04-21T23:13:46.895 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:46 smithi174 ceph-mon[31409]: osdmap e58: 9 total, 8 up, 9 in 2024-04-21T23:13:46.895 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:46 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T23:13:46.895 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:46 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T23:13:46.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:46 smithi040 ceph-mon[27984]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-04-21T23:13:46.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:46 smithi040 ceph-mon[27984]: osdmap e58: 9 total, 8 up, 9 in 2024-04-21T23:13:46.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:46 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T23:13:46.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:46 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T23:13:46.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:46 smithi111 ceph-mon[31085]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-04-21T23:13:46.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:46 smithi111 ceph-mon[31085]: osdmap e58: 9 total, 8 up, 9 in 2024-04-21T23:13:46.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:46 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T23:13:46.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:46 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T23:13:47.289 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi040:vg_nvme/lv_1 2024-04-21T23:13:47.549 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:13:47.857 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:47 smithi040 ceph-mon[27984]: purged_snaps scrub starts 2024-04-21T23:13:47.858 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:47 smithi040 ceph-mon[27984]: purged_snaps scrub ok 2024-04-21T23:13:47.858 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:47 smithi040 ceph-mon[27984]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:13:47.858 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:47 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T23:13:47.858 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:47 smithi040 ceph-mon[27984]: osd.8 [v2:172.21.15.174:6816/3817848746,v1:172.21.15.174:6817/3817848746] boot 2024-04-21T23:13:47.858 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:47 smithi040 ceph-mon[27984]: osdmap e59: 9 total, 9 up, 9 in 2024-04-21T23:13:47.858 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:47 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T23:13:47.858 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:47 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:13:47.858 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:47 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:13:47.858 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:47 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:13:47.858 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:47 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:13:47.858 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:47 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:13:47.858 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:47 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:13:47.858 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:47 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:13:47.859 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:47 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:13:47.859 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:47 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:13:47.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:47 smithi174 ceph-mon[31409]: purged_snaps scrub starts 2024-04-21T23:13:47.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:47 smithi174 ceph-mon[31409]: purged_snaps scrub ok 2024-04-21T23:13:47.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:47 smithi174 ceph-mon[31409]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:13:47.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:47 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T23:13:47.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:47 smithi174 ceph-mon[31409]: osd.8 [v2:172.21.15.174:6816/3817848746,v1:172.21.15.174:6817/3817848746] boot 2024-04-21T23:13:47.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:47 smithi174 ceph-mon[31409]: osdmap e59: 9 total, 9 up, 9 in 2024-04-21T23:13:47.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:47 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T23:13:47.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:47 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:13:47.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:47 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:13:47.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:47 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:13:47.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:47 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:13:47.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:47 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:13:47.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:47 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:13:47.945 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:47 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:13:47.945 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:47 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:13:47.945 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:47 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:13:47.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:47 smithi111 ceph-mon[31085]: purged_snaps scrub starts 2024-04-21T23:13:47.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:47 smithi111 ceph-mon[31085]: purged_snaps scrub ok 2024-04-21T23:13:47.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:47 smithi111 ceph-mon[31085]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:13:47.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:47 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T23:13:47.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:47 smithi111 ceph-mon[31085]: osd.8 [v2:172.21.15.174:6816/3817848746,v1:172.21.15.174:6817/3817848746] boot 2024-04-21T23:13:47.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:47 smithi111 ceph-mon[31085]: osdmap e59: 9 total, 9 up, 9 in 2024-04-21T23:13:47.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:47 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T23:13:47.946 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:47 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:13:47.946 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:47 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:13:47.946 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:47 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:13:47.946 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:47 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:13:47.946 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:47 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:13:47.946 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:47 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:13:47.946 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:47 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:13:47.946 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:47 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:13:47.946 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:47 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:13:48.703 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:48 smithi040 ceph-mon[27984]: Detected new or changed devices on smithi174 2024-04-21T23:13:48.703 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:48 smithi040 ceph-mon[27984]: Adjusting osd_memory_target on smithi174 to 5696M 2024-04-21T23:13:48.703 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:48 smithi040 ceph-mon[27984]: osdmap e60: 9 total, 9 up, 9 in 2024-04-21T23:13:48.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:48 smithi174 ceph-mon[31409]: Detected new or changed devices on smithi174 2024-04-21T23:13:48.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:48 smithi174 ceph-mon[31409]: Adjusting osd_memory_target on smithi174 to 5696M 2024-04-21T23:13:48.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:48 smithi174 ceph-mon[31409]: osdmap e60: 9 total, 9 up, 9 in 2024-04-21T23:13:48.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:48 smithi111 ceph-mon[31085]: Detected new or changed devices on smithi174 2024-04-21T23:13:48.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:48 smithi111 ceph-mon[31085]: Adjusting osd_memory_target on smithi174 to 5696M 2024-04-21T23:13:48.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:48 smithi111 ceph-mon[31085]: osdmap e60: 9 total, 9 up, 9 in 2024-04-21T23:13:49.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:13:49.232+0000 7fd447a29700 1 -- 172.21.15.40:0/1866072979 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4401011a0 msgr2=0x7fd4401015c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:49.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:13:49.232+0000 7fd447a29700 1 --2- 172.21.15.40:0/1866072979 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4401011a0 0x7fd4401015c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fd43c009920 tx=0x7fd43c009c30 comp rx=0 tx=0).stop 2024-04-21T23:13:49.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:13:49.232+0000 7fd447a29700 1 -- 172.21.15.40:0/1866072979 shutdown_connections 2024-04-21T23:13:49.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:13:49.232+0000 7fd447a29700 1 --2- 172.21.15.40:0/1866072979 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd440102d40 0x7fd4401095f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:49.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:13:49.232+0000 7fd447a29700 1 --2- 172.21.15.40:0/1866072979 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4401011a0 0x7fd4401015c0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:49.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:13:49.232+0000 7fd447a29700 1 --2- 172.21.15.40:0/1866072979 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd4401023a0 0x7fd440102800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:49.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:13:49.232+0000 7fd447a29700 1 -- 172.21.15.40:0/1866072979 >> 172.21.15.40:0/1866072979 conn(0x7fd4400fc760 msgr2=0x7fd4400feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:49.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:13:49.233+0000 7fd447a29700 1 -- 172.21.15.40:0/1866072979 shutdown_connections 2024-04-21T23:13:49.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:13:49.233+0000 7fd447a29700 1 -- 172.21.15.40:0/1866072979 wait complete. 2024-04-21T23:13:49.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:13:49.233+0000 7fd447a29700 1 Processor -- start 2024-04-21T23:13:49.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:13:49.233+0000 7fd447a29700 1 -- start start 2024-04-21T23:13:49.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:13:49.233+0000 7fd447a29700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4401011a0 0x7fd44006b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:49.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:13:49.233+0000 7fd447a29700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4401023a0 0x7fd44019d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:49.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:13:49.233+0000 7fd447a29700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd440102d40 0x7fd44019f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:49.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:13:49.233+0000 7fd447a29700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd44010cd60 con 0x7fd4401023a0 2024-04-21T23:13:49.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:13:49.233+0000 7fd447a29700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd44010cbe0 con 0x7fd4401011a0 2024-04-21T23:13:49.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:13:49.233+0000 7fd447a29700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd44010cee0 con 0x7fd440102d40 2024-04-21T23:13:49.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:13:49.233+0000 7fd444fc4700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4401023a0 0x7fd44019d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:49.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:13:49.233+0000 7fd444fc4700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4401023a0 0x7fd44019d2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54422/0 (socket says 172.21.15.40:54422) 2024-04-21T23:13:49.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:13:49.233+0000 7fd4457c5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4401011a0 0x7fd44006b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:49.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:13:49.233+0000 7fd444fc4700 1 -- 172.21.15.40:0/435210413 learned_addr learned my addr 172.21.15.40:0/435210413 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:13:49.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:13:49.234+0000 7fd445fc6700 1 --2- 172.21.15.40:0/435210413 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd440102d40 0x7fd44019f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:49.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:13:49.234+0000 7fd4457c5700 1 -- 172.21.15.40:0/435210413 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd440102d40 msgr2=0x7fd44019f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:49.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:13:49.234+0000 7fd4457c5700 1 --2- 172.21.15.40:0/435210413 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd440102d40 0x7fd44019f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:49.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:13:49.234+0000 7fd4457c5700 1 -- 172.21.15.40:0/435210413 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4401023a0 msgr2=0x7fd44019d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:49.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:13:49.234+0000 7fd4457c5700 1 --2- 172.21.15.40:0/435210413 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4401023a0 0x7fd44019d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:49.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:13:49.234+0000 7fd4457c5700 1 -- 172.21.15.40:0/435210413 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4401a0220 con 0x7fd4401011a0 2024-04-21T23:13:49.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:13:49.234+0000 7fd445fc6700 1 --2- 172.21.15.40:0/435210413 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd440102d40 0x7fd44019f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:13:49.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:13:49.234+0000 7fd4457c5700 1 --2- 172.21.15.40:0/435210413 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4401011a0 0x7fd44006b540 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd43c000c00 tx=0x7fd43c010f80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:49.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:13:49.234+0000 7fd4367fc700 1 -- 172.21.15.40:0/435210413 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd43c027070 con 0x7fd4401011a0 2024-04-21T23:13:49.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:13:49.234+0000 7fd447a29700 1 -- 172.21.15.40:0/435210413 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4401a04b0 con 0x7fd4401011a0 2024-04-21T23:13:49.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:13:49.234+0000 7fd4367fc700 1 -- 172.21.15.40:0/435210413 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd43c0056d0 con 0x7fd4401011a0 2024-04-21T23:13:49.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:13:49.234+0000 7fd4367fc700 1 -- 172.21.15.40:0/435210413 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd43c0049c0 con 0x7fd4401011a0 2024-04-21T23:13:49.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:13:49.234+0000 7fd447a29700 1 -- 172.21.15.40:0/435210413 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4401a0960 con 0x7fd4401011a0 2024-04-21T23:13:49.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:13:49.235+0000 7fd447a29700 1 -- 172.21.15.40:0/435210413 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd440068f20 con 0x7fd4401011a0 2024-04-21T23:13:49.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:13:49.237+0000 7fd4367fc700 1 -- 172.21.15.40:0/435210413 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fd43c02a070 con 0x7fd4401011a0 2024-04-21T23:13:49.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:13:49.237+0000 7fd4367fc700 1 --2- 172.21.15.40:0/435210413 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fd42c0a2c20 0x7fd42c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:49.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:13:49.237+0000 7fd444fc4700 1 --2- 172.21.15.40:0/435210413 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fd42c0a2c20 0x7fd42c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:49.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:13:49.237+0000 7fd4367fc700 1 -- 172.21.15.40:0/435210413 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5858+0+0 (secure 0 0 0) 0x7fd43c0cf360 con 0x7fd4401011a0 2024-04-21T23:13:49.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:13:49.238+0000 7fd444fc4700 1 --2- 172.21.15.40:0/435210413 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fd42c0a2c20 0x7fd42c0a50e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd430005d10 tx=0x7fd430009310 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:49.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:13:49.241+0000 7fd4367fc700 1 -- 172.21.15.40:0/435210413 <== mon.2 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fd43c05e6f0 con 0x7fd4401011a0 2024-04-21T23:13:49.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:13:49.456+0000 7fd447a29700 1 -- 172.21.15.40:0/435210413 --> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fd44002dfa0 con 0x7fd42c0a2c20 2024-04-21T23:13:49.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:49 smithi040 ceph-mon[27984]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail 2024-04-21T23:13:49.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:49 smithi040 ceph-mon[27984]: osdmap e61: 9 total, 9 up, 9 in 2024-04-21T23:13:49.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:49 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:13:49.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:49 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:13:49.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:49 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:13:49.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:49 smithi174 ceph-mon[31409]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail 2024-04-21T23:13:49.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:49 smithi174 ceph-mon[31409]: osdmap e61: 9 total, 9 up, 9 in 2024-04-21T23:13:49.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:49 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:13:49.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:49 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:13:49.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:49 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:13:49.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:49 smithi111 ceph-mon[31085]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail 2024-04-21T23:13:49.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:49 smithi111 ceph-mon[31085]: osdmap e61: 9 total, 9 up, 9 in 2024-04-21T23:13:49.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:49 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:13:49.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:49 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:13:49.946 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:49 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:13:50.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:50 smithi040 ceph-mon[27984]: from='client.24326 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:13:50.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:50 smithi040 ceph-mon[27984]: osdmap e62: 9 total, 9 up, 9 in 2024-04-21T23:13:50.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:50 smithi174 ceph-mon[31409]: from='client.24326 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:13:50.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:50 smithi174 ceph-mon[31409]: osdmap e62: 9 total, 9 up, 9 in 2024-04-21T23:13:50.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:50 smithi111 ceph-mon[31085]: from='client.24326 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:13:50.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:50 smithi111 ceph-mon[31085]: osdmap e62: 9 total, 9 up, 9 in 2024-04-21T23:13:51.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:51 smithi040 ceph-mon[27984]: pgmap v202: 1 pgs: 1 peering; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-21T23:13:51.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:51 smithi040 ceph-mon[27984]: osdmap e63: 9 total, 9 up, 9 in 2024-04-21T23:13:51.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:51 smithi174 ceph-mon[31409]: pgmap v202: 1 pgs: 1 peering; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-21T23:13:51.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:51 smithi174 ceph-mon[31409]: osdmap e63: 9 total, 9 up, 9 in 2024-04-21T23:13:51.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:51 smithi111 ceph-mon[31085]: pgmap v202: 1 pgs: 1 peering; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-21T23:13:51.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:51 smithi111 ceph-mon[31085]: osdmap e63: 9 total, 9 up, 9 in 2024-04-21T23:13:52.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:52 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/2433974569' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9a6b6a56-6ae5-49fe-9e5b-cb17d686e0dd"}]: dispatch 2024-04-21T23:13:52.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:52 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/2433974569' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9a6b6a56-6ae5-49fe-9e5b-cb17d686e0dd"}]': finished 2024-04-21T23:13:52.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:52 smithi040 ceph-mon[27984]: osdmap e64: 10 total, 9 up, 10 in 2024-04-21T23:13:52.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:52 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T23:13:52.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:52 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/2433974569' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9a6b6a56-6ae5-49fe-9e5b-cb17d686e0dd"}]: dispatch 2024-04-21T23:13:52.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:52 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/2433974569' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9a6b6a56-6ae5-49fe-9e5b-cb17d686e0dd"}]': finished 2024-04-21T23:13:52.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:52 smithi174 ceph-mon[31409]: osdmap e64: 10 total, 9 up, 10 in 2024-04-21T23:13:52.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:52 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T23:13:52.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:52 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/2433974569' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9a6b6a56-6ae5-49fe-9e5b-cb17d686e0dd"}]: dispatch 2024-04-21T23:13:52.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:52 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/2433974569' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9a6b6a56-6ae5-49fe-9e5b-cb17d686e0dd"}]': finished 2024-04-21T23:13:52.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:52 smithi111 ceph-mon[31085]: osdmap e64: 10 total, 9 up, 10 in 2024-04-21T23:13:52.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:52 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T23:13:53.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:53 smithi040 ceph-mon[27984]: pgmap v205: 1 pgs: 1 peering; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-21T23:13:53.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:53 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/3540178659' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:13:53.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:53 smithi174 ceph-mon[31409]: pgmap v205: 1 pgs: 1 peering; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-21T23:13:53.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:53 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/3540178659' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:13:53.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:53 smithi111 ceph-mon[31085]: pgmap v205: 1 pgs: 1 peering; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-21T23:13:53.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:53 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/3540178659' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:13:55.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:55 smithi040 ceph-mon[27984]: pgmap v206: 1 pgs: 1 peering; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-04-21T23:13:55.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:55 smithi174 ceph-mon[31409]: pgmap v206: 1 pgs: 1 peering; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-04-21T23:13:55.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:55 smithi111 ceph-mon[31085]: pgmap v206: 1 pgs: 1 peering; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-04-21T23:13:57.604 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:57 smithi040 ceph-mon[27984]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-04-21T23:13:57.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:57 smithi174 ceph-mon[31409]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-04-21T23:13:57.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:57 smithi111 ceph-mon[31085]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-04-21T23:13:59.842 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:13:59 smithi040 ceph-mon[27984]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-04-21T23:13:59.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:13:59 smithi174 ceph-mon[31409]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-04-21T23:13:59.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:13:59 smithi111 ceph-mon[31085]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-04-21T23:14:00.762 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:00 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-21T23:14:00.762 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:00 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:14:00.762 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:00 smithi040 ceph-mon[27984]: Deploying daemon osd.9 on smithi040 2024-04-21T23:14:00.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:00 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-21T23:14:00.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:00 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:14:00.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:00 smithi174 ceph-mon[31409]: Deploying daemon osd.9 on smithi040 2024-04-21T23:14:00.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:00 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-21T23:14:00.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:00 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:14:00.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:00 smithi111 ceph-mon[31085]: Deploying daemon osd.9 on smithi040 2024-04-21T23:14:01.819 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:01 smithi040 ceph-mon[27984]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-04-21T23:14:01.947 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:01 smithi111 ceph-mon[31085]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-04-21T23:14:01.948 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:01 smithi174 ceph-mon[31409]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-04-21T23:14:03.891 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:03 smithi040 ceph-mon[27984]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-04-21T23:14:03.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:03 smithi174 ceph-mon[31409]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-04-21T23:14:03.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:03 smithi111 ceph-mon[31085]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-04-21T23:14:05.848 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:05 smithi040 ceph-mon[27984]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-04-21T23:14:05.937 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:05 smithi174 ceph-mon[31409]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-04-21T23:14:05.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:05 smithi111 ceph-mon[31085]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-04-21T23:14:07.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:06 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:14:07.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:06 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:14:07.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:06 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:14:07.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:06 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:14:07.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:06 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:14:07.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:06 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:14:07.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:06 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:14:07.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:06 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:14:07.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:06 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:14:08.260 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:07 smithi040 ceph-mon[27984]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-04-21T23:14:08.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:07 smithi174 ceph-mon[31409]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-04-21T23:14:08.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:07 smithi111 ceph-mon[31085]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-04-21T23:14:10.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:14:10.070+0000 7fd4367fc700 1 -- 172.21.15.40:0/435210413 <== mgr.14148 v2:172.21.15.40:6800/2500760788 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fd44002dfa0 con 0x7fd42c0a2c20 2024-04-21T23:14:10.070 INFO:teuthology.orchestra.run.smithi040.stdout:Created osd(s) 9 on host 'smithi040' 2024-04-21T23:14:10.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:14:10.073+0000 7fd447a29700 1 -- 172.21.15.40:0/435210413 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fd42c0a2c20 msgr2=0x7fd42c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:10.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:14:10.073+0000 7fd447a29700 1 --2- 172.21.15.40:0/435210413 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fd42c0a2c20 0x7fd42c0a50e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd430005d10 tx=0x7fd430009310 comp rx=0 tx=0).stop 2024-04-21T23:14:10.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:14:10.073+0000 7fd447a29700 1 -- 172.21.15.40:0/435210413 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4401011a0 msgr2=0x7fd44006b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:10.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:14:10.073+0000 7fd447a29700 1 --2- 172.21.15.40:0/435210413 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4401011a0 0x7fd44006b540 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd43c000c00 tx=0x7fd43c010f80 comp rx=0 tx=0).stop 2024-04-21T23:14:10.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:14:10.073+0000 7fd447a29700 1 -- 172.21.15.40:0/435210413 shutdown_connections 2024-04-21T23:14:10.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:14:10.073+0000 7fd447a29700 1 --2- 172.21.15.40:0/435210413 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd440102d40 0x7fd44019f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:10.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:14:10.073+0000 7fd447a29700 1 --2- 172.21.15.40:0/435210413 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fd42c0a2c20 0x7fd42c0a50e0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:10.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:14:10.073+0000 7fd447a29700 1 --2- 172.21.15.40:0/435210413 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4401011a0 0x7fd44006b540 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:10.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:14:10.073+0000 7fd447a29700 1 --2- 172.21.15.40:0/435210413 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4401023a0 0x7fd44019d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:10.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:14:10.073+0000 7fd447a29700 1 -- 172.21.15.40:0/435210413 >> 172.21.15.40:0/435210413 conn(0x7fd4400fc760 msgr2=0x7fd4401057c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:10.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:14:10.073+0000 7fd447a29700 1 -- 172.21.15.40:0/435210413 shutdown_connections 2024-04-21T23:14:10.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:14:10.073+0000 7fd447a29700 1 -- 172.21.15.40:0/435210413 wait complete. 2024-04-21T23:14:10.426 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:10 smithi040 ceph-mon[27984]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-04-21T23:14:10.427 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:10 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:14:10.427 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:10 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:14:10.427 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:10 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:14:10.427 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:10 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:14:10.427 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:10 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:14:10.427 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:10 smithi040 ceph-mon[27984]: from='osd.9 [v2:172.21.15.40:6826/1649446531,v1:172.21.15.40:6827/1649446531]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-21T23:14:10.427 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:10 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:14:10.427 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:10 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:14:10.427 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:10 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:14:10.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:10 smithi174 ceph-mon[31409]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-04-21T23:14:10.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:10 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:14:10.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:10 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:14:10.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:10 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:14:10.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:10 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:14:10.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:10 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:14:10.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:10 smithi174 ceph-mon[31409]: from='osd.9 [v2:172.21.15.40:6826/1649446531,v1:172.21.15.40:6827/1649446531]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-21T23:14:10.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:10 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:14:10.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:10 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:14:10.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:10 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:14:10.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:10 smithi111 ceph-mon[31085]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-04-21T23:14:10.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:10 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:14:10.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:10 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:14:10.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:10 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:14:10.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:10 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:14:10.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:10 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:14:10.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:10 smithi111 ceph-mon[31085]: from='osd.9 [v2:172.21.15.40:6826/1649446531,v1:172.21.15.40:6827/1649446531]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-21T23:14:10.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:10 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:14:10.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:10 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:14:10.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:10 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:14:10.764 DEBUG:teuthology.orchestra.run.smithi040:osd.9> sudo journalctl -f -n 0 -u ceph-d814e4dc-0033-11ef-bc93-c7b262605968@osd.9.service 2024-04-21T23:14:10.766 INFO:tasks.cephadm:Deploying osd.10 on smithi111 with /dev/vg_nvme/lv_1... 2024-04-21T23:14:10.767 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-21T23:14:11.011 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.b/config 2024-04-21T23:14:11.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:11 smithi040 ceph-mon[27984]: from='osd.9 [v2:172.21.15.40:6826/1649446531,v1:172.21.15.40:6827/1649446531]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-21T23:14:11.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:11 smithi040 ceph-mon[27984]: osdmap e65: 10 total, 9 up, 10 in 2024-04-21T23:14:11.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:11 smithi040 ceph-mon[27984]: from='osd.9 [v2:172.21.15.40:6826/1649446531,v1:172.21.15.40:6827/1649446531]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-21T23:14:11.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:11 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T23:14:11.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:11 smithi111 ceph-mon[31085]: from='osd.9 [v2:172.21.15.40:6826/1649446531,v1:172.21.15.40:6827/1649446531]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-21T23:14:11.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:11 smithi111 ceph-mon[31085]: osdmap e65: 10 total, 9 up, 10 in 2024-04-21T23:14:11.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:11 smithi111 ceph-mon[31085]: from='osd.9 [v2:172.21.15.40:6826/1649446531,v1:172.21.15.40:6827/1649446531]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-21T23:14:11.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:11 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T23:14:11.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:11 smithi174 ceph-mon[31409]: from='osd.9 [v2:172.21.15.40:6826/1649446531,v1:172.21.15.40:6827/1649446531]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-21T23:14:11.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:11 smithi174 ceph-mon[31409]: osdmap e65: 10 total, 9 up, 10 in 2024-04-21T23:14:11.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:11 smithi174 ceph-mon[31409]: from='osd.9 [v2:172.21.15.40:6826/1649446531,v1:172.21.15.40:6827/1649446531]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-21T23:14:11.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:11 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T23:14:12.179 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:14:11 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:14:11.855+0000 7fef20e2b700 -1 osd.9 0 waiting for initial osdmap 2024-04-21T23:14:12.179 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:14:11 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:14:11.867+0000 7fef18c16700 -1 osd.9 66 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T23:14:12.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:12 smithi040 ceph-mon[27984]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-04-21T23:14:12.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:12 smithi040 ceph-mon[27984]: from='osd.9 [v2:172.21.15.40:6826/1649446531,v1:172.21.15.40:6827/1649446531]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-04-21T23:14:12.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:12 smithi040 ceph-mon[27984]: osdmap e66: 10 total, 9 up, 10 in 2024-04-21T23:14:12.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:12 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T23:14:12.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:12 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T23:14:12.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:12 smithi040 ceph-mon[27984]: from='osd.9 [v2:172.21.15.40:6826/1649446531,v1:172.21.15.40:6827/1649446531]' entity='osd.9' 2024-04-21T23:14:12.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:12 smithi111 ceph-mon[31085]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-04-21T23:14:12.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:12 smithi111 ceph-mon[31085]: from='osd.9 [v2:172.21.15.40:6826/1649446531,v1:172.21.15.40:6827/1649446531]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-04-21T23:14:12.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:12 smithi111 ceph-mon[31085]: osdmap e66: 10 total, 9 up, 10 in 2024-04-21T23:14:12.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:12 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T23:14:12.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:12 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T23:14:12.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:12 smithi111 ceph-mon[31085]: from='osd.9 [v2:172.21.15.40:6826/1649446531,v1:172.21.15.40:6827/1649446531]' entity='osd.9' 2024-04-21T23:14:12.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:12 smithi174 ceph-mon[31409]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-04-21T23:14:12.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:12 smithi174 ceph-mon[31409]: from='osd.9 [v2:172.21.15.40:6826/1649446531,v1:172.21.15.40:6827/1649446531]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-04-21T23:14:12.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:12 smithi174 ceph-mon[31409]: osdmap e66: 10 total, 9 up, 10 in 2024-04-21T23:14:12.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:12 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T23:14:12.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:12 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T23:14:12.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:12 smithi174 ceph-mon[31409]: from='osd.9 [v2:172.21.15.40:6826/1649446531,v1:172.21.15.40:6827/1649446531]' entity='osd.9' 2024-04-21T23:14:13.033 INFO:teuthology.orchestra.run.smithi111.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-21T23:14:13.033 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-21T23:14:13.033 INFO:teuthology.orchestra.run.smithi111.stderr: stderr: 10+0 records in 2024-04-21T23:14:13.033 INFO:teuthology.orchestra.run.smithi111.stderr:10+0 records out 2024-04-21T23:14:13.033 INFO:teuthology.orchestra.run.smithi111.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0111165 s, 943 MB/s 2024-04-21T23:14:13.033 INFO:teuthology.orchestra.run.smithi111.stderr: stderr: 2024-04-21T23:14:13.033 INFO:teuthology.orchestra.run.smithi111.stderr:--> Zapping successful for: 2024-04-21T23:14:13.496 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:13 smithi111 ceph-mon[31085]: purged_snaps scrub starts 2024-04-21T23:14:13.496 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:13 smithi111 ceph-mon[31085]: purged_snaps scrub ok 2024-04-21T23:14:13.496 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:13 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T23:14:13.496 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:13 smithi111 ceph-mon[31085]: osd.9 [v2:172.21.15.40:6826/1649446531,v1:172.21.15.40:6827/1649446531] boot 2024-04-21T23:14:13.496 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:13 smithi111 ceph-mon[31085]: osdmap e67: 10 total, 10 up, 10 in 2024-04-21T23:14:13.497 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:13 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T23:14:13.547 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:13 smithi040 ceph-mon[27984]: purged_snaps scrub starts 2024-04-21T23:14:13.547 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:13 smithi040 ceph-mon[27984]: purged_snaps scrub ok 2024-04-21T23:14:13.547 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:13 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T23:14:13.547 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:13 smithi040 ceph-mon[27984]: osd.9 [v2:172.21.15.40:6826/1649446531,v1:172.21.15.40:6827/1649446531] boot 2024-04-21T23:14:13.548 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:13 smithi040 ceph-mon[27984]: osdmap e67: 10 total, 10 up, 10 in 2024-04-21T23:14:13.548 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:13 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T23:14:13.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:13 smithi174 ceph-mon[31409]: purged_snaps scrub starts 2024-04-21T23:14:13.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:13 smithi174 ceph-mon[31409]: purged_snaps scrub ok 2024-04-21T23:14:13.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:13 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T23:14:13.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:13 smithi174 ceph-mon[31409]: osd.9 [v2:172.21.15.40:6826/1649446531,v1:172.21.15.40:6827/1649446531] boot 2024-04-21T23:14:13.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:13 smithi174 ceph-mon[31409]: osdmap e67: 10 total, 10 up, 10 in 2024-04-21T23:14:13.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:13 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T23:14:13.762 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi111:vg_nvme/lv_1 2024-04-21T23:14:14.004 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.b/config 2024-04-21T23:14:14.397 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:14 smithi111 ceph-mon[31085]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-04-21T23:14:14.397 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:14 smithi111 ceph-mon[31085]: osdmap e68: 10 total, 10 up, 10 in 2024-04-21T23:14:14.397 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:14 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:14:14.397 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:14 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:14:14.397 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:14 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:14:14.397 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:14 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:14:14.397 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:14 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:14:14.398 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:14 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:14:14.398 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:14 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:14:14.398 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:14 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:14:14.398 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:14 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:14:14.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:14 smithi040 ceph-mon[27984]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-04-21T23:14:14.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:14 smithi040 ceph-mon[27984]: osdmap e68: 10 total, 10 up, 10 in 2024-04-21T23:14:14.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:14 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:14:14.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:14 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:14:14.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:14 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:14:14.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:14 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:14:14.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:14 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:14:14.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:14 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:14:14.433 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:14 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:14:14.433 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:14 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:14:14.433 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:14 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:14:14.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:14 smithi174 ceph-mon[31409]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-04-21T23:14:14.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:14 smithi174 ceph-mon[31409]: osdmap e68: 10 total, 10 up, 10 in 2024-04-21T23:14:14.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:14 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:14:14.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:14 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:14:14.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:14 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:14:14.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:14 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:14:14.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:14 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:14:14.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:14 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:14:14.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:14 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:14:14.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:14 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:14:14.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:14 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:14:15.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:15 smithi111 ceph-mon[31085]: Detected new or changed devices on smithi040 2024-04-21T23:14:15.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:15 smithi111 ceph-mon[31085]: Adjusting osd_memory_target on smithi040 to 4273M 2024-04-21T23:14:15.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:15 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:14:15.674 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:14:15.675+0000 7fc08f878700 1 -- 172.21.15.111:0/1282189294 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc088102d50 msgr2=0x7fc088109600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:15.675 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:14:15.675+0000 7fc08f878700 1 --2- 172.21.15.111:0/1282189294 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc088102d50 0x7fc088109600 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fc08000b0d0 tx=0x7fc08000b3e0 comp rx=0 tx=0).stop 2024-04-21T23:14:15.675 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:14:15.675+0000 7fc08f878700 1 -- 172.21.15.111:0/1282189294 shutdown_connections 2024-04-21T23:14:15.675 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:14:15.675+0000 7fc08f878700 1 --2- 172.21.15.111:0/1282189294 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc0881011b0 0x7fc0881015d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:15.675 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:14:15.675+0000 7fc08f878700 1 --2- 172.21.15.111:0/1282189294 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc088102d50 0x7fc088109600 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:15.675 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:14:15.675+0000 7fc08f878700 1 --2- 172.21.15.111:0/1282189294 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0881023b0 0x7fc088102810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:15.675 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:14:15.675+0000 7fc08f878700 1 -- 172.21.15.111:0/1282189294 >> 172.21.15.111:0/1282189294 conn(0x7fc0880fc790 msgr2=0x7fc0880febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:15.675 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:14:15.675+0000 7fc08f878700 1 -- 172.21.15.111:0/1282189294 shutdown_connections 2024-04-21T23:14:15.675 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:14:15.675+0000 7fc08f878700 1 -- 172.21.15.111:0/1282189294 wait complete. 2024-04-21T23:14:15.677 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:14:15.676+0000 7fc08f878700 1 Processor -- start 2024-04-21T23:14:15.677 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:14:15.676+0000 7fc08f878700 1 -- start start 2024-04-21T23:14:15.677 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:14:15.676+0000 7fc08f878700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0881011b0 0x7fc0881983c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:15.678 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:14:15.676+0000 7fc08f878700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc0881023b0 0x7fc088198900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:15.678 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:14:15.676+0000 7fc08f878700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc088102d50 0x7fc08819f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:15.678 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:14:15.676+0000 7fc08f878700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc08810cda0 con 0x7fc0881011b0 2024-04-21T23:14:15.678 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:14:15.676+0000 7fc08f878700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc08810cc20 con 0x7fc088102d50 2024-04-21T23:14:15.678 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:14:15.676+0000 7fc08f878700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc08810cf20 con 0x7fc0881023b0 2024-04-21T23:14:15.678 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:14:15.676+0000 7fc08de15700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc088102d50 0x7fc08819f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:15.678 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:14:15.676+0000 7fc08de15700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc088102d50 0x7fc08819f930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55194/0 (socket says 172.21.15.111:55194) 2024-04-21T23:14:15.678 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:14:15.676+0000 7fc08de15700 1 -- 172.21.15.111:0/336003213 learned_addr learned my addr 172.21.15.111:0/336003213 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-04-21T23:14:15.678 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:14:15.676+0000 7fc08ce13700 1 --2- 172.21.15.111:0/336003213 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc0881023b0 0x7fc088198900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:15.678 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:14:15.676+0000 7fc08d614700 1 --2- 172.21.15.111:0/336003213 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0881011b0 0x7fc0881983c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:15.679 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:14:15.677+0000 7fc08de15700 1 -- 172.21.15.111:0/336003213 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc0881023b0 msgr2=0x7fc088198900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:15.679 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:14:15.677+0000 7fc08de15700 1 --2- 172.21.15.111:0/336003213 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc0881023b0 0x7fc088198900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:15.679 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:14:15.677+0000 7fc08de15700 1 -- 172.21.15.111:0/336003213 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0881011b0 msgr2=0x7fc0881983c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:15.679 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:14:15.677+0000 7fc08de15700 1 --2- 172.21.15.111:0/336003213 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0881011b0 0x7fc0881983c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:15.679 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:14:15.677+0000 7fc08de15700 1 -- 172.21.15.111:0/336003213 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc08819ffd0 con 0x7fc088102d50 2024-04-21T23:14:15.679 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:14:15.677+0000 7fc08de15700 1 --2- 172.21.15.111:0/336003213 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc088102d50 0x7fc08819f930 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc080006070 tx=0x7fc080007dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:15.679 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:14:15.677+0000 7fc07e7fc700 1 -- 172.21.15.111:0/336003213 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc080021020 con 0x7fc088102d50 2024-04-21T23:14:15.679 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:14:15.677+0000 7fc07e7fc700 1 -- 172.21.15.111:0/336003213 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc08001d950 con 0x7fc088102d50 2024-04-21T23:14:15.679 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:14:15.677+0000 7fc08f878700 1 -- 172.21.15.111:0/336003213 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0881a0260 con 0x7fc088102d50 2024-04-21T23:14:15.679 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:14:15.677+0000 7fc07e7fc700 1 -- 172.21.15.111:0/336003213 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc080005510 con 0x7fc088102d50 2024-04-21T23:14:15.679 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:14:15.677+0000 7fc08f878700 1 -- 172.21.15.111:0/336003213 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0881a07a0 con 0x7fc088102d50 2024-04-21T23:14:15.679 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:14:15.678+0000 7fc08f878700 1 -- 172.21.15.111:0/336003213 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc088068f20 con 0x7fc088102d50 2024-04-21T23:14:15.680 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:14:15.679+0000 7fc07e7fc700 1 -- 172.21.15.111:0/336003213 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fc080024020 con 0x7fc088102d50 2024-04-21T23:14:15.680 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:14:15.680+0000 7fc07e7fc700 1 --2- 172.21.15.111:0/336003213 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fc0740a2c20 0x7fc0740a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:15.680 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:14:15.680+0000 7fc07e7fc700 1 -- 172.21.15.111:0/336003213 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 6261+0+0 (secure 0 0 0) 0x7fc08000f020 con 0x7fc088102d50 2024-04-21T23:14:15.680 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:14:15.680+0000 7fc08d614700 1 --2- 172.21.15.111:0/336003213 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fc0740a2c20 0x7fc0740a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:15.680 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:14:15.680+0000 7fc08d614700 1 --2- 172.21.15.111:0/336003213 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fc0740a2c20 0x7fc0740a50e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fc078005950 tx=0x7fc0780058e0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:15.681 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:15 smithi040 ceph-mon[27984]: Detected new or changed devices on smithi040 2024-04-21T23:14:15.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:15 smithi040 ceph-mon[27984]: Adjusting osd_memory_target on smithi040 to 4273M 2024-04-21T23:14:15.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:15 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:14:15.682 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:14:15.683+0000 7fc07e7fc700 1 -- 172.21.15.111:0/336003213 <== mon.2 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fc08009d640 con 0x7fc088102d50 2024-04-21T23:14:15.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:15 smithi174 ceph-mon[31409]: Detected new or changed devices on smithi040 2024-04-21T23:14:15.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:15 smithi174 ceph-mon[31409]: Adjusting osd_memory_target on smithi040 to 4273M 2024-04-21T23:14:15.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:15 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:14:15.886 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:14:15.887+0000 7fc08f878700 1 -- 172.21.15.111:0/336003213 --> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fc0881a0ac0 con 0x7fc0740a2c20 2024-04-21T23:14:16.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:16 smithi174 ceph-mon[31409]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 284 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:14:16.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:16 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:14:16.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:16 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:14:16.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:16 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:14:16.499 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:16 smithi111 ceph-mon[31085]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 284 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:14:16.499 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:16 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:14:16.499 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:16 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:14:16.499 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:16 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:14:16.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:16 smithi040 ceph-mon[27984]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 284 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:14:16.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:16 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:14:16.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:16 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:14:16.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:16 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:14:17.638 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:17 smithi111 ceph-mon[31085]: from='client.24347 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:14:17.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:17 smithi040 ceph-mon[27984]: from='client.24347 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:14:17.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:17 smithi174 ceph-mon[31409]: from='client.24347 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:14:18.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:18 smithi040 ceph-mon[27984]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 284 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:14:18.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:18 smithi174 ceph-mon[31409]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 284 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:14:18.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:18 smithi111 ceph-mon[31085]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 284 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:14:19.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:19 smithi040 ceph-mon[27984]: from='client.? 172.21.15.111:0/34260666' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a70038ba-e2ad-4170-85cc-ea0470a2edef"}]: dispatch 2024-04-21T23:14:19.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:19 smithi040 ceph-mon[27984]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a70038ba-e2ad-4170-85cc-ea0470a2edef"}]: dispatch 2024-04-21T23:14:19.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:19 smithi040 ceph-mon[27984]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a70038ba-e2ad-4170-85cc-ea0470a2edef"}]': finished 2024-04-21T23:14:19.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:19 smithi040 ceph-mon[27984]: osdmap e69: 11 total, 10 up, 11 in 2024-04-21T23:14:19.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:19 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T23:14:19.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:19 smithi111 ceph-mon[31085]: from='client.? 172.21.15.111:0/34260666' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a70038ba-e2ad-4170-85cc-ea0470a2edef"}]: dispatch 2024-04-21T23:14:19.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:19 smithi111 ceph-mon[31085]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a70038ba-e2ad-4170-85cc-ea0470a2edef"}]: dispatch 2024-04-21T23:14:19.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:19 smithi111 ceph-mon[31085]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a70038ba-e2ad-4170-85cc-ea0470a2edef"}]': finished 2024-04-21T23:14:19.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:19 smithi111 ceph-mon[31085]: osdmap e69: 11 total, 10 up, 11 in 2024-04-21T23:14:19.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:19 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T23:14:19.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:19 smithi174 ceph-mon[31409]: from='client.? 172.21.15.111:0/34260666' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a70038ba-e2ad-4170-85cc-ea0470a2edef"}]: dispatch 2024-04-21T23:14:19.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:19 smithi174 ceph-mon[31409]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a70038ba-e2ad-4170-85cc-ea0470a2edef"}]: dispatch 2024-04-21T23:14:19.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:19 smithi174 ceph-mon[31409]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a70038ba-e2ad-4170-85cc-ea0470a2edef"}]': finished 2024-04-21T23:14:19.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:19 smithi174 ceph-mon[31409]: osdmap e69: 11 total, 10 up, 11 in 2024-04-21T23:14:19.695 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:19 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T23:14:20.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:20 smithi040 ceph-mon[27984]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 284 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:14:20.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:20 smithi040 ceph-mon[27984]: from='client.? 172.21.15.111:0/2701378125' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:14:20.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:20 smithi174 ceph-mon[31409]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 284 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:14:20.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:20 smithi174 ceph-mon[31409]: from='client.? 172.21.15.111:0/2701378125' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:14:20.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:20 smithi111 ceph-mon[31085]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 284 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:14:20.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:20 smithi111 ceph-mon[31085]: from='client.? 172.21.15.111:0/2701378125' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:14:22.480 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:22 smithi111 ceph-mon[31085]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 284 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:14:22.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:22 smithi040 ceph-mon[27984]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 284 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:14:22.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:22 smithi174 ceph-mon[31409]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 284 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:14:24.347 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:24 smithi111 ceph-mon[31085]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:14:24.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:24 smithi040 ceph-mon[27984]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:14:24.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:24 smithi174 ceph-mon[31409]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:14:26.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:26 smithi040 ceph-mon[27984]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:14:26.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:26 smithi174 ceph-mon[31409]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:14:26.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:26 smithi111 ceph-mon[31085]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:14:27.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:27 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-21T23:14:27.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:27 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:14:27.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:27 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-21T23:14:27.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:27 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:14:27.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:27 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-21T23:14:27.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:27 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:14:28.264 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:28 smithi111 ceph-mon[31085]: Deploying daemon osd.10 on smithi111 2024-04-21T23:14:28.264 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:28 smithi111 ceph-mon[31085]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:14:28.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:28 smithi040 ceph-mon[27984]: Deploying daemon osd.10 on smithi111 2024-04-21T23:14:28.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:28 smithi040 ceph-mon[27984]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:14:28.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:28 smithi174 ceph-mon[31409]: Deploying daemon osd.10 on smithi111 2024-04-21T23:14:28.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:28 smithi174 ceph-mon[31409]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:14:30.435 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:30 smithi111 ceph-mon[31085]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:14:30.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:30 smithi040 ceph-mon[27984]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:14:30.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:30 smithi174 ceph-mon[31409]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:14:32.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:32 smithi111 ceph-mon[31085]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:14:32.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:32 smithi040 ceph-mon[27984]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:14:32.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:32 smithi174 ceph-mon[31409]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:14:33.478 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:33 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:14:33.478 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:33 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:14:33.478 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:33 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:14:33.478 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:33 smithi111 ceph-mon[31085]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:14:33.595 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:33 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:14:33.595 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:33 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:14:33.595 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:33 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:14:33.595 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:33 smithi174 ceph-mon[31409]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:14:33.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:33 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:14:33.683 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:33 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:14:33.683 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:33 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:14:33.683 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:33 smithi040 ceph-mon[27984]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:14:35.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:35 smithi040 ceph-mon[27984]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:14:35.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:35 smithi174 ceph-mon[31409]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:14:35.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:35 smithi111 ceph-mon[31085]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:14:36.312 INFO:teuthology.orchestra.run.smithi111.stdout:Created osd(s) 10 on host 'smithi111' 2024-04-21T23:14:36.313 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:14:36.313+0000 7fc07e7fc700 1 -- 172.21.15.111:0/336003213 <== mgr.14148 v2:172.21.15.40:6800/2500760788 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7fc0881a0ac0 con 0x7fc0740a2c20 2024-04-21T23:14:36.315 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:14:36.315+0000 7fc08f878700 1 -- 172.21.15.111:0/336003213 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fc0740a2c20 msgr2=0x7fc0740a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:36.315 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:14:36.315+0000 7fc08f878700 1 --2- 172.21.15.111:0/336003213 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fc0740a2c20 0x7fc0740a50e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fc078005950 tx=0x7fc0780058e0 comp rx=0 tx=0).stop 2024-04-21T23:14:36.315 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:14:36.315+0000 7fc08f878700 1 -- 172.21.15.111:0/336003213 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc088102d50 msgr2=0x7fc08819f930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:36.315 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:14:36.315+0000 7fc08f878700 1 --2- 172.21.15.111:0/336003213 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc088102d50 0x7fc08819f930 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc080006070 tx=0x7fc080007dc0 comp rx=0 tx=0).stop 2024-04-21T23:14:36.315 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:14:36.315+0000 7fc08f878700 1 -- 172.21.15.111:0/336003213 shutdown_connections 2024-04-21T23:14:36.315 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:14:36.315+0000 7fc08f878700 1 --2- 172.21.15.111:0/336003213 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fc0740a2c20 0x7fc0740a50e0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:36.315 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:14:36.315+0000 7fc08f878700 1 --2- 172.21.15.111:0/336003213 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc088102d50 0x7fc08819f930 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:36.315 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:14:36.315+0000 7fc08f878700 1 --2- 172.21.15.111:0/336003213 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0881011b0 0x7fc0881983c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:36.315 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:14:36.316+0000 7fc08f878700 1 --2- 172.21.15.111:0/336003213 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc0881023b0 0x7fc088198900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:36.315 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:14:36.316+0000 7fc08f878700 1 -- 172.21.15.111:0/336003213 >> 172.21.15.111:0/336003213 conn(0x7fc0880fc790 msgr2=0x7fc0880fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:36.315 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:14:36.316+0000 7fc08f878700 1 -- 172.21.15.111:0/336003213 shutdown_connections 2024-04-21T23:14:36.316 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:14:36.316+0000 7fc08f878700 1 -- 172.21.15.111:0/336003213 wait complete. 2024-04-21T23:14:36.635 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:36 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:14:36.635 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:36 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:14:36.635 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:36 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:14:36.635 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:36 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:14:36.635 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:36 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:14:36.635 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:36 smithi111 ceph-mon[31085]: from='osd.10 [v2:172.21.15.111:6824/1227538935,v1:172.21.15.111:6825/1227538935]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-21T23:14:36.635 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:36 smithi111 ceph-mon[31085]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-21T23:14:36.635 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:36 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:14:36.635 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:36 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:14:36.635 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:36 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:14:36.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:36 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:14:36.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:36 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:14:36.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:36 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:14:36.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:36 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:14:36.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:36 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:14:36.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:36 smithi174 ceph-mon[31409]: from='osd.10 [v2:172.21.15.111:6824/1227538935,v1:172.21.15.111:6825/1227538935]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-21T23:14:36.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:36 smithi174 ceph-mon[31409]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-21T23:14:36.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:36 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:14:36.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:36 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:14:36.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:36 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:14:36.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:36 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:14:36.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:36 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:14:36.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:36 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:14:36.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:36 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:14:36.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:36 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:14:36.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:36 smithi040 ceph-mon[27984]: from='osd.10 [v2:172.21.15.111:6824/1227538935,v1:172.21.15.111:6825/1227538935]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-21T23:14:36.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:36 smithi040 ceph-mon[27984]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-21T23:14:36.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:36 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:14:36.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:36 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:14:36.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:36 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:14:36.968 DEBUG:teuthology.orchestra.run.smithi111:osd.10> sudo journalctl -f -n 0 -u ceph-d814e4dc-0033-11ef-bc93-c7b262605968@osd.10.service 2024-04-21T23:14:36.971 INFO:tasks.cephadm:Deploying osd.11 on smithi174 with /dev/vg_nvme/lv_1... 2024-04-21T23:14:36.971 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-21T23:14:37.217 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.c/config 2024-04-21T23:14:37.523 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:37 smithi111 ceph-mon[31085]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:14:37.523 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:37 smithi111 ceph-mon[31085]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-21T23:14:37.523 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:37 smithi111 ceph-mon[31085]: from='osd.10 [v2:172.21.15.111:6824/1227538935,v1:172.21.15.111:6825/1227538935]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-21T23:14:37.523 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:37 smithi111 ceph-mon[31085]: osdmap e70: 11 total, 10 up, 11 in 2024-04-21T23:14:37.523 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:37 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T23:14:37.523 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:37 smithi111 ceph-mon[31085]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-21T23:14:37.595 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:37 smithi174 ceph-mon[31409]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:14:37.596 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:37 smithi174 ceph-mon[31409]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-21T23:14:37.596 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:37 smithi174 ceph-mon[31409]: from='osd.10 [v2:172.21.15.111:6824/1227538935,v1:172.21.15.111:6825/1227538935]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-21T23:14:37.596 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:37 smithi174 ceph-mon[31409]: osdmap e70: 11 total, 10 up, 11 in 2024-04-21T23:14:37.596 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:37 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T23:14:37.596 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:37 smithi174 ceph-mon[31409]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-21T23:14:37.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:37 smithi040 ceph-mon[27984]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:14:37.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:37 smithi040 ceph-mon[27984]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-21T23:14:37.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:37 smithi040 ceph-mon[27984]: from='osd.10 [v2:172.21.15.111:6824/1227538935,v1:172.21.15.111:6825/1227538935]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-21T23:14:37.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:37 smithi040 ceph-mon[27984]: osdmap e70: 11 total, 10 up, 11 in 2024-04-21T23:14:37.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:37 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T23:14:37.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:37 smithi040 ceph-mon[27984]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-21T23:14:38.328 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:14:38 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:14:38.140+0000 7f981f24a700 -1 osd.10 0 waiting for initial osdmap 2024-04-21T23:14:38.328 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:14:38 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:14:38.151+0000 7f9817035700 -1 osd.10 72 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T23:14:38.622 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:38 smithi111 ceph-mon[31085]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-04-21T23:14:38.623 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:38 smithi111 ceph-mon[31085]: osdmap e71: 11 total, 10 up, 11 in 2024-04-21T23:14:38.623 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:38 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T23:14:38.623 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:38 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T23:14:38.623 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:38 smithi111 ceph-mon[31085]: osdmap e72: 11 total, 10 up, 11 in 2024-04-21T23:14:38.623 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:38 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T23:14:38.623 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:38 smithi111 ceph-mon[31085]: from='osd.10 ' entity='osd.10' 2024-04-21T23:14:38.624 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:38 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T23:14:38.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:38 smithi174 ceph-mon[31409]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-04-21T23:14:38.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:38 smithi174 ceph-mon[31409]: osdmap e71: 11 total, 10 up, 11 in 2024-04-21T23:14:38.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:38 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T23:14:38.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:38 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T23:14:38.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:38 smithi174 ceph-mon[31409]: osdmap e72: 11 total, 10 up, 11 in 2024-04-21T23:14:38.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:38 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T23:14:38.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:38 smithi174 ceph-mon[31409]: from='osd.10 ' entity='osd.10' 2024-04-21T23:14:38.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:38 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T23:14:38.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:38 smithi040 ceph-mon[27984]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-04-21T23:14:38.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:38 smithi040 ceph-mon[27984]: osdmap e71: 11 total, 10 up, 11 in 2024-04-21T23:14:38.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:38 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T23:14:38.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:38 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T23:14:38.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:38 smithi040 ceph-mon[27984]: osdmap e72: 11 total, 10 up, 11 in 2024-04-21T23:14:38.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:38 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T23:14:38.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:38 smithi040 ceph-mon[27984]: from='osd.10 ' entity='osd.10' 2024-04-21T23:14:38.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:38 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T23:14:39.508 INFO:teuthology.orchestra.run.smithi174.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-21T23:14:39.508 INFO:teuthology.orchestra.run.smithi174.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-21T23:14:39.508 INFO:teuthology.orchestra.run.smithi174.stderr: stderr: 10+0 records in 2024-04-21T23:14:39.508 INFO:teuthology.orchestra.run.smithi174.stderr:10+0 records out 2024-04-21T23:14:39.508 INFO:teuthology.orchestra.run.smithi174.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0115611 s, 907 MB/s 2024-04-21T23:14:39.508 INFO:teuthology.orchestra.run.smithi174.stderr:--> Zapping successful for: 2024-04-21T23:14:40.137 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:39 smithi111 ceph-mon[31085]: purged_snaps scrub starts 2024-04-21T23:14:40.137 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:39 smithi111 ceph-mon[31085]: purged_snaps scrub ok 2024-04-21T23:14:40.137 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:39 smithi111 ceph-mon[31085]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:14:40.137 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:39 smithi111 ceph-mon[31085]: osd.10 [v2:172.21.15.111:6824/1227538935,v1:172.21.15.111:6825/1227538935] boot 2024-04-21T23:14:40.137 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:39 smithi111 ceph-mon[31085]: osdmap e73: 11 total, 11 up, 11 in 2024-04-21T23:14:40.137 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:39 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T23:14:40.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:39 smithi040 ceph-mon[27984]: purged_snaps scrub starts 2024-04-21T23:14:40.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:39 smithi040 ceph-mon[27984]: purged_snaps scrub ok 2024-04-21T23:14:40.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:39 smithi040 ceph-mon[27984]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:14:40.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:39 smithi040 ceph-mon[27984]: osd.10 [v2:172.21.15.111:6824/1227538935,v1:172.21.15.111:6825/1227538935] boot 2024-04-21T23:14:40.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:39 smithi040 ceph-mon[27984]: osdmap e73: 11 total, 11 up, 11 in 2024-04-21T23:14:40.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:39 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T23:14:40.191 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:39 smithi174 ceph-mon[31409]: purged_snaps scrub starts 2024-04-21T23:14:40.191 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:39 smithi174 ceph-mon[31409]: purged_snaps scrub ok 2024-04-21T23:14:40.191 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:39 smithi174 ceph-mon[31409]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:14:40.191 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:39 smithi174 ceph-mon[31409]: osd.10 [v2:172.21.15.111:6824/1227538935,v1:172.21.15.111:6825/1227538935] boot 2024-04-21T23:14:40.192 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:39 smithi174 ceph-mon[31409]: osdmap e73: 11 total, 11 up, 11 in 2024-04-21T23:14:40.192 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:39 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T23:14:40.224 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi174:vg_nvme/lv_1 2024-04-21T23:14:40.473 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.c/config 2024-04-21T23:14:41.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:40 smithi040 ceph-mon[27984]: osdmap e74: 11 total, 11 up, 11 in 2024-04-21T23:14:41.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:40 smithi040 ceph-mon[27984]: Detected new or changed devices on smithi111 2024-04-21T23:14:41.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:40 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:14:41.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:40 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:14:41.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:40 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:14:41.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:40 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:14:41.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:40 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:14:41.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:40 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:14:41.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:40 smithi040 ceph-mon[27984]: Adjusting osd_memory_target on smithi111 to 4272M 2024-04-21T23:14:41.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:40 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:14:41.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:40 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:14:41.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:40 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:14:41.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:40 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:14:41.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:40 smithi174 ceph-mon[31409]: osdmap e74: 11 total, 11 up, 11 in 2024-04-21T23:14:41.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:40 smithi174 ceph-mon[31409]: Detected new or changed devices on smithi111 2024-04-21T23:14:41.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:40 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:14:41.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:40 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:14:41.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:40 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:14:41.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:40 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:14:41.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:40 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:14:41.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:40 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:14:41.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:40 smithi174 ceph-mon[31409]: Adjusting osd_memory_target on smithi111 to 4272M 2024-04-21T23:14:41.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:40 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:14:41.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:40 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:14:41.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:40 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:14:41.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:40 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:14:41.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:40 smithi111 ceph-mon[31085]: osdmap e74: 11 total, 11 up, 11 in 2024-04-21T23:14:41.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:40 smithi111 ceph-mon[31085]: Detected new or changed devices on smithi111 2024-04-21T23:14:41.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:40 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:14:41.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:40 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:14:41.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:40 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:14:41.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:40 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:14:41.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:40 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:14:41.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:40 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:14:41.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:40 smithi111 ceph-mon[31085]: Adjusting osd_memory_target on smithi111 to 4272M 2024-04-21T23:14:41.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:40 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:14:41.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:40 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:14:41.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:40 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:14:41.196 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:40 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:14:42.083 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:41 smithi174 ceph-mon[31409]: pgmap v239: 1 pgs: 1 remapped+peering; 577 KiB data, 316 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:14:42.084 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:41 smithi174 ceph-mon[31409]: osdmap e75: 11 total, 11 up, 11 in 2024-04-21T23:14:42.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:41 smithi040 ceph-mon[27984]: pgmap v239: 1 pgs: 1 remapped+peering; 577 KiB data, 316 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:14:42.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:41 smithi040 ceph-mon[27984]: osdmap e75: 11 total, 11 up, 11 in 2024-04-21T23:14:42.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:41 smithi111 ceph-mon[31085]: pgmap v239: 1 pgs: 1 remapped+peering; 577 KiB data, 316 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:14:42.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:41 smithi111 ceph-mon[31085]: osdmap e75: 11 total, 11 up, 11 in 2024-04-21T23:14:42.307 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:14:42.307+0000 7f55f639b700 1 -- 172.21.15.174:0/1765429297 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f55f0102d70 msgr2=0x7f55f0105160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:42.307 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:14:42.307+0000 7f55f639b700 1 --2- 172.21.15.174:0/1765429297 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f55f0102d70 0x7f55f0105160 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f55e4009990 tx=0x7f55e4009ca0 comp rx=0 tx=0).stop 2024-04-21T23:14:42.307 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:14:42.307+0000 7f55f639b700 1 -- 172.21.15.174:0/1765429297 shutdown_connections 2024-04-21T23:14:42.307 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:14:42.307+0000 7f55f639b700 1 --2- 172.21.15.174:0/1765429297 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f55f0102d70 0x7f55f0105160 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:42.307 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:14:42.307+0000 7f55f639b700 1 --2- 172.21.15.174:0/1765429297 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55f0107fd0 0x7f55f010a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:42.307 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:14:42.307+0000 7f55f639b700 1 --2- 172.21.15.174:0/1765429297 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55f01056a0 0x7f55f0107a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:42.307 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:14:42.307+0000 7f55f639b700 1 -- 172.21.15.174:0/1765429297 >> 172.21.15.174:0/1765429297 conn(0x7f55f00fc720 msgr2=0x7f55f00feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:42.307 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:14:42.307+0000 7f55f639b700 1 -- 172.21.15.174:0/1765429297 shutdown_connections 2024-04-21T23:14:42.307 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:14:42.307+0000 7f55f639b700 1 -- 172.21.15.174:0/1765429297 wait complete. 2024-04-21T23:14:42.309 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:14:42.308+0000 7f55f639b700 1 Processor -- start 2024-04-21T23:14:42.309 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:14:42.308+0000 7f55f639b700 1 -- start start 2024-04-21T23:14:42.309 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:14:42.308+0000 7f55f639b700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f55f0102d70 0x7f55f01026d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:42.309 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:14:42.308+0000 7f55f639b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55f01056a0 0x7f55f0100d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:42.310 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:14:42.308+0000 7f55f639b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55f0107fd0 0x7f55f0101260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:42.310 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:14:42.308+0000 7f55f639b700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f55f010ded0 con 0x7f55f0107fd0 2024-04-21T23:14:42.310 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:14:42.308+0000 7f55f639b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f55f010dd50 con 0x7f55f01056a0 2024-04-21T23:14:42.310 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:14:42.308+0000 7f55f639b700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f55f010e050 con 0x7f55f0102d70 2024-04-21T23:14:42.310 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:14:42.308+0000 7f55effff700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f55f0102d70 0x7f55f01026d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:42.310 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:14:42.308+0000 7f55effff700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f55f0102d70 0x7f55f01026d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.174:38398/0 (socket says 172.21.15.174:38398) 2024-04-21T23:14:42.310 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:14:42.308+0000 7f55effff700 1 -- 172.21.15.174:0/3136566220 learned_addr learned my addr 172.21.15.174:0/3136566220 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-04-21T23:14:42.310 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:14:42.308+0000 7f55ef7fe700 1 --2- 172.21.15.174:0/3136566220 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55f01056a0 0x7f55f0100d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:42.310 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:14:42.308+0000 7f55f4938700 1 --2- 172.21.15.174:0/3136566220 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55f0107fd0 0x7f55f0101260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:42.310 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:14:42.308+0000 7f55effff700 1 -- 172.21.15.174:0/3136566220 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55f01056a0 msgr2=0x7f55f0100d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:42.310 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:14:42.309+0000 7f55effff700 1 --2- 172.21.15.174:0/3136566220 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55f01056a0 0x7f55f0100d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:42.310 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:14:42.309+0000 7f55effff700 1 -- 172.21.15.174:0/3136566220 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55f0107fd0 msgr2=0x7f55f0101260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:42.310 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:14:42.309+0000 7f55effff700 1 --2- 172.21.15.174:0/3136566220 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55f0107fd0 0x7f55f0101260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:42.311 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:14:42.309+0000 7f55effff700 1 -- 172.21.15.174:0/3136566220 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55f0101b10 con 0x7f55f0102d70 2024-04-21T23:14:42.311 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:14:42.309+0000 7f55effff700 1 --2- 172.21.15.174:0/3136566220 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f55f0102d70 0x7f55f01026d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f55e4005910 tx=0x7f55e4009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:42.311 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:14:42.309+0000 7f55ed7fa700 1 -- 172.21.15.174:0/3136566220 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55e4027070 con 0x7f55f0102d70 2024-04-21T23:14:42.311 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:14:42.309+0000 7f55ed7fa700 1 -- 172.21.15.174:0/3136566220 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f55e4004150 con 0x7f55f0102d70 2024-04-21T23:14:42.311 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:14:42.309+0000 7f55f639b700 1 -- 172.21.15.174:0/3136566220 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55f01a25d0 con 0x7f55f0102d70 2024-04-21T23:14:42.311 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:14:42.309+0000 7f55ed7fa700 1 -- 172.21.15.174:0/3136566220 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55e4022db0 con 0x7f55f0102d70 2024-04-21T23:14:42.311 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:14:42.309+0000 7f55f639b700 1 -- 172.21.15.174:0/3136566220 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55f01a2aa0 con 0x7f55f0102d70 2024-04-21T23:14:42.311 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:14:42.310+0000 7f55f639b700 1 -- 172.21.15.174:0/3136566220 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55f0068f20 con 0x7f55f0102d70 2024-04-21T23:14:42.312 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:14:42.311+0000 7f55ed7fa700 1 -- 172.21.15.174:0/3136566220 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f55e402a070 con 0x7f55f0102d70 2024-04-21T23:14:42.312 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:14:42.311+0000 7f55ed7fa700 1 --2- 172.21.15.174:0/3136566220 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f55d80a2c20 0x7f55d80a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:42.312 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:14:42.312+0000 7f55ed7fa700 1 -- 172.21.15.174:0/3136566220 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 6694+0+0 (secure 0 0 0) 0x7f55e40cf800 con 0x7f55f0102d70 2024-04-21T23:14:42.312 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:14:42.312+0000 7f55ef7fe700 1 --2- 172.21.15.174:0/3136566220 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f55d80a2c20 0x7f55d80a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:42.312 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:14:42.312+0000 7f55ef7fe700 1 --2- 172.21.15.174:0/3136566220 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f55d80a2c20 0x7f55d80a50e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f55e0005950 tx=0x7f55e000b410 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:42.315 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:14:42.316+0000 7f55ed7fa700 1 -- 172.21.15.174:0/3136566220 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f55e405e6f0 con 0x7f55f0102d70 2024-04-21T23:14:42.526 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:14:42.527+0000 7f55f639b700 1 -- 172.21.15.174:0/3136566220 --> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f55f0062670 con 0x7f55d80a2c20 2024-04-21T23:14:43.141 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:42 smithi174 ceph-mon[31409]: osdmap e76: 11 total, 11 up, 11 in 2024-04-21T23:14:43.142 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:42 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:14:43.142 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:42 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:14:43.142 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:42 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:14:43.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:42 smithi040 ceph-mon[27984]: osdmap e76: 11 total, 11 up, 11 in 2024-04-21T23:14:43.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:42 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:14:43.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:42 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:14:43.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:42 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:14:43.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:42 smithi111 ceph-mon[31085]: osdmap e76: 11 total, 11 up, 11 in 2024-04-21T23:14:43.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:42 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:14:43.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:42 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:14:43.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:42 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:14:44.166 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:43 smithi174 ceph-mon[31409]: pgmap v242: 1 pgs: 1 remapped+peering; 577 KiB data, 316 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:14:44.166 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:43 smithi174 ceph-mon[31409]: from='client.24355 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:14:44.166 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:43 smithi174 ceph-mon[31409]: osdmap e77: 11 total, 11 up, 11 in 2024-04-21T23:14:44.181 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:43 smithi040 ceph-mon[27984]: pgmap v242: 1 pgs: 1 remapped+peering; 577 KiB data, 316 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:14:44.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:43 smithi040 ceph-mon[27984]: from='client.24355 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:14:44.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:43 smithi040 ceph-mon[27984]: osdmap e77: 11 total, 11 up, 11 in 2024-04-21T23:14:44.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:43 smithi111 ceph-mon[31085]: pgmap v242: 1 pgs: 1 remapped+peering; 577 KiB data, 316 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:14:44.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:43 smithi111 ceph-mon[31085]: from='client.24355 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:14:44.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:43 smithi111 ceph-mon[31085]: osdmap e77: 11 total, 11 up, 11 in 2024-04-21T23:14:46.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:45 smithi040 ceph-mon[27984]: pgmap v244: 1 pgs: 1 remapped+peering; 577 KiB data, 316 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:14:46.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:45 smithi040 ceph-mon[27984]: from='client.? 172.21.15.174:0/1215189801' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "781532bb-7a5e-44e3-8eb7-b584b0054666"}]: dispatch 2024-04-21T23:14:46.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:45 smithi040 ceph-mon[27984]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "781532bb-7a5e-44e3-8eb7-b584b0054666"}]: dispatch 2024-04-21T23:14:46.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:45 smithi040 ceph-mon[27984]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "781532bb-7a5e-44e3-8eb7-b584b0054666"}]': finished 2024-04-21T23:14:46.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:45 smithi040 ceph-mon[27984]: osdmap e78: 12 total, 11 up, 12 in 2024-04-21T23:14:46.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:45 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T23:14:46.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:45 smithi174 ceph-mon[31409]: pgmap v244: 1 pgs: 1 remapped+peering; 577 KiB data, 316 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:14:46.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:45 smithi174 ceph-mon[31409]: from='client.? 172.21.15.174:0/1215189801' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "781532bb-7a5e-44e3-8eb7-b584b0054666"}]: dispatch 2024-04-21T23:14:46.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:45 smithi174 ceph-mon[31409]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "781532bb-7a5e-44e3-8eb7-b584b0054666"}]: dispatch 2024-04-21T23:14:46.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:45 smithi174 ceph-mon[31409]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "781532bb-7a5e-44e3-8eb7-b584b0054666"}]': finished 2024-04-21T23:14:46.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:45 smithi174 ceph-mon[31409]: osdmap e78: 12 total, 11 up, 12 in 2024-04-21T23:14:46.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:45 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T23:14:46.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:45 smithi111 ceph-mon[31085]: pgmap v244: 1 pgs: 1 remapped+peering; 577 KiB data, 316 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:14:46.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:45 smithi111 ceph-mon[31085]: from='client.? 172.21.15.174:0/1215189801' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "781532bb-7a5e-44e3-8eb7-b584b0054666"}]: dispatch 2024-04-21T23:14:46.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:45 smithi111 ceph-mon[31085]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "781532bb-7a5e-44e3-8eb7-b584b0054666"}]: dispatch 2024-04-21T23:14:46.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:45 smithi111 ceph-mon[31085]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "781532bb-7a5e-44e3-8eb7-b584b0054666"}]': finished 2024-04-21T23:14:46.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:45 smithi111 ceph-mon[31085]: osdmap e78: 12 total, 11 up, 12 in 2024-04-21T23:14:46.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:45 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T23:14:47.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:46 smithi040 ceph-mon[27984]: from='client.? 172.21.15.174:0/4032146530' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:14:47.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:46 smithi174 ceph-mon[31409]: from='client.? 172.21.15.174:0/4032146530' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:14:47.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:46 smithi111 ceph-mon[31085]: from='client.? 172.21.15.174:0/4032146530' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:14:48.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:47 smithi040 ceph-mon[27984]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-04-21T23:14:48.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:47 smithi174 ceph-mon[31409]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-04-21T23:14:48.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:47 smithi111 ceph-mon[31085]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-04-21T23:14:50.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:49 smithi040 ceph-mon[27984]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-04-21T23:14:50.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:49 smithi174 ceph-mon[31409]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-04-21T23:14:50.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:49 smithi111 ceph-mon[31085]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-04-21T23:14:52.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:51 smithi040 ceph-mon[27984]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-21T23:14:52.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:51 smithi174 ceph-mon[31409]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-21T23:14:52.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:51 smithi111 ceph-mon[31085]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-21T23:14:54.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:53 smithi040 ceph-mon[27984]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-04-21T23:14:54.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:53 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-21T23:14:54.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:53 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:14:54.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:53 smithi040 ceph-mon[27984]: Deploying daemon osd.11 on smithi174 2024-04-21T23:14:54.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:53 smithi174 ceph-mon[31409]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-04-21T23:14:54.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:53 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-21T23:14:54.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:53 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:14:54.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:53 smithi174 ceph-mon[31409]: Deploying daemon osd.11 on smithi174 2024-04-21T23:14:54.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:53 smithi111 ceph-mon[31085]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-04-21T23:14:54.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:53 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-21T23:14:54.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:53 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:14:54.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:53 smithi111 ceph-mon[31085]: Deploying daemon osd.11 on smithi174 2024-04-21T23:14:56.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:55 smithi040 ceph-mon[27984]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-21T23:14:56.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:55 smithi174 ceph-mon[31409]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-21T23:14:56.193 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:55 smithi111 ceph-mon[31085]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-21T23:14:58.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:57 smithi040 ceph-mon[27984]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:14:58.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:57 smithi174 ceph-mon[31409]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:14:58.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:57 smithi111 ceph-mon[31085]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:14:59.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:59 smithi174 ceph-mon[31409]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:14:59.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:59 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:14:59.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:59 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:14:59.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:14:59 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:00.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:59 smithi040 ceph-mon[27984]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:15:00.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:59 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:15:00.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:59 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:00.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:14:59 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:00.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:59 smithi111 ceph-mon[31085]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:15:00.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:59 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:15:00.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:59 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:00.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:14:59 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:02.105 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:01 smithi174 ceph-mon[31409]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:15:02.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:01 smithi040 ceph-mon[27984]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:15:02.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:01 smithi111 ceph-mon[31085]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:15:03.303 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:15:03.303+0000 7f55ed7fa700 1 -- 172.21.15.174:0/3136566220 <== mgr.14148 v2:172.21.15.40:6800/2500760788 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f55f0062670 con 0x7f55d80a2c20 2024-04-21T23:15:03.303 INFO:teuthology.orchestra.run.smithi174.stdout:Created osd(s) 11 on host 'smithi174' 2024-04-21T23:15:03.305 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:15:03.305+0000 7f55f639b700 1 -- 172.21.15.174:0/3136566220 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f55d80a2c20 msgr2=0x7f55d80a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:03.305 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:15:03.305+0000 7f55f639b700 1 --2- 172.21.15.174:0/3136566220 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f55d80a2c20 0x7f55d80a50e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f55e0005950 tx=0x7f55e000b410 comp rx=0 tx=0).stop 2024-04-21T23:15:03.305 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:15:03.305+0000 7f55f639b700 1 -- 172.21.15.174:0/3136566220 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f55f0102d70 msgr2=0x7f55f01026d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:03.305 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:15:03.305+0000 7f55f639b700 1 --2- 172.21.15.174:0/3136566220 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f55f0102d70 0x7f55f01026d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f55e4005910 tx=0x7f55e4009580 comp rx=0 tx=0).stop 2024-04-21T23:15:03.305 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:15:03.305+0000 7f55f639b700 1 -- 172.21.15.174:0/3136566220 shutdown_connections 2024-04-21T23:15:03.306 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:15:03.305+0000 7f55f639b700 1 --2- 172.21.15.174:0/3136566220 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f55f0102d70 0x7f55f01026d0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:03.306 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:15:03.305+0000 7f55f639b700 1 --2- 172.21.15.174:0/3136566220 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f55d80a2c20 0x7f55d80a50e0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:03.306 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:15:03.305+0000 7f55f639b700 1 --2- 172.21.15.174:0/3136566220 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55f0107fd0 0x7f55f0101260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:03.306 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:15:03.305+0000 7f55f639b700 1 --2- 172.21.15.174:0/3136566220 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55f01056a0 0x7f55f0100d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:03.306 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:15:03.306+0000 7f55f639b700 1 -- 172.21.15.174:0/3136566220 >> 172.21.15.174:0/3136566220 conn(0x7f55f00fc720 msgr2=0x7f55f01036b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:03.306 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:15:03.306+0000 7f55f639b700 1 -- 172.21.15.174:0/3136566220 shutdown_connections 2024-04-21T23:15:03.306 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:15:03.306+0000 7f55f639b700 1 -- 172.21.15.174:0/3136566220 wait complete. 2024-04-21T23:15:03.604 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:03 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:03.604 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:03 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:03.605 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:03 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:15:03.605 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:03 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:15:03.605 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:03 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:03.605 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:03 smithi174 ceph-mon[31409]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:15:03.605 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:03 smithi174 ceph-mon[31409]: from='osd.11 [v2:172.21.15.174:6824/2045865553,v1:172.21.15.174:6825/2045865553]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-21T23:15:03.605 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:03 smithi174 ceph-mon[31409]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-21T23:15:03.605 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:03 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:15:03.605 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:03 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:03.605 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:03 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:03.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:03 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:03.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:03 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:03.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:03 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:15:03.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:03 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:15:03.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:03 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:03.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:03 smithi040 ceph-mon[27984]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:15:03.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:03 smithi040 ceph-mon[27984]: from='osd.11 [v2:172.21.15.174:6824/2045865553,v1:172.21.15.174:6825/2045865553]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-21T23:15:03.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:03 smithi040 ceph-mon[27984]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-21T23:15:03.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:03 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:15:03.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:03 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:03.683 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:03 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:03.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:03 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:03.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:03 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:03.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:03 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:15:03.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:03 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:15:03.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:03 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:03.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:03 smithi111 ceph-mon[31085]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:15:03.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:03 smithi111 ceph-mon[31085]: from='osd.11 [v2:172.21.15.174:6824/2045865553,v1:172.21.15.174:6825/2045865553]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-21T23:15:03.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:03 smithi111 ceph-mon[31085]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-21T23:15:03.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:03 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:15:03.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:03 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:03.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:03 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:03.942 DEBUG:teuthology.orchestra.run.smithi174:osd.11> sudo journalctl -f -n 0 -u ceph-d814e4dc-0033-11ef-bc93-c7b262605968@osd.11.service 2024-04-21T23:15:03.944 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-04-21T23:15:03.944 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-21T23:15:04.201 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:15:04.496 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:04 smithi174 ceph-mon[31409]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-21T23:15:04.496 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:04 smithi174 ceph-mon[31409]: osdmap e79: 12 total, 11 up, 12 in 2024-04-21T23:15:04.497 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:04 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T23:15:04.497 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:04 smithi174 ceph-mon[31409]: from='osd.11 [v2:172.21.15.174:6824/2045865553,v1:172.21.15.174:6825/2045865553]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-04-21T23:15:04.497 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:04 smithi174 ceph-mon[31409]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-04-21T23:15:04.497 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:15:04 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:15:04.313+0000 7f4e00691700 -1 osd.11 0 waiting for initial osdmap 2024-04-21T23:15:04.497 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:15:04 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:15:04.330+0000 7f4dfc484700 -1 osd.11 80 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T23:15:04.598 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:04 smithi040 ceph-mon[27984]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-21T23:15:04.599 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:04 smithi040 ceph-mon[27984]: osdmap e79: 12 total, 11 up, 12 in 2024-04-21T23:15:04.599 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:04 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T23:15:04.599 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:04 smithi040 ceph-mon[27984]: from='osd.11 [v2:172.21.15.174:6824/2045865553,v1:172.21.15.174:6825/2045865553]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-04-21T23:15:04.599 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:04 smithi040 ceph-mon[27984]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-04-21T23:15:04.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:04 smithi111 ceph-mon[31085]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-21T23:15:04.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:04 smithi111 ceph-mon[31085]: osdmap e79: 12 total, 11 up, 12 in 2024-04-21T23:15:04.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:04 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T23:15:04.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:04 smithi111 ceph-mon[31085]: from='osd.11 [v2:172.21.15.174:6824/2045865553,v1:172.21.15.174:6825/2045865553]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-04-21T23:15:04.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:04 smithi111 ceph-mon[31085]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-04-21T23:15:05.404 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:05 smithi174 ceph-mon[31409]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-04-21T23:15:05.404 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:05 smithi174 ceph-mon[31409]: osdmap e80: 12 total, 11 up, 12 in 2024-04-21T23:15:05.404 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:05 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T23:15:05.404 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:05 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T23:15:05.404 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:05 smithi174 ceph-mon[31409]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:15:05.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:05 smithi040 ceph-mon[27984]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-04-21T23:15:05.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:05 smithi040 ceph-mon[27984]: osdmap e80: 12 total, 11 up, 12 in 2024-04-21T23:15:05.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:05 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T23:15:05.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:05 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T23:15:05.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:05 smithi040 ceph-mon[27984]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:15:05.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:05 smithi111 ceph-mon[31085]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-04-21T23:15:05.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:05 smithi111 ceph-mon[31085]: osdmap e80: 12 total, 11 up, 12 in 2024-04-21T23:15:05.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:05 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T23:15:05.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:05 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T23:15:05.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:05 smithi111 ceph-mon[31085]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:15:06.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:06.212+0000 7f3bd9bd7700 1 -- 172.21.15.40:0/737789734 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3bd41023b0 msgr2=0x7f3bd4102810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:06.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:06.212+0000 7f3bd9bd7700 1 --2- 172.21.15.40:0/737789734 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3bd41023b0 0x7f3bd4102810 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc0099c0 tx=0x7f3bbc009cd0 comp rx=0 tx=0).stop 2024-04-21T23:15:06.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:06.212+0000 7f3bd9bd7700 1 -- 172.21.15.40:0/737789734 shutdown_connections 2024-04-21T23:15:06.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:06.212+0000 7f3bd9bd7700 1 --2- 172.21.15.40:0/737789734 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3bd41011b0 0x7f3bd41015d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:06.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:06.212+0000 7f3bd9bd7700 1 --2- 172.21.15.40:0/737789734 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3bd4102d50 0x7f3bd4109600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:06.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:06.212+0000 7f3bd9bd7700 1 --2- 172.21.15.40:0/737789734 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3bd41023b0 0x7f3bd4102810 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:06.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:06.212+0000 7f3bd9bd7700 1 -- 172.21.15.40:0/737789734 >> 172.21.15.40:0/737789734 conn(0x7f3bd40fc750 msgr2=0x7f3bd40feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:06.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:06.212+0000 7f3bd9bd7700 1 -- 172.21.15.40:0/737789734 shutdown_connections 2024-04-21T23:15:06.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:06.212+0000 7f3bd9bd7700 1 -- 172.21.15.40:0/737789734 wait complete. 2024-04-21T23:15:06.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:06.213+0000 7f3bd9bd7700 1 Processor -- start 2024-04-21T23:15:06.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:06.213+0000 7f3bd9bd7700 1 -- start start 2024-04-21T23:15:06.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:06.213+0000 7f3bd9bd7700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3bd41011b0 0x7f3bd41973c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:06.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:06.213+0000 7f3bd9bd7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3bd41023b0 0x7f3bd4197900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:06.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:06.214+0000 7f3bd9bd7700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3bd4102d50 0x7f3bd419e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:06.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:06.214+0000 7f3bd9bd7700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3bd410cd80 con 0x7f3bd41011b0 2024-04-21T23:15:06.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:06.214+0000 7f3bd9bd7700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3bd410cc00 con 0x7f3bd41023b0 2024-04-21T23:15:06.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:06.214+0000 7f3bd9bd7700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3bd410cf00 con 0x7f3bd4102d50 2024-04-21T23:15:06.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:06.214+0000 7f3bd37fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3bd41011b0 0x7f3bd41973c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:06.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:06.214+0000 7f3bd37fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3bd41011b0 0x7f3bd41973c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37026/0 (socket says 172.21.15.40:37026) 2024-04-21T23:15:06.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:06.214+0000 7f3bd37fe700 1 -- 172.21.15.40:0/2491182433 learned_addr learned my addr 172.21.15.40:0/2491182433 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:15:06.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:06.214+0000 7f3bd3fff700 1 --2- 172.21.15.40:0/2491182433 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3bd4102d50 0x7f3bd419e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:06.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:06.214+0000 7f3bcaffd700 1 --2- 172.21.15.40:0/2491182433 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3bd41023b0 0x7f3bd4197900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:06.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:06.214+0000 7f3bd37fe700 1 -- 172.21.15.40:0/2491182433 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3bd4102d50 msgr2=0x7f3bd419e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:06.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:06.214+0000 7f3bd37fe700 1 --2- 172.21.15.40:0/2491182433 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3bd4102d50 0x7f3bd419e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:06.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:06.214+0000 7f3bd37fe700 1 -- 172.21.15.40:0/2491182433 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3bd41023b0 msgr2=0x7f3bd4197900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:06.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:06.214+0000 7f3bd37fe700 1 --2- 172.21.15.40:0/2491182433 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3bd41023b0 0x7f3bd4197900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:06.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:06.214+0000 7f3bd37fe700 1 -- 172.21.15.40:0/2491182433 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bd406aa60 con 0x7f3bd41011b0 2024-04-21T23:15:06.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:06.214+0000 7f3bcaffd700 1 --2- 172.21.15.40:0/2491182433 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3bd41023b0 0x7f3bd4197900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:15:06.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:06.214+0000 7f3bd37fe700 1 --2- 172.21.15.40:0/2491182433 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3bd41011b0 0x7f3bd41973c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f3bc400d980 tx=0x7f3bc400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:06.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:06.214+0000 7f3bd17fa700 1 -- 172.21.15.40:0/2491182433 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bc4017030 con 0x7f3bd41011b0 2024-04-21T23:15:06.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:06.214+0000 7f3bd17fa700 1 -- 172.21.15.40:0/2491182433 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3bc4004500 con 0x7f3bd41011b0 2024-04-21T23:15:06.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:06.214+0000 7f3bd17fa700 1 -- 172.21.15.40:0/2491182433 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bc4003aa0 con 0x7f3bd41011b0 2024-04-21T23:15:06.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:06.215+0000 7f3bd9bd7700 1 -- 172.21.15.40:0/2491182433 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bd406ad50 con 0x7f3bd41011b0 2024-04-21T23:15:06.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:06.215+0000 7f3bd9bd7700 1 -- 172.21.15.40:0/2491182433 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bd406b290 con 0x7f3bd41011b0 2024-04-21T23:15:06.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:06.216+0000 7f3bd9bd7700 1 -- 172.21.15.40:0/2491182433 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3bd4068f20 con 0x7f3bd41011b0 2024-04-21T23:15:06.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:06.217+0000 7f3bd17fa700 1 -- 172.21.15.40:0/2491182433 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f3bc401a070 con 0x7f3bd41011b0 2024-04-21T23:15:06.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:06.217+0000 7f3bd17fa700 1 --2- 172.21.15.40:0/2491182433 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f3bc00a2c20 0x7f3bc00a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:06.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:06.217+0000 7f3bcaffd700 1 --2- 172.21.15.40:0/2491182433 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f3bc00a2c20 0x7f3bc00a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:06.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:06.217+0000 7f3bd17fa700 1 -- 172.21.15.40:0/2491182433 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f3bc40ce4a0 con 0x7f3bd41011b0 2024-04-21T23:15:06.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:06.218+0000 7f3bcaffd700 1 --2- 172.21.15.40:0/2491182433 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f3bc00a2c20 0x7f3bc00a50e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc000c00 tx=0x7f3bbc005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:06.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:06.222+0000 7f3bd17fa700 1 -- 172.21.15.40:0/2491182433 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f3bc409b8a0 con 0x7f3bd41011b0 2024-04-21T23:15:06.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:06.432+0000 7f3bd9bd7700 1 -- 172.21.15.40:0/2491182433 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f3bd402dc30 con 0x7f3bd41011b0 2024-04-21T23:15:06.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:06.432+0000 7f3bd17fa700 1 -- 172.21.15.40:0/2491182433 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v82) ==== 74+0+133 (secure 0 0 0) 0x7f3bc409b1a0 con 0x7f3bd41011b0 2024-04-21T23:15:06.432 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:15:06.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:06.435+0000 7f3bd9bd7700 1 -- 172.21.15.40:0/2491182433 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f3bc00a2c20 msgr2=0x7f3bc00a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:06.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:06.435+0000 7f3bd9bd7700 1 --2- 172.21.15.40:0/2491182433 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f3bc00a2c20 0x7f3bc00a50e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc000c00 tx=0x7f3bbc005fd0 comp rx=0 tx=0).stop 2024-04-21T23:15:06.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:06.435+0000 7f3bd9bd7700 1 -- 172.21.15.40:0/2491182433 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3bd41011b0 msgr2=0x7f3bd41973c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:06.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:06.435+0000 7f3bd9bd7700 1 --2- 172.21.15.40:0/2491182433 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3bd41011b0 0x7f3bd41973c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f3bc400d980 tx=0x7f3bc400dc90 comp rx=0 tx=0).stop 2024-04-21T23:15:06.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:06.435+0000 7f3bd9bd7700 1 -- 172.21.15.40:0/2491182433 shutdown_connections 2024-04-21T23:15:06.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:06.435+0000 7f3bd9bd7700 1 --2- 172.21.15.40:0/2491182433 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3bd4102d50 0x7f3bd419e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:06.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:06.435+0000 7f3bd9bd7700 1 --2- 172.21.15.40:0/2491182433 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f3bc00a2c20 0x7f3bc00a50e0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:06.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:06.435+0000 7f3bd9bd7700 1 --2- 172.21.15.40:0/2491182433 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3bd41011b0 0x7f3bd41973c0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:06.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:06.435+0000 7f3bd9bd7700 1 --2- 172.21.15.40:0/2491182433 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3bd41023b0 0x7f3bd4197900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:06.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:06.435+0000 7f3bd9bd7700 1 -- 172.21.15.40:0/2491182433 >> 172.21.15.40:0/2491182433 conn(0x7f3bd40fc750 msgr2=0x7f3bd41056b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:06.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:06.435+0000 7f3bd9bd7700 1 -- 172.21.15.40:0/2491182433 shutdown_connections 2024-04-21T23:15:06.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:06.435+0000 7f3bd9bd7700 1 -- 172.21.15.40:0/2491182433 wait complete. 2024-04-21T23:15:06.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:06 smithi040 ceph-mon[27984]: purged_snaps scrub starts 2024-04-21T23:15:06.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:06 smithi040 ceph-mon[27984]: purged_snaps scrub ok 2024-04-21T23:15:06.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:06 smithi040 ceph-mon[27984]: osd.11 [v2:172.21.15.174:6824/2045865553,v1:172.21.15.174:6825/2045865553] boot 2024-04-21T23:15:06.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:06 smithi040 ceph-mon[27984]: osdmap e81: 12 total, 12 up, 12 in 2024-04-21T23:15:06.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:06 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T23:15:06.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:06 smithi174 ceph-mon[31409]: purged_snaps scrub starts 2024-04-21T23:15:06.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:06 smithi174 ceph-mon[31409]: purged_snaps scrub ok 2024-04-21T23:15:06.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:06 smithi174 ceph-mon[31409]: osd.11 [v2:172.21.15.174:6824/2045865553,v1:172.21.15.174:6825/2045865553] boot 2024-04-21T23:15:06.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:06 smithi174 ceph-mon[31409]: osdmap e81: 12 total, 12 up, 12 in 2024-04-21T23:15:06.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:06 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T23:15:06.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:06 smithi111 ceph-mon[31085]: purged_snaps scrub starts 2024-04-21T23:15:06.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:06 smithi111 ceph-mon[31085]: purged_snaps scrub ok 2024-04-21T23:15:06.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:06 smithi111 ceph-mon[31085]: osd.11 [v2:172.21.15.174:6824/2045865553,v1:172.21.15.174:6825/2045865553] boot 2024-04-21T23:15:06.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:06 smithi111 ceph-mon[31085]: osdmap e81: 12 total, 12 up, 12 in 2024-04-21T23:15:06.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:06 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T23:15:07.096 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":82,"num_osds":12,"num_up_osds":12,"osd_up_since":1713741305,"num_in_osds":12,"osd_in_since":1713741285,"num_remapped_pgs":0} 2024-04-21T23:15:07.097 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-21T23:15:07.359 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:15:07.588 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:07 smithi040 ceph-mon[27984]: osdmap e82: 12 total, 12 up, 12 in 2024-04-21T23:15:07.588 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:07 smithi040 ceph-mon[27984]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:07.588 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:07 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/2491182433' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-21T23:15:07.588 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:07 smithi040 ceph-mon[27984]: Detected new or changed devices on smithi174 2024-04-21T23:15:07.588 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:07 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:07.588 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:07 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:07.588 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:07 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:15:07.588 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:07 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:15:07.588 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:07 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:15:07.588 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:07 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:15:07.588 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:07 smithi040 ceph-mon[27984]: Adjusting osd_memory_target on smithi174 to 4272M 2024-04-21T23:15:07.589 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:07 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:07.589 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:07 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:15:07.589 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:07 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:15:07.589 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:07 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:07.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:07 smithi174 ceph-mon[31409]: osdmap e82: 12 total, 12 up, 12 in 2024-04-21T23:15:07.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:07 smithi174 ceph-mon[31409]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:07.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:07 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/2491182433' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-21T23:15:07.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:07 smithi174 ceph-mon[31409]: Detected new or changed devices on smithi174 2024-04-21T23:15:07.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:07 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:07.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:07 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:07.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:07 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:15:07.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:07 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:15:07.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:07 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:15:07.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:07 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:15:07.695 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:07 smithi174 ceph-mon[31409]: Adjusting osd_memory_target on smithi174 to 4272M 2024-04-21T23:15:07.695 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:07 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:07.695 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:07 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:15:07.695 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:07 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:15:07.695 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:07 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:07.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:07 smithi111 ceph-mon[31085]: osdmap e82: 12 total, 12 up, 12 in 2024-04-21T23:15:07.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:07 smithi111 ceph-mon[31085]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:07.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:07 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/2491182433' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-21T23:15:07.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:07 smithi111 ceph-mon[31085]: Detected new or changed devices on smithi174 2024-04-21T23:15:07.696 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:07 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:07.696 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:07 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:07.696 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:07 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:15:07.696 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:07 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:15:07.696 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:07 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:15:07.696 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:07 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:15:07.696 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:07 smithi111 ceph-mon[31085]: Adjusting osd_memory_target on smithi174 to 4272M 2024-04-21T23:15:07.696 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:07 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:07.696 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:07 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:15:07.696 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:07 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:15:07.696 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:07 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:08.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:08.909+0000 7f80a6e30700 1 -- 172.21.15.40:0/2420278002 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f80a006a8e0 msgr2=0x7f80a010f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:08.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:08.909+0000 7f80a6e30700 1 --2- 172.21.15.40:0/2420278002 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f80a006a8e0 0x7f80a010f0f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f809800b0a0 tx=0x7f809800b3b0 comp rx=0 tx=0).stop 2024-04-21T23:15:08.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:08.909+0000 7f80a6e30700 1 -- 172.21.15.40:0/2420278002 shutdown_connections 2024-04-21T23:15:08.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:08.909+0000 7f80a6e30700 1 --2- 172.21.15.40:0/2420278002 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f80a01082b0 0x7f80a0108690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:08.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:08.909+0000 7f80a6e30700 1 --2- 172.21.15.40:0/2420278002 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f80a006a8e0 0x7f80a010f0f0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:08.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:08.909+0000 7f80a6e30700 1 --2- 172.21.15.40:0/2420278002 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80a0069f40 0x7f80a006a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:08.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:08.909+0000 7f80a6e30700 1 -- 172.21.15.40:0/2420278002 >> 172.21.15.40:0/2420278002 conn(0x7f80a00780e0 msgr2=0x7f80a00784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:08.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:08.909+0000 7f80a6e30700 1 -- 172.21.15.40:0/2420278002 shutdown_connections 2024-04-21T23:15:08.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:08.909+0000 7f80a6e30700 1 -- 172.21.15.40:0/2420278002 wait complete. 2024-04-21T23:15:08.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:08.910+0000 7f80a6e30700 1 Processor -- start 2024-04-21T23:15:08.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:08.910+0000 7f80a6e30700 1 -- start start 2024-04-21T23:15:08.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:08.910+0000 7f80a6e30700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f80a0069f40 0x7f80a0101200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:08.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:08.910+0000 7f80a6e30700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80a006a8e0 0x7f80a0101740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:08.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:08.910+0000 7f80a6e30700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f80a01082b0 0x7f80a0101c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:08.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:08.910+0000 7f80a6e30700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f80a0112570 con 0x7f80a0069f40 2024-04-21T23:15:08.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:08.910+0000 7f80a6e30700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f80a01123f0 con 0x7f80a006a8e0 2024-04-21T23:15:08.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:08.910+0000 7f80a6e30700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f80a01126f0 con 0x7f80a01082b0 2024-04-21T23:15:08.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:08.910+0000 7f80a4bcc700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f80a0069f40 0x7f80a0101200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:08.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:08.910+0000 7f80a4bcc700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f80a0069f40 0x7f80a0101200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37038/0 (socket says 172.21.15.40:37038) 2024-04-21T23:15:08.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:08.910+0000 7f80a4bcc700 1 -- 172.21.15.40:0/362240644 learned_addr learned my addr 172.21.15.40:0/362240644 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:15:08.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:08.910+0000 7f809ffff700 1 --2- 172.21.15.40:0/362240644 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80a006a8e0 0x7f80a0101740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:08.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:08.910+0000 7f80a53cd700 1 --2- 172.21.15.40:0/362240644 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f80a01082b0 0x7f80a0101c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:08.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:08.910+0000 7f80a4bcc700 1 -- 172.21.15.40:0/362240644 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f80a01082b0 msgr2=0x7f80a0101c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:08.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:08.910+0000 7f80a4bcc700 1 --2- 172.21.15.40:0/362240644 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f80a01082b0 0x7f80a0101c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:08.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:08.910+0000 7f80a4bcc700 1 -- 172.21.15.40:0/362240644 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80a006a8e0 msgr2=0x7f80a0101740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:08.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:08.910+0000 7f80a4bcc700 1 --2- 172.21.15.40:0/362240644 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80a006a8e0 0x7f80a0101740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:08.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:08.910+0000 7f80a4bcc700 1 -- 172.21.15.40:0/362240644 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80a01a6f10 con 0x7f80a0069f40 2024-04-21T23:15:08.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:08.911+0000 7f80a4bcc700 1 --2- 172.21.15.40:0/362240644 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f80a0069f40 0x7f80a0101200 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f808800b530 tx=0x7f808800b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:08.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:08.911+0000 7f809dffb700 1 -- 172.21.15.40:0/362240644 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8088015020 con 0x7f80a0069f40 2024-04-21T23:15:08.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:08.911+0000 7f809dffb700 1 -- 172.21.15.40:0/362240644 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8088011d10 con 0x7f80a0069f40 2024-04-21T23:15:08.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:08.911+0000 7f80a6e30700 1 -- 172.21.15.40:0/362240644 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80a01a7200 con 0x7f80a0069f40 2024-04-21T23:15:08.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:08.911+0000 7f809dffb700 1 -- 172.21.15.40:0/362240644 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8088010e50 con 0x7f80a0069f40 2024-04-21T23:15:08.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:08.911+0000 7f80a6e30700 1 -- 172.21.15.40:0/362240644 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80a01a76f0 con 0x7f80a0069f40 2024-04-21T23:15:08.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:08.912+0000 7f80a6e30700 1 -- 172.21.15.40:0/362240644 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80a010a820 con 0x7f80a0069f40 2024-04-21T23:15:08.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:08.913+0000 7f809dffb700 1 -- 172.21.15.40:0/362240644 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f8088018020 con 0x7f80a0069f40 2024-04-21T23:15:08.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:08.914+0000 7f809dffb700 1 --2- 172.21.15.40:0/362240644 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f808c0a2dc0 0x7f808c0a5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:08.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:08.914+0000 7f809ffff700 1 --2- 172.21.15.40:0/362240644 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f808c0a2dc0 0x7f808c0a5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:08.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:08.914+0000 7f809dffb700 1 -- 172.21.15.40:0/362240644 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f80880ce4a0 con 0x7f80a0069f40 2024-04-21T23:15:08.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:08.914+0000 7f809ffff700 1 --2- 172.21.15.40:0/362240644 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f808c0a2dc0 0x7f808c0a5280 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f8090005950 tx=0x7f80900058e0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:08.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:08.918+0000 7f809dffb700 1 -- 172.21.15.40:0/362240644 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f808801f020 con 0x7f80a0069f40 2024-04-21T23:15:09.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:09.126+0000 7f80a6e30700 1 -- 172.21.15.40:0/362240644 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f80a0102730 con 0x7f80a0069f40 2024-04-21T23:15:09.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:09.127+0000 7f809dffb700 1 -- 172.21.15.40:0/362240644 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+18466 (secure 0 0 0) 0x7f808809b8a0 con 0x7f80a0069f40 2024-04-21T23:15:09.126 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:15:09.127 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":82,"fsid":"d814e4dc-0033-11ef-bc93-c7b262605968","created":"2024-04-21T23:07:52.659207+0000","modified":"2024-04-21T23:15:06.317496+0000","last_up_change":"2024-04-21T23:15:05.308821+0000","last_in_change":"2024-04-21T23:14:45.447491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:11:14.824775+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0e8d692a-22fe-4a87-8d99-9151f96cbb02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6803","nonce":2009758784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6805","nonce":2009758784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6809","nonce":2009758784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6807","nonce":2009758784}]},"public_addr":"172.21.15.40:6803/2009758784","cluster_addr":"172.21.15.40:6805/2009758784","heartbeat_back_addr":"172.21.15.40:6809/2009758784","heartbeat_front_addr":"172.21.15.40:6807/2009758784","state":["exists","up"]},{"osd":1,"uuid":"fe5fe11b-0e84-4359-83a7-1ba45105755a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6801","nonce":1763903836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6803","nonce":1763903836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6807","nonce":1763903836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6805","nonce":1763903836}]},"public_addr":"172.21.15.111:6801/1763903836","cluster_addr":"172.21.15.111:6803/1763903836","heartbeat_back_addr":"172.21.15.111:6807/1763903836","heartbeat_front_addr":"172.21.15.111:6805/1763903836","state":["exists","up"]},{"osd":2,"uuid":"813ad735-5cc2-4da0-ae16-6db238395fbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6801","nonce":439695825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6803","nonce":439695825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6807","nonce":439695825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6805","nonce":439695825}]},"public_addr":"172.21.15.174:6801/439695825","cluster_addr":"172.21.15.174:6803/439695825","heartbeat_back_addr":"172.21.15.174:6807/439695825","heartbeat_front_addr":"172.21.15.174:6805/439695825","state":["exists","up"]},{"osd":3,"uuid":"09becb03-b7ce-4ffc-917d-b7900ec37e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6811","nonce":3454312436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6813","nonce":3454312436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6817","nonce":3454312436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6815","nonce":3454312436}]},"public_addr":"172.21.15.40:6811/3454312436","cluster_addr":"172.21.15.40:6813/3454312436","heartbeat_back_addr":"172.21.15.40:6817/3454312436","heartbeat_front_addr":"172.21.15.40:6815/3454312436","state":["exists","up"]},{"osd":4,"uuid":"46eabfd7-d88a-4efd-94eb-04354d3a9d7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6809","nonce":1810866070}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6811","nonce":1810866070}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6815","nonce":1810866070}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6813","nonce":1810866070}]},"public_addr":"172.21.15.111:6809/1810866070","cluster_addr":"172.21.15.111:6811/1810866070","heartbeat_back_addr":"172.21.15.111:6815/1810866070","heartbeat_front_addr":"172.21.15.111:6813/1810866070","state":["exists","up"]},{"osd":5,"uuid":"6f73b582-6c65-41db-aab0-9b360f83d5c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6809","nonce":3752986711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6811","nonce":3752986711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6815","nonce":3752986711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6813","nonce":3752986711}]},"public_addr":"172.21.15.174:6809/3752986711","cluster_addr":"172.21.15.174:6811/3752986711","heartbeat_back_addr":"172.21.15.174:6815/3752986711","heartbeat_front_addr":"172.21.15.174:6813/3752986711","state":["exists","up"]},{"osd":6,"uuid":"39f305f0-12d2-4732-aaa0-cd66700af538","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6819","nonce":1618855427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6821","nonce":1618855427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6825","nonce":1618855427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6823","nonce":1618855427}]},"public_addr":"172.21.15.40:6819/1618855427","cluster_addr":"172.21.15.40:6821/1618855427","heartbeat_back_addr":"172.21.15.40:6825/1618855427","heartbeat_front_addr":"172.21.15.40:6823/1618855427","state":["exists","up"]},{"osd":7,"uuid":"769f2468-5102-490b-89f8-d0444d3169a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6817","nonce":927054162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6819","nonce":927054162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6823","nonce":927054162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6821","nonce":927054162}]},"public_addr":"172.21.15.111:6817/927054162","cluster_addr":"172.21.15.111:6819/927054162","heartbeat_back_addr":"172.21.15.111:6823/927054162","heartbeat_front_addr":"172.21.15.111:6821/927054162","state":["exists","up"]},{"osd":8,"uuid":"1a444e80-007d-473b-b972-4c814a813c22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6817","nonce":3817848746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6819","nonce":3817848746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6823","nonce":3817848746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6821","nonce":3817848746}]},"public_addr":"172.21.15.174:6817/3817848746","cluster_addr":"172.21.15.174:6819/3817848746","heartbeat_back_addr":"172.21.15.174:6823/3817848746","heartbeat_front_addr":"172.21.15.174:6821/3817848746","state":["exists","up"]},{"osd":9,"uuid":"9a6b6a56-6ae5-49fe-9e5b-cb17d686e0dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6827","nonce":1649446531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6829","nonce":1649446531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6833","nonce":1649446531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6831","nonce":1649446531}]},"public_addr":"172.21.15.40:6827/1649446531","cluster_addr":"172.21.15.40:6829/1649446531","heartbeat_back_addr":"172.21.15.40:6833/1649446531","heartbeat_front_addr":"172.21.15.40:6831/1649446531","state":["exists","up"]},{"osd":10,"uuid":"a70038ba-e2ad-4170-85cc-ea0470a2edef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6825","nonce":1227538935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6827","nonce":1227538935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6831","nonce":1227538935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6829","nonce":1227538935}]},"public_addr":"172.21.15.111:6825/1227538935","cluster_addr":"172.21.15.111:6827/1227538935","heartbeat_back_addr":"172.21.15.111:6831/1227538935","heartbeat_front_addr":"172.21.15.111:6829/1227538935","state":["exists","up"]},{"osd":11,"uuid":"781532bb-7a5e-44e3-8eb7-b584b0054666","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6825","nonce":2045865553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6827","nonce":2045865553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6831","nonce":2045865553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6829","nonce":2045865553}]},"public_addr":"172.21.15.174:6825/2045865553","cluster_addr":"172.21.15.174:6827/2045865553","heartbeat_back_addr":"172.21.15.174:6831/2045865553","heartbeat_front_addr":"172.21.15.174:6829/2045865553","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:10:24.761290+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:10:48.565428+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:11:12.307818+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:11:37.951529+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:02.832523+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:27.568879+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:52.997707+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:13:18.937274+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:13:44.708657+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:14:10.833855+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:14:37.078272+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:15:03.878076+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:6801/526554319":"2024-04-22T23:08:50.305770+0000","172.21.15.40:6800/526554319":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/3178763471":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/4045911364":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/276300080":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/3689649814":"2024-04-22T23:08:23.743030+0000","172.21.15.40:0/2241116715":"2024-04-22T23:08:23.743030+0000","172.21.15.40:6801/1479040326":"2024-04-22T23:08:23.743030+0000","172.21.15.40:6800/1479040326":"2024-04-22T23:08:23.743030+0000","172.21.15.40:0/681298099":"2024-04-22T23:08:23.743030+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T23:15:09.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:09.129+0000 7f80a6e30700 1 -- 172.21.15.40:0/362240644 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f808c0a2dc0 msgr2=0x7f808c0a5280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:09.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:09.129+0000 7f80a6e30700 1 --2- 172.21.15.40:0/362240644 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f808c0a2dc0 0x7f808c0a5280 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f8090005950 tx=0x7f80900058e0 comp rx=0 tx=0).stop 2024-04-21T23:15:09.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:09.129+0000 7f80a6e30700 1 -- 172.21.15.40:0/362240644 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f80a0069f40 msgr2=0x7f80a0101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:09.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:09.129+0000 7f80a6e30700 1 --2- 172.21.15.40:0/362240644 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f80a0069f40 0x7f80a0101200 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f808800b530 tx=0x7f808800b8f0 comp rx=0 tx=0).stop 2024-04-21T23:15:09.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:09.129+0000 7f80a6e30700 1 -- 172.21.15.40:0/362240644 shutdown_connections 2024-04-21T23:15:09.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:09.129+0000 7f80a6e30700 1 --2- 172.21.15.40:0/362240644 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f80a01082b0 0x7f80a0101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:09.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:09.129+0000 7f80a6e30700 1 --2- 172.21.15.40:0/362240644 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f808c0a2dc0 0x7f808c0a5280 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:09.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:09.129+0000 7f80a6e30700 1 --2- 172.21.15.40:0/362240644 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f80a0069f40 0x7f80a0101200 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:09.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:09.129+0000 7f80a6e30700 1 --2- 172.21.15.40:0/362240644 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80a006a8e0 0x7f80a0101740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:09.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:09.129+0000 7f80a6e30700 1 -- 172.21.15.40:0/362240644 >> 172.21.15.40:0/362240644 conn(0x7f80a00780e0 msgr2=0x7f80a00ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:09.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:09.129+0000 7f80a6e30700 1 -- 172.21.15.40:0/362240644 shutdown_connections 2024-04-21T23:15:09.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:09.129+0000 7f80a6e30700 1 -- 172.21.15.40:0/362240644 wait complete. 2024-04-21T23:15:09.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:09 smithi174 ceph-mon[31409]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:09.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:09 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/362240644' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:15:09.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:09 smithi111 ceph-mon[31085]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:09.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:09 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/362240644' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:15:09.810 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:09 smithi040 ceph-mon[27984]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:09.810 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:09 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/362240644' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:15:09.844 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T23:11:14.824775+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-21T23:15:09.845 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-21T23:15:10.096 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:15:11.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:11 smithi174 ceph-mon[31409]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:11.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:11 smithi111 ceph-mon[31085]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:11.723 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:11 smithi040 ceph-mon[27984]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:11.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:11.740+0000 7f7d89823700 1 -- 172.21.15.40:0/2061796975 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d84107420 msgr2=0x7f7d84107800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:11.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:11.740+0000 7f7d89823700 1 --2- 172.21.15.40:0/2061796975 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d84107420 0x7f7d84107800 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7d740099c0 tx=0x7f7d74009cd0 comp rx=0 tx=0).stop 2024-04-21T23:15:11.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:11.740+0000 7f7d89823700 1 -- 172.21.15.40:0/2061796975 shutdown_connections 2024-04-21T23:15:11.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:11.740+0000 7f7d89823700 1 --2- 172.21.15.40:0/2061796975 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d841026c0 0x7f7d8410e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:11.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:11.740+0000 7f7d89823700 1 --2- 172.21.15.40:0/2061796975 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d84107420 0x7f7d84107800 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:11.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:11.740+0000 7f7d89823700 1 --2- 172.21.15.40:0/2061796975 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d84101d00 0x7f7d84102180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:11.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:11.740+0000 7f7d89823700 1 -- 172.21.15.40:0/2061796975 >> 172.21.15.40:0/2061796975 conn(0x7f7d84076450 msgr2=0x7f7d84076860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:11.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:11.740+0000 7f7d89823700 1 -- 172.21.15.40:0/2061796975 shutdown_connections 2024-04-21T23:15:11.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:11.740+0000 7f7d89823700 1 -- 172.21.15.40:0/2061796975 wait complete. 2024-04-21T23:15:11.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:11.740+0000 7f7d89823700 1 Processor -- start 2024-04-21T23:15:11.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:11.741+0000 7f7d89823700 1 -- start start 2024-04-21T23:15:11.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:11.741+0000 7f7d89823700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d84101d00 0x7f7d8419e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:11.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:11.741+0000 7f7d89823700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d841026c0 0x7f7d8419ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:11.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:11.741+0000 7f7d89823700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d84107420 0x7f7d841988c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:11.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:11.741+0000 7f7d89823700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7d84111590 con 0x7f7d841026c0 2024-04-21T23:15:11.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:11.741+0000 7f7d89823700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f7d84111410 con 0x7f7d84107420 2024-04-21T23:15:11.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:11.741+0000 7f7d89823700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7d84111710 con 0x7f7d84101d00 2024-04-21T23:15:11.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:11.741+0000 7f7d827fc700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d841026c0 0x7f7d8419ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:11.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:11.741+0000 7f7d827fc700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d841026c0 0x7f7d8419ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33386/0 (socket says 172.21.15.40:33386) 2024-04-21T23:15:11.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:11.741+0000 7f7d827fc700 1 -- 172.21.15.40:0/2280601725 learned_addr learned my addr 172.21.15.40:0/2280601725 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:15:11.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:11.741+0000 7f7d837fe700 1 --2- 172.21.15.40:0/2280601725 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d84107420 0x7f7d841988c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:11.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:11.741+0000 7f7d82ffd700 1 --2- 172.21.15.40:0/2280601725 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d84101d00 0x7f7d8419e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:11.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:11.741+0000 7f7d827fc700 1 -- 172.21.15.40:0/2280601725 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d84101d00 msgr2=0x7f7d8419e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:11.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:11.741+0000 7f7d827fc700 1 --2- 172.21.15.40:0/2280601725 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d84101d00 0x7f7d8419e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:11.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:11.741+0000 7f7d827fc700 1 -- 172.21.15.40:0/2280601725 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d84107420 msgr2=0x7f7d841988c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:11.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:11.741+0000 7f7d827fc700 1 --2- 172.21.15.40:0/2280601725 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d84107420 0x7f7d841988c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:11.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:11.741+0000 7f7d827fc700 1 -- 172.21.15.40:0/2280601725 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d84199020 con 0x7f7d841026c0 2024-04-21T23:15:11.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:11.741+0000 7f7d82ffd700 1 --2- 172.21.15.40:0/2280601725 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d84101d00 0x7f7d8419e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:15:11.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:11.742+0000 7f7d827fc700 1 --2- 172.21.15.40:0/2280601725 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d841026c0 0x7f7d8419ec40 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7d7800e9c0 tx=0x7f7d7800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:11.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:11.742+0000 7f7d88821700 1 -- 172.21.15.40:0/2280601725 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d7800cbd0 con 0x7f7d841026c0 2024-04-21T23:15:11.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:11.742+0000 7f7d89823700 1 -- 172.21.15.40:0/2280601725 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d84199310 con 0x7f7d841026c0 2024-04-21T23:15:11.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:11.742+0000 7f7d88821700 1 -- 172.21.15.40:0/2280601725 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d7800cd70 con 0x7f7d841026c0 2024-04-21T23:15:11.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:11.742+0000 7f7d88821700 1 -- 172.21.15.40:0/2280601725 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d78010430 con 0x7f7d841026c0 2024-04-21T23:15:11.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:11.742+0000 7f7d89823700 1 -- 172.21.15.40:0/2280601725 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d841a55d0 con 0x7f7d841026c0 2024-04-21T23:15:11.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:11.743+0000 7f7d89823700 1 -- 172.21.15.40:0/2280601725 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d84068f20 con 0x7f7d841026c0 2024-04-21T23:15:11.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:11.744+0000 7f7d88821700 1 -- 172.21.15.40:0/2280601725 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f7d78004750 con 0x7f7d841026c0 2024-04-21T23:15:11.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:11.744+0000 7f7d88821700 1 --2- 172.21.15.40:0/2280601725 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f7d6c0a2c20 0x7f7d6c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:11.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:11.745+0000 7f7d82ffd700 1 --2- 172.21.15.40:0/2280601725 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f7d6c0a2c20 0x7f7d6c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:11.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:11.745+0000 7f7d88821700 1 -- 172.21.15.40:0/2280601725 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f7d780ceee0 con 0x7f7d841026c0 2024-04-21T23:15:11.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:11.745+0000 7f7d82ffd700 1 --2- 172.21.15.40:0/2280601725 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f7d6c0a2c20 0x7f7d6c0a50e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7d74000c00 tx=0x7f7d74005940 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:11.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:11.749+0000 7f7d88821700 1 -- 172.21.15.40:0/2280601725 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f7d7809c240 con 0x7f7d841026c0 2024-04-21T23:15:11.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:11.954+0000 7f7d89823700 1 -- 172.21.15.40:0/2280601725 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f7d8402db30 con 0x7f7d841026c0 2024-04-21T23:15:11.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:11.954+0000 7f7d88821700 1 -- 172.21.15.40:0/2280601725 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) ==== 93+0+10 (secure 0 0 0) 0x7f7d7809bb40 con 0x7f7d841026c0 2024-04-21T23:15:11.954 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-04-21T23:15:11.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:11.957+0000 7f7d89823700 1 -- 172.21.15.40:0/2280601725 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f7d6c0a2c20 msgr2=0x7f7d6c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:11.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:11.957+0000 7f7d89823700 1 --2- 172.21.15.40:0/2280601725 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f7d6c0a2c20 0x7f7d6c0a50e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7d74000c00 tx=0x7f7d74005940 comp rx=0 tx=0).stop 2024-04-21T23:15:11.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:11.957+0000 7f7d89823700 1 -- 172.21.15.40:0/2280601725 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d841026c0 msgr2=0x7f7d8419ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:11.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:11.957+0000 7f7d89823700 1 --2- 172.21.15.40:0/2280601725 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d841026c0 0x7f7d8419ec40 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7d7800e9c0 tx=0x7f7d7800ecd0 comp rx=0 tx=0).stop 2024-04-21T23:15:11.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:11.957+0000 7f7d89823700 1 -- 172.21.15.40:0/2280601725 shutdown_connections 2024-04-21T23:15:11.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:11.957+0000 7f7d89823700 1 --2- 172.21.15.40:0/2280601725 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d84101d00 0x7f7d8419e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:11.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:11.957+0000 7f7d89823700 1 --2- 172.21.15.40:0/2280601725 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f7d6c0a2c20 0x7f7d6c0a50e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:11.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:11.957+0000 7f7d89823700 1 --2- 172.21.15.40:0/2280601725 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d84107420 0x7f7d841988c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:11.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:11.957+0000 7f7d89823700 1 --2- 172.21.15.40:0/2280601725 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d841026c0 0x7f7d8419ec40 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:11.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:11.957+0000 7f7d89823700 1 -- 172.21.15.40:0/2280601725 >> 172.21.15.40:0/2280601725 conn(0x7f7d84076450 msgr2=0x7f7d84100cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:11.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:11.957+0000 7f7d89823700 1 -- 172.21.15.40:0/2280601725 shutdown_connections 2024-04-21T23:15:11.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:11.957+0000 7f7d89823700 1 -- 172.21.15.40:0/2280601725 wait complete. 2024-04-21T23:15:12.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:12 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/2280601725' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:15:12.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:12 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/2280601725' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:15:12.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:12 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/2280601725' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:15:12.742 INFO:tasks.cephadm:Adding mds.a on smithi040 2024-04-21T23:15:12.742 INFO:tasks.cephadm:Adding mds.d on smithi040 2024-04-21T23:15:12.743 INFO:tasks.cephadm:Adding mds.g on smithi040 2024-04-21T23:15:12.743 INFO:tasks.cephadm:Adding mds.j on smithi040 2024-04-21T23:15:12.743 INFO:tasks.cephadm:Adding mds.b on smithi111 2024-04-21T23:15:12.743 INFO:tasks.cephadm:Adding mds.e on smithi111 2024-04-21T23:15:12.743 INFO:tasks.cephadm:Adding mds.h on smithi111 2024-04-21T23:15:12.743 INFO:tasks.cephadm:Adding mds.k on smithi111 2024-04-21T23:15:12.743 INFO:tasks.cephadm:Adding mds.c on smithi174 2024-04-21T23:15:12.743 INFO:tasks.cephadm:Adding mds.f on smithi174 2024-04-21T23:15:12.743 INFO:tasks.cephadm:Adding mds.i on smithi174 2024-04-21T23:15:12.743 INFO:tasks.cephadm:Adding mds.l on smithi174 2024-04-21T23:15:12.743 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph orch apply mds all '12;smithi040=a;smithi040=d;smithi040=g;smithi040=j;smithi111=b;smithi111=e;smithi111=h;smithi111=k;smithi174=c;smithi174=f;smithi174=i;smithi174=l' 2024-04-21T23:15:12.993 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.c/config 2024-04-21T23:15:13.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:13 smithi174 ceph-mon[31409]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:13.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:13 smithi040 ceph-mon[27984]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:13.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:13 smithi111 ceph-mon[31085]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:14.662 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:15:14.663+0000 7f729e6cc700 1 -- 172.21.15.174:0/2187922890 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7298102d20 msgr2=0x7f72981095d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:14.662 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:15:14.663+0000 7f729e6cc700 1 --2- 172.21.15.174:0/2187922890 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7298102d20 0x7f72981095d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f729000b0a0 tx=0x7f729000b3b0 comp rx=0 tx=0).stop 2024-04-21T23:15:14.662 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:15:14.663+0000 7f729e6cc700 1 -- 172.21.15.174:0/2187922890 shutdown_connections 2024-04-21T23:15:14.663 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:15:14.663+0000 7f729e6cc700 1 --2- 172.21.15.174:0/2187922890 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7298102d20 0x7f72981095d0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:14.663 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:15:14.663+0000 7f729e6cc700 1 --2- 172.21.15.174:0/2187922890 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7298101180 0x7f72981015a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:14.663 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:15:14.663+0000 7f729e6cc700 1 --2- 172.21.15.174:0/2187922890 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7298102380 0x7f72981027e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:14.663 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:15:14.663+0000 7f729e6cc700 1 -- 172.21.15.174:0/2187922890 >> 172.21.15.174:0/2187922890 conn(0x7f72980fc760 msgr2=0x7f72980feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:14.663 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:15:14.663+0000 7f729e6cc700 1 -- 172.21.15.174:0/2187922890 shutdown_connections 2024-04-21T23:15:14.663 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:15:14.663+0000 7f729e6cc700 1 -- 172.21.15.174:0/2187922890 wait complete. 2024-04-21T23:15:14.665 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:15:14.664+0000 7f729e6cc700 1 Processor -- start 2024-04-21T23:15:14.665 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:15:14.664+0000 7f729e6cc700 1 -- start start 2024-04-21T23:15:14.665 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:15:14.664+0000 7f729e6cc700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7298101180 0x7f7298198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:14.665 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:15:14.665+0000 7f729e6cc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7298102380 0x7f72981989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:14.665 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:15:14.665+0000 7f729e6cc700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7298102d20 0x7f729819fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:14.665 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:15:14.665+0000 7f729e6cc700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f729810dee0 con 0x7f7298101180 2024-04-21T23:15:14.665 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:15:14.665+0000 7f729e6cc700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f729810dd60 con 0x7f7298102380 2024-04-21T23:15:14.665 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:15:14.665+0000 7f729e6cc700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f729810e060 con 0x7f7298102d20 2024-04-21T23:15:14.665 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:15:14.665+0000 7f729cc69700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7298102d20 0x7f729819fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:14.665 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:15:14.665+0000 7f729cc69700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7298102d20 0x7f729819fa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.174:37338/0 (socket says 172.21.15.174:37338) 2024-04-21T23:15:14.665 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:15:14.665+0000 7f729cc69700 1 -- 172.21.15.174:0/826076074 learned_addr learned my addr 172.21.15.174:0/826076074 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-04-21T23:15:14.665 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:15:14.665+0000 7f72977fe700 1 --2- 172.21.15.174:0/826076074 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7298102380 0x7f72981989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:14.665 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:15:14.665+0000 7f7297fff700 1 --2- 172.21.15.174:0/826076074 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7298101180 0x7f7298198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:14.666 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:15:14.665+0000 7f729cc69700 1 -- 172.21.15.174:0/826076074 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7298102380 msgr2=0x7f72981989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:14.666 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:15:14.665+0000 7f729cc69700 1 --2- 172.21.15.174:0/826076074 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7298102380 0x7f72981989b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:14.666 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:15:14.665+0000 7f729cc69700 1 -- 172.21.15.174:0/826076074 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7298101180 msgr2=0x7f7298198470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:14.666 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:15:14.665+0000 7f729cc69700 1 --2- 172.21.15.174:0/826076074 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7298101180 0x7f7298198470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:14.666 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:15:14.665+0000 7f729cc69700 1 -- 172.21.15.174:0/826076074 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72981a00d0 con 0x7f7298102d20 2024-04-21T23:15:14.666 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:15:14.665+0000 7f729cc69700 1 --2- 172.21.15.174:0/826076074 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7298102d20 0x7f729819fa30 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f729000be60 tx=0x7f729000be90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:14.666 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:15:14.665+0000 7f72957fa700 1 -- 172.21.15.174:0/826076074 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7290021020 con 0x7f7298102d20 2024-04-21T23:15:14.666 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:15:14.665+0000 7f729e6cc700 1 -- 172.21.15.174:0/826076074 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72981a0360 con 0x7f7298102d20 2024-04-21T23:15:14.666 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:15:14.665+0000 7f72957fa700 1 -- 172.21.15.174:0/826076074 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7290005520 con 0x7f7298102d20 2024-04-21T23:15:14.666 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:15:14.665+0000 7f72957fa700 1 -- 172.21.15.174:0/826076074 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f729001b900 con 0x7f7298102d20 2024-04-21T23:15:14.666 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:15:14.665+0000 7f729e6cc700 1 -- 172.21.15.174:0/826076074 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72981a0890 con 0x7f7298102d20 2024-04-21T23:15:14.666 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:15:14.666+0000 7f729e6cc700 1 -- 172.21.15.174:0/826076074 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7298068f20 con 0x7f7298102d20 2024-04-21T23:15:14.667 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:15:14.668+0000 7f72957fa700 1 -- 172.21.15.174:0/826076074 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f7290024020 con 0x7f7298102d20 2024-04-21T23:15:14.667 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:15:14.669+0000 7f72957fa700 1 --2- 172.21.15.174:0/826076074 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f72800a2c20 0x7f72800a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:14.667 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:15:14.669+0000 7f72957fa700 1 -- 172.21.15.174:0/826076074 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f729000f020 con 0x7f7298102d20 2024-04-21T23:15:14.668 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:15:14.669+0000 7f7297fff700 1 --2- 172.21.15.174:0/826076074 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f72800a2c20 0x7f72800a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:14.668 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:15:14.670+0000 7f7297fff700 1 --2- 172.21.15.174:0/826076074 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f72800a2c20 0x7f72800a50e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f72981021e0 tx=0x7f7288009190 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:14.671 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:15:14.673+0000 7f72957fa700 1 -- 172.21.15.174:0/826076074 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f7290011e50 con 0x7f7298102d20 2024-04-21T23:15:14.877 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:15:14.878+0000 7f729e6cc700 1 -- 172.21.15.174:0/826076074 --> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi040=a;smithi040=d;smithi040=g;smithi040=j;smithi111=b;smithi111=e;smithi111=h;smithi111=k;smithi174=c;smithi174=f;smithi174=i;smithi174=l", "target": ["mon-mgr", ""]}) -- 0x7f7298062770 con 0x7f72800a2c20 2024-04-21T23:15:14.885 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:15:14.886+0000 7f72957fa700 1 -- 172.21.15.174:0/826076074 <== mgr.14148 v2:172.21.15.40:6800/2500760788 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f7298062770 con 0x7f72800a2c20 2024-04-21T23:15:14.885 INFO:teuthology.orchestra.run.smithi174.stdout:Scheduled mds.all update... 2024-04-21T23:15:14.887 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:15:14.889+0000 7f729e6cc700 1 -- 172.21.15.174:0/826076074 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f72800a2c20 msgr2=0x7f72800a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:14.888 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:15:14.889+0000 7f729e6cc700 1 --2- 172.21.15.174:0/826076074 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f72800a2c20 0x7f72800a50e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f72981021e0 tx=0x7f7288009190 comp rx=0 tx=0).stop 2024-04-21T23:15:14.888 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:15:14.889+0000 7f729e6cc700 1 -- 172.21.15.174:0/826076074 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7298102d20 msgr2=0x7f729819fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:14.888 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:15:14.889+0000 7f729e6cc700 1 --2- 172.21.15.174:0/826076074 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7298102d20 0x7f729819fa30 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f729000be60 tx=0x7f729000be90 comp rx=0 tx=0).stop 2024-04-21T23:15:14.888 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:15:14.889+0000 7f729e6cc700 1 -- 172.21.15.174:0/826076074 shutdown_connections 2024-04-21T23:15:14.888 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:15:14.889+0000 7f729e6cc700 1 --2- 172.21.15.174:0/826076074 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7298102d20 0x7f729819fa30 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:14.888 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:15:14.889+0000 7f729e6cc700 1 --2- 172.21.15.174:0/826076074 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f72800a2c20 0x7f72800a50e0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:14.888 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:15:14.889+0000 7f729e6cc700 1 --2- 172.21.15.174:0/826076074 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7298101180 0x7f7298198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:14.888 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:15:14.889+0000 7f729e6cc700 1 --2- 172.21.15.174:0/826076074 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7298102380 0x7f72981989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:14.888 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:15:14.889+0000 7f729e6cc700 1 -- 172.21.15.174:0/826076074 >> 172.21.15.174:0/826076074 conn(0x7f72980fc760 msgr2=0x7f7298105680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:14.888 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:15:14.889+0000 7f729e6cc700 1 -- 172.21.15.174:0/826076074 shutdown_connections 2024-04-21T23:15:14.888 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-21T23:15:14.889+0000 7f729e6cc700 1 -- 172.21.15.174:0/826076074 wait complete. 2024-04-21T23:15:15.752 DEBUG:teuthology.orchestra.run.smithi040:mds.a> sudo journalctl -f -n 0 -u ceph-d814e4dc-0033-11ef-bc93-c7b262605968@mds.a.service 2024-04-21T23:15:15.755 DEBUG:teuthology.orchestra.run.smithi040:mds.d> sudo journalctl -f -n 0 -u ceph-d814e4dc-0033-11ef-bc93-c7b262605968@mds.d.service 2024-04-21T23:15:15.798 DEBUG:teuthology.orchestra.run.smithi040:mds.g> sudo journalctl -f -n 0 -u ceph-d814e4dc-0033-11ef-bc93-c7b262605968@mds.g.service 2024-04-21T23:15:15.841 DEBUG:teuthology.orchestra.run.smithi040:mds.j> sudo journalctl -f -n 0 -u ceph-d814e4dc-0033-11ef-bc93-c7b262605968@mds.j.service 2024-04-21T23:15:15.885 DEBUG:teuthology.orchestra.run.smithi111:mds.b> sudo journalctl -f -n 0 -u ceph-d814e4dc-0033-11ef-bc93-c7b262605968@mds.b.service 2024-04-21T23:15:15.887 DEBUG:teuthology.orchestra.run.smithi111:mds.e> sudo journalctl -f -n 0 -u ceph-d814e4dc-0033-11ef-bc93-c7b262605968@mds.e.service 2024-04-21T23:15:15.931 DEBUG:teuthology.orchestra.run.smithi111:mds.h> sudo journalctl -f -n 0 -u ceph-d814e4dc-0033-11ef-bc93-c7b262605968@mds.h.service 2024-04-21T23:15:15.973 DEBUG:teuthology.orchestra.run.smithi111:mds.k> sudo journalctl -f -n 0 -u ceph-d814e4dc-0033-11ef-bc93-c7b262605968@mds.k.service 2024-04-21T23:15:16.017 DEBUG:teuthology.orchestra.run.smithi174:mds.c> sudo journalctl -f -n 0 -u ceph-d814e4dc-0033-11ef-bc93-c7b262605968@mds.c.service 2024-04-21T23:15:16.019 DEBUG:teuthology.orchestra.run.smithi174:mds.f> sudo journalctl -f -n 0 -u ceph-d814e4dc-0033-11ef-bc93-c7b262605968@mds.f.service 2024-04-21T23:15:16.039 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:15 smithi174 ceph-mon[31409]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:16.039 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:15 smithi174 ceph-mon[31409]: from='client.24397 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi040=a;smithi040=d;smithi040=g;smithi040=j;smithi111=b;smithi111=e;smithi111=h;smithi111=k;smithi174=c;smithi174=f;smithi174=i;smithi174=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:15:16.039 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:15 smithi174 ceph-mon[31409]: Saving service mds.all spec with placement smithi040=a;smithi040=d;smithi040=g;smithi040=j;smithi111=b;smithi111=e;smithi111=h;smithi111=k;smithi174=c;smithi174=f;smithi174=i;smithi174=l;count:12 2024-04-21T23:15:16.039 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:15 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:16.039 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:15 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:15:16.039 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:15 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:15:16.039 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:15 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:15:16.039 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:15 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:16.039 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:15 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:15:16.039 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:15 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:15:16.039 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:15 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:15:16.040 DEBUG:teuthology.orchestra.run.smithi174:mds.i> sudo journalctl -f -n 0 -u ceph-d814e4dc-0033-11ef-bc93-c7b262605968@mds.i.service 2024-04-21T23:15:16.083 DEBUG:teuthology.orchestra.run.smithi174:mds.l> sudo journalctl -f -n 0 -u ceph-d814e4dc-0033-11ef-bc93-c7b262605968@mds.l.service 2024-04-21T23:15:16.126 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-04-21T23:15:16.126 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T23:15:16.150 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:15 smithi040 ceph-mon[27984]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:16.150 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:15 smithi040 ceph-mon[27984]: from='client.24397 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi040=a;smithi040=d;smithi040=g;smithi040=j;smithi111=b;smithi111=e;smithi111=h;smithi111=k;smithi174=c;smithi174=f;smithi174=i;smithi174=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:15:16.150 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:15 smithi040 ceph-mon[27984]: Saving service mds.all spec with placement smithi040=a;smithi040=d;smithi040=g;smithi040=j;smithi111=b;smithi111=e;smithi111=h;smithi111=k;smithi174=c;smithi174=f;smithi174=i;smithi174=l;count:12 2024-04-21T23:15:16.150 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:15 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:16.150 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:15 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:15:16.150 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:15 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:15:16.150 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:15 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:15:16.151 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:15 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:16.151 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:15 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:15:16.151 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:15 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:15:16.151 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:15 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:15:16.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:15 smithi111 ceph-mon[31085]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:16.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:15 smithi111 ceph-mon[31085]: from='client.24397 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi040=a;smithi040=d;smithi040=g;smithi040=j;smithi111=b;smithi111=e;smithi111=h;smithi111=k;smithi174=c;smithi174=f;smithi174=i;smithi174=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:15:16.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:15 smithi111 ceph-mon[31085]: Saving service mds.all spec with placement smithi040=a;smithi040=d;smithi040=g;smithi040=j;smithi111=b;smithi111=e;smithi111=h;smithi111=k;smithi174=c;smithi174=f;smithi174=i;smithi174=l;count:12 2024-04-21T23:15:16.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:15 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:16.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:15 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:15:16.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:15 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:15:16.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:15 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:15:16.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:15 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:16.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:15 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:15:16.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:15 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:15:16.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:15 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:15:16.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.263+0000 7faa7e888640 1 -- 172.21.15.40:0/397056699 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa78102250 msgr2=0x7faa78108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:16.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.263+0000 7faa7e888640 1 --2- 172.21.15.40:0/397056699 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa78102250 0x7faa78108ae0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7faa70009ca0 tx=0x7faa7002c0f0 comp rx=0 tx=0).stop 2024-04-21T23:15:16.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.263+0000 7faa7e888640 1 -- 172.21.15.40:0/397056699 shutdown_connections 2024-04-21T23:15:16.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.263+0000 7faa7e888640 1 --2- 172.21.15.40:0/397056699 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa78102250 0x7faa78108ae0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:16.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.263+0000 7faa7e888640 1 --2- 172.21.15.40:0/397056699 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa78101890 0x7faa78101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:16.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.263+0000 7faa7e888640 1 --2- 172.21.15.40:0/397056699 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa781006a0 0x7faa78100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:16.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.263+0000 7faa7e888640 1 -- 172.21.15.40:0/397056699 >> 172.21.15.40:0/397056699 conn(0x7faa780fbc30 msgr2=0x7faa780fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:16.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.263+0000 7faa7e888640 1 -- 172.21.15.40:0/397056699 shutdown_connections 2024-04-21T23:15:16.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.263+0000 7faa7e888640 1 -- 172.21.15.40:0/397056699 wait complete. 2024-04-21T23:15:16.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.264+0000 7faa7e888640 1 Processor -- start 2024-04-21T23:15:16.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.264+0000 7faa7e888640 1 -- start start 2024-04-21T23:15:16.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.264+0000 7faa7e888640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa781006a0 0x7faa7810b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:16.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.264+0000 7faa7e888640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa78101890 0x7faa78109660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:16.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.264+0000 7faa7e888640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa78102250 0x7faa78109ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:16.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.264+0000 7faa7e888640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7faa7810d480 con 0x7faa78101890 2024-04-21T23:15:16.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.264+0000 7faa7e888640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7faa7810d300 con 0x7faa78102250 2024-04-21T23:15:16.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.264+0000 7faa7e888640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7faa7810d600 con 0x7faa781006a0 2024-04-21T23:15:16.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.264+0000 7faa777fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa78101890 0x7faa78109660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:16.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.264+0000 7faa777fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa78101890 0x7faa78109660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33400/0 (socket says 172.21.15.40:33400) 2024-04-21T23:15:16.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.264+0000 7faa777fe640 1 -- 172.21.15.40:0/3673862127 learned_addr learned my addr 172.21.15.40:0/3673862127 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:15:16.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.264+0000 7faa77fff640 1 --2- 172.21.15.40:0/3673862127 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa781006a0 0x7faa7810b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:16.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.264+0000 7faa7ce25640 1 --2- 172.21.15.40:0/3673862127 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa78102250 0x7faa78109ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:16.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.264+0000 7faa777fe640 1 -- 172.21.15.40:0/3673862127 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa781006a0 msgr2=0x7faa7810b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:16.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.264+0000 7faa777fe640 1 --2- 172.21.15.40:0/3673862127 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa781006a0 0x7faa7810b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:16.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.264+0000 7faa777fe640 1 -- 172.21.15.40:0/3673862127 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa78102250 msgr2=0x7faa78109ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:16.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.264+0000 7faa777fe640 1 --2- 172.21.15.40:0/3673862127 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa78102250 0x7faa78109ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:16.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.264+0000 7faa777fe640 1 -- 172.21.15.40:0/3673862127 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa7810a430 con 0x7faa78101890 2024-04-21T23:15:16.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.264+0000 7faa7ce25640 1 --2- 172.21.15.40:0/3673862127 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa78102250 0x7faa78109ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:15:16.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.265+0000 7faa777fe640 1 --2- 172.21.15.40:0/3673862127 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa78101890 0x7faa78109660 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7faa6c00b4a0 tx=0x7faa6c00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:16.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.265+0000 7faa757fa640 1 -- 172.21.15.40:0/3673862127 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa6c014070 con 0x7faa78101890 2024-04-21T23:15:16.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.265+0000 7faa757fa640 1 -- 172.21.15.40:0/3673862127 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa6c010af0 con 0x7faa78101890 2024-04-21T23:15:16.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.265+0000 7faa7e888640 1 -- 172.21.15.40:0/3673862127 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa781a1bb0 con 0x7faa78101890 2024-04-21T23:15:16.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.265+0000 7faa757fa640 1 -- 172.21.15.40:0/3673862127 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa6c017d10 con 0x7faa78101890 2024-04-21T23:15:16.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.265+0000 7faa7e888640 1 -- 172.21.15.40:0/3673862127 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa781a2100 con 0x7faa78101890 2024-04-21T23:15:16.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.266+0000 7faa7e888640 1 -- 172.21.15.40:0/3673862127 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa781079c0 con 0x7faa78101890 2024-04-21T23:15:16.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.267+0000 7faa757fa640 1 -- 172.21.15.40:0/3673862127 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7faa6c017070 con 0x7faa78101890 2024-04-21T23:15:16.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.267+0000 7faa757fa640 1 --2- 172.21.15.40:0/3673862127 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7faa600a2b90 0x7faa600a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:16.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.268+0000 7faa77fff640 1 --2- 172.21.15.40:0/3673862127 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7faa600a2b90 0x7faa600a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:16.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.268+0000 7faa757fa640 1 -- 172.21.15.40:0/3673862127 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7faa6c0cdff0 con 0x7faa78101890 2024-04-21T23:15:16.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.268+0000 7faa77fff640 1 --2- 172.21.15.40:0/3673862127 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7faa600a2b90 0x7faa600a5050 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7faa781016f0 tx=0x7faa6800b390 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:16.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.272+0000 7faa757fa640 1 -- 172.21.15.40:0/3673862127 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7faa6c09b3f0 con 0x7faa78101890 2024-04-21T23:15:16.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.459+0000 7faa7e888640 1 -- 172.21.15.40:0/3673862127 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7faa7802d390 con 0x7faa78101890 2024-04-21T23:15:16.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.460+0000 7faa757fa640 1 -- 172.21.15.40:0/3673862127 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+18466 (secure 0 0 0) 0x7faa6c09acf0 con 0x7faa78101890 2024-04-21T23:15:16.459 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:15:16.460 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":82,"fsid":"d814e4dc-0033-11ef-bc93-c7b262605968","created":"2024-04-21T23:07:52.659207+0000","modified":"2024-04-21T23:15:06.317496+0000","last_up_change":"2024-04-21T23:15:05.308821+0000","last_in_change":"2024-04-21T23:14:45.447491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:11:14.824775+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0e8d692a-22fe-4a87-8d99-9151f96cbb02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6803","nonce":2009758784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6805","nonce":2009758784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6809","nonce":2009758784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6807","nonce":2009758784}]},"public_addr":"172.21.15.40:6803/2009758784","cluster_addr":"172.21.15.40:6805/2009758784","heartbeat_back_addr":"172.21.15.40:6809/2009758784","heartbeat_front_addr":"172.21.15.40:6807/2009758784","state":["exists","up"]},{"osd":1,"uuid":"fe5fe11b-0e84-4359-83a7-1ba45105755a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6801","nonce":1763903836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6803","nonce":1763903836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6807","nonce":1763903836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6805","nonce":1763903836}]},"public_addr":"172.21.15.111:6801/1763903836","cluster_addr":"172.21.15.111:6803/1763903836","heartbeat_back_addr":"172.21.15.111:6807/1763903836","heartbeat_front_addr":"172.21.15.111:6805/1763903836","state":["exists","up"]},{"osd":2,"uuid":"813ad735-5cc2-4da0-ae16-6db238395fbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6801","nonce":439695825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6803","nonce":439695825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6807","nonce":439695825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6805","nonce":439695825}]},"public_addr":"172.21.15.174:6801/439695825","cluster_addr":"172.21.15.174:6803/439695825","heartbeat_back_addr":"172.21.15.174:6807/439695825","heartbeat_front_addr":"172.21.15.174:6805/439695825","state":["exists","up"]},{"osd":3,"uuid":"09becb03-b7ce-4ffc-917d-b7900ec37e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6811","nonce":3454312436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6813","nonce":3454312436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6817","nonce":3454312436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6815","nonce":3454312436}]},"public_addr":"172.21.15.40:6811/3454312436","cluster_addr":"172.21.15.40:6813/3454312436","heartbeat_back_addr":"172.21.15.40:6817/3454312436","heartbeat_front_addr":"172.21.15.40:6815/3454312436","state":["exists","up"]},{"osd":4,"uuid":"46eabfd7-d88a-4efd-94eb-04354d3a9d7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6809","nonce":1810866070}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6811","nonce":1810866070}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6815","nonce":1810866070}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6813","nonce":1810866070}]},"public_addr":"172.21.15.111:6809/1810866070","cluster_addr":"172.21.15.111:6811/1810866070","heartbeat_back_addr":"172.21.15.111:6815/1810866070","heartbeat_front_addr":"172.21.15.111:6813/1810866070","state":["exists","up"]},{"osd":5,"uuid":"6f73b582-6c65-41db-aab0-9b360f83d5c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6809","nonce":3752986711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6811","nonce":3752986711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6815","nonce":3752986711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6813","nonce":3752986711}]},"public_addr":"172.21.15.174:6809/3752986711","cluster_addr":"172.21.15.174:6811/3752986711","heartbeat_back_addr":"172.21.15.174:6815/3752986711","heartbeat_front_addr":"172.21.15.174:6813/3752986711","state":["exists","up"]},{"osd":6,"uuid":"39f305f0-12d2-4732-aaa0-cd66700af538","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6819","nonce":1618855427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6821","nonce":1618855427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6825","nonce":1618855427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6823","nonce":1618855427}]},"public_addr":"172.21.15.40:6819/1618855427","cluster_addr":"172.21.15.40:6821/1618855427","heartbeat_back_addr":"172.21.15.40:6825/1618855427","heartbeat_front_addr":"172.21.15.40:6823/1618855427","state":["exists","up"]},{"osd":7,"uuid":"769f2468-5102-490b-89f8-d0444d3169a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6817","nonce":927054162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6819","nonce":927054162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6823","nonce":927054162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6821","nonce":927054162}]},"public_addr":"172.21.15.111:6817/927054162","cluster_addr":"172.21.15.111:6819/927054162","heartbeat_back_addr":"172.21.15.111:6823/927054162","heartbeat_front_addr":"172.21.15.111:6821/927054162","state":["exists","up"]},{"osd":8,"uuid":"1a444e80-007d-473b-b972-4c814a813c22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6817","nonce":3817848746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6819","nonce":3817848746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6823","nonce":3817848746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6821","nonce":3817848746}]},"public_addr":"172.21.15.174:6817/3817848746","cluster_addr":"172.21.15.174:6819/3817848746","heartbeat_back_addr":"172.21.15.174:6823/3817848746","heartbeat_front_addr":"172.21.15.174:6821/3817848746","state":["exists","up"]},{"osd":9,"uuid":"9a6b6a56-6ae5-49fe-9e5b-cb17d686e0dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6827","nonce":1649446531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6829","nonce":1649446531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6833","nonce":1649446531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6831","nonce":1649446531}]},"public_addr":"172.21.15.40:6827/1649446531","cluster_addr":"172.21.15.40:6829/1649446531","heartbeat_back_addr":"172.21.15.40:6833/1649446531","heartbeat_front_addr":"172.21.15.40:6831/1649446531","state":["exists","up"]},{"osd":10,"uuid":"a70038ba-e2ad-4170-85cc-ea0470a2edef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6825","nonce":1227538935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6827","nonce":1227538935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6831","nonce":1227538935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6829","nonce":1227538935}]},"public_addr":"172.21.15.111:6825/1227538935","cluster_addr":"172.21.15.111:6827/1227538935","heartbeat_back_addr":"172.21.15.111:6831/1227538935","heartbeat_front_addr":"172.21.15.111:6829/1227538935","state":["exists","up"]},{"osd":11,"uuid":"781532bb-7a5e-44e3-8eb7-b584b0054666","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6825","nonce":2045865553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6827","nonce":2045865553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6831","nonce":2045865553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6829","nonce":2045865553}]},"public_addr":"172.21.15.174:6825/2045865553","cluster_addr":"172.21.15.174:6827/2045865553","heartbeat_back_addr":"172.21.15.174:6831/2045865553","heartbeat_front_addr":"172.21.15.174:6829/2045865553","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:10:24.761290+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:10:48.565428+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:11:12.307818+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:11:37.951529+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:02.832523+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:27.568879+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:52.997707+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:13:18.937274+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:13:44.708657+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:14:10.833855+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:14:37.078272+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:15:03.878076+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:6801/526554319":"2024-04-22T23:08:50.305770+0000","172.21.15.40:6800/526554319":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/3178763471":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/4045911364":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/276300080":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/3689649814":"2024-04-22T23:08:23.743030+0000","172.21.15.40:0/2241116715":"2024-04-22T23:08:23.743030+0000","172.21.15.40:6801/1479040326":"2024-04-22T23:08:23.743030+0000","172.21.15.40:6800/1479040326":"2024-04-22T23:08:23.743030+0000","172.21.15.40:0/681298099":"2024-04-22T23:08:23.743030+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T23:15:16.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.462+0000 7faa7e888640 1 -- 172.21.15.40:0/3673862127 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7faa600a2b90 msgr2=0x7faa600a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:16.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.462+0000 7faa7e888640 1 --2- 172.21.15.40:0/3673862127 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7faa600a2b90 0x7faa600a5050 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7faa781016f0 tx=0x7faa6800b390 comp rx=0 tx=0).stop 2024-04-21T23:15:16.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.462+0000 7faa7e888640 1 -- 172.21.15.40:0/3673862127 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa78101890 msgr2=0x7faa78109660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:16.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.462+0000 7faa7e888640 1 --2- 172.21.15.40:0/3673862127 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa78101890 0x7faa78109660 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7faa6c00b4a0 tx=0x7faa6c00b960 comp rx=0 tx=0).stop 2024-04-21T23:15:16.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.462+0000 7faa7e888640 1 -- 172.21.15.40:0/3673862127 shutdown_connections 2024-04-21T23:15:16.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.462+0000 7faa7e888640 1 --2- 172.21.15.40:0/3673862127 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7faa600a2b90 0x7faa600a5050 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:16.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.462+0000 7faa7e888640 1 --2- 172.21.15.40:0/3673862127 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa78102250 0x7faa78109ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:16.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.462+0000 7faa7e888640 1 --2- 172.21.15.40:0/3673862127 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa78101890 0x7faa78109660 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:16.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.462+0000 7faa7e888640 1 --2- 172.21.15.40:0/3673862127 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa781006a0 0x7faa7810b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:16.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.462+0000 7faa7e888640 1 -- 172.21.15.40:0/3673862127 >> 172.21.15.40:0/3673862127 conn(0x7faa780fbc30 msgr2=0x7faa78104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:16.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.462+0000 7faa7e888640 1 -- 172.21.15.40:0/3673862127 shutdown_connections 2024-04-21T23:15:16.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.462+0000 7faa7e888640 1 -- 172.21.15.40:0/3673862127 wait complete. 2024-04-21T23:15:16.472 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T23:11:14.824775+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-21T23:15:16.473 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T23:15:16.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.617+0000 7f81a2ea3640 1 -- 172.21.15.40:0/39539422 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f819c101890 msgr2=0x7f819c101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:16.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.617+0000 7f81a2ea3640 1 --2- 172.21.15.40:0/39539422 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f819c101890 0x7f819c101d10 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f8184009900 tx=0x7f818402bc30 comp rx=0 tx=0).stop 2024-04-21T23:15:16.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.617+0000 7f81a2ea3640 1 -- 172.21.15.40:0/39539422 shutdown_connections 2024-04-21T23:15:16.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.617+0000 7f81a2ea3640 1 --2- 172.21.15.40:0/39539422 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f819c102250 0x7f819c108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:16.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.617+0000 7f81a2ea3640 1 --2- 172.21.15.40:0/39539422 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f819c101890 0x7f819c101d10 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:16.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.617+0000 7f81a2ea3640 1 --2- 172.21.15.40:0/39539422 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f819c1006a0 0x7f819c100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:16.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.617+0000 7f81a2ea3640 1 -- 172.21.15.40:0/39539422 >> 172.21.15.40:0/39539422 conn(0x7f819c0fbc30 msgr2=0x7f819c0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:16.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.617+0000 7f81a2ea3640 1 -- 172.21.15.40:0/39539422 shutdown_connections 2024-04-21T23:15:16.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.617+0000 7f81a2ea3640 1 -- 172.21.15.40:0/39539422 wait complete. 2024-04-21T23:15:16.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.618+0000 7f81a2ea3640 1 Processor -- start 2024-04-21T23:15:16.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.618+0000 7f81a2ea3640 1 -- start start 2024-04-21T23:15:16.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.618+0000 7f81a2ea3640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f819c1006a0 0x7f819c06a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:16.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.618+0000 7f81a2ea3640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f819c101890 0x7f819c19c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:16.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.618+0000 7f81a2ea3640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f819c102250 0x7f819c19ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:16.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.618+0000 7f81a2ea3640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f819c10c300 con 0x7f819c1006a0 2024-04-21T23:15:16.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.618+0000 7f81a2ea3640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f819c10c180 con 0x7f819c101890 2024-04-21T23:15:16.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.618+0000 7f81a2ea3640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f819c10c480 con 0x7f819c102250 2024-04-21T23:15:16.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.618+0000 7f81a0c3f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f819c1006a0 0x7f819c06a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:16.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.618+0000 7f81a0c3f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f819c1006a0 0x7f819c06a910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33418/0 (socket says 172.21.15.40:33418) 2024-04-21T23:15:16.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.618+0000 7f81a0c3f640 1 -- 172.21.15.40:0/2504145147 learned_addr learned my addr 172.21.15.40:0/2504145147 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:15:16.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.618+0000 7f81a1440640 1 --2- 172.21.15.40:0/2504145147 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f819c102250 0x7f819c19ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:16.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.618+0000 7f819bfff640 1 --2- 172.21.15.40:0/2504145147 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f819c101890 0x7f819c19c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:16.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.618+0000 7f81a0c3f640 1 -- 172.21.15.40:0/2504145147 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f819c102250 msgr2=0x7f819c19ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:16.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.618+0000 7f81a0c3f640 1 --2- 172.21.15.40:0/2504145147 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f819c102250 0x7f819c19ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:16.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.618+0000 7f81a0c3f640 1 -- 172.21.15.40:0/2504145147 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f819c101890 msgr2=0x7f819c19c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:16.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.618+0000 7f81a0c3f640 1 --2- 172.21.15.40:0/2504145147 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f819c101890 0x7f819c19c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:16.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.618+0000 7f81a0c3f640 1 -- 172.21.15.40:0/2504145147 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f819c19f710 con 0x7f819c1006a0 2024-04-21T23:15:16.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.618+0000 7f81a1440640 1 --2- 172.21.15.40:0/2504145147 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f819c102250 0x7f819c19ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:15:16.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.618+0000 7f819bfff640 1 --2- 172.21.15.40:0/2504145147 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f819c101890 0x7f819c19c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:15:16.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.619+0000 7f81a0c3f640 1 --2- 172.21.15.40:0/2504145147 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f819c1006a0 0x7f819c06a910 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f818c00d950 tx=0x7f818c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:16.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.619+0000 7f8199ffb640 1 -- 172.21.15.40:0/2504145147 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f818c017020 con 0x7f819c1006a0 2024-04-21T23:15:16.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.619+0000 7f8199ffb640 1 -- 172.21.15.40:0/2504145147 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f818c009c30 con 0x7f819c1006a0 2024-04-21T23:15:16.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.619+0000 7f8199ffb640 1 -- 172.21.15.40:0/2504145147 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f818c00f720 con 0x7f819c1006a0 2024-04-21T23:15:16.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.619+0000 7f81a2ea3640 1 -- 172.21.15.40:0/2504145147 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f819c19fa00 con 0x7f819c1006a0 2024-04-21T23:15:16.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.619+0000 7f81a2ea3640 1 -- 172.21.15.40:0/2504145147 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f819c19fed0 con 0x7f819c1006a0 2024-04-21T23:15:16.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.620+0000 7f81a2ea3640 1 -- 172.21.15.40:0/2504145147 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f819c107a90 con 0x7f819c1006a0 2024-04-21T23:15:16.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.621+0000 7f8199ffb640 1 -- 172.21.15.40:0/2504145147 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f818c01a020 con 0x7f819c1006a0 2024-04-21T23:15:16.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.622+0000 7f8199ffb640 1 --2- 172.21.15.40:0/2504145147 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f81880a2b90 0x7f81880a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:16.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.622+0000 7f819bfff640 1 --2- 172.21.15.40:0/2504145147 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f81880a2b90 0x7f81880a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:16.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.622+0000 7f8199ffb640 1 -- 172.21.15.40:0/2504145147 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f818c0cec50 con 0x7f819c1006a0 2024-04-21T23:15:16.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.622+0000 7f819bfff640 1 --2- 172.21.15.40:0/2504145147 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f81880a2b90 0x7f81880a5050 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8184006010 tx=0x7f818402c8a0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:16.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.626+0000 7f8199ffb640 1 -- 172.21.15.40:0/2504145147 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f818c021020 con 0x7f819c1006a0 2024-04-21T23:15:16.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.810+0000 7f81a2ea3640 1 -- 172.21.15.40:0/2504145147 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f819c02d240 con 0x7f819c1006a0 2024-04-21T23:15:16.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.811+0000 7f8199ffb640 1 -- 172.21.15.40:0/2504145147 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) ==== 93+0+10 (secure 0 0 0) 0x7f818c009de0 con 0x7f819c1006a0 2024-04-21T23:15:16.810 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-04-21T23:15:16.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.813+0000 7f81a2ea3640 1 -- 172.21.15.40:0/2504145147 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f81880a2b90 msgr2=0x7f81880a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:16.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.813+0000 7f81a2ea3640 1 --2- 172.21.15.40:0/2504145147 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f81880a2b90 0x7f81880a5050 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8184006010 tx=0x7f818402c8a0 comp rx=0 tx=0).stop 2024-04-21T23:15:16.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.813+0000 7f81a2ea3640 1 -- 172.21.15.40:0/2504145147 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f819c1006a0 msgr2=0x7f819c06a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:16.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.813+0000 7f81a2ea3640 1 --2- 172.21.15.40:0/2504145147 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f819c1006a0 0x7f819c06a910 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f818c00d950 tx=0x7f818c00de10 comp rx=0 tx=0).stop 2024-04-21T23:15:16.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.813+0000 7f81a2ea3640 1 -- 172.21.15.40:0/2504145147 shutdown_connections 2024-04-21T23:15:16.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.813+0000 7f81a2ea3640 1 --2- 172.21.15.40:0/2504145147 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f81880a2b90 0x7f81880a5050 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:16.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.813+0000 7f81a2ea3640 1 --2- 172.21.15.40:0/2504145147 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f819c101890 0x7f819c19c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:16.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.813+0000 7f81a2ea3640 1 --2- 172.21.15.40:0/2504145147 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f819c102250 0x7f819c19ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:16.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.813+0000 7f81a2ea3640 1 --2- 172.21.15.40:0/2504145147 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f819c1006a0 0x7f819c06a910 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:16.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.813+0000 7f81a2ea3640 1 -- 172.21.15.40:0/2504145147 >> 172.21.15.40:0/2504145147 conn(0x7f819c0fbc30 msgr2=0x7f819c104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:16.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.813+0000 7f81a2ea3640 1 -- 172.21.15.40:0/2504145147 shutdown_connections 2024-04-21T23:15:16.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.813+0000 7f81a2ea3640 1 -- 172.21.15.40:0/2504145147 wait complete. 2024-04-21T23:15:16.823 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:15:16.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.926+0000 7f05e8177640 1 -- 172.21.15.40:0/627241895 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05e00ff8a0 msgr2=0x7f05e00ffd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:16.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.926+0000 7f05e8177640 1 --2- 172.21.15.40:0/627241895 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05e00ff8a0 0x7f05e00ffd20 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f05d0009900 tx=0x7f05d002bc30 comp rx=0 tx=0).stop 2024-04-21T23:15:16.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.926+0000 7f05e8177640 1 -- 172.21.15.40:0/627241895 shutdown_connections 2024-04-21T23:15:16.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.926+0000 7f05e8177640 1 --2- 172.21.15.40:0/627241895 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f05e0100260 0x7f05e0109e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:16.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.926+0000 7f05e8177640 1 --2- 172.21.15.40:0/627241895 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05e00ff8a0 0x7f05e00ffd20 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:16.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.926+0000 7f05e8177640 1 --2- 172.21.15.40:0/627241895 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05e0102e30 0x7f05e0103250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:16.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.926+0000 7f05e8177640 1 -- 172.21.15.40:0/627241895 >> 172.21.15.40:0/627241895 conn(0x7f05e00fb420 msgr2=0x7f05e00fd860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:16.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.926+0000 7f05e8177640 1 -- 172.21.15.40:0/627241895 shutdown_connections 2024-04-21T23:15:16.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.926+0000 7f05e8177640 1 -- 172.21.15.40:0/627241895 wait complete. 2024-04-21T23:15:16.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.926+0000 7f05e8177640 1 Processor -- start 2024-04-21T23:15:16.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.926+0000 7f05e8177640 1 -- start start 2024-04-21T23:15:16.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.927+0000 7f05e8177640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05e00ff8a0 0x7f05e0102530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:16.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.927+0000 7f05e8177640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f05e0100260 0x7f05e0100b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:16.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.927+0000 7f05e8177640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05e0102e30 0x7f05e01010b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:16.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.927+0000 7f05e8177640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f05e010d560 con 0x7f05e0102e30 2024-04-21T23:15:16.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.927+0000 7f05e8177640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f05e010d3e0 con 0x7f05e00ff8a0 2024-04-21T23:15:16.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.927+0000 7f05e8177640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f05e010d6e0 con 0x7f05e0100260 2024-04-21T23:15:16.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.927+0000 7f05e6714640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05e0102e30 0x7f05e01010b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:16.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.927+0000 7f05e6714640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05e0102e30 0x7f05e01010b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33428/0 (socket says 172.21.15.40:33428) 2024-04-21T23:15:16.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.927+0000 7f05e6714640 1 -- 172.21.15.40:0/2527324952 learned_addr learned my addr 172.21.15.40:0/2527324952 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:15:16.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.927+0000 7f05e5f13640 1 --2- 172.21.15.40:0/2527324952 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05e00ff8a0 0x7f05e0102530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:16.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.927+0000 7f05e5712640 1 --2- 172.21.15.40:0/2527324952 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f05e0100260 0x7f05e0100b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:16.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.927+0000 7f05e6714640 1 -- 172.21.15.40:0/2527324952 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f05e0100260 msgr2=0x7f05e0100b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:16.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.927+0000 7f05e6714640 1 --2- 172.21.15.40:0/2527324952 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f05e0100260 0x7f05e0100b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:16.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.927+0000 7f05e6714640 1 -- 172.21.15.40:0/2527324952 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05e00ff8a0 msgr2=0x7f05e0102530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:16.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.927+0000 7f05e6714640 1 --2- 172.21.15.40:0/2527324952 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05e00ff8a0 0x7f05e0102530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:16.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.927+0000 7f05e6714640 1 -- 172.21.15.40:0/2527324952 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05e0101940 con 0x7f05e0102e30 2024-04-21T23:15:16.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.927+0000 7f05e6714640 1 --2- 172.21.15.40:0/2527324952 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05e0102e30 0x7f05e01010b0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f05d800b970 tx=0x7f05d800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:16.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.927+0000 7f05d6ffd640 1 -- 172.21.15.40:0/2527324952 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05d8019070 con 0x7f05e0102e30 2024-04-21T23:15:16.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.927+0000 7f05d6ffd640 1 -- 172.21.15.40:0/2527324952 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f05d800cb40 con 0x7f05e0102e30 2024-04-21T23:15:16.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.927+0000 7f05d6ffd640 1 -- 172.21.15.40:0/2527324952 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05d8007650 con 0x7f05e0102e30 2024-04-21T23:15:16.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.927+0000 7f05e8177640 1 -- 172.21.15.40:0/2527324952 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05e01a1c30 con 0x7f05e0102e30 2024-04-21T23:15:16.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.928+0000 7f05e8177640 1 -- 172.21.15.40:0/2527324952 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05e01a2150 con 0x7f05e0102e30 2024-04-21T23:15:16.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.928+0000 7f05e8177640 1 -- 172.21.15.40:0/2527324952 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05e00689b0 con 0x7f05e0102e30 2024-04-21T23:15:16.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.929+0000 7f05d6ffd640 1 -- 172.21.15.40:0/2527324952 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f05d801c070 con 0x7f05e0102e30 2024-04-21T23:15:16.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.930+0000 7f05d6ffd640 1 --2- 172.21.15.40:0/2527324952 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f05cc0a2c10 0x7f05cc0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:16.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.930+0000 7f05e5f13640 1 --2- 172.21.15.40:0/2527324952 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f05cc0a2c10 0x7f05cc0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:16.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.930+0000 7f05d6ffd640 1 -- 172.21.15.40:0/2527324952 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f05d80ce130 con 0x7f05e0102e30 2024-04-21T23:15:16.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.930+0000 7f05e5f13640 1 --2- 172.21.15.40:0/2527324952 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f05cc0a2c10 0x7f05cc0a50d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f05e0069b20 tx=0x7f05dc005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:16.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:16.935+0000 7f05d6ffd640 1 -- 172.21.15.40:0/2527324952 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f05d809aee0 con 0x7f05e0102e30 2024-04-21T23:15:17.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:17.156+0000 7f05e8177640 1 -- 172.21.15.40:0/2527324952 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f05e002d170 con 0x7f05e0102e30 2024-04-21T23:15:17.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:17.157+0000 7f05d6ffd640 1 -- 172.21.15.40:0/2527324952 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7f05d809f1e0 con 0x7f05e0102e30 2024-04-21T23:15:17.156 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:15:17.156 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-04-21T23:15:17.156 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 1 2024-04-21T23:15:17.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:17.159+0000 7f05e8177640 1 -- 172.21.15.40:0/2527324952 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f05cc0a2c10 msgr2=0x7f05cc0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:17.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:17.159+0000 7f05e8177640 1 --2- 172.21.15.40:0/2527324952 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f05cc0a2c10 0x7f05cc0a50d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f05e0069b20 tx=0x7f05dc005f90 comp rx=0 tx=0).stop 2024-04-21T23:15:17.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:17.159+0000 7f05e8177640 1 -- 172.21.15.40:0/2527324952 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05e0102e30 msgr2=0x7f05e01010b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:17.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:17.159+0000 7f05e8177640 1 --2- 172.21.15.40:0/2527324952 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05e0102e30 0x7f05e01010b0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f05d800b970 tx=0x7f05d800be30 comp rx=0 tx=0).stop 2024-04-21T23:15:17.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:17.159+0000 7f05e8177640 1 -- 172.21.15.40:0/2527324952 shutdown_connections 2024-04-21T23:15:17.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:17.159+0000 7f05e8177640 1 --2- 172.21.15.40:0/2527324952 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f05cc0a2c10 0x7f05cc0a50d0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:17.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:17.159+0000 7f05e8177640 1 --2- 172.21.15.40:0/2527324952 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05e0102e30 0x7f05e01010b0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:17.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:17.159+0000 7f05e8177640 1 --2- 172.21.15.40:0/2527324952 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f05e0100260 0x7f05e0100b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:17.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:17.159+0000 7f05e8177640 1 --2- 172.21.15.40:0/2527324952 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05e00ff8a0 0x7f05e0102530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:17.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:17.159+0000 7f05e8177640 1 -- 172.21.15.40:0/2527324952 >> 172.21.15.40:0/2527324952 conn(0x7f05e00fb420 msgr2=0x7f05e0105df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:17.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:17.159+0000 7f05e8177640 1 -- 172.21.15.40:0/2527324952 shutdown_connections 2024-04-21T23:15:17.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:17.159+0000 7f05e8177640 1 -- 172.21.15.40:0/2527324952 wait complete. 2024-04-21T23:15:17.167 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:16 smithi040 ceph-mon[27984]: Deploying daemon mds.l on smithi174 2024-04-21T23:15:17.167 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:16 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/3673862127' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:15:17.168 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:16 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/2504145147' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:15:17.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:16 smithi111 ceph-mon[31085]: Deploying daemon mds.l on smithi174 2024-04-21T23:15:17.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:16 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/3673862127' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:15:17.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:16 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/2504145147' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:15:17.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:16 smithi174 ceph-mon[31409]: Deploying daemon mds.l on smithi174 2024-04-21T23:15:17.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:16 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/3673862127' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:15:17.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:16 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/2504145147' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:15:17.890 INFO:journalctl@ceph.mds.l.smithi174.stdout:Apr 21 23:15:17 smithi174 systemd[1]: Starting Ceph mds.l for d814e4dc-0033-11ef-bc93-c7b262605968... 2024-04-21T23:15:18.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:17 smithi040 ceph-mon[27984]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:18.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:17 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/2527324952' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:15:18.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:17 smithi174 ceph-mon[31409]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:18.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:17 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/2527324952' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:15:18.194 INFO:journalctl@ceph.mds.l.smithi174.stdout:Apr 21 23:15:17 smithi174 podman[61162]: 2024-04-21 23:15:17.841788169 +0000 UTC m=+0.018367442 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-21T23:15:18.194 INFO:journalctl@ceph.mds.l.smithi174.stdout:Apr 21 23:15:17 smithi174 podman[61162]: 2024-04-21 23:15:17.966838209 +0000 UTC m=+0.143417462 container create 816c950d8d17ddb7b2cd1b3bde65e09c161fb657cbad216be5cd018b91a1834f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-l, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, RELEASE=squid-820a2f8) 2024-04-21T23:15:18.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:17 smithi111 ceph-mon[31085]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:18.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:17 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/2527324952' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:15:18.694 INFO:journalctl@ceph.mds.l.smithi174.stdout:Apr 21 23:15:18 smithi174 podman[61162]: 2024-04-21 23:15:18.227814748 +0000 UTC m=+0.404394001 container init 816c950d8d17ddb7b2cd1b3bde65e09c161fb657cbad216be5cd018b91a1834f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-l, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, RELEASE=squid-820a2f8, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-21T23:15:18.694 INFO:journalctl@ceph.mds.l.smithi174.stdout:Apr 21 23:15:18 smithi174 podman[61162]: 2024-04-21 23:15:18.230266374 +0000 UTC m=+0.406845627 container start 816c950d8d17ddb7b2cd1b3bde65e09c161fb657cbad216be5cd018b91a1834f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-l, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=squid-820a2f8, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2) 2024-04-21T23:15:18.694 INFO:journalctl@ceph.mds.l.smithi174.stdout:Apr 21 23:15:18 smithi174 ceph-mds[61181]: -- 172.21.15.174:0/2943531886 <== mon.1 v2:172.21.15.174:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56271dff5a40 con 0x56271dfc8c00 2024-04-21T23:15:18.694 INFO:journalctl@ceph.mds.l.smithi174.stdout:Apr 21 23:15:18 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-l[61177]: starting mds.l at 2024-04-21T23:15:18.694 INFO:journalctl@ceph.mds.l.smithi174.stdout:Apr 21 23:15:18 smithi174 bash[61162]: 816c950d8d17ddb7b2cd1b3bde65e09c161fb657cbad216be5cd018b91a1834f 2024-04-21T23:15:18.694 INFO:journalctl@ceph.mds.l.smithi174.stdout:Apr 21 23:15:18 smithi174 systemd[1]: Started Ceph mds.l for d814e4dc-0033-11ef-bc93-c7b262605968. 2024-04-21T23:15:19.170 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:15:19.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:19.285+0000 7f1cfa7bf640 1 -- 172.21.15.40:0/241650930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1cf4103360 msgr2=0x7f1cf4109bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:19.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:19.285+0000 7f1cfa7bf640 1 --2- 172.21.15.40:0/241650930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1cf4103360 0x7f1cf4109bf0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f1cec009ca0 tx=0x7f1cec02c0f0 comp rx=0 tx=0).stop 2024-04-21T23:15:19.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:19.285+0000 7f1cfa7bf640 1 -- 172.21.15.40:0/241650930 shutdown_connections 2024-04-21T23:15:19.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:19.285+0000 7f1cfa7bf640 1 --2- 172.21.15.40:0/241650930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1cf4103360 0x7f1cf4109bf0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:19.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:19.285+0000 7f1cfa7bf640 1 --2- 172.21.15.40:0/241650930 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1cf41029a0 0x7f1cf4102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:19.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:19.285+0000 7f1cfa7bf640 1 --2- 172.21.15.40:0/241650930 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1cf41017b0 0x7f1cf4101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:19.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:19.285+0000 7f1cfa7bf640 1 -- 172.21.15.40:0/241650930 >> 172.21.15.40:0/241650930 conn(0x7f1cf40fcd40 msgr2=0x7f1cf40ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:19.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:19.285+0000 7f1cfa7bf640 1 -- 172.21.15.40:0/241650930 shutdown_connections 2024-04-21T23:15:19.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:19.285+0000 7f1cfa7bf640 1 -- 172.21.15.40:0/241650930 wait complete. 2024-04-21T23:15:19.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:19.286+0000 7f1cfa7bf640 1 Processor -- start 2024-04-21T23:15:19.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:19.286+0000 7f1cfa7bf640 1 -- start start 2024-04-21T23:15:19.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:19.286+0000 7f1cfa7bf640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1cf41017b0 0x7f1cf4197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:19.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:19.286+0000 7f1cfa7bf640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1cf41029a0 0x7f1cf4197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:19.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:19.286+0000 7f1cfa7bf640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1cf4103360 0x7f1cf419efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:19.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:19.286+0000 7f1cfa7bf640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1cf410d410 con 0x7f1cf41029a0 2024-04-21T23:15:19.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:19.286+0000 7f1cfa7bf640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f1cf410d290 con 0x7f1cf4103360 2024-04-21T23:15:19.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:19.286+0000 7f1cfa7bf640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1cf410d590 con 0x7f1cf41017b0 2024-04-21T23:15:19.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:19.287+0000 7f1cf37fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1cf41029a0 0x7f1cf4197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:19.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:19.287+0000 7f1cf37fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1cf41029a0 0x7f1cf4197f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33446/0 (socket says 172.21.15.40:33446) 2024-04-21T23:15:19.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:19.287+0000 7f1cf37fe640 1 -- 172.21.15.40:0/666082377 learned_addr learned my addr 172.21.15.40:0/666082377 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:15:19.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:19.287+0000 7f1cf8d5c640 1 --2- 172.21.15.40:0/666082377 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1cf4103360 0x7f1cf419efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:19.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:19.287+0000 7f1cf3fff640 1 --2- 172.21.15.40:0/666082377 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1cf41017b0 0x7f1cf4197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:19.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:19.287+0000 7f1cf37fe640 1 -- 172.21.15.40:0/666082377 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1cf41017b0 msgr2=0x7f1cf4197a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:19.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:19.287+0000 7f1cf37fe640 1 --2- 172.21.15.40:0/666082377 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1cf41017b0 0x7f1cf4197a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:19.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:19.287+0000 7f1cf37fe640 1 -- 172.21.15.40:0/666082377 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1cf4103360 msgr2=0x7f1cf419efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:19.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:19.287+0000 7f1cf37fe640 1 --2- 172.21.15.40:0/666082377 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1cf4103360 0x7f1cf419efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:19.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:19.287+0000 7f1cf37fe640 1 -- 172.21.15.40:0/666082377 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cf419f640 con 0x7f1cf41029a0 2024-04-21T23:15:19.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:19.287+0000 7f1cf37fe640 1 --2- 172.21.15.40:0/666082377 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1cf41029a0 0x7f1cf4197f60 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1ce400d950 tx=0x7f1ce400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:19.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:19.287+0000 7f1cf17fa640 1 -- 172.21.15.40:0/666082377 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ce4017070 con 0x7f1cf41029a0 2024-04-21T23:15:19.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:19.287+0000 7f1cfa7bf640 1 -- 172.21.15.40:0/666082377 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cf419f930 con 0x7f1cf41029a0 2024-04-21T23:15:19.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:19.287+0000 7f1cf17fa640 1 -- 172.21.15.40:0/666082377 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ce4009c30 con 0x7f1cf41029a0 2024-04-21T23:15:19.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:19.287+0000 7f1cf17fa640 1 -- 172.21.15.40:0/666082377 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ce400f6b0 con 0x7f1cf41029a0 2024-04-21T23:15:19.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:19.287+0000 7f1cfa7bf640 1 -- 172.21.15.40:0/666082377 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cf419fe80 con 0x7f1cf41029a0 2024-04-21T23:15:19.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:19.288+0000 7f1cfa7bf640 1 -- 172.21.15.40:0/666082377 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1cf4108a00 con 0x7f1cf41029a0 2024-04-21T23:15:19.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:19.289+0000 7f1cf17fa640 1 -- 172.21.15.40:0/666082377 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f1ce40109d0 con 0x7f1cf41029a0 2024-04-21T23:15:19.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:19.290+0000 7f1cf17fa640 1 --2- 172.21.15.40:0/666082377 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f1cdc0a2b90 0x7f1cdc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:19.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:19.290+0000 7f1cf3fff640 1 --2- 172.21.15.40:0/666082377 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f1cdc0a2b90 0x7f1cdc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:19.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:19.290+0000 7f1cf17fa640 1 -- 172.21.15.40:0/666082377 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f1ce40ce870 con 0x7f1cf41029a0 2024-04-21T23:15:19.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:19.291+0000 7f1cf3fff640 1 --2- 172.21.15.40:0/666082377 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f1cdc0a2b90 0x7f1cdc0a5050 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1cf4102800 tx=0x7f1ce8005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:19.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:19.294+0000 7f1cf17fa640 1 -- 172.21.15.40:0/666082377 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f1ce4009de0 con 0x7f1cf41029a0 2024-04-21T23:15:19.388 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:19 smithi111 ceph-mon[31085]: mds.? [v2:172.21.15.174:6832/3438493203,v1:172.21.15.174:6833/3438493203] up:boot 2024-04-21T23:15:19.389 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:19 smithi111 ceph-mon[31085]: fsmap 1 up:standby 2024-04-21T23:15:19.389 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:19 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-21T23:15:19.389 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:19 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:19.389 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:19 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:19.389 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:19 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:19.389 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:19 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:15:19.389 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:19 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:15:19.389 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:19 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:15:19.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:19.517+0000 7f1cfa7bf640 1 -- 172.21.15.40:0/666082377 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1cf402d170 con 0x7f1cf41029a0 2024-04-21T23:15:19.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:19.518+0000 7f1cf17fa640 1 -- 172.21.15.40:0/666082377 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7f1ce409bbd0 con 0x7f1cf41029a0 2024-04-21T23:15:19.517 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:15:19.517 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/3438493203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":3438493203},{"type":"v1","addr":"172.21.15.174:6833","nonce":3438493203}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-21T23:15:19.517 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 2 2024-04-21T23:15:19.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:19.519+0000 7f1cfa7bf640 1 -- 172.21.15.40:0/666082377 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f1cdc0a2b90 msgr2=0x7f1cdc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:19.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:19.519+0000 7f1cfa7bf640 1 --2- 172.21.15.40:0/666082377 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f1cdc0a2b90 0x7f1cdc0a5050 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1cf4102800 tx=0x7f1ce8005f90 comp rx=0 tx=0).stop 2024-04-21T23:15:19.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:19.519+0000 7f1cfa7bf640 1 -- 172.21.15.40:0/666082377 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1cf41029a0 msgr2=0x7f1cf4197f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:19.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:19.520+0000 7f1cfa7bf640 1 --2- 172.21.15.40:0/666082377 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1cf41029a0 0x7f1cf4197f60 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1ce400d950 tx=0x7f1ce400de10 comp rx=0 tx=0).stop 2024-04-21T23:15:19.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:19.520+0000 7f1cfa7bf640 1 -- 172.21.15.40:0/666082377 shutdown_connections 2024-04-21T23:15:19.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:19.520+0000 7f1cfa7bf640 1 --2- 172.21.15.40:0/666082377 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f1cdc0a2b90 0x7f1cdc0a5050 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:19.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:19.520+0000 7f1cfa7bf640 1 --2- 172.21.15.40:0/666082377 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1cf4103360 0x7f1cf419efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:19.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:19.520+0000 7f1cfa7bf640 1 --2- 172.21.15.40:0/666082377 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1cf41029a0 0x7f1cf4197f60 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:19.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:19.520+0000 7f1cfa7bf640 1 --2- 172.21.15.40:0/666082377 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1cf41017b0 0x7f1cf4197a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:19.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:19.520+0000 7f1cfa7bf640 1 -- 172.21.15.40:0/666082377 >> 172.21.15.40:0/666082377 conn(0x7f1cf40fcd40 msgr2=0x7f1cf4105bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:19.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:19.520+0000 7f1cfa7bf640 1 -- 172.21.15.40:0/666082377 shutdown_connections 2024-04-21T23:15:19.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:19.520+0000 7f1cfa7bf640 1 -- 172.21.15.40:0/666082377 wait complete. 2024-04-21T23:15:19.528 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:19 smithi040 ceph-mon[27984]: mds.? [v2:172.21.15.174:6832/3438493203,v1:172.21.15.174:6833/3438493203] up:boot 2024-04-21T23:15:19.528 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:19 smithi040 ceph-mon[27984]: fsmap 1 up:standby 2024-04-21T23:15:19.529 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:19 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-21T23:15:19.529 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:19 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:19.529 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:19 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:19.529 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:19 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:19.529 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:19 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:15:19.529 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:19 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:15:19.529 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:19 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:15:19.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:19 smithi174 ceph-mon[31409]: mds.? [v2:172.21.15.174:6832/3438493203,v1:172.21.15.174:6833/3438493203] up:boot 2024-04-21T23:15:19.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:19 smithi174 ceph-mon[31409]: fsmap 1 up:standby 2024-04-21T23:15:19.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:19 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-21T23:15:19.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:19 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:19.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:19 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:19.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:19 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:19.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:19 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:15:19.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:19 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:15:19.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:19 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:15:20.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:20 smithi111 ceph-mon[31085]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:20.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:20 smithi111 ceph-mon[31085]: Deploying daemon mds.k on smithi111 2024-04-21T23:15:20.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:20 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/666082377' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:15:20.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:20 smithi040 ceph-mon[27984]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:20.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:20 smithi040 ceph-mon[27984]: Deploying daemon mds.k on smithi111 2024-04-21T23:15:20.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:20 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/666082377' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:15:20.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:20 smithi174 ceph-mon[31409]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:20.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:20 smithi174 ceph-mon[31409]: Deploying daemon mds.k on smithi111 2024-04-21T23:15:20.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:20 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/666082377' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:15:20.944 INFO:journalctl@ceph.mds.k.smithi111.stdout:Apr 21 23:15:20 smithi111 systemd[1]: Starting Ceph mds.k for d814e4dc-0033-11ef-bc93-c7b262605968... 2024-04-21T23:15:21.367 INFO:journalctl@ceph.mds.k.smithi111.stdout:Apr 21 23:15:21 smithi111 podman[60460]: 2024-04-21 23:15:20.907628676 +0000 UTC m=+0.017825790 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-21T23:15:21.367 INFO:journalctl@ceph.mds.k.smithi111.stdout:Apr 21 23:15:21 smithi111 podman[60460]: 2024-04-21 23:15:21.02328116 +0000 UTC m=+0.133478250 container create 2fd4155c0bf06458b1c6ec9d9b05037039cd48499eaff7e109a62961e2b1f86b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-k, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-820a2f8, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5) 2024-04-21T23:15:21.367 INFO:journalctl@ceph.mds.k.smithi111.stdout:Apr 21 23:15:21 smithi111 podman[60460]: 2024-04-21 23:15:21.368128355 +0000 UTC m=+0.478325446 container init 2fd4155c0bf06458b1c6ec9d9b05037039cd48499eaff7e109a62961e2b1f86b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-k, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, RELEASE=squid-820a2f8, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-21T23:15:21.531 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:15:21.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:21.639+0000 7fa7e8937640 1 -- 172.21.15.40:0/427913329 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7e4107460 msgr2=0x7fa7e41098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:21.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:21.639+0000 7fa7e8937640 1 --2- 172.21.15.40:0/427913329 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7e4107460 0x7fa7e41098b0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa7dc009ca0 tx=0x7fa7dc02c0f0 comp rx=0 tx=0).stop 2024-04-21T23:15:21.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:21.640+0000 7fa7e8937640 1 -- 172.21.15.40:0/427913329 shutdown_connections 2024-04-21T23:15:21.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:21.640+0000 7fa7e8937640 1 --2- 172.21.15.40:0/427913329 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7e4104b30 0x7fa7e4106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:21.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:21.640+0000 7fa7e8937640 1 --2- 172.21.15.40:0/427913329 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7e4107460 0x7fa7e41098b0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:21.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:21.640+0000 7fa7e8937640 1 --2- 172.21.15.40:0/427913329 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa7e4102200 0x7fa7e41045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:21.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:21.640+0000 7fa7e8937640 1 -- 172.21.15.40:0/427913329 >> 172.21.15.40:0/427913329 conn(0x7fa7e40fbbf0 msgr2=0x7fa7e40fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:21.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:21.640+0000 7fa7e8937640 1 -- 172.21.15.40:0/427913329 shutdown_connections 2024-04-21T23:15:21.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:21.640+0000 7fa7e8937640 1 -- 172.21.15.40:0/427913329 wait complete. 2024-04-21T23:15:21.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:21.640+0000 7fa7e8937640 1 Processor -- start 2024-04-21T23:15:21.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:21.641+0000 7fa7e8937640 1 -- start start 2024-04-21T23:15:21.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:21.641+0000 7fa7e8937640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7e4102200 0x7fa7e4101b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:21.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:21.641+0000 7fa7e8937640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa7e4104b30 0x7fa7e41001b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:21.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:21.641+0000 7fa7e8937640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7e4107460 0x7fa7e41006f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:21.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:21.641+0000 7fa7e8937640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa7e410d400 con 0x7fa7e4102200 2024-04-21T23:15:21.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:21.641+0000 7fa7e8937640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fa7e410d280 con 0x7fa7e4107460 2024-04-21T23:15:21.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:21.641+0000 7fa7e8937640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa7e410d580 con 0x7fa7e4104b30 2024-04-21T23:15:21.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:21.641+0000 7fa7e259c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7e4102200 0x7fa7e4101b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:21.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:21.641+0000 7fa7e259c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7e4102200 0x7fa7e4101b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33468/0 (socket says 172.21.15.40:33468) 2024-04-21T23:15:21.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:21.641+0000 7fa7e259c640 1 -- 172.21.15.40:0/589572395 learned_addr learned my addr 172.21.15.40:0/589572395 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:15:21.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:21.641+0000 7fa7e1d9b640 1 --2- 172.21.15.40:0/589572395 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa7e4104b30 0x7fa7e41001b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:21.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:21.641+0000 7fa7e2d9d640 1 --2- 172.21.15.40:0/589572395 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7e4107460 0x7fa7e41006f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:21.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:21.641+0000 7fa7e259c640 1 -- 172.21.15.40:0/589572395 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa7e4104b30 msgr2=0x7fa7e41001b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:21.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:21.641+0000 7fa7e259c640 1 --2- 172.21.15.40:0/589572395 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa7e4104b30 0x7fa7e41001b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:21.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:21.641+0000 7fa7e259c640 1 -- 172.21.15.40:0/589572395 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7e4107460 msgr2=0x7fa7e41006f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:21.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:21.641+0000 7fa7e259c640 1 --2- 172.21.15.40:0/589572395 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7e4107460 0x7fa7e41006f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:21.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:21.641+0000 7fa7e259c640 1 -- 172.21.15.40:0/589572395 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7e4100f80 con 0x7fa7e4102200 2024-04-21T23:15:21.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:21.641+0000 7fa7e1d9b640 1 --2- 172.21.15.40:0/589572395 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa7e4104b30 0x7fa7e41001b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:15:21.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:21.641+0000 7fa7e2d9d640 1 --2- 172.21.15.40:0/589572395 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7e4107460 0x7fa7e41006f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:15:21.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:21.641+0000 7fa7e259c640 1 --2- 172.21.15.40:0/589572395 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7e4102200 0x7fa7e4101b70 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa7d400d950 tx=0x7fa7d400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:21.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:21.641+0000 7fa7d37fe640 1 -- 172.21.15.40:0/589572395 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7d4017070 con 0x7fa7e4102200 2024-04-21T23:15:21.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:21.642+0000 7fa7e8937640 1 -- 172.21.15.40:0/589572395 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7e41a1b70 con 0x7fa7e4102200 2024-04-21T23:15:21.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:21.642+0000 7fa7d37fe640 1 -- 172.21.15.40:0/589572395 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7d4009c30 con 0x7fa7e4102200 2024-04-21T23:15:21.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:21.642+0000 7fa7e8937640 1 -- 172.21.15.40:0/589572395 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7e41a2110 con 0x7fa7e4102200 2024-04-21T23:15:21.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:21.642+0000 7fa7d37fe640 1 -- 172.21.15.40:0/589572395 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7d400f6b0 con 0x7fa7e4102200 2024-04-21T23:15:21.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:21.643+0000 7fa7e8937640 1 -- 172.21.15.40:0/589572395 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7e40fd6e0 con 0x7fa7e4102200 2024-04-21T23:15:21.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:21.645+0000 7fa7d37fe640 1 -- 172.21.15.40:0/589572395 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fa7d40109d0 con 0x7fa7e4102200 2024-04-21T23:15:21.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:21.645+0000 7fa7d37fe640 1 --2- 172.21.15.40:0/589572395 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fa7cc0a2b90 0x7fa7cc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:21.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:21.645+0000 7fa7e1d9b640 1 --2- 172.21.15.40:0/589572395 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fa7cc0a2b90 0x7fa7cc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:21.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:21.645+0000 7fa7d37fe640 1 -- 172.21.15.40:0/589572395 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fa7d40ce850 con 0x7fa7e4102200 2024-04-21T23:15:21.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:21.646+0000 7fa7e1d9b640 1 --2- 172.21.15.40:0/589572395 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fa7cc0a2b90 0x7fa7cc0a5050 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa7e41018e0 tx=0x7fa7d8005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:21.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:21.648+0000 7fa7d37fe640 1 -- 172.21.15.40:0/589572395 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fa7d4009de0 con 0x7fa7e4102200 2024-04-21T23:15:21.694 INFO:journalctl@ceph.mds.k.smithi111.stdout:Apr 21 23:15:21 smithi111 podman[60460]: 2024-04-21 23:15:21.370957285 +0000 UTC m=+0.481154385 container start 2fd4155c0bf06458b1c6ec9d9b05037039cd48499eaff7e109a62961e2b1f86b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-k, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-820a2f8) 2024-04-21T23:15:21.694 INFO:journalctl@ceph.mds.k.smithi111.stdout:Apr 21 23:15:21 smithi111 ceph-mds[60479]: -- 172.21.15.111:0/327058939 <== mon.2 v2:172.21.15.111:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d5e1d57a40 con 0x55d5e1d2ac00 2024-04-21T23:15:21.694 INFO:journalctl@ceph.mds.k.smithi111.stdout:Apr 21 23:15:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-k[60475]: starting mds.k at 2024-04-21T23:15:21.694 INFO:journalctl@ceph.mds.k.smithi111.stdout:Apr 21 23:15:21 smithi111 bash[60460]: 2fd4155c0bf06458b1c6ec9d9b05037039cd48499eaff7e109a62961e2b1f86b 2024-04-21T23:15:21.694 INFO:journalctl@ceph.mds.k.smithi111.stdout:Apr 21 23:15:21 smithi111 systemd[1]: Started Ceph mds.k for d814e4dc-0033-11ef-bc93-c7b262605968. 2024-04-21T23:15:21.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:21.871+0000 7fa7e8937640 1 -- 172.21.15.40:0/589572395 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa7e402d170 con 0x7fa7e4102200 2024-04-21T23:15:21.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:21.871+0000 7fa7d37fe640 1 -- 172.21.15.40:0/589572395 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7fa7e402d170 con 0x7fa7e4102200 2024-04-21T23:15:21.871 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:15:21.871 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/3438493203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":3438493203},{"type":"v1","addr":"172.21.15.174:6833","nonce":3438493203}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-21T23:15:21.871 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 2 2024-04-21T23:15:21.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:21.874+0000 7fa7e8937640 1 -- 172.21.15.40:0/589572395 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fa7cc0a2b90 msgr2=0x7fa7cc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:21.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:21.874+0000 7fa7e8937640 1 --2- 172.21.15.40:0/589572395 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fa7cc0a2b90 0x7fa7cc0a5050 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa7e41018e0 tx=0x7fa7d8005f90 comp rx=0 tx=0).stop 2024-04-21T23:15:21.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:21.874+0000 7fa7e8937640 1 -- 172.21.15.40:0/589572395 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7e4102200 msgr2=0x7fa7e4101b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:21.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:21.874+0000 7fa7e8937640 1 --2- 172.21.15.40:0/589572395 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7e4102200 0x7fa7e4101b70 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa7d400d950 tx=0x7fa7d400de10 comp rx=0 tx=0).stop 2024-04-21T23:15:21.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:21.874+0000 7fa7e8937640 1 -- 172.21.15.40:0/589572395 shutdown_connections 2024-04-21T23:15:21.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:21.874+0000 7fa7e8937640 1 --2- 172.21.15.40:0/589572395 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fa7cc0a2b90 0x7fa7cc0a5050 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:21.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:21.874+0000 7fa7e8937640 1 --2- 172.21.15.40:0/589572395 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7e4107460 0x7fa7e41006f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:21.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:21.874+0000 7fa7e8937640 1 --2- 172.21.15.40:0/589572395 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa7e4104b30 0x7fa7e41001b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:21.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:21.874+0000 7fa7e8937640 1 --2- 172.21.15.40:0/589572395 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7e4102200 0x7fa7e4101b70 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:21.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:21.874+0000 7fa7e8937640 1 -- 172.21.15.40:0/589572395 >> 172.21.15.40:0/589572395 conn(0x7fa7e40fbbf0 msgr2=0x7fa7e4102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:21.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:21.874+0000 7fa7e8937640 1 -- 172.21.15.40:0/589572395 shutdown_connections 2024-04-21T23:15:21.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:21.874+0000 7fa7e8937640 1 -- 172.21.15.40:0/589572395 wait complete. 2024-04-21T23:15:22.516 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:22 smithi111 ceph-mon[31085]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:22.516 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:22 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:22.516 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:22 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:22.516 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:22 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:22.516 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:22 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:15:22.516 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:22 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:15:22.517 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:22 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:15:22.517 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:22 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/589572395' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:15:22.517 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:22 smithi040 ceph-mon[27984]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:22.517 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:22 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:22.517 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:22 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:22.517 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:22 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:22.517 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:22 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:15:22.517 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:22 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:15:22.517 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:22 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:15:22.517 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:22 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/589572395' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:15:22.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:22 smithi174 ceph-mon[31409]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:22.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:22 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:22.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:22 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:22.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:22 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:22.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:22 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:15:22.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:22 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:15:22.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:22 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:15:22.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:22 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/589572395' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:15:23.666 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:23 smithi040 ceph-mon[27984]: Deploying daemon mds.j on smithi040 2024-04-21T23:15:23.666 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:23 smithi040 ceph-mon[27984]: mds.? [v2:172.21.15.111:6832/3105752678,v1:172.21.15.111:6833/3105752678] up:boot 2024-04-21T23:15:23.666 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:23 smithi040 ceph-mon[27984]: fsmap 2 up:standby 2024-04-21T23:15:23.666 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:23 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-21T23:15:23.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:23 smithi174 ceph-mon[31409]: Deploying daemon mds.j on smithi040 2024-04-21T23:15:23.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:23 smithi174 ceph-mon[31409]: mds.? [v2:172.21.15.111:6832/3105752678,v1:172.21.15.111:6833/3105752678] up:boot 2024-04-21T23:15:23.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:23 smithi174 ceph-mon[31409]: fsmap 2 up:standby 2024-04-21T23:15:23.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:23 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-21T23:15:23.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:23 smithi111 ceph-mon[31085]: Deploying daemon mds.j on smithi040 2024-04-21T23:15:23.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:23 smithi111 ceph-mon[31085]: mds.? [v2:172.21.15.111:6832/3105752678,v1:172.21.15.111:6833/3105752678] up:boot 2024-04-21T23:15:23.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:23 smithi111 ceph-mon[31085]: fsmap 2 up:standby 2024-04-21T23:15:23.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:23 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-21T23:15:23.885 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:15:24.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:24.001+0000 7f107f235640 1 -- 172.21.15.40:0/1857064240 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1078101870 msgr2=0x7f1078101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:24.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:24.001+0000 7f107f235640 1 --2- 172.21.15.40:0/1857064240 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1078101870 0x7f1078101cf0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f1060009900 tx=0x7f106002bc30 comp rx=0 tx=0).stop 2024-04-21T23:15:24.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:24.001+0000 7f107f235640 1 -- 172.21.15.40:0/1857064240 shutdown_connections 2024-04-21T23:15:24.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:24.001+0000 7f107f235640 1 --2- 172.21.15.40:0/1857064240 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1078102230 0x7f1078108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:24.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:24.001+0000 7f107f235640 1 --2- 172.21.15.40:0/1857064240 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1078101870 0x7f1078101cf0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:24.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:24.001+0000 7f107f235640 1 --2- 172.21.15.40:0/1857064240 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1078100680 0x7f1078100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:24.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:24.001+0000 7f107f235640 1 -- 172.21.15.40:0/1857064240 >> 172.21.15.40:0/1857064240 conn(0x7f10780fbbf0 msgr2=0x7f10780fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:24.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:24.001+0000 7f107f235640 1 -- 172.21.15.40:0/1857064240 shutdown_connections 2024-04-21T23:15:24.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:24.001+0000 7f107f235640 1 -- 172.21.15.40:0/1857064240 wait complete. 2024-04-21T23:15:24.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:24.002+0000 7f107f235640 1 Processor -- start 2024-04-21T23:15:24.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:24.002+0000 7f107f235640 1 -- start start 2024-04-21T23:15:24.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:24.002+0000 7f107f235640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1078100680 0x7f1078069440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:24.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:24.002+0000 7f107f235640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1078101870 0x7f107819c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:24.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:24.002+0000 7f107f235640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1078102230 0x7f107819edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:24.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:24.002+0000 7f107f235640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f107810c350 con 0x7f1078102230 2024-04-21T23:15:24.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:24.002+0000 7f107f235640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f107810c1d0 con 0x7f1078100680 2024-04-21T23:15:24.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:24.002+0000 7f107f235640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f107810c4d0 con 0x7f1078101870 2024-04-21T23:15:24.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:24.002+0000 7f107d7d2640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1078102230 0x7f107819edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:24.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:24.002+0000 7f107d7d2640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1078102230 0x7f107819edd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51756/0 (socket says 172.21.15.40:51756) 2024-04-21T23:15:24.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:24.002+0000 7f107d7d2640 1 -- 172.21.15.40:0/146017475 learned_addr learned my addr 172.21.15.40:0/146017475 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:15:24.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:24.002+0000 7f107cfd1640 1 --2- 172.21.15.40:0/146017475 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1078100680 0x7f1078069440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:24.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:24.002+0000 7f1077fff640 1 --2- 172.21.15.40:0/146017475 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1078101870 0x7f107819c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:24.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:24.002+0000 7f107d7d2640 1 -- 172.21.15.40:0/146017475 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1078101870 msgr2=0x7f107819c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:24.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:24.002+0000 7f107d7d2640 1 --2- 172.21.15.40:0/146017475 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1078101870 0x7f107819c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:24.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:24.002+0000 7f107d7d2640 1 -- 172.21.15.40:0/146017475 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1078100680 msgr2=0x7f1078069440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:24.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:24.002+0000 7f107d7d2640 1 --2- 172.21.15.40:0/146017475 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1078100680 0x7f1078069440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:24.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:24.002+0000 7f107d7d2640 1 -- 172.21.15.40:0/146017475 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f107819f6e0 con 0x7f1078102230 2024-04-21T23:15:24.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:24.002+0000 7f107cfd1640 1 --2- 172.21.15.40:0/146017475 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1078100680 0x7f1078069440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:15:24.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:24.002+0000 7f107d7d2640 1 --2- 172.21.15.40:0/146017475 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1078102230 0x7f107819edd0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f107000eaa0 tx=0x7f107000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:24.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:24.002+0000 7f1075ffb640 1 -- 172.21.15.40:0/146017475 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f107000cdb0 con 0x7f1078102230 2024-04-21T23:15:24.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:24.003+0000 7f1075ffb640 1 -- 172.21.15.40:0/146017475 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1070005020 con 0x7f1078102230 2024-04-21T23:15:24.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:24.003+0000 7f107f235640 1 -- 172.21.15.40:0/146017475 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f107819f970 con 0x7f1078102230 2024-04-21T23:15:24.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:24.003+0000 7f1075ffb640 1 -- 172.21.15.40:0/146017475 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10700116b0 con 0x7f1078102230 2024-04-21T23:15:24.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:24.003+0000 7f107f235640 1 -- 172.21.15.40:0/146017475 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f107819fe90 con 0x7f1078102230 2024-04-21T23:15:24.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:24.004+0000 7f107f235640 1 -- 172.21.15.40:0/146017475 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10781079a0 con 0x7f1078102230 2024-04-21T23:15:24.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:24.005+0000 7f1075ffb640 1 -- 172.21.15.40:0/146017475 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f1070005560 con 0x7f1078102230 2024-04-21T23:15:24.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:24.005+0000 7f1075ffb640 1 --2- 172.21.15.40:0/146017475 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f10640a2b90 0x7f10640a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:24.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:24.005+0000 7f1075ffb640 1 -- 172.21.15.40:0/146017475 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f10700cee10 con 0x7f1078102230 2024-04-21T23:15:24.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:24.005+0000 7f107cfd1640 1 --2- 172.21.15.40:0/146017475 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f10640a2b90 0x7f10640a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:24.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:24.006+0000 7f107cfd1640 1 --2- 172.21.15.40:0/146017475 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f10640a2b90 0x7f10640a5050 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f10781016d0 tx=0x7f106800b390 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:24.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:24.010+0000 7f1075ffb640 1 -- 172.21.15.40:0/146017475 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f1070016030 con 0x7f1078102230 2024-04-21T23:15:24.182 INFO:journalctl@ceph.mds.j.smithi040.stdout:Apr 21 23:15:24 smithi040 podman[66039]: 2024-04-21 23:15:24.10342702 +0000 UTC m=+0.148349568 container create bf807801f4277f168efad1a744c587a9718715fb561e58f1401eb4647962aa03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-j, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=squid-820a2f8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True) 2024-04-21T23:15:24.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:24.239+0000 7f107f235640 1 -- 172.21.15.40:0/146017475 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f107802d320 con 0x7f1078102230 2024-04-21T23:15:24.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:24.239+0000 7f1075ffb640 1 -- 172.21.15.40:0/146017475 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2224 (secure 0 0 0) 0x7f107009c210 con 0x7f1078102230 2024-04-21T23:15:24.239 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:15:24.239 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/3438493203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":3438493203},{"type":"v1","addr":"172.21.15.174:6833","nonce":3438493203}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/3105752678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":3105752678},{"type":"v1","addr":"172.21.15.111:6833","nonce":3105752678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-21T23:15:24.239 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 3 2024-04-21T23:15:24.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:24.241+0000 7f107f235640 1 -- 172.21.15.40:0/146017475 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f10640a2b90 msgr2=0x7f10640a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:24.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:24.241+0000 7f107f235640 1 --2- 172.21.15.40:0/146017475 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f10640a2b90 0x7f10640a5050 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f10781016d0 tx=0x7f106800b390 comp rx=0 tx=0).stop 2024-04-21T23:15:24.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:24.241+0000 7f107f235640 1 -- 172.21.15.40:0/146017475 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1078102230 msgr2=0x7f107819edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:24.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:24.241+0000 7f107f235640 1 --2- 172.21.15.40:0/146017475 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1078102230 0x7f107819edd0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f107000eaa0 tx=0x7f107000ef60 comp rx=0 tx=0).stop 2024-04-21T23:15:24.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:24.241+0000 7f107f235640 1 -- 172.21.15.40:0/146017475 shutdown_connections 2024-04-21T23:15:24.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:24.241+0000 7f107f235640 1 --2- 172.21.15.40:0/146017475 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f10640a2b90 0x7f10640a5050 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:24.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:24.241+0000 7f107f235640 1 --2- 172.21.15.40:0/146017475 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1078102230 0x7f107819edd0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:24.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:24.241+0000 7f107f235640 1 --2- 172.21.15.40:0/146017475 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1078101870 0x7f107819c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:24.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:24.241+0000 7f107f235640 1 --2- 172.21.15.40:0/146017475 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1078100680 0x7f1078069440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:24.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:24.241+0000 7f107f235640 1 -- 172.21.15.40:0/146017475 >> 172.21.15.40:0/146017475 conn(0x7f10780fbbf0 msgr2=0x7f1078104b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:24.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:24.242+0000 7f107f235640 1 -- 172.21.15.40:0/146017475 shutdown_connections 2024-04-21T23:15:24.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:24.242+0000 7f107f235640 1 -- 172.21.15.40:0/146017475 wait complete. 2024-04-21T23:15:24.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:24 smithi040 ceph-mon[27984]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:24.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:24 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/146017475' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:15:24.682 INFO:journalctl@ceph.mds.j.smithi040.stdout:Apr 21 23:15:24 smithi040 podman[66039]: 2024-04-21 23:15:24.339967189 +0000 UTC m=+0.384889736 container init bf807801f4277f168efad1a744c587a9718715fb561e58f1401eb4647962aa03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-j, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , RELEASE=squid-820a2f8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5) 2024-04-21T23:15:24.682 INFO:journalctl@ceph.mds.j.smithi040.stdout:Apr 21 23:15:24 smithi040 podman[66039]: 2024-04-21 23:15:24.34266982 +0000 UTC m=+0.387592370 container start bf807801f4277f168efad1a744c587a9718715fb561e58f1401eb4647962aa03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-j, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, ceph=True, RELEASE=squid-820a2f8, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-21T23:15:24.682 INFO:journalctl@ceph.mds.j.smithi040.stdout:Apr 21 23:15:24 smithi040 ceph-mds[66079]: -- 172.21.15.40:0/3216854164 <== mon.0 v2:172.21.15.40:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564eb973da40 con 0x564eb9711000 2024-04-21T23:15:24.683 INFO:journalctl@ceph.mds.j.smithi040.stdout:Apr 21 23:15:24 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-j[66073]: starting mds.j at 2024-04-21T23:15:24.683 INFO:journalctl@ceph.mds.j.smithi040.stdout:Apr 21 23:15:24 smithi040 bash[66039]: bf807801f4277f168efad1a744c587a9718715fb561e58f1401eb4647962aa03 2024-04-21T23:15:24.683 INFO:journalctl@ceph.mds.j.smithi040.stdout:Apr 21 23:15:24 smithi040 systemd[1]: Started Ceph mds.j for d814e4dc-0033-11ef-bc93-c7b262605968. 2024-04-21T23:15:24.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:24 smithi174 ceph-mon[31409]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:24.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:24 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/146017475' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:15:24.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:24 smithi111 ceph-mon[31085]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:24.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:24 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/146017475' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:15:25.838 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:25 smithi040 ceph-mon[27984]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:25.838 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:25 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:25.838 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:25 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:25.838 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:25 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:25.839 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:25 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:15:25.839 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:25 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:15:25.839 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:25 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:15:25.839 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:25 smithi040 ceph-mon[27984]: Deploying daemon mds.d on smithi040 2024-04-21T23:15:25.839 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:25 smithi040 ceph-mon[27984]: mds.? [v2:172.21.15.40:6834/1571948830,v1:172.21.15.40:6835/1571948830] up:boot 2024-04-21T23:15:25.839 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:25 smithi040 ceph-mon[27984]: fsmap 3 up:standby 2024-04-21T23:15:25.839 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:25 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-21T23:15:25.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:25 smithi174 ceph-mon[31409]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:25.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:25 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:25.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:25 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:25.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:25 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:25.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:25 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:15:25.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:25 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:15:25.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:25 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:15:25.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:25 smithi174 ceph-mon[31409]: Deploying daemon mds.d on smithi040 2024-04-21T23:15:25.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:25 smithi174 ceph-mon[31409]: mds.? [v2:172.21.15.40:6834/1571948830,v1:172.21.15.40:6835/1571948830] up:boot 2024-04-21T23:15:25.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:25 smithi174 ceph-mon[31409]: fsmap 3 up:standby 2024-04-21T23:15:25.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:25 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-21T23:15:25.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:25 smithi111 ceph-mon[31085]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:25.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:25 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:25.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:25 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:25.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:25 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:25.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:25 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:15:25.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:25 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:15:25.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:25 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:15:25.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:25 smithi111 ceph-mon[31085]: Deploying daemon mds.d on smithi040 2024-04-21T23:15:25.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:25 smithi111 ceph-mon[31085]: mds.? [v2:172.21.15.40:6834/1571948830,v1:172.21.15.40:6835/1571948830] up:boot 2024-04-21T23:15:25.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:25 smithi111 ceph-mon[31085]: fsmap 3 up:standby 2024-04-21T23:15:25.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:25 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-21T23:15:26.252 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:15:26.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:26.380+0000 7f14660d4640 1 -- 172.21.15.40:0/3608175138 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14600ffa40 msgr2=0x7f14600ffe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:26.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:26.380+0000 7f14660d4640 1 --2- 172.21.15.40:0/3608175138 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14600ffa40 0x7f14600ffe60 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1450009930 tx=0x7f145002bc30 comp rx=0 tx=0).stop 2024-04-21T23:15:26.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:26.380+0000 7f14660d4640 1 -- 172.21.15.40:0/3608175138 shutdown_connections 2024-04-21T23:15:26.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:26.380+0000 7f14660d4640 1 --2- 172.21.15.40:0/3608175138 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1460102a10 0x7f1460102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:26.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:26.380+0000 7f14660d4640 1 --2- 172.21.15.40:0/3608175138 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f14601033d0 0x7f1460109c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:26.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:26.380+0000 7f14660d4640 1 --2- 172.21.15.40:0/3608175138 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14600ffa40 0x7f14600ffe60 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:26.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:26.380+0000 7f14660d4640 1 -- 172.21.15.40:0/3608175138 >> 172.21.15.40:0/3608175138 conn(0x7f14600fb370 msgr2=0x7f14600fd7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:26.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:26.380+0000 7f14660d4640 1 -- 172.21.15.40:0/3608175138 shutdown_connections 2024-04-21T23:15:26.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:26.380+0000 7f14660d4640 1 -- 172.21.15.40:0/3608175138 wait complete. 2024-04-21T23:15:26.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:26.381+0000 7f14660d4640 1 Processor -- start 2024-04-21T23:15:26.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:26.381+0000 7f14660d4640 1 -- start start 2024-04-21T23:15:26.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:26.381+0000 7f14660d4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f14600ffa40 0x7f1460197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:26.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:26.381+0000 7f14660d4640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1460102a10 0x7f1460198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:26.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:26.381+0000 7f14660d4640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14601033d0 0x7f146019f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:26.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:26.381+0000 7f14660d4640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f146010d3f0 con 0x7f1460102a10 2024-04-21T23:15:26.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:26.381+0000 7f14660d4640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f146010d270 con 0x7f14601033d0 2024-04-21T23:15:26.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:26.381+0000 7f14660d4640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f146010d570 con 0x7f14600ffa40 2024-04-21T23:15:26.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:26.381+0000 7f145effd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1460102a10 0x7f1460198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:26.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:26.381+0000 7f145effd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1460102a10 0x7f1460198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51796/0 (socket says 172.21.15.40:51796) 2024-04-21T23:15:26.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:26.381+0000 7f145effd640 1 -- 172.21.15.40:0/3245047154 learned_addr learned my addr 172.21.15.40:0/3245047154 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:15:26.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:26.381+0000 7f145f7fe640 1 --2- 172.21.15.40:0/3245047154 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f14600ffa40 0x7f1460197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:26.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:26.381+0000 7f145ffff640 1 --2- 172.21.15.40:0/3245047154 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14601033d0 0x7f146019f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:26.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:26.382+0000 7f145effd640 1 -- 172.21.15.40:0/3245047154 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f14600ffa40 msgr2=0x7f1460197ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:26.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:26.382+0000 7f145effd640 1 --2- 172.21.15.40:0/3245047154 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f14600ffa40 0x7f1460197ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:26.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:26.382+0000 7f145effd640 1 -- 172.21.15.40:0/3245047154 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14601033d0 msgr2=0x7f146019f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:26.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:26.382+0000 7f145effd640 1 --2- 172.21.15.40:0/3245047154 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14601033d0 0x7f146019f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:26.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:26.382+0000 7f145effd640 1 -- 172.21.15.40:0/3245047154 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f146019f6e0 con 0x7f1460102a10 2024-04-21T23:15:26.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:26.382+0000 7f145effd640 1 --2- 172.21.15.40:0/3245047154 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1460102a10 0x7f1460198000 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f145400e9f0 tx=0x7f145400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:26.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:26.382+0000 7f145cff9640 1 -- 172.21.15.40:0/3245047154 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f145400cd80 con 0x7f1460102a10 2024-04-21T23:15:26.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:26.382+0000 7f145cff9640 1 -- 172.21.15.40:0/3245047154 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1454010910 con 0x7f1460102a10 2024-04-21T23:15:26.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:26.382+0000 7f14660d4640 1 -- 172.21.15.40:0/3245047154 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f146019f9d0 con 0x7f1460102a10 2024-04-21T23:15:26.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:26.382+0000 7f145cff9640 1 -- 172.21.15.40:0/3245047154 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1454018990 con 0x7f1460102a10 2024-04-21T23:15:26.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:26.382+0000 7f14660d4640 1 -- 172.21.15.40:0/3245047154 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f146019ff20 con 0x7f1460102a10 2024-04-21T23:15:26.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:26.383+0000 7f14660d4640 1 -- 172.21.15.40:0/3245047154 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1460108b40 con 0x7f1460102a10 2024-04-21T23:15:26.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:26.384+0000 7f145cff9640 1 -- 172.21.15.40:0/3245047154 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f1454010ac0 con 0x7f1460102a10 2024-04-21T23:15:26.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:26.385+0000 7f145cff9640 1 --2- 172.21.15.40:0/3245047154 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f14480a2b90 0x7f14480a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:26.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:26.385+0000 7f145f7fe640 1 --2- 172.21.15.40:0/3245047154 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f14480a2b90 0x7f14480a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:26.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:26.385+0000 7f145cff9640 1 -- 172.21.15.40:0/3245047154 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f14540cefe0 con 0x7f1460102a10 2024-04-21T23:15:26.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:26.385+0000 7f145f7fe640 1 --2- 172.21.15.40:0/3245047154 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f14480a2b90 0x7f14480a5050 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f145002c7b0 tx=0x7f1450000bc0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:26.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:26.389+0000 7f145cff9640 1 -- 172.21.15.40:0/3245047154 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f145409c340 con 0x7f1460102a10 2024-04-21T23:15:26.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:26.624+0000 7f14660d4640 1 -- 172.21.15.40:0/3245047154 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f146002d170 con 0x7f1460102a10 2024-04-21T23:15:26.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:26.624+0000 7f145cff9640 1 -- 172.21.15.40:0/3245047154 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3049 (secure 0 0 0) 0x7f145409bc40 con 0x7f1460102a10 2024-04-21T23:15:26.624 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:15:26.624 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/1571948830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1571948830},{"type":"v1","addr":"172.21.15.40:6835","nonce":1571948830}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/3438493203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":3438493203},{"type":"v1","addr":"172.21.15.174:6833","nonce":3438493203}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/3105752678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":3105752678},{"type":"v1","addr":"172.21.15.111:6833","nonce":3105752678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-21T23:15:26.624 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 4 2024-04-21T23:15:26.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:26.626+0000 7f14660d4640 1 -- 172.21.15.40:0/3245047154 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f14480a2b90 msgr2=0x7f14480a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:26.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:26.626+0000 7f14660d4640 1 --2- 172.21.15.40:0/3245047154 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f14480a2b90 0x7f14480a5050 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f145002c7b0 tx=0x7f1450000bc0 comp rx=0 tx=0).stop 2024-04-21T23:15:26.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:26.626+0000 7f14660d4640 1 -- 172.21.15.40:0/3245047154 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1460102a10 msgr2=0x7f1460198000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:26.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:26.626+0000 7f14660d4640 1 --2- 172.21.15.40:0/3245047154 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1460102a10 0x7f1460198000 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f145400e9f0 tx=0x7f145400eeb0 comp rx=0 tx=0).stop 2024-04-21T23:15:26.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:26.626+0000 7f14660d4640 1 -- 172.21.15.40:0/3245047154 shutdown_connections 2024-04-21T23:15:26.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:26.626+0000 7f14660d4640 1 --2- 172.21.15.40:0/3245047154 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f14480a2b90 0x7f14480a5050 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:26.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:26.626+0000 7f14660d4640 1 --2- 172.21.15.40:0/3245047154 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14601033d0 0x7f146019f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:26.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:26.626+0000 7f14660d4640 1 --2- 172.21.15.40:0/3245047154 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1460102a10 0x7f1460198000 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:26.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:26.626+0000 7f14660d4640 1 --2- 172.21.15.40:0/3245047154 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f14600ffa40 0x7f1460197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:26.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:26.626+0000 7f14660d4640 1 -- 172.21.15.40:0/3245047154 >> 172.21.15.40:0/3245047154 conn(0x7f14600fb370 msgr2=0x7f1460105d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:26.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:26.626+0000 7f14660d4640 1 -- 172.21.15.40:0/3245047154 shutdown_connections 2024-04-21T23:15:26.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:26.627+0000 7f14660d4640 1 -- 172.21.15.40:0/3245047154 wait complete. 2024-04-21T23:15:26.930 INFO:journalctl@ceph.mds.d.smithi040.stdout:Apr 21 23:15:26 smithi040 systemd[1]: Starting Ceph mds.d for d814e4dc-0033-11ef-bc93-c7b262605968... 2024-04-21T23:15:27.182 INFO:journalctl@ceph.mds.d.smithi040.stdout:Apr 21 23:15:26 smithi040 podman[66406]: 2024-04-21 23:15:26.834107059 +0000 UTC m=+0.018234453 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-21T23:15:27.182 INFO:journalctl@ceph.mds.d.smithi040.stdout:Apr 21 23:15:26 smithi040 podman[66406]: 2024-04-21 23:15:26.947199709 +0000 UTC m=+0.131327075 container create 968c7523f37cff51b1f6600ce8560f57046bf95d68b56d96b26cb4bacc0d3900 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-d, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-820a2f8, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-21T23:15:27.182 INFO:journalctl@ceph.mds.d.smithi040.stdout:Apr 21 23:15:27 smithi040 podman[66406]: 2024-04-21 23:15:27.173342773 +0000 UTC m=+0.357470149 container init 968c7523f37cff51b1f6600ce8560f57046bf95d68b56d96b26cb4bacc0d3900 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-d, RELEASE=squid-820a2f8, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T23:15:27.182 INFO:journalctl@ceph.mds.d.smithi040.stdout:Apr 21 23:15:27 smithi040 podman[66406]: 2024-04-21 23:15:27.175956 +0000 UTC m=+0.360083375 container start 968c7523f37cff51b1f6600ce8560f57046bf95d68b56d96b26cb4bacc0d3900 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-d, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-820a2f8, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240416) 2024-04-21T23:15:27.491 INFO:journalctl@ceph.mds.d.smithi040.stdout:Apr 21 23:15:27 smithi040 ceph-mds[66426]: -- 172.21.15.40:0/1382900210 <== mon.0 v2:172.21.15.40:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55aebb9fda40 con 0x55aebb9d0800 2024-04-21T23:15:27.491 INFO:journalctl@ceph.mds.d.smithi040.stdout:Apr 21 23:15:27 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-d[66422]: starting mds.d at 2024-04-21T23:15:27.491 INFO:journalctl@ceph.mds.d.smithi040.stdout:Apr 21 23:15:27 smithi040 bash[66406]: 968c7523f37cff51b1f6600ce8560f57046bf95d68b56d96b26cb4bacc0d3900 2024-04-21T23:15:27.491 INFO:journalctl@ceph.mds.d.smithi040.stdout:Apr 21 23:15:27 smithi040 systemd[1]: Started Ceph mds.d for d814e4dc-0033-11ef-bc93-c7b262605968. 2024-04-21T23:15:27.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:27 smithi040 ceph-mon[27984]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:27.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:27 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/3245047154' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:15:27.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:27 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:27.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:27 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:27.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:27 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:27.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:27 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:15:27.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:27 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:15:27.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:27 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:15:27.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:27 smithi174 ceph-mon[31409]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:27.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:27 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/3245047154' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:15:27.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:27 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:27.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:27 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:27.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:27 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:27.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:27 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:15:27.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:27 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:15:27.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:27 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:15:27.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:27 smithi111 ceph-mon[31085]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:27.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:27 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/3245047154' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:15:27.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:27 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:27.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:27 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:27.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:27 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:27.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:27 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:15:27.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:27 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:15:27.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:27 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:15:28.637 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:15:28.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:28 smithi111 ceph-mon[31085]: Deploying daemon mds.e on smithi111 2024-04-21T23:15:28.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:28 smithi111 ceph-mon[31085]: mds.? [v2:172.21.15.40:6836/969356642,v1:172.21.15.40:6837/969356642] up:boot 2024-04-21T23:15:28.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:28 smithi111 ceph-mon[31085]: fsmap 4 up:standby 2024-04-21T23:15:28.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:28 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-21T23:15:28.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:28.755+0000 7f8dd7c8b640 1 -- 172.21.15.40:0/2785117977 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8dd0102230 msgr2=0x7f8dd0108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:28.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:28.755+0000 7f8dd7c8b640 1 --2- 172.21.15.40:0/2785117977 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8dd0102230 0x7f8dd0108ac0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f8dc8009ca0 tx=0x7f8dc802c0f0 comp rx=0 tx=0).stop 2024-04-21T23:15:28.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:28.755+0000 7f8dd7c8b640 1 -- 172.21.15.40:0/2785117977 shutdown_connections 2024-04-21T23:15:28.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:28.755+0000 7f8dd7c8b640 1 --2- 172.21.15.40:0/2785117977 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8dd0102230 0x7f8dd0108ac0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:28.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:28.755+0000 7f8dd7c8b640 1 --2- 172.21.15.40:0/2785117977 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8dd0101870 0x7f8dd0101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:28.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:28.755+0000 7f8dd7c8b640 1 --2- 172.21.15.40:0/2785117977 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8dd0100680 0x7f8dd0100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:28.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:28.755+0000 7f8dd7c8b640 1 -- 172.21.15.40:0/2785117977 >> 172.21.15.40:0/2785117977 conn(0x7f8dd00fbbf0 msgr2=0x7f8dd00fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:28.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:28.755+0000 7f8dd7c8b640 1 -- 172.21.15.40:0/2785117977 shutdown_connections 2024-04-21T23:15:28.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:28.755+0000 7f8dd7c8b640 1 -- 172.21.15.40:0/2785117977 wait complete. 2024-04-21T23:15:28.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:28.756+0000 7f8dd7c8b640 1 Processor -- start 2024-04-21T23:15:28.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:28.756+0000 7f8dd7c8b640 1 -- start start 2024-04-21T23:15:28.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:28.756+0000 7f8dd7c8b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8dd0100680 0x7f8dd00694a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:28.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:28.756+0000 7f8dd7c8b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8dd0101870 0x7f8dd019c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:28.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:28.756+0000 7f8dd7c8b640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8dd0102230 0x7f8dd019eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:28.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:28.756+0000 7f8dd7c8b640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8dd010c300 con 0x7f8dd0101870 2024-04-21T23:15:28.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:28.756+0000 7f8dd7c8b640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f8dd010c180 con 0x7f8dd0102230 2024-04-21T23:15:28.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:28.756+0000 7f8dd7c8b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8dd010c480 con 0x7f8dd0100680 2024-04-21T23:15:28.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:28.756+0000 7f8dd5226640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8dd0101870 0x7f8dd019c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:28.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:28.756+0000 7f8dd5226640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8dd0101870 0x7f8dd019c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51808/0 (socket says 172.21.15.40:51808) 2024-04-21T23:15:28.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:28.756+0000 7f8dd5a27640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8dd0100680 0x7f8dd00694a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:28.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:28.756+0000 7f8dd5226640 1 -- 172.21.15.40:0/2256158350 learned_addr learned my addr 172.21.15.40:0/2256158350 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:15:28.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:28.756+0000 7f8dd6228640 1 --2- 172.21.15.40:0/2256158350 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8dd0102230 0x7f8dd019eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:28.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:28.756+0000 7f8dd5226640 1 -- 172.21.15.40:0/2256158350 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8dd0100680 msgr2=0x7f8dd00694a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:28.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:28.756+0000 7f8dd5226640 1 --2- 172.21.15.40:0/2256158350 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8dd0100680 0x7f8dd00694a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:28.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:28.756+0000 7f8dd5226640 1 -- 172.21.15.40:0/2256158350 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8dd0102230 msgr2=0x7f8dd019eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:28.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:28.756+0000 7f8dd5226640 1 --2- 172.21.15.40:0/2256158350 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8dd0102230 0x7f8dd019eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:28.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:28.756+0000 7f8dd5226640 1 -- 172.21.15.40:0/2256158350 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8dd019f660 con 0x7f8dd0101870 2024-04-21T23:15:28.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:28.757+0000 7f8dd5226640 1 --2- 172.21.15.40:0/2256158350 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8dd0101870 0x7f8dd019c730 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f8dc000d770 tx=0x7f8dc000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:28.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:28.757+0000 7f8dc6ffd640 1 -- 172.21.15.40:0/2256158350 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dc0017070 con 0x7f8dd0101870 2024-04-21T23:15:28.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:28.757+0000 7f8dc6ffd640 1 -- 172.21.15.40:0/2256158350 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8dc0005500 con 0x7f8dd0101870 2024-04-21T23:15:28.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:28.757+0000 7f8dd7c8b640 1 -- 172.21.15.40:0/2256158350 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8dd019f950 con 0x7f8dd0101870 2024-04-21T23:15:28.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:28.757+0000 7f8dc6ffd640 1 -- 172.21.15.40:0/2256158350 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dc0004e00 con 0x7f8dd0101870 2024-04-21T23:15:28.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:28.757+0000 7f8dd7c8b640 1 -- 172.21.15.40:0/2256158350 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8dd019fe50 con 0x7f8dd0101870 2024-04-21T23:15:28.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:28.758+0000 7f8dd7c8b640 1 -- 172.21.15.40:0/2256158350 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8dd01079a0 con 0x7f8dd0101870 2024-04-21T23:15:28.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:28.759+0000 7f8dc6ffd640 1 -- 172.21.15.40:0/2256158350 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f8dc0004fa0 con 0x7f8dd0101870 2024-04-21T23:15:28.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:28.760+0000 7f8dc6ffd640 1 --2- 172.21.15.40:0/2256158350 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f8dbc0a2b90 0x7f8dbc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:28.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:28.760+0000 7f8dd5a27640 1 --2- 172.21.15.40:0/2256158350 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f8dbc0a2b90 0x7f8dbc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:28.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:28.760+0000 7f8dc6ffd640 1 -- 172.21.15.40:0/2256158350 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f8dc00ce9d0 con 0x7f8dd0101870 2024-04-21T23:15:28.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:28.760+0000 7f8dd5a27640 1 --2- 172.21.15.40:0/2256158350 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f8dbc0a2b90 0x7f8dbc0a5050 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f8dd01016d0 tx=0x7f8dcc005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:28.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:28.764+0000 7f8dc6ffd640 1 -- 172.21.15.40:0/2256158350 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f8dc009bdd0 con 0x7f8dd0101870 2024-04-21T23:15:28.931 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:28 smithi040 ceph-mon[27984]: Deploying daemon mds.e on smithi111 2024-04-21T23:15:28.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:28 smithi040 ceph-mon[27984]: mds.? [v2:172.21.15.40:6836/969356642,v1:172.21.15.40:6837/969356642] up:boot 2024-04-21T23:15:28.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:28 smithi040 ceph-mon[27984]: fsmap 4 up:standby 2024-04-21T23:15:28.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:28 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-21T23:15:28.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:28 smithi174 ceph-mon[31409]: Deploying daemon mds.e on smithi111 2024-04-21T23:15:28.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:28 smithi174 ceph-mon[31409]: mds.? [v2:172.21.15.40:6836/969356642,v1:172.21.15.40:6837/969356642] up:boot 2024-04-21T23:15:28.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:28 smithi174 ceph-mon[31409]: fsmap 4 up:standby 2024-04-21T23:15:28.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:28 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-21T23:15:28.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:28.989+0000 7f8dd7c8b640 1 -- 172.21.15.40:0/2256158350 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8dd002d170 con 0x7f8dd0101870 2024-04-21T23:15:28.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:28.989+0000 7f8dc6ffd640 1 -- 172.21.15.40:0/2256158350 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3871 (secure 0 0 0) 0x7f8dc009b6d0 con 0x7f8dd0101870 2024-04-21T23:15:28.988 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:15:28.988 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/1571948830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1571948830},{"type":"v1","addr":"172.21.15.40:6835","nonce":1571948830}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/969356642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":969356642},{"type":"v1","addr":"172.21.15.40:6837","nonce":969356642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/3438493203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":3438493203},{"type":"v1","addr":"172.21.15.174:6833","nonce":3438493203}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/3105752678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":3105752678},{"type":"v1","addr":"172.21.15.111:6833","nonce":3105752678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-21T23:15:28.989 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 5 2024-04-21T23:15:28.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:28.991+0000 7f8dd7c8b640 1 -- 172.21.15.40:0/2256158350 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f8dbc0a2b90 msgr2=0x7f8dbc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:28.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:28.991+0000 7f8dd7c8b640 1 --2- 172.21.15.40:0/2256158350 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f8dbc0a2b90 0x7f8dbc0a5050 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f8dd01016d0 tx=0x7f8dcc005f90 comp rx=0 tx=0).stop 2024-04-21T23:15:28.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:28.991+0000 7f8dd7c8b640 1 -- 172.21.15.40:0/2256158350 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8dd0101870 msgr2=0x7f8dd019c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:28.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:28.991+0000 7f8dd7c8b640 1 --2- 172.21.15.40:0/2256158350 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8dd0101870 0x7f8dd019c730 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f8dc000d770 tx=0x7f8dc000dc30 comp rx=0 tx=0).stop 2024-04-21T23:15:28.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:28.991+0000 7f8dd7c8b640 1 -- 172.21.15.40:0/2256158350 shutdown_connections 2024-04-21T23:15:28.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:28.991+0000 7f8dd7c8b640 1 --2- 172.21.15.40:0/2256158350 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f8dbc0a2b90 0x7f8dbc0a5050 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:28.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:28.991+0000 7f8dd7c8b640 1 --2- 172.21.15.40:0/2256158350 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8dd0102230 0x7f8dd019eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:28.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:28.991+0000 7f8dd7c8b640 1 --2- 172.21.15.40:0/2256158350 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8dd0101870 0x7f8dd019c730 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:28.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:28.991+0000 7f8dd7c8b640 1 --2- 172.21.15.40:0/2256158350 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8dd0100680 0x7f8dd00694a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:28.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:28.991+0000 7f8dd7c8b640 1 -- 172.21.15.40:0/2256158350 >> 172.21.15.40:0/2256158350 conn(0x7f8dd00fbbf0 msgr2=0x7f8dd0104b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:28.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:28.991+0000 7f8dd7c8b640 1 -- 172.21.15.40:0/2256158350 shutdown_connections 2024-04-21T23:15:28.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:28.991+0000 7f8dd7c8b640 1 -- 172.21.15.40:0/2256158350 wait complete. 2024-04-21T23:15:29.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:29 smithi111 ceph-mon[31085]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:29.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:29 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/2256158350' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:15:29.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:29 smithi040 ceph-mon[27984]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:29.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:29 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/2256158350' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:15:29.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:29 smithi174 ceph-mon[31409]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:29.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:29 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/2256158350' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:15:29.944 INFO:journalctl@ceph.mds.e.smithi111.stdout:Apr 21 23:15:29 smithi111 podman[60784]: 2024-04-21 23:15:29.887844197 +0000 UTC m=+0.139251380 container create e9670c79289b5d48e3b42be2cb7d3c250276be3dbc2deefbe339eb2b213a4647 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-e, org.label-schema.build-date=20240416, RELEASE=squid-820a2f8, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-04-21T23:15:30.444 INFO:journalctl@ceph.mds.e.smithi111.stdout:Apr 21 23:15:30 smithi111 podman[60784]: 2024-04-21 23:15:30.165371496 +0000 UTC m=+0.416778634 container init e9670c79289b5d48e3b42be2cb7d3c250276be3dbc2deefbe339eb2b213a4647 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-e, RELEASE=squid-820a2f8, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-21T23:15:30.444 INFO:journalctl@ceph.mds.e.smithi111.stdout:Apr 21 23:15:30 smithi111 podman[60784]: 2024-04-21 23:15:30.168002674 +0000 UTC m=+0.419409812 container start e9670c79289b5d48e3b42be2cb7d3c250276be3dbc2deefbe339eb2b213a4647 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-e, RELEASE=squid-820a2f8, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-21T23:15:30.444 INFO:journalctl@ceph.mds.e.smithi111.stdout:Apr 21 23:15:30 smithi111 ceph-mds[60803]: -- 172.21.15.111:0/4244905502 <== mon.2 v2:172.21.15.111:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555bd3d2fa40 con 0x555bd3d02800 2024-04-21T23:15:30.444 INFO:journalctl@ceph.mds.e.smithi111.stdout:Apr 21 23:15:30 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-e[60799]: starting mds.e at 2024-04-21T23:15:30.444 INFO:journalctl@ceph.mds.e.smithi111.stdout:Apr 21 23:15:30 smithi111 bash[60784]: e9670c79289b5d48e3b42be2cb7d3c250276be3dbc2deefbe339eb2b213a4647 2024-04-21T23:15:30.444 INFO:journalctl@ceph.mds.e.smithi111.stdout:Apr 21 23:15:30 smithi111 systemd[1]: Started Ceph mds.e for d814e4dc-0033-11ef-bc93-c7b262605968. 2024-04-21T23:15:31.003 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:15:31.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:31.117+0000 7feb83c4e640 1 -- 172.21.15.40:0/3616803788 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb7c101880 msgr2=0x7feb7c101d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:31.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:31.117+0000 7feb83c4e640 1 --2- 172.21.15.40:0/3616803788 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb7c101880 0x7feb7c101d00 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7feb6c009900 tx=0x7feb6c02bc30 comp rx=0 tx=0).stop 2024-04-21T23:15:31.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:31.117+0000 7feb83c4e640 1 -- 172.21.15.40:0/3616803788 shutdown_connections 2024-04-21T23:15:31.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:31.117+0000 7feb83c4e640 1 --2- 172.21.15.40:0/3616803788 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feb7c102240 0x7feb7c108ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:31.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:31.117+0000 7feb83c4e640 1 --2- 172.21.15.40:0/3616803788 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb7c101880 0x7feb7c101d00 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:31.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:31.117+0000 7feb83c4e640 1 --2- 172.21.15.40:0/3616803788 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb7c100690 0x7feb7c100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:31.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:31.117+0000 7feb83c4e640 1 -- 172.21.15.40:0/3616803788 >> 172.21.15.40:0/3616803788 conn(0x7feb7c0fbc20 msgr2=0x7feb7c0fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:31.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:31.118+0000 7feb83c4e640 1 -- 172.21.15.40:0/3616803788 shutdown_connections 2024-04-21T23:15:31.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:31.118+0000 7feb83c4e640 1 -- 172.21.15.40:0/3616803788 wait complete. 2024-04-21T23:15:31.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:31.118+0000 7feb83c4e640 1 Processor -- start 2024-04-21T23:15:31.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:31.118+0000 7feb83c4e640 1 -- start start 2024-04-21T23:15:31.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:31.118+0000 7feb83c4e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb7c100690 0x7feb7c104620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:31.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:31.118+0000 7feb83c4e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb7c101880 0x7feb7c102c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:31.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:31.118+0000 7feb83c4e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feb7c102240 0x7feb7c1031a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:31.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:31.118+0000 7feb83c4e640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7feb7c10c2f0 con 0x7feb7c101880 2024-04-21T23:15:31.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:31.118+0000 7feb83c4e640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7feb7c10c170 con 0x7feb7c100690 2024-04-21T23:15:31.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:31.118+0000 7feb83c4e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7feb7c10c470 con 0x7feb7c102240 2024-04-21T23:15:31.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:31.119+0000 7feb811e9640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb7c101880 0x7feb7c102c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:31.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:31.119+0000 7feb811e9640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb7c101880 0x7feb7c102c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51820/0 (socket says 172.21.15.40:51820) 2024-04-21T23:15:31.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:31.119+0000 7feb811e9640 1 -- 172.21.15.40:0/984130549 learned_addr learned my addr 172.21.15.40:0/984130549 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:15:31.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:31.119+0000 7feb819ea640 1 --2- 172.21.15.40:0/984130549 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb7c100690 0x7feb7c104620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:31.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:31.119+0000 7feb821eb640 1 --2- 172.21.15.40:0/984130549 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feb7c102240 0x7feb7c1031a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:31.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:31.119+0000 7feb811e9640 1 -- 172.21.15.40:0/984130549 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feb7c102240 msgr2=0x7feb7c1031a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:31.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:31.119+0000 7feb811e9640 1 --2- 172.21.15.40:0/984130549 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feb7c102240 0x7feb7c1031a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:31.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:31.119+0000 7feb811e9640 1 -- 172.21.15.40:0/984130549 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb7c100690 msgr2=0x7feb7c104620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:31.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:31.119+0000 7feb811e9640 1 --2- 172.21.15.40:0/984130549 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb7c100690 0x7feb7c104620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:31.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:31.119+0000 7feb811e9640 1 -- 172.21.15.40:0/984130549 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb7c103a30 con 0x7feb7c101880 2024-04-21T23:15:31.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:31.119+0000 7feb819ea640 1 --2- 172.21.15.40:0/984130549 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb7c100690 0x7feb7c104620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:15:31.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:31.119+0000 7feb811e9640 1 --2- 172.21.15.40:0/984130549 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb7c101880 0x7feb7c102c60 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7feb6c005ab0 tx=0x7feb6c02c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:31.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:31.119+0000 7feb72ffd640 1 -- 172.21.15.40:0/984130549 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb6c041070 con 0x7feb7c101880 2024-04-21T23:15:31.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:31.119+0000 7feb72ffd640 1 -- 172.21.15.40:0/984130549 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb6c004450 con 0x7feb7c101880 2024-04-21T23:15:31.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:31.119+0000 7feb83c4e640 1 -- 172.21.15.40:0/984130549 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb7c1a1c40 con 0x7feb7c101880 2024-04-21T23:15:31.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:31.119+0000 7feb72ffd640 1 -- 172.21.15.40:0/984130549 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb6c003d70 con 0x7feb7c101880 2024-04-21T23:15:31.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:31.119+0000 7feb83c4e640 1 -- 172.21.15.40:0/984130549 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb7c1a2190 con 0x7feb7c101880 2024-04-21T23:15:31.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:31.120+0000 7feb83c4e640 1 -- 172.21.15.40:0/984130549 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb7c1a2360 con 0x7feb7c101880 2024-04-21T23:15:31.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:31.122+0000 7feb72ffd640 1 -- 172.21.15.40:0/984130549 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7feb6c044070 con 0x7feb7c101880 2024-04-21T23:15:31.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:31.122+0000 7feb72ffd640 1 --2- 172.21.15.40:0/984130549 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7feb680a2c10 0x7feb680a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:31.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:31.122+0000 7feb819ea640 1 --2- 172.21.15.40:0/984130549 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7feb680a2c10 0x7feb680a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:31.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:31.122+0000 7feb72ffd640 1 -- 172.21.15.40:0/984130549 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7feb6c0f0880 con 0x7feb7c101880 2024-04-21T23:15:31.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:31.123+0000 7feb819ea640 1 --2- 172.21.15.40:0/984130549 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7feb680a2c10 0x7feb680a50d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7feb7c1016e0 tx=0x7feb78005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:31.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:31.126+0000 7feb72ffd640 1 -- 172.21.15.40:0/984130549 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7feb6c0bdc80 con 0x7feb7c101880 2024-04-21T23:15:31.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:31.354+0000 7feb83c4e640 1 -- 172.21.15.40:0/984130549 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feb7c1a2530 con 0x7feb7c101880 2024-04-21T23:15:31.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:31.354+0000 7feb72ffd640 1 -- 172.21.15.40:0/984130549 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4699 (secure 0 0 0) 0x7feb6c0bd580 con 0x7feb7c101880 2024-04-21T23:15:31.354 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:15:31.354 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/1571948830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1571948830},{"type":"v1","addr":"172.21.15.40:6835","nonce":1571948830}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/969356642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":969356642},{"type":"v1","addr":"172.21.15.40:6837","nonce":969356642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/3438493203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":3438493203},{"type":"v1","addr":"172.21.15.174:6833","nonce":3438493203}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/3105752678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":3105752678},{"type":"v1","addr":"172.21.15.111:6833","nonce":3105752678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24461,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/2704462092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2704462092},{"type":"v1","addr":"172.21.15.111:6835","nonce":2704462092}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-04-21T23:15:31.355 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 6 2024-04-21T23:15:31.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:31.356+0000 7feb83c4e640 1 -- 172.21.15.40:0/984130549 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7feb680a2c10 msgr2=0x7feb680a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:31.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:31.356+0000 7feb83c4e640 1 --2- 172.21.15.40:0/984130549 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7feb680a2c10 0x7feb680a50d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7feb7c1016e0 tx=0x7feb78005f90 comp rx=0 tx=0).stop 2024-04-21T23:15:31.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:31.356+0000 7feb83c4e640 1 -- 172.21.15.40:0/984130549 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb7c101880 msgr2=0x7feb7c102c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:31.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:31.356+0000 7feb83c4e640 1 --2- 172.21.15.40:0/984130549 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb7c101880 0x7feb7c102c60 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7feb6c005ab0 tx=0x7feb6c02c6c0 comp rx=0 tx=0).stop 2024-04-21T23:15:31.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:31.356+0000 7feb83c4e640 1 -- 172.21.15.40:0/984130549 shutdown_connections 2024-04-21T23:15:31.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:31.356+0000 7feb83c4e640 1 --2- 172.21.15.40:0/984130549 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7feb680a2c10 0x7feb680a50d0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:31.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:31.356+0000 7feb83c4e640 1 --2- 172.21.15.40:0/984130549 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feb7c102240 0x7feb7c1031a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:31.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:31.357+0000 7feb83c4e640 1 --2- 172.21.15.40:0/984130549 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb7c101880 0x7feb7c102c60 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:31.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:31.357+0000 7feb83c4e640 1 --2- 172.21.15.40:0/984130549 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb7c100690 0x7feb7c104620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:31.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:31.357+0000 7feb83c4e640 1 -- 172.21.15.40:0/984130549 >> 172.21.15.40:0/984130549 conn(0x7feb7c0fbc20 msgr2=0x7feb7c0fd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:31.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:31.357+0000 7feb83c4e640 1 -- 172.21.15.40:0/984130549 shutdown_connections 2024-04-21T23:15:31.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:31.357+0000 7feb83c4e640 1 -- 172.21.15.40:0/984130549 wait complete. 2024-04-21T23:15:31.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:31 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:31.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:31 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:31.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:31 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:31.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:31 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:15:31.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:31 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:15:31.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:31 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:15:31.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:31 smithi040 ceph-mon[27984]: Deploying daemon mds.f on smithi174 2024-04-21T23:15:31.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:31 smithi040 ceph-mon[27984]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:31.683 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:31 smithi040 ceph-mon[27984]: mds.? [v2:172.21.15.111:6834/2704462092,v1:172.21.15.111:6835/2704462092] up:boot 2024-04-21T23:15:31.683 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:31 smithi040 ceph-mon[27984]: fsmap 5 up:standby 2024-04-21T23:15:31.683 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:31 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-21T23:15:31.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:31 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:31.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:31 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:31.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:31 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:31.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:31 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:15:31.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:31 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:15:31.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:31 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:15:31.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:31 smithi174 ceph-mon[31409]: Deploying daemon mds.f on smithi174 2024-04-21T23:15:31.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:31 smithi174 ceph-mon[31409]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:31.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:31 smithi174 ceph-mon[31409]: mds.? [v2:172.21.15.111:6834/2704462092,v1:172.21.15.111:6835/2704462092] up:boot 2024-04-21T23:15:31.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:31 smithi174 ceph-mon[31409]: fsmap 5 up:standby 2024-04-21T23:15:31.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:31 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-21T23:15:31.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:31 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:31.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:31 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:31.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:31 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:31.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:31 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:15:31.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:31 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:15:31.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:31 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:15:31.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:31 smithi111 ceph-mon[31085]: Deploying daemon mds.f on smithi174 2024-04-21T23:15:31.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:31 smithi111 ceph-mon[31085]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:31.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:31 smithi111 ceph-mon[31085]: mds.? [v2:172.21.15.111:6834/2704462092,v1:172.21.15.111:6835/2704462092] up:boot 2024-04-21T23:15:31.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:31 smithi111 ceph-mon[31085]: fsmap 5 up:standby 2024-04-21T23:15:31.696 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:31 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-21T23:15:32.575 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:32 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/984130549' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:15:32.575 INFO:journalctl@ceph.mds.f.smithi174.stdout:Apr 21 23:15:32 smithi174 systemd[1]: Starting Ceph mds.f for d814e4dc-0033-11ef-bc93-c7b262605968... 2024-04-21T23:15:32.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:32 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/984130549' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:15:32.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:32 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/984130549' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:15:33.055 INFO:journalctl@ceph.mds.f.smithi174.stdout:Apr 21 23:15:32 smithi174 podman[61466]: 2024-04-21 23:15:32.693308234 +0000 UTC m=+0.017488461 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-21T23:15:33.055 INFO:journalctl@ceph.mds.f.smithi174.stdout:Apr 21 23:15:32 smithi174 podman[61466]: 2024-04-21 23:15:32.812627616 +0000 UTC m=+0.136807830 container create b1cc187a796df3f83505468a095f75c5f78cf2a4ff907606f3b7d4d542004542 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-f, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, RELEASE=squid-820a2f8, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T23:15:33.055 INFO:journalctl@ceph.mds.f.smithi174.stdout:Apr 21 23:15:33 smithi174 podman[61466]: 2024-04-21 23:15:33.056842619 +0000 UTC m=+0.381022834 container init b1cc187a796df3f83505468a095f75c5f78cf2a4ff907606f3b7d4d542004542 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-f, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, RELEASE=squid-820a2f8, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5) 2024-04-21T23:15:33.368 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:15:33.443 INFO:journalctl@ceph.mds.f.smithi174.stdout:Apr 21 23:15:33 smithi174 podman[61466]: 2024-04-21 23:15:33.059604806 +0000 UTC m=+0.383785024 container start b1cc187a796df3f83505468a095f75c5f78cf2a4ff907606f3b7d4d542004542 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-f, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=squid-820a2f8, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T23:15:33.444 INFO:journalctl@ceph.mds.f.smithi174.stdout:Apr 21 23:15:33 smithi174 ceph-mds[61485]: -- 172.21.15.174:0/1713387785 <== mon.1 v2:172.21.15.174:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55df7c79da40 con 0x55df7c770800 2024-04-21T23:15:33.444 INFO:journalctl@ceph.mds.f.smithi174.stdout:Apr 21 23:15:33 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-f[61481]: starting mds.f at 2024-04-21T23:15:33.444 INFO:journalctl@ceph.mds.f.smithi174.stdout:Apr 21 23:15:33 smithi174 bash[61466]: b1cc187a796df3f83505468a095f75c5f78cf2a4ff907606f3b7d4d542004542 2024-04-21T23:15:33.444 INFO:journalctl@ceph.mds.f.smithi174.stdout:Apr 21 23:15:33 smithi174 systemd[1]: Started Ceph mds.f for d814e4dc-0033-11ef-bc93-c7b262605968. 2024-04-21T23:15:33.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:33.484+0000 7fef7806b640 1 -- 172.21.15.40:0/3076117012 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef701017b0 msgr2=0x7fef70101bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:33.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:33.484+0000 7fef7806b640 1 --2- 172.21.15.40:0/3076117012 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef701017b0 0x7fef70101bd0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fef6c009900 tx=0x7fef6c02bc30 comp rx=0 tx=0).stop 2024-04-21T23:15:33.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:33.484+0000 7fef7806b640 1 -- 172.21.15.40:0/3076117012 shutdown_connections 2024-04-21T23:15:33.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:33.484+0000 7fef7806b640 1 --2- 172.21.15.40:0/3076117012 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fef701029a0 0x7fef70102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:33.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:33.484+0000 7fef7806b640 1 --2- 172.21.15.40:0/3076117012 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef70103360 0x7fef70109bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:33.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:33.484+0000 7fef7806b640 1 --2- 172.21.15.40:0/3076117012 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef701017b0 0x7fef70101bd0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:33.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:33.484+0000 7fef7806b640 1 -- 172.21.15.40:0/3076117012 >> 172.21.15.40:0/3076117012 conn(0x7fef700fcd40 msgr2=0x7fef700ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:33.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:33.484+0000 7fef7806b640 1 -- 172.21.15.40:0/3076117012 shutdown_connections 2024-04-21T23:15:33.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:33.484+0000 7fef7806b640 1 -- 172.21.15.40:0/3076117012 wait complete. 2024-04-21T23:15:33.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:33.485+0000 7fef7806b640 1 Processor -- start 2024-04-21T23:15:33.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:33.485+0000 7fef7806b640 1 -- start start 2024-04-21T23:15:33.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:33.485+0000 7fef7806b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef701017b0 0x7fef700754f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:33.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:33.485+0000 7fef7806b640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fef701029a0 0x7fef70073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:33.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:33.485+0000 7fef7806b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef70103360 0x7fef70074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:33.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:33.485+0000 7fef7806b640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fef7010d450 con 0x7fef701017b0 2024-04-21T23:15:33.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:33.485+0000 7fef7806b640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fef7010d2d0 con 0x7fef701029a0 2024-04-21T23:15:33.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:33.485+0000 7fef7806b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fef7010d5d0 con 0x7fef70103360 2024-04-21T23:15:33.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:33.485+0000 7fef75e07640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef701017b0 0x7fef700754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:33.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:33.485+0000 7fef75e07640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef701017b0 0x7fef700754f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55926/0 (socket says 172.21.15.40:55926) 2024-04-21T23:15:33.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:33.485+0000 7fef75e07640 1 -- 172.21.15.40:0/1873030383 learned_addr learned my addr 172.21.15.40:0/1873030383 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:15:33.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:33.485+0000 7fef76608640 1 --2- 172.21.15.40:0/1873030383 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef70103360 0x7fef70074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:33.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:33.485+0000 7fef75606640 1 --2- 172.21.15.40:0/1873030383 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fef701029a0 0x7fef70073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:33.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:33.485+0000 7fef75e07640 1 -- 172.21.15.40:0/1873030383 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef70103360 msgr2=0x7fef70074070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:33.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:33.485+0000 7fef75e07640 1 --2- 172.21.15.40:0/1873030383 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef70103360 0x7fef70074070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:33.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:33.485+0000 7fef75e07640 1 -- 172.21.15.40:0/1873030383 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fef701029a0 msgr2=0x7fef70073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:33.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:33.485+0000 7fef75e07640 1 --2- 172.21.15.40:0/1873030383 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fef701029a0 0x7fef70073b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:33.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:33.485+0000 7fef75e07640 1 -- 172.21.15.40:0/1873030383 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef70074900 con 0x7fef701017b0 2024-04-21T23:15:33.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:33.485+0000 7fef75e07640 1 --2- 172.21.15.40:0/1873030383 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef701017b0 0x7fef700754f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fef6c000c00 tx=0x7fef6c02c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:33.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:33.486+0000 7fef66ffd640 1 -- 172.21.15.40:0/1873030383 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef6c043020 con 0x7fef701017b0 2024-04-21T23:15:33.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:33.486+0000 7fef66ffd640 1 -- 172.21.15.40:0/1873030383 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fef6c0054e0 con 0x7fef701017b0 2024-04-21T23:15:33.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:33.486+0000 7fef7806b640 1 -- 172.21.15.40:0/1873030383 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef701a1bc0 con 0x7fef701017b0 2024-04-21T23:15:33.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:33.486+0000 7fef66ffd640 1 -- 172.21.15.40:0/1873030383 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef6c003ce0 con 0x7fef701017b0 2024-04-21T23:15:33.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:33.486+0000 7fef7806b640 1 -- 172.21.15.40:0/1873030383 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef701a20e0 con 0x7fef701017b0 2024-04-21T23:15:33.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:33.487+0000 7fef7806b640 1 -- 172.21.15.40:0/1873030383 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef70108ad0 con 0x7fef701017b0 2024-04-21T23:15:33.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:33.488+0000 7fef66ffd640 1 -- 172.21.15.40:0/1873030383 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fef6c046020 con 0x7fef701017b0 2024-04-21T23:15:33.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:33.489+0000 7fef66ffd640 1 --2- 172.21.15.40:0/1873030383 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fef5c0a2b90 0x7fef5c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:33.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:33.489+0000 7fef75606640 1 --2- 172.21.15.40:0/1873030383 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fef5c0a2b90 0x7fef5c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:33.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:33.489+0000 7fef66ffd640 1 -- 172.21.15.40:0/1873030383 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fef6c0f0da0 con 0x7fef701017b0 2024-04-21T23:15:33.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:33.489+0000 7fef75606640 1 --2- 172.21.15.40:0/1873030383 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fef5c0a2b90 0x7fef5c0a5050 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fef60005fd0 tx=0x7fef60005c70 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:33.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:33.493+0000 7fef66ffd640 1 -- 172.21.15.40:0/1873030383 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fef6c04d020 con 0x7fef701017b0 2024-04-21T23:15:33.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:33.716+0000 7fef7806b640 1 -- 172.21.15.40:0/1873030383 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fef7002d3b0 con 0x7fef701017b0 2024-04-21T23:15:33.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:33.717+0000 7fef66ffd640 1 -- 172.21.15.40:0/1873030383 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5527 (secure 0 0 0) 0x7fef6c0be100 con 0x7fef701017b0 2024-04-21T23:15:33.716 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:15:33.716 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/1571948830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1571948830},{"type":"v1","addr":"172.21.15.40:6835","nonce":1571948830}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/969356642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":969356642},{"type":"v1","addr":"172.21.15.40:6837","nonce":969356642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/3438493203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":3438493203},{"type":"v1","addr":"172.21.15.174:6833","nonce":3438493203}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/3105752678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":3105752678},{"type":"v1","addr":"172.21.15.111:6833","nonce":3105752678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24461,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/2704462092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2704462092},{"type":"v1","addr":"172.21.15.111:6835","nonce":2704462092}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/3905733045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3905733045},{"type":"v1","addr":"172.21.15.174:6835","nonce":3905733045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-04-21T23:15:33.717 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 7 2024-04-21T23:15:33.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:33.719+0000 7fef7806b640 1 -- 172.21.15.40:0/1873030383 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fef5c0a2b90 msgr2=0x7fef5c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:33.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:33.719+0000 7fef7806b640 1 --2- 172.21.15.40:0/1873030383 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fef5c0a2b90 0x7fef5c0a5050 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fef60005fd0 tx=0x7fef60005c70 comp rx=0 tx=0).stop 2024-04-21T23:15:33.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:33.719+0000 7fef7806b640 1 -- 172.21.15.40:0/1873030383 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef701017b0 msgr2=0x7fef700754f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:33.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:33.719+0000 7fef7806b640 1 --2- 172.21.15.40:0/1873030383 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef701017b0 0x7fef700754f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fef6c000c00 tx=0x7fef6c02c6c0 comp rx=0 tx=0).stop 2024-04-21T23:15:33.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:33.719+0000 7fef7806b640 1 -- 172.21.15.40:0/1873030383 shutdown_connections 2024-04-21T23:15:33.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:33.719+0000 7fef7806b640 1 --2- 172.21.15.40:0/1873030383 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fef5c0a2b90 0x7fef5c0a5050 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:33.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:33.719+0000 7fef7806b640 1 --2- 172.21.15.40:0/1873030383 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fef701029a0 0x7fef70073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:33.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:33.719+0000 7fef7806b640 1 --2- 172.21.15.40:0/1873030383 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef70103360 0x7fef70074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:33.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:33.719+0000 7fef7806b640 1 --2- 172.21.15.40:0/1873030383 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef701017b0 0x7fef700754f0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:33.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:33.719+0000 7fef7806b640 1 -- 172.21.15.40:0/1873030383 >> 172.21.15.40:0/1873030383 conn(0x7fef700fcd40 msgr2=0x7fef70105ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:33.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:33.719+0000 7fef7806b640 1 -- 172.21.15.40:0/1873030383 shutdown_connections 2024-04-21T23:15:33.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:33.719+0000 7fef7806b640 1 -- 172.21.15.40:0/1873030383 wait complete. 2024-04-21T23:15:34.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:34 smithi174 ceph-mon[31409]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:34.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:34 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:34.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:34 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:34.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:34 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:34.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:34 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:15:34.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:34 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:15:34.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:34 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:15:34.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:34 smithi174 ceph-mon[31409]: Deploying daemon mds.c on smithi174 2024-04-21T23:15:34.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:34 smithi174 ceph-mon[31409]: mds.? [v2:172.21.15.174:6834/3905733045,v1:172.21.15.174:6835/3905733045] up:boot 2024-04-21T23:15:34.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:34 smithi174 ceph-mon[31409]: fsmap 6 up:standby 2024-04-21T23:15:34.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:34 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-21T23:15:34.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:34 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/1873030383' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:15:34.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:34 smithi040 ceph-mon[27984]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:34.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:34 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:34.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:34 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:34.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:34 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:34.683 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:34 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:15:34.683 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:34 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:15:34.683 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:34 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:15:34.683 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:34 smithi040 ceph-mon[27984]: Deploying daemon mds.c on smithi174 2024-04-21T23:15:34.683 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:34 smithi040 ceph-mon[27984]: mds.? [v2:172.21.15.174:6834/3905733045,v1:172.21.15.174:6835/3905733045] up:boot 2024-04-21T23:15:34.683 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:34 smithi040 ceph-mon[27984]: fsmap 6 up:standby 2024-04-21T23:15:34.683 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:34 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-21T23:15:34.683 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:34 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/1873030383' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:15:34.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:34 smithi111 ceph-mon[31085]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:34.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:34 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:34.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:34 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:34.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:34 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:34.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:34 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:15:34.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:34 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:15:34.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:34 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:15:34.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:34 smithi111 ceph-mon[31085]: Deploying daemon mds.c on smithi174 2024-04-21T23:15:34.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:34 smithi111 ceph-mon[31085]: mds.? [v2:172.21.15.174:6834/3905733045,v1:172.21.15.174:6835/3905733045] up:boot 2024-04-21T23:15:34.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:34 smithi111 ceph-mon[31085]: fsmap 6 up:standby 2024-04-21T23:15:34.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:34 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-21T23:15:34.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:34 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/1873030383' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:15:35.444 INFO:journalctl@ceph.mds.c.smithi174.stdout:Apr 21 23:15:35 smithi174 systemd[1]: Starting Ceph mds.c for d814e4dc-0033-11ef-bc93-c7b262605968... 2024-04-21T23:15:35.730 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:15:35.843 INFO:journalctl@ceph.mds.c.smithi174.stdout:Apr 21 23:15:35 smithi174 podman[61787]: 2024-04-21 23:15:35.458973369 +0000 UTC m=+0.017756517 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-21T23:15:35.843 INFO:journalctl@ceph.mds.c.smithi174.stdout:Apr 21 23:15:35 smithi174 podman[61787]: 2024-04-21 23:15:35.588462789 +0000 UTC m=+0.147245926 container create 92e760e59bf0ab44bb316b8ce80313b265f5d4e50eef9e03560f68294cd3bd9b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-c, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-820a2f8, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T23:15:35.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:35.846+0000 7f3b20bde640 1 -- 172.21.15.40:0/46596744 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b1c1006a0 msgr2=0x7f3b1c100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:35.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:35.846+0000 7f3b20bde640 1 --2- 172.21.15.40:0/46596744 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b1c1006a0 0x7f3b1c100ac0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f3b0c009900 tx=0x7f3b0c02bc30 comp rx=0 tx=0).stop 2024-04-21T23:15:35.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:35.846+0000 7f3b20bde640 1 -- 172.21.15.40:0/46596744 shutdown_connections 2024-04-21T23:15:35.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:35.846+0000 7f3b20bde640 1 --2- 172.21.15.40:0/46596744 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b1c102250 0x7f3b1c108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:35.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:35.846+0000 7f3b20bde640 1 --2- 172.21.15.40:0/46596744 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3b1c101890 0x7f3b1c101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:35.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:35.846+0000 7f3b20bde640 1 --2- 172.21.15.40:0/46596744 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b1c1006a0 0x7f3b1c100ac0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:35.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:35.846+0000 7f3b20bde640 1 -- 172.21.15.40:0/46596744 >> 172.21.15.40:0/46596744 conn(0x7f3b1c0fbc30 msgr2=0x7f3b1c0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:35.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:35.846+0000 7f3b20bde640 1 -- 172.21.15.40:0/46596744 shutdown_connections 2024-04-21T23:15:35.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:35.846+0000 7f3b20bde640 1 -- 172.21.15.40:0/46596744 wait complete. 2024-04-21T23:15:35.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:35.847+0000 7f3b20bde640 1 Processor -- start 2024-04-21T23:15:35.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:35.847+0000 7f3b20bde640 1 -- start start 2024-04-21T23:15:35.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:35.847+0000 7f3b20bde640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3b1c1006a0 0x7f3b1c06a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:35.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:35.847+0000 7f3b20bde640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b1c101890 0x7f3b1c19c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:35.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:35.847+0000 7f3b20bde640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b1c102250 0x7f3b1c19ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:35.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:35.847+0000 7f3b20bde640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3b1c10c340 con 0x7f3b1c102250 2024-04-21T23:15:35.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:35.847+0000 7f3b20bde640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3b1c10c1c0 con 0x7f3b1c101890 2024-04-21T23:15:35.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:35.847+0000 7f3b20bde640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3b1c10c4c0 con 0x7f3b1c1006a0 2024-04-21T23:15:35.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:35.847+0000 7f3b1ad9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b1c102250 0x7f3b1c19ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:35.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:35.847+0000 7f3b1a59c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3b1c1006a0 0x7f3b1c06a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:35.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:35.847+0000 7f3b1ad9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b1c102250 0x7f3b1c19ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55944/0 (socket says 172.21.15.40:55944) 2024-04-21T23:15:35.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:35.847+0000 7f3b1ad9d640 1 -- 172.21.15.40:0/87004953 learned_addr learned my addr 172.21.15.40:0/87004953 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:15:35.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:35.847+0000 7f3b13fff640 1 --2- 172.21.15.40:0/87004953 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b1c101890 0x7f3b1c19c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:35.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:35.848+0000 7f3b1ad9d640 1 -- 172.21.15.40:0/87004953 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3b1c1006a0 msgr2=0x7f3b1c06a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:35.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:35.848+0000 7f3b1ad9d640 1 --2- 172.21.15.40:0/87004953 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3b1c1006a0 0x7f3b1c06a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:35.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:35.848+0000 7f3b1ad9d640 1 -- 172.21.15.40:0/87004953 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b1c101890 msgr2=0x7f3b1c19c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:35.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:35.848+0000 7f3b1ad9d640 1 --2- 172.21.15.40:0/87004953 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b1c101890 0x7f3b1c19c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:35.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:35.848+0000 7f3b1ad9d640 1 -- 172.21.15.40:0/87004953 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b1c19f6d0 con 0x7f3b1c102250 2024-04-21T23:15:35.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:35.848+0000 7f3b1ad9d640 1 --2- 172.21.15.40:0/87004953 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b1c102250 0x7f3b1c19ee10 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f3b1400eaa0 tx=0x7f3b1400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:35.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:35.848+0000 7f3b137fe640 1 -- 172.21.15.40:0/87004953 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b1400cdb0 con 0x7f3b1c102250 2024-04-21T23:15:35.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:35.848+0000 7f3b137fe640 1 -- 172.21.15.40:0/87004953 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b14005020 con 0x7f3b1c102250 2024-04-21T23:15:35.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:35.848+0000 7f3b20bde640 1 -- 172.21.15.40:0/87004953 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b1c19f9c0 con 0x7f3b1c102250 2024-04-21T23:15:35.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:35.848+0000 7f3b137fe640 1 -- 172.21.15.40:0/87004953 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b140116b0 con 0x7f3b1c102250 2024-04-21T23:15:35.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:35.848+0000 7f3b20bde640 1 -- 172.21.15.40:0/87004953 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b1c19fec0 con 0x7f3b1c102250 2024-04-21T23:15:35.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:35.849+0000 7f3b20bde640 1 -- 172.21.15.40:0/87004953 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b1c1079c0 con 0x7f3b1c102250 2024-04-21T23:15:35.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:35.850+0000 7f3b137fe640 1 -- 172.21.15.40:0/87004953 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f3b14005560 con 0x7f3b1c102250 2024-04-21T23:15:35.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:35.851+0000 7f3b137fe640 1 --2- 172.21.15.40:0/87004953 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f3b080a2b90 0x7f3b080a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:35.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:35.851+0000 7f3b137fe640 1 -- 172.21.15.40:0/87004953 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f3b140ced70 con 0x7f3b1c102250 2024-04-21T23:15:35.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:35.851+0000 7f3b1a59c640 1 --2- 172.21.15.40:0/87004953 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f3b080a2b90 0x7f3b080a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:35.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:35.852+0000 7f3b1a59c640 1 --2- 172.21.15.40:0/87004953 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f3b080a2b90 0x7f3b080a5050 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f3b0c02c170 tx=0x7f3b0c02df90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:35.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:35.856+0000 7f3b137fe640 1 -- 172.21.15.40:0/87004953 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f3b14016030 con 0x7f3b1c102250 2024-04-21T23:15:36.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:36.083+0000 7f3b20bde640 1 -- 172.21.15.40:0/87004953 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3b1c02d170 con 0x7f3b1c102250 2024-04-21T23:15:36.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:36.083+0000 7f3b137fe640 1 -- 172.21.15.40:0/87004953 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5527 (secure 0 0 0) 0x7f3b1409bb20 con 0x7f3b1c102250 2024-04-21T23:15:36.082 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:15:36.083 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/1571948830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1571948830},{"type":"v1","addr":"172.21.15.40:6835","nonce":1571948830}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/969356642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":969356642},{"type":"v1","addr":"172.21.15.40:6837","nonce":969356642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/3438493203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":3438493203},{"type":"v1","addr":"172.21.15.174:6833","nonce":3438493203}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/3105752678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":3105752678},{"type":"v1","addr":"172.21.15.111:6833","nonce":3105752678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24461,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/2704462092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2704462092},{"type":"v1","addr":"172.21.15.111:6835","nonce":2704462092}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/3905733045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3905733045},{"type":"v1","addr":"172.21.15.174:6835","nonce":3905733045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-04-21T23:15:36.083 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 7 2024-04-21T23:15:36.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:36.085+0000 7f3b20bde640 1 -- 172.21.15.40:0/87004953 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f3b080a2b90 msgr2=0x7f3b080a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:36.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:36.085+0000 7f3b20bde640 1 --2- 172.21.15.40:0/87004953 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f3b080a2b90 0x7f3b080a5050 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f3b0c02c170 tx=0x7f3b0c02df90 comp rx=0 tx=0).stop 2024-04-21T23:15:36.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:36.085+0000 7f3b20bde640 1 -- 172.21.15.40:0/87004953 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b1c102250 msgr2=0x7f3b1c19ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:36.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:36.085+0000 7f3b20bde640 1 --2- 172.21.15.40:0/87004953 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b1c102250 0x7f3b1c19ee10 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f3b1400eaa0 tx=0x7f3b1400ef60 comp rx=0 tx=0).stop 2024-04-21T23:15:36.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:36.085+0000 7f3b20bde640 1 -- 172.21.15.40:0/87004953 shutdown_connections 2024-04-21T23:15:36.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:36.085+0000 7f3b20bde640 1 --2- 172.21.15.40:0/87004953 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f3b080a2b90 0x7f3b080a5050 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:36.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:36.085+0000 7f3b20bde640 1 --2- 172.21.15.40:0/87004953 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b1c101890 0x7f3b1c19c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:36.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:36.085+0000 7f3b20bde640 1 --2- 172.21.15.40:0/87004953 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b1c102250 0x7f3b1c19ee10 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:36.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:36.085+0000 7f3b20bde640 1 --2- 172.21.15.40:0/87004953 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3b1c1006a0 0x7f3b1c06a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:36.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:36.085+0000 7f3b20bde640 1 -- 172.21.15.40:0/87004953 >> 172.21.15.40:0/87004953 conn(0x7f3b1c0fbc30 msgr2=0x7f3b1c104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:36.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:36.085+0000 7f3b20bde640 1 -- 172.21.15.40:0/87004953 shutdown_connections 2024-04-21T23:15:36.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:36.085+0000 7f3b20bde640 1 -- 172.21.15.40:0/87004953 wait complete. 2024-04-21T23:15:36.194 INFO:journalctl@ceph.mds.c.smithi174.stdout:Apr 21 23:15:35 smithi174 podman[61787]: 2024-04-21 23:15:35.844961505 +0000 UTC m=+0.403744643 container init 92e760e59bf0ab44bb316b8ce80313b265f5d4e50eef9e03560f68294cd3bd9b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-c, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=squid-820a2f8, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-21T23:15:36.194 INFO:journalctl@ceph.mds.c.smithi174.stdout:Apr 21 23:15:35 smithi174 podman[61787]: 2024-04-21 23:15:35.847751753 +0000 UTC m=+0.406534891 container start 92e760e59bf0ab44bb316b8ce80313b265f5d4e50eef9e03560f68294cd3bd9b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-c, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-820a2f8, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, ceph=True) 2024-04-21T23:15:36.194 INFO:journalctl@ceph.mds.c.smithi174.stdout:Apr 21 23:15:35 smithi174 ceph-mds[61807]: -- 172.21.15.174:0/710478517 <== mon.1 v2:172.21.15.174:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ccfad79a40 con 0x55ccfad4cc00 2024-04-21T23:15:36.194 INFO:journalctl@ceph.mds.c.smithi174.stdout:Apr 21 23:15:35 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-c[61803]: starting mds.c at 2024-04-21T23:15:36.194 INFO:journalctl@ceph.mds.c.smithi174.stdout:Apr 21 23:15:35 smithi174 bash[61787]: 92e760e59bf0ab44bb316b8ce80313b265f5d4e50eef9e03560f68294cd3bd9b 2024-04-21T23:15:36.194 INFO:journalctl@ceph.mds.c.smithi174.stdout:Apr 21 23:15:35 smithi174 systemd[1]: Started Ceph mds.c for d814e4dc-0033-11ef-bc93-c7b262605968. 2024-04-21T23:15:36.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:36 smithi040 ceph-mon[27984]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:36.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:36 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:36.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:36 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:36.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:36 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:36.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:36 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:15:36.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:36 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:15:36.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:36 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:15:36.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:36 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/87004953' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:15:36.654 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:36 smithi111 ceph-mon[31085]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:36.655 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:36 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:36.655 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:36 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:36.655 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:36 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:36.655 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:36 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:15:36.655 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:36 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:15:36.655 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:36 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:15:36.655 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:36 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/87004953' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:15:36.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:36 smithi174 ceph-mon[31409]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:36.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:36 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:36.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:36 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:36.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:36 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:36.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:36 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:15:36.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:36 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:15:36.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:36 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:15:36.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:36 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/87004953' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:15:37.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:37 smithi040 ceph-mon[27984]: Deploying daemon mds.h on smithi111 2024-04-21T23:15:37.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:37 smithi040 ceph-mon[27984]: mds.? [v2:172.21.15.174:6836/465579746,v1:172.21.15.174:6837/465579746] up:boot 2024-04-21T23:15:37.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:37 smithi040 ceph-mon[27984]: fsmap 7 up:standby 2024-04-21T23:15:37.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:37 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-21T23:15:37.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:37 smithi174 ceph-mon[31409]: Deploying daemon mds.h on smithi111 2024-04-21T23:15:37.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:37 smithi174 ceph-mon[31409]: mds.? [v2:172.21.15.174:6836/465579746,v1:172.21.15.174:6837/465579746] up:boot 2024-04-21T23:15:37.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:37 smithi174 ceph-mon[31409]: fsmap 7 up:standby 2024-04-21T23:15:37.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:37 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-21T23:15:37.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:37 smithi111 ceph-mon[31085]: Deploying daemon mds.h on smithi111 2024-04-21T23:15:37.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:37 smithi111 ceph-mon[31085]: mds.? [v2:172.21.15.174:6836/465579746,v1:172.21.15.174:6837/465579746] up:boot 2024-04-21T23:15:37.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:37 smithi111 ceph-mon[31085]: fsmap 7 up:standby 2024-04-21T23:15:37.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:37 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-21T23:15:38.096 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:15:38.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:38.214+0000 7f6fd2d1e640 1 -- 172.21.15.40:0/361052261 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6fcc102980 msgr2=0x7f6fcc102e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:38.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:38.214+0000 7f6fd2d1e640 1 --2- 172.21.15.40:0/361052261 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6fcc102980 0x7f6fcc102e00 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f6fb8009900 tx=0x7f6fb802bc30 comp rx=0 tx=0).stop 2024-04-21T23:15:38.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:38.214+0000 7f6fd2d1e640 1 -- 172.21.15.40:0/361052261 shutdown_connections 2024-04-21T23:15:38.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:38.214+0000 7f6fd2d1e640 1 --2- 172.21.15.40:0/361052261 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6fcc103340 0x7f6fcc109bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:38.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:38.214+0000 7f6fd2d1e640 1 --2- 172.21.15.40:0/361052261 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6fcc102980 0x7f6fcc102e00 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:38.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:38.214+0000 7f6fd2d1e640 1 --2- 172.21.15.40:0/361052261 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6fcc101790 0x7f6fcc101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:38.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:38.214+0000 7f6fd2d1e640 1 -- 172.21.15.40:0/361052261 >> 172.21.15.40:0/361052261 conn(0x7f6fcc0fcd20 msgr2=0x7f6fcc0ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:38.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:38.214+0000 7f6fd2d1e640 1 -- 172.21.15.40:0/361052261 shutdown_connections 2024-04-21T23:15:38.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:38.214+0000 7f6fd2d1e640 1 -- 172.21.15.40:0/361052261 wait complete. 2024-04-21T23:15:38.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:38.215+0000 7f6fd2d1e640 1 Processor -- start 2024-04-21T23:15:38.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:38.215+0000 7f6fd2d1e640 1 -- start start 2024-04-21T23:15:38.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:38.215+0000 7f6fd2d1e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6fcc101790 0x7f6fcc199ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:38.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:38.215+0000 7f6fd2d1e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6fcc102980 0x7f6fcc19a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:38.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:38.215+0000 7f6fd2d1e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6fcc103340 0x7f6fcc1a1240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:38.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:38.215+0000 7f6fd2d1e640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6fcc10d440 con 0x7f6fcc102980 2024-04-21T23:15:38.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:38.215+0000 7f6fd2d1e640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f6fcc10d2c0 con 0x7f6fcc103340 2024-04-21T23:15:38.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:38.215+0000 7f6fd2d1e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6fcc10d5c0 con 0x7f6fcc101790 2024-04-21T23:15:38.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:38.215+0000 7f6fcbfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6fcc102980 0x7f6fcc19a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:38.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:38.215+0000 7f6fcbfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6fcc102980 0x7f6fcc19a1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55964/0 (socket says 172.21.15.40:55964) 2024-04-21T23:15:38.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:38.215+0000 7f6fcbfff640 1 -- 172.21.15.40:0/3105469679 learned_addr learned my addr 172.21.15.40:0/3105469679 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:15:38.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:38.215+0000 7f6fd12bb640 1 --2- 172.21.15.40:0/3105469679 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6fcc103340 0x7f6fcc1a1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:38.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:38.215+0000 7f6fd0aba640 1 --2- 172.21.15.40:0/3105469679 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6fcc101790 0x7f6fcc199ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:38.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:38.216+0000 7f6fcbfff640 1 -- 172.21.15.40:0/3105469679 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6fcc101790 msgr2=0x7f6fcc199ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:38.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:38.216+0000 7f6fcbfff640 1 --2- 172.21.15.40:0/3105469679 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6fcc101790 0x7f6fcc199ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:38.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:38.216+0000 7f6fcbfff640 1 -- 172.21.15.40:0/3105469679 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6fcc103340 msgr2=0x7f6fcc1a1240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:38.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:38.216+0000 7f6fcbfff640 1 --2- 172.21.15.40:0/3105469679 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6fcc103340 0x7f6fcc1a1240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:38.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:38.216+0000 7f6fcbfff640 1 -- 172.21.15.40:0/3105469679 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fcc1a18e0 con 0x7f6fcc102980 2024-04-21T23:15:38.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:38.216+0000 7f6fd12bb640 1 --2- 172.21.15.40:0/3105469679 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6fcc103340 0x7f6fcc1a1240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:15:38.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:38.216+0000 7f6fcbfff640 1 --2- 172.21.15.40:0/3105469679 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6fcc102980 0x7f6fcc19a1e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6fb8005ab0 tx=0x7f6fb802c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:38.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:38.216+0000 7f6fc9ffb640 1 -- 172.21.15.40:0/3105469679 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fb8043020 con 0x7f6fcc102980 2024-04-21T23:15:38.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:38.216+0000 7f6fc9ffb640 1 -- 172.21.15.40:0/3105469679 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6fb80054e0 con 0x7f6fcc102980 2024-04-21T23:15:38.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:38.216+0000 7f6fd2d1e640 1 -- 172.21.15.40:0/3105469679 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fcc1a1b70 con 0x7f6fcc102980 2024-04-21T23:15:38.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:38.216+0000 7f6fc9ffb640 1 -- 172.21.15.40:0/3105469679 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fb8003ce0 con 0x7f6fcc102980 2024-04-21T23:15:38.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:38.216+0000 7f6fd2d1e640 1 -- 172.21.15.40:0/3105469679 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fcc1a20c0 con 0x7f6fcc102980 2024-04-21T23:15:38.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:38.217+0000 7f6fd2d1e640 1 -- 172.21.15.40:0/3105469679 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6fcc1a2290 con 0x7f6fcc102980 2024-04-21T23:15:38.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:38.219+0000 7f6fc9ffb640 1 -- 172.21.15.40:0/3105469679 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f6fb8046020 con 0x7f6fcc102980 2024-04-21T23:15:38.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:38.219+0000 7f6fc9ffb640 1 --2- 172.21.15.40:0/3105469679 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f6fbc0a2b90 0x7f6fbc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:38.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:38.219+0000 7f6fd0aba640 1 --2- 172.21.15.40:0/3105469679 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f6fbc0a2b90 0x7f6fbc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:38.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:38.219+0000 7f6fc9ffb640 1 -- 172.21.15.40:0/3105469679 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f6fb80f0c60 con 0x7f6fcc102980 2024-04-21T23:15:38.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:38.220+0000 7f6fd0aba640 1 --2- 172.21.15.40:0/3105469679 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f6fbc0a2b90 0x7f6fbc0a5050 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f6fcc19ab70 tx=0x7f6fc000b390 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:38.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:38.224+0000 7f6fc9ffb640 1 -- 172.21.15.40:0/3105469679 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f6fb804d020 con 0x7f6fcc102980 2024-04-21T23:15:38.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:38 smithi040 ceph-mon[27984]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:38.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:38 smithi111 ceph-mon[31085]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:38.444 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:15:38 smithi111 systemd[1]: Starting Ceph mds.h for d814e4dc-0033-11ef-bc93-c7b262605968... 2024-04-21T23:15:38.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:38.446+0000 7f6fd2d1e640 1 -- 172.21.15.40:0/3105469679 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6fcc1a2620 con 0x7f6fcc102980 2024-04-21T23:15:38.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:38.447+0000 7f6fc9ffb640 1 -- 172.21.15.40:0/3105469679 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6352 (secure 0 0 0) 0x7f6fb80be060 con 0x7f6fcc102980 2024-04-21T23:15:38.446 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:15:38.446 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/1571948830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1571948830},{"type":"v1","addr":"172.21.15.40:6835","nonce":1571948830}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/969356642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":969356642},{"type":"v1","addr":"172.21.15.40:6837","nonce":969356642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/3438493203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":3438493203},{"type":"v1","addr":"172.21.15.174:6833","nonce":3438493203}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/3105752678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":3105752678},{"type":"v1","addr":"172.21.15.111:6833","nonce":3105752678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24461,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/2704462092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2704462092},{"type":"v1","addr":"172.21.15.111:6835","nonce":2704462092}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/3905733045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3905733045},{"type":"v1","addr":"172.21.15.174:6835","nonce":3905733045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24484,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/465579746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":465579746},{"type":"v1","addr":"172.21.15.174:6837","nonce":465579746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-04-21T23:15:38.447 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 8 2024-04-21T23:15:38.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:38.449+0000 7f6fd2d1e640 1 -- 172.21.15.40:0/3105469679 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f6fbc0a2b90 msgr2=0x7f6fbc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:38.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:38.449+0000 7f6fd2d1e640 1 --2- 172.21.15.40:0/3105469679 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f6fbc0a2b90 0x7f6fbc0a5050 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f6fcc19ab70 tx=0x7f6fc000b390 comp rx=0 tx=0).stop 2024-04-21T23:15:38.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:38.449+0000 7f6fd2d1e640 1 -- 172.21.15.40:0/3105469679 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6fcc102980 msgr2=0x7f6fcc19a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:38.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:38.449+0000 7f6fd2d1e640 1 --2- 172.21.15.40:0/3105469679 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6fcc102980 0x7f6fcc19a1e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6fb8005ab0 tx=0x7f6fb802c6c0 comp rx=0 tx=0).stop 2024-04-21T23:15:38.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:38.449+0000 7f6fd2d1e640 1 -- 172.21.15.40:0/3105469679 shutdown_connections 2024-04-21T23:15:38.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:38.449+0000 7f6fd2d1e640 1 --2- 172.21.15.40:0/3105469679 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f6fbc0a2b90 0x7f6fbc0a5050 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:38.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:38.449+0000 7f6fd2d1e640 1 --2- 172.21.15.40:0/3105469679 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6fcc103340 0x7f6fcc1a1240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:38.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:38.449+0000 7f6fd2d1e640 1 --2- 172.21.15.40:0/3105469679 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6fcc102980 0x7f6fcc19a1e0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:38.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:38.449+0000 7f6fd2d1e640 1 --2- 172.21.15.40:0/3105469679 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6fcc101790 0x7f6fcc199ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:38.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:38.449+0000 7f6fd2d1e640 1 -- 172.21.15.40:0/3105469679 >> 172.21.15.40:0/3105469679 conn(0x7f6fcc0fcd20 msgr2=0x7f6fcc073970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:38.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:38.449+0000 7f6fd2d1e640 1 -- 172.21.15.40:0/3105469679 shutdown_connections 2024-04-21T23:15:38.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:38.450+0000 7f6fd2d1e640 1 -- 172.21.15.40:0/3105469679 wait complete. 2024-04-21T23:15:38.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:38 smithi174 ceph-mon[31409]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:38.732 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:15:38 smithi111 podman[61094]: 2024-04-21 23:15:38.358397223 +0000 UTC m=+0.017863362 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-21T23:15:38.732 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:15:38 smithi111 podman[61094]: 2024-04-21 23:15:38.471692129 +0000 UTC m=+0.131158291 container create c4271d3bd94d138b8e00f8cca0e2be7e9c461a91d439be0039946cb7c1d34b64 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-820a2f8, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416) 2024-04-21T23:15:38.732 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:15:38 smithi111 podman[61094]: 2024-04-21 23:15:38.733212366 +0000 UTC m=+0.392678482 container init c4271d3bd94d138b8e00f8cca0e2be7e9c461a91d439be0039946cb7c1d34b64 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-820a2f8, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-21T23:15:39.194 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:15:38 smithi111 podman[61094]: 2024-04-21 23:15:38.735946403 +0000 UTC m=+0.395412526 container start c4271d3bd94d138b8e00f8cca0e2be7e9c461a91d439be0039946cb7c1d34b64 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=squid-820a2f8, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-21T23:15:39.194 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:15:38 smithi111 ceph-mds[61113]: -- 172.21.15.111:0/2786412181 <== mon.2 v2:172.21.15.111:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5572f1ea5a40 con 0x5572f1e78c00 2024-04-21T23:15:39.194 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:15:38 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: starting mds.h at 2024-04-21T23:15:39.194 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:15:38 smithi111 bash[61094]: c4271d3bd94d138b8e00f8cca0e2be7e9c461a91d439be0039946cb7c1d34b64 2024-04-21T23:15:39.194 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:15:38 smithi111 systemd[1]: Started Ceph mds.h for d814e4dc-0033-11ef-bc93-c7b262605968. 2024-04-21T23:15:39.536 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:39 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/3105469679' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:15:39.537 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:39 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:39.537 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:39 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:39.537 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:39 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:39.537 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:39 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:15:39.537 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:39 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:15:39.537 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:39 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:15:39.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:39 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/3105469679' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:15:39.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:39 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:39.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:39 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:39.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:39 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:39.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:39 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:15:39.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:39 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:15:39.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:39 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:15:39.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:39 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/3105469679' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:15:39.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:39 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:39.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:39 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:39.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:39 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:39.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:39 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:15:39.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:39 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:15:39.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:39 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:15:40.460 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:15:40.486 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:40 smithi040 ceph-mon[27984]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:40.487 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:40 smithi040 ceph-mon[27984]: Deploying daemon mds.i on smithi174 2024-04-21T23:15:40.487 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:40 smithi040 ceph-mon[27984]: mds.? [v2:172.21.15.111:6836/2703764182,v1:172.21.15.111:6837/2703764182] up:boot 2024-04-21T23:15:40.487 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:40 smithi040 ceph-mon[27984]: fsmap 8 up:standby 2024-04-21T23:15:40.487 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:40 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-21T23:15:40.534 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:40 smithi174 ceph-mon[31409]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:40.535 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:40 smithi174 ceph-mon[31409]: Deploying daemon mds.i on smithi174 2024-04-21T23:15:40.535 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:40 smithi174 ceph-mon[31409]: mds.? [v2:172.21.15.111:6836/2703764182,v1:172.21.15.111:6837/2703764182] up:boot 2024-04-21T23:15:40.535 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:40 smithi174 ceph-mon[31409]: fsmap 8 up:standby 2024-04-21T23:15:40.535 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:40 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-21T23:15:40.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:40.578+0000 7fe5b4cd4640 1 -- 172.21.15.40:0/1220999946 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe5b0100680 msgr2=0x7fe5b0100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:40.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:40.578+0000 7fe5b4cd4640 1 --2- 172.21.15.40:0/1220999946 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe5b0100680 0x7fe5b0100aa0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fe5a0009900 tx=0x7fe5a002bc30 comp rx=0 tx=0).stop 2024-04-21T23:15:40.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:40.578+0000 7fe5b4cd4640 1 -- 172.21.15.40:0/1220999946 shutdown_connections 2024-04-21T23:15:40.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:40.578+0000 7fe5b4cd4640 1 --2- 172.21.15.40:0/1220999946 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5b0102230 0x7fe5b0108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:40.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:40.578+0000 7fe5b4cd4640 1 --2- 172.21.15.40:0/1220999946 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe5b0101870 0x7fe5b0101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:40.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:40.578+0000 7fe5b4cd4640 1 --2- 172.21.15.40:0/1220999946 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe5b0100680 0x7fe5b0100aa0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:40.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:40.578+0000 7fe5b4cd4640 1 -- 172.21.15.40:0/1220999946 >> 172.21.15.40:0/1220999946 conn(0x7fe5b00fbbf0 msgr2=0x7fe5b00fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:40.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:40.578+0000 7fe5b4cd4640 1 -- 172.21.15.40:0/1220999946 shutdown_connections 2024-04-21T23:15:40.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:40.578+0000 7fe5b4cd4640 1 -- 172.21.15.40:0/1220999946 wait complete. 2024-04-21T23:15:40.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:40.579+0000 7fe5b4cd4640 1 Processor -- start 2024-04-21T23:15:40.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:40.579+0000 7fe5b4cd4640 1 -- start start 2024-04-21T23:15:40.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:40.579+0000 7fe5b4cd4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe5b0100680 0x7fe5b0104610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:40.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:40.579+0000 7fe5b4cd4640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5b0101870 0x7fe5b0102c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:40.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:40.579+0000 7fe5b4cd4640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe5b0102230 0x7fe5b0103190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:40.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:40.579+0000 7fe5b4cd4640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe5b010c300 con 0x7fe5b0102230 2024-04-21T23:15:40.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:40.579+0000 7fe5b4cd4640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe5b010c180 con 0x7fe5b0101870 2024-04-21T23:15:40.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:40.579+0000 7fe5b4cd4640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe5b010c480 con 0x7fe5b0100680 2024-04-21T23:15:40.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:40.579+0000 7fe5aed9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe5b0102230 0x7fe5b0103190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:40.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:40.579+0000 7fe5aed9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe5b0102230 0x7fe5b0103190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55976/0 (socket says 172.21.15.40:55976) 2024-04-21T23:15:40.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:40.579+0000 7fe5aed9d640 1 -- 172.21.15.40:0/2825922411 learned_addr learned my addr 172.21.15.40:0/2825922411 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:15:40.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:40.579+0000 7fe5ae59c640 1 --2- 172.21.15.40:0/2825922411 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe5b0100680 0x7fe5b0104610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:40.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:40.579+0000 7fe5a7fff640 1 --2- 172.21.15.40:0/2825922411 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5b0101870 0x7fe5b0102c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:40.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:40.579+0000 7fe5aed9d640 1 -- 172.21.15.40:0/2825922411 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe5b0100680 msgr2=0x7fe5b0104610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:40.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:40.579+0000 7fe5aed9d640 1 --2- 172.21.15.40:0/2825922411 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe5b0100680 0x7fe5b0104610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:40.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:40.579+0000 7fe5aed9d640 1 -- 172.21.15.40:0/2825922411 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5b0101870 msgr2=0x7fe5b0102c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:40.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:40.579+0000 7fe5aed9d640 1 --2- 172.21.15.40:0/2825922411 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5b0101870 0x7fe5b0102c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:40.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:40.579+0000 7fe5aed9d640 1 -- 172.21.15.40:0/2825922411 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5b0103a20 con 0x7fe5b0102230 2024-04-21T23:15:40.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:40.579+0000 7fe5aed9d640 1 --2- 172.21.15.40:0/2825922411 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe5b0102230 0x7fe5b0103190 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fe5a800eaa0 tx=0x7fe5a800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:40.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:40.580+0000 7fe5a77fe640 1 -- 172.21.15.40:0/2825922411 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5a800cdb0 con 0x7fe5b0102230 2024-04-21T23:15:40.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:40.580+0000 7fe5b4cd4640 1 -- 172.21.15.40:0/2825922411 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5b01a1bf0 con 0x7fe5b0102230 2024-04-21T23:15:40.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:40.580+0000 7fe5a77fe640 1 -- 172.21.15.40:0/2825922411 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe5a8005020 con 0x7fe5b0102230 2024-04-21T23:15:40.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:40.580+0000 7fe5a77fe640 1 -- 172.21.15.40:0/2825922411 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5a80116b0 con 0x7fe5b0102230 2024-04-21T23:15:40.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:40.580+0000 7fe5b4cd4640 1 -- 172.21.15.40:0/2825922411 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5b01a2140 con 0x7fe5b0102230 2024-04-21T23:15:40.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:40.581+0000 7fe5b4cd4640 1 -- 172.21.15.40:0/2825922411 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5b01a2370 con 0x7fe5b0102230 2024-04-21T23:15:40.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:40.582+0000 7fe5a77fe640 1 -- 172.21.15.40:0/2825922411 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fe5a8005560 con 0x7fe5b0102230 2024-04-21T23:15:40.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:40.583+0000 7fe5a77fe640 1 --2- 172.21.15.40:0/2825922411 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fe59c0a2b90 0x7fe59c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:40.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:40.583+0000 7fe5ae59c640 1 --2- 172.21.15.40:0/2825922411 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fe59c0a2b90 0x7fe59c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:40.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:40.583+0000 7fe5a77fe640 1 -- 172.21.15.40:0/2825922411 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fe5a80cee90 con 0x7fe5b0102230 2024-04-21T23:15:40.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:40.583+0000 7fe5ae59c640 1 --2- 172.21.15.40:0/2825922411 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fe59c0a2b90 0x7fe59c0a5050 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe5a0000c00 tx=0x7fe5a0005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:40.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:40.587+0000 7fe5a77fe640 1 -- 172.21.15.40:0/2825922411 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fe5a8016030 con 0x7fe5b0102230 2024-04-21T23:15:40.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:40 smithi111 ceph-mon[31085]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:40.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:40 smithi111 ceph-mon[31085]: Deploying daemon mds.i on smithi174 2024-04-21T23:15:40.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:40 smithi111 ceph-mon[31085]: mds.? [v2:172.21.15.111:6836/2703764182,v1:172.21.15.111:6837/2703764182] up:boot 2024-04-21T23:15:40.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:40 smithi111 ceph-mon[31085]: fsmap 8 up:standby 2024-04-21T23:15:40.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:40 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-21T23:15:40.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:40.811+0000 7fe5b4cd4640 1 -- 172.21.15.40:0/2825922411 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe5b01a2660 con 0x7fe5b0102230 2024-04-21T23:15:40.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:40.812+0000 7fe5a77fe640 1 -- 172.21.15.40:0/2825922411 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7180 (secure 0 0 0) 0x7fe5a809c290 con 0x7fe5b0102230 2024-04-21T23:15:40.811 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:15:40.811 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/1571948830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1571948830},{"type":"v1","addr":"172.21.15.40:6835","nonce":1571948830}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/969356642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":969356642},{"type":"v1","addr":"172.21.15.40:6837","nonce":969356642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/3438493203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":3438493203},{"type":"v1","addr":"172.21.15.174:6833","nonce":3438493203}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/3105752678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":3105752678},{"type":"v1","addr":"172.21.15.111:6833","nonce":3105752678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24461,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/2704462092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2704462092},{"type":"v1","addr":"172.21.15.111:6835","nonce":2704462092}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/3905733045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3905733045},{"type":"v1","addr":"172.21.15.174:6835","nonce":3905733045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24484,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/465579746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":465579746},{"type":"v1","addr":"172.21.15.174:6837","nonce":465579746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24494,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6837/2703764182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":2703764182},{"type":"v1","addr":"172.21.15.111:6837","nonce":2703764182}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-04-21T23:15:40.812 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 9 2024-04-21T23:15:40.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:40.814+0000 7fe5b4cd4640 1 -- 172.21.15.40:0/2825922411 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fe59c0a2b90 msgr2=0x7fe59c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:40.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:40.814+0000 7fe5b4cd4640 1 --2- 172.21.15.40:0/2825922411 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fe59c0a2b90 0x7fe59c0a5050 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe5a0000c00 tx=0x7fe5a0005fd0 comp rx=0 tx=0).stop 2024-04-21T23:15:40.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:40.814+0000 7fe5b4cd4640 1 -- 172.21.15.40:0/2825922411 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe5b0102230 msgr2=0x7fe5b0103190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:40.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:40.814+0000 7fe5b4cd4640 1 --2- 172.21.15.40:0/2825922411 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe5b0102230 0x7fe5b0103190 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fe5a800eaa0 tx=0x7fe5a800ef60 comp rx=0 tx=0).stop 2024-04-21T23:15:40.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:40.814+0000 7fe5b4cd4640 1 -- 172.21.15.40:0/2825922411 shutdown_connections 2024-04-21T23:15:40.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:40.814+0000 7fe5b4cd4640 1 --2- 172.21.15.40:0/2825922411 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fe59c0a2b90 0x7fe59c0a5050 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:40.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:40.814+0000 7fe5b4cd4640 1 --2- 172.21.15.40:0/2825922411 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5b0101870 0x7fe5b0102c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:40.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:40.814+0000 7fe5b4cd4640 1 --2- 172.21.15.40:0/2825922411 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe5b0102230 0x7fe5b0103190 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:40.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:40.814+0000 7fe5b4cd4640 1 --2- 172.21.15.40:0/2825922411 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe5b0100680 0x7fe5b0104610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:40.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:40.814+0000 7fe5b4cd4640 1 -- 172.21.15.40:0/2825922411 >> 172.21.15.40:0/2825922411 conn(0x7fe5b00fbbf0 msgr2=0x7fe5b00fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:40.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:40.814+0000 7fe5b4cd4640 1 -- 172.21.15.40:0/2825922411 shutdown_connections 2024-04-21T23:15:40.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:40.814+0000 7fe5b4cd4640 1 -- 172.21.15.40:0/2825922411 wait complete. 2024-04-21T23:15:41.194 INFO:journalctl@ceph.mds.i.smithi174.stdout:Apr 21 23:15:41 smithi174 systemd[1]: Starting Ceph mds.i for d814e4dc-0033-11ef-bc93-c7b262605968... 2024-04-21T23:15:41.498 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:41 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/2825922411' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:15:41.498 INFO:journalctl@ceph.mds.i.smithi174.stdout:Apr 21 23:15:41 smithi174 podman[62096]: 2024-04-21 23:15:41.152016871 +0000 UTC m=+0.018032939 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-21T23:15:41.498 INFO:journalctl@ceph.mds.i.smithi174.stdout:Apr 21 23:15:41 smithi174 podman[62096]: 2024-04-21 23:15:41.272180844 +0000 UTC m=+0.138196911 container create 27bf130082a01f74839f7b092e89104f9ecd21aa215be6f0c5c2d0b38eccd22f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-i, RELEASE=squid-820a2f8, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-21T23:15:41.499 INFO:journalctl@ceph.mds.i.smithi174.stdout:Apr 21 23:15:41 smithi174 podman[62096]: 2024-04-21 23:15:41.499985185 +0000 UTC m=+0.366001252 container init 27bf130082a01f74839f7b092e89104f9ecd21aa215be6f0c5c2d0b38eccd22f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-i, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=squid-820a2f8, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T23:15:41.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:41 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/2825922411' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:15:41.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:41 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/2825922411' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:15:41.944 INFO:journalctl@ceph.mds.i.smithi174.stdout:Apr 21 23:15:41 smithi174 podman[62096]: 2024-04-21 23:15:41.502599796 +0000 UTC m=+0.368615865 container start 27bf130082a01f74839f7b092e89104f9ecd21aa215be6f0c5c2d0b38eccd22f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-i, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-820a2f8, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T23:15:41.944 INFO:journalctl@ceph.mds.i.smithi174.stdout:Apr 21 23:15:41 smithi174 ceph-mds[62115]: -- 172.21.15.174:0/4271756623 <== mon.1 v2:172.21.15.174:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557c3c107a40 con 0x557c3c0da800 2024-04-21T23:15:41.944 INFO:journalctl@ceph.mds.i.smithi174.stdout:Apr 21 23:15:41 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-i[62111]: starting mds.i at 2024-04-21T23:15:41.944 INFO:journalctl@ceph.mds.i.smithi174.stdout:Apr 21 23:15:41 smithi174 bash[62096]: 27bf130082a01f74839f7b092e89104f9ecd21aa215be6f0c5c2d0b38eccd22f 2024-04-21T23:15:41.944 INFO:journalctl@ceph.mds.i.smithi174.stdout:Apr 21 23:15:41 smithi174 systemd[1]: Started Ceph mds.i for d814e4dc-0033-11ef-bc93-c7b262605968. 2024-04-21T23:15:42.569 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:42 smithi040 ceph-mon[27984]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:42.569 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:42 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:42.570 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:42 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:42.570 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:42 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:42.570 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:42 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:15:42.570 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:42 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:15:42.570 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:42 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:15:42.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:42 smithi174 ceph-mon[31409]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:42.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:42 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:42.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:42 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:42.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:42 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:42.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:42 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:15:42.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:42 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:15:42.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:42 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:15:42.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:42 smithi111 ceph-mon[31085]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:42.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:42 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:42.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:42 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:42.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:42 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:42.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:42 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:15:42.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:42 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:15:42.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:42 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:15:42.825 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:15:42.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:42.933+0000 7fc201a6d640 1 -- 172.21.15.40:0/1497952242 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1fc1017b0 msgr2=0x7fc1fc101bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:42.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:42.933+0000 7fc201a6d640 1 --2- 172.21.15.40:0/1497952242 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1fc1017b0 0x7fc1fc101bd0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fc1ec009900 tx=0x7fc1ec02bc30 comp rx=0 tx=0).stop 2024-04-21T23:15:42.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:42.933+0000 7fc201a6d640 1 -- 172.21.15.40:0/1497952242 shutdown_connections 2024-04-21T23:15:42.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:42.933+0000 7fc201a6d640 1 --2- 172.21.15.40:0/1497952242 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1fc103360 0x7fc1fc109bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:42.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:42.933+0000 7fc201a6d640 1 --2- 172.21.15.40:0/1497952242 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1fc1029a0 0x7fc1fc102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:42.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:42.933+0000 7fc201a6d640 1 --2- 172.21.15.40:0/1497952242 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1fc1017b0 0x7fc1fc101bd0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:42.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:42.933+0000 7fc201a6d640 1 -- 172.21.15.40:0/1497952242 >> 172.21.15.40:0/1497952242 conn(0x7fc1fc0fcd20 msgr2=0x7fc1fc0ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:42.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:42.933+0000 7fc201a6d640 1 -- 172.21.15.40:0/1497952242 shutdown_connections 2024-04-21T23:15:42.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:42.933+0000 7fc201a6d640 1 -- 172.21.15.40:0/1497952242 wait complete. 2024-04-21T23:15:42.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:42.934+0000 7fc201a6d640 1 Processor -- start 2024-04-21T23:15:42.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:42.934+0000 7fc201a6d640 1 -- start start 2024-04-21T23:15:42.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:42.934+0000 7fc201a6d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1fc1017b0 0x7fc1fc197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:42.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:42.934+0000 7fc201a6d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1fc1029a0 0x7fc1fc198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:42.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:42.934+0000 7fc201a6d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1fc103360 0x7fc1fc19f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:42.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:42.934+0000 7fc201a6d640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc1fc10d470 con 0x7fc1fc1029a0 2024-04-21T23:15:42.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:42.934+0000 7fc201a6d640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc1fc10d2f0 con 0x7fc1fc1017b0 2024-04-21T23:15:42.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:42.934+0000 7fc201a6d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc1fc10d5f0 con 0x7fc1fc103360 2024-04-21T23:15:42.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:42.934+0000 7fc1faffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1fc1029a0 0x7fc1fc198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:42.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:42.934+0000 7fc1faffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1fc1029a0 0x7fc1fc198060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48758/0 (socket says 172.21.15.40:48758) 2024-04-21T23:15:42.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:42.934+0000 7fc1faffd640 1 -- 172.21.15.40:0/1489445408 learned_addr learned my addr 172.21.15.40:0/1489445408 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:15:42.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:42.934+0000 7fc1fbfff640 1 --2- 172.21.15.40:0/1489445408 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1fc103360 0x7fc1fc19f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:42.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:42.934+0000 7fc1fb7fe640 1 --2- 172.21.15.40:0/1489445408 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1fc1017b0 0x7fc1fc197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:42.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:42.935+0000 7fc1faffd640 1 -- 172.21.15.40:0/1489445408 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1fc103360 msgr2=0x7fc1fc19f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:42.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:42.935+0000 7fc1faffd640 1 --2- 172.21.15.40:0/1489445408 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1fc103360 0x7fc1fc19f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:42.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:42.935+0000 7fc1faffd640 1 -- 172.21.15.40:0/1489445408 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1fc1017b0 msgr2=0x7fc1fc197b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:42.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:42.935+0000 7fc1faffd640 1 --2- 172.21.15.40:0/1489445408 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1fc1017b0 0x7fc1fc197b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:42.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:42.935+0000 7fc1faffd640 1 -- 172.21.15.40:0/1489445408 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1fc19f740 con 0x7fc1fc1029a0 2024-04-21T23:15:42.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:42.935+0000 7fc1fb7fe640 1 --2- 172.21.15.40:0/1489445408 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1fc1017b0 0x7fc1fc197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:15:42.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:42.935+0000 7fc1faffd640 1 --2- 172.21.15.40:0/1489445408 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1fc1029a0 0x7fc1fc198060 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fc1f000d770 tx=0x7fc1f000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:42.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:42.935+0000 7fc1f8ff9640 1 -- 172.21.15.40:0/1489445408 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1f0017070 con 0x7fc1fc1029a0 2024-04-21T23:15:42.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:42.935+0000 7fc201a6d640 1 -- 172.21.15.40:0/1489445408 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1fc19fa30 con 0x7fc1fc1029a0 2024-04-21T23:15:42.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:42.935+0000 7fc1f8ff9640 1 -- 172.21.15.40:0/1489445408 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1f0005500 con 0x7fc1fc1029a0 2024-04-21T23:15:42.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:42.935+0000 7fc1f8ff9640 1 -- 172.21.15.40:0/1489445408 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1f0004e00 con 0x7fc1fc1029a0 2024-04-21T23:15:42.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:42.935+0000 7fc201a6d640 1 -- 172.21.15.40:0/1489445408 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1fc19ff30 con 0x7fc1fc1029a0 2024-04-21T23:15:42.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:42.937+0000 7fc201a6d640 1 -- 172.21.15.40:0/1489445408 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1fc108b50 con 0x7fc1fc1029a0 2024-04-21T23:15:42.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:42.938+0000 7fc1f8ff9640 1 -- 172.21.15.40:0/1489445408 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fc1f0004fa0 con 0x7fc1fc1029a0 2024-04-21T23:15:42.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:42.939+0000 7fc1f8ff9640 1 --2- 172.21.15.40:0/1489445408 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fc1e40a2b90 0x7fc1e40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:42.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:42.939+0000 7fc1fb7fe640 1 --2- 172.21.15.40:0/1489445408 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fc1e40a2b90 0x7fc1e40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:42.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:42.939+0000 7fc1f8ff9640 1 -- 172.21.15.40:0/1489445408 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fc1f00ce5d0 con 0x7fc1fc1029a0 2024-04-21T23:15:42.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:42.940+0000 7fc1fb7fe640 1 --2- 172.21.15.40:0/1489445408 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fc1e40a2b90 0x7fc1e40a5050 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fc1ec02c170 tx=0x7fc1ec02df90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:42.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:42.944+0000 7fc1f8ff9640 1 -- 172.21.15.40:0/1489445408 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fc1f009b9d0 con 0x7fc1fc1029a0 2024-04-21T23:15:43.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:43.167+0000 7fc201a6d640 1 -- 172.21.15.40:0/1489445408 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc1fc02d370 con 0x7fc1fc1029a0 2024-04-21T23:15:43.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:43.167+0000 7fc1f8ff9640 1 -- 172.21.15.40:0/1489445408 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8010 (secure 0 0 0) 0x7fc1f009b2d0 con 0x7fc1fc1029a0 2024-04-21T23:15:43.167 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:15:43.167 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/1571948830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1571948830},{"type":"v1","addr":"172.21.15.40:6835","nonce":1571948830}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/969356642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":969356642},{"type":"v1","addr":"172.21.15.40:6837","nonce":969356642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/3438493203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":3438493203},{"type":"v1","addr":"172.21.15.174:6833","nonce":3438493203}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/3105752678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":3105752678},{"type":"v1","addr":"172.21.15.111:6833","nonce":3105752678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24461,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/2704462092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2704462092},{"type":"v1","addr":"172.21.15.111:6835","nonce":2704462092}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/3905733045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3905733045},{"type":"v1","addr":"172.21.15.174:6835","nonce":3905733045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24484,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/465579746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":465579746},{"type":"v1","addr":"172.21.15.174:6837","nonce":465579746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24494,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6837/2703764182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":2703764182},{"type":"v1","addr":"172.21.15.111:6837","nonce":2703764182}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24502,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6839/2467509274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":2467509274},{"type":"v1","addr":"172.21.15.174:6839","nonce":2467509274}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-21T23:15:43.167 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 10 2024-04-21T23:15:43.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:43.169+0000 7fc201a6d640 1 -- 172.21.15.40:0/1489445408 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fc1e40a2b90 msgr2=0x7fc1e40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:43.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:43.169+0000 7fc201a6d640 1 --2- 172.21.15.40:0/1489445408 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fc1e40a2b90 0x7fc1e40a5050 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fc1ec02c170 tx=0x7fc1ec02df90 comp rx=0 tx=0).stop 2024-04-21T23:15:43.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:43.169+0000 7fc201a6d640 1 -- 172.21.15.40:0/1489445408 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1fc1029a0 msgr2=0x7fc1fc198060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:43.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:43.169+0000 7fc201a6d640 1 --2- 172.21.15.40:0/1489445408 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1fc1029a0 0x7fc1fc198060 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fc1f000d770 tx=0x7fc1f000dc30 comp rx=0 tx=0).stop 2024-04-21T23:15:43.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:43.169+0000 7fc201a6d640 1 -- 172.21.15.40:0/1489445408 shutdown_connections 2024-04-21T23:15:43.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:43.169+0000 7fc201a6d640 1 --2- 172.21.15.40:0/1489445408 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fc1e40a2b90 0x7fc1e40a5050 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:43.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:43.169+0000 7fc201a6d640 1 --2- 172.21.15.40:0/1489445408 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1fc103360 0x7fc1fc19f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:43.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:43.169+0000 7fc201a6d640 1 --2- 172.21.15.40:0/1489445408 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1fc1029a0 0x7fc1fc198060 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:43.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:43.169+0000 7fc201a6d640 1 --2- 172.21.15.40:0/1489445408 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1fc1017b0 0x7fc1fc197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:43.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:43.169+0000 7fc201a6d640 1 -- 172.21.15.40:0/1489445408 >> 172.21.15.40:0/1489445408 conn(0x7fc1fc0fcd20 msgr2=0x7fc1fc105d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:43.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:43.170+0000 7fc201a6d640 1 -- 172.21.15.40:0/1489445408 shutdown_connections 2024-04-21T23:15:43.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:43.170+0000 7fc201a6d640 1 -- 172.21.15.40:0/1489445408 wait complete. 2024-04-21T23:15:43.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:43 smithi040 ceph-mon[27984]: Deploying daemon mds.g on smithi040 2024-04-21T23:15:43.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:43 smithi040 ceph-mon[27984]: mds.? [v2:172.21.15.174:6838/2467509274,v1:172.21.15.174:6839/2467509274] up:boot 2024-04-21T23:15:43.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:43 smithi040 ceph-mon[27984]: fsmap 9 up:standby 2024-04-21T23:15:43.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:43 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-21T23:15:43.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:43 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/1489445408' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:15:43.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:43 smithi174 ceph-mon[31409]: Deploying daemon mds.g on smithi040 2024-04-21T23:15:43.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:43 smithi174 ceph-mon[31409]: mds.? [v2:172.21.15.174:6838/2467509274,v1:172.21.15.174:6839/2467509274] up:boot 2024-04-21T23:15:43.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:43 smithi174 ceph-mon[31409]: fsmap 9 up:standby 2024-04-21T23:15:43.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:43 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-21T23:15:43.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:43 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/1489445408' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:15:43.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:43 smithi111 ceph-mon[31085]: Deploying daemon mds.g on smithi040 2024-04-21T23:15:43.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:43 smithi111 ceph-mon[31085]: mds.? [v2:172.21.15.174:6838/2467509274,v1:172.21.15.174:6839/2467509274] up:boot 2024-04-21T23:15:43.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:43 smithi111 ceph-mon[31085]: fsmap 9 up:standby 2024-04-21T23:15:43.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:43 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-21T23:15:43.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:43 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/1489445408' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:15:44.182 INFO:journalctl@ceph.mds.g.smithi040.stdout:Apr 21 23:15:44 smithi040 podman[67013]: 2024-04-21 23:15:44.104944211 +0000 UTC m=+0.123711160 container create 780ba2a634dc4f2403855e8edbca892dc0504172a27563315143969bb66c2357 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-g, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-820a2f8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-21T23:15:44.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:44 smithi040 ceph-mon[27984]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:44.682 INFO:journalctl@ceph.mds.g.smithi040.stdout:Apr 21 23:15:44 smithi040 podman[67013]: 2024-04-21 23:15:44.341289358 +0000 UTC m=+0.360056305 container init 780ba2a634dc4f2403855e8edbca892dc0504172a27563315143969bb66c2357 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-g, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, RELEASE=squid-820a2f8, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-21T23:15:44.682 INFO:journalctl@ceph.mds.g.smithi040.stdout:Apr 21 23:15:44 smithi040 podman[67013]: 2024-04-21 23:15:44.344137803 +0000 UTC m=+0.362904752 container start 780ba2a634dc4f2403855e8edbca892dc0504172a27563315143969bb66c2357 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-g, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, ceph=True, RELEASE=squid-820a2f8, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T23:15:44.682 INFO:journalctl@ceph.mds.g.smithi040.stdout:Apr 21 23:15:44 smithi040 ceph-mds[67032]: -- 172.21.15.40:0/1630848103 <== mon.0 v2:172.21.15.40:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ce94767a40 con 0x55ce9473a800 2024-04-21T23:15:44.682 INFO:journalctl@ceph.mds.g.smithi040.stdout:Apr 21 23:15:44 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-g[67028]: starting mds.g at 2024-04-21T23:15:44.682 INFO:journalctl@ceph.mds.g.smithi040.stdout:Apr 21 23:15:44 smithi040 bash[67013]: 780ba2a634dc4f2403855e8edbca892dc0504172a27563315143969bb66c2357 2024-04-21T23:15:44.682 INFO:journalctl@ceph.mds.g.smithi040.stdout:Apr 21 23:15:44 smithi040 systemd[1]: Started Ceph mds.g for d814e4dc-0033-11ef-bc93-c7b262605968. 2024-04-21T23:15:44.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:44 smithi174 ceph-mon[31409]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:44.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:44 smithi111 ceph-mon[31085]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:45.180 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:15:45.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:45.289+0000 7f1de67a1640 1 -- 172.21.15.40:0/135430132 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1de0103360 msgr2=0x7f1de0109bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:45.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:45.289+0000 7f1de67a1640 1 --2- 172.21.15.40:0/135430132 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1de0103360 0x7f1de0109bf0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f1dd8009ca0 tx=0x7f1dd802c0f0 comp rx=0 tx=0).stop 2024-04-21T23:15:45.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:45.289+0000 7f1de67a1640 1 -- 172.21.15.40:0/135430132 shutdown_connections 2024-04-21T23:15:45.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:45.289+0000 7f1de67a1640 1 --2- 172.21.15.40:0/135430132 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1de0103360 0x7f1de0109bf0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:45.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:45.289+0000 7f1de67a1640 1 --2- 172.21.15.40:0/135430132 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1de01029a0 0x7f1de0102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:45.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:45.289+0000 7f1de67a1640 1 --2- 172.21.15.40:0/135430132 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1de01017b0 0x7f1de0101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:45.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:45.289+0000 7f1de67a1640 1 -- 172.21.15.40:0/135430132 >> 172.21.15.40:0/135430132 conn(0x7f1de00fcd20 msgr2=0x7f1de00ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:45.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:45.289+0000 7f1de67a1640 1 -- 172.21.15.40:0/135430132 shutdown_connections 2024-04-21T23:15:45.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:45.289+0000 7f1de67a1640 1 -- 172.21.15.40:0/135430132 wait complete. 2024-04-21T23:15:45.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:45.290+0000 7f1de67a1640 1 Processor -- start 2024-04-21T23:15:45.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:45.290+0000 7f1de67a1640 1 -- start start 2024-04-21T23:15:45.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:45.290+0000 7f1de67a1640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1de01017b0 0x7f1de0199c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:45.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:45.290+0000 7f1de67a1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1de01029a0 0x7f1de019a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:45.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:45.290+0000 7f1de67a1640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1de0103360 0x7f1de01a1230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:45.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:45.290+0000 7f1de67a1640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1de010d430 con 0x7f1de01017b0 2024-04-21T23:15:45.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:45.290+0000 7f1de67a1640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f1de010d2b0 con 0x7f1de0103360 2024-04-21T23:15:45.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:45.290+0000 7f1de67a1640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1de010d5b0 con 0x7f1de01029a0 2024-04-21T23:15:45.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:45.290+0000 7f1ddffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1de01017b0 0x7f1de0199c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:45.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:45.290+0000 7f1ddffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1de01017b0 0x7f1de0199c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48798/0 (socket says 172.21.15.40:48798) 2024-04-21T23:15:45.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:45.290+0000 7f1ddffff640 1 -- 172.21.15.40:0/4227799720 learned_addr learned my addr 172.21.15.40:0/4227799720 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:15:45.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:45.290+0000 7f1ddf7fe640 1 --2- 172.21.15.40:0/4227799720 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1de01029a0 0x7f1de019a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:45.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:45.290+0000 7f1de4d3e640 1 --2- 172.21.15.40:0/4227799720 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1de0103360 0x7f1de01a1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:45.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:45.291+0000 7f1ddffff640 1 -- 172.21.15.40:0/4227799720 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1de01029a0 msgr2=0x7f1de019a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:45.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:45.291+0000 7f1ddffff640 1 --2- 172.21.15.40:0/4227799720 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1de01029a0 0x7f1de019a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:45.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:45.291+0000 7f1ddffff640 1 -- 172.21.15.40:0/4227799720 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1de0103360 msgr2=0x7f1de01a1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:45.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:45.291+0000 7f1ddffff640 1 --2- 172.21.15.40:0/4227799720 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1de0103360 0x7f1de01a1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:45.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:45.291+0000 7f1ddffff640 1 -- 172.21.15.40:0/4227799720 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1de01a18d0 con 0x7f1de01017b0 2024-04-21T23:15:45.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:45.291+0000 7f1de4d3e640 1 --2- 172.21.15.40:0/4227799720 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1de0103360 0x7f1de01a1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:15:45.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:45.291+0000 7f1ddffff640 1 --2- 172.21.15.40:0/4227799720 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1de01017b0 0x7f1de0199c90 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f1dd000c970 tx=0x7f1dd000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:45.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:45.291+0000 7f1ddd7fa640 1 -- 172.21.15.40:0/4227799720 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dd0004220 con 0x7f1de01017b0 2024-04-21T23:15:45.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:45.291+0000 7f1ddd7fa640 1 -- 172.21.15.40:0/4227799720 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1dd00043c0 con 0x7f1de01017b0 2024-04-21T23:15:45.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:45.291+0000 7f1de67a1640 1 -- 172.21.15.40:0/4227799720 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1de01a1bc0 con 0x7f1de01017b0 2024-04-21T23:15:45.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:45.291+0000 7f1ddd7fa640 1 -- 172.21.15.40:0/4227799720 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dd000f450 con 0x7f1de01017b0 2024-04-21T23:15:45.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:45.291+0000 7f1de67a1640 1 -- 172.21.15.40:0/4227799720 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1de01a2110 con 0x7f1de01017b0 2024-04-21T23:15:45.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:45.292+0000 7f1de67a1640 1 -- 172.21.15.40:0/4227799720 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1de01a22e0 con 0x7f1de01017b0 2024-04-21T23:15:45.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:45.293+0000 7f1ddd7fa640 1 -- 172.21.15.40:0/4227799720 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f1dd00079b0 con 0x7f1de01017b0 2024-04-21T23:15:45.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:45.294+0000 7f1ddd7fa640 1 --2- 172.21.15.40:0/4227799720 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f1dc80a2b90 0x7f1dc80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:45.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:45.294+0000 7f1ddf7fe640 1 --2- 172.21.15.40:0/4227799720 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f1dc80a2b90 0x7f1dc80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:45.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:45.294+0000 7f1ddd7fa640 1 -- 172.21.15.40:0/4227799720 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f1dd00ced50 con 0x7f1de01017b0 2024-04-21T23:15:45.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:45.295+0000 7f1ddf7fe640 1 --2- 172.21.15.40:0/4227799720 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f1dc80a2b90 0x7f1dc80a5050 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f1de019b190 tx=0x7f1dd400b390 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:45.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:45.299+0000 7f1ddd7fa640 1 -- 172.21.15.40:0/4227799720 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f1dd0014030 con 0x7f1de01017b0 2024-04-21T23:15:45.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:45.528+0000 7f1de67a1640 1 -- 172.21.15.40:0/4227799720 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1de01a24b0 con 0x7f1de01017b0 2024-04-21T23:15:45.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:45.528+0000 7f1ddd7fa640 1 -- 172.21.15.40:0/4227799720 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8836 (secure 0 0 0) 0x7f1dd009c130 con 0x7f1de01017b0 2024-04-21T23:15:45.528 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:15:45.528 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/1571948830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1571948830},{"type":"v1","addr":"172.21.15.40:6835","nonce":1571948830}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/969356642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":969356642},{"type":"v1","addr":"172.21.15.40:6837","nonce":969356642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6839/3025057845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":3025057845},{"type":"v1","addr":"172.21.15.40:6839","nonce":3025057845}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/3438493203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":3438493203},{"type":"v1","addr":"172.21.15.174:6833","nonce":3438493203}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/3105752678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":3105752678},{"type":"v1","addr":"172.21.15.111:6833","nonce":3105752678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24461,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/2704462092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2704462092},{"type":"v1","addr":"172.21.15.111:6835","nonce":2704462092}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/3905733045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3905733045},{"type":"v1","addr":"172.21.15.174:6835","nonce":3905733045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24484,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/465579746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":465579746},{"type":"v1","addr":"172.21.15.174:6837","nonce":465579746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24494,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6837/2703764182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":2703764182},{"type":"v1","addr":"172.21.15.111:6837","nonce":2703764182}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24502,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6839/2467509274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":2467509274},{"type":"v1","addr":"172.21.15.174:6839","nonce":2467509274}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-21T23:15:45.528 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 11 2024-04-21T23:15:45.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:45.531+0000 7f1de67a1640 1 -- 172.21.15.40:0/4227799720 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f1dc80a2b90 msgr2=0x7f1dc80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:45.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:45.531+0000 7f1de67a1640 1 --2- 172.21.15.40:0/4227799720 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f1dc80a2b90 0x7f1dc80a5050 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f1de019b190 tx=0x7f1dd400b390 comp rx=0 tx=0).stop 2024-04-21T23:15:45.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:45.531+0000 7f1de67a1640 1 -- 172.21.15.40:0/4227799720 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1de01017b0 msgr2=0x7f1de0199c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:45.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:45.531+0000 7f1de67a1640 1 --2- 172.21.15.40:0/4227799720 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1de01017b0 0x7f1de0199c90 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f1dd000c970 tx=0x7f1dd000ce30 comp rx=0 tx=0).stop 2024-04-21T23:15:45.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:45.531+0000 7f1de67a1640 1 -- 172.21.15.40:0/4227799720 shutdown_connections 2024-04-21T23:15:45.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:45.531+0000 7f1de67a1640 1 --2- 172.21.15.40:0/4227799720 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f1dc80a2b90 0x7f1dc80a5050 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:45.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:45.531+0000 7f1de67a1640 1 --2- 172.21.15.40:0/4227799720 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1de0103360 0x7f1de01a1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:45.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:45.531+0000 7f1de67a1640 1 --2- 172.21.15.40:0/4227799720 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1de01029a0 0x7f1de019a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:45.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:45.531+0000 7f1de67a1640 1 --2- 172.21.15.40:0/4227799720 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1de01017b0 0x7f1de0199c90 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:45.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:45.531+0000 7f1de67a1640 1 -- 172.21.15.40:0/4227799720 >> 172.21.15.40:0/4227799720 conn(0x7f1de00fcd20 msgr2=0x7f1de0073970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:45.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:45.531+0000 7f1de67a1640 1 -- 172.21.15.40:0/4227799720 shutdown_connections 2024-04-21T23:15:45.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:45.531+0000 7f1de67a1640 1 -- 172.21.15.40:0/4227799720 wait complete. 2024-04-21T23:15:45.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:45 smithi040 ceph-mon[27984]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:45.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:45 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:45.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:45 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:45.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:45 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:45.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:45 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:15:45.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:45 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:15:45.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:45 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:15:45.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:45 smithi040 ceph-mon[27984]: Deploying daemon mds.a on smithi040 2024-04-21T23:15:45.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:45 smithi040 ceph-mon[27984]: mds.? [v2:172.21.15.40:6838/3025057845,v1:172.21.15.40:6839/3025057845] up:boot 2024-04-21T23:15:45.683 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:45 smithi040 ceph-mon[27984]: fsmap 10 up:standby 2024-04-21T23:15:45.683 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:45 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-21T23:15:45.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:45 smithi174 ceph-mon[31409]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:45.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:45 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:45.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:45 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:45.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:45 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:45.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:45 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:15:45.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:45 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:15:45.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:45 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:15:45.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:45 smithi174 ceph-mon[31409]: Deploying daemon mds.a on smithi040 2024-04-21T23:15:45.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:45 smithi174 ceph-mon[31409]: mds.? [v2:172.21.15.40:6838/3025057845,v1:172.21.15.40:6839/3025057845] up:boot 2024-04-21T23:15:45.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:45 smithi174 ceph-mon[31409]: fsmap 10 up:standby 2024-04-21T23:15:45.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:45 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-21T23:15:45.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:45 smithi111 ceph-mon[31085]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:45.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:45 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:45.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:45 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:45.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:45 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:45.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:45 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:15:45.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:45 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:15:45.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:45 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:15:45.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:45 smithi111 ceph-mon[31085]: Deploying daemon mds.a on smithi040 2024-04-21T23:15:45.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:45 smithi111 ceph-mon[31085]: mds.? [v2:172.21.15.40:6838/3025057845,v1:172.21.15.40:6839/3025057845] up:boot 2024-04-21T23:15:45.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:45 smithi111 ceph-mon[31085]: fsmap 10 up:standby 2024-04-21T23:15:45.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:45 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-21T23:15:46.905 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:46 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/4227799720' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:15:46.905 INFO:journalctl@ceph.mds.a.smithi040.stdout:Apr 21 23:15:46 smithi040 systemd[1]: Starting Ceph mds.a for d814e4dc-0033-11ef-bc93-c7b262605968... 2024-04-21T23:15:46.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:46 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/4227799720' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:15:46.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:46 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/4227799720' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:15:47.172 INFO:journalctl@ceph.mds.a.smithi040.stdout:Apr 21 23:15:46 smithi040 podman[67362]: 2024-04-21 23:15:46.807522662 +0000 UTC m=+0.018302628 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-21T23:15:47.172 INFO:journalctl@ceph.mds.a.smithi040.stdout:Apr 21 23:15:46 smithi040 podman[67362]: 2024-04-21 23:15:46.911378218 +0000 UTC m=+0.122158206 container create dfb13ba672d111681c92978fa4b27449e3f65cca3556499c23dbfae6d18a5c41 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-a, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=squid-820a2f8, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0) 2024-04-21T23:15:47.172 INFO:journalctl@ceph.mds.a.smithi040.stdout:Apr 21 23:15:47 smithi040 podman[67362]: 2024-04-21 23:15:47.147381312 +0000 UTC m=+0.358161261 container init dfb13ba672d111681c92978fa4b27449e3f65cca3556499c23dbfae6d18a5c41 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-a, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-820a2f8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-21T23:15:47.173 INFO:journalctl@ceph.mds.a.smithi040.stdout:Apr 21 23:15:47 smithi040 podman[67362]: 2024-04-21 23:15:47.150039779 +0000 UTC m=+0.360819725 container start dfb13ba672d111681c92978fa4b27449e3f65cca3556499c23dbfae6d18a5c41 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-a, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-820a2f8, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.vendor=CentOS) 2024-04-21T23:15:47.432 INFO:journalctl@ceph.mds.a.smithi040.stdout:Apr 21 23:15:47 smithi040 ceph-mds[67380]: -- 172.21.15.40:0/1200035580 <== mon.0 v2:172.21.15.40:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e189c8ba40 con 0x55e189c5f000 2024-04-21T23:15:47.432 INFO:journalctl@ceph.mds.a.smithi040.stdout:Apr 21 23:15:47 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-a[67376]: starting mds.a at 2024-04-21T23:15:47.432 INFO:journalctl@ceph.mds.a.smithi040.stdout:Apr 21 23:15:47 smithi040 bash[67362]: dfb13ba672d111681c92978fa4b27449e3f65cca3556499c23dbfae6d18a5c41 2024-04-21T23:15:47.432 INFO:journalctl@ceph.mds.a.smithi040.stdout:Apr 21 23:15:47 smithi040 systemd[1]: Started Ceph mds.a for d814e4dc-0033-11ef-bc93-c7b262605968. 2024-04-21T23:15:47.542 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:15:47.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:47.649+0000 7fd6f8ae6640 1 -- 172.21.15.40:0/2828334853 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6f41006a0 msgr2=0x7fd6f4100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:47.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:47.649+0000 7fd6f8ae6640 1 --2- 172.21.15.40:0/2828334853 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6f41006a0 0x7fd6f4100ac0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc009900 tx=0x7fd6dc02bc30 comp rx=0 tx=0).stop 2024-04-21T23:15:47.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:47.649+0000 7fd6f8ae6640 1 -- 172.21.15.40:0/2828334853 shutdown_connections 2024-04-21T23:15:47.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:47.649+0000 7fd6f8ae6640 1 --2- 172.21.15.40:0/2828334853 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6f4101890 0x7fd6f4101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:47.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:47.649+0000 7fd6f8ae6640 1 --2- 172.21.15.40:0/2828334853 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd6f4102250 0x7fd6f4108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:47.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:47.649+0000 7fd6f8ae6640 1 --2- 172.21.15.40:0/2828334853 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6f41006a0 0x7fd6f4100ac0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:47.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:47.649+0000 7fd6f8ae6640 1 -- 172.21.15.40:0/2828334853 >> 172.21.15.40:0/2828334853 conn(0x7fd6f40fbc30 msgr2=0x7fd6f40fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:47.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:47.649+0000 7fd6f8ae6640 1 -- 172.21.15.40:0/2828334853 shutdown_connections 2024-04-21T23:15:47.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:47.649+0000 7fd6f8ae6640 1 -- 172.21.15.40:0/2828334853 wait complete. 2024-04-21T23:15:47.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:47.650+0000 7fd6f8ae6640 1 Processor -- start 2024-04-21T23:15:47.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:47.650+0000 7fd6f8ae6640 1 -- start start 2024-04-21T23:15:47.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:47.650+0000 7fd6f8ae6640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6f41006a0 0x7fd6f4197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:47.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:47.650+0000 7fd6f8ae6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6f4101890 0x7fd6f4197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:47.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:47.650+0000 7fd6f8ae6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd6f4102250 0x7fd6f419f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:47.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:47.650+0000 7fd6f8ae6640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd6f410c320 con 0x7fd6f4101890 2024-04-21T23:15:47.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:47.650+0000 7fd6f8ae6640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd6f410c1a0 con 0x7fd6f41006a0 2024-04-21T23:15:47.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:47.650+0000 7fd6f8ae6640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd6f410c4a0 con 0x7fd6f4102250 2024-04-21T23:15:47.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:47.651+0000 7fd6f1d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6f4101890 0x7fd6f4197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:47.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:47.651+0000 7fd6f1d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6f4101890 0x7fd6f4197fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48824/0 (socket says 172.21.15.40:48824) 2024-04-21T23:15:47.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:47.651+0000 7fd6f1d9b640 1 -- 172.21.15.40:0/1834134983 learned_addr learned my addr 172.21.15.40:0/1834134983 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:15:47.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:47.651+0000 7fd6f259c640 1 --2- 172.21.15.40:0/1834134983 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6f41006a0 0x7fd6f4197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:47.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:47.651+0000 7fd6f2d9d640 1 --2- 172.21.15.40:0/1834134983 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd6f4102250 0x7fd6f419f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:47.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:47.651+0000 7fd6f1d9b640 1 -- 172.21.15.40:0/1834134983 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd6f4102250 msgr2=0x7fd6f419f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:47.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:47.651+0000 7fd6f1d9b640 1 --2- 172.21.15.40:0/1834134983 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd6f4102250 0x7fd6f419f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:47.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:47.651+0000 7fd6f1d9b640 1 -- 172.21.15.40:0/1834134983 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6f41006a0 msgr2=0x7fd6f4197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:47.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:47.651+0000 7fd6f1d9b640 1 --2- 172.21.15.40:0/1834134983 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6f41006a0 0x7fd6f4197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:47.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:47.651+0000 7fd6f1d9b640 1 -- 172.21.15.40:0/1834134983 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6f419f6b0 con 0x7fd6f4101890 2024-04-21T23:15:47.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:47.651+0000 7fd6f2d9d640 1 --2- 172.21.15.40:0/1834134983 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd6f4102250 0x7fd6f419f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:15:47.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:47.651+0000 7fd6f1d9b640 1 --2- 172.21.15.40:0/1834134983 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6f4101890 0x7fd6f4197fd0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fd6e400e9f0 tx=0x7fd6e400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:47.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:47.651+0000 7fd6eb7fe640 1 -- 172.21.15.40:0/1834134983 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6e400cd80 con 0x7fd6f4101890 2024-04-21T23:15:47.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:47.651+0000 7fd6eb7fe640 1 -- 172.21.15.40:0/1834134983 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6e4010910 con 0x7fd6f4101890 2024-04-21T23:15:47.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:47.651+0000 7fd6eb7fe640 1 -- 172.21.15.40:0/1834134983 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6e4018990 con 0x7fd6f4101890 2024-04-21T23:15:47.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:47.651+0000 7fd6f8ae6640 1 -- 172.21.15.40:0/1834134983 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6f419f9a0 con 0x7fd6f4101890 2024-04-21T23:15:47.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:47.651+0000 7fd6f8ae6640 1 -- 172.21.15.40:0/1834134983 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6f419ff40 con 0x7fd6f4101890 2024-04-21T23:15:47.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:47.652+0000 7fd6f8ae6640 1 -- 172.21.15.40:0/1834134983 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6f41a0170 con 0x7fd6f4101890 2024-04-21T23:15:47.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:47.653+0000 7fd6eb7fe640 1 -- 172.21.15.40:0/1834134983 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fd6e4010ac0 con 0x7fd6f4101890 2024-04-21T23:15:47.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:47.654+0000 7fd6eb7fe640 1 --2- 172.21.15.40:0/1834134983 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fd6e00a2c10 0x7fd6e00a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:47.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:47.654+0000 7fd6f259c640 1 --2- 172.21.15.40:0/1834134983 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fd6e00a2c10 0x7fd6e00a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:47.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:47.654+0000 7fd6eb7fe640 1 -- 172.21.15.40:0/1834134983 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fd6e40cf2b0 con 0x7fd6f4101890 2024-04-21T23:15:47.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:47.654+0000 7fd6f259c640 1 --2- 172.21.15.40:0/1834134983 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fd6e00a2c10 0x7fd6e00a50d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc000c00 tx=0x7fd6dc005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:47.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:47.657+0000 7fd6eb7fe640 1 -- 172.21.15.40:0/1834134983 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fd6e409c6b0 con 0x7fd6f4101890 2024-04-21T23:15:47.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:47.881+0000 7fd6f8ae6640 1 -- 172.21.15.40:0/1834134983 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd6f41a0340 con 0x7fd6f4101890 2024-04-21T23:15:47.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:47.881+0000 7fd6eb7fe640 1 -- 172.21.15.40:0/1834134983 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9662 (secure 0 0 0) 0x7fd6e409bfb0 con 0x7fd6f4101890 2024-04-21T23:15:47.881 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:15:47.881 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/1571948830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1571948830},{"type":"v1","addr":"172.21.15.40:6835","nonce":1571948830}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/969356642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":969356642},{"type":"v1","addr":"172.21.15.40:6837","nonce":969356642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6839/3025057845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":3025057845},{"type":"v1","addr":"172.21.15.40:6839","nonce":3025057845}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6841/3350488804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":3350488804},{"type":"v1","addr":"172.21.15.40:6841","nonce":3350488804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/3438493203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":3438493203},{"type":"v1","addr":"172.21.15.174:6833","nonce":3438493203}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/3105752678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":3105752678},{"type":"v1","addr":"172.21.15.111:6833","nonce":3105752678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24461,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/2704462092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2704462092},{"type":"v1","addr":"172.21.15.111:6835","nonce":2704462092}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/3905733045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3905733045},{"type":"v1","addr":"172.21.15.174:6835","nonce":3905733045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24484,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/465579746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":465579746},{"type":"v1","addr":"172.21.15.174:6837","nonce":465579746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24494,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6837/2703764182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":2703764182},{"type":"v1","addr":"172.21.15.111:6837","nonce":2703764182}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24502,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6839/2467509274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":2467509274},{"type":"v1","addr":"172.21.15.174:6839","nonce":2467509274}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-21T23:15:47.881 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 12 2024-04-21T23:15:47.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:47.883+0000 7fd6f8ae6640 1 -- 172.21.15.40:0/1834134983 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fd6e00a2c10 msgr2=0x7fd6e00a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:47.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:47.883+0000 7fd6f8ae6640 1 --2- 172.21.15.40:0/1834134983 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fd6e00a2c10 0x7fd6e00a50d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc000c00 tx=0x7fd6dc005fd0 comp rx=0 tx=0).stop 2024-04-21T23:15:47.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:47.883+0000 7fd6f8ae6640 1 -- 172.21.15.40:0/1834134983 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6f4101890 msgr2=0x7fd6f4197fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:47.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:47.883+0000 7fd6f8ae6640 1 --2- 172.21.15.40:0/1834134983 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6f4101890 0x7fd6f4197fd0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fd6e400e9f0 tx=0x7fd6e400eeb0 comp rx=0 tx=0).stop 2024-04-21T23:15:47.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:47.883+0000 7fd6f8ae6640 1 -- 172.21.15.40:0/1834134983 shutdown_connections 2024-04-21T23:15:47.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:47.883+0000 7fd6f8ae6640 1 --2- 172.21.15.40:0/1834134983 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fd6e00a2c10 0x7fd6e00a50d0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:47.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:47.883+0000 7fd6f8ae6640 1 --2- 172.21.15.40:0/1834134983 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd6f4102250 0x7fd6f419f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:47.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:47.884+0000 7fd6f8ae6640 1 --2- 172.21.15.40:0/1834134983 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6f4101890 0x7fd6f4197fd0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:47.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:47.884+0000 7fd6f8ae6640 1 --2- 172.21.15.40:0/1834134983 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6f41006a0 0x7fd6f4197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:47.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:47.884+0000 7fd6f8ae6640 1 -- 172.21.15.40:0/1834134983 >> 172.21.15.40:0/1834134983 conn(0x7fd6f40fbc30 msgr2=0x7fd6f40fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:47.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:47.884+0000 7fd6f8ae6640 1 -- 172.21.15.40:0/1834134983 shutdown_connections 2024-04-21T23:15:47.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:47.884+0000 7fd6f8ae6640 1 -- 172.21.15.40:0/1834134983 wait complete. 2024-04-21T23:15:47.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:47 smithi040 ceph-mon[27984]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:47.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:47 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:47.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:47 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:47.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:47 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:47.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:47 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:15:47.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:47 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:15:47.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:47 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:15:47.929 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:47 smithi111 ceph-mon[31085]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:47.930 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:47 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:47.930 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:47 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:47.930 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:47 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:47.930 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:47 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:15:47.930 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:47 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:15:47.930 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:47 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:15:47.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:47 smithi174 ceph-mon[31409]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:47.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:47 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:47.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:47 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:47.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:47 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:47.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:47 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:15:47.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:47 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:15:47.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:47 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:15:48.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:48 smithi040 ceph-mon[27984]: Deploying daemon mds.b on smithi111 2024-04-21T23:15:48.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:48 smithi040 ceph-mon[27984]: mds.? [v2:172.21.15.40:6840/3350488804,v1:172.21.15.40:6841/3350488804] up:boot 2024-04-21T23:15:48.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:48 smithi040 ceph-mon[27984]: fsmap 11 up:standby 2024-04-21T23:15:48.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:48 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-21T23:15:48.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:48 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/1834134983' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:15:48.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:48 smithi174 ceph-mon[31409]: Deploying daemon mds.b on smithi111 2024-04-21T23:15:48.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:48 smithi174 ceph-mon[31409]: mds.? [v2:172.21.15.40:6840/3350488804,v1:172.21.15.40:6841/3350488804] up:boot 2024-04-21T23:15:48.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:48 smithi174 ceph-mon[31409]: fsmap 11 up:standby 2024-04-21T23:15:48.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:48 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-21T23:15:48.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:48 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/1834134983' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:15:48.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:48 smithi111 ceph-mon[31085]: Deploying daemon mds.b on smithi111 2024-04-21T23:15:48.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:48 smithi111 ceph-mon[31085]: mds.? [v2:172.21.15.40:6840/3350488804,v1:172.21.15.40:6841/3350488804] up:boot 2024-04-21T23:15:48.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:48 smithi111 ceph-mon[31085]: fsmap 11 up:standby 2024-04-21T23:15:48.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:48 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-21T23:15:48.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:48 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/1834134983' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:15:49.689 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:49 smithi111 ceph-mon[31085]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:49.895 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:15:49.921 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:49 smithi040 ceph-mon[27984]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:49.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:49 smithi174 ceph-mon[31409]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:50.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:50.011+0000 7f34a44cd640 1 -- 172.21.15.40:0/3778327179 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f349c102200 msgr2=0x7f349c1045f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:50.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:50.011+0000 7f34a44cd640 1 --2- 172.21.15.40:0/3778327179 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f349c102200 0x7f349c1045f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f3498009900 tx=0x7f349802bc30 comp rx=0 tx=0).stop 2024-04-21T23:15:50.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:50.011+0000 7f34a44cd640 1 -- 172.21.15.40:0/3778327179 shutdown_connections 2024-04-21T23:15:50.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:50.011+0000 7f34a44cd640 1 --2- 172.21.15.40:0/3778327179 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f349c104b30 0x7f349c106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:50.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:50.011+0000 7f34a44cd640 1 --2- 172.21.15.40:0/3778327179 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f349c107460 0x7f349c1098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:50.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:50.011+0000 7f34a44cd640 1 --2- 172.21.15.40:0/3778327179 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f349c102200 0x7f349c1045f0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:50.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:50.011+0000 7f34a44cd640 1 -- 172.21.15.40:0/3778327179 >> 172.21.15.40:0/3778327179 conn(0x7f349c0fbbf0 msgr2=0x7f349c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:50.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:50.011+0000 7f34a44cd640 1 -- 172.21.15.40:0/3778327179 shutdown_connections 2024-04-21T23:15:50.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:50.011+0000 7f34a44cd640 1 -- 172.21.15.40:0/3778327179 wait complete. 2024-04-21T23:15:50.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:50.011+0000 7f34a44cd640 1 Processor -- start 2024-04-21T23:15:50.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:50.012+0000 7f34a44cd640 1 -- start start 2024-04-21T23:15:50.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:50.012+0000 7f34a44cd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f349c102200 0x7f349c197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:50.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:50.012+0000 7f34a44cd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f349c104b30 0x7f349c197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:50.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:50.012+0000 7f34a44cd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f349c107460 0x7f349c19f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:50.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:50.012+0000 7f34a44cd640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f349c10d400 con 0x7f349c102200 2024-04-21T23:15:50.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:50.012+0000 7f34a44cd640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f349c10d280 con 0x7f349c104b30 2024-04-21T23:15:50.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:50.012+0000 7f34a44cd640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f349c10d580 con 0x7f349c107460 2024-04-21T23:15:50.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:50.012+0000 7f34a2269640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f349c102200 0x7f349c197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:50.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:50.012+0000 7f34a2269640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f349c102200 0x7f349c197aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48846/0 (socket says 172.21.15.40:48846) 2024-04-21T23:15:50.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:50.012+0000 7f34a2269640 1 -- 172.21.15.40:0/1000241903 learned_addr learned my addr 172.21.15.40:0/1000241903 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:15:50.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:50.012+0000 7f34a2a6a640 1 --2- 172.21.15.40:0/1000241903 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f349c107460 0x7f349c19f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:50.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:50.012+0000 7f34a1a68640 1 --2- 172.21.15.40:0/1000241903 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f349c104b30 0x7f349c197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:50.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:50.012+0000 7f34a2269640 1 -- 172.21.15.40:0/1000241903 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f349c107460 msgr2=0x7f349c19f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:50.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:50.012+0000 7f34a2269640 1 --2- 172.21.15.40:0/1000241903 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f349c107460 0x7f349c19f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:50.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:50.012+0000 7f34a2269640 1 -- 172.21.15.40:0/1000241903 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f349c104b30 msgr2=0x7f349c197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:50.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:50.012+0000 7f34a2269640 1 --2- 172.21.15.40:0/1000241903 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f349c104b30 0x7f349c197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:50.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:50.012+0000 7f34a2269640 1 -- 172.21.15.40:0/1000241903 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f349c19f6c0 con 0x7f349c102200 2024-04-21T23:15:50.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:50.012+0000 7f34a1a68640 1 --2- 172.21.15.40:0/1000241903 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f349c104b30 0x7f349c197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:15:50.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:50.012+0000 7f34a2269640 1 --2- 172.21.15.40:0/1000241903 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f349c102200 0x7f349c197aa0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f3498009230 tx=0x7f3498005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:50.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:50.013+0000 7f34937fe640 1 -- 172.21.15.40:0/1000241903 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3498041070 con 0x7f349c102200 2024-04-21T23:15:50.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:50.013+0000 7f34937fe640 1 -- 172.21.15.40:0/1000241903 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34980044d0 con 0x7f349c102200 2024-04-21T23:15:50.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:50.013+0000 7f34a44cd640 1 -- 172.21.15.40:0/1000241903 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f349c19f950 con 0x7f349c102200 2024-04-21T23:15:50.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:50.013+0000 7f34937fe640 1 -- 172.21.15.40:0/1000241903 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3498034440 con 0x7f349c102200 2024-04-21T23:15:50.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:50.013+0000 7f34a44cd640 1 -- 172.21.15.40:0/1000241903 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f349c19fe40 con 0x7f349c102200 2024-04-21T23:15:50.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:50.014+0000 7f34a44cd640 1 -- 172.21.15.40:0/1000241903 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f349c0fd6e0 con 0x7f349c102200 2024-04-21T23:15:50.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:50.015+0000 7f34937fe640 1 -- 172.21.15.40:0/1000241903 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f3498044070 con 0x7f349c102200 2024-04-21T23:15:50.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:50.015+0000 7f34937fe640 1 --2- 172.21.15.40:0/1000241903 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f34880a2b90 0x7f34880a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:50.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:50.015+0000 7f34a1a68640 1 --2- 172.21.15.40:0/1000241903 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f34880a2b90 0x7f34880a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:50.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:50.015+0000 7f34937fe640 1 -- 172.21.15.40:0/1000241903 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f34980f0810 con 0x7f349c102200 2024-04-21T23:15:50.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:50.016+0000 7f34a1a68640 1 --2- 172.21.15.40:0/1000241903 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f34880a2b90 0x7f34880a5050 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f349c198fb0 tx=0x7f348c005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:50.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:50.020+0000 7f34937fe640 1 -- 172.21.15.40:0/1000241903 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f34980bdc10 con 0x7f349c102200 2024-04-21T23:15:50.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:50.251+0000 7f34a44cd640 1 -- 172.21.15.40:0/1000241903 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f349c02d170 con 0x7f349c102200 2024-04-21T23:15:50.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:50.251+0000 7f34937fe640 1 -- 172.21.15.40:0/1000241903 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9662 (secure 0 0 0) 0x7f34980bd510 con 0x7f349c102200 2024-04-21T23:15:50.251 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:15:50.251 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/1571948830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1571948830},{"type":"v1","addr":"172.21.15.40:6835","nonce":1571948830}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/969356642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":969356642},{"type":"v1","addr":"172.21.15.40:6837","nonce":969356642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6839/3025057845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":3025057845},{"type":"v1","addr":"172.21.15.40:6839","nonce":3025057845}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6841/3350488804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":3350488804},{"type":"v1","addr":"172.21.15.40:6841","nonce":3350488804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/3438493203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":3438493203},{"type":"v1","addr":"172.21.15.174:6833","nonce":3438493203}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/3105752678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":3105752678},{"type":"v1","addr":"172.21.15.111:6833","nonce":3105752678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24461,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/2704462092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2704462092},{"type":"v1","addr":"172.21.15.111:6835","nonce":2704462092}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/3905733045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3905733045},{"type":"v1","addr":"172.21.15.174:6835","nonce":3905733045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24484,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/465579746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":465579746},{"type":"v1","addr":"172.21.15.174:6837","nonce":465579746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24494,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6837/2703764182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":2703764182},{"type":"v1","addr":"172.21.15.111:6837","nonce":2703764182}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24502,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6839/2467509274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":2467509274},{"type":"v1","addr":"172.21.15.174:6839","nonce":2467509274}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-21T23:15:50.251 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 12 2024-04-21T23:15:50.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:50.253+0000 7f34a44cd640 1 -- 172.21.15.40:0/1000241903 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f34880a2b90 msgr2=0x7f34880a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:50.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:50.253+0000 7f34a44cd640 1 --2- 172.21.15.40:0/1000241903 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f34880a2b90 0x7f34880a5050 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f349c198fb0 tx=0x7f348c005f90 comp rx=0 tx=0).stop 2024-04-21T23:15:50.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:50.253+0000 7f34a44cd640 1 -- 172.21.15.40:0/1000241903 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f349c102200 msgr2=0x7f349c197aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:50.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:50.253+0000 7f34a44cd640 1 --2- 172.21.15.40:0/1000241903 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f349c102200 0x7f349c197aa0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f3498009230 tx=0x7f3498005d50 comp rx=0 tx=0).stop 2024-04-21T23:15:50.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:50.254+0000 7f34a44cd640 1 -- 172.21.15.40:0/1000241903 shutdown_connections 2024-04-21T23:15:50.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:50.254+0000 7f34a44cd640 1 --2- 172.21.15.40:0/1000241903 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f34880a2b90 0x7f34880a5050 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:50.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:50.254+0000 7f34a44cd640 1 --2- 172.21.15.40:0/1000241903 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f349c104b30 0x7f349c197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:50.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:50.254+0000 7f34a44cd640 1 --2- 172.21.15.40:0/1000241903 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f349c107460 0x7f349c19f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:50.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:50.254+0000 7f34a44cd640 1 --2- 172.21.15.40:0/1000241903 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f349c102200 0x7f349c197aa0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:50.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:50.254+0000 7f34a44cd640 1 -- 172.21.15.40:0/1000241903 >> 172.21.15.40:0/1000241903 conn(0x7f349c0fbbf0 msgr2=0x7f349c102c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:50.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:50.254+0000 7f34a44cd640 1 -- 172.21.15.40:0/1000241903 shutdown_connections 2024-04-21T23:15:50.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:50.254+0000 7f34a44cd640 1 -- 172.21.15.40:0/1000241903 wait complete. 2024-04-21T23:15:50.444 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 21 23:15:50 smithi111 systemd[1]: Started Ceph mds.b for d814e4dc-0033-11ef-bc93-c7b262605968. 2024-04-21T23:15:51.384 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:51 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:51.384 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:51 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:51.384 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:51 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:51.384 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:51 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:51.384 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:51 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:51.384 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:51 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/1000241903' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:15:51.384 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:51 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:15:51.384 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:51 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:51.384 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:51 smithi111 ceph-mon[31085]: mds.? [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] up:boot 2024-04-21T23:15:51.384 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:51 smithi111 ceph-mon[31085]: fsmap 12 up:standby 2024-04-21T23:15:51.384 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:51 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-21T23:15:51.418 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:51 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:51.418 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:51 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:51.418 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:51 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:51.418 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:51 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:51.418 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:51 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:51.418 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:51 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/1000241903' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:15:51.418 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:51 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:15:51.418 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:51 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:51.418 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:51 smithi174 ceph-mon[31409]: mds.? [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] up:boot 2024-04-21T23:15:51.419 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:51 smithi174 ceph-mon[31409]: fsmap 12 up:standby 2024-04-21T23:15:51.419 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:51 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-21T23:15:51.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:51 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:51.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:51 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:51.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:51 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:51.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:51 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:51.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:51 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:51.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:51 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/1000241903' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:15:51.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:51 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:15:51.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:51 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:51.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:51 smithi040 ceph-mon[27984]: mds.? [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] up:boot 2024-04-21T23:15:51.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:51 smithi040 ceph-mon[27984]: fsmap 12 up:standby 2024-04-21T23:15:51.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:51 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-21T23:15:52.264 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:15:52.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.368+0000 7f85b4288640 1 -- 172.21.15.40:0/1998493826 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85ac101890 msgr2=0x7f85ac101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:52.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.368+0000 7f85b4288640 1 --2- 172.21.15.40:0/1998493826 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85ac101890 0x7f85ac101d10 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f85a0009900 tx=0x7f85a002bc30 comp rx=0 tx=0).stop 2024-04-21T23:15:52.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.368+0000 7f85b4288640 1 -- 172.21.15.40:0/1998493826 shutdown_connections 2024-04-21T23:15:52.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.368+0000 7f85b4288640 1 --2- 172.21.15.40:0/1998493826 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f85ac102250 0x7f85ac108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:52.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.368+0000 7f85b4288640 1 --2- 172.21.15.40:0/1998493826 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85ac101890 0x7f85ac101d10 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:52.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.368+0000 7f85b4288640 1 --2- 172.21.15.40:0/1998493826 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f85ac1006a0 0x7f85ac100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:52.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.368+0000 7f85b4288640 1 -- 172.21.15.40:0/1998493826 >> 172.21.15.40:0/1998493826 conn(0x7f85ac0fbc30 msgr2=0x7f85ac0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:52.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.369+0000 7f85b4288640 1 -- 172.21.15.40:0/1998493826 shutdown_connections 2024-04-21T23:15:52.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.369+0000 7f85b4288640 1 -- 172.21.15.40:0/1998493826 wait complete. 2024-04-21T23:15:52.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.369+0000 7f85b4288640 1 Processor -- start 2024-04-21T23:15:52.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.369+0000 7f85b4288640 1 -- start start 2024-04-21T23:15:52.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.369+0000 7f85b4288640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85ac1006a0 0x7f85ac197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:52.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.369+0000 7f85b4288640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f85ac101890 0x7f85ac198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:52.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.369+0000 7f85b4288640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f85ac102250 0x7f85ac19f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:52.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.369+0000 7f85b4288640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f85ac10c320 con 0x7f85ac1006a0 2024-04-21T23:15:52.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.369+0000 7f85b4288640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f85ac10c1a0 con 0x7f85ac102250 2024-04-21T23:15:52.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.369+0000 7f85b4288640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f85ac10c4a0 con 0x7f85ac101890 2024-04-21T23:15:52.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.369+0000 7f85b2024640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85ac1006a0 0x7f85ac197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:52.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.369+0000 7f85b2024640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85ac1006a0 0x7f85ac197b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46344/0 (socket says 172.21.15.40:46344) 2024-04-21T23:15:52.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.369+0000 7f85b2024640 1 -- 172.21.15.40:0/4220739181 learned_addr learned my addr 172.21.15.40:0/4220739181 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:15:52.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.369+0000 7f85b1823640 1 --2- 172.21.15.40:0/4220739181 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f85ac101890 0x7f85ac198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:52.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.370+0000 7f85b2825640 1 --2- 172.21.15.40:0/4220739181 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f85ac102250 0x7f85ac19f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:52.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.370+0000 7f85b2024640 1 -- 172.21.15.40:0/4220739181 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f85ac101890 msgr2=0x7f85ac198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:52.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.370+0000 7f85b2024640 1 --2- 172.21.15.40:0/4220739181 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f85ac101890 0x7f85ac198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:52.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.370+0000 7f85b2024640 1 -- 172.21.15.40:0/4220739181 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f85ac102250 msgr2=0x7f85ac19f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:52.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.370+0000 7f85b2024640 1 --2- 172.21.15.40:0/4220739181 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f85ac102250 0x7f85ac19f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:52.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.370+0000 7f85b2024640 1 -- 172.21.15.40:0/4220739181 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85ac19f6e0 con 0x7f85ac1006a0 2024-04-21T23:15:52.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.370+0000 7f85b2024640 1 --2- 172.21.15.40:0/4220739181 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85ac1006a0 0x7f85ac197b10 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f859c00eb70 tx=0x7f859c00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:52.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.370+0000 7f85ab7fe640 1 -- 172.21.15.40:0/4220739181 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f859c003b70 con 0x7f85ac1006a0 2024-04-21T23:15:52.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.370+0000 7f85ab7fe640 1 -- 172.21.15.40:0/4220739181 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f859c00ee70 con 0x7f85ac1006a0 2024-04-21T23:15:52.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.370+0000 7f85b4288640 1 -- 172.21.15.40:0/4220739181 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85ac19f9d0 con 0x7f85ac1006a0 2024-04-21T23:15:52.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.370+0000 7f85ab7fe640 1 -- 172.21.15.40:0/4220739181 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f859c010830 con 0x7f85ac1006a0 2024-04-21T23:15:52.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.370+0000 7f85b4288640 1 -- 172.21.15.40:0/4220739181 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85ac19fef0 con 0x7f85ac1006a0 2024-04-21T23:15:52.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.371+0000 7f85b4288640 1 -- 172.21.15.40:0/4220739181 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85ac1a0120 con 0x7f85ac1006a0 2024-04-21T23:15:52.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.372+0000 7f85ab7fe640 1 -- 172.21.15.40:0/4220739181 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f859c004020 con 0x7f85ac1006a0 2024-04-21T23:15:52.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.373+0000 7f85ab7fe640 1 --2- 172.21.15.40:0/4220739181 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f85940a2c10 0x7f85940a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:52.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.373+0000 7f85b1823640 1 --2- 172.21.15.40:0/4220739181 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f85940a2c10 0x7f85940a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:52.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.373+0000 7f85ab7fe640 1 -- 172.21.15.40:0/4220739181 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f859c0cf170 con 0x7f85ac1006a0 2024-04-21T23:15:52.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.374+0000 7f85b1823640 1 --2- 172.21.15.40:0/4220739181 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f85940a2c10 0x7f85940a50d0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f85a002c7b0 tx=0x7f85a0005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:52.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.378+0000 7f85ab7fe640 1 -- 172.21.15.40:0/4220739181 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f859c09c570 con 0x7f85ac1006a0 2024-04-21T23:15:52.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:52 smithi040 ceph-mon[27984]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:52.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:52 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:52.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:52 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:52.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:52 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:52.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:52 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:52.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:52 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:52.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:52 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:52.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:52 smithi174 ceph-mon[31409]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:52.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:52 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:52.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:52 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:52.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:52 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:52.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:52 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:52.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:52 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:52.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:52 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:52.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:52 smithi111 ceph-mon[31085]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:52.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:52 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:52.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:52 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:52.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:52 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:52.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:52 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:52.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:52 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:52.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:52 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:52.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.601+0000 7f85b4288640 1 -- 172.21.15.40:0/4220739181 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f85ac1a0390 con 0x7f85ac1006a0 2024-04-21T23:15:52.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.602+0000 7f85ab7fe640 1 -- 172.21.15.40:0/4220739181 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10491 (secure 0 0 0) 0x7f859c09be70 con 0x7f85ac1006a0 2024-04-21T23:15:52.601 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:15:52.601 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/1571948830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1571948830},{"type":"v1","addr":"172.21.15.40:6835","nonce":1571948830}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/969356642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":969356642},{"type":"v1","addr":"172.21.15.40:6837","nonce":969356642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6839/3025057845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":3025057845},{"type":"v1","addr":"172.21.15.40:6839","nonce":3025057845}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6841/3350488804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":3350488804},{"type":"v1","addr":"172.21.15.40:6841","nonce":3350488804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/3438493203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":3438493203},{"type":"v1","addr":"172.21.15.174:6833","nonce":3438493203}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/3105752678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":3105752678},{"type":"v1","addr":"172.21.15.111:6833","nonce":3105752678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24461,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/2704462092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2704462092},{"type":"v1","addr":"172.21.15.111:6835","nonce":2704462092}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/3905733045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3905733045},{"type":"v1","addr":"172.21.15.174:6835","nonce":3905733045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24484,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/465579746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":465579746},{"type":"v1","addr":"172.21.15.174:6837","nonce":465579746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24494,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6837/2703764182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":2703764182},{"type":"v1","addr":"172.21.15.111:6837","nonce":2703764182}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24502,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6839/2467509274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":2467509274},{"type":"v1","addr":"172.21.15.174:6839","nonce":2467509274}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24521,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6839/3712235548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3712235548},{"type":"v1","addr":"172.21.15.111:6839","nonce":3712235548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-04-21T23:15:52.601 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 13 2024-04-21T23:15:52.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.604+0000 7f85b4288640 1 -- 172.21.15.40:0/4220739181 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f85940a2c10 msgr2=0x7f85940a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:52.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.604+0000 7f85b4288640 1 --2- 172.21.15.40:0/4220739181 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f85940a2c10 0x7f85940a50d0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f85a002c7b0 tx=0x7f85a0005fd0 comp rx=0 tx=0).stop 2024-04-21T23:15:52.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.604+0000 7f85b4288640 1 -- 172.21.15.40:0/4220739181 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85ac1006a0 msgr2=0x7f85ac197b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:52.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.604+0000 7f85b4288640 1 --2- 172.21.15.40:0/4220739181 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85ac1006a0 0x7f85ac197b10 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f859c00eb70 tx=0x7f859c00c6a0 comp rx=0 tx=0).stop 2024-04-21T23:15:52.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.604+0000 7f85b4288640 1 -- 172.21.15.40:0/4220739181 shutdown_connections 2024-04-21T23:15:52.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.604+0000 7f85b4288640 1 --2- 172.21.15.40:0/4220739181 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f85940a2c10 0x7f85940a50d0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:52.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.604+0000 7f85b4288640 1 --2- 172.21.15.40:0/4220739181 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f85ac102250 0x7f85ac19f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:52.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.604+0000 7f85b4288640 1 --2- 172.21.15.40:0/4220739181 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f85ac101890 0x7f85ac198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:52.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.604+0000 7f85b4288640 1 --2- 172.21.15.40:0/4220739181 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85ac1006a0 0x7f85ac197b10 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:52.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.604+0000 7f85b4288640 1 -- 172.21.15.40:0/4220739181 >> 172.21.15.40:0/4220739181 conn(0x7f85ac0fbc30 msgr2=0x7f85ac0fd710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:52.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.604+0000 7f85b4288640 1 -- 172.21.15.40:0/4220739181 shutdown_connections 2024-04-21T23:15:52.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.604+0000 7f85b4288640 1 -- 172.21.15.40:0/4220739181 wait complete. 2024-04-21T23:15:52.614 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T23:15:52.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.718+0000 7f620278a640 1 -- 172.21.15.40:0/1491931133 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f61fc1029a0 msgr2=0x7f61fc102e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:52.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.718+0000 7f620278a640 1 --2- 172.21.15.40:0/1491931133 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f61fc1029a0 0x7f61fc102e20 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f61f0009900 tx=0x7f61f002bc30 comp rx=0 tx=0).stop 2024-04-21T23:15:52.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.718+0000 7f620278a640 1 -- 172.21.15.40:0/1491931133 shutdown_connections 2024-04-21T23:15:52.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.718+0000 7f620278a640 1 --2- 172.21.15.40:0/1491931133 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f61fc103360 0x7f61fc109bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:52.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.718+0000 7f620278a640 1 --2- 172.21.15.40:0/1491931133 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f61fc1029a0 0x7f61fc102e20 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:52.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.718+0000 7f620278a640 1 --2- 172.21.15.40:0/1491931133 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f61fc1017b0 0x7f61fc101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:52.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.718+0000 7f620278a640 1 -- 172.21.15.40:0/1491931133 >> 172.21.15.40:0/1491931133 conn(0x7f61fc0fcd40 msgr2=0x7f61fc0ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:52.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.718+0000 7f620278a640 1 -- 172.21.15.40:0/1491931133 shutdown_connections 2024-04-21T23:15:52.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.718+0000 7f620278a640 1 -- 172.21.15.40:0/1491931133 wait complete. 2024-04-21T23:15:52.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.719+0000 7f620278a640 1 Processor -- start 2024-04-21T23:15:52.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.719+0000 7f620278a640 1 -- start start 2024-04-21T23:15:52.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.719+0000 7f620278a640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f61fc1017b0 0x7f61fc199c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:52.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.719+0000 7f620278a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f61fc1029a0 0x7f61fc19a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:52.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.719+0000 7f620278a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f61fc103360 0x7f61fc1a11c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:52.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.719+0000 7f620278a640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f61fc10d430 con 0x7f61fc1029a0 2024-04-21T23:15:52.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.719+0000 7f620278a640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f61fc10d2b0 con 0x7f61fc1017b0 2024-04-21T23:15:52.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.719+0000 7f620278a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f61fc10d5b0 con 0x7f61fc103360 2024-04-21T23:15:52.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.719+0000 7f61fb7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f61fc1029a0 0x7f61fc19a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:52.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.719+0000 7f61fb7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f61fc1029a0 0x7f61fc19a190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46370/0 (socket says 172.21.15.40:46370) 2024-04-21T23:15:52.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.719+0000 7f61fb7fe640 1 -- 172.21.15.40:0/2381959100 learned_addr learned my addr 172.21.15.40:0/2381959100 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:15:52.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.719+0000 7f61fbfff640 1 --2- 172.21.15.40:0/2381959100 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f61fc1017b0 0x7f61fc199c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:52.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.719+0000 7f6200d27640 1 --2- 172.21.15.40:0/2381959100 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f61fc103360 0x7f61fc1a11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:52.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.719+0000 7f61fb7fe640 1 -- 172.21.15.40:0/2381959100 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f61fc103360 msgr2=0x7f61fc1a11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:52.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.719+0000 7f61fb7fe640 1 --2- 172.21.15.40:0/2381959100 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f61fc103360 0x7f61fc1a11c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:52.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.719+0000 7f61fb7fe640 1 -- 172.21.15.40:0/2381959100 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f61fc1017b0 msgr2=0x7f61fc199c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:52.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.719+0000 7f61fb7fe640 1 --2- 172.21.15.40:0/2381959100 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f61fc1017b0 0x7f61fc199c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:52.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.719+0000 7f61fb7fe640 1 -- 172.21.15.40:0/2381959100 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61fc1a1860 con 0x7f61fc1029a0 2024-04-21T23:15:52.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.720+0000 7f61fb7fe640 1 --2- 172.21.15.40:0/2381959100 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f61fc1029a0 0x7f61fc19a190 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f61f0009230 tx=0x7f61f0005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:52.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.720+0000 7f61f97fa640 1 -- 172.21.15.40:0/2381959100 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61f0041070 con 0x7f61fc1029a0 2024-04-21T23:15:52.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.720+0000 7f61f97fa640 1 -- 172.21.15.40:0/2381959100 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f61f00054e0 con 0x7f61fc1029a0 2024-04-21T23:15:52.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.720+0000 7f620278a640 1 -- 172.21.15.40:0/2381959100 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61fc1a1af0 con 0x7f61fc1029a0 2024-04-21T23:15:52.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.720+0000 7f61f97fa640 1 -- 172.21.15.40:0/2381959100 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61f0034440 con 0x7f61fc1029a0 2024-04-21T23:15:52.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.720+0000 7f620278a640 1 -- 172.21.15.40:0/2381959100 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61fc1a1fe0 con 0x7f61fc1029a0 2024-04-21T23:15:52.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.721+0000 7f620278a640 1 -- 172.21.15.40:0/2381959100 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61fc1a21b0 con 0x7f61fc1029a0 2024-04-21T23:15:52.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.722+0000 7f61f97fa640 1 -- 172.21.15.40:0/2381959100 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f61f0044070 con 0x7f61fc1029a0 2024-04-21T23:15:52.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.723+0000 7f61f97fa640 1 --2- 172.21.15.40:0/2381959100 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f61e40a2b90 0x7f61e40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:52.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.723+0000 7f61fbfff640 1 --2- 172.21.15.40:0/2381959100 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f61e40a2b90 0x7f61e40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:52.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.723+0000 7f61f97fa640 1 -- 172.21.15.40:0/2381959100 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f61f00f0e00 con 0x7f61fc1029a0 2024-04-21T23:15:52.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.723+0000 7f61fbfff640 1 --2- 172.21.15.40:0/2381959100 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f61e40a2b90 0x7f61e40a5050 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f61fc19b120 tx=0x7f61ec005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:52.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.726+0000 7f61f97fa640 1 -- 172.21.15.40:0/2381959100 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f61f00be160 con 0x7f61fc1029a0 2024-04-21T23:15:52.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.915+0000 7f620278a640 1 -- 172.21.15.40:0/2381959100 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f61fc1a2540 con 0x7f61fc1029a0 2024-04-21T23:15:52.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.916+0000 7f61f97fa640 1 -- 172.21.15.40:0/2381959100 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+18466 (secure 0 0 0) 0x7f61f00bda60 con 0x7f61fc1029a0 2024-04-21T23:15:52.916 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:15:52.916 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":82,"fsid":"d814e4dc-0033-11ef-bc93-c7b262605968","created":"2024-04-21T23:07:52.659207+0000","modified":"2024-04-21T23:15:06.317496+0000","last_up_change":"2024-04-21T23:15:05.308821+0000","last_in_change":"2024-04-21T23:14:45.447491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:11:14.824775+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0e8d692a-22fe-4a87-8d99-9151f96cbb02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6803","nonce":2009758784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6805","nonce":2009758784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6809","nonce":2009758784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6807","nonce":2009758784}]},"public_addr":"172.21.15.40:6803/2009758784","cluster_addr":"172.21.15.40:6805/2009758784","heartbeat_back_addr":"172.21.15.40:6809/2009758784","heartbeat_front_addr":"172.21.15.40:6807/2009758784","state":["exists","up"]},{"osd":1,"uuid":"fe5fe11b-0e84-4359-83a7-1ba45105755a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6801","nonce":1763903836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6803","nonce":1763903836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6807","nonce":1763903836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6805","nonce":1763903836}]},"public_addr":"172.21.15.111:6801/1763903836","cluster_addr":"172.21.15.111:6803/1763903836","heartbeat_back_addr":"172.21.15.111:6807/1763903836","heartbeat_front_addr":"172.21.15.111:6805/1763903836","state":["exists","up"]},{"osd":2,"uuid":"813ad735-5cc2-4da0-ae16-6db238395fbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6801","nonce":439695825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6803","nonce":439695825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6807","nonce":439695825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6805","nonce":439695825}]},"public_addr":"172.21.15.174:6801/439695825","cluster_addr":"172.21.15.174:6803/439695825","heartbeat_back_addr":"172.21.15.174:6807/439695825","heartbeat_front_addr":"172.21.15.174:6805/439695825","state":["exists","up"]},{"osd":3,"uuid":"09becb03-b7ce-4ffc-917d-b7900ec37e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6811","nonce":3454312436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6813","nonce":3454312436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6817","nonce":3454312436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6815","nonce":3454312436}]},"public_addr":"172.21.15.40:6811/3454312436","cluster_addr":"172.21.15.40:6813/3454312436","heartbeat_back_addr":"172.21.15.40:6817/3454312436","heartbeat_front_addr":"172.21.15.40:6815/3454312436","state":["exists","up"]},{"osd":4,"uuid":"46eabfd7-d88a-4efd-94eb-04354d3a9d7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6809","nonce":1810866070}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6811","nonce":1810866070}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6815","nonce":1810866070}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6813","nonce":1810866070}]},"public_addr":"172.21.15.111:6809/1810866070","cluster_addr":"172.21.15.111:6811/1810866070","heartbeat_back_addr":"172.21.15.111:6815/1810866070","heartbeat_front_addr":"172.21.15.111:6813/1810866070","state":["exists","up"]},{"osd":5,"uuid":"6f73b582-6c65-41db-aab0-9b360f83d5c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6809","nonce":3752986711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6811","nonce":3752986711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6815","nonce":3752986711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6813","nonce":3752986711}]},"public_addr":"172.21.15.174:6809/3752986711","cluster_addr":"172.21.15.174:6811/3752986711","heartbeat_back_addr":"172.21.15.174:6815/3752986711","heartbeat_front_addr":"172.21.15.174:6813/3752986711","state":["exists","up"]},{"osd":6,"uuid":"39f305f0-12d2-4732-aaa0-cd66700af538","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6819","nonce":1618855427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6821","nonce":1618855427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6825","nonce":1618855427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6823","nonce":1618855427}]},"public_addr":"172.21.15.40:6819/1618855427","cluster_addr":"172.21.15.40:6821/1618855427","heartbeat_back_addr":"172.21.15.40:6825/1618855427","heartbeat_front_addr":"172.21.15.40:6823/1618855427","state":["exists","up"]},{"osd":7,"uuid":"769f2468-5102-490b-89f8-d0444d3169a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6817","nonce":927054162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6819","nonce":927054162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6823","nonce":927054162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6821","nonce":927054162}]},"public_addr":"172.21.15.111:6817/927054162","cluster_addr":"172.21.15.111:6819/927054162","heartbeat_back_addr":"172.21.15.111:6823/927054162","heartbeat_front_addr":"172.21.15.111:6821/927054162","state":["exists","up"]},{"osd":8,"uuid":"1a444e80-007d-473b-b972-4c814a813c22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6817","nonce":3817848746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6819","nonce":3817848746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6823","nonce":3817848746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6821","nonce":3817848746}]},"public_addr":"172.21.15.174:6817/3817848746","cluster_addr":"172.21.15.174:6819/3817848746","heartbeat_back_addr":"172.21.15.174:6823/3817848746","heartbeat_front_addr":"172.21.15.174:6821/3817848746","state":["exists","up"]},{"osd":9,"uuid":"9a6b6a56-6ae5-49fe-9e5b-cb17d686e0dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6827","nonce":1649446531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6829","nonce":1649446531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6833","nonce":1649446531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6831","nonce":1649446531}]},"public_addr":"172.21.15.40:6827/1649446531","cluster_addr":"172.21.15.40:6829/1649446531","heartbeat_back_addr":"172.21.15.40:6833/1649446531","heartbeat_front_addr":"172.21.15.40:6831/1649446531","state":["exists","up"]},{"osd":10,"uuid":"a70038ba-e2ad-4170-85cc-ea0470a2edef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6825","nonce":1227538935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6827","nonce":1227538935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6831","nonce":1227538935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6829","nonce":1227538935}]},"public_addr":"172.21.15.111:6825/1227538935","cluster_addr":"172.21.15.111:6827/1227538935","heartbeat_back_addr":"172.21.15.111:6831/1227538935","heartbeat_front_addr":"172.21.15.111:6829/1227538935","state":["exists","up"]},{"osd":11,"uuid":"781532bb-7a5e-44e3-8eb7-b584b0054666","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6825","nonce":2045865553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6827","nonce":2045865553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6831","nonce":2045865553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6829","nonce":2045865553}]},"public_addr":"172.21.15.174:6825/2045865553","cluster_addr":"172.21.15.174:6827/2045865553","heartbeat_back_addr":"172.21.15.174:6831/2045865553","heartbeat_front_addr":"172.21.15.174:6829/2045865553","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:10:24.761290+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:10:48.565428+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:11:12.307818+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:11:37.951529+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:02.832523+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:27.568879+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:52.997707+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:13:18.937274+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:13:44.708657+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:14:10.833855+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:14:37.078272+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:15:03.878076+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:6801/526554319":"2024-04-22T23:08:50.305770+0000","172.21.15.40:6800/526554319":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/3178763471":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/4045911364":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/276300080":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/3689649814":"2024-04-22T23:08:23.743030+0000","172.21.15.40:0/2241116715":"2024-04-22T23:08:23.743030+0000","172.21.15.40:6801/1479040326":"2024-04-22T23:08:23.743030+0000","172.21.15.40:6800/1479040326":"2024-04-22T23:08:23.743030+0000","172.21.15.40:0/681298099":"2024-04-22T23:08:23.743030+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T23:15:52.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.918+0000 7f620278a640 1 -- 172.21.15.40:0/2381959100 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f61e40a2b90 msgr2=0x7f61e40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:52.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.918+0000 7f620278a640 1 --2- 172.21.15.40:0/2381959100 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f61e40a2b90 0x7f61e40a5050 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f61fc19b120 tx=0x7f61ec005f90 comp rx=0 tx=0).stop 2024-04-21T23:15:52.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.918+0000 7f620278a640 1 -- 172.21.15.40:0/2381959100 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f61fc1029a0 msgr2=0x7f61fc19a190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:52.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.918+0000 7f620278a640 1 --2- 172.21.15.40:0/2381959100 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f61fc1029a0 0x7f61fc19a190 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f61f0009230 tx=0x7f61f0005dd0 comp rx=0 tx=0).stop 2024-04-21T23:15:52.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.918+0000 7f620278a640 1 -- 172.21.15.40:0/2381959100 shutdown_connections 2024-04-21T23:15:52.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.918+0000 7f620278a640 1 --2- 172.21.15.40:0/2381959100 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f61e40a2b90 0x7f61e40a5050 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:52.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.918+0000 7f620278a640 1 --2- 172.21.15.40:0/2381959100 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f61fc103360 0x7f61fc1a11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:52.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.918+0000 7f620278a640 1 --2- 172.21.15.40:0/2381959100 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f61fc1029a0 0x7f61fc19a190 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:52.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.918+0000 7f620278a640 1 --2- 172.21.15.40:0/2381959100 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f61fc1017b0 0x7f61fc199c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:52.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.919+0000 7f620278a640 1 -- 172.21.15.40:0/2381959100 >> 172.21.15.40:0/2381959100 conn(0x7f61fc0fcd40 msgr2=0x7f61fc073970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:52.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.919+0000 7f620278a640 1 -- 172.21.15.40:0/2381959100 shutdown_connections 2024-04-21T23:15:52.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:52.919+0000 7f620278a640 1 -- 172.21.15.40:0/2381959100 wait complete. 2024-04-21T23:15:52.929 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T23:11:14.824775+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-21T23:15:52.929 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T23:15:53.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.036+0000 7f67cdd6c640 1 -- 172.21.15.40:0/3055184799 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67c8102240 msgr2=0x7f67c8108ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:53.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.036+0000 7f67cdd6c640 1 --2- 172.21.15.40:0/3055184799 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67c8102240 0x7f67c8108ad0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f67c0009ca0 tx=0x7f67c002c0f0 comp rx=0 tx=0).stop 2024-04-21T23:15:53.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.036+0000 7f67cdd6c640 1 -- 172.21.15.40:0/3055184799 shutdown_connections 2024-04-21T23:15:53.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.036+0000 7f67cdd6c640 1 --2- 172.21.15.40:0/3055184799 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f67c8101880 0x7f67c8101d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:53.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.036+0000 7f67cdd6c640 1 --2- 172.21.15.40:0/3055184799 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67c8102240 0x7f67c8108ad0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:53.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.036+0000 7f67cdd6c640 1 --2- 172.21.15.40:0/3055184799 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f67c8100690 0x7f67c8100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:53.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.036+0000 7f67cdd6c640 1 -- 172.21.15.40:0/3055184799 >> 172.21.15.40:0/3055184799 conn(0x7f67c80fbc00 msgr2=0x7f67c80fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:53.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.036+0000 7f67cdd6c640 1 -- 172.21.15.40:0/3055184799 shutdown_connections 2024-04-21T23:15:53.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.036+0000 7f67cdd6c640 1 -- 172.21.15.40:0/3055184799 wait complete. 2024-04-21T23:15:53.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.037+0000 7f67cdd6c640 1 Processor -- start 2024-04-21T23:15:53.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.037+0000 7f67cdd6c640 1 -- start start 2024-04-21T23:15:53.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.037+0000 7f67cdd6c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f67c8100690 0x7f67c8199d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:53.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.037+0000 7f67cdd6c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67c8101880 0x7f67c819a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:53.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.037+0000 7f67cdd6c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f67c8102240 0x7f67c81a1270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:53.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.037+0000 7f67cdd6c640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f67c810c310 con 0x7f67c8101880 2024-04-21T23:15:53.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.037+0000 7f67cdd6c640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f67c810c190 con 0x7f67c8100690 2024-04-21T23:15:53.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.037+0000 7f67cdd6c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f67c810c490 con 0x7f67c8102240 2024-04-21T23:15:53.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.037+0000 7f67c6ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67c8101880 0x7f67c819a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:53.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.037+0000 7f67c6ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67c8101880 0x7f67c819a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46380/0 (socket says 172.21.15.40:46380) 2024-04-21T23:15:53.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.037+0000 7f67c6ffd640 1 -- 172.21.15.40:0/695951518 learned_addr learned my addr 172.21.15.40:0/695951518 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:15:53.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.037+0000 7f67c77fe640 1 --2- 172.21.15.40:0/695951518 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f67c8100690 0x7f67c8199d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:53.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.037+0000 7f67c7fff640 1 --2- 172.21.15.40:0/695951518 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f67c8102240 0x7f67c81a1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:53.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.038+0000 7f67c6ffd640 1 -- 172.21.15.40:0/695951518 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f67c8102240 msgr2=0x7f67c81a1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:53.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.038+0000 7f67c6ffd640 1 --2- 172.21.15.40:0/695951518 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f67c8102240 0x7f67c81a1270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:53.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.038+0000 7f67c6ffd640 1 -- 172.21.15.40:0/695951518 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f67c8100690 msgr2=0x7f67c8199d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:53.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.038+0000 7f67c6ffd640 1 --2- 172.21.15.40:0/695951518 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f67c8100690 0x7f67c8199d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:53.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.038+0000 7f67c6ffd640 1 -- 172.21.15.40:0/695951518 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67c81a1910 con 0x7f67c8101880 2024-04-21T23:15:53.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.038+0000 7f67c6ffd640 1 --2- 172.21.15.40:0/695951518 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67c8101880 0x7f67c819a240 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f67bc00e9f0 tx=0x7f67bc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:53.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.038+0000 7f67c4ff9640 1 -- 172.21.15.40:0/695951518 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67bc00cd80 con 0x7f67c8101880 2024-04-21T23:15:53.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.038+0000 7f67c4ff9640 1 -- 172.21.15.40:0/695951518 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67bc010910 con 0x7f67c8101880 2024-04-21T23:15:53.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.038+0000 7f67cdd6c640 1 -- 172.21.15.40:0/695951518 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67c81a1c00 con 0x7f67c8101880 2024-04-21T23:15:53.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.038+0000 7f67c4ff9640 1 -- 172.21.15.40:0/695951518 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67bc018990 con 0x7f67c8101880 2024-04-21T23:15:53.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.038+0000 7f67cdd6c640 1 -- 172.21.15.40:0/695951518 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67c81a2150 con 0x7f67c8101880 2024-04-21T23:15:53.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.039+0000 7f67cdd6c640 1 -- 172.21.15.40:0/695951518 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67c81a23e0 con 0x7f67c8101880 2024-04-21T23:15:53.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.041+0000 7f67c4ff9640 1 -- 172.21.15.40:0/695951518 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f67bc010ac0 con 0x7f67c8101880 2024-04-21T23:15:53.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.042+0000 7f67c4ff9640 1 --2- 172.21.15.40:0/695951518 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f67b00a2b90 0x7f67b00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:53.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.042+0000 7f67c4ff9640 1 -- 172.21.15.40:0/695951518 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f67bc0cf420 con 0x7f67c8101880 2024-04-21T23:15:53.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.042+0000 7f67c77fe640 1 --2- 172.21.15.40:0/695951518 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f67b00a2b90 0x7f67b00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:53.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.042+0000 7f67c77fe640 1 --2- 172.21.15.40:0/695951518 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f67b00a2b90 0x7f67b00a5050 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f67c819b1d0 tx=0x7f67b8005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:53.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.044+0000 7f67c4ff9640 1 -- 172.21.15.40:0/695951518 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f67bc09c130 con 0x7f67c8101880 2024-04-21T23:15:53.181 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:53 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/4220739181' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:15:53.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:53 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:15:53.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:53 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:15:53.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:53 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:53.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:53 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:15:53.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:53 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/2381959100' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:15:53.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.234+0000 7f67cdd6c640 1 -- 172.21.15.40:0/695951518 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f67c81a25b0 con 0x7f67c8101880 2024-04-21T23:15:53.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.235+0000 7f67c4ff9640 1 -- 172.21.15.40:0/695951518 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) ==== 93+0+10 (secure 0 0 0) 0x7f67bc0a0430 con 0x7f67c8101880 2024-04-21T23:15:53.234 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-04-21T23:15:53.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.236+0000 7f67cdd6c640 1 -- 172.21.15.40:0/695951518 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f67b00a2b90 msgr2=0x7f67b00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:53.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.236+0000 7f67cdd6c640 1 --2- 172.21.15.40:0/695951518 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f67b00a2b90 0x7f67b00a5050 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f67c819b1d0 tx=0x7f67b8005f90 comp rx=0 tx=0).stop 2024-04-21T23:15:53.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.236+0000 7f67cdd6c640 1 -- 172.21.15.40:0/695951518 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67c8101880 msgr2=0x7f67c819a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:53.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.236+0000 7f67cdd6c640 1 --2- 172.21.15.40:0/695951518 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67c8101880 0x7f67c819a240 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f67bc00e9f0 tx=0x7f67bc00eeb0 comp rx=0 tx=0).stop 2024-04-21T23:15:53.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.236+0000 7f67cdd6c640 1 -- 172.21.15.40:0/695951518 shutdown_connections 2024-04-21T23:15:53.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.236+0000 7f67cdd6c640 1 --2- 172.21.15.40:0/695951518 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f67b00a2b90 0x7f67b00a5050 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:53.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.236+0000 7f67cdd6c640 1 --2- 172.21.15.40:0/695951518 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f67c8102240 0x7f67c81a1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:53.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.237+0000 7f67cdd6c640 1 --2- 172.21.15.40:0/695951518 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67c8101880 0x7f67c819a240 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:53.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.237+0000 7f67cdd6c640 1 --2- 172.21.15.40:0/695951518 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f67c8100690 0x7f67c8199d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:53.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.237+0000 7f67cdd6c640 1 -- 172.21.15.40:0/695951518 >> 172.21.15.40:0/695951518 conn(0x7f67c80fbc00 msgr2=0x7f67c8069ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:53.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.237+0000 7f67cdd6c640 1 -- 172.21.15.40:0/695951518 shutdown_connections 2024-04-21T23:15:53.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.237+0000 7f67cdd6c640 1 -- 172.21.15.40:0/695951518 wait complete. 2024-04-21T23:15:53.247 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-21T23:15:53.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.354+0000 7f5211f9e640 1 -- 172.21.15.40:0/3946542513 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f520c1022b0 msgr2=0x7f520c1046a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:53.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.354+0000 7f5211f9e640 1 --2- 172.21.15.40:0/3946542513 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f520c1022b0 0x7f520c1046a0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f51fc009900 tx=0x7f51fc02bc30 comp rx=0 tx=0).stop 2024-04-21T23:15:53.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.354+0000 7f5211f9e640 1 -- 172.21.15.40:0/3946542513 shutdown_connections 2024-04-21T23:15:53.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.354+0000 7f5211f9e640 1 --2- 172.21.15.40:0/3946542513 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f520c107510 0x7f520c109900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:53.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.354+0000 7f5211f9e640 1 --2- 172.21.15.40:0/3946542513 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f520c104be0 0x7f520c106fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:53.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.354+0000 7f5211f9e640 1 --2- 172.21.15.40:0/3946542513 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f520c1022b0 0x7f520c1046a0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:53.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.354+0000 7f5211f9e640 1 -- 172.21.15.40:0/3946542513 >> 172.21.15.40:0/3946542513 conn(0x7f520c0fbc20 msgr2=0x7f520c0fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:53.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.354+0000 7f5211f9e640 1 -- 172.21.15.40:0/3946542513 shutdown_connections 2024-04-21T23:15:53.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.354+0000 7f5211f9e640 1 -- 172.21.15.40:0/3946542513 wait complete. 2024-04-21T23:15:53.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.355+0000 7f5211f9e640 1 Processor -- start 2024-04-21T23:15:53.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.355+0000 7f5211f9e640 1 -- start start 2024-04-21T23:15:53.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.355+0000 7f5211f9e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f520c1022b0 0x7f520c197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:53.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.355+0000 7f5211f9e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f520c104be0 0x7f520c198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:53.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.355+0000 7f5211f9e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f520c107510 0x7f520c19f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:53.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.355+0000 7f5211f9e640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f520c10d430 con 0x7f520c107510 2024-04-21T23:15:53.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.355+0000 7f5211f9e640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f520c10d2b0 con 0x7f520c1022b0 2024-04-21T23:15:53.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.355+0000 7f5211f9e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f520c10d5b0 con 0x7f520c104be0 2024-04-21T23:15:53.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.355+0000 7f5202ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f520c104be0 0x7f520c198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:53.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.355+0000 7f520b7fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f520c1022b0 0x7f520c197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:53.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.355+0000 7f5202ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f520c104be0 0x7f520c198080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.40:51886/0 (socket says 172.21.15.40:51886) 2024-04-21T23:15:53.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.355+0000 7f5202ffd640 1 -- 172.21.15.40:0/1627108156 learned_addr learned my addr 172.21.15.40:0/1627108156 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:15:53.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.355+0000 7f520bfff640 1 --2- 172.21.15.40:0/1627108156 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f520c107510 0x7f520c19f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:53.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.356+0000 7f5202ffd640 1 -- 172.21.15.40:0/1627108156 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f520c1022b0 msgr2=0x7f520c197b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:53.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.356+0000 7f5202ffd640 1 --2- 172.21.15.40:0/1627108156 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f520c1022b0 0x7f520c197b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:53.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.356+0000 7f5202ffd640 1 -- 172.21.15.40:0/1627108156 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f520c107510 msgr2=0x7f520c19f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:53.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.356+0000 7f5202ffd640 1 --2- 172.21.15.40:0/1627108156 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f520c107510 0x7f520c19f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:53.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.356+0000 7f5202ffd640 1 -- 172.21.15.40:0/1627108156 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f520c19f710 con 0x7f520c104be0 2024-04-21T23:15:53.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.356+0000 7f520b7fe640 1 --2- 172.21.15.40:0/1627108156 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f520c1022b0 0x7f520c197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:15:53.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.356+0000 7f520bfff640 1 --2- 172.21.15.40:0/1627108156 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f520c107510 0x7f520c19f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:15:53.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.356+0000 7f5202ffd640 1 --2- 172.21.15.40:0/1627108156 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f520c104be0 0x7f520c198080 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f51f400d740 tx=0x7f51f400dc00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:53.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.356+0000 7f52097fa640 1 -- 172.21.15.40:0/1627108156 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51f4017070 con 0x7f520c104be0 2024-04-21T23:15:53.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.356+0000 7f52097fa640 1 -- 172.21.15.40:0/1627108156 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51f400de50 con 0x7f520c104be0 2024-04-21T23:15:53.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.356+0000 7f5211f9e640 1 -- 172.21.15.40:0/1627108156 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f520c19fa00 con 0x7f520c104be0 2024-04-21T23:15:53.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.356+0000 7f52097fa640 1 -- 172.21.15.40:0/1627108156 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51f4004e00 con 0x7f520c104be0 2024-04-21T23:15:53.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.356+0000 7f5211f9e640 1 -- 172.21.15.40:0/1627108156 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f520c19ff00 con 0x7f520c104be0 2024-04-21T23:15:53.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.357+0000 7f5211f9e640 1 -- 172.21.15.40:0/1627108156 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f520c0fd710 con 0x7f520c104be0 2024-04-21T23:15:53.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.358+0000 7f52097fa640 1 -- 172.21.15.40:0/1627108156 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f51f4004fa0 con 0x7f520c104be0 2024-04-21T23:15:53.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.359+0000 7f52097fa640 1 --2- 172.21.15.40:0/1627108156 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f51f80a2b90 0x7f51f80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:53.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.359+0000 7f520b7fe640 1 --2- 172.21.15.40:0/1627108156 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f51f80a2b90 0x7f51f80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:53.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.359+0000 7f52097fa640 1 -- 172.21.15.40:0/1627108156 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f51f40ce520 con 0x7f520c104be0 2024-04-21T23:15:53.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.360+0000 7f520b7fe640 1 --2- 172.21.15.40:0/1627108156 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f51f80a2b90 0x7f51f80a5050 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f51fc02c7b0 tx=0x7f51fc005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:53.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.363+0000 7f52097fa640 1 -- 172.21.15.40:0/1627108156 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f51f409b230 con 0x7f520c104be0 2024-04-21T23:15:53.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:53 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/4220739181' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:15:53.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:53 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:15:53.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:53 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:15:53.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:53 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:53.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:53 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:15:53.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:53 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/2381959100' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:15:53.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:53 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/4220739181' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:15:53.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:53 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:15:53.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:53 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:15:53.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:53 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:53.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:53 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:15:53.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:53 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/2381959100' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:15:53.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.555+0000 7f5211f9e640 1 -- 172.21.15.40:0/1627108156 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f520c02d3c0 con 0x7f520c104be0 2024-04-21T23:15:53.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.555+0000 7f52097fa640 1 -- 172.21.15.40:0/1627108156 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v82) ==== 84+0+65 (secure 0 0 0) 0x7f51f409f530 con 0x7f520c104be0 2024-04-21T23:15:53.554 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:15:53.555 INFO:teuthology.orchestra.run.smithi040.stdout:[ 2024-04-21T23:15:53.555 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:15:53.555 INFO:teuthology.orchestra.run.smithi040.stdout: "poolnum": 1, 2024-04-21T23:15:53.555 INFO:teuthology.orchestra.run.smithi040.stdout: "poolname": ".mgr" 2024-04-21T23:15:53.555 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:15:53.555 INFO:teuthology.orchestra.run.smithi040.stdout:] 2024-04-21T23:15:53.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.557+0000 7f5211f9e640 1 -- 172.21.15.40:0/1627108156 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f51f80a2b90 msgr2=0x7f51f80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:53.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.557+0000 7f5211f9e640 1 --2- 172.21.15.40:0/1627108156 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f51f80a2b90 0x7f51f80a5050 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f51fc02c7b0 tx=0x7f51fc005fd0 comp rx=0 tx=0).stop 2024-04-21T23:15:53.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.557+0000 7f5211f9e640 1 -- 172.21.15.40:0/1627108156 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f520c104be0 msgr2=0x7f520c198080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:53.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.557+0000 7f5211f9e640 1 --2- 172.21.15.40:0/1627108156 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f520c104be0 0x7f520c198080 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f51f400d740 tx=0x7f51f400dc00 comp rx=0 tx=0).stop 2024-04-21T23:15:53.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.557+0000 7f5211f9e640 1 -- 172.21.15.40:0/1627108156 shutdown_connections 2024-04-21T23:15:53.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.557+0000 7f5211f9e640 1 --2- 172.21.15.40:0/1627108156 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f51f80a2b90 0x7f51f80a5050 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:53.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.557+0000 7f5211f9e640 1 --2- 172.21.15.40:0/1627108156 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f520c107510 0x7f520c19f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:53.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.557+0000 7f5211f9e640 1 --2- 172.21.15.40:0/1627108156 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f520c104be0 0x7f520c198080 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:53.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.557+0000 7f5211f9e640 1 --2- 172.21.15.40:0/1627108156 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f520c1022b0 0x7f520c197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:53.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.557+0000 7f5211f9e640 1 -- 172.21.15.40:0/1627108156 >> 172.21.15.40:0/1627108156 conn(0x7f520c0fbc20 msgr2=0x7f520c1030a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:53.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.557+0000 7f5211f9e640 1 -- 172.21.15.40:0/1627108156 shutdown_connections 2024-04-21T23:15:53.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.557+0000 7f5211f9e640 1 -- 172.21.15.40:0/1627108156 wait complete. 2024-04-21T23:15:53.567 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-21T23:15:53.567 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-21T23:15:53.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.675+0000 7f09a9e6d640 1 -- 172.21.15.40:0/3893709647 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09a41006a0 msgr2=0x7f09a4100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:53.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.675+0000 7f09a9e6d640 1 --2- 172.21.15.40:0/3893709647 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09a41006a0 0x7f09a4100ac0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f0994009900 tx=0x7f099402bc30 comp rx=0 tx=0).stop 2024-04-21T23:15:53.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.675+0000 7f09a9e6d640 1 -- 172.21.15.40:0/3893709647 shutdown_connections 2024-04-21T23:15:53.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.675+0000 7f09a9e6d640 1 --2- 172.21.15.40:0/3893709647 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09a4101890 0x7f09a4101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:53.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.675+0000 7f09a9e6d640 1 --2- 172.21.15.40:0/3893709647 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f09a4102250 0x7f09a4108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:53.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.675+0000 7f09a9e6d640 1 --2- 172.21.15.40:0/3893709647 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09a41006a0 0x7f09a4100ac0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:53.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.675+0000 7f09a9e6d640 1 -- 172.21.15.40:0/3893709647 >> 172.21.15.40:0/3893709647 conn(0x7f09a40fbc30 msgr2=0x7f09a40fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:53.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.675+0000 7f09a9e6d640 1 -- 172.21.15.40:0/3893709647 shutdown_connections 2024-04-21T23:15:53.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.675+0000 7f09a9e6d640 1 -- 172.21.15.40:0/3893709647 wait complete. 2024-04-21T23:15:53.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.675+0000 7f09a9e6d640 1 Processor -- start 2024-04-21T23:15:53.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.676+0000 7f09a9e6d640 1 -- start start 2024-04-21T23:15:53.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.676+0000 7f09a9e6d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09a41006a0 0x7f09a4197b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:53.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.676+0000 7f09a9e6d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09a4101890 0x7f09a41980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:53.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.676+0000 7f09a9e6d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f09a4102250 0x7f09a419f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:53.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.676+0000 7f09a9e6d640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f09a410d440 con 0x7f09a41006a0 2024-04-21T23:15:53.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.676+0000 7f09a9e6d640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f09a410d2c0 con 0x7f09a4101890 2024-04-21T23:15:53.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.676+0000 7f09a9e6d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f09a410d5c0 con 0x7f09a4102250 2024-04-21T23:15:53.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.676+0000 7f09a37fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09a41006a0 0x7f09a4197b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:53.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.676+0000 7f09a37fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09a41006a0 0x7f09a4197b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46414/0 (socket says 172.21.15.40:46414) 2024-04-21T23:15:53.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.676+0000 7f09a3fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f09a4102250 0x7f09a419f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:53.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.676+0000 7f09a37fe640 1 -- 172.21.15.40:0/2193148310 learned_addr learned my addr 172.21.15.40:0/2193148310 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:15:53.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.676+0000 7f09a2ffd640 1 --2- 172.21.15.40:0/2193148310 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09a4101890 0x7f09a41980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:53.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.676+0000 7f09a37fe640 1 -- 172.21.15.40:0/2193148310 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f09a4102250 msgr2=0x7f09a419f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:53.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.676+0000 7f09a37fe640 1 --2- 172.21.15.40:0/2193148310 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f09a4102250 0x7f09a419f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:53.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.676+0000 7f09a37fe640 1 -- 172.21.15.40:0/2193148310 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09a4101890 msgr2=0x7f09a41980c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:53.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.676+0000 7f09a37fe640 1 --2- 172.21.15.40:0/2193148310 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09a4101890 0x7f09a41980c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:53.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.676+0000 7f09a37fe640 1 -- 172.21.15.40:0/2193148310 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09a419f750 con 0x7f09a41006a0 2024-04-21T23:15:53.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.676+0000 7f09a3fff640 1 --2- 172.21.15.40:0/2193148310 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f09a4102250 0x7f09a419f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:15:53.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.676+0000 7f09a37fe640 1 --2- 172.21.15.40:0/2193148310 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09a41006a0 0x7f09a4197b80 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f0994000c00 tx=0x7f0994004960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:53.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.677+0000 7f09a0ff9640 1 -- 172.21.15.40:0/2193148310 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0994041070 con 0x7f09a41006a0 2024-04-21T23:15:53.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.677+0000 7f09a0ff9640 1 -- 172.21.15.40:0/2193148310 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0994004450 con 0x7f09a41006a0 2024-04-21T23:15:53.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.677+0000 7f09a0ff9640 1 -- 172.21.15.40:0/2193148310 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0994003df0 con 0x7f09a41006a0 2024-04-21T23:15:53.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.677+0000 7f09a9e6d640 1 -- 172.21.15.40:0/2193148310 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09a419f9e0 con 0x7f09a41006a0 2024-04-21T23:15:53.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.677+0000 7f09a9e6d640 1 -- 172.21.15.40:0/2193148310 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09a419fed0 con 0x7f09a41006a0 2024-04-21T23:15:53.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.678+0000 7f09a9e6d640 1 -- 172.21.15.40:0/2193148310 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09a4107ab0 con 0x7f09a41006a0 2024-04-21T23:15:53.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.679+0000 7f09a0ff9640 1 -- 172.21.15.40:0/2193148310 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f0994044070 con 0x7f09a41006a0 2024-04-21T23:15:53.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.680+0000 7f09a0ff9640 1 --2- 172.21.15.40:0/2193148310 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f09900a2b90 0x7f09900a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:53.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.680+0000 7f09a2ffd640 1 --2- 172.21.15.40:0/2193148310 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f09900a2b90 0x7f09900a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:53.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.680+0000 7f09a0ff9640 1 -- 172.21.15.40:0/2193148310 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f09940f0620 con 0x7f09a41006a0 2024-04-21T23:15:53.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.680+0000 7f09a2ffd640 1 --2- 172.21.15.40:0/2193148310 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f09900a2b90 0x7f09900a5050 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f09a4199040 tx=0x7f098c005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:53.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.685+0000 7f09a0ff9640 1 -- 172.21.15.40:0/2193148310 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f09940bd3d0 con 0x7f09a41006a0 2024-04-21T23:15:53.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:53.874+0000 7f09a9e6d640 1 -- 172.21.15.40:0/2193148310 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f09a402d360 con 0x7f09a41006a0 2024-04-21T23:15:54.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:54.373+0000 7f09a0ff9640 1 -- 172.21.15.40:0/2193148310 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v83) ==== 138+0+0 (secure 0 0 0) 0x7f09940c16d0 con 0x7f09a41006a0 2024-04-21T23:15:54.373 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_metadata' created 2024-04-21T23:15:54.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:54.377+0000 7f09a9e6d640 1 -- 172.21.15.40:0/2193148310 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f09900a2b90 msgr2=0x7f09900a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:54.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:54.377+0000 7f09a9e6d640 1 --2- 172.21.15.40:0/2193148310 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f09900a2b90 0x7f09900a5050 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f09a4199040 tx=0x7f098c005f90 comp rx=0 tx=0).stop 2024-04-21T23:15:54.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:54.377+0000 7f09a9e6d640 1 -- 172.21.15.40:0/2193148310 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09a41006a0 msgr2=0x7f09a4197b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:54.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:54.377+0000 7f09a9e6d640 1 --2- 172.21.15.40:0/2193148310 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09a41006a0 0x7f09a4197b80 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f0994000c00 tx=0x7f0994004960 comp rx=0 tx=0).stop 2024-04-21T23:15:54.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:54.377+0000 7f09a9e6d640 1 -- 172.21.15.40:0/2193148310 shutdown_connections 2024-04-21T23:15:54.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:54.377+0000 7f09a9e6d640 1 --2- 172.21.15.40:0/2193148310 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f09900a2b90 0x7f09900a5050 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:54.377+0000 7f09a9e6d640 1 --2- 172.21.15.40:0/2193148310 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09a4101890 0x7f09a41980c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:54.377+0000 7f09a9e6d640 1 --2- 172.21.15.40:0/2193148310 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f09a4102250 0x7f09a419f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:54.377+0000 7f09a9e6d640 1 --2- 172.21.15.40:0/2193148310 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09a41006a0 0x7f09a4197b80 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:54.377+0000 7f09a9e6d640 1 -- 172.21.15.40:0/2193148310 >> 172.21.15.40:0/2193148310 conn(0x7f09a40fbc30 msgr2=0x7f09a4104c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:54.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:54.377+0000 7f09a9e6d640 1 -- 172.21.15.40:0/2193148310 shutdown_connections 2024-04-21T23:15:54.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:54.378+0000 7f09a9e6d640 1 -- 172.21.15.40:0/2193148310 wait complete. 2024-04-21T23:15:54.391 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-04-21T23:15:54.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:54.510+0000 7ff4a7713640 1 -- 172.21.15.40:0/4023453356 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff4a00741a0 msgr2=0x7ff4a0074620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:54.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:54.510+0000 7ff4a7713640 1 --2- 172.21.15.40:0/4023453356 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff4a00741a0 0x7ff4a0074620 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ff494009900 tx=0x7ff49402bc30 comp rx=0 tx=0).stop 2024-04-21T23:15:54.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:54.510+0000 7ff4a7713640 1 -- 172.21.15.40:0/4023453356 shutdown_connections 2024-04-21T23:15:54.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:54.510+0000 7ff4a7713640 1 --2- 172.21.15.40:0/4023453356 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff4a01096a0 0x7ff4a010bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:54.510+0000 7ff4a7713640 1 --2- 172.21.15.40:0/4023453356 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff4a00741a0 0x7ff4a0074620 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:54.510+0000 7ff4a7713640 1 --2- 172.21.15.40:0/4023453356 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4a0075780 0x7ff4a0073bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:54.510+0000 7ff4a7713640 1 -- 172.21.15.40:0/4023453356 >> 172.21.15.40:0/4023453356 conn(0x7ff4a00fcd30 msgr2=0x7ff4a00ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:54.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:54.510+0000 7ff4a7713640 1 -- 172.21.15.40:0/4023453356 shutdown_connections 2024-04-21T23:15:54.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:54.510+0000 7ff4a7713640 1 -- 172.21.15.40:0/4023453356 wait complete. 2024-04-21T23:15:54.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:54.511+0000 7ff4a7713640 1 Processor -- start 2024-04-21T23:15:54.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:54.511+0000 7ff4a7713640 1 -- start start 2024-04-21T23:15:54.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:54.511+0000 7ff4a7713640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff4a00741a0 0x7ff4a0199c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:54.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:54.511+0000 7ff4a7713640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4a0075780 0x7ff4a019a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:54.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:54.511+0000 7ff4a7713640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff4a01096a0 0x7ff4a01a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:54.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:54.511+0000 7ff4a7713640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff4a010f5b0 con 0x7ff4a01096a0 2024-04-21T23:15:54.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:54.511+0000 7ff4a7713640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff4a010f430 con 0x7ff4a0075780 2024-04-21T23:15:54.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:54.511+0000 7ff4a7713640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff4a010f730 con 0x7ff4a00741a0 2024-04-21T23:15:54.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:54.511+0000 7ff4a5cb0640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff4a01096a0 0x7ff4a01a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:54.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:54.511+0000 7ff4a5cb0640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff4a01096a0 0x7ff4a01a11a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46430/0 (socket says 172.21.15.40:46430) 2024-04-21T23:15:54.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:54.511+0000 7ff4a5cb0640 1 -- 172.21.15.40:0/2855732045 learned_addr learned my addr 172.21.15.40:0/2855732045 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:15:54.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:54.511+0000 7ff4a54af640 1 --2- 172.21.15.40:0/2855732045 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff4a00741a0 0x7ff4a0199c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:54.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:54.511+0000 7ff4a4cae640 1 --2- 172.21.15.40:0/2855732045 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4a0075780 0x7ff4a019a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:54.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:54.511+0000 7ff4a5cb0640 1 -- 172.21.15.40:0/2855732045 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff4a00741a0 msgr2=0x7ff4a0199c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:54.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:54.511+0000 7ff4a5cb0640 1 --2- 172.21.15.40:0/2855732045 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff4a00741a0 0x7ff4a0199c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:54.511+0000 7ff4a5cb0640 1 -- 172.21.15.40:0/2855732045 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4a0075780 msgr2=0x7ff4a019a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:54.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:54.511+0000 7ff4a5cb0640 1 --2- 172.21.15.40:0/2855732045 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4a0075780 0x7ff4a019a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:54.511+0000 7ff4a5cb0640 1 -- 172.21.15.40:0/2855732045 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4a01a1840 con 0x7ff4a01096a0 2024-04-21T23:15:54.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:54.512+0000 7ff4a5cb0640 1 --2- 172.21.15.40:0/2855732045 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff4a01096a0 0x7ff4a01a11a0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ff49800cce0 tx=0x7ff49800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:54.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:54.512+0000 7ff4927fc640 1 -- 172.21.15.40:0/2855732045 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff49800f450 con 0x7ff4a01096a0 2024-04-21T23:15:54.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:54.512+0000 7ff4a7713640 1 -- 172.21.15.40:0/2855732045 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4a01a1b30 con 0x7ff4a01096a0 2024-04-21T23:15:54.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:54.512+0000 7ff4a7713640 1 -- 172.21.15.40:0/2855732045 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4a01a2080 con 0x7ff4a01096a0 2024-04-21T23:15:54.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:54.512+0000 7ff4927fc640 1 -- 172.21.15.40:0/2855732045 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff49800fad0 con 0x7ff4a01096a0 2024-04-21T23:15:54.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:54.512+0000 7ff4927fc640 1 -- 172.21.15.40:0/2855732045 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff498018b70 con 0x7ff4a01096a0 2024-04-21T23:15:54.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:54.513+0000 7ff4a7713640 1 -- 172.21.15.40:0/2855732045 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4a01083d0 con 0x7ff4a01096a0 2024-04-21T23:15:54.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:54.514+0000 7ff4927fc640 1 -- 172.21.15.40:0/2855732045 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7ff498018d10 con 0x7ff4a01096a0 2024-04-21T23:15:54.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:54.515+0000 7ff4927fc640 1 --2- 172.21.15.40:0/2855732045 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7ff48c0a2b90 0x7ff48c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:54.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:54.515+0000 7ff4a54af640 1 --2- 172.21.15.40:0/2855732045 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7ff48c0a2b90 0x7ff48c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:54.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:54.515+0000 7ff4927fc640 1 -- 172.21.15.40:0/2855732045 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7497+0+0 (secure 0 0 0) 0x7ff4980cf200 con 0x7ff4a01096a0 2024-04-21T23:15:54.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:54.516+0000 7ff4a54af640 1 --2- 172.21.15.40:0/2855732045 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7ff48c0a2b90 0x7ff48c0a5050 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff4a019b100 tx=0x7ff49c005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:54.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:54.519+0000 7ff4927fc640 1 -- 172.21.15.40:0/2855732045 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7ff498014030 con 0x7ff4a01096a0 2024-04-21T23:15:54.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:54 smithi040 ceph-mon[27984]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:54.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:54 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/695951518' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:15:54.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:54 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/1627108156' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-21T23:15:54.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:54 smithi174 ceph-mon[31409]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:54.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:54 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/695951518' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:15:54.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:54 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/1627108156' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-21T23:15:54.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:54 smithi111 ceph-mon[31085]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:54.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:54 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/695951518' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:15:54.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:54 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/1627108156' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-21T23:15:54.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:54.730+0000 7ff4a7713640 1 -- 172.21.15.40:0/2855732045 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7ff4a002d170 con 0x7ff4a01096a0 2024-04-21T23:15:55.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:55.380+0000 7ff4927fc640 1 -- 172.21.15.40:0/2855732045 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v84) ==== 171+0+0 (secure 0 0 0) 0x7ff49809c3f0 con 0x7ff4a01096a0 2024-04-21T23:15:55.380 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_data' created 2024-04-21T23:15:55.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:55.383+0000 7ff4a7713640 1 -- 172.21.15.40:0/2855732045 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7ff48c0a2b90 msgr2=0x7ff48c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:55.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:55.383+0000 7ff4a7713640 1 --2- 172.21.15.40:0/2855732045 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7ff48c0a2b90 0x7ff48c0a5050 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff4a019b100 tx=0x7ff49c005f90 comp rx=0 tx=0).stop 2024-04-21T23:15:55.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:55.383+0000 7ff4a7713640 1 -- 172.21.15.40:0/2855732045 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff4a01096a0 msgr2=0x7ff4a01a11a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:55.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:55.384+0000 7ff4a7713640 1 --2- 172.21.15.40:0/2855732045 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff4a01096a0 0x7ff4a01a11a0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ff49800cce0 tx=0x7ff49800a620 comp rx=0 tx=0).stop 2024-04-21T23:15:55.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:55.384+0000 7ff4a7713640 1 -- 172.21.15.40:0/2855732045 shutdown_connections 2024-04-21T23:15:55.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:55.384+0000 7ff4a7713640 1 --2- 172.21.15.40:0/2855732045 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7ff48c0a2b90 0x7ff48c0a5050 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:55.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:55.384+0000 7ff4a7713640 1 --2- 172.21.15.40:0/2855732045 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4a0075780 0x7ff4a019a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:55.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:55.384+0000 7ff4a7713640 1 --2- 172.21.15.40:0/2855732045 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff4a01096a0 0x7ff4a01a11a0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:55.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:55.384+0000 7ff4a7713640 1 --2- 172.21.15.40:0/2855732045 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff4a00741a0 0x7ff4a0199c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:55.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:55.384+0000 7ff4a7713640 1 -- 172.21.15.40:0/2855732045 >> 172.21.15.40:0/2855732045 conn(0x7ff4a00fcd30 msgr2=0x7ff4a01055a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:55.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:55.384+0000 7ff4a7713640 1 -- 172.21.15.40:0/2855732045 shutdown_connections 2024-04-21T23:15:55.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:55.384+0000 7ff4a7713640 1 -- 172.21.15.40:0/2855732045 wait complete. 2024-04-21T23:15:55.403 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-21T23:15:55.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:55.509+0000 7f64bc30f640 1 -- 172.21.15.40:0/1923836582 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f64b4102230 msgr2=0x7f64b4108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:55.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:55.509+0000 7f64bc30f640 1 --2- 172.21.15.40:0/1923836582 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f64b4102230 0x7f64b4108ac0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f64ac009ca0 tx=0x7f64ac02c0f0 comp rx=0 tx=0).stop 2024-04-21T23:15:55.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:55.509+0000 7f64bc30f640 1 -- 172.21.15.40:0/1923836582 shutdown_connections 2024-04-21T23:15:55.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:55.509+0000 7f64bc30f640 1 --2- 172.21.15.40:0/1923836582 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f64b4102230 0x7f64b4108ac0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:55.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:55.509+0000 7f64bc30f640 1 --2- 172.21.15.40:0/1923836582 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f64b4101870 0x7f64b4101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:55.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:55.509+0000 7f64bc30f640 1 --2- 172.21.15.40:0/1923836582 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64b4100680 0x7f64b4100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:55.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:55.509+0000 7f64bc30f640 1 -- 172.21.15.40:0/1923836582 >> 172.21.15.40:0/1923836582 conn(0x7f64b40fbbf0 msgr2=0x7f64b40fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:55.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:55.509+0000 7f64bc30f640 1 -- 172.21.15.40:0/1923836582 shutdown_connections 2024-04-21T23:15:55.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:55.509+0000 7f64bc30f640 1 -- 172.21.15.40:0/1923836582 wait complete. 2024-04-21T23:15:55.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:55.509+0000 7f64bc30f640 1 Processor -- start 2024-04-21T23:15:55.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:55.509+0000 7f64bc30f640 1 -- start start 2024-04-21T23:15:55.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:55.510+0000 7f64bc30f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f64b4100680 0x7f64b4197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:55.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:55.510+0000 7f64bc30f640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64b4101870 0x7f64b4198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:55.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:55.510+0000 7f64bc30f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f64b4102230 0x7f64b419f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:55.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:55.510+0000 7f64bc30f640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f64b410c360 con 0x7f64b4100680 2024-04-21T23:15:55.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:55.510+0000 7f64bc30f640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f64b410c1e0 con 0x7f64b4101870 2024-04-21T23:15:55.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:55.510+0000 7f64bc30f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f64b410c4e0 con 0x7f64b4102230 2024-04-21T23:15:55.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:55.510+0000 7f64ba0ab640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f64b4100680 0x7f64b4197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:55.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:55.510+0000 7f64ba0ab640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f64b4100680 0x7f64b4197af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46456/0 (socket says 172.21.15.40:46456) 2024-04-21T23:15:55.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:55.510+0000 7f64ba0ab640 1 -- 172.21.15.40:0/1796688418 learned_addr learned my addr 172.21.15.40:0/1796688418 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:15:55.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:55.510+0000 7f64b98aa640 1 --2- 172.21.15.40:0/1796688418 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64b4101870 0x7f64b4198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:55.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:55.510+0000 7f64ba8ac640 1 --2- 172.21.15.40:0/1796688418 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f64b4102230 0x7f64b419f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:55.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:55.510+0000 7f64ba0ab640 1 -- 172.21.15.40:0/1796688418 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f64b4102230 msgr2=0x7f64b419f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:55.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:55.510+0000 7f64ba0ab640 1 --2- 172.21.15.40:0/1796688418 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f64b4102230 0x7f64b419f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:55.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:55.510+0000 7f64ba0ab640 1 -- 172.21.15.40:0/1796688418 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64b4101870 msgr2=0x7f64b4198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:55.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:55.510+0000 7f64ba0ab640 1 --2- 172.21.15.40:0/1796688418 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64b4101870 0x7f64b4198030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:55.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:55.510+0000 7f64ba0ab640 1 -- 172.21.15.40:0/1796688418 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64b419f710 con 0x7f64b4100680 2024-04-21T23:15:55.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:55.510+0000 7f64ba8ac640 1 --2- 172.21.15.40:0/1796688418 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f64b4102230 0x7f64b419f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:15:55.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:55.510+0000 7f64ba0ab640 1 --2- 172.21.15.40:0/1796688418 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f64b4100680 0x7f64b4197af0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f64b000e9c0 tx=0x7f64b000edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:55.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:55.510+0000 7f64ab7fe640 1 -- 172.21.15.40:0/1796688418 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64b000ccc0 con 0x7f64b4100680 2024-04-21T23:15:55.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:55.510+0000 7f64ab7fe640 1 -- 172.21.15.40:0/1796688418 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f64b00035f0 con 0x7f64b4100680 2024-04-21T23:15:55.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:55.510+0000 7f64ab7fe640 1 -- 172.21.15.40:0/1796688418 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64b000ce60 con 0x7f64b4100680 2024-04-21T23:15:55.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:55.510+0000 7f64bc30f640 1 -- 172.21.15.40:0/1796688418 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64b419fa00 con 0x7f64b4100680 2024-04-21T23:15:55.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:55.511+0000 7f64bc30f640 1 -- 172.21.15.40:0/1796688418 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64b419ffa0 con 0x7f64b4100680 2024-04-21T23:15:55.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:55.511+0000 7f64bc30f640 1 -- 172.21.15.40:0/1796688418 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64b41a0230 con 0x7f64b4100680 2024-04-21T23:15:55.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:55.513+0000 7f64ab7fe640 1 -- 172.21.15.40:0/1796688418 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f64b00037a0 con 0x7f64b4100680 2024-04-21T23:15:55.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:55.513+0000 7f64ab7fe640 1 --2- 172.21.15.40:0/1796688418 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f64a00a2c10 0x7f64a00a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:55.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:55.513+0000 7f64b98aa640 1 --2- 172.21.15.40:0/1796688418 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f64a00a2c10 0x7f64a00a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:55.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:55.513+0000 7f64ab7fe640 1 -- 172.21.15.40:0/1796688418 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7879+0+0 (secure 0 0 0) 0x7f64b00cf340 con 0x7f64b4100680 2024-04-21T23:15:55.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:55.514+0000 7f64b98aa640 1 --2- 172.21.15.40:0/1796688418 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f64a00a2c10 0x7f64a00a50d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f64b4199000 tx=0x7f64a4009110 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:55.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:55.517+0000 7f64ab7fe640 1 -- 172.21.15.40:0/1796688418 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f64b009c450 con 0x7f64b4100680 2024-04-21T23:15:55.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:55 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/2193148310' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-21T23:15:55.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:55 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/2193148310' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-21T23:15:55.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:55 smithi040 ceph-mon[27984]: osdmap e83: 12 total, 12 up, 12 in 2024-04-21T23:15:55.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:55 smithi040 ceph-mon[27984]: pgmap v285: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:55.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:55 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:55.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:55 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:55.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:55 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:55.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:55 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:55.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:55 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:55.683 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:55 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:55.683 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:55 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:15:55.683 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:55 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:15:55.683 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:55 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:55.683 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:55 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/2855732045' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-21T23:15:55.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:55 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/2193148310' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-21T23:15:55.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:55 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/2193148310' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-21T23:15:55.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:55 smithi174 ceph-mon[31409]: osdmap e83: 12 total, 12 up, 12 in 2024-04-21T23:15:55.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:55 smithi174 ceph-mon[31409]: pgmap v285: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:55.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:55 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:55.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:55 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:55.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:55 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:55.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:55 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:55.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:55 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:55.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:55 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:55.695 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:55 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:15:55.695 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:55 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:15:55.695 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:55 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:55.695 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:55 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/2855732045' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-21T23:15:55.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:55 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/2193148310' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-21T23:15:55.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:55 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/2193148310' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-21T23:15:55.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:55 smithi111 ceph-mon[31085]: osdmap e83: 12 total, 12 up, 12 in 2024-04-21T23:15:55.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:55 smithi111 ceph-mon[31085]: pgmap v285: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:55.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:55 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:55.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:55 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:55.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:55 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:55.696 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:55 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:55.696 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:55 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:55.696 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:55 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:55.696 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:55 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:15:55.696 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:55 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:15:55.696 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:55 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:15:55.696 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:55 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/2855732045' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-21T23:15:55.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:55.740+0000 7f64bc30f640 1 -- 172.21.15.40:0/1796688418 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f64b41a0a10 con 0x7f64b4100680 2024-04-21T23:15:56.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:56.381+0000 7f64ab7fe640 1 -- 172.21.15.40:0/1796688418 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7f64b009bd50 con 0x7f64b4100680 2024-04-21T23:15:56.381 INFO:teuthology.orchestra.run.smithi040.stderr:new fs with metadata pool 2 and data pool 3 2024-04-21T23:15:56.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:56.384+0000 7f64bc30f640 1 -- 172.21.15.40:0/1796688418 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f64a00a2c10 msgr2=0x7f64a00a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:56.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:56.384+0000 7f64bc30f640 1 --2- 172.21.15.40:0/1796688418 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f64a00a2c10 0x7f64a00a50d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f64b4199000 tx=0x7f64a4009110 comp rx=0 tx=0).stop 2024-04-21T23:15:56.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:56.384+0000 7f64bc30f640 1 -- 172.21.15.40:0/1796688418 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f64b4100680 msgr2=0x7f64b4197af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:56.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:56.384+0000 7f64bc30f640 1 --2- 172.21.15.40:0/1796688418 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f64b4100680 0x7f64b4197af0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f64b000e9c0 tx=0x7f64b000edc0 comp rx=0 tx=0).stop 2024-04-21T23:15:56.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:56.384+0000 7f64bc30f640 1 -- 172.21.15.40:0/1796688418 shutdown_connections 2024-04-21T23:15:56.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:56.384+0000 7f64bc30f640 1 --2- 172.21.15.40:0/1796688418 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f64a00a2c10 0x7f64a00a50d0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:56.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:56.384+0000 7f64bc30f640 1 --2- 172.21.15.40:0/1796688418 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64b4101870 0x7f64b4198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:56.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:56.384+0000 7f64bc30f640 1 --2- 172.21.15.40:0/1796688418 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f64b4102230 0x7f64b419f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:56.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:56.384+0000 7f64bc30f640 1 --2- 172.21.15.40:0/1796688418 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f64b4100680 0x7f64b4197af0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:56.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:56.384+0000 7f64bc30f640 1 -- 172.21.15.40:0/1796688418 >> 172.21.15.40:0/1796688418 conn(0x7f64b40fbbf0 msgr2=0x7f64b40fd680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:56.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:56.384+0000 7f64bc30f640 1 -- 172.21.15.40:0/1796688418 shutdown_connections 2024-04-21T23:15:56.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:56.384+0000 7f64bc30f640 1 -- 172.21.15.40:0/1796688418 wait complete. 2024-04-21T23:15:56.399 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-04-21T23:15:56.399 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-04-21T23:15:56.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:56.510+0000 7ff00c9e7640 1 -- 172.21.15.40:0/3907342239 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff008103350 msgr2=0x7ff008109be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:56.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:56.510+0000 7ff00c9e7640 1 --2- 172.21.15.40:0/3907342239 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff008103350 0x7ff008109be0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7ff000009ca0 tx=0x7ff00002c0f0 comp rx=0 tx=0).stop 2024-04-21T23:15:56.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:56.510+0000 7ff00c9e7640 1 -- 172.21.15.40:0/3907342239 shutdown_connections 2024-04-21T23:15:56.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:56.510+0000 7ff00c9e7640 1 --2- 172.21.15.40:0/3907342239 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff008103350 0x7ff008109be0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:56.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:56.510+0000 7ff00c9e7640 1 --2- 172.21.15.40:0/3907342239 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff008102990 0x7ff008102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:56.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:56.510+0000 7ff00c9e7640 1 --2- 172.21.15.40:0/3907342239 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0081017a0 0x7ff008101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:56.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:56.510+0000 7ff00c9e7640 1 -- 172.21.15.40:0/3907342239 >> 172.21.15.40:0/3907342239 conn(0x7ff0080fcd30 msgr2=0x7ff0080ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:56.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:56.510+0000 7ff00c9e7640 1 -- 172.21.15.40:0/3907342239 shutdown_connections 2024-04-21T23:15:56.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:56.510+0000 7ff00c9e7640 1 -- 172.21.15.40:0/3907342239 wait complete. 2024-04-21T23:15:56.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:56.511+0000 7ff00c9e7640 1 Processor -- start 2024-04-21T23:15:56.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:56.511+0000 7ff00c9e7640 1 -- start start 2024-04-21T23:15:56.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:56.511+0000 7ff00c9e7640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0081017a0 0x7ff0081979b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:56.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:56.511+0000 7ff00c9e7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff008102990 0x7ff008197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:56.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:56.511+0000 7ff00c9e7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff008103350 0x7ff00819ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:56.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:56.511+0000 7ff00c9e7640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff00810d420 con 0x7ff008102990 2024-04-21T23:15:56.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:56.511+0000 7ff00c9e7640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff00810d2a0 con 0x7ff0081017a0 2024-04-21T23:15:56.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:56.511+0000 7ff00c9e7640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff00810d5a0 con 0x7ff008103350 2024-04-21T23:15:56.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:56.511+0000 7feffffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff008102990 0x7ff008197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:56.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:56.511+0000 7feffffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff008102990 0x7ff008197ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46478/0 (socket says 172.21.15.40:46478) 2024-04-21T23:15:56.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:56.511+0000 7feffffff640 1 -- 172.21.15.40:0/3013856374 learned_addr learned my addr 172.21.15.40:0/3013856374 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:15:56.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:56.511+0000 7ff006d9d640 1 --2- 172.21.15.40:0/3013856374 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff008103350 0x7ff00819ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:56.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:56.511+0000 7ff00659c640 1 --2- 172.21.15.40:0/3013856374 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0081017a0 0x7ff0081979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:56.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:56.511+0000 7feffffff640 1 -- 172.21.15.40:0/3013856374 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff008103350 msgr2=0x7ff00819ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:56.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:56.511+0000 7feffffff640 1 --2- 172.21.15.40:0/3013856374 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff008103350 0x7ff00819ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:56.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:56.511+0000 7feffffff640 1 -- 172.21.15.40:0/3013856374 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0081017a0 msgr2=0x7ff0081979b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:56.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:56.511+0000 7feffffff640 1 --2- 172.21.15.40:0/3013856374 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0081017a0 0x7ff0081979b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:56.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:56.511+0000 7feffffff640 1 -- 172.21.15.40:0/3013856374 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff00819f5d0 con 0x7ff008102990 2024-04-21T23:15:56.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:56.511+0000 7feffffff640 1 --2- 172.21.15.40:0/3013856374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff008102990 0x7ff008197ef0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7feff000d770 tx=0x7feff000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:56.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:56.512+0000 7fefff7fe640 1 -- 172.21.15.40:0/3013856374 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feff0017070 con 0x7ff008102990 2024-04-21T23:15:56.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:56.512+0000 7fefff7fe640 1 -- 172.21.15.40:0/3013856374 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feff0005500 con 0x7ff008102990 2024-04-21T23:15:56.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:56.512+0000 7ff00c9e7640 1 -- 172.21.15.40:0/3013856374 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff00819f8c0 con 0x7ff008102990 2024-04-21T23:15:56.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:56.512+0000 7fefff7fe640 1 -- 172.21.15.40:0/3013856374 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feff0004e00 con 0x7ff008102990 2024-04-21T23:15:56.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:56.512+0000 7ff00c9e7640 1 -- 172.21.15.40:0/3013856374 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff00819fdc0 con 0x7ff008102990 2024-04-21T23:15:56.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:56.513+0000 7ff00c9e7640 1 -- 172.21.15.40:0/3013856374 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0081089f0 con 0x7ff008102990 2024-04-21T23:15:56.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:56.514+0000 7fefff7fe640 1 -- 172.21.15.40:0/3013856374 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7feff0004fa0 con 0x7ff008102990 2024-04-21T23:15:56.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:56.514+0000 7fefff7fe640 1 --2- 172.21.15.40:0/3013856374 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7feff40a2b90 0x7feff40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:56.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:56.514+0000 7fefff7fe640 1 -- 172.21.15.40:0/3013856374 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7feff00cecf0 con 0x7ff008102990 2024-04-21T23:15:56.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:56.515+0000 7ff00659c640 1 --2- 172.21.15.40:0/3013856374 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7feff40a2b90 0x7feff40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:56.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:56.515+0000 7ff00659c640 1 --2- 172.21.15.40:0/3013856374 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7feff40a2b90 0x7feff40a5050 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff0081027f0 tx=0x7feff8005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:56.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:56.518+0000 7fefff7fe640 1 -- 172.21.15.40:0/3013856374 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7feff009b750 con 0x7ff008102990 2024-04-21T23:15:56.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:56 smithi040 ceph-mon[27984]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-21T23:15:56.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:56 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/2855732045' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-04-21T23:15:56.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:56 smithi040 ceph-mon[27984]: osdmap e84: 12 total, 12 up, 12 in 2024-04-21T23:15:56.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:56 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/1796688418' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-21T23:15:56.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:56 smithi040 ceph-mon[27984]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-21T23:15:56.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:56 smithi174 ceph-mon[31409]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-21T23:15:56.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:56 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/2855732045' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-04-21T23:15:56.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:56 smithi174 ceph-mon[31409]: osdmap e84: 12 total, 12 up, 12 in 2024-04-21T23:15:56.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:56 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/1796688418' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-21T23:15:56.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:56 smithi174 ceph-mon[31409]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-21T23:15:56.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:56 smithi111 ceph-mon[31085]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-21T23:15:56.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:56 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/2855732045' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-04-21T23:15:56.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:56 smithi111 ceph-mon[31085]: osdmap e84: 12 total, 12 up, 12 in 2024-04-21T23:15:56.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:56 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/1796688418' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-21T23:15:56.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:56 smithi111 ceph-mon[31085]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-21T23:15:56.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:56.711+0000 7ff00c9e7640 1 -- 172.21.15.40:0/3013856374 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7ff00802d320 con 0x7ff008102990 2024-04-21T23:15:57.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:57.328+0000 7fefff7fe640 1 -- 172.21.15.40:0/3013856374 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v86) ==== 157+0+0 (secure 0 0 0) 0x7feff009fa50 con 0x7ff008102990 2024-04-21T23:15:57.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:57.330+0000 7ff00c9e7640 1 -- 172.21.15.40:0/3013856374 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7feff40a2b90 msgr2=0x7feff40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:57.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:57.330+0000 7ff00c9e7640 1 --2- 172.21.15.40:0/3013856374 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7feff40a2b90 0x7feff40a5050 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff0081027f0 tx=0x7feff8005f90 comp rx=0 tx=0).stop 2024-04-21T23:15:57.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:57.330+0000 7ff00c9e7640 1 -- 172.21.15.40:0/3013856374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff008102990 msgr2=0x7ff008197ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:57.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:57.330+0000 7ff00c9e7640 1 --2- 172.21.15.40:0/3013856374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff008102990 0x7ff008197ef0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7feff000d770 tx=0x7feff000dc30 comp rx=0 tx=0).stop 2024-04-21T23:15:57.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:57.330+0000 7ff00c9e7640 1 -- 172.21.15.40:0/3013856374 shutdown_connections 2024-04-21T23:15:57.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:57.330+0000 7ff00c9e7640 1 --2- 172.21.15.40:0/3013856374 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7feff40a2b90 0x7feff40a5050 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:57.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:57.330+0000 7ff00c9e7640 1 --2- 172.21.15.40:0/3013856374 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff008103350 0x7ff00819ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:57.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:57.330+0000 7ff00c9e7640 1 --2- 172.21.15.40:0/3013856374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff008102990 0x7ff008197ef0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:57.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:57.330+0000 7ff00c9e7640 1 --2- 172.21.15.40:0/3013856374 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0081017a0 0x7ff0081979b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:57.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:57.330+0000 7ff00c9e7640 1 -- 172.21.15.40:0/3013856374 >> 172.21.15.40:0/3013856374 conn(0x7ff0080fcd30 msgr2=0x7ff008105bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:57.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:57.331+0000 7ff00c9e7640 1 -- 172.21.15.40:0/3013856374 shutdown_connections 2024-04-21T23:15:57.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:57.331+0000 7ff00c9e7640 1 -- 172.21.15.40:0/3013856374 wait complete. 2024-04-21T23:15:57.345 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-04-21T23:15:57.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:57.463+0000 7fda43c15640 1 -- 172.21.15.40:0/272969521 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda3c101880 msgr2=0x7fda3c101d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:57.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:57.463+0000 7fda43c15640 1 --2- 172.21.15.40:0/272969521 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda3c101880 0x7fda3c101d00 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fda2c009900 tx=0x7fda2c02bc30 comp rx=0 tx=0).stop 2024-04-21T23:15:57.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:57.463+0000 7fda43c15640 1 -- 172.21.15.40:0/272969521 shutdown_connections 2024-04-21T23:15:57.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:57.463+0000 7fda43c15640 1 --2- 172.21.15.40:0/272969521 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fda3c102240 0x7fda3c108ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:57.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:57.463+0000 7fda43c15640 1 --2- 172.21.15.40:0/272969521 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda3c101880 0x7fda3c101d00 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:57.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:57.463+0000 7fda43c15640 1 --2- 172.21.15.40:0/272969521 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fda3c100690 0x7fda3c100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:57.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:57.463+0000 7fda43c15640 1 -- 172.21.15.40:0/272969521 >> 172.21.15.40:0/272969521 conn(0x7fda3c0fbc20 msgr2=0x7fda3c0fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:57.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:57.463+0000 7fda43c15640 1 -- 172.21.15.40:0/272969521 shutdown_connections 2024-04-21T23:15:57.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:57.463+0000 7fda43c15640 1 -- 172.21.15.40:0/272969521 wait complete. 2024-04-21T23:15:57.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:57.463+0000 7fda43c15640 1 Processor -- start 2024-04-21T23:15:57.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:57.463+0000 7fda43c15640 1 -- start start 2024-04-21T23:15:57.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:57.464+0000 7fda43c15640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fda3c100690 0x7fda3c197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:57.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:57.464+0000 7fda43c15640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda3c101880 0x7fda3c1980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:57.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:57.464+0000 7fda43c15640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fda3c102240 0x7fda3c19f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:57.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:57.464+0000 7fda43c15640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fda3c10c370 con 0x7fda3c101880 2024-04-21T23:15:57.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:57.464+0000 7fda43c15640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fda3c10c1f0 con 0x7fda3c102240 2024-04-21T23:15:57.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:57.464+0000 7fda43c15640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fda3c10c4f0 con 0x7fda3c100690 2024-04-21T23:15:57.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:57.464+0000 7fda411b0640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda3c101880 0x7fda3c1980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:57.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:57.464+0000 7fda411b0640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda3c101880 0x7fda3c1980b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46486/0 (socket says 172.21.15.40:46486) 2024-04-21T23:15:57.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:57.464+0000 7fda411b0640 1 -- 172.21.15.40:0/137827637 learned_addr learned my addr 172.21.15.40:0/137827637 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:15:57.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:57.464+0000 7fda419b1640 1 --2- 172.21.15.40:0/137827637 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fda3c100690 0x7fda3c197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:57.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:57.464+0000 7fda421b2640 1 --2- 172.21.15.40:0/137827637 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fda3c102240 0x7fda3c19f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:57.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:57.464+0000 7fda411b0640 1 -- 172.21.15.40:0/137827637 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fda3c100690 msgr2=0x7fda3c197b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:57.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:57.464+0000 7fda411b0640 1 --2- 172.21.15.40:0/137827637 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fda3c100690 0x7fda3c197b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:57.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:57.464+0000 7fda411b0640 1 -- 172.21.15.40:0/137827637 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fda3c102240 msgr2=0x7fda3c19f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:57.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:57.464+0000 7fda411b0640 1 --2- 172.21.15.40:0/137827637 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fda3c102240 0x7fda3c19f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:57.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:57.464+0000 7fda411b0640 1 -- 172.21.15.40:0/137827637 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda3c19f740 con 0x7fda3c101880 2024-04-21T23:15:57.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:57.464+0000 7fda421b2640 1 --2- 172.21.15.40:0/137827637 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fda3c102240 0x7fda3c19f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:15:57.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:57.464+0000 7fda411b0640 1 --2- 172.21.15.40:0/137827637 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda3c101880 0x7fda3c1980b0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fda2c02c8e0 tx=0x7fda2c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:57.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:57.464+0000 7fda32ffd640 1 -- 172.21.15.40:0/137827637 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda2c041070 con 0x7fda3c101880 2024-04-21T23:15:57.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:57.464+0000 7fda32ffd640 1 -- 172.21.15.40:0/137827637 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda2c0054e0 con 0x7fda3c101880 2024-04-21T23:15:57.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:57.464+0000 7fda43c15640 1 -- 172.21.15.40:0/137827637 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda3c19f9d0 con 0x7fda3c101880 2024-04-21T23:15:57.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:57.464+0000 7fda32ffd640 1 -- 172.21.15.40:0/137827637 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda2c0346a0 con 0x7fda3c101880 2024-04-21T23:15:57.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:57.465+0000 7fda43c15640 1 -- 172.21.15.40:0/137827637 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda3c19fee0 con 0x7fda3c101880 2024-04-21T23:15:57.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:57.465+0000 7fda43c15640 1 -- 172.21.15.40:0/137827637 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda3c1a01d0 con 0x7fda3c101880 2024-04-21T23:15:57.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:57.467+0000 7fda32ffd640 1 -- 172.21.15.40:0/137827637 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fda2c044070 con 0x7fda3c101880 2024-04-21T23:15:57.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:57.467+0000 7fda32ffd640 1 --2- 172.21.15.40:0/137827637 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fda280a2c10 0x7fda280a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:57.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:57.467+0000 7fda419b1640 1 --2- 172.21.15.40:0/137827637 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fda280a2c10 0x7fda280a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:57.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:57.467+0000 7fda32ffd640 1 -- 172.21.15.40:0/137827637 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8277+0+0 (secure 0 0 0) 0x7fda2c0f0ea0 con 0x7fda3c101880 2024-04-21T23:15:57.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:57.468+0000 7fda419b1640 1 --2- 172.21.15.40:0/137827637 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fda280a2c10 0x7fda280a50d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fda380058c0 tx=0x7fda38005850 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:57.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:57.472+0000 7fda32ffd640 1 -- 172.21.15.40:0/137827637 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fda2c0bddd0 con 0x7fda3c101880 2024-04-21T23:15:57.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:57.658+0000 7fda43c15640 1 -- 172.21.15.40:0/137827637 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fda3c1a08c0 con 0x7fda3c101880 2024-04-21T23:15:57.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:57 smithi040 ceph-mon[27984]: osdmap e85: 12 total, 12 up, 12 in 2024-04-21T23:15:57.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:57 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/1796688418' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-21T23:15:57.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:57 smithi040 ceph-mon[27984]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-21T23:15:57.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:57 smithi040 ceph-mon[27984]: pgmap v288: 97 pgs: 31 creating+peering, 61 unknown, 5 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:57.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:57 smithi040 ceph-mon[27984]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-21T23:15:57.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:57 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/3013856374' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-21T23:15:57.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:57 smithi174 ceph-mon[31409]: osdmap e85: 12 total, 12 up, 12 in 2024-04-21T23:15:57.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:57 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/1796688418' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-21T23:15:57.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:57 smithi174 ceph-mon[31409]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-21T23:15:57.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:57 smithi174 ceph-mon[31409]: pgmap v288: 97 pgs: 31 creating+peering, 61 unknown, 5 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:57.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:57 smithi174 ceph-mon[31409]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-21T23:15:57.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:57 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/3013856374' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-21T23:15:57.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:57 smithi111 ceph-mon[31085]: osdmap e85: 12 total, 12 up, 12 in 2024-04-21T23:15:57.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:57 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/1796688418' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-21T23:15:57.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:57 smithi111 ceph-mon[31085]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-21T23:15:57.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:57 smithi111 ceph-mon[31085]: pgmap v288: 97 pgs: 31 creating+peering, 61 unknown, 5 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:15:57.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:57 smithi111 ceph-mon[31085]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-21T23:15:57.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:57 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/3013856374' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-21T23:15:58.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:58 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/3013856374' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-04-21T23:15:58.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:58 smithi040 ceph-mon[27984]: osdmap e86: 12 total, 12 up, 12 in 2024-04-21T23:15:58.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:58 smithi040 ceph-mon[27984]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-21T23:15:58.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:58 smithi040 ceph-mon[27984]: Cluster is now healthy 2024-04-21T23:15:58.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:58 smithi040 ceph-mon[27984]: mds.? [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] up:active 2024-04-21T23:15:58.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:58 smithi040 ceph-mon[27984]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T23:15:58.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:58 smithi040 ceph-mon[27984]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T23:15:58.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:58 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/137827637' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-21T23:15:58.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:58 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/3013856374' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-04-21T23:15:58.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:58 smithi174 ceph-mon[31409]: osdmap e86: 12 total, 12 up, 12 in 2024-04-21T23:15:58.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:58 smithi174 ceph-mon[31409]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-21T23:15:58.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:58 smithi174 ceph-mon[31409]: Cluster is now healthy 2024-04-21T23:15:58.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:58 smithi174 ceph-mon[31409]: mds.? [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] up:active 2024-04-21T23:15:58.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:58 smithi174 ceph-mon[31409]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T23:15:58.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:58 smithi174 ceph-mon[31409]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T23:15:58.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:58 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/137827637' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-21T23:15:58.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:58 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/3013856374' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-04-21T23:15:58.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:58 smithi111 ceph-mon[31085]: osdmap e86: 12 total, 12 up, 12 in 2024-04-21T23:15:58.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:58 smithi111 ceph-mon[31085]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-21T23:15:58.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:58 smithi111 ceph-mon[31085]: Cluster is now healthy 2024-04-21T23:15:58.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:58 smithi111 ceph-mon[31085]: mds.? [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] up:active 2024-04-21T23:15:58.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:58 smithi111 ceph-mon[31085]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T23:15:58.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:58 smithi111 ceph-mon[31085]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T23:15:58.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:58 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/137827637' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-21T23:15:59.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:59.400+0000 7fda32ffd640 1 -- 172.21.15.40:0/137827637 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v88) ==== 228+0+0 (secure 0 0 0) 0x7fda2c0bd6d0 con 0x7fda3c101880 2024-04-21T23:15:59.400 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_data_ec' created 2024-04-21T23:15:59.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:59.403+0000 7fda43c15640 1 -- 172.21.15.40:0/137827637 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fda280a2c10 msgr2=0x7fda280a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:59.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:59.403+0000 7fda43c15640 1 --2- 172.21.15.40:0/137827637 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fda280a2c10 0x7fda280a50d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fda380058c0 tx=0x7fda38005850 comp rx=0 tx=0).stop 2024-04-21T23:15:59.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:59.403+0000 7fda43c15640 1 -- 172.21.15.40:0/137827637 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda3c101880 msgr2=0x7fda3c1980b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:59.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:59.403+0000 7fda43c15640 1 --2- 172.21.15.40:0/137827637 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda3c101880 0x7fda3c1980b0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fda2c02c8e0 tx=0x7fda2c004270 comp rx=0 tx=0).stop 2024-04-21T23:15:59.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:59.403+0000 7fda43c15640 1 -- 172.21.15.40:0/137827637 shutdown_connections 2024-04-21T23:15:59.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:59.403+0000 7fda43c15640 1 --2- 172.21.15.40:0/137827637 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fda280a2c10 0x7fda280a50d0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:59.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:59.403+0000 7fda43c15640 1 --2- 172.21.15.40:0/137827637 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fda3c102240 0x7fda3c19f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:59.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:59.403+0000 7fda43c15640 1 --2- 172.21.15.40:0/137827637 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda3c101880 0x7fda3c1980b0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:59.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:59.403+0000 7fda43c15640 1 --2- 172.21.15.40:0/137827637 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fda3c100690 0x7fda3c197b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:59.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:59.403+0000 7fda43c15640 1 -- 172.21.15.40:0/137827637 >> 172.21.15.40:0/137827637 conn(0x7fda3c0fbc20 msgr2=0x7fda3c0fd640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:59.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:59.404+0000 7fda43c15640 1 -- 172.21.15.40:0/137827637 shutdown_connections 2024-04-21T23:15:59.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:59.404+0000 7fda43c15640 1 -- 172.21.15.40:0/137827637 wait complete. 2024-04-21T23:15:59.421 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-04-21T23:15:59.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:59.534+0000 7f3f6324b640 1 -- 172.21.15.40:0/605935248 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f5c103330 msgr2=0x7f3f5c109bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:59.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:59.534+0000 7f3f6324b640 1 --2- 172.21.15.40:0/605935248 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f5c103330 0x7f3f5c109bc0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f3f54009cc0 tx=0x7f3f54030510 comp rx=0 tx=0).stop 2024-04-21T23:15:59.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:59.534+0000 7f3f6324b640 1 -- 172.21.15.40:0/605935248 shutdown_connections 2024-04-21T23:15:59.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:59.534+0000 7f3f6324b640 1 --2- 172.21.15.40:0/605935248 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f5c102970 0x7f3f5c102df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:59.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:59.534+0000 7f3f6324b640 1 --2- 172.21.15.40:0/605935248 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f5c103330 0x7f3f5c109bc0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:59.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:59.534+0000 7f3f6324b640 1 --2- 172.21.15.40:0/605935248 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f5c101780 0x7f3f5c101ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:59.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:59.534+0000 7f3f6324b640 1 -- 172.21.15.40:0/605935248 >> 172.21.15.40:0/605935248 conn(0x7f3f5c0fccf0 msgr2=0x7f3f5c0ff130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:59.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:59.534+0000 7f3f6324b640 1 -- 172.21.15.40:0/605935248 shutdown_connections 2024-04-21T23:15:59.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:59.535+0000 7f3f6324b640 1 -- 172.21.15.40:0/605935248 wait complete. 2024-04-21T23:15:59.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:59.535+0000 7f3f6324b640 1 Processor -- start 2024-04-21T23:15:59.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:59.535+0000 7f3f6324b640 1 -- start start 2024-04-21T23:15:59.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:59.535+0000 7f3f6324b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f5c101780 0x7f3f5c195800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:59.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:59.535+0000 7f3f6324b640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f5c102970 0x7f3f5c195d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:59.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:59.535+0000 7f3f6324b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f5c103330 0x7f3f5c19cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:59.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:59.535+0000 7f3f6324b640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3f5c10b220 con 0x7f3f5c101780 2024-04-21T23:15:59.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:59.535+0000 7f3f6324b640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3f5c10b0a0 con 0x7f3f5c102970 2024-04-21T23:15:59.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:59.535+0000 7f3f6324b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3f5c10b3a0 con 0x7f3f5c103330 2024-04-21T23:15:59.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:59.535+0000 7f3f60fe7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f5c101780 0x7f3f5c195800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:59.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:59.535+0000 7f3f60fe7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f5c101780 0x7f3f5c195800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46496/0 (socket says 172.21.15.40:46496) 2024-04-21T23:15:59.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:59.535+0000 7f3f60fe7640 1 -- 172.21.15.40:0/1633666901 learned_addr learned my addr 172.21.15.40:0/1633666901 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:15:59.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:59.535+0000 7f3f617e8640 1 --2- 172.21.15.40:0/1633666901 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f5c103330 0x7f3f5c19cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:59.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:59.535+0000 7f3f53fff640 1 --2- 172.21.15.40:0/1633666901 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f5c102970 0x7f3f5c195d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:59.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:59.536+0000 7f3f60fe7640 1 -- 172.21.15.40:0/1633666901 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f5c103330 msgr2=0x7f3f5c19cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:59.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:59.536+0000 7f3f60fe7640 1 --2- 172.21.15.40:0/1633666901 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f5c103330 0x7f3f5c19cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:59.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:59.536+0000 7f3f60fe7640 1 -- 172.21.15.40:0/1633666901 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f5c102970 msgr2=0x7f3f5c195d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:59.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:59.536+0000 7f3f60fe7640 1 --2- 172.21.15.40:0/1633666901 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f5c102970 0x7f3f5c195d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:59.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:59.536+0000 7f3f60fe7640 1 -- 172.21.15.40:0/1633666901 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f5c19d420 con 0x7f3f5c101780 2024-04-21T23:15:59.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:59.536+0000 7f3f60fe7640 1 --2- 172.21.15.40:0/1633666901 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f5c101780 0x7f3f5c195800 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f3f5800d950 tx=0x7f3f5800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:59.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:59.536+0000 7f3f51ffb640 1 -- 172.21.15.40:0/1633666901 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f58017070 con 0x7f3f5c101780 2024-04-21T23:15:59.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:59.536+0000 7f3f51ffb640 1 -- 172.21.15.40:0/1633666901 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f58009c30 con 0x7f3f5c101780 2024-04-21T23:15:59.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:59.536+0000 7f3f6324b640 1 -- 172.21.15.40:0/1633666901 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f5c19d710 con 0x7f3f5c101780 2024-04-21T23:15:59.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:59.536+0000 7f3f51ffb640 1 -- 172.21.15.40:0/1633666901 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f5800f600 con 0x7f3f5c101780 2024-04-21T23:15:59.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:59.536+0000 7f3f6324b640 1 -- 172.21.15.40:0/1633666901 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f5c19dc60 con 0x7f3f5c101780 2024-04-21T23:15:59.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:59.537+0000 7f3f6324b640 1 -- 172.21.15.40:0/1633666901 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f5c108b10 con 0x7f3f5c101780 2024-04-21T23:15:59.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:59.539+0000 7f3f51ffb640 1 -- 172.21.15.40:0/1633666901 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f3f5801a070 con 0x7f3f5c101780 2024-04-21T23:15:59.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:59.539+0000 7f3f51ffb640 1 --2- 172.21.15.40:0/1633666901 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f3f440a2c10 0x7f3f440a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:59.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:59.539+0000 7f3f51ffb640 1 -- 172.21.15.40:0/1633666901 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8770+0+0 (secure 0 0 0) 0x7f3f580ceba0 con 0x7f3f5c101780 2024-04-21T23:15:59.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:59.539+0000 7f3f53fff640 1 --2- 172.21.15.40:0/1633666901 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f3f440a2c10 0x7f3f440a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:59.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:59.540+0000 7f3f53fff640 1 --2- 172.21.15.40:0/1633666901 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f3f440a2c10 0x7f3f440a50d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f3f5c196d10 tx=0x7f3f4c00b390 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:59.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:59.543+0000 7f3f51ffb640 1 -- 172.21.15.40:0/1633666901 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f3f58009de0 con 0x7f3f5c101780 2024-04-21T23:15:59.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:59 smithi040 ceph-mon[27984]: pgmap v290: 97 pgs: 31 creating+peering, 38 unknown, 28 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 509 B/s wr, 5 op/s 2024-04-21T23:15:59.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:15:59 smithi040 ceph-mon[27984]: osdmap e87: 12 total, 12 up, 12 in 2024-04-21T23:15:59.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:59 smithi174 ceph-mon[31409]: pgmap v290: 97 pgs: 31 creating+peering, 38 unknown, 28 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 509 B/s wr, 5 op/s 2024-04-21T23:15:59.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:15:59 smithi174 ceph-mon[31409]: osdmap e87: 12 total, 12 up, 12 in 2024-04-21T23:15:59.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:59 smithi111 ceph-mon[31085]: pgmap v290: 97 pgs: 31 creating+peering, 38 unknown, 28 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 509 B/s wr, 5 op/s 2024-04-21T23:15:59.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:15:59 smithi111 ceph-mon[31085]: osdmap e87: 12 total, 12 up, 12 in 2024-04-21T23:15:59.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:15:59.730+0000 7f3f6324b640 1 -- 172.21.15.40:0/1633666901 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7f3f5c02d170 con 0x7f3f5c101780 2024-04-21T23:16:00.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:00.408+0000 7f3f51ffb640 1 -- 172.21.15.40:0/1633666901 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v89) ==== 169+0+0 (secure 0 0 0) 0x7f3f5809b310 con 0x7f3f5c101780 2024-04-21T23:16:00.407 INFO:teuthology.orchestra.run.smithi040.stderr:set pool 4 allow_ec_overwrites to true 2024-04-21T23:16:00.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:00.411+0000 7f3f6324b640 1 -- 172.21.15.40:0/1633666901 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f3f440a2c10 msgr2=0x7f3f440a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:00.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:00.411+0000 7f3f6324b640 1 --2- 172.21.15.40:0/1633666901 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f3f440a2c10 0x7f3f440a50d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f3f5c196d10 tx=0x7f3f4c00b390 comp rx=0 tx=0).stop 2024-04-21T23:16:00.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:00.411+0000 7f3f6324b640 1 -- 172.21.15.40:0/1633666901 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f5c101780 msgr2=0x7f3f5c195800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:00.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:00.411+0000 7f3f6324b640 1 --2- 172.21.15.40:0/1633666901 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f5c101780 0x7f3f5c195800 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f3f5800d950 tx=0x7f3f5800de10 comp rx=0 tx=0).stop 2024-04-21T23:16:00.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:00.411+0000 7f3f6324b640 1 -- 172.21.15.40:0/1633666901 shutdown_connections 2024-04-21T23:16:00.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:00.411+0000 7f3f6324b640 1 --2- 172.21.15.40:0/1633666901 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f3f440a2c10 0x7f3f440a50d0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:00.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:00.411+0000 7f3f6324b640 1 --2- 172.21.15.40:0/1633666901 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f5c102970 0x7f3f5c195d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:00.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:00.411+0000 7f3f6324b640 1 --2- 172.21.15.40:0/1633666901 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f5c103330 0x7f3f5c19cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:00.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:00.411+0000 7f3f6324b640 1 --2- 172.21.15.40:0/1633666901 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f5c101780 0x7f3f5c195800 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:00.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:00.411+0000 7f3f6324b640 1 -- 172.21.15.40:0/1633666901 >> 172.21.15.40:0/1633666901 conn(0x7f3f5c0fccf0 msgr2=0x7f3f5c105ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:00.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:00.411+0000 7f3f6324b640 1 -- 172.21.15.40:0/1633666901 shutdown_connections 2024-04-21T23:16:00.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:00.411+0000 7f3f6324b640 1 -- 172.21.15.40:0/1633666901 wait complete. 2024-04-21T23:16:00.431 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-04-21T23:16:00.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:00.545+0000 7fa5e011e640 1 -- 172.21.15.40:0/607065688 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5d8102250 msgr2=0x7fa5d8108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:00.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:00.545+0000 7fa5e011e640 1 --2- 172.21.15.40:0/607065688 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5d8102250 0x7fa5d8108ae0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fa5d0009ca0 tx=0x7fa5d002c0f0 comp rx=0 tx=0).stop 2024-04-21T23:16:00.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:00.545+0000 7fa5e011e640 1 -- 172.21.15.40:0/607065688 shutdown_connections 2024-04-21T23:16:00.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:00.545+0000 7fa5e011e640 1 --2- 172.21.15.40:0/607065688 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5d8101890 0x7fa5d8101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:00.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:00.545+0000 7fa5e011e640 1 --2- 172.21.15.40:0/607065688 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5d8102250 0x7fa5d8108ae0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:00.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:00.545+0000 7fa5e011e640 1 --2- 172.21.15.40:0/607065688 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa5d81006a0 0x7fa5d8100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:00.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:00.545+0000 7fa5e011e640 1 -- 172.21.15.40:0/607065688 >> 172.21.15.40:0/607065688 conn(0x7fa5d80fbc30 msgr2=0x7fa5d80fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:00.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:00.545+0000 7fa5e011e640 1 -- 172.21.15.40:0/607065688 shutdown_connections 2024-04-21T23:16:00.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:00.545+0000 7fa5e011e640 1 -- 172.21.15.40:0/607065688 wait complete. 2024-04-21T23:16:00.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:00.545+0000 7fa5e011e640 1 Processor -- start 2024-04-21T23:16:00.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:00.545+0000 7fa5e011e640 1 -- start start 2024-04-21T23:16:00.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:00.546+0000 7fa5e011e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa5d81006a0 0x7fa5d8197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:00.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:00.546+0000 7fa5e011e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5d8101890 0x7fa5d8197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:00.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:00.546+0000 7fa5e011e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5d8102250 0x7fa5d819f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:00.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:00.546+0000 7fa5e011e640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa5d810c320 con 0x7fa5d8102250 2024-04-21T23:16:00.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:00.546+0000 7fa5e011e640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fa5d810c1a0 con 0x7fa5d8101890 2024-04-21T23:16:00.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:00.546+0000 7fa5e011e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa5d810c4a0 con 0x7fa5d81006a0 2024-04-21T23:16:00.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:00.546+0000 7fa5de6bb640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5d8102250 0x7fa5d819f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:00.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:00.546+0000 7fa5de6bb640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5d8102250 0x7fa5d819f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46508/0 (socket says 172.21.15.40:46508) 2024-04-21T23:16:00.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:00.546+0000 7fa5de6bb640 1 -- 172.21.15.40:0/2127534840 learned_addr learned my addr 172.21.15.40:0/2127534840 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:16:00.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:00.546+0000 7fa5dd6b9640 1 --2- 172.21.15.40:0/2127534840 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5d8101890 0x7fa5d8197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:00.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:00.546+0000 7fa5ddeba640 1 --2- 172.21.15.40:0/2127534840 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa5d81006a0 0x7fa5d8197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:00.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:00.546+0000 7fa5de6bb640 1 -- 172.21.15.40:0/2127534840 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa5d81006a0 msgr2=0x7fa5d8197a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:00.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:00.546+0000 7fa5de6bb640 1 --2- 172.21.15.40:0/2127534840 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa5d81006a0 0x7fa5d8197a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:00.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:00.546+0000 7fa5de6bb640 1 -- 172.21.15.40:0/2127534840 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5d8101890 msgr2=0x7fa5d8197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:00.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:00.546+0000 7fa5de6bb640 1 --2- 172.21.15.40:0/2127534840 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5d8101890 0x7fa5d8197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:00.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:00.546+0000 7fa5de6bb640 1 -- 172.21.15.40:0/2127534840 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5d819f6a0 con 0x7fa5d8102250 2024-04-21T23:16:00.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:00.546+0000 7fa5de6bb640 1 --2- 172.21.15.40:0/2127534840 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5d8102250 0x7fa5d819f030 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fa5d002c700 tx=0x7fa5d0032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:00.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:00.546+0000 7fa5ceffd640 1 -- 172.21.15.40:0/2127534840 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5d0039040 con 0x7fa5d8102250 2024-04-21T23:16:00.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:00.546+0000 7fa5e011e640 1 -- 172.21.15.40:0/2127534840 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5d819f930 con 0x7fa5d8102250 2024-04-21T23:16:00.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:00.546+0000 7fa5ceffd640 1 -- 172.21.15.40:0/2127534840 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa5d0044070 con 0x7fa5d8102250 2024-04-21T23:16:00.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:00.546+0000 7fa5ceffd640 1 -- 172.21.15.40:0/2127534840 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5d00042a0 con 0x7fa5d8102250 2024-04-21T23:16:00.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:00.546+0000 7fa5e011e640 1 -- 172.21.15.40:0/2127534840 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5d819fe70 con 0x7fa5d8102250 2024-04-21T23:16:00.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:00.547+0000 7fa5e011e640 1 -- 172.21.15.40:0/2127534840 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5d81a00a0 con 0x7fa5d8102250 2024-04-21T23:16:00.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:00.549+0000 7fa5ceffd640 1 -- 172.21.15.40:0/2127534840 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fa5d0003970 con 0x7fa5d8102250 2024-04-21T23:16:00.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:00.549+0000 7fa5ceffd640 1 --2- 172.21.15.40:0/2127534840 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fa5c40a2b90 0x7fa5c40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:00.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:00.549+0000 7fa5ddeba640 1 --2- 172.21.15.40:0/2127534840 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fa5c40a2b90 0x7fa5c40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:00.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:00.549+0000 7fa5ceffd640 1 -- 172.21.15.40:0/2127534840 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8770+0+0 (secure 0 0 0) 0x7fa5d00f1db0 con 0x7fa5d8102250 2024-04-21T23:16:00.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:00.550+0000 7fa5ddeba640 1 --2- 172.21.15.40:0/2127534840 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fa5c40a2b90 0x7fa5c40a5050 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa5d40058c0 tx=0x7fa5d4005c80 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:00.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:00.553+0000 7fa5ceffd640 1 -- 172.21.15.40:0/2127534840 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fa5d00beb80 con 0x7fa5d8102250 2024-04-21T23:16:00.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:00 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/137827637' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-21T23:16:00.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:00 smithi040 ceph-mon[27984]: osdmap e88: 12 total, 12 up, 12 in 2024-04-21T23:16:00.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:00 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/1633666901' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-21T23:16:00.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:00 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/137827637' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-21T23:16:00.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:00 smithi174 ceph-mon[31409]: osdmap e88: 12 total, 12 up, 12 in 2024-04-21T23:16:00.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:00 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/1633666901' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-21T23:16:00.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:00 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/137827637' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-21T23:16:00.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:00 smithi111 ceph-mon[31085]: osdmap e88: 12 total, 12 up, 12 in 2024-04-21T23:16:00.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:00 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/1633666901' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-21T23:16:00.695 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 21 23:16:00 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-b[61419]: 2024-04-21T23:16:00.513+0000 7fccc5509700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-21T23:16:00.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:00.779+0000 7fa5e011e640 1 -- 172.21.15.40:0/2127534840 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7fa5d81a0270 con 0x7fa5d8102250 2024-04-21T23:16:01.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.410+0000 7fa5ceffd640 1 -- 172.21.15.40:0/2127534840 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) ==== 137+0+0 (secure 0 0 0) 0x7fa5d00be480 con 0x7fa5d8102250 2024-04-21T23:16:01.409 INFO:teuthology.orchestra.run.smithi040.stderr:added data pool 4 to fsmap 2024-04-21T23:16:01.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.413+0000 7fa5e011e640 1 -- 172.21.15.40:0/2127534840 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fa5c40a2b90 msgr2=0x7fa5c40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:01.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.413+0000 7fa5e011e640 1 --2- 172.21.15.40:0/2127534840 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fa5c40a2b90 0x7fa5c40a5050 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa5d40058c0 tx=0x7fa5d4005c80 comp rx=0 tx=0).stop 2024-04-21T23:16:01.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.413+0000 7fa5e011e640 1 -- 172.21.15.40:0/2127534840 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5d8102250 msgr2=0x7fa5d819f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:01.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.413+0000 7fa5e011e640 1 --2- 172.21.15.40:0/2127534840 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5d8102250 0x7fa5d819f030 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fa5d002c700 tx=0x7fa5d0032f20 comp rx=0 tx=0).stop 2024-04-21T23:16:01.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.413+0000 7fa5e011e640 1 -- 172.21.15.40:0/2127534840 shutdown_connections 2024-04-21T23:16:01.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.413+0000 7fa5e011e640 1 --2- 172.21.15.40:0/2127534840 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fa5c40a2b90 0x7fa5c40a5050 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:01.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.413+0000 7fa5e011e640 1 --2- 172.21.15.40:0/2127534840 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5d8101890 0x7fa5d8197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:01.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.413+0000 7fa5e011e640 1 --2- 172.21.15.40:0/2127534840 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5d8102250 0x7fa5d819f030 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:01.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.413+0000 7fa5e011e640 1 --2- 172.21.15.40:0/2127534840 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa5d81006a0 0x7fa5d8197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:01.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.413+0000 7fa5e011e640 1 -- 172.21.15.40:0/2127534840 >> 172.21.15.40:0/2127534840 conn(0x7fa5d80fbc30 msgr2=0x7fa5d80fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:01.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.413+0000 7fa5e011e640 1 -- 172.21.15.40:0/2127534840 shutdown_connections 2024-04-21T23:16:01.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.413+0000 7fa5e011e640 1 -- 172.21.15.40:0/2127534840 wait complete. 2024-04-21T23:16:01.427 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:16:01.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.535+0000 7f2d1d2cf640 1 -- 172.21.15.40:0/1940068504 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d18103360 msgr2=0x7f2d18109bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:01.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.535+0000 7f2d1d2cf640 1 --2- 172.21.15.40:0/1940068504 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d18103360 0x7f2d18109bf0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f2d08009ca0 tx=0x7f2d0802c0f0 comp rx=0 tx=0).stop 2024-04-21T23:16:01.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.535+0000 7f2d1d2cf640 1 -- 172.21.15.40:0/1940068504 shutdown_connections 2024-04-21T23:16:01.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.536+0000 7f2d1d2cf640 1 --2- 172.21.15.40:0/1940068504 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d18103360 0x7f2d18109bf0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:01.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.536+0000 7f2d1d2cf640 1 --2- 172.21.15.40:0/1940068504 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2d181029a0 0x7f2d18102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:01.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.536+0000 7f2d1d2cf640 1 --2- 172.21.15.40:0/1940068504 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2d181017b0 0x7f2d18101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:01.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.536+0000 7f2d1d2cf640 1 -- 172.21.15.40:0/1940068504 >> 172.21.15.40:0/1940068504 conn(0x7f2d180fcd40 msgr2=0x7f2d180ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:01.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.536+0000 7f2d1d2cf640 1 -- 172.21.15.40:0/1940068504 shutdown_connections 2024-04-21T23:16:01.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.536+0000 7f2d1d2cf640 1 -- 172.21.15.40:0/1940068504 wait complete. 2024-04-21T23:16:01.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.536+0000 7f2d1d2cf640 1 Processor -- start 2024-04-21T23:16:01.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.536+0000 7f2d1d2cf640 1 -- start start 2024-04-21T23:16:01.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.536+0000 7f2d1d2cf640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2d181017b0 0x7f2d18197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:01.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.536+0000 7f2d1d2cf640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d181029a0 0x7f2d18197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:01.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.536+0000 7f2d1d2cf640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2d18103360 0x7f2d1819efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:01.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.536+0000 7f2d1d2cf640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2d1810d490 con 0x7f2d181029a0 2024-04-21T23:16:01.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.536+0000 7f2d1d2cf640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f2d1810d310 con 0x7f2d181017b0 2024-04-21T23:16:01.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.536+0000 7f2d1d2cf640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2d1810d610 con 0x7f2d18103360 2024-04-21T23:16:01.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.537+0000 7f2d167fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d181029a0 0x7f2d18197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:01.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.537+0000 7f2d167fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d181029a0 0x7f2d18197f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46528/0 (socket says 172.21.15.40:46528) 2024-04-21T23:16:01.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.537+0000 7f2d167fc640 1 -- 172.21.15.40:0/1527454186 learned_addr learned my addr 172.21.15.40:0/1527454186 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:16:01.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.537+0000 7f2d16ffd640 1 --2- 172.21.15.40:0/1527454186 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2d181017b0 0x7f2d18197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:01.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.537+0000 7f2d177fe640 1 --2- 172.21.15.40:0/1527454186 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2d18103360 0x7f2d1819efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:01.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.537+0000 7f2d167fc640 1 -- 172.21.15.40:0/1527454186 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2d18103360 msgr2=0x7f2d1819efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:01.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.537+0000 7f2d167fc640 1 --2- 172.21.15.40:0/1527454186 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2d18103360 0x7f2d1819efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:01.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.537+0000 7f2d167fc640 1 -- 172.21.15.40:0/1527454186 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2d181017b0 msgr2=0x7f2d18197a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:01.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.537+0000 7f2d167fc640 1 --2- 172.21.15.40:0/1527454186 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2d181017b0 0x7f2d18197a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:01.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.537+0000 7f2d167fc640 1 -- 172.21.15.40:0/1527454186 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d1819f630 con 0x7f2d181029a0 2024-04-21T23:16:01.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.537+0000 7f2d167fc640 1 --2- 172.21.15.40:0/1527454186 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d181029a0 0x7f2d18197f50 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f2d0c00b4a0 tx=0x7f2d0c00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:01.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.537+0000 7f2cfffff640 1 -- 172.21.15.40:0/1527454186 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d0c014070 con 0x7f2d181029a0 2024-04-21T23:16:01.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.537+0000 7f2cfffff640 1 -- 172.21.15.40:0/1527454186 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d0c010af0 con 0x7f2d181029a0 2024-04-21T23:16:01.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.537+0000 7f2d1d2cf640 1 -- 172.21.15.40:0/1527454186 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d1819f920 con 0x7f2d181029a0 2024-04-21T23:16:01.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.537+0000 7f2cfffff640 1 -- 172.21.15.40:0/1527454186 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d0c017d10 con 0x7f2d181029a0 2024-04-21T23:16:01.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.537+0000 7f2d1d2cf640 1 -- 172.21.15.40:0/1527454186 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d1819fe20 con 0x7f2d181029a0 2024-04-21T23:16:01.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.538+0000 7f2d1d2cf640 1 -- 172.21.15.40:0/1527454186 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d18108a00 con 0x7f2d181029a0 2024-04-21T23:16:01.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.539+0000 7f2cfffff640 1 -- 172.21.15.40:0/1527454186 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f2d0c017070 con 0x7f2d181029a0 2024-04-21T23:16:01.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.540+0000 7f2cfffff640 1 --2- 172.21.15.40:0/1527454186 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f2d000a2b90 0x7f2d000a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:01.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.540+0000 7f2d16ffd640 1 --2- 172.21.15.40:0/1527454186 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f2d000a2b90 0x7f2d000a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:01.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.540+0000 7f2cfffff640 1 -- 172.21.15.40:0/1527454186 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f2d0c0ce720 con 0x7f2d181029a0 2024-04-21T23:16:01.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.540+0000 7f2d16ffd640 1 --2- 172.21.15.40:0/1527454186 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f2d000a2b90 0x7f2d000a5050 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f2d18102800 tx=0x7f2d10005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:01.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.544+0000 7f2cfffff640 1 -- 172.21.15.40:0/1527454186 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f2d0c09b3e0 con 0x7f2d181029a0 2024-04-21T23:16:01.681 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:01 smithi040 ceph-mon[27984]: pgmap v293: 129 pgs: 22 creating+peering, 25 unknown, 82 active+clean; 579 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s wr, 10 op/s 2024-04-21T23:16:01.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:01 smithi040 ceph-mon[27984]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-21T23:16:01.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:01 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/1633666901' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-04-21T23:16:01.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:01 smithi040 ceph-mon[27984]: osdmap e89: 12 total, 12 up, 12 in 2024-04-21T23:16:01.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:01 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/2127534840' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-21T23:16:01.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:01 smithi174 ceph-mon[31409]: pgmap v293: 129 pgs: 22 creating+peering, 25 unknown, 82 active+clean; 579 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s wr, 10 op/s 2024-04-21T23:16:01.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:01 smithi174 ceph-mon[31409]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-21T23:16:01.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:01 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/1633666901' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-04-21T23:16:01.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:01 smithi174 ceph-mon[31409]: osdmap e89: 12 total, 12 up, 12 in 2024-04-21T23:16:01.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:01 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/2127534840' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-21T23:16:01.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:01 smithi111 ceph-mon[31085]: pgmap v293: 129 pgs: 22 creating+peering, 25 unknown, 82 active+clean; 579 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s wr, 10 op/s 2024-04-21T23:16:01.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:01 smithi111 ceph-mon[31085]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-21T23:16:01.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:01 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/1633666901' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-04-21T23:16:01.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:01 smithi111 ceph-mon[31085]: osdmap e89: 12 total, 12 up, 12 in 2024-04-21T23:16:01.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:01 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/2127534840' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-21T23:16:01.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.770+0000 7f2d1d2cf640 1 -- 172.21.15.40:0/1527454186 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2d1802d170 con 0x7f2d181029a0 2024-04-21T23:16:01.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.770+0000 7f2cfffff640 1 -- 172.21.15.40:0/1527454186 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11809 (secure 0 0 0) 0x7f2d0c09ace0 con 0x7f2d181029a0 2024-04-21T23:16:01.770 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:16:01.770 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/1571948830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1571948830},{"type":"v1","addr":"172.21.15.40:6835","nonce":1571948830}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/969356642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":969356642},{"type":"v1","addr":"172.21.15.40:6837","nonce":969356642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6839/3025057845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":3025057845},{"type":"v1","addr":"172.21.15.40:6839","nonce":3025057845}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6841/3350488804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":3350488804},{"type":"v1","addr":"172.21.15.40:6841","nonce":3350488804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/3438493203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":3438493203},{"type":"v1","addr":"172.21.15.174:6833","nonce":3438493203}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/3105752678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":3105752678},{"type":"v1","addr":"172.21.15.111:6833","nonce":3105752678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24461,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/2704462092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2704462092},{"type":"v1","addr":"172.21.15.111:6835","nonce":2704462092}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/3905733045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3905733045},{"type":"v1","addr":"172.21.15.174:6835","nonce":3905733045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24484,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/465579746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":465579746},{"type":"v1","addr":"172.21.15.174:6837","nonce":465579746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24494,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6837/2703764182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":2703764182},{"type":"v1","addr":"172.21.15.111:6837","nonce":2703764182}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24502,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6839/2467509274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":2467509274},{"type":"v1","addr":"172.21.15.174:6839","nonce":2467509274}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-21T23:15:55.742233+0000","modified":"2024-04-21T23:16:00.780997+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24521},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3712235548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3712235548},{"type":"v1","addr":"172.21.15.111:6839","nonce":3712235548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24521,"qdb_cluster":[24521]},"id":1}]} 2024-04-21T23:16:01.770 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 17 2024-04-21T23:16:01.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.772+0000 7f2d1d2cf640 1 -- 172.21.15.40:0/1527454186 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f2d000a2b90 msgr2=0x7f2d000a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:01.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.772+0000 7f2d1d2cf640 1 --2- 172.21.15.40:0/1527454186 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f2d000a2b90 0x7f2d000a5050 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f2d18102800 tx=0x7f2d10005f90 comp rx=0 tx=0).stop 2024-04-21T23:16:01.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.772+0000 7f2d1d2cf640 1 -- 172.21.15.40:0/1527454186 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d181029a0 msgr2=0x7f2d18197f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:01.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.772+0000 7f2d1d2cf640 1 --2- 172.21.15.40:0/1527454186 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d181029a0 0x7f2d18197f50 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f2d0c00b4a0 tx=0x7f2d0c00b960 comp rx=0 tx=0).stop 2024-04-21T23:16:01.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.772+0000 7f2d1d2cf640 1 -- 172.21.15.40:0/1527454186 shutdown_connections 2024-04-21T23:16:01.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.772+0000 7f2d1d2cf640 1 --2- 172.21.15.40:0/1527454186 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f2d000a2b90 0x7f2d000a5050 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:01.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.772+0000 7f2d1d2cf640 1 --2- 172.21.15.40:0/1527454186 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2d18103360 0x7f2d1819efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:01.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.772+0000 7f2d1d2cf640 1 --2- 172.21.15.40:0/1527454186 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d181029a0 0x7f2d18197f50 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:01.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.772+0000 7f2d1d2cf640 1 --2- 172.21.15.40:0/1527454186 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2d181017b0 0x7f2d18197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:01.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.772+0000 7f2d1d2cf640 1 -- 172.21.15.40:0/1527454186 >> 172.21.15.40:0/1527454186 conn(0x7f2d180fcd40 msgr2=0x7f2d18105bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:01.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.773+0000 7f2d1d2cf640 1 -- 172.21.15.40:0/1527454186 shutdown_connections 2024-04-21T23:16:01.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.773+0000 7f2d1d2cf640 1 -- 172.21.15.40:0/1527454186 wait complete. 2024-04-21T23:16:01.783 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T23:16:01.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.886+0000 7f0cc255c640 1 -- 172.21.15.40:0/2510114319 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0cbc102230 msgr2=0x7f0cbc108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:01.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.886+0000 7f0cc255c640 1 --2- 172.21.15.40:0/2510114319 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0cbc102230 0x7f0cbc108ac0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f0cb4009ca0 tx=0x7f0cb402c0f0 comp rx=0 tx=0).stop 2024-04-21T23:16:01.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.886+0000 7f0cc255c640 1 -- 172.21.15.40:0/2510114319 shutdown_connections 2024-04-21T23:16:01.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.886+0000 7f0cc255c640 1 --2- 172.21.15.40:0/2510114319 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0cbc101870 0x7f0cbc101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:01.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.886+0000 7f0cc255c640 1 --2- 172.21.15.40:0/2510114319 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0cbc102230 0x7f0cbc108ac0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:01.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.886+0000 7f0cc255c640 1 --2- 172.21.15.40:0/2510114319 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0cbc100680 0x7f0cbc100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:01.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.886+0000 7f0cc255c640 1 -- 172.21.15.40:0/2510114319 >> 172.21.15.40:0/2510114319 conn(0x7f0cbc0fbbf0 msgr2=0x7f0cbc0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:01.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.886+0000 7f0cc255c640 1 -- 172.21.15.40:0/2510114319 shutdown_connections 2024-04-21T23:16:01.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.886+0000 7f0cc255c640 1 -- 172.21.15.40:0/2510114319 wait complete. 2024-04-21T23:16:01.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.886+0000 7f0cc255c640 1 Processor -- start 2024-04-21T23:16:01.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.886+0000 7f0cc255c640 1 -- start start 2024-04-21T23:16:01.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.886+0000 7f0cc255c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0cbc100680 0x7f0cbc069440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:01.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.887+0000 7f0cc255c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0cbc101870 0x7f0cbc19c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:01.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.887+0000 7f0cc255c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0cbc102230 0x7f0cbc19ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:01.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.887+0000 7f0cc255c640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0cbc10c300 con 0x7f0cbc101870 2024-04-21T23:16:01.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.887+0000 7f0cc255c640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f0cbc10c180 con 0x7f0cbc102230 2024-04-21T23:16:01.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.887+0000 7f0cc255c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0cbc10c480 con 0x7f0cbc100680 2024-04-21T23:16:01.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.887+0000 7f0cbb7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0cbc101870 0x7f0cbc19c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:01.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.887+0000 7f0cbb7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0cbc101870 0x7f0cbc19c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33126/0 (socket says 172.21.15.40:33126) 2024-04-21T23:16:01.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.887+0000 7f0cbb7fe640 1 -- 172.21.15.40:0/2525197907 learned_addr learned my addr 172.21.15.40:0/2525197907 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:16:01.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.887+0000 7f0cbbfff640 1 --2- 172.21.15.40:0/2525197907 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0cbc100680 0x7f0cbc069440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:01.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.887+0000 7f0cc0af9640 1 --2- 172.21.15.40:0/2525197907 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0cbc102230 0x7f0cbc19ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:01.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.887+0000 7f0cbb7fe640 1 -- 172.21.15.40:0/2525197907 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0cbc100680 msgr2=0x7f0cbc069440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:01.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.887+0000 7f0cbb7fe640 1 --2- 172.21.15.40:0/2525197907 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0cbc100680 0x7f0cbc069440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:01.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.887+0000 7f0cbb7fe640 1 -- 172.21.15.40:0/2525197907 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0cbc102230 msgr2=0x7f0cbc19ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:01.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.887+0000 7f0cbb7fe640 1 --2- 172.21.15.40:0/2525197907 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0cbc102230 0x7f0cbc19ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:01.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.887+0000 7f0cbb7fe640 1 -- 172.21.15.40:0/2525197907 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cbc19f730 con 0x7f0cbc101870 2024-04-21T23:16:01.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.887+0000 7f0cbb7fe640 1 --2- 172.21.15.40:0/2525197907 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0cbc101870 0x7f0cbc19c800 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f0cac00d950 tx=0x7f0cac00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:01.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.887+0000 7f0cb97fa640 1 -- 172.21.15.40:0/2525197907 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cac017070 con 0x7f0cbc101870 2024-04-21T23:16:01.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.887+0000 7f0cb97fa640 1 -- 172.21.15.40:0/2525197907 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0cac009c30 con 0x7f0cbc101870 2024-04-21T23:16:01.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.887+0000 7f0cc255c640 1 -- 172.21.15.40:0/2525197907 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0cbc19f9c0 con 0x7f0cbc101870 2024-04-21T23:16:01.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.887+0000 7f0cb97fa640 1 -- 172.21.15.40:0/2525197907 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cac00f6b0 con 0x7f0cbc101870 2024-04-21T23:16:01.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.887+0000 7f0cc255c640 1 -- 172.21.15.40:0/2525197907 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0cbc19ff10 con 0x7f0cbc101870 2024-04-21T23:16:01.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.888+0000 7f0cc255c640 1 -- 172.21.15.40:0/2525197907 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0cbc107940 con 0x7f0cbc101870 2024-04-21T23:16:01.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.890+0000 7f0cb97fa640 1 -- 172.21.15.40:0/2525197907 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f0cac0109d0 con 0x7f0cbc101870 2024-04-21T23:16:01.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.890+0000 7f0cb97fa640 1 --2- 172.21.15.40:0/2525197907 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f0ca80a2b90 0x7f0ca80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:01.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.890+0000 7f0cbbfff640 1 --2- 172.21.15.40:0/2525197907 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f0ca80a2b90 0x7f0ca80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:01.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.891+0000 7f0cb97fa640 1 -- 172.21.15.40:0/2525197907 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f0cac0ceed0 con 0x7f0cbc101870 2024-04-21T23:16:01.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.891+0000 7f0cbbfff640 1 --2- 172.21.15.40:0/2525197907 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f0ca80a2b90 0x7f0ca80a5050 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f0ca40058c0 tx=0x7f0ca400a2d0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:01.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:01.895+0000 7f0cb97fa640 1 -- 172.21.15.40:0/2525197907 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f0cac009de0 con 0x7f0cbc101870 2024-04-21T23:16:02.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.079+0000 7f0cc255c640 1 -- 172.21.15.40:0/2525197907 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0cbc062370 con 0x7f0cbc101870 2024-04-21T23:16:02.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.081+0000 7f0cb97fa640 1 -- 172.21.15.40:0/2525197907 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24138 (secure 0 0 0) 0x7f0cac09bbf0 con 0x7f0cbc101870 2024-04-21T23:16:02.080 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:16:02.080 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":90,"fsid":"d814e4dc-0033-11ef-bc93-c7b262605968","created":"2024-04-21T23:07:52.659207+0000","modified":"2024-04-21T23:16:00.780824+0000","last_up_change":"2024-04-21T23:15:05.308821+0000","last_in_change":"2024-04-21T23:14:45.447491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:11:14.824775+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T23:15:54.322571+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T23:15:55.164493+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T23:15:58.838266+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"0e8d692a-22fe-4a87-8d99-9151f96cbb02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6803","nonce":2009758784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6805","nonce":2009758784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6809","nonce":2009758784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6807","nonce":2009758784}]},"public_addr":"172.21.15.40:6803/2009758784","cluster_addr":"172.21.15.40:6805/2009758784","heartbeat_back_addr":"172.21.15.40:6809/2009758784","heartbeat_front_addr":"172.21.15.40:6807/2009758784","state":["exists","up"]},{"osd":1,"uuid":"fe5fe11b-0e84-4359-83a7-1ba45105755a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6801","nonce":1763903836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6803","nonce":1763903836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6807","nonce":1763903836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6805","nonce":1763903836}]},"public_addr":"172.21.15.111:6801/1763903836","cluster_addr":"172.21.15.111:6803/1763903836","heartbeat_back_addr":"172.21.15.111:6807/1763903836","heartbeat_front_addr":"172.21.15.111:6805/1763903836","state":["exists","up"]},{"osd":2,"uuid":"813ad735-5cc2-4da0-ae16-6db238395fbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6801","nonce":439695825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6803","nonce":439695825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6807","nonce":439695825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6805","nonce":439695825}]},"public_addr":"172.21.15.174:6801/439695825","cluster_addr":"172.21.15.174:6803/439695825","heartbeat_back_addr":"172.21.15.174:6807/439695825","heartbeat_front_addr":"172.21.15.174:6805/439695825","state":["exists","up"]},{"osd":3,"uuid":"09becb03-b7ce-4ffc-917d-b7900ec37e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6811","nonce":3454312436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6813","nonce":3454312436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6817","nonce":3454312436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6815","nonce":3454312436}]},"public_addr":"172.21.15.40:6811/3454312436","cluster_addr":"172.21.15.40:6813/3454312436","heartbeat_back_addr":"172.21.15.40:6817/3454312436","heartbeat_front_addr":"172.21.15.40:6815/3454312436","state":["exists","up"]},{"osd":4,"uuid":"46eabfd7-d88a-4efd-94eb-04354d3a9d7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6809","nonce":1810866070}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6811","nonce":1810866070}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6815","nonce":1810866070}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6813","nonce":1810866070}]},"public_addr":"172.21.15.111:6809/1810866070","cluster_addr":"172.21.15.111:6811/1810866070","heartbeat_back_addr":"172.21.15.111:6815/1810866070","heartbeat_front_addr":"172.21.15.111:6813/1810866070","state":["exists","up"]},{"osd":5,"uuid":"6f73b582-6c65-41db-aab0-9b360f83d5c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6809","nonce":3752986711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6811","nonce":3752986711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6815","nonce":3752986711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6813","nonce":3752986711}]},"public_addr":"172.21.15.174:6809/3752986711","cluster_addr":"172.21.15.174:6811/3752986711","heartbeat_back_addr":"172.21.15.174:6815/3752986711","heartbeat_front_addr":"172.21.15.174:6813/3752986711","state":["exists","up"]},{"osd":6,"uuid":"39f305f0-12d2-4732-aaa0-cd66700af538","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6819","nonce":1618855427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6821","nonce":1618855427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6825","nonce":1618855427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6823","nonce":1618855427}]},"public_addr":"172.21.15.40:6819/1618855427","cluster_addr":"172.21.15.40:6821/1618855427","heartbeat_back_addr":"172.21.15.40:6825/1618855427","heartbeat_front_addr":"172.21.15.40:6823/1618855427","state":["exists","up"]},{"osd":7,"uuid":"769f2468-5102-490b-89f8-d0444d3169a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6817","nonce":927054162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6819","nonce":927054162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6823","nonce":927054162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6821","nonce":927054162}]},"public_addr":"172.21.15.111:6817/927054162","cluster_addr":"172.21.15.111:6819/927054162","heartbeat_back_addr":"172.21.15.111:6823/927054162","heartbeat_front_addr":"172.21.15.111:6821/927054162","state":["exists","up"]},{"osd":8,"uuid":"1a444e80-007d-473b-b972-4c814a813c22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6817","nonce":3817848746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6819","nonce":3817848746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6823","nonce":3817848746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6821","nonce":3817848746}]},"public_addr":"172.21.15.174:6817/3817848746","cluster_addr":"172.21.15.174:6819/3817848746","heartbeat_back_addr":"172.21.15.174:6823/3817848746","heartbeat_front_addr":"172.21.15.174:6821/3817848746","state":["exists","up"]},{"osd":9,"uuid":"9a6b6a56-6ae5-49fe-9e5b-cb17d686e0dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6827","nonce":1649446531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6829","nonce":1649446531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6833","nonce":1649446531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6831","nonce":1649446531}]},"public_addr":"172.21.15.40:6827/1649446531","cluster_addr":"172.21.15.40:6829/1649446531","heartbeat_back_addr":"172.21.15.40:6833/1649446531","heartbeat_front_addr":"172.21.15.40:6831/1649446531","state":["exists","up"]},{"osd":10,"uuid":"a70038ba-e2ad-4170-85cc-ea0470a2edef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6825","nonce":1227538935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6827","nonce":1227538935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6831","nonce":1227538935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6829","nonce":1227538935}]},"public_addr":"172.21.15.111:6825/1227538935","cluster_addr":"172.21.15.111:6827/1227538935","heartbeat_back_addr":"172.21.15.111:6831/1227538935","heartbeat_front_addr":"172.21.15.111:6829/1227538935","state":["exists","up"]},{"osd":11,"uuid":"781532bb-7a5e-44e3-8eb7-b584b0054666","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6825","nonce":2045865553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6827","nonce":2045865553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6831","nonce":2045865553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6829","nonce":2045865553}]},"public_addr":"172.21.15.174:6825/2045865553","cluster_addr":"172.21.15.174:6827/2045865553","heartbeat_back_addr":"172.21.15.174:6831/2045865553","heartbeat_front_addr":"172.21.15.174:6829/2045865553","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:10:24.761290+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:10:48.565428+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:11:12.307818+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:11:37.951529+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:02.832523+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:27.568879+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:52.997707+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:13:18.937274+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:13:44.708657+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:14:10.833855+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:14:37.078272+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:15:03.878076+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:6801/526554319":"2024-04-22T23:08:50.305770+0000","172.21.15.40:6800/526554319":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/3178763471":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/4045911364":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/276300080":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/3689649814":"2024-04-22T23:08:23.743030+0000","172.21.15.40:0/2241116715":"2024-04-22T23:08:23.743030+0000","172.21.15.40:6801/1479040326":"2024-04-22T23:08:23.743030+0000","172.21.15.40:6800/1479040326":"2024-04-22T23:08:23.743030+0000","172.21.15.40:0/681298099":"2024-04-22T23:08:23.743030+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T23:16:02.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.083+0000 7f0cc255c640 1 -- 172.21.15.40:0/2525197907 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f0ca80a2b90 msgr2=0x7f0ca80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:02.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.083+0000 7f0cc255c640 1 --2- 172.21.15.40:0/2525197907 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f0ca80a2b90 0x7f0ca80a5050 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f0ca40058c0 tx=0x7f0ca400a2d0 comp rx=0 tx=0).stop 2024-04-21T23:16:02.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.083+0000 7f0cc255c640 1 -- 172.21.15.40:0/2525197907 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0cbc101870 msgr2=0x7f0cbc19c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:02.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.083+0000 7f0cc255c640 1 --2- 172.21.15.40:0/2525197907 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0cbc101870 0x7f0cbc19c800 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f0cac00d950 tx=0x7f0cac00de10 comp rx=0 tx=0).stop 2024-04-21T23:16:02.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.083+0000 7f0cc255c640 1 -- 172.21.15.40:0/2525197907 shutdown_connections 2024-04-21T23:16:02.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.083+0000 7f0cc255c640 1 --2- 172.21.15.40:0/2525197907 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f0ca80a2b90 0x7f0ca80a5050 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:02.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.083+0000 7f0cc255c640 1 --2- 172.21.15.40:0/2525197907 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0cbc102230 0x7f0cbc19ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:02.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.083+0000 7f0cc255c640 1 --2- 172.21.15.40:0/2525197907 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0cbc101870 0x7f0cbc19c800 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:02.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.083+0000 7f0cc255c640 1 --2- 172.21.15.40:0/2525197907 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0cbc100680 0x7f0cbc069440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:02.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.083+0000 7f0cc255c640 1 -- 172.21.15.40:0/2525197907 >> 172.21.15.40:0/2525197907 conn(0x7f0cbc0fbbf0 msgr2=0x7f0cbc104b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:02.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.083+0000 7f0cc255c640 1 -- 172.21.15.40:0/2525197907 shutdown_connections 2024-04-21T23:16:02.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.083+0000 7f0cc255c640 1 -- 172.21.15.40:0/2525197907 wait complete. 2024-04-21T23:16:02.094 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T23:16:02.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.196+0000 7f2260d65640 1 -- 172.21.15.40:0/3224158821 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f225c100680 msgr2=0x7f225c100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:02.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.196+0000 7f2260d65640 1 --2- 172.21.15.40:0/3224158821 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f225c100680 0x7f225c100aa0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f2254009900 tx=0x7f225402bc30 comp rx=0 tx=0).stop 2024-04-21T23:16:02.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.197+0000 7f2260d65640 1 -- 172.21.15.40:0/3224158821 shutdown_connections 2024-04-21T23:16:02.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.197+0000 7f2260d65640 1 --2- 172.21.15.40:0/3224158821 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f225c102230 0x7f225c108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:02.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.197+0000 7f2260d65640 1 --2- 172.21.15.40:0/3224158821 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f225c101870 0x7f225c101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:02.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.197+0000 7f2260d65640 1 --2- 172.21.15.40:0/3224158821 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f225c100680 0x7f225c100aa0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:02.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.197+0000 7f2260d65640 1 -- 172.21.15.40:0/3224158821 >> 172.21.15.40:0/3224158821 conn(0x7f225c0fbbf0 msgr2=0x7f225c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:02.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.197+0000 7f2260d65640 1 -- 172.21.15.40:0/3224158821 shutdown_connections 2024-04-21T23:16:02.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.197+0000 7f2260d65640 1 -- 172.21.15.40:0/3224158821 wait complete. 2024-04-21T23:16:02.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.197+0000 7f2260d65640 1 Processor -- start 2024-04-21T23:16:02.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.197+0000 7f2260d65640 1 -- start start 2024-04-21T23:16:02.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.197+0000 7f2260d65640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f225c100680 0x7f225c10ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:02.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.198+0000 7f2260d65640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f225c101870 0x7f225c1094c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:02.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.198+0000 7f2260d65640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f225c102230 0x7f225c109a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:02.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.198+0000 7f2260d65640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f225c10d3e0 con 0x7f225c102230 2024-04-21T23:16:02.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.198+0000 7f2260d65640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f225c10d260 con 0x7f225c100680 2024-04-21T23:16:02.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.198+0000 7f2260d65640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f225c10d560 con 0x7f225c101870 2024-04-21T23:16:02.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.198+0000 7f225ad9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f225c102230 0x7f225c109a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:02.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.198+0000 7f225ad9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f225c102230 0x7f225c109a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33136/0 (socket says 172.21.15.40:33136) 2024-04-21T23:16:02.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.198+0000 7f225ad9d640 1 -- 172.21.15.40:0/3709224332 learned_addr learned my addr 172.21.15.40:0/3709224332 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:16:02.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.198+0000 7f225a59c640 1 --2- 172.21.15.40:0/3709224332 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f225c100680 0x7f225c10ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:02.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.198+0000 7f225ad9d640 1 -- 172.21.15.40:0/3709224332 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f225c101870 msgr2=0x7f225c1094c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:02.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.198+0000 7f2259d9b640 1 --2- 172.21.15.40:0/3709224332 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f225c101870 0x7f225c1094c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:02.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.198+0000 7f225ad9d640 1 --2- 172.21.15.40:0/3709224332 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f225c101870 0x7f225c1094c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:02.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.198+0000 7f225ad9d640 1 -- 172.21.15.40:0/3709224332 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f225c100680 msgr2=0x7f225c10ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:02.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.198+0000 7f225ad9d640 1 --2- 172.21.15.40:0/3709224332 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f225c100680 0x7f225c10ae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:02.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.198+0000 7f225ad9d640 1 -- 172.21.15.40:0/3709224332 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f225c10a290 con 0x7f225c102230 2024-04-21T23:16:02.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.198+0000 7f2259d9b640 1 --2- 172.21.15.40:0/3709224332 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f225c101870 0x7f225c1094c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:16:02.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.198+0000 7f225ad9d640 1 --2- 172.21.15.40:0/3709224332 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f225c102230 0x7f225c109a00 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f224c00eaa0 tx=0x7f224c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:02.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.198+0000 7f22537fe640 1 -- 172.21.15.40:0/3709224332 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f224c00cdb0 con 0x7f225c102230 2024-04-21T23:16:02.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.198+0000 7f2260d65640 1 -- 172.21.15.40:0/3709224332 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f225c1a1bd0 con 0x7f225c102230 2024-04-21T23:16:02.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.198+0000 7f22537fe640 1 -- 172.21.15.40:0/3709224332 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f224c005020 con 0x7f225c102230 2024-04-21T23:16:02.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.198+0000 7f22537fe640 1 -- 172.21.15.40:0/3709224332 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f224c0116b0 con 0x7f225c102230 2024-04-21T23:16:02.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.198+0000 7f2260d65640 1 -- 172.21.15.40:0/3709224332 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f225c1a20d0 con 0x7f225c102230 2024-04-21T23:16:02.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.199+0000 7f2260d65640 1 -- 172.21.15.40:0/3709224332 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f225c107a20 con 0x7f225c102230 2024-04-21T23:16:02.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.201+0000 7f22537fe640 1 -- 172.21.15.40:0/3709224332 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f224c005560 con 0x7f225c102230 2024-04-21T23:16:02.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.201+0000 7f22537fe640 1 --2- 172.21.15.40:0/3709224332 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f22480a2b90 0x7f22480a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:02.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.201+0000 7f225a59c640 1 --2- 172.21.15.40:0/3709224332 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f22480a2b90 0x7f22480a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:02.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.201+0000 7f22537fe640 1 -- 172.21.15.40:0/3709224332 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f224c0cf910 con 0x7f225c102230 2024-04-21T23:16:02.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.202+0000 7f225a59c640 1 --2- 172.21.15.40:0/3709224332 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f22480a2b90 0x7f22480a5050 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f2254000c00 tx=0x7f2254005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:02.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.204+0000 7f22537fe640 1 -- 172.21.15.40:0/3709224332 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f224c016030 con 0x7f225c102230 2024-04-21T23:16:02.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.390+0000 7f2260d65640 1 -- 172.21.15.40:0/3709224332 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f225c02d2f0 con 0x7f225c102230 2024-04-21T23:16:02.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.391+0000 7f22537fe640 1 -- 172.21.15.40:0/3709224332 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24138 (secure 0 0 0) 0x7f224c09c630 con 0x7f225c102230 2024-04-21T23:16:02.391 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:16:02.391 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":90,"fsid":"d814e4dc-0033-11ef-bc93-c7b262605968","created":"2024-04-21T23:07:52.659207+0000","modified":"2024-04-21T23:16:00.780824+0000","last_up_change":"2024-04-21T23:15:05.308821+0000","last_in_change":"2024-04-21T23:14:45.447491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:11:14.824775+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T23:15:54.322571+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T23:15:55.164493+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T23:15:58.838266+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"0e8d692a-22fe-4a87-8d99-9151f96cbb02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6803","nonce":2009758784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6805","nonce":2009758784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6809","nonce":2009758784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6807","nonce":2009758784}]},"public_addr":"172.21.15.40:6803/2009758784","cluster_addr":"172.21.15.40:6805/2009758784","heartbeat_back_addr":"172.21.15.40:6809/2009758784","heartbeat_front_addr":"172.21.15.40:6807/2009758784","state":["exists","up"]},{"osd":1,"uuid":"fe5fe11b-0e84-4359-83a7-1ba45105755a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6801","nonce":1763903836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6803","nonce":1763903836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6807","nonce":1763903836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6805","nonce":1763903836}]},"public_addr":"172.21.15.111:6801/1763903836","cluster_addr":"172.21.15.111:6803/1763903836","heartbeat_back_addr":"172.21.15.111:6807/1763903836","heartbeat_front_addr":"172.21.15.111:6805/1763903836","state":["exists","up"]},{"osd":2,"uuid":"813ad735-5cc2-4da0-ae16-6db238395fbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6801","nonce":439695825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6803","nonce":439695825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6807","nonce":439695825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6805","nonce":439695825}]},"public_addr":"172.21.15.174:6801/439695825","cluster_addr":"172.21.15.174:6803/439695825","heartbeat_back_addr":"172.21.15.174:6807/439695825","heartbeat_front_addr":"172.21.15.174:6805/439695825","state":["exists","up"]},{"osd":3,"uuid":"09becb03-b7ce-4ffc-917d-b7900ec37e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6811","nonce":3454312436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6813","nonce":3454312436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6817","nonce":3454312436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6815","nonce":3454312436}]},"public_addr":"172.21.15.40:6811/3454312436","cluster_addr":"172.21.15.40:6813/3454312436","heartbeat_back_addr":"172.21.15.40:6817/3454312436","heartbeat_front_addr":"172.21.15.40:6815/3454312436","state":["exists","up"]},{"osd":4,"uuid":"46eabfd7-d88a-4efd-94eb-04354d3a9d7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6809","nonce":1810866070}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6811","nonce":1810866070}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6815","nonce":1810866070}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6813","nonce":1810866070}]},"public_addr":"172.21.15.111:6809/1810866070","cluster_addr":"172.21.15.111:6811/1810866070","heartbeat_back_addr":"172.21.15.111:6815/1810866070","heartbeat_front_addr":"172.21.15.111:6813/1810866070","state":["exists","up"]},{"osd":5,"uuid":"6f73b582-6c65-41db-aab0-9b360f83d5c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6809","nonce":3752986711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6811","nonce":3752986711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6815","nonce":3752986711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6813","nonce":3752986711}]},"public_addr":"172.21.15.174:6809/3752986711","cluster_addr":"172.21.15.174:6811/3752986711","heartbeat_back_addr":"172.21.15.174:6815/3752986711","heartbeat_front_addr":"172.21.15.174:6813/3752986711","state":["exists","up"]},{"osd":6,"uuid":"39f305f0-12d2-4732-aaa0-cd66700af538","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6819","nonce":1618855427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6821","nonce":1618855427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6825","nonce":1618855427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6823","nonce":1618855427}]},"public_addr":"172.21.15.40:6819/1618855427","cluster_addr":"172.21.15.40:6821/1618855427","heartbeat_back_addr":"172.21.15.40:6825/1618855427","heartbeat_front_addr":"172.21.15.40:6823/1618855427","state":["exists","up"]},{"osd":7,"uuid":"769f2468-5102-490b-89f8-d0444d3169a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6817","nonce":927054162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6819","nonce":927054162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6823","nonce":927054162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6821","nonce":927054162}]},"public_addr":"172.21.15.111:6817/927054162","cluster_addr":"172.21.15.111:6819/927054162","heartbeat_back_addr":"172.21.15.111:6823/927054162","heartbeat_front_addr":"172.21.15.111:6821/927054162","state":["exists","up"]},{"osd":8,"uuid":"1a444e80-007d-473b-b972-4c814a813c22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6817","nonce":3817848746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6819","nonce":3817848746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6823","nonce":3817848746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6821","nonce":3817848746}]},"public_addr":"172.21.15.174:6817/3817848746","cluster_addr":"172.21.15.174:6819/3817848746","heartbeat_back_addr":"172.21.15.174:6823/3817848746","heartbeat_front_addr":"172.21.15.174:6821/3817848746","state":["exists","up"]},{"osd":9,"uuid":"9a6b6a56-6ae5-49fe-9e5b-cb17d686e0dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6827","nonce":1649446531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6829","nonce":1649446531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6833","nonce":1649446531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6831","nonce":1649446531}]},"public_addr":"172.21.15.40:6827/1649446531","cluster_addr":"172.21.15.40:6829/1649446531","heartbeat_back_addr":"172.21.15.40:6833/1649446531","heartbeat_front_addr":"172.21.15.40:6831/1649446531","state":["exists","up"]},{"osd":10,"uuid":"a70038ba-e2ad-4170-85cc-ea0470a2edef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6825","nonce":1227538935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6827","nonce":1227538935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6831","nonce":1227538935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6829","nonce":1227538935}]},"public_addr":"172.21.15.111:6825/1227538935","cluster_addr":"172.21.15.111:6827/1227538935","heartbeat_back_addr":"172.21.15.111:6831/1227538935","heartbeat_front_addr":"172.21.15.111:6829/1227538935","state":["exists","up"]},{"osd":11,"uuid":"781532bb-7a5e-44e3-8eb7-b584b0054666","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6825","nonce":2045865553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6827","nonce":2045865553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6831","nonce":2045865553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6829","nonce":2045865553}]},"public_addr":"172.21.15.174:6825/2045865553","cluster_addr":"172.21.15.174:6827/2045865553","heartbeat_back_addr":"172.21.15.174:6831/2045865553","heartbeat_front_addr":"172.21.15.174:6829/2045865553","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:10:24.761290+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:10:48.565428+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:11:12.307818+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:11:37.951529+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:02.832523+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:27.568879+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:52.997707+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:13:18.937274+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:13:44.708657+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:14:10.833855+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:14:37.078272+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:15:03.878076+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:6801/526554319":"2024-04-22T23:08:50.305770+0000","172.21.15.40:6800/526554319":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/3178763471":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/4045911364":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/276300080":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/3689649814":"2024-04-22T23:08:23.743030+0000","172.21.15.40:0/2241116715":"2024-04-22T23:08:23.743030+0000","172.21.15.40:6801/1479040326":"2024-04-22T23:08:23.743030+0000","172.21.15.40:6800/1479040326":"2024-04-22T23:08:23.743030+0000","172.21.15.40:0/681298099":"2024-04-22T23:08:23.743030+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T23:16:02.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.393+0000 7f2260d65640 1 -- 172.21.15.40:0/3709224332 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f22480a2b90 msgr2=0x7f22480a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:02.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.393+0000 7f2260d65640 1 --2- 172.21.15.40:0/3709224332 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f22480a2b90 0x7f22480a5050 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f2254000c00 tx=0x7f2254005fd0 comp rx=0 tx=0).stop 2024-04-21T23:16:02.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.393+0000 7f2260d65640 1 -- 172.21.15.40:0/3709224332 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f225c102230 msgr2=0x7f225c109a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:02.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.393+0000 7f2260d65640 1 --2- 172.21.15.40:0/3709224332 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f225c102230 0x7f225c109a00 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f224c00eaa0 tx=0x7f224c00ef60 comp rx=0 tx=0).stop 2024-04-21T23:16:02.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.393+0000 7f2260d65640 1 -- 172.21.15.40:0/3709224332 shutdown_connections 2024-04-21T23:16:02.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.393+0000 7f2260d65640 1 --2- 172.21.15.40:0/3709224332 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f22480a2b90 0x7f22480a5050 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:02.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.393+0000 7f2260d65640 1 --2- 172.21.15.40:0/3709224332 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f225c102230 0x7f225c109a00 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:02.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.393+0000 7f2260d65640 1 --2- 172.21.15.40:0/3709224332 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f225c101870 0x7f225c1094c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:02.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.393+0000 7f2260d65640 1 --2- 172.21.15.40:0/3709224332 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f225c100680 0x7f225c10ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:02.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.393+0000 7f2260d65640 1 -- 172.21.15.40:0/3709224332 >> 172.21.15.40:0/3709224332 conn(0x7f225c0fbbf0 msgr2=0x7f225c104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:02.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.393+0000 7f2260d65640 1 -- 172.21.15.40:0/3709224332 shutdown_connections 2024-04-21T23:16:02.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.393+0000 7f2260d65640 1 -- 172.21.15.40:0/3709224332 wait complete. 2024-04-21T23:16:02.418 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:16:02.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.522+0000 7f7746a66640 1 -- 172.21.15.40:0/3258390420 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f77401029a0 msgr2=0x7f7740102e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:02.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.522+0000 7f7746a66640 1 --2- 172.21.15.40:0/3258390420 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f77401029a0 0x7f7740102e20 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f7730009900 tx=0x7f773002bc30 comp rx=0 tx=0).stop 2024-04-21T23:16:02.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.522+0000 7f7746a66640 1 -- 172.21.15.40:0/3258390420 shutdown_connections 2024-04-21T23:16:02.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.522+0000 7f7746a66640 1 --2- 172.21.15.40:0/3258390420 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7740103360 0x7f7740109bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:02.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.522+0000 7f7746a66640 1 --2- 172.21.15.40:0/3258390420 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f77401029a0 0x7f7740102e20 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:02.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.522+0000 7f7746a66640 1 --2- 172.21.15.40:0/3258390420 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77401017b0 0x7f7740101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:02.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.522+0000 7f7746a66640 1 -- 172.21.15.40:0/3258390420 >> 172.21.15.40:0/3258390420 conn(0x7f77400fcd40 msgr2=0x7f77400ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:02.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.522+0000 7f7746a66640 1 -- 172.21.15.40:0/3258390420 shutdown_connections 2024-04-21T23:16:02.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.522+0000 7f7746a66640 1 -- 172.21.15.40:0/3258390420 wait complete. 2024-04-21T23:16:02.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.522+0000 7f7746a66640 1 Processor -- start 2024-04-21T23:16:02.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.522+0000 7f7746a66640 1 -- start start 2024-04-21T23:16:02.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.523+0000 7f7746a66640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77401017b0 0x7f7740197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:02.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.523+0000 7f7746a66640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f77401029a0 0x7f7740198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:02.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.523+0000 7f7746a66640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7740103360 0x7f774019f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:02.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.523+0000 7f7746a66640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f774010d430 con 0x7f7740103360 2024-04-21T23:16:02.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.523+0000 7f7746a66640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f774010d2b0 con 0x7f77401017b0 2024-04-21T23:16:02.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.523+0000 7f7746a66640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f774010d5b0 con 0x7f77401029a0 2024-04-21T23:16:02.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.523+0000 7f7745003640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7740103360 0x7f774019f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:02.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.523+0000 7f7745003640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7740103360 0x7f774019f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33164/0 (socket says 172.21.15.40:33164) 2024-04-21T23:16:02.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.523+0000 7f7745003640 1 -- 172.21.15.40:0/3929145817 learned_addr learned my addr 172.21.15.40:0/3929145817 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:16:02.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.523+0000 7f7744802640 1 --2- 172.21.15.40:0/3929145817 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77401017b0 0x7f7740197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:02.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.523+0000 7f773ffff640 1 --2- 172.21.15.40:0/3929145817 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f77401029a0 0x7f7740198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:02.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.523+0000 7f7745003640 1 -- 172.21.15.40:0/3929145817 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f77401029a0 msgr2=0x7f7740198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:02.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.523+0000 7f7745003640 1 --2- 172.21.15.40:0/3929145817 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f77401029a0 0x7f7740198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:02.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.523+0000 7f7745003640 1 -- 172.21.15.40:0/3929145817 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77401017b0 msgr2=0x7f7740197b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:02.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.523+0000 7f7745003640 1 --2- 172.21.15.40:0/3929145817 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77401017b0 0x7f7740197b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:02.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.523+0000 7f7745003640 1 -- 172.21.15.40:0/3929145817 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f774019f720 con 0x7f7740103360 2024-04-21T23:16:02.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.523+0000 7f7745003640 1 --2- 172.21.15.40:0/3929145817 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7740103360 0x7f774019f0b0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f772800c970 tx=0x7f772800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:02.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.523+0000 7f773dffb640 1 -- 172.21.15.40:0/3929145817 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7728007bb0 con 0x7f7740103360 2024-04-21T23:16:02.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.523+0000 7f7746a66640 1 -- 172.21.15.40:0/3929145817 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f774019fa10 con 0x7f7740103360 2024-04-21T23:16:02.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.523+0000 7f773dffb640 1 -- 172.21.15.40:0/3929145817 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7728007d50 con 0x7f7740103360 2024-04-21T23:16:02.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.523+0000 7f773dffb640 1 -- 172.21.15.40:0/3929145817 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f772800f450 con 0x7f7740103360 2024-04-21T23:16:02.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.524+0000 7f7746a66640 1 -- 172.21.15.40:0/3929145817 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f774019ff80 con 0x7f7740103360 2024-04-21T23:16:02.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.525+0000 7f7746a66640 1 -- 172.21.15.40:0/3929145817 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7740108ad0 con 0x7f7740103360 2024-04-21T23:16:02.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.526+0000 7f773dffb640 1 -- 172.21.15.40:0/3929145817 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f7728004f20 con 0x7f7740103360 2024-04-21T23:16:02.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.526+0000 7f773dffb640 1 --2- 172.21.15.40:0/3929145817 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f772c0a2b90 0x7f772c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:02.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.526+0000 7f7744802640 1 --2- 172.21.15.40:0/3929145817 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f772c0a2b90 0x7f772c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:02.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.526+0000 7f773dffb640 1 -- 172.21.15.40:0/3929145817 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f77280cee70 con 0x7f7740103360 2024-04-21T23:16:02.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.527+0000 7f7744802640 1 --2- 172.21.15.40:0/3929145817 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f772c0a2b90 0x7f772c0a5050 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f7740102800 tx=0x7f773800a590 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:02.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.531+0000 7f773dffb640 1 -- 172.21.15.40:0/3929145817 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f7728014030 con 0x7f7740103360 2024-04-21T23:16:02.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:02 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/2127534840' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-04-21T23:16:02.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:02 smithi040 ceph-mon[27984]: osdmap e90: 12 total, 12 up, 12 in 2024-04-21T23:16:02.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:02 smithi040 ceph-mon[27984]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T23:16:02.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:02 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/1527454186' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:16:02.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:02 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/2525197907' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:16:02.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:02 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/3709224332' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:16:02.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:02 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/2127534840' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-04-21T23:16:02.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:02 smithi174 ceph-mon[31409]: osdmap e90: 12 total, 12 up, 12 in 2024-04-21T23:16:02.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:02 smithi174 ceph-mon[31409]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T23:16:02.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:02 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/1527454186' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:16:02.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:02 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/2525197907' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:16:02.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:02 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/3709224332' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:16:02.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:02 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/2127534840' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-04-21T23:16:02.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:02 smithi111 ceph-mon[31085]: osdmap e90: 12 total, 12 up, 12 in 2024-04-21T23:16:02.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:02 smithi111 ceph-mon[31085]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T23:16:02.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:02 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/1527454186' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:16:02.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:02 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/2525197907' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:16:02.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:02 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/3709224332' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:16:02.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.760+0000 7f7746a66640 1 -- 172.21.15.40:0/3929145817 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f774002d170 con 0x7f7740103360 2024-04-21T23:16:02.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.761+0000 7f773dffb640 1 -- 172.21.15.40:0/3929145817 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11809 (secure 0 0 0) 0x7f772809b540 con 0x7f7740103360 2024-04-21T23:16:02.760 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:16:02.761 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/1571948830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1571948830},{"type":"v1","addr":"172.21.15.40:6835","nonce":1571948830}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/969356642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":969356642},{"type":"v1","addr":"172.21.15.40:6837","nonce":969356642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6839/3025057845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":3025057845},{"type":"v1","addr":"172.21.15.40:6839","nonce":3025057845}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6841/3350488804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":3350488804},{"type":"v1","addr":"172.21.15.40:6841","nonce":3350488804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/3438493203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":3438493203},{"type":"v1","addr":"172.21.15.174:6833","nonce":3438493203}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/3105752678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":3105752678},{"type":"v1","addr":"172.21.15.111:6833","nonce":3105752678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24461,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/2704462092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2704462092},{"type":"v1","addr":"172.21.15.111:6835","nonce":2704462092}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/3905733045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3905733045},{"type":"v1","addr":"172.21.15.174:6835","nonce":3905733045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24484,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/465579746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":465579746},{"type":"v1","addr":"172.21.15.174:6837","nonce":465579746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24494,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6837/2703764182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":2703764182},{"type":"v1","addr":"172.21.15.111:6837","nonce":2703764182}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24502,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6839/2467509274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":2467509274},{"type":"v1","addr":"172.21.15.174:6839","nonce":2467509274}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-21T23:15:55.742233+0000","modified":"2024-04-21T23:16:00.780997+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24521},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3712235548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3712235548},{"type":"v1","addr":"172.21.15.111:6839","nonce":3712235548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24521,"qdb_cluster":[24521]},"id":1}]} 2024-04-21T23:16:02.761 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 17 2024-04-21T23:16:02.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.764+0000 7f7746a66640 1 -- 172.21.15.40:0/3929145817 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f772c0a2b90 msgr2=0x7f772c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:02.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.764+0000 7f7746a66640 1 --2- 172.21.15.40:0/3929145817 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f772c0a2b90 0x7f772c0a5050 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f7740102800 tx=0x7f773800a590 comp rx=0 tx=0).stop 2024-04-21T23:16:02.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.764+0000 7f7746a66640 1 -- 172.21.15.40:0/3929145817 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7740103360 msgr2=0x7f774019f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:02.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.764+0000 7f7746a66640 1 --2- 172.21.15.40:0/3929145817 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7740103360 0x7f774019f0b0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f772800c970 tx=0x7f772800ce30 comp rx=0 tx=0).stop 2024-04-21T23:16:02.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.764+0000 7f7746a66640 1 -- 172.21.15.40:0/3929145817 shutdown_connections 2024-04-21T23:16:02.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.764+0000 7f7746a66640 1 --2- 172.21.15.40:0/3929145817 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f772c0a2b90 0x7f772c0a5050 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:02.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.764+0000 7f7746a66640 1 --2- 172.21.15.40:0/3929145817 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7740103360 0x7f774019f0b0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:02.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.764+0000 7f7746a66640 1 --2- 172.21.15.40:0/3929145817 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f77401029a0 0x7f7740198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:02.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.764+0000 7f7746a66640 1 --2- 172.21.15.40:0/3929145817 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77401017b0 0x7f7740197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:02.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.764+0000 7f7746a66640 1 -- 172.21.15.40:0/3929145817 >> 172.21.15.40:0/3929145817 conn(0x7f77400fcd40 msgr2=0x7f7740105ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:02.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.764+0000 7f7746a66640 1 -- 172.21.15.40:0/3929145817 shutdown_connections 2024-04-21T23:16:02.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.764+0000 7f7746a66640 1 -- 172.21.15.40:0/3929145817 wait complete. 2024-04-21T23:16:02.774 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T23:16:02.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.877+0000 7efc8b442640 1 -- 172.21.15.40:0/3047578286 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc84102220 msgr2=0x7efc84104610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:02.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.877+0000 7efc8b442640 1 --2- 172.21.15.40:0/3047578286 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc84102220 0x7efc84104610 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7efc80009900 tx=0x7efc8002bc30 comp rx=0 tx=0).stop 2024-04-21T23:16:02.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.877+0000 7efc8b442640 1 -- 172.21.15.40:0/3047578286 shutdown_connections 2024-04-21T23:16:02.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.877+0000 7efc8b442640 1 --2- 172.21.15.40:0/3047578286 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc84107480 0x7efc841098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:02.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.877+0000 7efc8b442640 1 --2- 172.21.15.40:0/3047578286 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efc84104b50 0x7efc84106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:02.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.877+0000 7efc8b442640 1 --2- 172.21.15.40:0/3047578286 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc84102220 0x7efc84104610 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:02.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.877+0000 7efc8b442640 1 -- 172.21.15.40:0/3047578286 >> 172.21.15.40:0/3047578286 conn(0x7efc840fbc30 msgr2=0x7efc840fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:02.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.878+0000 7efc8b442640 1 -- 172.21.15.40:0/3047578286 shutdown_connections 2024-04-21T23:16:02.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.878+0000 7efc8b442640 1 -- 172.21.15.40:0/3047578286 wait complete. 2024-04-21T23:16:02.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.878+0000 7efc8b442640 1 Processor -- start 2024-04-21T23:16:02.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.878+0000 7efc8b442640 1 -- start start 2024-04-21T23:16:02.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.878+0000 7efc8b442640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc84102220 0x7efc84101b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:02.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.878+0000 7efc8b442640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc84104b50 0x7efc841001d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:02.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.878+0000 7efc8b442640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efc84107480 0x7efc84100710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:02.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.878+0000 7efc8b442640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7efc8410d450 con 0x7efc84102220 2024-04-21T23:16:02.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.878+0000 7efc8b442640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7efc8410d2d0 con 0x7efc84104b50 2024-04-21T23:16:02.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.878+0000 7efc8b442640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7efc8410d5d0 con 0x7efc84107480 2024-04-21T23:16:02.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.878+0000 7efc891de640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc84102220 0x7efc84101b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:02.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.878+0000 7efc891de640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc84102220 0x7efc84101b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33180/0 (socket says 172.21.15.40:33180) 2024-04-21T23:16:02.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.878+0000 7efc891de640 1 -- 172.21.15.40:0/2555176576 learned_addr learned my addr 172.21.15.40:0/2555176576 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:16:02.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.879+0000 7efc889dd640 1 --2- 172.21.15.40:0/2555176576 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc84104b50 0x7efc841001d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:02.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.879+0000 7efc891de640 1 -- 172.21.15.40:0/2555176576 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efc84107480 msgr2=0x7efc84100710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:02.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.879+0000 7efc891de640 1 --2- 172.21.15.40:0/2555176576 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efc84107480 0x7efc84100710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:02.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.879+0000 7efc891de640 1 -- 172.21.15.40:0/2555176576 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc84104b50 msgr2=0x7efc841001d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:02.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.879+0000 7efc891de640 1 --2- 172.21.15.40:0/2555176576 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc84104b50 0x7efc841001d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:02.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.879+0000 7efc891de640 1 -- 172.21.15.40:0/2555176576 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc84100fa0 con 0x7efc84102220 2024-04-21T23:16:02.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.879+0000 7efc889dd640 1 --2- 172.21.15.40:0/2555176576 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc84104b50 0x7efc841001d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:16:02.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.879+0000 7efc891de640 1 --2- 172.21.15.40:0/2555176576 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc84102220 0x7efc84101b90 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7efc80009230 tx=0x7efc80005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:02.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.879+0000 7efc7a7fc640 1 -- 172.21.15.40:0/2555176576 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc80041070 con 0x7efc84102220 2024-04-21T23:16:02.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.879+0000 7efc7a7fc640 1 -- 172.21.15.40:0/2555176576 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc800054e0 con 0x7efc84102220 2024-04-21T23:16:02.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.879+0000 7efc8b442640 1 -- 172.21.15.40:0/2555176576 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc841a1b50 con 0x7efc84102220 2024-04-21T23:16:02.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.879+0000 7efc7a7fc640 1 -- 172.21.15.40:0/2555176576 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc80034440 con 0x7efc84102220 2024-04-21T23:16:02.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.879+0000 7efc8b442640 1 -- 172.21.15.40:0/2555176576 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc841a20f0 con 0x7efc84102220 2024-04-21T23:16:02.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.880+0000 7efc8b442640 1 -- 172.21.15.40:0/2555176576 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc840fd720 con 0x7efc84102220 2024-04-21T23:16:02.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.881+0000 7efc7a7fc640 1 -- 172.21.15.40:0/2555176576 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7efc80044070 con 0x7efc84102220 2024-04-21T23:16:02.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.882+0000 7efc7a7fc640 1 --2- 172.21.15.40:0/2555176576 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7efc700a2c10 0x7efc700a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:02.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.882+0000 7efc889dd640 1 --2- 172.21.15.40:0/2555176576 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7efc700a2c10 0x7efc700a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:02.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.882+0000 7efc7a7fc640 1 -- 172.21.15.40:0/2555176576 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7efc800f0fa0 con 0x7efc84102220 2024-04-21T23:16:02.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.882+0000 7efc889dd640 1 --2- 172.21.15.40:0/2555176576 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7efc700a2c10 0x7efc700a50d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7efc84101900 tx=0x7efc74005e50 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:02.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:02.887+0000 7efc7a7fc640 1 -- 172.21.15.40:0/2555176576 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7efc800bdcc0 con 0x7efc84102220 2024-04-21T23:16:03.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.076+0000 7efc8b442640 1 -- 172.21.15.40:0/2555176576 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7efc8402d390 con 0x7efc84102220 2024-04-21T23:16:03.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.077+0000 7efc7a7fc640 1 -- 172.21.15.40:0/2555176576 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24138 (secure 0 0 0) 0x7efc800bd5c0 con 0x7efc84102220 2024-04-21T23:16:03.078 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:16:03.079 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":90,"fsid":"d814e4dc-0033-11ef-bc93-c7b262605968","created":"2024-04-21T23:07:52.659207+0000","modified":"2024-04-21T23:16:00.780824+0000","last_up_change":"2024-04-21T23:15:05.308821+0000","last_in_change":"2024-04-21T23:14:45.447491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:11:14.824775+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T23:15:54.322571+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T23:15:55.164493+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T23:15:58.838266+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"0e8d692a-22fe-4a87-8d99-9151f96cbb02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6803","nonce":2009758784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6805","nonce":2009758784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6809","nonce":2009758784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6807","nonce":2009758784}]},"public_addr":"172.21.15.40:6803/2009758784","cluster_addr":"172.21.15.40:6805/2009758784","heartbeat_back_addr":"172.21.15.40:6809/2009758784","heartbeat_front_addr":"172.21.15.40:6807/2009758784","state":["exists","up"]},{"osd":1,"uuid":"fe5fe11b-0e84-4359-83a7-1ba45105755a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6801","nonce":1763903836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6803","nonce":1763903836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6807","nonce":1763903836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6805","nonce":1763903836}]},"public_addr":"172.21.15.111:6801/1763903836","cluster_addr":"172.21.15.111:6803/1763903836","heartbeat_back_addr":"172.21.15.111:6807/1763903836","heartbeat_front_addr":"172.21.15.111:6805/1763903836","state":["exists","up"]},{"osd":2,"uuid":"813ad735-5cc2-4da0-ae16-6db238395fbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6801","nonce":439695825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6803","nonce":439695825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6807","nonce":439695825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6805","nonce":439695825}]},"public_addr":"172.21.15.174:6801/439695825","cluster_addr":"172.21.15.174:6803/439695825","heartbeat_back_addr":"172.21.15.174:6807/439695825","heartbeat_front_addr":"172.21.15.174:6805/439695825","state":["exists","up"]},{"osd":3,"uuid":"09becb03-b7ce-4ffc-917d-b7900ec37e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6811","nonce":3454312436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6813","nonce":3454312436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6817","nonce":3454312436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6815","nonce":3454312436}]},"public_addr":"172.21.15.40:6811/3454312436","cluster_addr":"172.21.15.40:6813/3454312436","heartbeat_back_addr":"172.21.15.40:6817/3454312436","heartbeat_front_addr":"172.21.15.40:6815/3454312436","state":["exists","up"]},{"osd":4,"uuid":"46eabfd7-d88a-4efd-94eb-04354d3a9d7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6809","nonce":1810866070}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6811","nonce":1810866070}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6815","nonce":1810866070}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6813","nonce":1810866070}]},"public_addr":"172.21.15.111:6809/1810866070","cluster_addr":"172.21.15.111:6811/1810866070","heartbeat_back_addr":"172.21.15.111:6815/1810866070","heartbeat_front_addr":"172.21.15.111:6813/1810866070","state":["exists","up"]},{"osd":5,"uuid":"6f73b582-6c65-41db-aab0-9b360f83d5c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6809","nonce":3752986711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6811","nonce":3752986711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6815","nonce":3752986711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6813","nonce":3752986711}]},"public_addr":"172.21.15.174:6809/3752986711","cluster_addr":"172.21.15.174:6811/3752986711","heartbeat_back_addr":"172.21.15.174:6815/3752986711","heartbeat_front_addr":"172.21.15.174:6813/3752986711","state":["exists","up"]},{"osd":6,"uuid":"39f305f0-12d2-4732-aaa0-cd66700af538","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6819","nonce":1618855427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6821","nonce":1618855427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6825","nonce":1618855427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6823","nonce":1618855427}]},"public_addr":"172.21.15.40:6819/1618855427","cluster_addr":"172.21.15.40:6821/1618855427","heartbeat_back_addr":"172.21.15.40:6825/1618855427","heartbeat_front_addr":"172.21.15.40:6823/1618855427","state":["exists","up"]},{"osd":7,"uuid":"769f2468-5102-490b-89f8-d0444d3169a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6817","nonce":927054162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6819","nonce":927054162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6823","nonce":927054162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6821","nonce":927054162}]},"public_addr":"172.21.15.111:6817/927054162","cluster_addr":"172.21.15.111:6819/927054162","heartbeat_back_addr":"172.21.15.111:6823/927054162","heartbeat_front_addr":"172.21.15.111:6821/927054162","state":["exists","up"]},{"osd":8,"uuid":"1a444e80-007d-473b-b972-4c814a813c22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6817","nonce":3817848746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6819","nonce":3817848746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6823","nonce":3817848746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6821","nonce":3817848746}]},"public_addr":"172.21.15.174:6817/3817848746","cluster_addr":"172.21.15.174:6819/3817848746","heartbeat_back_addr":"172.21.15.174:6823/3817848746","heartbeat_front_addr":"172.21.15.174:6821/3817848746","state":["exists","up"]},{"osd":9,"uuid":"9a6b6a56-6ae5-49fe-9e5b-cb17d686e0dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6827","nonce":1649446531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6829","nonce":1649446531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6833","nonce":1649446531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6831","nonce":1649446531}]},"public_addr":"172.21.15.40:6827/1649446531","cluster_addr":"172.21.15.40:6829/1649446531","heartbeat_back_addr":"172.21.15.40:6833/1649446531","heartbeat_front_addr":"172.21.15.40:6831/1649446531","state":["exists","up"]},{"osd":10,"uuid":"a70038ba-e2ad-4170-85cc-ea0470a2edef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6825","nonce":1227538935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6827","nonce":1227538935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6831","nonce":1227538935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6829","nonce":1227538935}]},"public_addr":"172.21.15.111:6825/1227538935","cluster_addr":"172.21.15.111:6827/1227538935","heartbeat_back_addr":"172.21.15.111:6831/1227538935","heartbeat_front_addr":"172.21.15.111:6829/1227538935","state":["exists","up"]},{"osd":11,"uuid":"781532bb-7a5e-44e3-8eb7-b584b0054666","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6825","nonce":2045865553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6827","nonce":2045865553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6831","nonce":2045865553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6829","nonce":2045865553}]},"public_addr":"172.21.15.174:6825/2045865553","cluster_addr":"172.21.15.174:6827/2045865553","heartbeat_back_addr":"172.21.15.174:6831/2045865553","heartbeat_front_addr":"172.21.15.174:6829/2045865553","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:10:24.761290+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:10:48.565428+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:11:12.307818+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:11:37.951529+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:02.832523+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:27.568879+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:52.997707+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:13:18.937274+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:13:44.708657+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:14:10.833855+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:14:37.078272+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:15:03.878076+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:6801/526554319":"2024-04-22T23:08:50.305770+0000","172.21.15.40:6800/526554319":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/3178763471":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/4045911364":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/276300080":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/3689649814":"2024-04-22T23:08:23.743030+0000","172.21.15.40:0/2241116715":"2024-04-22T23:08:23.743030+0000","172.21.15.40:6801/1479040326":"2024-04-22T23:08:23.743030+0000","172.21.15.40:6800/1479040326":"2024-04-22T23:08:23.743030+0000","172.21.15.40:0/681298099":"2024-04-22T23:08:23.743030+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T23:16:03.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.079+0000 7efc8b442640 1 -- 172.21.15.40:0/2555176576 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7efc700a2c10 msgr2=0x7efc700a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:03.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.079+0000 7efc8b442640 1 --2- 172.21.15.40:0/2555176576 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7efc700a2c10 0x7efc700a50d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7efc84101900 tx=0x7efc74005e50 comp rx=0 tx=0).stop 2024-04-21T23:16:03.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.079+0000 7efc8b442640 1 -- 172.21.15.40:0/2555176576 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc84102220 msgr2=0x7efc84101b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:03.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.079+0000 7efc8b442640 1 --2- 172.21.15.40:0/2555176576 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc84102220 0x7efc84101b90 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7efc80009230 tx=0x7efc80005dd0 comp rx=0 tx=0).stop 2024-04-21T23:16:03.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.080+0000 7efc8b442640 1 -- 172.21.15.40:0/2555176576 shutdown_connections 2024-04-21T23:16:03.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.080+0000 7efc8b442640 1 --2- 172.21.15.40:0/2555176576 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7efc700a2c10 0x7efc700a50d0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:03.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.080+0000 7efc8b442640 1 --2- 172.21.15.40:0/2555176576 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc84104b50 0x7efc841001d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:03.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.080+0000 7efc8b442640 1 --2- 172.21.15.40:0/2555176576 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efc84107480 0x7efc84100710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:03.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.080+0000 7efc8b442640 1 --2- 172.21.15.40:0/2555176576 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc84102220 0x7efc84101b90 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:03.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.080+0000 7efc8b442640 1 -- 172.21.15.40:0/2555176576 >> 172.21.15.40:0/2555176576 conn(0x7efc840fbc30 msgr2=0x7efc84102b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:03.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.080+0000 7efc8b442640 1 -- 172.21.15.40:0/2555176576 shutdown_connections 2024-04-21T23:16:03.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.080+0000 7efc8b442640 1 -- 172.21.15.40:0/2555176576 wait complete. 2024-04-21T23:16:03.090 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-21T23:15:55.742233+0000', 'modified': '2024-04-21T23:16:00.780997+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24521}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24521': {'gid': 24521, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.111:6839/3712235548', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6838', 'nonce': 3712235548}, {'type': 'v1', 'addr': '172.21.15.111:6839', 'nonce': 3712235548}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 24521, 'qdb_cluster': [24521]} 2024-04-21T23:16:03.090 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-21T23:16:03.091 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-21T23:16:03.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.193+0000 7f8a329af640 1 -- 172.21.15.40:0/2485335367 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a2c102230 msgr2=0x7f8a2c108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:03.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.193+0000 7f8a329af640 1 --2- 172.21.15.40:0/2485335367 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a2c102230 0x7f8a2c108ac0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f8a24009ca0 tx=0x7f8a2402c0f0 comp rx=0 tx=0).stop 2024-04-21T23:16:03.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.193+0000 7f8a329af640 1 -- 172.21.15.40:0/2485335367 shutdown_connections 2024-04-21T23:16:03.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.193+0000 7f8a329af640 1 --2- 172.21.15.40:0/2485335367 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a2c101870 0x7f8a2c101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:03.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.193+0000 7f8a329af640 1 --2- 172.21.15.40:0/2485335367 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a2c102230 0x7f8a2c108ac0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:03.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.193+0000 7f8a329af640 1 --2- 172.21.15.40:0/2485335367 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a2c100680 0x7f8a2c100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:03.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.193+0000 7f8a329af640 1 -- 172.21.15.40:0/2485335367 >> 172.21.15.40:0/2485335367 conn(0x7f8a2c0fbbf0 msgr2=0x7f8a2c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:03.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.193+0000 7f8a329af640 1 -- 172.21.15.40:0/2485335367 shutdown_connections 2024-04-21T23:16:03.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.193+0000 7f8a329af640 1 -- 172.21.15.40:0/2485335367 wait complete. 2024-04-21T23:16:03.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.194+0000 7f8a329af640 1 Processor -- start 2024-04-21T23:16:03.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.194+0000 7f8a329af640 1 -- start start 2024-04-21T23:16:03.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.194+0000 7f8a329af640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a2c100680 0x7f8a2c199c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:03.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.194+0000 7f8a329af640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a2c101870 0x7f8a2c19a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:03.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.194+0000 7f8a329af640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a2c102230 0x7f8a2c1a1230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:03.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.194+0000 7f8a329af640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8a2c10c300 con 0x7f8a2c102230 2024-04-21T23:16:03.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.194+0000 7f8a329af640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f8a2c10c180 con 0x7f8a2c101870 2024-04-21T23:16:03.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.194+0000 7f8a329af640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8a2c10c480 con 0x7f8a2c100680 2024-04-21T23:16:03.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.194+0000 7f8a30f4c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a2c102230 0x7f8a2c1a1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:03.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.194+0000 7f8a30f4c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a2c102230 0x7f8a2c1a1230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33208/0 (socket says 172.21.15.40:33208) 2024-04-21T23:16:03.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.194+0000 7f8a30f4c640 1 -- 172.21.15.40:0/357065638 learned_addr learned my addr 172.21.15.40:0/357065638 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:16:03.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.195+0000 7f8a2bfff640 1 --2- 172.21.15.40:0/357065638 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a2c100680 0x7f8a2c199c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:03.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.195+0000 7f8a2b7fe640 1 --2- 172.21.15.40:0/357065638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a2c101870 0x7f8a2c19a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:03.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.195+0000 7f8a30f4c640 1 -- 172.21.15.40:0/357065638 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a2c100680 msgr2=0x7f8a2c199c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:03.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.195+0000 7f8a30f4c640 1 --2- 172.21.15.40:0/357065638 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a2c100680 0x7f8a2c199c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:03.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.195+0000 7f8a30f4c640 1 -- 172.21.15.40:0/357065638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a2c101870 msgr2=0x7f8a2c19a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:03.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.195+0000 7f8a30f4c640 1 --2- 172.21.15.40:0/357065638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a2c101870 0x7f8a2c19a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:03.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.195+0000 7f8a30f4c640 1 -- 172.21.15.40:0/357065638 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a2c1a18d0 con 0x7f8a2c102230 2024-04-21T23:16:03.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.195+0000 7f8a30f4c640 1 --2- 172.21.15.40:0/357065638 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a2c102230 0x7f8a2c1a1230 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f8a2402c700 tx=0x7f8a24032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:03.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.195+0000 7f8a297fa640 1 -- 172.21.15.40:0/357065638 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a24039040 con 0x7f8a2c102230 2024-04-21T23:16:03.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.195+0000 7f8a329af640 1 -- 172.21.15.40:0/357065638 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a2c1a1b60 con 0x7f8a2c102230 2024-04-21T23:16:03.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.195+0000 7f8a297fa640 1 -- 172.21.15.40:0/357065638 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a24044070 con 0x7f8a2c102230 2024-04-21T23:16:03.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.195+0000 7f8a297fa640 1 -- 172.21.15.40:0/357065638 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a240042a0 con 0x7f8a2c102230 2024-04-21T23:16:03.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.195+0000 7f8a329af640 1 -- 172.21.15.40:0/357065638 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a2c1a2050 con 0x7f8a2c102230 2024-04-21T23:16:03.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.197+0000 7f8a297fa640 1 -- 172.21.15.40:0/357065638 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f8a24003970 con 0x7f8a2c102230 2024-04-21T23:16:03.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.197+0000 7f8a35590740 1 Processor -- start 2024-04-21T23:16:03.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.197+0000 7f8a35590740 1 -- start start 2024-04-21T23:16:03.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.198+0000 7f8a35590740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4f3615070 0x55b4f3615490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:03.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.198+0000 7f8a35590740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4f3615a60 0x55b4f361a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:03.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.198+0000 7f8a35590740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b4f361abd0 0x55b4f361d060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:03.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.198+0000 7f8a35590740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55b4f3579df0 con 0x55b4f3615a60 2024-04-21T23:16:03.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.198+0000 7f8a35590740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55b4f3522640 con 0x55b4f361abd0 2024-04-21T23:16:03.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.198+0000 7f8a35590740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55b4f3575db0 con 0x55b4f3615070 2024-04-21T23:16:03.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.198+0000 7f8a297fa640 1 --2- 172.21.15.40:0/357065638 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f8a180a2b90 0x7f8a180a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:03.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.198+0000 7f8a2bfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4f3615a60 0x55b4f361a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:03.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.198+0000 7f8a2bfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4f3615a60 0x55b4f361a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33212/0 (socket says 172.21.15.40:33212) 2024-04-21T23:16:03.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.198+0000 7f8a2bfff640 1 -- 172.21.15.40:0/3916981839 learned_addr learned my addr 172.21.15.40:0/3916981839 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:16:03.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.198+0000 7f8a30f4c640 1 --2- 172.21.15.40:0/357065638 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f8a180a2b90 0x7f8a180a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:03.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.198+0000 7f8a2b7fe640 1 --2- 172.21.15.40:0/3916981839 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b4f361abd0 0x55b4f361d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:03.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.198+0000 7f8a297fa640 1 -- 172.21.15.40:0/357065638 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f8a240f18b0 con 0x7f8a2c102230 2024-04-21T23:16:03.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.198+0000 7f8a2b7fe640 1 --2- 172.21.15.40:0/3916981839 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4f3615070 0x55b4f3615490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:03.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.198+0000 7f8a2bfff640 1 -- 172.21.15.40:0/3916981839 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4f3615070 msgr2=0x55b4f3615490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:03.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.198+0000 7f8a2bfff640 1 --2- 172.21.15.40:0/3916981839 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4f3615070 0x55b4f3615490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:03.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.198+0000 7f8a2bfff640 1 -- 172.21.15.40:0/3916981839 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b4f361abd0 msgr2=0x55b4f361d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:03.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.198+0000 7f8a2bfff640 1 --2- 172.21.15.40:0/3916981839 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b4f361abd0 0x55b4f361d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:03.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.198+0000 7f8a2bfff640 1 -- 172.21.15.40:0/3916981839 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b4f328e2d0 con 0x55b4f3615a60 2024-04-21T23:16:03.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.198+0000 7f8a2b7fe640 1 --2- 172.21.15.40:0/3916981839 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4f3615070 0x55b4f3615490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:16:03.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.198+0000 7f8a2bfff640 1 --2- 172.21.15.40:0/3916981839 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4f3615a60 0x55b4f361a560 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f8a1400bc60 tx=0x7f8a1400f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96c6dcf75890ec3f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:03.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.198+0000 7f8a227fc640 1 -- 172.21.15.40:0/3916981839 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a14015070 con 0x55b4f3615a60 2024-04-21T23:16:03.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.198+0000 7f8a30f4c640 1 --2- 172.21.15.40:0/357065638 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f8a180a2b90 0x7f8a180a5050 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f8a240091b0 tx=0x7f8a24088ad0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:03.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.199+0000 7f8a227fc640 1 -- 172.21.15.40:0/3916981839 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a14011920 con 0x55b4f3615a60 2024-04-21T23:16:03.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.199+0000 7f8a227fc640 1 -- 172.21.15.40:0/3916981839 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a14010890 con 0x55b4f3615a60 2024-04-21T23:16:03.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.199+0000 7f8a35590740 1 -- 172.21.15.40:0/3916981839 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4f3615a60 msgr2=0x55b4f361a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:03.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.199+0000 7f8a35590740 1 --2- 172.21.15.40:0/3916981839 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4f3615a60 0x55b4f361a560 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f8a1400bc60 tx=0x7f8a1400f730 comp rx=0 tx=0).stop 2024-04-21T23:16:03.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.199+0000 7f8a35590740 1 -- 172.21.15.40:0/3916981839 shutdown_connections 2024-04-21T23:16:03.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.199+0000 7f8a35590740 1 --2- 172.21.15.40:0/3916981839 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b4f361abd0 0x55b4f361d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:03.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.199+0000 7f8a35590740 1 --2- 172.21.15.40:0/3916981839 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4f3615a60 0x55b4f361a560 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:03.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.199+0000 7f8a35590740 1 --2- 172.21.15.40:0/3916981839 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4f3615070 0x55b4f3615490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:03.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.199+0000 7f8a35590740 1 -- 172.21.15.40:0/3916981839 >> 172.21.15.40:0/3916981839 conn(0x55b4f36105e0 msgr2=0x55b4f36109e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:03.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.199+0000 7f8a35590740 1 -- 172.21.15.40:0/3916981839 shutdown_connections 2024-04-21T23:16:03.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.199+0000 7f8a35590740 1 -- 172.21.15.40:0/3916981839 wait complete. 2024-04-21T23:16:03.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.199+0000 7f8a35590740 1 -- start start 2024-04-21T23:16:03.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.200+0000 7f8a35590740 1 Processor -- start 2024-04-21T23:16:03.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.200+0000 7f8a35590740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4f3615070 0x55b4f361f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:03.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.200+0000 7f8a35590740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4f3615a60 0x55b4f361fb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:03.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.200+0000 7f8a35590740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b4f361abd0 0x55b4f3626b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:03.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.200+0000 7f8a35590740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55b4f361db80 con 0x55b4f3615a60 2024-04-21T23:16:03.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.200+0000 7f8a35590740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55b4f3535be0 con 0x55b4f361abd0 2024-04-21T23:16:03.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.200+0000 7f8a35590740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55b4f361dd00 con 0x55b4f3615070 2024-04-21T23:16:03.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.200+0000 7f8a35590740 10 client.0 _ll_register_callbacks cb 0x55b4f35355d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:16:03.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.200+0000 7f8a2bfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4f3615a60 0x55b4f361fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:03.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.200+0000 7f8a2bfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4f3615a60 0x55b4f361fb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33214/0 (socket says 172.21.15.40:33214) 2024-04-21T23:16:03.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.200+0000 7f8a2bfff640 1 -- 172.21.15.40:0/2481651298 learned_addr learned my addr 172.21.15.40:0/2481651298 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:16:03.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.200+0000 7f8a2b7fe640 1 --2- 172.21.15.40:0/2481651298 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4f3615070 0x55b4f361f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:03.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.200+0000 7f8a2b7fe640 1 --2- 172.21.15.40:0/2481651298 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b4f361abd0 0x55b4f3626b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:03.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.200+0000 7f8a2bfff640 1 -- 172.21.15.40:0/2481651298 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4f3615070 msgr2=0x55b4f361f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:03.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.200+0000 7f8a2bfff640 1 --2- 172.21.15.40:0/2481651298 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4f3615070 0x55b4f361f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:03.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.200+0000 7f8a2bfff640 1 -- 172.21.15.40:0/2481651298 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b4f361abd0 msgr2=0x55b4f3626b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:03.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.200+0000 7f8a2bfff640 1 --2- 172.21.15.40:0/2481651298 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b4f361abd0 0x55b4f3626b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:03.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.200+0000 7f8a2bfff640 1 -- 172.21.15.40:0/2481651298 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b4f33586b0 con 0x55b4f3615a60 2024-04-21T23:16:03.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.200+0000 7f8a2b7fe640 1 --2- 172.21.15.40:0/2481651298 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b4f361abd0 0x55b4f3626b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:16:03.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.201+0000 7f8a2bfff640 1 --2- 172.21.15.40:0/2481651298 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4f3615a60 0x55b4f361fb20 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f8a14011650 tx=0x7f8a14011680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bf67c0e76bc087f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:03.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.201+0000 7f8a0b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-21T23:16:03.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.201+0000 7f8a0b7fe640 1 -- 172.21.15.40:0/2481651298 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a14015070 con 0x55b4f3615a60 2024-04-21T23:16:03.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.201+0000 7f8a0b7fe640 1 -- 172.21.15.40:0/2481651298 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f8a14011ce0 con 0x55b4f3615a60 2024-04-21T23:16:03.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.201+0000 7f8a35590740 1 -- 172.21.15.40:0/2481651298 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a14011920 con 0x55b4f3615a60 2024-04-21T23:16:03.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.201+0000 7f8a0b7fe640 1 -- 172.21.15.40:0/2481651298 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f8a1402e050 con 0x55b4f3615a60 2024-04-21T23:16:03.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.201+0000 7f8a35590740 10 client.14754 fetch_fsmap learned FSMap version 17 2024-04-21T23:16:03.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.201+0000 7f8a35590740 1 -- 172.21.15.40:0/2481651298 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x55b4f3629ab0 con 0x55b4f3615a60 2024-04-21T23:16:03.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.201+0000 7f8a0b7fe640 1 -- 172.21.15.40:0/2481651298 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a14011920 con 0x55b4f3615a60 2024-04-21T23:16:03.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.201+0000 7f8a0b7fe640 1 -- 172.21.15.40:0/2481651298 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7f8a14004ac0 con 0x55b4f3615a60 2024-04-21T23:16:03.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.201+0000 7f8a35590740 10 client.14754 fetch_fsmap finished waiting for FSMap version 17 2024-04-21T23:16:03.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.201+0000 7f8a35590740 10 client.14754 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-21T23:16:03.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.201+0000 7f8a35590740 20 client.14754 populate_metadata read hostname 'smithi040' 2024-04-21T23:16:03.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.201+0000 7f8a35590740 1 --2- 172.21.15.40:0/2481651298 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55b4f362bd10 0x55b4f362d180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:03.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.201+0000 7f8a35590740 4 client.14754 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:16:03.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.201+0000 7f8a35590740 1 -- 172.21.15.40:0/2481651298 --> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b4f3587330 con 0x55b4f362bd10 2024-04-21T23:16:03.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.202+0000 7f8a2b7fe640 1 --2- 172.21.15.40:0/2481651298 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55b4f362bd10 0x55b4f362d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:03.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.202+0000 7f8a2b7fe640 1 --2- 172.21.15.40:0/2481651298 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55b4f362bd10 0x55b4f362d180 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=653aad6e49b71be0 server_cookie=9552c8ec86c17e93 in_seq=0 out_seq=0 2024-04-21T23:16:03.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.202+0000 7f8a0b7fe640 10 client.14754 ms_handle_connect on v2:172.21.15.111:6838/3712235548 2024-04-21T23:16:03.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.203+0000 7f8a0b7fe640 1 -- 172.21.15.40:0/2481651298 <== mds.0 v2:172.21.15.111:6838/3712235548 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b4f3587330 con 0x55b4f362bd10 2024-04-21T23:16:03.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.203+0000 7f8a0b7fe640 10 client.14754 handle_command_reply: tid=0 2024-04-21T23:16:03.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.203+0000 7f8a35590740 1 client.14754 shutdown 2024-04-21T23:16:03.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.203+0000 7f8a35590740 1 -- 172.21.15.40:0/2481651298 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4f3615a60 msgr2=0x55b4f361fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:03.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.203+0000 7f8a35590740 1 --2- 172.21.15.40:0/2481651298 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4f3615a60 0x55b4f361fb20 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f8a14011650 tx=0x7f8a14011680 comp rx=0 tx=0).stop 2024-04-21T23:16:03.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.203+0000 7f8a35590740 1 -- 172.21.15.40:0/2481651298 shutdown_connections 2024-04-21T23:16:03.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.203+0000 7f8a35590740 1 --2- 172.21.15.40:0/2481651298 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55b4f362bd10 0x55b4f362d180 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:03.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.203+0000 7f8a35590740 1 --2- 172.21.15.40:0/2481651298 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b4f361abd0 0x55b4f3626b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:03.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.203+0000 7f8a35590740 1 --2- 172.21.15.40:0/2481651298 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4f3615a60 0x55b4f361fb20 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:03.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.203+0000 7f8a0b7fe640 0 client.14754 ms_handle_reset on v2:172.21.15.111:6838/3712235548 2024-04-21T23:16:03.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.203+0000 7f8a35590740 1 --2- 172.21.15.40:0/2481651298 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4f3615070 0x55b4f361f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:03.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.203+0000 7f8a35590740 1 -- 172.21.15.40:0/2481651298 >> 172.21.15.40:0/2481651298 conn(0x55b4f36105e0 msgr2=0x55b4f360f720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:03.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.203+0000 7f8a35590740 1 -- 172.21.15.40:0/2481651298 shutdown_connections 2024-04-21T23:16:03.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.203+0000 7f8a35590740 1 -- 172.21.15.40:0/2481651298 wait complete. 2024-04-21T23:16:03.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.203+0000 7f8a35590740 20 client.14754 trim_cache size 0 max 16384 2024-04-21T23:16:03.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.222+0000 7f8a35590740 1 Processor -- start 2024-04-21T23:16:03.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.222+0000 7f8a35590740 1 -- start start 2024-04-21T23:16:03.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.222+0000 7f8a35590740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4f3615070 0x55b4f361f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:03.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.222+0000 7f8a35590740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4f3615a60 0x55b4f3626b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:03.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.222+0000 7f8a35590740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b4f361abd0 0x55b4f361f730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:03.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.222+0000 7f8a35590740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55b4f362b4c0 con 0x55b4f3615a60 2024-04-21T23:16:03.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.222+0000 7f8a35590740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55b4f362bb90 con 0x55b4f361abd0 2024-04-21T23:16:03.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.222+0000 7f8a35590740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55b4f362c260 con 0x55b4f3615070 2024-04-21T23:16:03.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.222+0000 7f8a2bfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4f3615a60 0x55b4f3626b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:03.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.222+0000 7f8a2bfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4f3615a60 0x55b4f3626b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33222/0 (socket says 172.21.15.40:33222) 2024-04-21T23:16:03.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.222+0000 7f8a2bfff640 1 -- 172.21.15.40:0/2244032791 learned_addr learned my addr 172.21.15.40:0/2244032791 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:16:03.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.222+0000 7f8a2b7fe640 1 --2- 172.21.15.40:0/2244032791 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4f3615070 0x55b4f361f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:03.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.222+0000 7f8a2bfff640 1 -- 172.21.15.40:0/2244032791 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4f3615070 msgr2=0x55b4f361f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:03.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.222+0000 7f8a2bfff640 1 --2- 172.21.15.40:0/2244032791 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4f3615070 0x55b4f361f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:03.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.222+0000 7f8a2bfff640 1 -- 172.21.15.40:0/2244032791 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b4f361abd0 msgr2=0x55b4f361f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:03.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.222+0000 7f8a2bfff640 1 --2- 172.21.15.40:0/2244032791 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b4f361abd0 0x55b4f361f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:03.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.222+0000 7f8a2bfff640 1 -- 172.21.15.40:0/2244032791 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b4f3615e40 con 0x55b4f3615a60 2024-04-21T23:16:03.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.222+0000 7f8a2bfff640 1 --2- 172.21.15.40:0/2244032791 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4f3615a60 0x55b4f3626b20 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f8a14004ee0 tx=0x7f8a1400b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=498d8bd0b1776dd0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:03.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.222+0000 7f8a21ffb640 1 -- 172.21.15.40:0/2244032791 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a14037070 con 0x55b4f3615a60 2024-04-21T23:16:03.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.223+0000 7f8a21ffb640 1 -- 172.21.15.40:0/2244032791 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a1400fc30 con 0x55b4f3615a60 2024-04-21T23:16:03.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.223+0000 7f8a21ffb640 1 -- 172.21.15.40:0/2244032791 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a14015070 con 0x55b4f3615a60 2024-04-21T23:16:03.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.223+0000 7f8a35590740 1 -- 172.21.15.40:0/2244032791 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4f3615a60 msgr2=0x55b4f3626b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:03.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.223+0000 7f8a35590740 1 --2- 172.21.15.40:0/2244032791 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4f3615a60 0x55b4f3626b20 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f8a14004ee0 tx=0x7f8a1400b480 comp rx=0 tx=0).stop 2024-04-21T23:16:03.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.223+0000 7f8a35590740 1 -- 172.21.15.40:0/2244032791 shutdown_connections 2024-04-21T23:16:03.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.223+0000 7f8a35590740 1 --2- 172.21.15.40:0/2244032791 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b4f361abd0 0x55b4f361f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:03.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.223+0000 7f8a35590740 1 --2- 172.21.15.40:0/2244032791 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4f3615a60 0x55b4f3626b20 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:03.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.223+0000 7f8a35590740 1 --2- 172.21.15.40:0/2244032791 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4f3615070 0x55b4f361f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:03.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.223+0000 7f8a35590740 1 -- 172.21.15.40:0/2244032791 >> 172.21.15.40:0/2244032791 conn(0x55b4f36105e0 msgr2=0x55b4f36109c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:03.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.223+0000 7f8a35590740 1 -- 172.21.15.40:0/2244032791 shutdown_connections 2024-04-21T23:16:03.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.223+0000 7f8a35590740 1 -- 172.21.15.40:0/2244032791 wait complete. 2024-04-21T23:16:03.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.223+0000 7f8a35590740 1 -- start start 2024-04-21T23:16:03.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.224+0000 7f8a35590740 1 Processor -- start 2024-04-21T23:16:03.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.224+0000 7f8a35590740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b4f3615070 0x55b4f36221a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:03.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.224+0000 7f8a35590740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4f3615a60 0x55b4f36226e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:03.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.224+0000 7f8a35590740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4f361abd0 0x55b4f3625700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:03.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.224+0000 7f8a35590740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55b4f362d000 con 0x55b4f3615a60 2024-04-21T23:16:03.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.224+0000 7f8a35590740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55b4f362c930 con 0x55b4f3615070 2024-04-21T23:16:03.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.224+0000 7f8a35590740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55b4f362d6d0 con 0x55b4f361abd0 2024-04-21T23:16:03.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.224+0000 7f8a35590740 10 client.0 _ll_register_callbacks cb 0x7f8a1400b9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:16:03.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.224+0000 7f8a2bfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4f3615a60 0x55b4f36226e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:03.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.224+0000 7f8a2bfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4f3615a60 0x55b4f36226e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33226/0 (socket says 172.21.15.40:33226) 2024-04-21T23:16:03.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.224+0000 7f8a2bfff640 1 -- 172.21.15.40:0/1654716488 learned_addr learned my addr 172.21.15.40:0/1654716488 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:16:03.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.224+0000 7f8a2b7fe640 1 --2- 172.21.15.40:0/1654716488 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4f361abd0 0x55b4f3625700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:03.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.224+0000 7f8a2b7fe640 1 --2- 172.21.15.40:0/1654716488 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b4f3615070 0x55b4f36221a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:03.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.224+0000 7f8a2bfff640 1 -- 172.21.15.40:0/1654716488 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4f361abd0 msgr2=0x55b4f3625700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:03.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.224+0000 7f8a2bfff640 1 --2- 172.21.15.40:0/1654716488 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4f361abd0 0x55b4f3625700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:03.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.224+0000 7f8a2bfff640 1 -- 172.21.15.40:0/1654716488 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b4f3615070 msgr2=0x55b4f36221a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:03.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.224+0000 7f8a2bfff640 1 --2- 172.21.15.40:0/1654716488 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b4f3615070 0x55b4f36221a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:03.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.224+0000 7f8a2bfff640 1 -- 172.21.15.40:0/1654716488 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b4f3625cf0 con 0x55b4f3615a60 2024-04-21T23:16:03.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.224+0000 7f8a2bfff640 1 --2- 172.21.15.40:0/1654716488 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4f3615a60 0x55b4f36226e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f8a1400bc60 tx=0x7f8a1400bef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cb0645f8e5c9005 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:03.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.225+0000 7f8a0b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-21T23:16:03.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.225+0000 7f8a0b7fe640 1 -- 172.21.15.40:0/1654716488 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a14015050 con 0x55b4f3615a60 2024-04-21T23:16:03.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.225+0000 7f8a0b7fe640 1 -- 172.21.15.40:0/1654716488 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f8a14004040 con 0x55b4f3615a60 2024-04-21T23:16:03.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.225+0000 7f8a35590740 1 -- 172.21.15.40:0/1654716488 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a1400fc30 con 0x55b4f3615a60 2024-04-21T23:16:03.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.225+0000 7f8a0b7fe640 1 -- 172.21.15.40:0/1654716488 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f8a14037040 con 0x55b4f3615a60 2024-04-21T23:16:03.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.225+0000 7f8a35590740 10 client.14760 fetch_fsmap learned FSMap version 17 2024-04-21T23:16:03.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.225+0000 7f8a35590740 1 -- 172.21.15.40:0/1654716488 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x55b4f3691cc0 con 0x55b4f3615a60 2024-04-21T23:16:03.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.225+0000 7f8a0b7fe640 1 -- 172.21.15.40:0/1654716488 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a1400fc30 con 0x55b4f3615a60 2024-04-21T23:16:03.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.225+0000 7f8a0b7fe640 1 -- 172.21.15.40:0/1654716488 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7f8a14043030 con 0x55b4f3615a60 2024-04-21T23:16:03.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.225+0000 7f8a35590740 10 client.14760 fetch_fsmap finished waiting for FSMap version 17 2024-04-21T23:16:03.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.225+0000 7f8a35590740 10 client.14760 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-21T23:16:03.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.225+0000 7f8a35590740 20 client.14760 populate_metadata read hostname 'smithi040' 2024-04-21T23:16:03.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.225+0000 7f8a35590740 1 --2- 172.21.15.40:0/1654716488 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55b4f36964c0 0x55b4f36951d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:03.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.225+0000 7f8a35590740 4 client.14760 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-21T23:16:03.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.225+0000 7f8a35590740 1 -- 172.21.15.40:0/1654716488 --> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] -- command(tid 0: {"prefix": "status"}) -- 0x55b4f3581680 con 0x55b4f36964c0 2024-04-21T23:16:03.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.226+0000 7f8a2b7fe640 1 --2- 172.21.15.40:0/1654716488 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55b4f36964c0 0x55b4f36951d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:03.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.226+0000 7f8a0b7fe640 10 client.14760 ms_handle_connect on v2:172.21.15.111:6838/3712235548 2024-04-21T23:16:03.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.226+0000 7f8a2b7fe640 1 --2- 172.21.15.40:0/1654716488 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55b4f36964c0 0x55b4f36951d0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=82568e9677fe8c3e server_cookie=8691f02014121ed4 in_seq=0 out_seq=0 2024-04-21T23:16:03.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.226+0000 7f8a0b7fe640 1 -- 172.21.15.40:0/1654716488 <== mds.0 v2:172.21.15.111:6838/3712235548 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55b4f3581680 con 0x55b4f36964c0 2024-04-21T23:16:03.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.226+0000 7f8a0b7fe640 10 client.14760 handle_command_reply: tid=0 2024-04-21T23:16:03.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.226+0000 7f8a35590740 1 client.14760 shutdown 2024-04-21T23:16:03.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.226+0000 7f8a35590740 1 -- 172.21.15.40:0/1654716488 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4f3615a60 msgr2=0x55b4f36226e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:03.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.226+0000 7f8a35590740 1 --2- 172.21.15.40:0/1654716488 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4f3615a60 0x55b4f36226e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f8a1400bc60 tx=0x7f8a1400bef0 comp rx=0 tx=0).stop 2024-04-21T23:16:03.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.226+0000 7f8a35590740 1 -- 172.21.15.40:0/1654716488 shutdown_connections 2024-04-21T23:16:03.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.226+0000 7f8a35590740 1 --2- 172.21.15.40:0/1654716488 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55b4f36964c0 0x55b4f36951d0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:03.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.226+0000 7f8a35590740 1 --2- 172.21.15.40:0/1654716488 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4f361abd0 0x55b4f3625700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:03.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.226+0000 7f8a35590740 1 --2- 172.21.15.40:0/1654716488 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4f3615a60 0x55b4f36226e0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:03.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.226+0000 7f8a35590740 1 --2- 172.21.15.40:0/1654716488 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b4f3615070 0x55b4f36221a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:03.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.226+0000 7f8a0b7fe640 0 client.14760 ms_handle_reset on v2:172.21.15.111:6838/3712235548 2024-04-21T23:16:03.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.226+0000 7f8a35590740 1 -- 172.21.15.40:0/1654716488 >> 172.21.15.40:0/1654716488 conn(0x55b4f36105e0 msgr2=0x55b4f3613840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:03.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.226+0000 7f8a35590740 1 -- 172.21.15.40:0/1654716488 shutdown_connections 2024-04-21T23:16:03.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.226+0000 7f8a35590740 1 -- 172.21.15.40:0/1654716488 wait complete. 2024-04-21T23:16:03.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.227+0000 7f8a35590740 20 client.14760 trim_cache size 0 max 16384 2024-04-21T23:16:03.232 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-21T23:16:03.233 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "d814e4dc-0033-11ef-bc93-c7b262605968", 2024-04-21T23:16:03.233 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 0, 2024-04-21T23:16:03.233 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 24521, 2024-04-21T23:16:03.233 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-04-21T23:16:03.233 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-04-21T23:16:03.233 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-04-21T23:16:03.233 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 6.8432902440000003, 2024-04-21T23:16:03.233 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 17, 2024-04-21T23:16:03.233 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 90, 2024-04-21T23:16:03.233 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 89, 2024-04-21T23:16:03.233 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 13.234646595999999 2024-04-21T23:16:03.233 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-21T23:16:03.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.227+0000 7f8a329af640 1 -- 172.21.15.40:0/357065638 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f8a180a2b90 msgr2=0x7f8a180a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:03.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.227+0000 7f8a329af640 1 --2- 172.21.15.40:0/357065638 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f8a180a2b90 0x7f8a180a5050 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f8a240091b0 tx=0x7f8a24088ad0 comp rx=0 tx=0).stop 2024-04-21T23:16:03.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.227+0000 7f8a329af640 1 -- 172.21.15.40:0/357065638 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a2c102230 msgr2=0x7f8a2c1a1230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:03.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.227+0000 7f8a329af640 1 --2- 172.21.15.40:0/357065638 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a2c102230 0x7f8a2c1a1230 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f8a2402c700 tx=0x7f8a24032f20 comp rx=0 tx=0).stop 2024-04-21T23:16:03.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.227+0000 7f8a329af640 1 -- 172.21.15.40:0/357065638 shutdown_connections 2024-04-21T23:16:03.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.227+0000 7f8a329af640 1 --2- 172.21.15.40:0/357065638 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f8a180a2b90 0x7f8a180a5050 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:03.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.227+0000 7f8a329af640 1 --2- 172.21.15.40:0/357065638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a2c101870 0x7f8a2c19a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:03.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.227+0000 7f8a329af640 1 --2- 172.21.15.40:0/357065638 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a2c102230 0x7f8a2c1a1230 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:03.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.227+0000 7f8a329af640 1 --2- 172.21.15.40:0/357065638 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a2c100680 0x7f8a2c199c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:03.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.227+0000 7f8a329af640 1 -- 172.21.15.40:0/357065638 >> 172.21.15.40:0/357065638 conn(0x7f8a2c0fbbf0 msgr2=0x7f8a2c0fd680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:03.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.227+0000 7f8a329af640 1 -- 172.21.15.40:0/357065638 shutdown_connections 2024-04-21T23:16:03.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:03.227+0000 7f8a329af640 1 -- 172.21.15.40:0/357065638 wait complete. 2024-04-21T23:16:03.237 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-21T23:16:03.237 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-21T23:16:03.237 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph fs ls 2024-04-21T23:16:03.481 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:16:03.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:03 smithi040 ceph-mon[27984]: pgmap v296: 129 pgs: 7 creating+peering, 14 unknown, 108 active+clean; 579 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s wr, 4 op/s 2024-04-21T23:16:03.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:03 smithi040 ceph-mon[27984]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-21T23:16:03.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:03 smithi040 ceph-mon[27984]: Cluster is now healthy 2024-04-21T23:16:03.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:03 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/3929145817' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:16:03.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:03 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/2555176576' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:16:03.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:03 smithi111 ceph-mon[31085]: pgmap v296: 129 pgs: 7 creating+peering, 14 unknown, 108 active+clean; 579 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s wr, 4 op/s 2024-04-21T23:16:03.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:03 smithi111 ceph-mon[31085]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-21T23:16:03.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:03 smithi111 ceph-mon[31085]: Cluster is now healthy 2024-04-21T23:16:03.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:03 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/3929145817' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:16:03.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:03 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/2555176576' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:16:03.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:03 smithi174 ceph-mon[31409]: pgmap v296: 129 pgs: 7 creating+peering, 14 unknown, 108 active+clean; 579 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s wr, 4 op/s 2024-04-21T23:16:03.695 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:03 smithi174 ceph-mon[31409]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-21T23:16:03.695 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:03 smithi174 ceph-mon[31409]: Cluster is now healthy 2024-04-21T23:16:03.695 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:03 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/3929145817' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:16:03.695 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:03 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/2555176576' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:16:05.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:05.118+0000 7f36a6614700 1 -- 172.21.15.40:0/1848790393 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f36a0107570 msgr2=0x7f36a010f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:05.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:05.118+0000 7f36a6614700 1 --2- 172.21.15.40:0/1848790393 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f36a0107570 0x7f36a010f140 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f369800b0a0 tx=0x7f369800b3b0 comp rx=0 tx=0).stop 2024-04-21T23:16:05.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:05.118+0000 7f36a6614700 1 -- 172.21.15.40:0/1848790393 shutdown_connections 2024-04-21T23:16:05.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:05.118+0000 7f36a6614700 1 --2- 172.21.15.40:0/1848790393 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f36a01024e0 0x7f36a01028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:05.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:05.118+0000 7f36a6614700 1 --2- 172.21.15.40:0/1848790393 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f36a0107570 0x7f36a010f140 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:05.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:05.118+0000 7f36a6614700 1 --2- 172.21.15.40:0/1848790393 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f36a0102e90 0x7f36a0106e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:05.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:05.118+0000 7f36a6614700 1 -- 172.21.15.40:0/1848790393 >> 172.21.15.40:0/1848790393 conn(0x7f36a0079390 msgr2=0x7f36a00797a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:05.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:05.118+0000 7f36a6614700 1 -- 172.21.15.40:0/1848790393 shutdown_connections 2024-04-21T23:16:05.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:05.118+0000 7f36a6614700 1 -- 172.21.15.40:0/1848790393 wait complete. 2024-04-21T23:16:05.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:05.118+0000 7f36a6614700 1 Processor -- start 2024-04-21T23:16:05.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:05.118+0000 7f36a6614700 1 -- start start 2024-04-21T23:16:05.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:05.119+0000 7f36a6614700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f36a01024e0 0x7f36a019e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:05.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:05.119+0000 7f36a6614700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f36a0102e90 0x7f36a019eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:05.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:05.119+0000 7f36a6614700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f36a0107570 0x7f36a0198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:05.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:05.119+0000 7f36a6614700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f36a0112550 con 0x7f36a01024e0 2024-04-21T23:16:05.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:05.119+0000 7f36a6614700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f36a01123d0 con 0x7f36a0107570 2024-04-21T23:16:05.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:05.119+0000 7f36a6614700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f36a01126d0 con 0x7f36a0102e90 2024-04-21T23:16:05.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:05.119+0000 7f369ffff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f36a01024e0 0x7f36a019e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:05.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:05.119+0000 7f369ffff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f36a01024e0 0x7f36a019e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33246/0 (socket says 172.21.15.40:33246) 2024-04-21T23:16:05.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:05.119+0000 7f369ffff700 1 -- 172.21.15.40:0/466962341 learned_addr learned my addr 172.21.15.40:0/466962341 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:16:05.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:05.119+0000 7f36a4bb1700 1 --2- 172.21.15.40:0/466962341 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f36a0107570 0x7f36a0198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:05.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:05.119+0000 7f369f7fe700 1 --2- 172.21.15.40:0/466962341 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f36a0102e90 0x7f36a019eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:05.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:05.119+0000 7f369ffff700 1 -- 172.21.15.40:0/466962341 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f36a0102e90 msgr2=0x7f36a019eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:05.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:05.119+0000 7f369ffff700 1 --2- 172.21.15.40:0/466962341 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f36a0102e90 0x7f36a019eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:05.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:05.119+0000 7f369ffff700 1 -- 172.21.15.40:0/466962341 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f36a0107570 msgr2=0x7f36a0198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:05.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:05.119+0000 7f369ffff700 1 --2- 172.21.15.40:0/466962341 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f36a0107570 0x7f36a0198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:05.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:05.119+0000 7f369ffff700 1 -- 172.21.15.40:0/466962341 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36a0198ff0 con 0x7f36a01024e0 2024-04-21T23:16:05.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:05.119+0000 7f369f7fe700 1 --2- 172.21.15.40:0/466962341 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f36a0102e90 0x7f36a019eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:16:05.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:05.119+0000 7f369ffff700 1 --2- 172.21.15.40:0/466962341 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f36a01024e0 0x7f36a019e640 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f369000c9b0 tx=0x7f369000ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:05.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:05.119+0000 7f369d7fa700 1 -- 172.21.15.40:0/466962341 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3690007b00 con 0x7f36a01024e0 2024-04-21T23:16:05.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:05.120+0000 7f369d7fa700 1 -- 172.21.15.40:0/466962341 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3690007ca0 con 0x7f36a01024e0 2024-04-21T23:16:05.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:05.120+0000 7f36a6614700 1 -- 172.21.15.40:0/466962341 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36a01992e0 con 0x7f36a01024e0 2024-04-21T23:16:05.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:05.120+0000 7f369d7fa700 1 -- 172.21.15.40:0/466962341 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3690018730 con 0x7f36a01024e0 2024-04-21T23:16:05.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:05.120+0000 7f36a6614700 1 -- 172.21.15.40:0/466962341 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36a01a55c0 con 0x7f36a01024e0 2024-04-21T23:16:05.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:05.121+0000 7f36a6614700 1 -- 172.21.15.40:0/466962341 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36a0103970 con 0x7f36a01024e0 2024-04-21T23:16:05.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:05.122+0000 7f369d7fa700 1 -- 172.21.15.40:0/466962341 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f36900188d0 con 0x7f36a01024e0 2024-04-21T23:16:05.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:05.122+0000 7f369d7fa700 1 --2- 172.21.15.40:0/466962341 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f36880a2c20 0x7f36880a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:05.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:05.123+0000 7f369f7fe700 1 --2- 172.21.15.40:0/466962341 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f36880a2c20 0x7f36880a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:05.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:05.123+0000 7f369d7fa700 1 -- 172.21.15.40:0/466962341 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f36900cf1f0 con 0x7f36a01024e0 2024-04-21T23:16:05.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:05.123+0000 7f369f7fe700 1 --2- 172.21.15.40:0/466962341 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f36880a2c20 0x7f36880a50e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f36a0199ed0 tx=0x7f3694009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:05.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:05.127+0000 7f369d7fa700 1 -- 172.21.15.40:0/466962341 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f3690014030 con 0x7f36a01024e0 2024-04-21T23:16:05.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:05.382+0000 7f36a6614700 1 -- 172.21.15.40:0/466962341 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f36a0199c40 con 0x7f36a01024e0 2024-04-21T23:16:05.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:05.382+0000 7f369d7fa700 1 -- 172.21.15.40:0/466962341 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+88 (secure 0 0 0) 0x7f369009bfb0 con 0x7f36a01024e0 2024-04-21T23:16:05.381 INFO:teuthology.orchestra.run.smithi040.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-21T23:16:05.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:05.384+0000 7f36a6614700 1 -- 172.21.15.40:0/466962341 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f36880a2c20 msgr2=0x7f36880a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:05.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:05.384+0000 7f36a6614700 1 --2- 172.21.15.40:0/466962341 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f36880a2c20 0x7f36880a50e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f36a0199ed0 tx=0x7f3694009260 comp rx=0 tx=0).stop 2024-04-21T23:16:05.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:05.384+0000 7f36a6614700 1 -- 172.21.15.40:0/466962341 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f36a01024e0 msgr2=0x7f36a019e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:05.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:05.384+0000 7f36a6614700 1 --2- 172.21.15.40:0/466962341 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f36a01024e0 0x7f36a019e640 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f369000c9b0 tx=0x7f369000ccc0 comp rx=0 tx=0).stop 2024-04-21T23:16:05.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:05.384+0000 7f36a6614700 1 -- 172.21.15.40:0/466962341 shutdown_connections 2024-04-21T23:16:05.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:05.384+0000 7f36a6614700 1 --2- 172.21.15.40:0/466962341 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f36880a2c20 0x7f36880a50e0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:05.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:05.384+0000 7f36a6614700 1 --2- 172.21.15.40:0/466962341 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f36a0107570 0x7f36a0198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:05.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:05.384+0000 7f36a6614700 1 --2- 172.21.15.40:0/466962341 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f36a01024e0 0x7f36a019e640 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:05.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:05.384+0000 7f36a6614700 1 --2- 172.21.15.40:0/466962341 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f36a0102e90 0x7f36a019eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:05.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:05.384+0000 7f36a6614700 1 -- 172.21.15.40:0/466962341 >> 172.21.15.40:0/466962341 conn(0x7f36a0079390 msgr2=0x7f36a0100290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:05.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:05.384+0000 7f36a6614700 1 -- 172.21.15.40:0/466962341 shutdown_connections 2024-04-21T23:16:05.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:05.384+0000 7f36a6614700 1 -- 172.21.15.40:0/466962341 wait complete. 2024-04-21T23:16:05.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:05 smithi040 ceph-mon[27984]: pgmap v297: 129 pgs: 7 creating+peering, 4 unknown, 118 active+clean; 579 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 2 KiB/s wr, 3 op/s 2024-04-21T23:16:05.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:05 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/466962341' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-21T23:16:05.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:05 smithi174 ceph-mon[31409]: pgmap v297: 129 pgs: 7 creating+peering, 4 unknown, 118 active+clean; 579 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 2 KiB/s wr, 3 op/s 2024-04-21T23:16:05.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:05 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/466962341' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-21T23:16:05.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:05 smithi111 ceph-mon[31085]: pgmap v297: 129 pgs: 7 creating+peering, 4 unknown, 118 active+clean; 579 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 2 KiB/s wr, 3 op/s 2024-04-21T23:16:05.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:05 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/466962341' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-21T23:16:06.192 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-21T23:16:06.192 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-21T23:16:06.192 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi040.front.sepia.ceph.com 2024-04-21T23:16:06.192 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-04-21T23:16:06.192 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-21T23:16:06.192 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-21T23:16:06.192 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-21T23:16:06.192 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-21T23:16:06.192 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-04-21T23:16:06.192 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:16:06.193 DEBUG:teuthology.orchestra.run.smithi040:> ip addr 2024-04-21T23:16:06.211 INFO:teuthology.orchestra.run.smithi040.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-21T23:16:06.211 INFO:teuthology.orchestra.run.smithi040.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-21T23:16:06.211 INFO:teuthology.orchestra.run.smithi040.stdout: inet 127.0.0.1/8 scope host lo 2024-04-21T23:16:06.211 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-04-21T23:16:06.211 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 ::1/128 scope host 2024-04-21T23:16:06.211 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-04-21T23:16:06.211 INFO:teuthology.orchestra.run.smithi040.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-21T23:16:06.212 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:6b:fd:16 brd ff:ff:ff:ff:ff:ff 2024-04-21T23:16:06.212 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp7s0f0 2024-04-21T23:16:06.212 INFO:teuthology.orchestra.run.smithi040.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-21T23:16:06.212 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:6b:fd:17 brd ff:ff:ff:ff:ff:ff 2024-04-21T23:16:06.212 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp7s0f1 2024-04-21T23:16:06.212 INFO:teuthology.orchestra.run.smithi040.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-21T23:16:06.212 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:bd:12:cc brd ff:ff:ff:ff:ff:ff 2024-04-21T23:16:06.212 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp1s0f0 2024-04-21T23:16:06.212 INFO:teuthology.orchestra.run.smithi040.stdout: inet 172.21.15.40/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-21T23:16:06.212 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft 42205sec preferred_lft 42205sec 2024-04-21T23:16:06.212 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 fe80::ec4:7aff:febd:12cc/64 scope link noprefixroute 2024-04-21T23:16:06.212 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-04-21T23:16:06.212 INFO:teuthology.orchestra.run.smithi040.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-21T23:16:06.212 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:bd:12:cd brd ff:ff:ff:ff:ff:ff 2024-04-21T23:16:06.212 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp1s0f1 2024-04-21T23:16:06.213 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-21T23:16:06.213 DEBUG:teuthology.orchestra.run.smithi040:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-21T23:16:06.213 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2024-04-21T23:16:06.213 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link add name ceph-brx type bridge 2024-04-21T23:16:06.213 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip addr flush dev ceph-brx 2024-04-21T23:16:06.214 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link set ceph-brx up 2024-04-21T23:16:06.214 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-21T23:16:06.214 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-04-21T23:16:06.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:06 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-21T23:16:06.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:06 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T23:16:06.408 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:16:06.408 DEBUG:teuthology.orchestra.run.smithi040:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-21T23:16:06.434 INFO:teuthology.orchestra.run.smithi040.stdout:1 2024-04-21T23:16:06.435 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:16:06.436 DEBUG:teuthology.orchestra.run.smithi040:> route 2024-04-21T23:16:06.510 INFO:teuthology.orchestra.run.smithi040.stdout:Kernel IP routing table 2024-04-21T23:16:06.511 INFO:teuthology.orchestra.run.smithi040.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-21T23:16:06.511 INFO:teuthology.orchestra.run.smithi040.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-21T23:16:06.511 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-21T23:16:06.511 INFO:teuthology.orchestra.run.smithi040.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-21T23:16:06.511 DEBUG:teuthology.orchestra.run.smithi040:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-21T23:16:06.511 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2024-04-21T23:16:06.512 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-21T23:16:06.512 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-21T23:16:06.512 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-21T23:16:06.512 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-04-21T23:16:06.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:06 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-21T23:16:06.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:06 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T23:16:06.777 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:16:06.777 DEBUG:teuthology.orchestra.run.smithi040:> ip netns list 2024-04-21T23:16:06.793 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:16:06.793 DEBUG:teuthology.orchestra.run.smithi040:> ip netns list-id 2024-04-21T23:16:06.848 DEBUG:teuthology.orchestra.run.smithi040:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-21T23:16:06.848 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2024-04-21T23:16:06.848 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-21T23:16:06.848 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-04-21T23:16:06.848 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-04-21T23:16:06.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:06 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-21T23:16:06.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:06 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T23:16:06.966 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-04-21T23:16:06.967 DEBUG:teuthology.orchestra.run.smithi040:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-21T23:16:06.967 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2024-04-21T23:16:06.967 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-04-21T23:16:06.967 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-21T23:16:06.967 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-04-21T23:16:06.967 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-04-21T23:16:06.967 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-04-21T23:16:06.967 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-04-21T23:16:07.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:07 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-21T23:16:07.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:07 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T23:16:07.093 DEBUG:teuthology.orchestra.run.smithi040:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-21T23:16:07.093 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2024-04-21T23:16:07.093 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link set brx.0 up 2024-04-21T23:16:07.093 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link set dev brx.0 master ceph-brx 2024-04-21T23:16:07.093 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-04-21T23:16:07.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:07 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-21T23:16:07.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:07 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T23:16:07.170 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-04-21T23:16:07.170 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-21T23:16:07.170 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-04-21T23:16:07.185 INFO:teuthology.orchestra.run.smithi040.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-04-21T23:16:07.186 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-21T23:16:07.186 DEBUG:teuthology.orchestra.run.smithi040:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-04-21T23:16:07.241 DEBUG:teuthology.orchestra.run.smithi040:> sudo modprobe fuse 2024-04-21T23:16:07.306 DEBUG:teuthology.orchestra.run.smithi040:> cat /proc/self/mounts | awk '{print $2}' 2024-04-21T23:16:07.363 INFO:teuthology.orchestra.run.smithi040.stdout:/proc 2024-04-21T23:16:07.363 INFO:teuthology.orchestra.run.smithi040.stdout:/sys 2024-04-21T23:16:07.363 INFO:teuthology.orchestra.run.smithi040.stdout:/dev 2024-04-21T23:16:07.363 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/security 2024-04-21T23:16:07.363 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/shm 2024-04-21T23:16:07.363 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/pts 2024-04-21T23:16:07.363 INFO:teuthology.orchestra.run.smithi040.stdout:/run 2024-04-21T23:16:07.363 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/cgroup 2024-04-21T23:16:07.363 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/pstore 2024-04-21T23:16:07.363 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/bpf 2024-04-21T23:16:07.363 INFO:teuthology.orchestra.run.smithi040.stdout:/ 2024-04-21T23:16:07.363 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/selinux 2024-04-21T23:16:07.363 INFO:teuthology.orchestra.run.smithi040.stdout:/proc/sys/fs/binfmt_misc 2024-04-21T23:16:07.364 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/hugepages 2024-04-21T23:16:07.364 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/mqueue 2024-04-21T23:16:07.364 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/debug 2024-04-21T23:16:07.364 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/tracing 2024-04-21T23:16:07.364 INFO:teuthology.orchestra.run.smithi040.stdout:/run/credentials/systemd-sysusers.service 2024-04-21T23:16:07.364 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/config 2024-04-21T23:16:07.364 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/fuse/connections 2024-04-21T23:16:07.364 INFO:teuthology.orchestra.run.smithi040.stdout:/run/credentials/systemd-sysctl.service 2024-04-21T23:16:07.364 INFO:teuthology.orchestra.run.smithi040.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-21T23:16:07.364 INFO:teuthology.orchestra.run.smithi040.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-21T23:16:07.364 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/nfs/rpc_pipefs 2024-04-21T23:16:07.364 INFO:teuthology.orchestra.run.smithi040.stdout:/run/user/1000 2024-04-21T23:16:07.364 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph 2024-04-21T23:16:07.364 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay 2024-04-21T23:16:07.364 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/ee97a42f3a3e584103a65e5401d26f9e90beee3b099774e27956a06f9aae3f04/merged 2024-04-21T23:16:07.364 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/5b578d117dff6c77ff42485355b1f3fbd0ba32be0fd7454e14957896ffde180b/merged 2024-04-21T23:16:07.365 INFO:teuthology.orchestra.run.smithi040.stdout:/run/user/0 2024-04-21T23:16:07.365 INFO:teuthology.orchestra.run.smithi040.stdout:/proc/sys/fs/binfmt_misc 2024-04-21T23:16:07.365 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/bf129acafa6e662f9d769e35d1abad3b8a73cce30705229d03896b165f2b4872/merged 2024-04-21T23:16:07.365 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/c435ffa4563f8d391453469885f0c52413600a8c901881dca35f7818d9476346/merged 2024-04-21T23:16:07.365 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/12f881f08e65218aacc8a6f86b8d1c811dd8e7658e6e8413801df6999f7b4c19/merged 2024-04-21T23:16:07.365 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/b0b1a36da5ee61cea069b200bcfca27244b02cdd87683301abbb66cf65f536b5/merged 2024-04-21T23:16:07.365 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/5f6305e2feed35661880eadf1335149f8d71d88c3a6744bb0d69757ff7359b05/merged 2024-04-21T23:16:07.365 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/63385edd6428bd56f26f165ab67b26b0d9049a9a4de274cf1b6ae99f2abc6c34/merged 2024-04-21T23:16:07.365 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/0d815bb55ee16425c0accc123ecc48ae735c4c714c19e49a953241cdbd8a4711/merged 2024-04-21T23:16:07.365 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/36fed82ac0fd5a4832d3465a289ff98ef2fd709230eff6e76c50acf5cf96d143/merged 2024-04-21T23:16:07.365 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns 2024-04-21T23:16:07.365 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-21T23:16:07.365 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-21T23:16:07.366 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:16:07.366 DEBUG:teuthology.orchestra.run.smithi040:> ls /sys/fs/fuse/connections 2024-04-21T23:16:07.418 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-21T23:16:07.418 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-04-21T23:16:07.460 DEBUG:teuthology.orchestra.run.smithi040:> sudo modprobe fuse 2024-04-21T23:16:07.484 DEBUG:teuthology.orchestra.run.smithi040:> cat /proc/self/mounts | awk '{print $2}' 2024-04-21T23:16:07.541 INFO:teuthology.orchestra.run.smithi040.stdout:/proc 2024-04-21T23:16:07.541 INFO:teuthology.orchestra.run.smithi040.stdout:/sys 2024-04-21T23:16:07.541 INFO:teuthology.orchestra.run.smithi040.stdout:/dev 2024-04-21T23:16:07.541 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/security 2024-04-21T23:16:07.541 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/shm 2024-04-21T23:16:07.541 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/pts 2024-04-21T23:16:07.541 INFO:teuthology.orchestra.run.smithi040.stdout:/run 2024-04-21T23:16:07.541 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/cgroup 2024-04-21T23:16:07.541 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/pstore 2024-04-21T23:16:07.541 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/bpf 2024-04-21T23:16:07.541 INFO:teuthology.orchestra.run.smithi040.stdout:/ 2024-04-21T23:16:07.541 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/selinux 2024-04-21T23:16:07.542 INFO:teuthology.orchestra.run.smithi040.stdout:/proc/sys/fs/binfmt_misc 2024-04-21T23:16:07.542 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/hugepages 2024-04-21T23:16:07.542 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/mqueue 2024-04-21T23:16:07.542 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/debug 2024-04-21T23:16:07.542 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/tracing 2024-04-21T23:16:07.542 INFO:teuthology.orchestra.run.smithi040.stdout:/run/credentials/systemd-sysusers.service 2024-04-21T23:16:07.542 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/config 2024-04-21T23:16:07.542 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/fuse/connections 2024-04-21T23:16:07.543 INFO:teuthology.orchestra.run.smithi040.stdout:/run/credentials/systemd-sysctl.service 2024-04-21T23:16:07.543 INFO:teuthology.orchestra.run.smithi040.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-21T23:16:07.543 INFO:teuthology.orchestra.run.smithi040.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-21T23:16:07.543 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/nfs/rpc_pipefs 2024-04-21T23:16:07.543 INFO:teuthology.orchestra.run.smithi040.stdout:/run/user/1000 2024-04-21T23:16:07.543 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph 2024-04-21T23:16:07.543 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay 2024-04-21T23:16:07.543 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/ee97a42f3a3e584103a65e5401d26f9e90beee3b099774e27956a06f9aae3f04/merged 2024-04-21T23:16:07.543 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/5b578d117dff6c77ff42485355b1f3fbd0ba32be0fd7454e14957896ffde180b/merged 2024-04-21T23:16:07.543 INFO:teuthology.orchestra.run.smithi040.stdout:/run/user/0 2024-04-21T23:16:07.543 INFO:teuthology.orchestra.run.smithi040.stdout:/proc/sys/fs/binfmt_misc 2024-04-21T23:16:07.543 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/bf129acafa6e662f9d769e35d1abad3b8a73cce30705229d03896b165f2b4872/merged 2024-04-21T23:16:07.544 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/c435ffa4563f8d391453469885f0c52413600a8c901881dca35f7818d9476346/merged 2024-04-21T23:16:07.544 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/12f881f08e65218aacc8a6f86b8d1c811dd8e7658e6e8413801df6999f7b4c19/merged 2024-04-21T23:16:07.544 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/b0b1a36da5ee61cea069b200bcfca27244b02cdd87683301abbb66cf65f536b5/merged 2024-04-21T23:16:07.544 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/5f6305e2feed35661880eadf1335149f8d71d88c3a6744bb0d69757ff7359b05/merged 2024-04-21T23:16:07.544 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/63385edd6428bd56f26f165ab67b26b0d9049a9a4de274cf1b6ae99f2abc6c34/merged 2024-04-21T23:16:07.544 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/0d815bb55ee16425c0accc123ecc48ae735c4c714c19e49a953241cdbd8a4711/merged 2024-04-21T23:16:07.544 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/36fed82ac0fd5a4832d3465a289ff98ef2fd709230eff6e76c50acf5cf96d143/merged 2024-04-21T23:16:07.544 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns 2024-04-21T23:16:07.544 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-21T23:16:07.544 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-21T23:16:07.544 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:16:07.545 DEBUG:teuthology.orchestra.run.smithi040:> ls /sys/fs/fuse/connections 2024-04-21T23:16:07.557 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi040.stderr:2024-04-21T23:16:07.557+0000 7fd22f16c480 -1 init, newargv = 0x55a37561f0a0 newargc=15 2024-04-21T23:16:07.557 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi040.stderr:ceph-fuse[69530]: starting ceph client 2024-04-21T23:16:07.563 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi040.stderr:ceph-fuse[69530]: starting fuse 2024-04-21T23:16:07.572 INFO:teuthology.orchestra.run.smithi040.stdout:115 2024-04-21T23:16:07.572 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [115] 2024-04-21T23:16:07.572 DEBUG:teuthology.orchestra.run.smithi040:> sudo stdin-killer -- python3 -c ' 2024-04-21T23:16:07.572 DEBUG:teuthology.orchestra.run.smithi040:> import glob 2024-04-21T23:16:07.572 DEBUG:teuthology.orchestra.run.smithi040:> import re 2024-04-21T23:16:07.572 DEBUG:teuthology.orchestra.run.smithi040:> import os 2024-04-21T23:16:07.573 DEBUG:teuthology.orchestra.run.smithi040:> import subprocess 2024-04-21T23:16:07.573 DEBUG:teuthology.orchestra.run.smithi040:> 2024-04-21T23:16:07.573 DEBUG:teuthology.orchestra.run.smithi040:> def _find_admin_socket(client_name): 2024-04-21T23:16:07.573 DEBUG:teuthology.orchestra.run.smithi040:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-04-21T23:16:07.573 DEBUG:teuthology.orchestra.run.smithi040:> files = glob.glob(asok_path) 2024-04-21T23:16:07.573 DEBUG:teuthology.orchestra.run.smithi040:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-04-21T23:16:07.573 DEBUG:teuthology.orchestra.run.smithi040:> 2024-04-21T23:16:07.573 DEBUG:teuthology.orchestra.run.smithi040:> # Given a non-glob path, it better be there 2024-04-21T23:16:07.573 DEBUG:teuthology.orchestra.run.smithi040:> if "*" not in asok_path: 2024-04-21T23:16:07.573 DEBUG:teuthology.orchestra.run.smithi040:> assert(len(files) == 1) 2024-04-21T23:16:07.573 DEBUG:teuthology.orchestra.run.smithi040:> return files[0] 2024-04-21T23:16:07.573 DEBUG:teuthology.orchestra.run.smithi040:> 2024-04-21T23:16:07.573 DEBUG:teuthology.orchestra.run.smithi040:> for f in files: 2024-04-21T23:16:07.573 DEBUG:teuthology.orchestra.run.smithi040:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-21T23:16:07.573 DEBUG:teuthology.orchestra.run.smithi040:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-21T23:16:07.573 DEBUG:teuthology.orchestra.run.smithi040:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-21T23:16:07.573 DEBUG:teuthology.orchestra.run.smithi040:> contents = proc_f.read() 2024-04-21T23:16:07.574 DEBUG:teuthology.orchestra.run.smithi040:> if mountpoint in contents: 2024-04-21T23:16:07.574 DEBUG:teuthology.orchestra.run.smithi040:> return f 2024-04-21T23:16:07.574 DEBUG:teuthology.orchestra.run.smithi040:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-21T23:16:07.574 DEBUG:teuthology.orchestra.run.smithi040:> 2024-04-21T23:16:07.574 DEBUG:teuthology.orchestra.run.smithi040:> print(_find_admin_socket("client.admin")) 2024-04-21T23:16:07.574 DEBUG:teuthology.orchestra.run.smithi040:> ' 2024-04-21T23:16:07.632 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:07 smithi040 ceph-mon[27984]: pgmap v298: 129 pgs: 129 active+clean; 579 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 2 op/s 2024-04-21T23:16:07.691 INFO:teuthology.orchestra.run.smithi040.stdout:/var/run/ceph/ceph-client.admin.69530.asok 2024-04-21T23:16:07.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:07 smithi174 ceph-mon[31409]: pgmap v298: 129 pgs: 129 active+clean; 579 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 2 op/s 2024-04-21T23:16:07.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:07 smithi111 ceph-mon[31085]: pgmap v298: 129 pgs: 129 active+clean; 579 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 2 op/s 2024-04-21T23:16:07.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:07 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T23:16:07.701 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.69530.asok 2024-04-21T23:16:07.702 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:16:07.702 DEBUG:teuthology.orchestra.run.smithi040:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.69530.asok status 2024-04-21T23:16:07.839 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-21T23:16:07.839 INFO:teuthology.orchestra.run.smithi040.stdout: "metadata": { 2024-04-21T23:16:07.839 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_sha1": "820a2f8ced66152ed3d765c31444b8076ea28c6b", 2024-04-21T23:16:07.839 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_version": "ceph version 19.0.0-2405-g820a2f8c (820a2f8ced66152ed3d765c31444b8076ea28c6b) squid (dev)", 2024-04-21T23:16:07.840 INFO:teuthology.orchestra.run.smithi040.stdout: "entity_id": "admin", 2024-04-21T23:16:07.840 INFO:teuthology.orchestra.run.smithi040.stdout: "hostname": "smithi040", 2024-04-21T23:16:07.840 INFO:teuthology.orchestra.run.smithi040.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-04-21T23:16:07.840 INFO:teuthology.orchestra.run.smithi040.stdout: "pid": "69530", 2024-04-21T23:16:07.840 INFO:teuthology.orchestra.run.smithi040.stdout: "root": "/" 2024-04-21T23:16:07.840 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:16:07.840 INFO:teuthology.orchestra.run.smithi040.stdout: "dentry_count": 0, 2024-04-21T23:16:07.840 INFO:teuthology.orchestra.run.smithi040.stdout: "dentry_pinned_count": 0, 2024-04-21T23:16:07.840 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 14772, 2024-04-21T23:16:07.840 INFO:teuthology.orchestra.run.smithi040.stdout: "inst": { 2024-04-21T23:16:07.840 INFO:teuthology.orchestra.run.smithi040.stdout: "name": { 2024-04-21T23:16:07.840 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "client", 2024-04-21T23:16:07.840 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 14772 2024-04-21T23:16:07.841 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:16:07.841 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": { 2024-04-21T23:16:07.841 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-21T23:16:07.841 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "192.168.0.1:0", 2024-04-21T23:16:07.841 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 738988465 2024-04-21T23:16:07.841 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:16:07.841 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:16:07.841 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": { 2024-04-21T23:16:07.841 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-21T23:16:07.841 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "192.168.0.1:0", 2024-04-21T23:16:07.841 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 738988465 2024-04-21T23:16:07.841 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:16:07.842 INFO:teuthology.orchestra.run.smithi040.stdout: "inst_str": "client.14772 192.168.0.1:0/738988465", 2024-04-21T23:16:07.842 INFO:teuthology.orchestra.run.smithi040.stdout: "addr_str": "192.168.0.1:0/738988465", 2024-04-21T23:16:07.842 INFO:teuthology.orchestra.run.smithi040.stdout: "inode_count": 1, 2024-04-21T23:16:07.842 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_epoch": 17, 2024-04-21T23:16:07.842 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_epoch": 90, 2024-04-21T23:16:07.842 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_epoch_barrier": 0, 2024-04-21T23:16:07.842 INFO:teuthology.orchestra.run.smithi040.stdout: "blocklisted": false, 2024-04-21T23:16:07.842 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs" 2024-04-21T23:16:07.842 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-21T23:16:07.848 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:16:07.848 DEBUG:teuthology.orchestra.run.smithi040:> stat --file-system '--printf=%T 2024-04-21T23:16:07.848 DEBUG:teuthology.orchestra.run.smithi040:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-04-21T23:16:07.864 INFO:teuthology.orchestra.run.smithi040.stdout:fuseblk 2024-04-21T23:16:07.864 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-04-21T23:16:07.864 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:16:07.864 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-04-21T23:16:07.929 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-04-21T23:16:08.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-21T23:16:08.039 INFO:teuthology.orchestra.run.smithi040.stdout:# file: . 2024-04-21T23:16:08.040 INFO:teuthology.orchestra.run.smithi040.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-04-21T23:16:08.040 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:16:08.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T23:16:08.046 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-21T23:16:08.046 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-21T23:16:08.063 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T23:16:08.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.205+0000 7f2a47240640 1 -- 172.21.15.40:0/180681032 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a40103340 msgr2=0x7f2a40109bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:08.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.205+0000 7f2a47240640 1 --2- 172.21.15.40:0/180681032 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a40103340 0x7f2a40109bd0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f2a28009900 tx=0x7f2a2802bc30 comp rx=0 tx=0).stop 2024-04-21T23:16:08.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.205+0000 7f2a47240640 1 -- 172.21.15.40:0/180681032 shutdown_connections 2024-04-21T23:16:08.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.205+0000 7f2a47240640 1 --2- 172.21.15.40:0/180681032 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a40103340 0x7f2a40109bd0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:08.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.205+0000 7f2a47240640 1 --2- 172.21.15.40:0/180681032 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2a40102980 0x7f2a40102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:08.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.205+0000 7f2a47240640 1 --2- 172.21.15.40:0/180681032 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2a40101790 0x7f2a40101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:08.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.205+0000 7f2a47240640 1 -- 172.21.15.40:0/180681032 >> 172.21.15.40:0/180681032 conn(0x7f2a400fcd40 msgr2=0x7f2a400ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:08.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.205+0000 7f2a47240640 1 -- 172.21.15.40:0/180681032 shutdown_connections 2024-04-21T23:16:08.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.205+0000 7f2a47240640 1 -- 172.21.15.40:0/180681032 wait complete. 2024-04-21T23:16:08.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.206+0000 7f2a47240640 1 Processor -- start 2024-04-21T23:16:08.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.206+0000 7f2a47240640 1 -- start start 2024-04-21T23:16:08.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.206+0000 7f2a47240640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2a40101790 0x7f2a40199c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:08.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.206+0000 7f2a47240640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2a40102980 0x7f2a4019a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:08.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.206+0000 7f2a47240640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a40103340 0x7f2a401a11c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:08.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.206+0000 7f2a47240640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2a40073f80 con 0x7f2a40103340 2024-04-21T23:16:08.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.206+0000 7f2a47240640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f2a40073e00 con 0x7f2a40101790 2024-04-21T23:16:08.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.206+0000 7f2a47240640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2a40074100 con 0x7f2a40102980 2024-04-21T23:16:08.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.206+0000 7f2a457dd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a40103340 0x7f2a401a11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:08.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.206+0000 7f2a457dd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a40103340 0x7f2a401a11c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33252/0 (socket says 172.21.15.40:33252) 2024-04-21T23:16:08.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.206+0000 7f2a457dd640 1 -- 172.21.15.40:0/3170790984 learned_addr learned my addr 172.21.15.40:0/3170790984 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:16:08.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.206+0000 7f2a44fdc640 1 --2- 172.21.15.40:0/3170790984 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2a40101790 0x7f2a40199c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:08.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.206+0000 7f2a3ffff640 1 --2- 172.21.15.40:0/3170790984 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2a40102980 0x7f2a4019a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:08.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.206+0000 7f2a457dd640 1 -- 172.21.15.40:0/3170790984 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2a40102980 msgr2=0x7f2a4019a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:08.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.206+0000 7f2a457dd640 1 --2- 172.21.15.40:0/3170790984 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2a40102980 0x7f2a4019a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:08.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.206+0000 7f2a457dd640 1 -- 172.21.15.40:0/3170790984 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2a40101790 msgr2=0x7f2a40199c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:08.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.206+0000 7f2a457dd640 1 --2- 172.21.15.40:0/3170790984 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2a40101790 0x7f2a40199c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:08.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.206+0000 7f2a457dd640 1 -- 172.21.15.40:0/3170790984 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a401a1860 con 0x7f2a40103340 2024-04-21T23:16:08.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.206+0000 7f2a44fdc640 1 --2- 172.21.15.40:0/3170790984 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2a40101790 0x7f2a40199c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:16:08.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.207+0000 7f2a457dd640 1 --2- 172.21.15.40:0/3170790984 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a40103340 0x7f2a401a11c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f2a2802c7b0 tx=0x7f2a28005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:08.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.207+0000 7f2a3dffb640 1 -- 172.21.15.40:0/3170790984 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a28041070 con 0x7f2a40103340 2024-04-21T23:16:08.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.207+0000 7f2a47240640 1 -- 172.21.15.40:0/3170790984 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a401a1af0 con 0x7f2a40103340 2024-04-21T23:16:08.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.207+0000 7f2a3dffb640 1 -- 172.21.15.40:0/3170790984 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a280054e0 con 0x7f2a40103340 2024-04-21T23:16:08.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.207+0000 7f2a3dffb640 1 -- 172.21.15.40:0/3170790984 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a28034440 con 0x7f2a40103340 2024-04-21T23:16:08.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.207+0000 7f2a47240640 1 -- 172.21.15.40:0/3170790984 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a401a1fe0 con 0x7f2a40103340 2024-04-21T23:16:08.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.208+0000 7f2a47240640 1 -- 172.21.15.40:0/3170790984 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a401a21b0 con 0x7f2a40103340 2024-04-21T23:16:08.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.209+0000 7f2a3dffb640 1 -- 172.21.15.40:0/3170790984 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f2a28044070 con 0x7f2a40103340 2024-04-21T23:16:08.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.210+0000 7f2a3dffb640 1 --2- 172.21.15.40:0/3170790984 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f2a2c0a2b90 0x7f2a2c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:08.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.210+0000 7f2a44fdc640 1 --2- 172.21.15.40:0/3170790984 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f2a2c0a2b90 0x7f2a2c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:08.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.210+0000 7f2a3dffb640 1 -- 172.21.15.40:0/3170790984 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f2a280f1100 con 0x7f2a40103340 2024-04-21T23:16:08.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.210+0000 7f2a44fdc640 1 --2- 172.21.15.40:0/3170790984 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f2a2c0a2b90 0x7f2a2c0a5050 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2a4019b120 tx=0x7f2a38009500 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:08.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.215+0000 7f2a3dffb640 1 -- 172.21.15.40:0/3170790984 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f2a280bddc0 con 0x7f2a40103340 2024-04-21T23:16:08.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.403+0000 7f2a47240640 1 -- 172.21.15.40:0/3170790984 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2a401a2540 con 0x7f2a40103340 2024-04-21T23:16:08.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.405+0000 7f2a3dffb640 1 -- 172.21.15.40:0/3170790984 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24138 (secure 0 0 0) 0x7f2a280bd6c0 con 0x7f2a40103340 2024-04-21T23:16:08.404 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:16:08.404 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":90,"fsid":"d814e4dc-0033-11ef-bc93-c7b262605968","created":"2024-04-21T23:07:52.659207+0000","modified":"2024-04-21T23:16:00.780824+0000","last_up_change":"2024-04-21T23:15:05.308821+0000","last_in_change":"2024-04-21T23:14:45.447491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:11:14.824775+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T23:15:54.322571+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T23:15:55.164493+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T23:15:58.838266+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"0e8d692a-22fe-4a87-8d99-9151f96cbb02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6803","nonce":2009758784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6805","nonce":2009758784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6809","nonce":2009758784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6807","nonce":2009758784}]},"public_addr":"172.21.15.40:6803/2009758784","cluster_addr":"172.21.15.40:6805/2009758784","heartbeat_back_addr":"172.21.15.40:6809/2009758784","heartbeat_front_addr":"172.21.15.40:6807/2009758784","state":["exists","up"]},{"osd":1,"uuid":"fe5fe11b-0e84-4359-83a7-1ba45105755a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6801","nonce":1763903836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6803","nonce":1763903836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6807","nonce":1763903836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6805","nonce":1763903836}]},"public_addr":"172.21.15.111:6801/1763903836","cluster_addr":"172.21.15.111:6803/1763903836","heartbeat_back_addr":"172.21.15.111:6807/1763903836","heartbeat_front_addr":"172.21.15.111:6805/1763903836","state":["exists","up"]},{"osd":2,"uuid":"813ad735-5cc2-4da0-ae16-6db238395fbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6801","nonce":439695825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6803","nonce":439695825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6807","nonce":439695825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6805","nonce":439695825}]},"public_addr":"172.21.15.174:6801/439695825","cluster_addr":"172.21.15.174:6803/439695825","heartbeat_back_addr":"172.21.15.174:6807/439695825","heartbeat_front_addr":"172.21.15.174:6805/439695825","state":["exists","up"]},{"osd":3,"uuid":"09becb03-b7ce-4ffc-917d-b7900ec37e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6811","nonce":3454312436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6813","nonce":3454312436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6817","nonce":3454312436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6815","nonce":3454312436}]},"public_addr":"172.21.15.40:6811/3454312436","cluster_addr":"172.21.15.40:6813/3454312436","heartbeat_back_addr":"172.21.15.40:6817/3454312436","heartbeat_front_addr":"172.21.15.40:6815/3454312436","state":["exists","up"]},{"osd":4,"uuid":"46eabfd7-d88a-4efd-94eb-04354d3a9d7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6809","nonce":1810866070}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6811","nonce":1810866070}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6815","nonce":1810866070}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6813","nonce":1810866070}]},"public_addr":"172.21.15.111:6809/1810866070","cluster_addr":"172.21.15.111:6811/1810866070","heartbeat_back_addr":"172.21.15.111:6815/1810866070","heartbeat_front_addr":"172.21.15.111:6813/1810866070","state":["exists","up"]},{"osd":5,"uuid":"6f73b582-6c65-41db-aab0-9b360f83d5c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6809","nonce":3752986711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6811","nonce":3752986711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6815","nonce":3752986711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6813","nonce":3752986711}]},"public_addr":"172.21.15.174:6809/3752986711","cluster_addr":"172.21.15.174:6811/3752986711","heartbeat_back_addr":"172.21.15.174:6815/3752986711","heartbeat_front_addr":"172.21.15.174:6813/3752986711","state":["exists","up"]},{"osd":6,"uuid":"39f305f0-12d2-4732-aaa0-cd66700af538","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6819","nonce":1618855427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6821","nonce":1618855427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6825","nonce":1618855427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6823","nonce":1618855427}]},"public_addr":"172.21.15.40:6819/1618855427","cluster_addr":"172.21.15.40:6821/1618855427","heartbeat_back_addr":"172.21.15.40:6825/1618855427","heartbeat_front_addr":"172.21.15.40:6823/1618855427","state":["exists","up"]},{"osd":7,"uuid":"769f2468-5102-490b-89f8-d0444d3169a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6817","nonce":927054162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6819","nonce":927054162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6823","nonce":927054162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6821","nonce":927054162}]},"public_addr":"172.21.15.111:6817/927054162","cluster_addr":"172.21.15.111:6819/927054162","heartbeat_back_addr":"172.21.15.111:6823/927054162","heartbeat_front_addr":"172.21.15.111:6821/927054162","state":["exists","up"]},{"osd":8,"uuid":"1a444e80-007d-473b-b972-4c814a813c22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6817","nonce":3817848746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6819","nonce":3817848746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6823","nonce":3817848746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6821","nonce":3817848746}]},"public_addr":"172.21.15.174:6817/3817848746","cluster_addr":"172.21.15.174:6819/3817848746","heartbeat_back_addr":"172.21.15.174:6823/3817848746","heartbeat_front_addr":"172.21.15.174:6821/3817848746","state":["exists","up"]},{"osd":9,"uuid":"9a6b6a56-6ae5-49fe-9e5b-cb17d686e0dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6827","nonce":1649446531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6829","nonce":1649446531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6833","nonce":1649446531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6831","nonce":1649446531}]},"public_addr":"172.21.15.40:6827/1649446531","cluster_addr":"172.21.15.40:6829/1649446531","heartbeat_back_addr":"172.21.15.40:6833/1649446531","heartbeat_front_addr":"172.21.15.40:6831/1649446531","state":["exists","up"]},{"osd":10,"uuid":"a70038ba-e2ad-4170-85cc-ea0470a2edef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6825","nonce":1227538935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6827","nonce":1227538935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6831","nonce":1227538935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6829","nonce":1227538935}]},"public_addr":"172.21.15.111:6825/1227538935","cluster_addr":"172.21.15.111:6827/1227538935","heartbeat_back_addr":"172.21.15.111:6831/1227538935","heartbeat_front_addr":"172.21.15.111:6829/1227538935","state":["exists","up"]},{"osd":11,"uuid":"781532bb-7a5e-44e3-8eb7-b584b0054666","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6825","nonce":2045865553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6827","nonce":2045865553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6831","nonce":2045865553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6829","nonce":2045865553}]},"public_addr":"172.21.15.174:6825/2045865553","cluster_addr":"172.21.15.174:6827/2045865553","heartbeat_back_addr":"172.21.15.174:6831/2045865553","heartbeat_front_addr":"172.21.15.174:6829/2045865553","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:10:24.761290+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:10:48.565428+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:11:12.307818+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:11:37.951529+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:02.832523+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:27.568879+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:52.997707+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:13:18.937274+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:13:44.708657+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:14:10.833855+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:14:37.078272+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:15:03.878076+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:6801/526554319":"2024-04-22T23:08:50.305770+0000","172.21.15.40:6800/526554319":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/3178763471":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/4045911364":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/276300080":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/3689649814":"2024-04-22T23:08:23.743030+0000","172.21.15.40:0/2241116715":"2024-04-22T23:08:23.743030+0000","172.21.15.40:6801/1479040326":"2024-04-22T23:08:23.743030+0000","172.21.15.40:6800/1479040326":"2024-04-22T23:08:23.743030+0000","172.21.15.40:0/681298099":"2024-04-22T23:08:23.743030+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T23:16:08.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.407+0000 7f2a47240640 1 -- 172.21.15.40:0/3170790984 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f2a2c0a2b90 msgr2=0x7f2a2c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:08.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.407+0000 7f2a47240640 1 --2- 172.21.15.40:0/3170790984 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f2a2c0a2b90 0x7f2a2c0a5050 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2a4019b120 tx=0x7f2a38009500 comp rx=0 tx=0).stop 2024-04-21T23:16:08.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.407+0000 7f2a47240640 1 -- 172.21.15.40:0/3170790984 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a40103340 msgr2=0x7f2a401a11c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:08.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.407+0000 7f2a47240640 1 --2- 172.21.15.40:0/3170790984 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a40103340 0x7f2a401a11c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f2a2802c7b0 tx=0x7f2a28005dd0 comp rx=0 tx=0).stop 2024-04-21T23:16:08.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.407+0000 7f2a47240640 1 -- 172.21.15.40:0/3170790984 shutdown_connections 2024-04-21T23:16:08.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.407+0000 7f2a47240640 1 --2- 172.21.15.40:0/3170790984 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f2a2c0a2b90 0x7f2a2c0a5050 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:08.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.407+0000 7f2a47240640 1 --2- 172.21.15.40:0/3170790984 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a40103340 0x7f2a401a11c0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:08.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.407+0000 7f2a47240640 1 --2- 172.21.15.40:0/3170790984 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2a40102980 0x7f2a4019a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:08.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.407+0000 7f2a47240640 1 --2- 172.21.15.40:0/3170790984 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2a40101790 0x7f2a40199c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:08.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.407+0000 7f2a47240640 1 -- 172.21.15.40:0/3170790984 >> 172.21.15.40:0/3170790984 conn(0x7f2a400fcd40 msgr2=0x7f2a400fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:08.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.407+0000 7f2a47240640 1 -- 172.21.15.40:0/3170790984 shutdown_connections 2024-04-21T23:16:08.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.407+0000 7f2a47240640 1 -- 172.21.15.40:0/3170790984 wait complete. 2024-04-21T23:16:08.417 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T23:11:14.824775+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T23:15:54.322571+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T23:15:55.164493+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T23:15:58.838266+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T23:16:08.418 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T23:16:08.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.527+0000 7f6a182c2640 1 -- 172.21.15.40:0/273725930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a10101890 msgr2=0x7f6a10101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:08.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.527+0000 7f6a182c2640 1 --2- 172.21.15.40:0/273725930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a10101890 0x7f6a10101d10 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f6a00009900 tx=0x7f6a0002bc30 comp rx=0 tx=0).stop 2024-04-21T23:16:08.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.527+0000 7f6a182c2640 1 -- 172.21.15.40:0/273725930 shutdown_connections 2024-04-21T23:16:08.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.527+0000 7f6a182c2640 1 --2- 172.21.15.40:0/273725930 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6a10102250 0x7f6a10108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:08.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.527+0000 7f6a182c2640 1 --2- 172.21.15.40:0/273725930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a10101890 0x7f6a10101d10 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:08.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.527+0000 7f6a182c2640 1 --2- 172.21.15.40:0/273725930 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a101006a0 0x7f6a10100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:08.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.527+0000 7f6a182c2640 1 -- 172.21.15.40:0/273725930 >> 172.21.15.40:0/273725930 conn(0x7f6a100fbc30 msgr2=0x7f6a100fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:08.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.527+0000 7f6a182c2640 1 -- 172.21.15.40:0/273725930 shutdown_connections 2024-04-21T23:16:08.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.527+0000 7f6a182c2640 1 -- 172.21.15.40:0/273725930 wait complete. 2024-04-21T23:16:08.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.528+0000 7f6a182c2640 1 Processor -- start 2024-04-21T23:16:08.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.528+0000 7f6a182c2640 1 -- start start 2024-04-21T23:16:08.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.528+0000 7f6a182c2640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a101006a0 0x7f6a10199d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:08.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.528+0000 7f6a182c2640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a10101890 0x7f6a1019a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:08.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.528+0000 7f6a182c2640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6a10102250 0x7f6a101a1270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:08.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.528+0000 7f6a182c2640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6a1010d440 con 0x7f6a101006a0 2024-04-21T23:16:08.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.528+0000 7f6a182c2640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f6a1010d2c0 con 0x7f6a10102250 2024-04-21T23:16:08.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.528+0000 7f6a182c2640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6a1010d5c0 con 0x7f6a10101890 2024-04-21T23:16:08.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.528+0000 7f6a1605e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a101006a0 0x7f6a10199d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:08.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.528+0000 7f6a1605e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a101006a0 0x7f6a10199d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33262/0 (socket says 172.21.15.40:33262) 2024-04-21T23:16:08.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.528+0000 7f6a1605e640 1 -- 172.21.15.40:0/4288853795 learned_addr learned my addr 172.21.15.40:0/4288853795 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:16:08.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.528+0000 7f6a1685f640 1 --2- 172.21.15.40:0/4288853795 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6a10102250 0x7f6a101a1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:08.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.528+0000 7f6a1585d640 1 --2- 172.21.15.40:0/4288853795 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a10101890 0x7f6a1019a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:08.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.528+0000 7f6a1605e640 1 -- 172.21.15.40:0/4288853795 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a10101890 msgr2=0x7f6a1019a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:08.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.528+0000 7f6a1605e640 1 --2- 172.21.15.40:0/4288853795 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a10101890 0x7f6a1019a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:08.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.528+0000 7f6a1605e640 1 -- 172.21.15.40:0/4288853795 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6a10102250 msgr2=0x7f6a101a1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:08.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.528+0000 7f6a1605e640 1 --2- 172.21.15.40:0/4288853795 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6a10102250 0x7f6a101a1270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:08.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.528+0000 7f6a1605e640 1 -- 172.21.15.40:0/4288853795 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a101a1910 con 0x7f6a101006a0 2024-04-21T23:16:08.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.528+0000 7f6a1585d640 1 --2- 172.21.15.40:0/4288853795 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a10101890 0x7f6a1019a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:16:08.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.528+0000 7f6a1685f640 1 --2- 172.21.15.40:0/4288853795 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6a10102250 0x7f6a101a1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:16:08.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.528+0000 7f6a1605e640 1 --2- 172.21.15.40:0/4288853795 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a101006a0 0x7f6a10199d00 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f6a0c00d950 tx=0x7f6a0c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:08.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.529+0000 7f6a077fe640 1 -- 172.21.15.40:0/4288853795 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a0c017070 con 0x7f6a101006a0 2024-04-21T23:16:08.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.529+0000 7f6a077fe640 1 -- 172.21.15.40:0/4288853795 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a0c009c30 con 0x7f6a101006a0 2024-04-21T23:16:08.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.529+0000 7f6a077fe640 1 -- 172.21.15.40:0/4288853795 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a0c00f600 con 0x7f6a101006a0 2024-04-21T23:16:08.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.529+0000 7f6a182c2640 1 -- 172.21.15.40:0/4288853795 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a101a1c00 con 0x7f6a101006a0 2024-04-21T23:16:08.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.529+0000 7f6a182c2640 1 -- 172.21.15.40:0/4288853795 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a101a2100 con 0x7f6a101006a0 2024-04-21T23:16:08.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.530+0000 7f6a182c2640 1 -- 172.21.15.40:0/4288853795 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a101a22d0 con 0x7f6a101006a0 2024-04-21T23:16:08.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.531+0000 7f6a077fe640 1 -- 172.21.15.40:0/4288853795 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f6a0c01a070 con 0x7f6a101006a0 2024-04-21T23:16:08.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.531+0000 7f6a077fe640 1 --2- 172.21.15.40:0/4288853795 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f69fc0a2c10 0x7f69fc0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:08.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.531+0000 7f6a077fe640 1 -- 172.21.15.40:0/4288853795 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f6a0c0cee40 con 0x7f6a101006a0 2024-04-21T23:16:08.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.532+0000 7f6a1585d640 1 --2- 172.21.15.40:0/4288853795 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f69fc0a2c10 0x7f69fc0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:08.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.532+0000 7f6a1585d640 1 --2- 172.21.15.40:0/4288853795 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f69fc0a2c10 0x7f69fc0a50d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f6a000096c0 tx=0x7f6a00000bc0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:08.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.536+0000 7f6a077fe640 1 -- 172.21.15.40:0/4288853795 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f6a0c009de0 con 0x7f6a101006a0 2024-04-21T23:16:08.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:08 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/3170790984' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:16:08.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:08 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/3170790984' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:16:08.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:08 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/3170790984' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:16:08.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.722+0000 7f6a182c2640 1 -- 172.21.15.40:0/4288853795 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f6a101a2b30 con 0x7f6a101006a0 2024-04-21T23:16:08.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.722+0000 7f6a077fe640 1 -- 172.21.15.40:0/4288853795 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) ==== 93+0+10 (secure 0 0 0) 0x7f6a0c09bb00 con 0x7f6a101006a0 2024-04-21T23:16:08.722 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-04-21T23:16:08.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.724+0000 7f6a182c2640 1 -- 172.21.15.40:0/4288853795 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f69fc0a2c10 msgr2=0x7f69fc0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:08.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.724+0000 7f6a182c2640 1 --2- 172.21.15.40:0/4288853795 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f69fc0a2c10 0x7f69fc0a50d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f6a000096c0 tx=0x7f6a00000bc0 comp rx=0 tx=0).stop 2024-04-21T23:16:08.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.724+0000 7f6a182c2640 1 -- 172.21.15.40:0/4288853795 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a101006a0 msgr2=0x7f6a10199d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:08.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.724+0000 7f6a182c2640 1 --2- 172.21.15.40:0/4288853795 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a101006a0 0x7f6a10199d00 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f6a0c00d950 tx=0x7f6a0c00de10 comp rx=0 tx=0).stop 2024-04-21T23:16:08.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.724+0000 7f6a182c2640 1 -- 172.21.15.40:0/4288853795 shutdown_connections 2024-04-21T23:16:08.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.724+0000 7f6a182c2640 1 --2- 172.21.15.40:0/4288853795 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f69fc0a2c10 0x7f69fc0a50d0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:08.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.724+0000 7f6a182c2640 1 --2- 172.21.15.40:0/4288853795 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6a10102250 0x7f6a101a1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:08.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.724+0000 7f6a182c2640 1 --2- 172.21.15.40:0/4288853795 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a10101890 0x7f6a1019a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:08.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.724+0000 7f6a182c2640 1 --2- 172.21.15.40:0/4288853795 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a101006a0 0x7f6a10199d00 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:08.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.724+0000 7f6a182c2640 1 -- 172.21.15.40:0/4288853795 >> 172.21.15.40:0/4288853795 conn(0x7f6a100fbc30 msgr2=0x7f6a101094a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:08.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.724+0000 7f6a182c2640 1 -- 172.21.15.40:0/4288853795 shutdown_connections 2024-04-21T23:16:08.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.724+0000 7f6a182c2640 1 -- 172.21.15.40:0/4288853795 wait complete. 2024-04-21T23:16:08.734 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T23:16:08.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.840+0000 7f99f0d27640 1 -- 172.21.15.40:0/603562010 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99ec101870 msgr2=0x7f99ec101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:08.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.840+0000 7f99f0d27640 1 --2- 172.21.15.40:0/603562010 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99ec101870 0x7f99ec101cf0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f99dc009900 tx=0x7f99dc02bc30 comp rx=0 tx=0).stop 2024-04-21T23:16:08.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.840+0000 7f99f0d27640 1 -- 172.21.15.40:0/603562010 shutdown_connections 2024-04-21T23:16:08.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.840+0000 7f99f0d27640 1 --2- 172.21.15.40:0/603562010 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99ec102230 0x7f99ec108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:08.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.840+0000 7f99f0d27640 1 --2- 172.21.15.40:0/603562010 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99ec101870 0x7f99ec101cf0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:08.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.840+0000 7f99f0d27640 1 --2- 172.21.15.40:0/603562010 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f99ec100680 0x7f99ec100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:08.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.840+0000 7f99f0d27640 1 -- 172.21.15.40:0/603562010 >> 172.21.15.40:0/603562010 conn(0x7f99ec0fbbf0 msgr2=0x7f99ec0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:08.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.840+0000 7f99f0d27640 1 -- 172.21.15.40:0/603562010 shutdown_connections 2024-04-21T23:16:08.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.840+0000 7f99f0d27640 1 -- 172.21.15.40:0/603562010 wait complete. 2024-04-21T23:16:08.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.840+0000 7f99f0d27640 1 Processor -- start 2024-04-21T23:16:08.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.841+0000 7f99f0d27640 1 -- start start 2024-04-21T23:16:08.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.841+0000 7f99f0d27640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99ec100680 0x7f99ec196980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:08.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.841+0000 7f99f0d27640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f99ec101870 0x7f99ec196ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:08.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.841+0000 7f99f0d27640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99ec102230 0x7f99ec19df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:08.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.841+0000 7f99f0d27640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f99ec10c2e0 con 0x7f99ec102230 2024-04-21T23:16:08.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.841+0000 7f99f0d27640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f99ec10c160 con 0x7f99ec100680 2024-04-21T23:16:08.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.841+0000 7f99f0d27640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f99ec10c460 con 0x7f99ec101870 2024-04-21T23:16:08.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.841+0000 7f99ead9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99ec102230 0x7f99ec19df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:08.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.841+0000 7f99ead9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99ec102230 0x7f99ec19df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33278/0 (socket says 172.21.15.40:33278) 2024-04-21T23:16:08.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.841+0000 7f99ead9d640 1 -- 172.21.15.40:0/3118698935 learned_addr learned my addr 172.21.15.40:0/3118698935 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:16:08.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.841+0000 7f99ea59c640 1 --2- 172.21.15.40:0/3118698935 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99ec100680 0x7f99ec196980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:08.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.841+0000 7f99e9d9b640 1 --2- 172.21.15.40:0/3118698935 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f99ec101870 0x7f99ec196ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:08.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.841+0000 7f99ead9d640 1 -- 172.21.15.40:0/3118698935 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f99ec101870 msgr2=0x7f99ec196ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:08.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.841+0000 7f99ead9d640 1 --2- 172.21.15.40:0/3118698935 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f99ec101870 0x7f99ec196ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:08.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.841+0000 7f99ead9d640 1 -- 172.21.15.40:0/3118698935 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99ec100680 msgr2=0x7f99ec196980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:08.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.841+0000 7f99ead9d640 1 --2- 172.21.15.40:0/3118698935 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99ec100680 0x7f99ec196980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:08.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.841+0000 7f99ead9d640 1 -- 172.21.15.40:0/3118698935 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99ec19e5a0 con 0x7f99ec102230 2024-04-21T23:16:08.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.841+0000 7f99e9d9b640 1 --2- 172.21.15.40:0/3118698935 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f99ec101870 0x7f99ec196ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:16:08.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.841+0000 7f99ead9d640 1 --2- 172.21.15.40:0/3118698935 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99ec102230 0x7f99ec19df30 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f99e400cce0 tx=0x7f99e400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:08.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.842+0000 7f99e37fe640 1 -- 172.21.15.40:0/3118698935 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99e400f450 con 0x7f99ec102230 2024-04-21T23:16:08.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.842+0000 7f99f0d27640 1 -- 172.21.15.40:0/3118698935 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99ec19e890 con 0x7f99ec102230 2024-04-21T23:16:08.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.842+0000 7f99e37fe640 1 -- 172.21.15.40:0/3118698935 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f99e400fad0 con 0x7f99ec102230 2024-04-21T23:16:08.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.842+0000 7f99e37fe640 1 -- 172.21.15.40:0/3118698935 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99e4018c10 con 0x7f99ec102230 2024-04-21T23:16:08.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.842+0000 7f99f0d27640 1 -- 172.21.15.40:0/3118698935 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99ec19edb0 con 0x7f99ec102230 2024-04-21T23:16:08.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.843+0000 7f99f0d27640 1 -- 172.21.15.40:0/3118698935 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99ec1079a0 con 0x7f99ec102230 2024-04-21T23:16:08.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.844+0000 7f99e37fe640 1 -- 172.21.15.40:0/3118698935 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f99e400f5f0 con 0x7f99ec102230 2024-04-21T23:16:08.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.844+0000 7f99e37fe640 1 --2- 172.21.15.40:0/3118698935 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f99d80a2b90 0x7f99d80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:08.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.844+0000 7f99ea59c640 1 --2- 172.21.15.40:0/3118698935 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f99d80a2b90 0x7f99d80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:08.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.844+0000 7f99e37fe640 1 -- 172.21.15.40:0/3118698935 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f99e40cfa10 con 0x7f99ec102230 2024-04-21T23:16:08.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.845+0000 7f99ea59c640 1 --2- 172.21.15.40:0/3118698935 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f99d80a2b90 0x7f99d80a5050 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f99d4005fd0 tx=0x7f99d4005850 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:08.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:08.849+0000 7f99e37fe640 1 -- 172.21.15.40:0/3118698935 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f99e4014030 con 0x7f99ec102230 2024-04-21T23:16:09.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.032+0000 7f99f0d27640 1 -- 172.21.15.40:0/3118698935 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f99ec062340 con 0x7f99ec102230 2024-04-21T23:16:09.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.033+0000 7f99e37fe640 1 -- 172.21.15.40:0/3118698935 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) ==== 104+0+11 (secure 0 0 0) 0x7f99e409c730 con 0x7f99ec102230 2024-04-21T23:16:09.032 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-21T23:16:09.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.034+0000 7f99f0d27640 1 -- 172.21.15.40:0/3118698935 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f99d80a2b90 msgr2=0x7f99d80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:09.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.034+0000 7f99f0d27640 1 --2- 172.21.15.40:0/3118698935 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f99d80a2b90 0x7f99d80a5050 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f99d4005fd0 tx=0x7f99d4005850 comp rx=0 tx=0).stop 2024-04-21T23:16:09.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.034+0000 7f99f0d27640 1 -- 172.21.15.40:0/3118698935 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99ec102230 msgr2=0x7f99ec19df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:09.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.034+0000 7f99f0d27640 1 --2- 172.21.15.40:0/3118698935 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99ec102230 0x7f99ec19df30 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f99e400cce0 tx=0x7f99e400a620 comp rx=0 tx=0).stop 2024-04-21T23:16:09.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.034+0000 7f99f0d27640 1 -- 172.21.15.40:0/3118698935 shutdown_connections 2024-04-21T23:16:09.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.034+0000 7f99f0d27640 1 --2- 172.21.15.40:0/3118698935 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f99d80a2b90 0x7f99d80a5050 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:09.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.034+0000 7f99f0d27640 1 --2- 172.21.15.40:0/3118698935 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99ec102230 0x7f99ec19df30 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:09.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.034+0000 7f99f0d27640 1 --2- 172.21.15.40:0/3118698935 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f99ec101870 0x7f99ec196ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:09.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.034+0000 7f99f0d27640 1 --2- 172.21.15.40:0/3118698935 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99ec100680 0x7f99ec196980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:09.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.034+0000 7f99f0d27640 1 -- 172.21.15.40:0/3118698935 >> 172.21.15.40:0/3118698935 conn(0x7f99ec0fbbf0 msgr2=0x7f99ec104b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:09.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.035+0000 7f99f0d27640 1 -- 172.21.15.40:0/3118698935 shutdown_connections 2024-04-21T23:16:09.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.035+0000 7f99f0d27640 1 -- 172.21.15.40:0/3118698935 wait complete. 2024-04-21T23:16:09.045 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T23:16:09.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.150+0000 7fd784e9d640 1 -- 172.21.15.40:0/3984473674 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd780102990 msgr2=0x7fd780102e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:09.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.150+0000 7fd784e9d640 1 --2- 172.21.15.40:0/3984473674 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd780102990 0x7fd780102e10 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fd768009900 tx=0x7fd76802bc30 comp rx=0 tx=0).stop 2024-04-21T23:16:09.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.150+0000 7fd784e9d640 1 -- 172.21.15.40:0/3984473674 shutdown_connections 2024-04-21T23:16:09.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.150+0000 7fd784e9d640 1 --2- 172.21.15.40:0/3984473674 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd780103350 0x7fd780109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:09.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.150+0000 7fd784e9d640 1 --2- 172.21.15.40:0/3984473674 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd780102990 0x7fd780102e10 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:09.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.150+0000 7fd784e9d640 1 --2- 172.21.15.40:0/3984473674 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd7801017a0 0x7fd780101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:09.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.150+0000 7fd784e9d640 1 -- 172.21.15.40:0/3984473674 >> 172.21.15.40:0/3984473674 conn(0x7fd7800fcd10 msgr2=0x7fd7800ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:09.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.150+0000 7fd784e9d640 1 -- 172.21.15.40:0/3984473674 shutdown_connections 2024-04-21T23:16:09.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.150+0000 7fd784e9d640 1 -- 172.21.15.40:0/3984473674 wait complete. 2024-04-21T23:16:09.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.151+0000 7fd784e9d640 1 Processor -- start 2024-04-21T23:16:09.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.151+0000 7fd784e9d640 1 -- start start 2024-04-21T23:16:09.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.151+0000 7fd784e9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd7801017a0 0x7fd780197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:09.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.151+0000 7fd784e9d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd780102990 0x7fd780197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:09.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.151+0000 7fd784e9d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd780103350 0x7fd78019f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:09.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.151+0000 7fd784e9d640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd78010d490 con 0x7fd7801017a0 2024-04-21T23:16:09.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.151+0000 7fd784e9d640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd78010d310 con 0x7fd780103350 2024-04-21T23:16:09.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.151+0000 7fd784e9d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd78010d610 con 0x7fd780102990 2024-04-21T23:16:09.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.151+0000 7fd77e59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd7801017a0 0x7fd780197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:09.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.151+0000 7fd77e59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd7801017a0 0x7fd780197aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33296/0 (socket says 172.21.15.40:33296) 2024-04-21T23:16:09.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.151+0000 7fd77e59c640 1 -- 172.21.15.40:0/2524485593 learned_addr learned my addr 172.21.15.40:0/2524485593 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:16:09.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.151+0000 7fd775d9b640 1 --2- 172.21.15.40:0/2524485593 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd780102990 0x7fd780197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:09.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.151+0000 7fd77e59c640 1 -- 172.21.15.40:0/2524485593 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd780102990 msgr2=0x7fd780197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:09.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.151+0000 7fd77e59c640 1 --2- 172.21.15.40:0/2524485593 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd780102990 0x7fd780197fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:09.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.152+0000 7fd77ed9d640 1 --2- 172.21.15.40:0/2524485593 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd780103350 0x7fd78019f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:09.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.152+0000 7fd77e59c640 1 -- 172.21.15.40:0/2524485593 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd780103350 msgr2=0x7fd78019f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:09.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.152+0000 7fd77e59c640 1 --2- 172.21.15.40:0/2524485593 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd780103350 0x7fd78019f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:09.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.152+0000 7fd77e59c640 1 -- 172.21.15.40:0/2524485593 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd78019f6c0 con 0x7fd7801017a0 2024-04-21T23:16:09.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.152+0000 7fd77ed9d640 1 --2- 172.21.15.40:0/2524485593 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd780103350 0x7fd78019f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:16:09.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.152+0000 7fd77e59c640 1 --2- 172.21.15.40:0/2524485593 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd7801017a0 0x7fd780197aa0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fd77000b450 tx=0x7fd77000b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:09.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.152+0000 7fd777fff640 1 -- 172.21.15.40:0/2524485593 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd770015020 con 0x7fd7801017a0 2024-04-21T23:16:09.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.152+0000 7fd784e9d640 1 -- 172.21.15.40:0/2524485593 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd78019f9b0 con 0x7fd7801017a0 2024-04-21T23:16:09.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.152+0000 7fd777fff640 1 -- 172.21.15.40:0/2524485593 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd770011d70 con 0x7fd7801017a0 2024-04-21T23:16:09.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.152+0000 7fd777fff640 1 -- 172.21.15.40:0/2524485593 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd77000f340 con 0x7fd7801017a0 2024-04-21T23:16:09.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.152+0000 7fd784e9d640 1 -- 172.21.15.40:0/2524485593 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd78019feb0 con 0x7fd7801017a0 2024-04-21T23:16:09.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.153+0000 7fd784e9d640 1 -- 172.21.15.40:0/2524485593 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd780108ac0 con 0x7fd7801017a0 2024-04-21T23:16:09.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.154+0000 7fd777fff640 1 -- 172.21.15.40:0/2524485593 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fd770018020 con 0x7fd7801017a0 2024-04-21T23:16:09.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.155+0000 7fd777fff640 1 --2- 172.21.15.40:0/2524485593 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fd76c0a2b90 0x7fd76c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:09.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.155+0000 7fd777fff640 1 -- 172.21.15.40:0/2524485593 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fd7700cf370 con 0x7fd7801017a0 2024-04-21T23:16:09.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.155+0000 7fd775d9b640 1 --2- 172.21.15.40:0/2524485593 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fd76c0a2b90 0x7fd76c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:09.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.155+0000 7fd775d9b640 1 --2- 172.21.15.40:0/2524485593 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fd76c0a2b90 0x7fd76c0a5050 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fd76802c8e0 tx=0x7fd768005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:09.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.158+0000 7fd777fff640 1 -- 172.21.15.40:0/2524485593 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fd77001f020 con 0x7fd7801017a0 2024-04-21T23:16:09.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.341+0000 7fd784e9d640 1 -- 172.21.15.40:0/2524485593 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fd7800689b0 con 0x7fd7801017a0 2024-04-21T23:16:09.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.342+0000 7fd777fff640 1 -- 172.21.15.40:0/2524485593 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) ==== 100+0+11 (secure 0 0 0) 0x7fd77000d040 con 0x7fd7801017a0 2024-04-21T23:16:09.341 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-04-21T23:16:09.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.344+0000 7fd784e9d640 1 -- 172.21.15.40:0/2524485593 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fd76c0a2b90 msgr2=0x7fd76c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:09.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.344+0000 7fd784e9d640 1 --2- 172.21.15.40:0/2524485593 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fd76c0a2b90 0x7fd76c0a5050 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fd76802c8e0 tx=0x7fd768005fd0 comp rx=0 tx=0).stop 2024-04-21T23:16:09.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.344+0000 7fd784e9d640 1 -- 172.21.15.40:0/2524485593 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd7801017a0 msgr2=0x7fd780197aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:09.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.344+0000 7fd784e9d640 1 --2- 172.21.15.40:0/2524485593 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd7801017a0 0x7fd780197aa0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fd77000b450 tx=0x7fd77000b910 comp rx=0 tx=0).stop 2024-04-21T23:16:09.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.344+0000 7fd784e9d640 1 -- 172.21.15.40:0/2524485593 shutdown_connections 2024-04-21T23:16:09.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.344+0000 7fd784e9d640 1 --2- 172.21.15.40:0/2524485593 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fd76c0a2b90 0x7fd76c0a5050 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:09.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.344+0000 7fd784e9d640 1 --2- 172.21.15.40:0/2524485593 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd780103350 0x7fd78019f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:09.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.344+0000 7fd784e9d640 1 --2- 172.21.15.40:0/2524485593 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd780102990 0x7fd780197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:09.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.344+0000 7fd784e9d640 1 --2- 172.21.15.40:0/2524485593 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd7801017a0 0x7fd780197aa0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:09.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.344+0000 7fd784e9d640 1 -- 172.21.15.40:0/2524485593 >> 172.21.15.40:0/2524485593 conn(0x7fd7800fcd10 msgr2=0x7fd780105c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:09.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.344+0000 7fd784e9d640 1 -- 172.21.15.40:0/2524485593 shutdown_connections 2024-04-21T23:16:09.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.344+0000 7fd784e9d640 1 -- 172.21.15.40:0/2524485593 wait complete. 2024-04-21T23:16:09.354 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T23:16:09.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.460+0000 7f8837525640 1 -- 172.21.15.40:0/2086415186 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8830101870 msgr2=0x7f8830101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:09.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.460+0000 7f8837525640 1 --2- 172.21.15.40:0/2086415186 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8830101870 0x7f8830101cf0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f8820009900 tx=0x7f882002bc30 comp rx=0 tx=0).stop 2024-04-21T23:16:09.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.460+0000 7f8837525640 1 -- 172.21.15.40:0/2086415186 shutdown_connections 2024-04-21T23:16:09.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.460+0000 7f8837525640 1 --2- 172.21.15.40:0/2086415186 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8830102230 0x7f8830108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:09.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.460+0000 7f8837525640 1 --2- 172.21.15.40:0/2086415186 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8830101870 0x7f8830101cf0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:09.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.460+0000 7f8837525640 1 --2- 172.21.15.40:0/2086415186 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8830100680 0x7f8830100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:09.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.460+0000 7f8837525640 1 -- 172.21.15.40:0/2086415186 >> 172.21.15.40:0/2086415186 conn(0x7f88300fbbf0 msgr2=0x7f88300fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:09.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.460+0000 7f8837525640 1 -- 172.21.15.40:0/2086415186 shutdown_connections 2024-04-21T23:16:09.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.460+0000 7f8837525640 1 -- 172.21.15.40:0/2086415186 wait complete. 2024-04-21T23:16:09.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.460+0000 7f8837525640 1 Processor -- start 2024-04-21T23:16:09.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.460+0000 7f8837525640 1 -- start start 2024-04-21T23:16:09.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.460+0000 7f8837525640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8830100680 0x7f8830069460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:09.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.460+0000 7f8837525640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8830101870 0x7f883019c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:09.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.460+0000 7f8837525640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8830102230 0x7f883019eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:09.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.460+0000 7f8837525640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f883010c360 con 0x7f8830101870 2024-04-21T23:16:09.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.460+0000 7f8837525640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f883010c1e0 con 0x7f8830100680 2024-04-21T23:16:09.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.460+0000 7f8837525640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f883010c4e0 con 0x7f8830102230 2024-04-21T23:16:09.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.461+0000 7f8834ac0640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8830101870 0x7f883019c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:09.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.461+0000 7f8834ac0640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8830101870 0x7f883019c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33304/0 (socket says 172.21.15.40:33304) 2024-04-21T23:16:09.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.461+0000 7f8834ac0640 1 -- 172.21.15.40:0/2730881234 learned_addr learned my addr 172.21.15.40:0/2730881234 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:16:09.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.461+0000 7f88352c1640 1 --2- 172.21.15.40:0/2730881234 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8830100680 0x7f8830069460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:09.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.461+0000 7f8835ac2640 1 --2- 172.21.15.40:0/2730881234 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8830102230 0x7f883019eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:09.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.461+0000 7f8834ac0640 1 -- 172.21.15.40:0/2730881234 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8830102230 msgr2=0x7f883019eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:09.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.461+0000 7f8834ac0640 1 --2- 172.21.15.40:0/2730881234 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8830102230 0x7f883019eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:09.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.461+0000 7f8834ac0640 1 -- 172.21.15.40:0/2730881234 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8830100680 msgr2=0x7f8830069460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:09.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.461+0000 7f8834ac0640 1 --2- 172.21.15.40:0/2730881234 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8830100680 0x7f8830069460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:09.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.461+0000 7f8834ac0640 1 -- 172.21.15.40:0/2730881234 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f883019f760 con 0x7f8830101870 2024-04-21T23:16:09.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.461+0000 7f8835ac2640 1 --2- 172.21.15.40:0/2730881234 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8830102230 0x7f883019eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:16:09.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.461+0000 7f88352c1640 1 --2- 172.21.15.40:0/2730881234 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8830100680 0x7f8830069460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:16:09.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.461+0000 7f8834ac0640 1 --2- 172.21.15.40:0/2730881234 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8830101870 0x7f883019c860 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f8820009230 tx=0x7f8820004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:09.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.461+0000 7f88267fc640 1 -- 172.21.15.40:0/2730881234 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8820041070 con 0x7f8830101870 2024-04-21T23:16:09.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.461+0000 7f88267fc640 1 -- 172.21.15.40:0/2730881234 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f88200054e0 con 0x7f8830101870 2024-04-21T23:16:09.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.461+0000 7f8837525640 1 -- 172.21.15.40:0/2730881234 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f883019f9f0 con 0x7f8830101870 2024-04-21T23:16:09.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.461+0000 7f88267fc640 1 -- 172.21.15.40:0/2730881234 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88200346a0 con 0x7f8830101870 2024-04-21T23:16:09.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.461+0000 7f8837525640 1 -- 172.21.15.40:0/2730881234 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f883019fee0 con 0x7f8830101870 2024-04-21T23:16:09.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.462+0000 7f8837525640 1 -- 172.21.15.40:0/2730881234 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8830107a20 con 0x7f8830101870 2024-04-21T23:16:09.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.464+0000 7f88267fc640 1 -- 172.21.15.40:0/2730881234 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f8820044070 con 0x7f8830101870 2024-04-21T23:16:09.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.464+0000 7f88267fc640 1 --2- 172.21.15.40:0/2730881234 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f881c0a2b90 0x7f881c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:09.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.464+0000 7f88267fc640 1 -- 172.21.15.40:0/2730881234 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f88200f1420 con 0x7f8830101870 2024-04-21T23:16:09.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.464+0000 7f88352c1640 1 --2- 172.21.15.40:0/2730881234 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f881c0a2b90 0x7f881c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:09.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.465+0000 7f88352c1640 1 --2- 172.21.15.40:0/2730881234 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f881c0a2b90 0x7f881c0a5050 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f88301016d0 tx=0x7f882c00a560 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:09.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.468+0000 7f88267fc640 1 -- 172.21.15.40:0/2730881234 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f88200be140 con 0x7f8830101870 2024-04-21T23:16:09.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.652+0000 7f8837525640 1 -- 172.21.15.40:0/2730881234 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f8830062340 con 0x7f8830101870 2024-04-21T23:16:09.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.653+0000 7f88267fc640 1 -- 172.21.15.40:0/2730881234 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v90) ==== 103+0+11 (secure 0 0 0) 0x7f88200bda40 con 0x7f8830101870 2024-04-21T23:16:09.652 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-21T23:16:09.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.654+0000 7f8837525640 1 -- 172.21.15.40:0/2730881234 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f881c0a2b90 msgr2=0x7f881c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:09.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.654+0000 7f8837525640 1 --2- 172.21.15.40:0/2730881234 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f881c0a2b90 0x7f881c0a5050 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f88301016d0 tx=0x7f882c00a560 comp rx=0 tx=0).stop 2024-04-21T23:16:09.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.654+0000 7f8837525640 1 -- 172.21.15.40:0/2730881234 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8830101870 msgr2=0x7f883019c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:09.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.654+0000 7f8837525640 1 --2- 172.21.15.40:0/2730881234 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8830101870 0x7f883019c860 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f8820009230 tx=0x7f8820004040 comp rx=0 tx=0).stop 2024-04-21T23:16:09.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.654+0000 7f8837525640 1 -- 172.21.15.40:0/2730881234 shutdown_connections 2024-04-21T23:16:09.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.654+0000 7f8837525640 1 --2- 172.21.15.40:0/2730881234 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f881c0a2b90 0x7f881c0a5050 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:09.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.654+0000 7f8837525640 1 --2- 172.21.15.40:0/2730881234 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8830102230 0x7f883019eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:09.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.654+0000 7f8837525640 1 --2- 172.21.15.40:0/2730881234 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8830101870 0x7f883019c860 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:09.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.654+0000 7f8837525640 1 --2- 172.21.15.40:0/2730881234 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8830100680 0x7f8830069460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:09.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.654+0000 7f8837525640 1 -- 172.21.15.40:0/2730881234 >> 172.21.15.40:0/2730881234 conn(0x7f88300fbbf0 msgr2=0x7f8830104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:09.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.655+0000 7f8837525640 1 -- 172.21.15.40:0/2730881234 shutdown_connections 2024-04-21T23:16:09.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.655+0000 7f8837525640 1 -- 172.21.15.40:0/2730881234 wait complete. 2024-04-21T23:16:09.664 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:09 smithi040 ceph-mon[27984]: pgmap v299: 129 pgs: 129 active+clean; 579 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-04-21T23:16:09.664 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:09 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/4288853795' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:16:09.664 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:09 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/3118698935' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T23:16:09.664 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:09 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/2524485593' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T23:16:09.665 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-21T23:16:09.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:09 smithi174 ceph-mon[31409]: pgmap v299: 129 pgs: 129 active+clean; 579 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-04-21T23:16:09.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:09 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/4288853795' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:16:09.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:09 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/3118698935' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T23:16:09.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:09 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/2524485593' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T23:16:09.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:09 smithi111 ceph-mon[31085]: pgmap v299: 129 pgs: 129 active+clean; 579 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-04-21T23:16:09.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:09 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/4288853795' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:16:09.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:09 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/3118698935' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T23:16:09.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:09 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/2524485593' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T23:16:09.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.770+0000 7f1be65a1640 1 -- 172.21.15.40:0/1024377549 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1be01017a0 msgr2=0x7f1be0101bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:09.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.770+0000 7f1be65a1640 1 --2- 172.21.15.40:0/1024377549 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1be01017a0 0x7f1be0101bc0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f1bd0009900 tx=0x7f1bd002bc30 comp rx=0 tx=0).stop 2024-04-21T23:16:09.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.770+0000 7f1be65a1640 1 -- 172.21.15.40:0/1024377549 shutdown_connections 2024-04-21T23:16:09.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.770+0000 7f1be65a1640 1 --2- 172.21.15.40:0/1024377549 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1be0103350 0x7f1be0109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:09.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.770+0000 7f1be65a1640 1 --2- 172.21.15.40:0/1024377549 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1be0102990 0x7f1be0102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:09.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.770+0000 7f1be65a1640 1 --2- 172.21.15.40:0/1024377549 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1be01017a0 0x7f1be0101bc0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:09.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.770+0000 7f1be65a1640 1 -- 172.21.15.40:0/1024377549 >> 172.21.15.40:0/1024377549 conn(0x7f1be00fcd10 msgr2=0x7f1be00ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:09.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.770+0000 7f1be65a1640 1 -- 172.21.15.40:0/1024377549 shutdown_connections 2024-04-21T23:16:09.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.770+0000 7f1be65a1640 1 -- 172.21.15.40:0/1024377549 wait complete. 2024-04-21T23:16:09.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.770+0000 7f1be65a1640 1 Processor -- start 2024-04-21T23:16:09.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.770+0000 7f1be65a1640 1 -- start start 2024-04-21T23:16:09.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.771+0000 7f1be65a1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1be01017a0 0x7f1be0197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:09.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.771+0000 7f1be65a1640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1be0102990 0x7f1be0197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:09.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.771+0000 7f1be65a1640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1be0103350 0x7f1be019f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:09.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.771+0000 7f1be65a1640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1be010d410 con 0x7f1be0102990 2024-04-21T23:16:09.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.771+0000 7f1be65a1640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f1be010d290 con 0x7f1be0103350 2024-04-21T23:16:09.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.771+0000 7f1be65a1640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1be010d590 con 0x7f1be01017a0 2024-04-21T23:16:09.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.771+0000 7f1bdf7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1be0102990 0x7f1be0197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:09.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.771+0000 7f1bdf7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1be0102990 0x7f1be0197fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33330/0 (socket says 172.21.15.40:33330) 2024-04-21T23:16:09.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.771+0000 7f1bdf7fe640 1 -- 172.21.15.40:0/3018181361 learned_addr learned my addr 172.21.15.40:0/3018181361 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:16:09.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.771+0000 7f1bdffff640 1 --2- 172.21.15.40:0/3018181361 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1be01017a0 0x7f1be0197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:09.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.771+0000 7f1be4b3e640 1 --2- 172.21.15.40:0/3018181361 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1be0103350 0x7f1be019f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:09.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.771+0000 7f1bdf7fe640 1 -- 172.21.15.40:0/3018181361 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1be01017a0 msgr2=0x7f1be0197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:09.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.771+0000 7f1bdf7fe640 1 --2- 172.21.15.40:0/3018181361 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1be01017a0 0x7f1be0197a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:09.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.771+0000 7f1bdf7fe640 1 -- 172.21.15.40:0/3018181361 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1be0103350 msgr2=0x7f1be019f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:09.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.771+0000 7f1bdf7fe640 1 --2- 172.21.15.40:0/3018181361 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1be0103350 0x7f1be019f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:09.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.771+0000 7f1bdf7fe640 1 -- 172.21.15.40:0/3018181361 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1be019f6b0 con 0x7f1be0102990 2024-04-21T23:16:09.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.771+0000 7f1bdf7fe640 1 --2- 172.21.15.40:0/3018181361 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1be0102990 0x7f1be0197fd0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f1bd400c970 tx=0x7f1bd400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:09.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.771+0000 7f1bdd7fa640 1 -- 172.21.15.40:0/3018181361 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bd4007bb0 con 0x7f1be0102990 2024-04-21T23:16:09.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.771+0000 7f1be65a1640 1 -- 172.21.15.40:0/3018181361 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1be019f9a0 con 0x7f1be0102990 2024-04-21T23:16:09.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.771+0000 7f1be65a1640 1 -- 172.21.15.40:0/3018181361 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1be019fef0 con 0x7f1be0102990 2024-04-21T23:16:09.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.771+0000 7f1bdd7fa640 1 -- 172.21.15.40:0/3018181361 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1bd4007d50 con 0x7f1be0102990 2024-04-21T23:16:09.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.771+0000 7f1bdd7fa640 1 -- 172.21.15.40:0/3018181361 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bd401d060 con 0x7f1be0102990 2024-04-21T23:16:09.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.773+0000 7f1be65a1640 1 -- 172.21.15.40:0/3018181361 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1be0108b40 con 0x7f1be0102990 2024-04-21T23:16:09.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.774+0000 7f1bdd7fa640 1 -- 172.21.15.40:0/3018181361 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f1bd4005000 con 0x7f1be0102990 2024-04-21T23:16:09.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.774+0000 7f1bdd7fa640 1 --2- 172.21.15.40:0/3018181361 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f1bc80a2b90 0x7f1bc80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:09.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.774+0000 7f1bdffff640 1 --2- 172.21.15.40:0/3018181361 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f1bc80a2b90 0x7f1bc80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:09.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.774+0000 7f1bdd7fa640 1 -- 172.21.15.40:0/3018181361 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f1bd40ced00 con 0x7f1be0102990 2024-04-21T23:16:09.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.775+0000 7f1bdffff640 1 --2- 172.21.15.40:0/3018181361 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f1bc80a2b90 0x7f1bc80a5050 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f1bd0036040 tx=0x7f1bd0005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:09.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.779+0000 7f1bdd7fa640 1 -- 172.21.15.40:0/3018181361 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f1bd4014030 con 0x7f1be0102990 2024-04-21T23:16:09.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.977+0000 7f1be65a1640 1 -- 172.21.15.40:0/3018181361 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f1be002d170 con 0x7f1be0102990 2024-04-21T23:16:09.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.977+0000 7f1bdd7fa640 1 -- 172.21.15.40:0/3018181361 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v90) ==== 81+0+588 (secure 0 0 0) 0x7f1bd409ba20 con 0x7f1be0102990 2024-04-21T23:16:09.977 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:6801/526554319 2024-04-22T23:08:50.305770+0000 2024-04-21T23:16:09.977 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:6800/526554319 2024-04-22T23:08:50.305770+0000 2024-04-21T23:16:09.977 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:0/3178763471 2024-04-22T23:08:50.305770+0000 2024-04-21T23:16:09.977 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:0/4045911364 2024-04-22T23:08:50.305770+0000 2024-04-21T23:16:09.977 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:0/276300080 2024-04-22T23:08:50.305770+0000 2024-04-21T23:16:09.977 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:0/3689649814 2024-04-22T23:08:23.743030+0000 2024-04-21T23:16:09.977 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:0/2241116715 2024-04-22T23:08:23.743030+0000 2024-04-21T23:16:09.977 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:6801/1479040326 2024-04-22T23:08:23.743030+0000 2024-04-21T23:16:09.977 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:6800/1479040326 2024-04-22T23:08:23.743030+0000 2024-04-21T23:16:09.977 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:0/681298099 2024-04-22T23:08:23.743030+0000 2024-04-21T23:16:09.978 INFO:teuthology.orchestra.run.smithi040.stderr:listed 10 entries 2024-04-21T23:16:09.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.979+0000 7f1be65a1640 1 -- 172.21.15.40:0/3018181361 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f1bc80a2b90 msgr2=0x7f1bc80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:09.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.979+0000 7f1be65a1640 1 --2- 172.21.15.40:0/3018181361 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f1bc80a2b90 0x7f1bc80a5050 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f1bd0036040 tx=0x7f1bd0005fd0 comp rx=0 tx=0).stop 2024-04-21T23:16:09.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.979+0000 7f1be65a1640 1 -- 172.21.15.40:0/3018181361 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1be0102990 msgr2=0x7f1be0197fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:09.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.979+0000 7f1be65a1640 1 --2- 172.21.15.40:0/3018181361 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1be0102990 0x7f1be0197fd0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f1bd400c970 tx=0x7f1bd400ce30 comp rx=0 tx=0).stop 2024-04-21T23:16:09.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.979+0000 7f1be65a1640 1 -- 172.21.15.40:0/3018181361 shutdown_connections 2024-04-21T23:16:09.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.979+0000 7f1be65a1640 1 --2- 172.21.15.40:0/3018181361 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f1bc80a2b90 0x7f1bc80a5050 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:09.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.979+0000 7f1be65a1640 1 --2- 172.21.15.40:0/3018181361 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1be0103350 0x7f1be019f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:09.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.979+0000 7f1be65a1640 1 --2- 172.21.15.40:0/3018181361 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1be0102990 0x7f1be0197fd0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:09.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.979+0000 7f1be65a1640 1 --2- 172.21.15.40:0/3018181361 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1be01017a0 0x7f1be0197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:09.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.979+0000 7f1be65a1640 1 -- 172.21.15.40:0/3018181361 >> 172.21.15.40:0/3018181361 conn(0x7f1be00fcd10 msgr2=0x7f1be0105d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:09.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.980+0000 7f1be65a1640 1 -- 172.21.15.40:0/3018181361 shutdown_connections 2024-04-21T23:16:09.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:09.980+0000 7f1be65a1640 1 -- 172.21.15.40:0/3018181361 wait complete. 2024-04-21T23:16:09.990 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-21T23:16:09.990 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-21T23:16:10.006 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T23:16:10.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.149+0000 7f26139e5640 1 -- 172.21.15.40:0/4004477610 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f260c1017b0 msgr2=0x7f260c101bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:10.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.149+0000 7f26139e5640 1 --2- 172.21.15.40:0/4004477610 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f260c1017b0 0x7f260c101bd0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f2604009ca0 tx=0x7f260402c0f0 comp rx=0 tx=0).stop 2024-04-21T23:16:10.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.149+0000 7f26139e5640 1 -- 172.21.15.40:0/4004477610 shutdown_connections 2024-04-21T23:16:10.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.149+0000 7f26139e5640 1 --2- 172.21.15.40:0/4004477610 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f260c1029a0 0x7f260c102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:10.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.149+0000 7f26139e5640 1 --2- 172.21.15.40:0/4004477610 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f260c103360 0x7f260c109bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:10.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.149+0000 7f26139e5640 1 --2- 172.21.15.40:0/4004477610 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f260c1017b0 0x7f260c101bd0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:10.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.149+0000 7f26139e5640 1 -- 172.21.15.40:0/4004477610 >> 172.21.15.40:0/4004477610 conn(0x7f260c0fcd40 msgr2=0x7f260c0ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:10.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.150+0000 7f26139e5640 1 -- 172.21.15.40:0/4004477610 shutdown_connections 2024-04-21T23:16:10.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.150+0000 7f26139e5640 1 -- 172.21.15.40:0/4004477610 wait complete. 2024-04-21T23:16:10.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.150+0000 7f26139e5640 1 Processor -- start 2024-04-21T23:16:10.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.150+0000 7f26139e5640 1 -- start start 2024-04-21T23:16:10.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.150+0000 7f26139e5640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f260c1017b0 0x7f260c197b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:10.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.150+0000 7f26139e5640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f260c1029a0 0x7f260c1980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:10.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.150+0000 7f26139e5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f260c103360 0x7f260c19f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:10.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.150+0000 7f26139e5640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f260c10d480 con 0x7f260c103360 2024-04-21T23:16:10.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.150+0000 7f26139e5640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f260c10d300 con 0x7f260c1017b0 2024-04-21T23:16:10.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.150+0000 7f26139e5640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f260c10d600 con 0x7f260c1029a0 2024-04-21T23:16:10.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.151+0000 7f2611f82640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f260c103360 0x7f260c19f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:10.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.151+0000 7f2611f82640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f260c103360 0x7f260c19f130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33356/0 (socket says 172.21.15.40:33356) 2024-04-21T23:16:10.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.151+0000 7f2611f82640 1 -- 172.21.15.40:0/3039093242 learned_addr learned my addr 172.21.15.40:0/3039093242 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:16:10.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.151+0000 7f2611781640 1 --2- 172.21.15.40:0/3039093242 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f260c1017b0 0x7f260c197b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:10.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.151+0000 7f2610f80640 1 --2- 172.21.15.40:0/3039093242 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f260c1029a0 0x7f260c1980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:10.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.151+0000 7f2611f82640 1 -- 172.21.15.40:0/3039093242 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f260c1029a0 msgr2=0x7f260c1980c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:10.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.151+0000 7f2611f82640 1 --2- 172.21.15.40:0/3039093242 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f260c1029a0 0x7f260c1980c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:10.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.151+0000 7f2611f82640 1 -- 172.21.15.40:0/3039093242 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f260c1017b0 msgr2=0x7f260c197b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:10.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.151+0000 7f2611f82640 1 --2- 172.21.15.40:0/3039093242 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f260c1017b0 0x7f260c197b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:10.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.151+0000 7f2611f82640 1 -- 172.21.15.40:0/3039093242 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f260c19f7a0 con 0x7f260c103360 2024-04-21T23:16:10.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.151+0000 7f2610f80640 1 --2- 172.21.15.40:0/3039093242 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f260c1029a0 0x7f260c1980c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:16:10.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.151+0000 7f2611f82640 1 --2- 172.21.15.40:0/3039093242 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f260c103360 0x7f260c19f130 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f260800c970 tx=0x7f260800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:10.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.151+0000 7f26027fc640 1 -- 172.21.15.40:0/3039093242 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2608007bb0 con 0x7f260c103360 2024-04-21T23:16:10.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.151+0000 7f26139e5640 1 -- 172.21.15.40:0/3039093242 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f260c19fa90 con 0x7f260c103360 2024-04-21T23:16:10.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.151+0000 7f26027fc640 1 -- 172.21.15.40:0/3039093242 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2608007d50 con 0x7f260c103360 2024-04-21T23:16:10.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.151+0000 7f26027fc640 1 -- 172.21.15.40:0/3039093242 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f260800f450 con 0x7f260c103360 2024-04-21T23:16:10.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.151+0000 7f26139e5640 1 -- 172.21.15.40:0/3039093242 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f260c19ffe0 con 0x7f260c103360 2024-04-21T23:16:10.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.152+0000 7f26139e5640 1 -- 172.21.15.40:0/3039093242 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f260c108b40 con 0x7f260c103360 2024-04-21T23:16:10.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.153+0000 7f26027fc640 1 -- 172.21.15.40:0/3039093242 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f2608004f60 con 0x7f260c103360 2024-04-21T23:16:10.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.154+0000 7f26027fc640 1 --2- 172.21.15.40:0/3039093242 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f25f80a2b90 0x7f25f80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:10.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.154+0000 7f2611781640 1 --2- 172.21.15.40:0/3039093242 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f25f80a2b90 0x7f25f80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:10.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.154+0000 7f26027fc640 1 -- 172.21.15.40:0/3039093242 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f26080cf0b0 con 0x7f260c103360 2024-04-21T23:16:10.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.154+0000 7f2611781640 1 --2- 172.21.15.40:0/3039093242 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f25f80a2b90 0x7f25f80a5050 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f260402c700 tx=0x7f260402cca0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:10.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.158+0000 7f26027fc640 1 -- 172.21.15.40:0/3039093242 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f2608014030 con 0x7f260c103360 2024-04-21T23:16:10.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.344+0000 7f26139e5640 1 -- 172.21.15.40:0/3039093242 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f260c062340 con 0x7f260c103360 2024-04-21T23:16:10.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.346+0000 7f26027fc640 1 -- 172.21.15.40:0/3039093242 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24138 (secure 0 0 0) 0x7f260809bd70 con 0x7f260c103360 2024-04-21T23:16:10.345 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:16:10.345 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":90,"fsid":"d814e4dc-0033-11ef-bc93-c7b262605968","created":"2024-04-21T23:07:52.659207+0000","modified":"2024-04-21T23:16:00.780824+0000","last_up_change":"2024-04-21T23:15:05.308821+0000","last_in_change":"2024-04-21T23:14:45.447491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:11:14.824775+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T23:15:54.322571+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T23:15:55.164493+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T23:15:58.838266+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"0e8d692a-22fe-4a87-8d99-9151f96cbb02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6803","nonce":2009758784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6805","nonce":2009758784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6809","nonce":2009758784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6807","nonce":2009758784}]},"public_addr":"172.21.15.40:6803/2009758784","cluster_addr":"172.21.15.40:6805/2009758784","heartbeat_back_addr":"172.21.15.40:6809/2009758784","heartbeat_front_addr":"172.21.15.40:6807/2009758784","state":["exists","up"]},{"osd":1,"uuid":"fe5fe11b-0e84-4359-83a7-1ba45105755a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6801","nonce":1763903836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6803","nonce":1763903836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6807","nonce":1763903836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6805","nonce":1763903836}]},"public_addr":"172.21.15.111:6801/1763903836","cluster_addr":"172.21.15.111:6803/1763903836","heartbeat_back_addr":"172.21.15.111:6807/1763903836","heartbeat_front_addr":"172.21.15.111:6805/1763903836","state":["exists","up"]},{"osd":2,"uuid":"813ad735-5cc2-4da0-ae16-6db238395fbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6801","nonce":439695825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6803","nonce":439695825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6807","nonce":439695825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6805","nonce":439695825}]},"public_addr":"172.21.15.174:6801/439695825","cluster_addr":"172.21.15.174:6803/439695825","heartbeat_back_addr":"172.21.15.174:6807/439695825","heartbeat_front_addr":"172.21.15.174:6805/439695825","state":["exists","up"]},{"osd":3,"uuid":"09becb03-b7ce-4ffc-917d-b7900ec37e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6811","nonce":3454312436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6813","nonce":3454312436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6817","nonce":3454312436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6815","nonce":3454312436}]},"public_addr":"172.21.15.40:6811/3454312436","cluster_addr":"172.21.15.40:6813/3454312436","heartbeat_back_addr":"172.21.15.40:6817/3454312436","heartbeat_front_addr":"172.21.15.40:6815/3454312436","state":["exists","up"]},{"osd":4,"uuid":"46eabfd7-d88a-4efd-94eb-04354d3a9d7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6809","nonce":1810866070}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6811","nonce":1810866070}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6815","nonce":1810866070}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6813","nonce":1810866070}]},"public_addr":"172.21.15.111:6809/1810866070","cluster_addr":"172.21.15.111:6811/1810866070","heartbeat_back_addr":"172.21.15.111:6815/1810866070","heartbeat_front_addr":"172.21.15.111:6813/1810866070","state":["exists","up"]},{"osd":5,"uuid":"6f73b582-6c65-41db-aab0-9b360f83d5c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6809","nonce":3752986711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6811","nonce":3752986711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6815","nonce":3752986711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6813","nonce":3752986711}]},"public_addr":"172.21.15.174:6809/3752986711","cluster_addr":"172.21.15.174:6811/3752986711","heartbeat_back_addr":"172.21.15.174:6815/3752986711","heartbeat_front_addr":"172.21.15.174:6813/3752986711","state":["exists","up"]},{"osd":6,"uuid":"39f305f0-12d2-4732-aaa0-cd66700af538","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6819","nonce":1618855427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6821","nonce":1618855427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6825","nonce":1618855427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6823","nonce":1618855427}]},"public_addr":"172.21.15.40:6819/1618855427","cluster_addr":"172.21.15.40:6821/1618855427","heartbeat_back_addr":"172.21.15.40:6825/1618855427","heartbeat_front_addr":"172.21.15.40:6823/1618855427","state":["exists","up"]},{"osd":7,"uuid":"769f2468-5102-490b-89f8-d0444d3169a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6817","nonce":927054162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6819","nonce":927054162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6823","nonce":927054162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6821","nonce":927054162}]},"public_addr":"172.21.15.111:6817/927054162","cluster_addr":"172.21.15.111:6819/927054162","heartbeat_back_addr":"172.21.15.111:6823/927054162","heartbeat_front_addr":"172.21.15.111:6821/927054162","state":["exists","up"]},{"osd":8,"uuid":"1a444e80-007d-473b-b972-4c814a813c22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6817","nonce":3817848746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6819","nonce":3817848746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6823","nonce":3817848746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6821","nonce":3817848746}]},"public_addr":"172.21.15.174:6817/3817848746","cluster_addr":"172.21.15.174:6819/3817848746","heartbeat_back_addr":"172.21.15.174:6823/3817848746","heartbeat_front_addr":"172.21.15.174:6821/3817848746","state":["exists","up"]},{"osd":9,"uuid":"9a6b6a56-6ae5-49fe-9e5b-cb17d686e0dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6827","nonce":1649446531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6829","nonce":1649446531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6833","nonce":1649446531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6831","nonce":1649446531}]},"public_addr":"172.21.15.40:6827/1649446531","cluster_addr":"172.21.15.40:6829/1649446531","heartbeat_back_addr":"172.21.15.40:6833/1649446531","heartbeat_front_addr":"172.21.15.40:6831/1649446531","state":["exists","up"]},{"osd":10,"uuid":"a70038ba-e2ad-4170-85cc-ea0470a2edef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6825","nonce":1227538935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6827","nonce":1227538935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6831","nonce":1227538935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6829","nonce":1227538935}]},"public_addr":"172.21.15.111:6825/1227538935","cluster_addr":"172.21.15.111:6827/1227538935","heartbeat_back_addr":"172.21.15.111:6831/1227538935","heartbeat_front_addr":"172.21.15.111:6829/1227538935","state":["exists","up"]},{"osd":11,"uuid":"781532bb-7a5e-44e3-8eb7-b584b0054666","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6825","nonce":2045865553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6827","nonce":2045865553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6831","nonce":2045865553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6829","nonce":2045865553}]},"public_addr":"172.21.15.174:6825/2045865553","cluster_addr":"172.21.15.174:6827/2045865553","heartbeat_back_addr":"172.21.15.174:6831/2045865553","heartbeat_front_addr":"172.21.15.174:6829/2045865553","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:10:24.761290+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:10:48.565428+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:11:12.307818+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:11:37.951529+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:02.832523+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:27.568879+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:52.997707+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:13:18.937274+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:13:44.708657+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:14:10.833855+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:14:37.078272+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:15:03.878076+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:6801/526554319":"2024-04-22T23:08:50.305770+0000","172.21.15.40:6800/526554319":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/3178763471":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/4045911364":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/276300080":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/3689649814":"2024-04-22T23:08:23.743030+0000","172.21.15.40:0/2241116715":"2024-04-22T23:08:23.743030+0000","172.21.15.40:6801/1479040326":"2024-04-22T23:08:23.743030+0000","172.21.15.40:6800/1479040326":"2024-04-22T23:08:23.743030+0000","172.21.15.40:0/681298099":"2024-04-22T23:08:23.743030+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T23:16:10.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.348+0000 7f26139e5640 1 -- 172.21.15.40:0/3039093242 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f25f80a2b90 msgr2=0x7f25f80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:10.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.348+0000 7f26139e5640 1 --2- 172.21.15.40:0/3039093242 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f25f80a2b90 0x7f25f80a5050 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f260402c700 tx=0x7f260402cca0 comp rx=0 tx=0).stop 2024-04-21T23:16:10.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.348+0000 7f26139e5640 1 -- 172.21.15.40:0/3039093242 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f260c103360 msgr2=0x7f260c19f130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:10.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.348+0000 7f26139e5640 1 --2- 172.21.15.40:0/3039093242 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f260c103360 0x7f260c19f130 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f260800c970 tx=0x7f260800ce30 comp rx=0 tx=0).stop 2024-04-21T23:16:10.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.348+0000 7f26139e5640 1 -- 172.21.15.40:0/3039093242 shutdown_connections 2024-04-21T23:16:10.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.348+0000 7f26139e5640 1 --2- 172.21.15.40:0/3039093242 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f25f80a2b90 0x7f25f80a5050 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:10.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.348+0000 7f26139e5640 1 --2- 172.21.15.40:0/3039093242 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f260c103360 0x7f260c19f130 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:10.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.348+0000 7f26139e5640 1 --2- 172.21.15.40:0/3039093242 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f260c1029a0 0x7f260c1980c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:10.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.348+0000 7f26139e5640 1 --2- 172.21.15.40:0/3039093242 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f260c1017b0 0x7f260c197b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:10.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.348+0000 7f26139e5640 1 -- 172.21.15.40:0/3039093242 >> 172.21.15.40:0/3039093242 conn(0x7f260c0fcd40 msgr2=0x7f260c105d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:10.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.348+0000 7f26139e5640 1 -- 172.21.15.40:0/3039093242 shutdown_connections 2024-04-21T23:16:10.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.348+0000 7f26139e5640 1 -- 172.21.15.40:0/3039093242 wait complete. 2024-04-21T23:16:10.359 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T23:11:14.824775+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T23:15:54.322571+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T23:15:55.164493+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T23:15:58.838266+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T23:16:10.359 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T23:16:10.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.467+0000 7fc59b14f640 1 -- 172.21.15.40:0/1456880423 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc594102230 msgr2=0x7fc594108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:10.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.467+0000 7fc59b14f640 1 --2- 172.21.15.40:0/1456880423 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc594102230 0x7fc594108ac0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fc58c009ca0 tx=0x7fc58c02c0f0 comp rx=0 tx=0).stop 2024-04-21T23:16:10.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.467+0000 7fc59b14f640 1 -- 172.21.15.40:0/1456880423 shutdown_connections 2024-04-21T23:16:10.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.467+0000 7fc59b14f640 1 --2- 172.21.15.40:0/1456880423 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc594102230 0x7fc594108ac0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:10.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.467+0000 7fc59b14f640 1 --2- 172.21.15.40:0/1456880423 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc594101870 0x7fc594101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:10.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.467+0000 7fc59b14f640 1 --2- 172.21.15.40:0/1456880423 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc594100680 0x7fc594100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:10.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.467+0000 7fc59b14f640 1 -- 172.21.15.40:0/1456880423 >> 172.21.15.40:0/1456880423 conn(0x7fc5940fbbf0 msgr2=0x7fc5940fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:10.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.467+0000 7fc59b14f640 1 -- 172.21.15.40:0/1456880423 shutdown_connections 2024-04-21T23:16:10.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.467+0000 7fc59b14f640 1 -- 172.21.15.40:0/1456880423 wait complete. 2024-04-21T23:16:10.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.468+0000 7fc59b14f640 1 Processor -- start 2024-04-21T23:16:10.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.468+0000 7fc59b14f640 1 -- start start 2024-04-21T23:16:10.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.468+0000 7fc59b14f640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc594100680 0x7fc594069460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:10.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.468+0000 7fc59b14f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc594101870 0x7fc59419c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:10.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.468+0000 7fc59b14f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc594102230 0x7fc59419ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:10.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.468+0000 7fc59b14f640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc59410c3a0 con 0x7fc594101870 2024-04-21T23:16:10.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.468+0000 7fc59b14f640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc59410c220 con 0x7fc594100680 2024-04-21T23:16:10.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.468+0000 7fc59b14f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc59410c520 con 0x7fc594102230 2024-04-21T23:16:10.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.469+0000 7fc593fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc594101870 0x7fc59419c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:10.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.469+0000 7fc593fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc594101870 0x7fc59419c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33376/0 (socket says 172.21.15.40:33376) 2024-04-21T23:16:10.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.469+0000 7fc593fff640 1 -- 172.21.15.40:0/281994763 learned_addr learned my addr 172.21.15.40:0/281994763 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:16:10.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.469+0000 7fc598eeb640 1 --2- 172.21.15.40:0/281994763 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc594100680 0x7fc594069460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:10.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.469+0000 7fc5996ec640 1 --2- 172.21.15.40:0/281994763 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc594102230 0x7fc59419ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:10.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.469+0000 7fc593fff640 1 -- 172.21.15.40:0/281994763 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc594102230 msgr2=0x7fc59419ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:10.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.469+0000 7fc593fff640 1 --2- 172.21.15.40:0/281994763 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc594102230 0x7fc59419ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:10.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.469+0000 7fc593fff640 1 -- 172.21.15.40:0/281994763 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc594100680 msgr2=0x7fc594069460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:10.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.469+0000 7fc593fff640 1 --2- 172.21.15.40:0/281994763 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc594100680 0x7fc594069460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:10.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.469+0000 7fc593fff640 1 -- 172.21.15.40:0/281994763 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc59419f720 con 0x7fc594101870 2024-04-21T23:16:10.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.469+0000 7fc5996ec640 1 --2- 172.21.15.40:0/281994763 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc594102230 0x7fc59419ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:16:10.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.469+0000 7fc593fff640 1 --2- 172.21.15.40:0/281994763 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc594101870 0x7fc59419c820 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fc58000d770 tx=0x7fc58000db70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:10.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.469+0000 7fc591ffb640 1 -- 172.21.15.40:0/281994763 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc580017070 con 0x7fc594101870 2024-04-21T23:16:10.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.469+0000 7fc591ffb640 1 -- 172.21.15.40:0/281994763 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc580009820 con 0x7fc594101870 2024-04-21T23:16:10.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.469+0000 7fc59b14f640 1 -- 172.21.15.40:0/281994763 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc59419fa10 con 0x7fc594101870 2024-04-21T23:16:10.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.469+0000 7fc591ffb640 1 -- 172.21.15.40:0/281994763 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc580004cd0 con 0x7fc594101870 2024-04-21T23:16:10.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.470+0000 7fc59b14f640 1 -- 172.21.15.40:0/281994763 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc59419ff10 con 0x7fc594101870 2024-04-21T23:16:10.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.470+0000 7fc59b14f640 1 -- 172.21.15.40:0/281994763 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5941079a0 con 0x7fc594101870 2024-04-21T23:16:10.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.471+0000 7fc591ffb640 1 -- 172.21.15.40:0/281994763 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fc5800099d0 con 0x7fc594101870 2024-04-21T23:16:10.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.472+0000 7fc591ffb640 1 --2- 172.21.15.40:0/281994763 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fc5840a2b90 0x7fc5840a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:10.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.472+0000 7fc598eeb640 1 --2- 172.21.15.40:0/281994763 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fc5840a2b90 0x7fc5840a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:10.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.472+0000 7fc591ffb640 1 -- 172.21.15.40:0/281994763 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fc5800ced20 con 0x7fc594101870 2024-04-21T23:16:10.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.473+0000 7fc598eeb640 1 --2- 172.21.15.40:0/281994763 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fc5840a2b90 0x7fc5840a5050 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc5941016d0 tx=0x7fc588009110 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:10.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.477+0000 7fc591ffb640 1 -- 172.21.15.40:0/281994763 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fc58009b9e0 con 0x7fc594101870 2024-04-21T23:16:10.645 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:10 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/2730881234' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T23:16:10.645 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:10 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/3018181361' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-21T23:16:10.645 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:10 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/3039093242' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:16:10.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.663+0000 7fc59b14f640 1 -- 172.21.15.40:0/281994763 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc594062340 con 0x7fc594101870 2024-04-21T23:16:10.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.664+0000 7fc591ffb640 1 -- 172.21.15.40:0/281994763 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) ==== 93+0+10 (secure 0 0 0) 0x7fc58009b2e0 con 0x7fc594101870 2024-04-21T23:16:10.663 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-04-21T23:16:10.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.666+0000 7fc59b14f640 1 -- 172.21.15.40:0/281994763 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fc5840a2b90 msgr2=0x7fc5840a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:10.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.666+0000 7fc59b14f640 1 --2- 172.21.15.40:0/281994763 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fc5840a2b90 0x7fc5840a5050 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc5941016d0 tx=0x7fc588009110 comp rx=0 tx=0).stop 2024-04-21T23:16:10.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.666+0000 7fc59b14f640 1 -- 172.21.15.40:0/281994763 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc594101870 msgr2=0x7fc59419c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:10.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.666+0000 7fc59b14f640 1 --2- 172.21.15.40:0/281994763 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc594101870 0x7fc59419c820 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fc58000d770 tx=0x7fc58000db70 comp rx=0 tx=0).stop 2024-04-21T23:16:10.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.666+0000 7fc59b14f640 1 -- 172.21.15.40:0/281994763 shutdown_connections 2024-04-21T23:16:10.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.666+0000 7fc59b14f640 1 --2- 172.21.15.40:0/281994763 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fc5840a2b90 0x7fc5840a5050 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:10.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.666+0000 7fc59b14f640 1 --2- 172.21.15.40:0/281994763 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc594102230 0x7fc59419ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:10.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.666+0000 7fc59b14f640 1 --2- 172.21.15.40:0/281994763 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc594101870 0x7fc59419c820 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:10.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.666+0000 7fc59b14f640 1 --2- 172.21.15.40:0/281994763 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc594100680 0x7fc594069460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:10.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.666+0000 7fc59b14f640 1 -- 172.21.15.40:0/281994763 >> 172.21.15.40:0/281994763 conn(0x7fc5940fbbf0 msgr2=0x7fc594104b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:10.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.666+0000 7fc59b14f640 1 -- 172.21.15.40:0/281994763 shutdown_connections 2024-04-21T23:16:10.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.666+0000 7fc59b14f640 1 -- 172.21.15.40:0/281994763 wait complete. 2024-04-21T23:16:10.676 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T23:16:10.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:10 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/2730881234' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T23:16:10.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:10 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/3018181361' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-21T23:16:10.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:10 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/3039093242' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:16:10.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:10 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/2730881234' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T23:16:10.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:10 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/3018181361' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-21T23:16:10.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:10 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/3039093242' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:16:10.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.783+0000 7fb8b8fb8640 1 -- 172.21.15.40:0/1754459309 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb8b4104b30 msgr2=0x7fb8b4106f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:10.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.783+0000 7fb8b8fb8640 1 --2- 172.21.15.40:0/1754459309 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb8b4104b30 0x7fb8b4106f20 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb8a8009900 tx=0x7fb8a802bc30 comp rx=0 tx=0).stop 2024-04-21T23:16:10.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.783+0000 7fb8b8fb8640 1 -- 172.21.15.40:0/1754459309 shutdown_connections 2024-04-21T23:16:10.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.783+0000 7fb8b8fb8640 1 --2- 172.21.15.40:0/1754459309 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8b4107460 0x7fb8b41098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:10.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.783+0000 7fb8b8fb8640 1 --2- 172.21.15.40:0/1754459309 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb8b4104b30 0x7fb8b4106f20 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:10.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.783+0000 7fb8b8fb8640 1 --2- 172.21.15.40:0/1754459309 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb8b4102200 0x7fb8b41045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:10.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.783+0000 7fb8b8fb8640 1 -- 172.21.15.40:0/1754459309 >> 172.21.15.40:0/1754459309 conn(0x7fb8b40fbbf0 msgr2=0x7fb8b40fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:10.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.783+0000 7fb8b8fb8640 1 -- 172.21.15.40:0/1754459309 shutdown_connections 2024-04-21T23:16:10.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.783+0000 7fb8b8fb8640 1 -- 172.21.15.40:0/1754459309 wait complete. 2024-04-21T23:16:10.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.784+0000 7fb8b8fb8640 1 Processor -- start 2024-04-21T23:16:10.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.784+0000 7fb8b8fb8640 1 -- start start 2024-04-21T23:16:10.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.784+0000 7fb8b8fb8640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb8b4102200 0x7fb8b4101b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:10.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.784+0000 7fb8b8fb8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb8b4104b30 0x7fb8b41001b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:10.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.784+0000 7fb8b8fb8640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8b4107460 0x7fb8b41006f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:10.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.784+0000 7fb8b8fb8640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb8b410d3f0 con 0x7fb8b4102200 2024-04-21T23:16:10.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.784+0000 7fb8b8fb8640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb8b410d270 con 0x7fb8b4107460 2024-04-21T23:16:10.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.784+0000 7fb8b8fb8640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb8b410d570 con 0x7fb8b4104b30 2024-04-21T23:16:10.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.784+0000 7fb8b259c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb8b4102200 0x7fb8b4101b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:10.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.784+0000 7fb8b259c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb8b4102200 0x7fb8b4101b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33390/0 (socket says 172.21.15.40:33390) 2024-04-21T23:16:10.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.784+0000 7fb8b259c640 1 -- 172.21.15.40:0/4078000877 learned_addr learned my addr 172.21.15.40:0/4078000877 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:16:10.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.784+0000 7fb8b1d9b640 1 --2- 172.21.15.40:0/4078000877 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb8b4104b30 0x7fb8b41001b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:10.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.785+0000 7fb8b2d9d640 1 --2- 172.21.15.40:0/4078000877 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8b4107460 0x7fb8b41006f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:10.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.785+0000 7fb8b259c640 1 -- 172.21.15.40:0/4078000877 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb8b4104b30 msgr2=0x7fb8b41001b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:10.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.785+0000 7fb8b259c640 1 --2- 172.21.15.40:0/4078000877 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb8b4104b30 0x7fb8b41001b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:10.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.785+0000 7fb8b259c640 1 -- 172.21.15.40:0/4078000877 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8b4107460 msgr2=0x7fb8b41006f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:10.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.785+0000 7fb8b259c640 1 --2- 172.21.15.40:0/4078000877 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8b4107460 0x7fb8b41006f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:10.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.785+0000 7fb8b259c640 1 -- 172.21.15.40:0/4078000877 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8b4100f80 con 0x7fb8b4102200 2024-04-21T23:16:10.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.785+0000 7fb8b259c640 1 --2- 172.21.15.40:0/4078000877 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb8b4102200 0x7fb8b4101b70 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac00cce0 tx=0x7fb8ac00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:10.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.785+0000 7fb8a37fe640 1 -- 172.21.15.40:0/4078000877 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8ac00f650 con 0x7fb8b4102200 2024-04-21T23:16:10.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.785+0000 7fb8a37fe640 1 -- 172.21.15.40:0/4078000877 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb8ac00fcd0 con 0x7fb8b4102200 2024-04-21T23:16:10.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.785+0000 7fb8b8fb8640 1 -- 172.21.15.40:0/4078000877 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8b41a1b50 con 0x7fb8b4102200 2024-04-21T23:16:10.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.785+0000 7fb8a37fe640 1 -- 172.21.15.40:0/4078000877 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8ac018c70 con 0x7fb8b4102200 2024-04-21T23:16:10.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.785+0000 7fb8b8fb8640 1 -- 172.21.15.40:0/4078000877 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8b41a2070 con 0x7fb8b4102200 2024-04-21T23:16:10.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.786+0000 7fb8b8fb8640 1 -- 172.21.15.40:0/4078000877 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8b4068560 con 0x7fb8b4102200 2024-04-21T23:16:10.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.787+0000 7fb8a37fe640 1 -- 172.21.15.40:0/4078000877 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fb8ac00f7f0 con 0x7fb8b4102200 2024-04-21T23:16:10.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.788+0000 7fb8a37fe640 1 --2- 172.21.15.40:0/4078000877 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fb89c0a2b90 0x7fb89c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:10.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.788+0000 7fb8b1d9b640 1 --2- 172.21.15.40:0/4078000877 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fb89c0a2b90 0x7fb89c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:10.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.788+0000 7fb8a37fe640 1 -- 172.21.15.40:0/4078000877 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fb8ac0cfa30 con 0x7fb8b4102200 2024-04-21T23:16:10.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.789+0000 7fb8b1d9b640 1 --2- 172.21.15.40:0/4078000877 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fb89c0a2b90 0x7fb89c0a5050 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fb8a8000c00 tx=0x7fb8a8004f10 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:10.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.791+0000 7fb8a37fe640 1 -- 172.21.15.40:0/4078000877 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fb8ac014030 con 0x7fb8b4102200 2024-04-21T23:16:10.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.981+0000 7fb8b8fb8640 1 -- 172.21.15.40:0/4078000877 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fb8b402d210 con 0x7fb8b4102200 2024-04-21T23:16:10.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.982+0000 7fb8a37fe640 1 -- 172.21.15.40:0/4078000877 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) ==== 104+0+11 (secure 0 0 0) 0x7fb8ac09c7e0 con 0x7fb8b4102200 2024-04-21T23:16:10.982 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-21T23:16:10.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.984+0000 7fb8b8fb8640 1 -- 172.21.15.40:0/4078000877 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fb89c0a2b90 msgr2=0x7fb89c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:10.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.984+0000 7fb8b8fb8640 1 --2- 172.21.15.40:0/4078000877 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fb89c0a2b90 0x7fb89c0a5050 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fb8a8000c00 tx=0x7fb8a8004f10 comp rx=0 tx=0).stop 2024-04-21T23:16:10.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.984+0000 7fb8b8fb8640 1 -- 172.21.15.40:0/4078000877 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb8b4102200 msgr2=0x7fb8b4101b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:10.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.984+0000 7fb8b8fb8640 1 --2- 172.21.15.40:0/4078000877 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb8b4102200 0x7fb8b4101b70 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac00cce0 tx=0x7fb8ac00a620 comp rx=0 tx=0).stop 2024-04-21T23:16:10.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.984+0000 7fb8b8fb8640 1 -- 172.21.15.40:0/4078000877 shutdown_connections 2024-04-21T23:16:10.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.984+0000 7fb8b8fb8640 1 --2- 172.21.15.40:0/4078000877 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fb89c0a2b90 0x7fb89c0a5050 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:10.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.984+0000 7fb8b8fb8640 1 --2- 172.21.15.40:0/4078000877 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8b4107460 0x7fb8b41006f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:10.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.984+0000 7fb8b8fb8640 1 --2- 172.21.15.40:0/4078000877 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb8b4104b30 0x7fb8b41001b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:10.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.984+0000 7fb8b8fb8640 1 --2- 172.21.15.40:0/4078000877 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb8b4102200 0x7fb8b4101b70 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:10.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.984+0000 7fb8b8fb8640 1 -- 172.21.15.40:0/4078000877 >> 172.21.15.40:0/4078000877 conn(0x7fb8b40fbbf0 msgr2=0x7fb8b4102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:10.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.984+0000 7fb8b8fb8640 1 -- 172.21.15.40:0/4078000877 shutdown_connections 2024-04-21T23:16:10.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:10.984+0000 7fb8b8fb8640 1 -- 172.21.15.40:0/4078000877 wait complete. 2024-04-21T23:16:10.995 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T23:16:11.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.103+0000 7ff0d065f640 1 -- 172.21.15.40:0/1056355904 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0c8102980 msgr2=0x7ff0c8102e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:11.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.103+0000 7ff0d065f640 1 --2- 172.21.15.40:0/1056355904 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0c8102980 0x7ff0c8102e00 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ff0bc009900 tx=0x7ff0bc02bc30 comp rx=0 tx=0).stop 2024-04-21T23:16:11.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.103+0000 7ff0d065f640 1 -- 172.21.15.40:0/1056355904 shutdown_connections 2024-04-21T23:16:11.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.103+0000 7ff0d065f640 1 --2- 172.21.15.40:0/1056355904 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0c8103340 0x7ff0c8109bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:11.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.103+0000 7ff0d065f640 1 --2- 172.21.15.40:0/1056355904 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0c8102980 0x7ff0c8102e00 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:11.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.103+0000 7ff0d065f640 1 --2- 172.21.15.40:0/1056355904 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff0c8101790 0x7ff0c8101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:11.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.103+0000 7ff0d065f640 1 -- 172.21.15.40:0/1056355904 >> 172.21.15.40:0/1056355904 conn(0x7ff0c80fcd20 msgr2=0x7ff0c80ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:11.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.103+0000 7ff0d065f640 1 -- 172.21.15.40:0/1056355904 shutdown_connections 2024-04-21T23:16:11.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.103+0000 7ff0d065f640 1 -- 172.21.15.40:0/1056355904 wait complete. 2024-04-21T23:16:11.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.104+0000 7ff0d065f640 1 Processor -- start 2024-04-21T23:16:11.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.104+0000 7ff0d065f640 1 -- start start 2024-04-21T23:16:11.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.104+0000 7ff0d065f640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0c8101790 0x7ff0c80754f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:11.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.104+0000 7ff0d065f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff0c8102980 0x7ff0c8073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:11.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.104+0000 7ff0d065f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0c8103340 0x7ff0c8074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:11.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.104+0000 7ff0d065f640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff0c810d410 con 0x7ff0c8103340 2024-04-21T23:16:11.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.104+0000 7ff0d065f640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff0c810d290 con 0x7ff0c8101790 2024-04-21T23:16:11.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.104+0000 7ff0d065f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff0c810d590 con 0x7ff0c8102980 2024-04-21T23:16:11.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.104+0000 7ff0cebfc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0c8103340 0x7ff0c8074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:11.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.104+0000 7ff0cebfc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0c8103340 0x7ff0c8074070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33404/0 (socket says 172.21.15.40:33404) 2024-04-21T23:16:11.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.104+0000 7ff0cebfc640 1 -- 172.21.15.40:0/846318354 learned_addr learned my addr 172.21.15.40:0/846318354 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:16:11.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.104+0000 7ff0cdbfa640 1 --2- 172.21.15.40:0/846318354 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff0c8102980 0x7ff0c8073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:11.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.104+0000 7ff0ce3fb640 1 --2- 172.21.15.40:0/846318354 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0c8101790 0x7ff0c80754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:11.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.104+0000 7ff0cebfc640 1 -- 172.21.15.40:0/846318354 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff0c8102980 msgr2=0x7ff0c8073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:11.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.104+0000 7ff0cebfc640 1 --2- 172.21.15.40:0/846318354 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff0c8102980 0x7ff0c8073b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:11.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.104+0000 7ff0cebfc640 1 -- 172.21.15.40:0/846318354 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0c8101790 msgr2=0x7ff0c80754f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:11.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.104+0000 7ff0cebfc640 1 --2- 172.21.15.40:0/846318354 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0c8101790 0x7ff0c80754f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:11.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.104+0000 7ff0cebfc640 1 -- 172.21.15.40:0/846318354 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0c8074900 con 0x7ff0c8103340 2024-04-21T23:16:11.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.104+0000 7ff0cebfc640 1 --2- 172.21.15.40:0/846318354 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0c8103340 0x7ff0c8074070 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ff0c000eb10 tx=0x7ff0c000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:11.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.105+0000 7ff0bb7fe640 1 -- 172.21.15.40:0/846318354 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0c0004b20 con 0x7ff0c8103340 2024-04-21T23:16:11.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.105+0000 7ff0bb7fe640 1 -- 172.21.15.40:0/846318354 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff0c0004cc0 con 0x7ff0c8103340 2024-04-21T23:16:11.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.105+0000 7ff0d065f640 1 -- 172.21.15.40:0/846318354 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0c81a1c10 con 0x7ff0c8103340 2024-04-21T23:16:11.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.105+0000 7ff0bb7fe640 1 -- 172.21.15.40:0/846318354 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0c0010760 con 0x7ff0c8103340 2024-04-21T23:16:11.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.105+0000 7ff0d065f640 1 -- 172.21.15.40:0/846318354 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0c81a2130 con 0x7ff0c8103340 2024-04-21T23:16:11.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.106+0000 7ff0d065f640 1 -- 172.21.15.40:0/846318354 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0c8068560 con 0x7ff0c8103340 2024-04-21T23:16:11.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.107+0000 7ff0bb7fe640 1 -- 172.21.15.40:0/846318354 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7ff0c0010900 con 0x7ff0c8103340 2024-04-21T23:16:11.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.108+0000 7ff0bb7fe640 1 --2- 172.21.15.40:0/846318354 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7ff0b40a2c10 0x7ff0b40a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:11.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.108+0000 7ff0ce3fb640 1 --2- 172.21.15.40:0/846318354 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7ff0b40a2c10 0x7ff0b40a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:11.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.108+0000 7ff0bb7fe640 1 -- 172.21.15.40:0/846318354 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7ff0c00cfb30 con 0x7ff0c8103340 2024-04-21T23:16:11.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.108+0000 7ff0ce3fb640 1 --2- 172.21.15.40:0/846318354 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7ff0b40a2c10 0x7ff0b40a50d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ff0c4005fd0 tx=0x7ff0c4005850 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:11.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.111+0000 7ff0bb7fe640 1 -- 172.21.15.40:0/846318354 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7ff0c00536f0 con 0x7ff0c8103340 2024-04-21T23:16:11.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.296+0000 7ff0d065f640 1 -- 172.21.15.40:0/846318354 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7ff0c802d1f0 con 0x7ff0c8103340 2024-04-21T23:16:11.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.297+0000 7ff0bb7fe640 1 -- 172.21.15.40:0/846318354 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) ==== 100+0+11 (secure 0 0 0) 0x7ff0c009c850 con 0x7ff0c8103340 2024-04-21T23:16:11.297 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-04-21T23:16:11.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.299+0000 7ff0d065f640 1 -- 172.21.15.40:0/846318354 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7ff0b40a2c10 msgr2=0x7ff0b40a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:11.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.299+0000 7ff0d065f640 1 --2- 172.21.15.40:0/846318354 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7ff0b40a2c10 0x7ff0b40a50d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ff0c4005fd0 tx=0x7ff0c4005850 comp rx=0 tx=0).stop 2024-04-21T23:16:11.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.299+0000 7ff0d065f640 1 -- 172.21.15.40:0/846318354 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0c8103340 msgr2=0x7ff0c8074070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:11.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.299+0000 7ff0d065f640 1 --2- 172.21.15.40:0/846318354 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0c8103340 0x7ff0c8074070 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ff0c000eb10 tx=0x7ff0c000c2d0 comp rx=0 tx=0).stop 2024-04-21T23:16:11.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.299+0000 7ff0d065f640 1 -- 172.21.15.40:0/846318354 shutdown_connections 2024-04-21T23:16:11.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.299+0000 7ff0d065f640 1 --2- 172.21.15.40:0/846318354 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7ff0b40a2c10 0x7ff0b40a50d0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:11.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.299+0000 7ff0d065f640 1 --2- 172.21.15.40:0/846318354 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0c8103340 0x7ff0c8074070 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:11.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.299+0000 7ff0d065f640 1 --2- 172.21.15.40:0/846318354 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff0c8102980 0x7ff0c8073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:11.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.299+0000 7ff0d065f640 1 --2- 172.21.15.40:0/846318354 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0c8101790 0x7ff0c80754f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:11.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.299+0000 7ff0d065f640 1 -- 172.21.15.40:0/846318354 >> 172.21.15.40:0/846318354 conn(0x7ff0c80fcd20 msgr2=0x7ff0c8105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:11.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.299+0000 7ff0d065f640 1 -- 172.21.15.40:0/846318354 shutdown_connections 2024-04-21T23:16:11.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.299+0000 7ff0d065f640 1 -- 172.21.15.40:0/846318354 wait complete. 2024-04-21T23:16:11.310 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T23:16:11.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.416+0000 7f37373c9640 1 -- 172.21.15.40:0/836446355 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3730102230 msgr2=0x7f3730108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:11.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.416+0000 7f37373c9640 1 --2- 172.21.15.40:0/836446355 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3730102230 0x7f3730108ac0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f3728009ca0 tx=0x7f372802c0f0 comp rx=0 tx=0).stop 2024-04-21T23:16:11.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.416+0000 7f37373c9640 1 -- 172.21.15.40:0/836446355 shutdown_connections 2024-04-21T23:16:11.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.416+0000 7f37373c9640 1 --2- 172.21.15.40:0/836446355 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3730101870 0x7f3730101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:11.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.416+0000 7f37373c9640 1 --2- 172.21.15.40:0/836446355 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3730102230 0x7f3730108ac0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:11.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.416+0000 7f37373c9640 1 --2- 172.21.15.40:0/836446355 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3730100680 0x7f3730100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:11.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.416+0000 7f37373c9640 1 -- 172.21.15.40:0/836446355 >> 172.21.15.40:0/836446355 conn(0x7f37300fbbf0 msgr2=0x7f37300fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:11.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.417+0000 7f37373c9640 1 -- 172.21.15.40:0/836446355 shutdown_connections 2024-04-21T23:16:11.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.417+0000 7f37373c9640 1 -- 172.21.15.40:0/836446355 wait complete. 2024-04-21T23:16:11.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.417+0000 7f37373c9640 1 Processor -- start 2024-04-21T23:16:11.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.417+0000 7f37373c9640 1 -- start start 2024-04-21T23:16:11.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.417+0000 7f37373c9640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3730100680 0x7f3730197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:11.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.417+0000 7f37373c9640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3730101870 0x7f3730197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:11.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.417+0000 7f37373c9640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3730102230 0x7f373019f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:11.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.417+0000 7f37373c9640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f373010d420 con 0x7f3730102230 2024-04-21T23:16:11.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.417+0000 7f37373c9640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f373010d2a0 con 0x7f3730100680 2024-04-21T23:16:11.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.417+0000 7f37373c9640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f373010d5a0 con 0x7f3730101870 2024-04-21T23:16:11.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.417+0000 7f3735966640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3730102230 0x7f373019f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:11.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.417+0000 7f3735966640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3730102230 0x7f373019f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33412/0 (socket says 172.21.15.40:33412) 2024-04-21T23:16:11.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.417+0000 7f3735966640 1 -- 172.21.15.40:0/2222654616 learned_addr learned my addr 172.21.15.40:0/2222654616 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:16:11.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.418+0000 7f3735165640 1 --2- 172.21.15.40:0/2222654616 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3730100680 0x7f3730197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:11.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.418+0000 7f3734964640 1 --2- 172.21.15.40:0/2222654616 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3730101870 0x7f3730197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:11.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.418+0000 7f3735966640 1 -- 172.21.15.40:0/2222654616 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3730101870 msgr2=0x7f3730197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:11.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.418+0000 7f3735966640 1 --2- 172.21.15.40:0/2222654616 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3730101870 0x7f3730197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:11.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.418+0000 7f3735966640 1 -- 172.21.15.40:0/2222654616 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3730100680 msgr2=0x7f3730197a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:11.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.418+0000 7f3735966640 1 --2- 172.21.15.40:0/2222654616 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3730100680 0x7f3730197a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:11.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.418+0000 7f3735966640 1 -- 172.21.15.40:0/2222654616 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f373019f690 con 0x7f3730102230 2024-04-21T23:16:11.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.418+0000 7f3735165640 1 --2- 172.21.15.40:0/2222654616 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3730100680 0x7f3730197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:16:11.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.418+0000 7f3735966640 1 --2- 172.21.15.40:0/2222654616 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3730102230 0x7f373019f020 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f3728037040 tx=0x7f3728032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:11.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.418+0000 7f37267fc640 1 -- 172.21.15.40:0/2222654616 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3728041020 con 0x7f3730102230 2024-04-21T23:16:11.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.418+0000 7f37373c9640 1 -- 172.21.15.40:0/2222654616 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f373019f920 con 0x7f3730102230 2024-04-21T23:16:11.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.418+0000 7f37267fc640 1 -- 172.21.15.40:0/2222654616 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3728034d90 con 0x7f3730102230 2024-04-21T23:16:11.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.418+0000 7f37267fc640 1 -- 172.21.15.40:0/2222654616 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3728039040 con 0x7f3730102230 2024-04-21T23:16:11.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.418+0000 7f37373c9640 1 -- 172.21.15.40:0/2222654616 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f373019fe10 con 0x7f3730102230 2024-04-21T23:16:11.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.419+0000 7f37373c9640 1 -- 172.21.15.40:0/2222654616 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37301079f0 con 0x7f3730102230 2024-04-21T23:16:11.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.421+0000 7f37267fc640 1 -- 172.21.15.40:0/2222654616 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f3728044020 con 0x7f3730102230 2024-04-21T23:16:11.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.421+0000 7f37267fc640 1 --2- 172.21.15.40:0/2222654616 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f371c0a2b90 0x7f371c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:11.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.421+0000 7f37267fc640 1 -- 172.21.15.40:0/2222654616 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f37280f12e0 con 0x7f3730102230 2024-04-21T23:16:11.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.421+0000 7f3735165640 1 --2- 172.21.15.40:0/2222654616 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f371c0a2b90 0x7f371c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:11.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.422+0000 7f3735165640 1 --2- 172.21.15.40:0/2222654616 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f371c0a2b90 0x7f371c0a5050 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f37301016d0 tx=0x7f372c005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:11.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.426+0000 7f37267fc640 1 -- 172.21.15.40:0/2222654616 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f372804b020 con 0x7f3730102230 2024-04-21T23:16:11.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.611+0000 7f37373c9640 1 -- 172.21.15.40:0/2222654616 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f3730062340 con 0x7f3730102230 2024-04-21T23:16:11.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.611+0000 7f37267fc640 1 -- 172.21.15.40:0/2222654616 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v90) ==== 103+0+11 (secure 0 0 0) 0x7f37280be000 con 0x7f3730102230 2024-04-21T23:16:11.611 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-21T23:16:11.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.613+0000 7f37373c9640 1 -- 172.21.15.40:0/2222654616 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f371c0a2b90 msgr2=0x7f371c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:11.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.613+0000 7f37373c9640 1 --2- 172.21.15.40:0/2222654616 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f371c0a2b90 0x7f371c0a5050 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f37301016d0 tx=0x7f372c005f90 comp rx=0 tx=0).stop 2024-04-21T23:16:11.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.613+0000 7f37373c9640 1 -- 172.21.15.40:0/2222654616 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3730102230 msgr2=0x7f373019f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:11.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.613+0000 7f37373c9640 1 --2- 172.21.15.40:0/2222654616 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3730102230 0x7f373019f020 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f3728037040 tx=0x7f3728032f20 comp rx=0 tx=0).stop 2024-04-21T23:16:11.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.613+0000 7f37373c9640 1 -- 172.21.15.40:0/2222654616 shutdown_connections 2024-04-21T23:16:11.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.613+0000 7f37373c9640 1 --2- 172.21.15.40:0/2222654616 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f371c0a2b90 0x7f371c0a5050 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:11.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.613+0000 7f37373c9640 1 --2- 172.21.15.40:0/2222654616 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3730102230 0x7f373019f020 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:11.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.613+0000 7f37373c9640 1 --2- 172.21.15.40:0/2222654616 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3730101870 0x7f3730197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:11.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.613+0000 7f37373c9640 1 --2- 172.21.15.40:0/2222654616 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3730100680 0x7f3730197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:11.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.613+0000 7f37373c9640 1 -- 172.21.15.40:0/2222654616 >> 172.21.15.40:0/2222654616 conn(0x7f37300fbbf0 msgr2=0x7f3730104bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:11.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.613+0000 7f37373c9640 1 -- 172.21.15.40:0/2222654616 shutdown_connections 2024-04-21T23:16:11.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.613+0000 7f37373c9640 1 -- 172.21.15.40:0/2222654616 wait complete. 2024-04-21T23:16:11.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:11 smithi040 ceph-mon[27984]: pgmap v300: 129 pgs: 129 active+clean; 581 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 307 B/s wr, 0 op/s 2024-04-21T23:16:11.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:11 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/281994763' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:16:11.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:11 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/4078000877' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T23:16:11.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:11 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/846318354' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T23:16:11.625 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-21T23:16:11.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:11 smithi174 ceph-mon[31409]: pgmap v300: 129 pgs: 129 active+clean; 581 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 307 B/s wr, 0 op/s 2024-04-21T23:16:11.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:11 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/281994763' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:16:11.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:11 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/4078000877' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T23:16:11.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:11 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/846318354' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T23:16:11.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:11 smithi111 ceph-mon[31085]: pgmap v300: 129 pgs: 129 active+clean; 581 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 307 B/s wr, 0 op/s 2024-04-21T23:16:11.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:11 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/281994763' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:16:11.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:11 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/4078000877' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T23:16:11.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:11 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/846318354' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T23:16:11.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.729+0000 7ff341488640 1 -- 172.21.15.40:0/3729175215 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff33c102250 msgr2=0x7ff33c108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:11.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.729+0000 7ff341488640 1 --2- 172.21.15.40:0/3729175215 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff33c102250 0x7ff33c108ae0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7ff334009ca0 tx=0x7ff33402c0f0 comp rx=0 tx=0).stop 2024-04-21T23:16:11.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.729+0000 7ff341488640 1 -- 172.21.15.40:0/3729175215 shutdown_connections 2024-04-21T23:16:11.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.729+0000 7ff341488640 1 --2- 172.21.15.40:0/3729175215 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff33c102250 0x7ff33c108ae0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:11.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.729+0000 7ff341488640 1 --2- 172.21.15.40:0/3729175215 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff33c101890 0x7ff33c101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:11.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.729+0000 7ff341488640 1 --2- 172.21.15.40:0/3729175215 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff33c1006a0 0x7ff33c100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:11.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.729+0000 7ff341488640 1 -- 172.21.15.40:0/3729175215 >> 172.21.15.40:0/3729175215 conn(0x7ff33c0fbc30 msgr2=0x7ff33c0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:11.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.729+0000 7ff341488640 1 -- 172.21.15.40:0/3729175215 shutdown_connections 2024-04-21T23:16:11.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.729+0000 7ff341488640 1 -- 172.21.15.40:0/3729175215 wait complete. 2024-04-21T23:16:11.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.729+0000 7ff341488640 1 Processor -- start 2024-04-21T23:16:11.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.729+0000 7ff341488640 1 -- start start 2024-04-21T23:16:11.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.730+0000 7ff341488640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff33c1006a0 0x7ff33c104630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:11.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.730+0000 7ff341488640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff33c101890 0x7ff33c102c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:11.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.730+0000 7ff341488640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff33c102250 0x7ff33c1031b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:11.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.730+0000 7ff341488640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff33c10c320 con 0x7ff33c101890 2024-04-21T23:16:11.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.730+0000 7ff341488640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff33c10c1a0 con 0x7ff33c102250 2024-04-21T23:16:11.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.730+0000 7ff341488640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff33c10c4a0 con 0x7ff33c1006a0 2024-04-21T23:16:11.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.730+0000 7ff33a7fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff33c101890 0x7ff33c102c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:11.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.730+0000 7ff33a7fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff33c101890 0x7ff33c102c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33434/0 (socket says 172.21.15.40:33434) 2024-04-21T23:16:11.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.730+0000 7ff33a7fc640 1 -- 172.21.15.40:0/3403742987 learned_addr learned my addr 172.21.15.40:0/3403742987 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:16:11.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.730+0000 7ff33affd640 1 --2- 172.21.15.40:0/3403742987 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff33c1006a0 0x7ff33c104630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:11.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.730+0000 7ff33b7fe640 1 --2- 172.21.15.40:0/3403742987 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff33c102250 0x7ff33c1031b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:11.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.730+0000 7ff33a7fc640 1 -- 172.21.15.40:0/3403742987 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff33c1006a0 msgr2=0x7ff33c104630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:11.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.730+0000 7ff33a7fc640 1 --2- 172.21.15.40:0/3403742987 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff33c1006a0 0x7ff33c104630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:11.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.730+0000 7ff33a7fc640 1 -- 172.21.15.40:0/3403742987 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff33c102250 msgr2=0x7ff33c1031b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:11.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.730+0000 7ff33a7fc640 1 --2- 172.21.15.40:0/3403742987 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff33c102250 0x7ff33c1031b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:11.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.730+0000 7ff33a7fc640 1 -- 172.21.15.40:0/3403742987 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff33c103a40 con 0x7ff33c101890 2024-04-21T23:16:11.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.730+0000 7ff33a7fc640 1 --2- 172.21.15.40:0/3403742987 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff33c101890 0x7ff33c102c70 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7ff33000c970 tx=0x7ff33000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:11.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.730+0000 7ff323fff640 1 -- 172.21.15.40:0/3403742987 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff330007bb0 con 0x7ff33c101890 2024-04-21T23:16:11.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.730+0000 7ff341488640 1 -- 172.21.15.40:0/3403742987 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff33c1a1b70 con 0x7ff33c101890 2024-04-21T23:16:11.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.730+0000 7ff323fff640 1 -- 172.21.15.40:0/3403742987 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff330007d50 con 0x7ff33c101890 2024-04-21T23:16:11.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.730+0000 7ff323fff640 1 -- 172.21.15.40:0/3403742987 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff33000f450 con 0x7ff33c101890 2024-04-21T23:16:11.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.730+0000 7ff341488640 1 -- 172.21.15.40:0/3403742987 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff33c1a2070 con 0x7ff33c101890 2024-04-21T23:16:11.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.731+0000 7ff341488640 1 -- 172.21.15.40:0/3403742987 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff33c1a22a0 con 0x7ff33c101890 2024-04-21T23:16:11.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.733+0000 7ff323fff640 1 -- 172.21.15.40:0/3403742987 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7ff330004fe0 con 0x7ff33c101890 2024-04-21T23:16:11.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.733+0000 7ff323fff640 1 --2- 172.21.15.40:0/3403742987 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7ff3240a2b90 0x7ff3240a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:11.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.733+0000 7ff323fff640 1 -- 172.21.15.40:0/3403742987 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7ff3300cf3c0 con 0x7ff33c101890 2024-04-21T23:16:11.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.733+0000 7ff33affd640 1 --2- 172.21.15.40:0/3403742987 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7ff3240a2b90 0x7ff3240a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:11.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.734+0000 7ff33affd640 1 --2- 172.21.15.40:0/3403742987 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7ff3240a2b90 0x7ff3240a5050 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ff33c1016f0 tx=0x7ff32c00a2d0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:11.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.737+0000 7ff323fff640 1 -- 172.21.15.40:0/3403742987 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7ff330014030 con 0x7ff33c101890 2024-04-21T23:16:11.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.935+0000 7ff341488640 1 -- 172.21.15.40:0/3403742987 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7ff33c1a2630 con 0x7ff33c101890 2024-04-21T23:16:11.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.935+0000 7ff323fff640 1 -- 172.21.15.40:0/3403742987 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v90) ==== 81+0+588 (secure 0 0 0) 0x7ff33009ba90 con 0x7ff33c101890 2024-04-21T23:16:11.936 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:6801/526554319 2024-04-22T23:08:50.305770+0000 2024-04-21T23:16:11.936 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:6800/526554319 2024-04-22T23:08:50.305770+0000 2024-04-21T23:16:11.936 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:0/3178763471 2024-04-22T23:08:50.305770+0000 2024-04-21T23:16:11.936 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:0/4045911364 2024-04-22T23:08:50.305770+0000 2024-04-21T23:16:11.936 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:0/276300080 2024-04-22T23:08:50.305770+0000 2024-04-21T23:16:11.936 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:0/3689649814 2024-04-22T23:08:23.743030+0000 2024-04-21T23:16:11.936 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:0/2241116715 2024-04-22T23:08:23.743030+0000 2024-04-21T23:16:11.936 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:6801/1479040326 2024-04-22T23:08:23.743030+0000 2024-04-21T23:16:11.936 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:6800/1479040326 2024-04-22T23:08:23.743030+0000 2024-04-21T23:16:11.936 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:0/681298099 2024-04-22T23:08:23.743030+0000 2024-04-21T23:16:11.937 INFO:teuthology.orchestra.run.smithi040.stderr:listed 10 entries 2024-04-21T23:16:11.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.937+0000 7ff341488640 1 -- 172.21.15.40:0/3403742987 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7ff3240a2b90 msgr2=0x7ff3240a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:11.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.937+0000 7ff341488640 1 --2- 172.21.15.40:0/3403742987 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7ff3240a2b90 0x7ff3240a5050 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ff33c1016f0 tx=0x7ff32c00a2d0 comp rx=0 tx=0).stop 2024-04-21T23:16:11.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.937+0000 7ff341488640 1 -- 172.21.15.40:0/3403742987 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff33c101890 msgr2=0x7ff33c102c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:11.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.937+0000 7ff341488640 1 --2- 172.21.15.40:0/3403742987 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff33c101890 0x7ff33c102c70 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7ff33000c970 tx=0x7ff33000ce30 comp rx=0 tx=0).stop 2024-04-21T23:16:11.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.937+0000 7ff341488640 1 -- 172.21.15.40:0/3403742987 shutdown_connections 2024-04-21T23:16:11.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.937+0000 7ff341488640 1 --2- 172.21.15.40:0/3403742987 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7ff3240a2b90 0x7ff3240a5050 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:11.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.937+0000 7ff341488640 1 --2- 172.21.15.40:0/3403742987 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff33c102250 0x7ff33c1031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:11.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.937+0000 7ff341488640 1 --2- 172.21.15.40:0/3403742987 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff33c101890 0x7ff33c102c70 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:11.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.937+0000 7ff341488640 1 --2- 172.21.15.40:0/3403742987 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff33c1006a0 0x7ff33c104630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:11.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.937+0000 7ff341488640 1 -- 172.21.15.40:0/3403742987 >> 172.21.15.40:0/3403742987 conn(0x7ff33c0fbc30 msgr2=0x7ff33c0fd6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:11.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.937+0000 7ff341488640 1 -- 172.21.15.40:0/3403742987 shutdown_connections 2024-04-21T23:16:11.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:11.937+0000 7ff341488640 1 -- 172.21.15.40:0/3403742987 wait complete. 2024-04-21T23:16:11.948 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi040.front.sepia.ceph.com... 2024-04-21T23:16:11.948 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:16:11.948 DEBUG:teuthology.orchestra.run.smithi040:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-04-21T23:16:11.971 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi040.stderr:ceph-fuse[69530]: fuse finished with error 0 and tester_r 0 2024-04-21T23:16:11.973 INFO:teuthology.orchestra.run:waiting for 300 2024-04-21T23:16:12.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:12 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/2222654616' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T23:16:12.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:12 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/3403742987' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-21T23:16:12.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:12 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/2222654616' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T23:16:12.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:12 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/3403742987' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-21T23:16:12.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:12 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/2222654616' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T23:16:12.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:12 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/3403742987' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-21T23:16:13.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:13 smithi174 ceph-mon[31409]: pgmap v301: 129 pgs: 129 active+clean; 581 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 279 B/s wr, 0 op/s 2024-04-21T23:16:13.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:13 smithi111 ceph-mon[31085]: pgmap v301: 129 pgs: 129 active+clean; 581 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 279 B/s wr, 0 op/s 2024-04-21T23:16:13.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:13 smithi040 ceph-mon[27984]: pgmap v301: 129 pgs: 129 active+clean; 581 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 279 B/s wr, 0 op/s 2024-04-21T23:16:15.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:15 smithi040 ceph-mon[27984]: pgmap v302: 129 pgs: 129 active+clean; 581 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-04-21T23:16:15.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:15 smithi174 ceph-mon[31409]: pgmap v302: 129 pgs: 129 active+clean; 581 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-04-21T23:16:15.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:15 smithi111 ceph-mon[31085]: pgmap v302: 129 pgs: 129 active+clean; 581 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-04-21T23:16:17.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:17 smithi040 ceph-mon[27984]: pgmap v303: 129 pgs: 129 active+clean; 581 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-21T23:16:17.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:17 smithi174 ceph-mon[31409]: pgmap v303: 129 pgs: 129 active+clean; 581 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-21T23:16:17.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:17 smithi111 ceph-mon[31085]: pgmap v303: 129 pgs: 129 active+clean; 581 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-21T23:16:18.075 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi040.front.sepia.ceph.com 2024-04-21T23:16:18.075 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:16:18.075 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-04-21T23:16:18.098 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T23:16:18.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.254+0000 7f3a7a9c8640 1 -- 172.21.15.40:0/1516403262 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a74102280 msgr2=0x7f3a74104670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:18.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.254+0000 7f3a7a9c8640 1 --2- 172.21.15.40:0/1516403262 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a74102280 0x7f3a74104670 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f3a5c009900 tx=0x7f3a5c02bc30 comp rx=0 tx=0).stop 2024-04-21T23:16:18.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.254+0000 7f3a7a9c8640 1 -- 172.21.15.40:0/1516403262 shutdown_connections 2024-04-21T23:16:18.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.254+0000 7f3a7a9c8640 1 --2- 172.21.15.40:0/1516403262 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a741074e0 0x7f3a741098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:18.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.254+0000 7f3a7a9c8640 1 --2- 172.21.15.40:0/1516403262 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3a74104bb0 0x7f3a74106fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:18.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.254+0000 7f3a7a9c8640 1 --2- 172.21.15.40:0/1516403262 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a74102280 0x7f3a74104670 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:18.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.254+0000 7f3a7a9c8640 1 -- 172.21.15.40:0/1516403262 >> 172.21.15.40:0/1516403262 conn(0x7f3a740fbc30 msgr2=0x7f3a740fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:18.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.254+0000 7f3a7a9c8640 1 -- 172.21.15.40:0/1516403262 shutdown_connections 2024-04-21T23:16:18.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.254+0000 7f3a7a9c8640 1 -- 172.21.15.40:0/1516403262 wait complete. 2024-04-21T23:16:18.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.254+0000 7f3a7a9c8640 1 Processor -- start 2024-04-21T23:16:18.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.254+0000 7f3a7a9c8640 1 -- start start 2024-04-21T23:16:18.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.255+0000 7f3a7a9c8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3a74102280 0x7f3a74101bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:18.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.255+0000 7f3a7a9c8640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a74104bb0 0x7f3a74100230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:18.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.255+0000 7f3a7a9c8640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a741074e0 0x7f3a74100770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:18.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.255+0000 7f3a7a9c8640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3a7410d440 con 0x7f3a74104bb0 2024-04-21T23:16:18.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.255+0000 7f3a7a9c8640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3a7410d2c0 con 0x7f3a741074e0 2024-04-21T23:16:18.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.255+0000 7f3a7a9c8640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3a7410d5c0 con 0x7f3a74102280 2024-04-21T23:16:18.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.255+0000 7f3a737fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a74104bb0 0x7f3a74100230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:18.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.255+0000 7f3a737fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a74104bb0 0x7f3a74100230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35510/0 (socket says 172.21.15.40:35510) 2024-04-21T23:16:18.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.255+0000 7f3a737fe640 1 -- 172.21.15.40:0/2349977229 learned_addr learned my addr 172.21.15.40:0/2349977229 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:16:18.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.255+0000 7f3a73fff640 1 --2- 172.21.15.40:0/2349977229 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3a74102280 0x7f3a74101bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:18.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.255+0000 7f3a78f65640 1 --2- 172.21.15.40:0/2349977229 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a741074e0 0x7f3a74100770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:18.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.255+0000 7f3a737fe640 1 -- 172.21.15.40:0/2349977229 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3a74102280 msgr2=0x7f3a74101bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:18.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.255+0000 7f3a737fe640 1 --2- 172.21.15.40:0/2349977229 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3a74102280 0x7f3a74101bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:18.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.255+0000 7f3a737fe640 1 -- 172.21.15.40:0/2349977229 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a741074e0 msgr2=0x7f3a74100770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:18.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.255+0000 7f3a737fe640 1 --2- 172.21.15.40:0/2349977229 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a741074e0 0x7f3a74100770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:18.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.255+0000 7f3a737fe640 1 -- 172.21.15.40:0/2349977229 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a74101000 con 0x7f3a74104bb0 2024-04-21T23:16:18.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.255+0000 7f3a737fe640 1 --2- 172.21.15.40:0/2349977229 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a74104bb0 0x7f3a74100230 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f3a6400d740 tx=0x7f3a6400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:18.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.256+0000 7f3a717fa640 1 -- 172.21.15.40:0/2349977229 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a64017070 con 0x7f3a74104bb0 2024-04-21T23:16:18.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.256+0000 7f3a717fa640 1 -- 172.21.15.40:0/2349977229 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a6400de50 con 0x7f3a74104bb0 2024-04-21T23:16:18.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.256+0000 7f3a7a9c8640 1 -- 172.21.15.40:0/2349977229 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a741a1c30 con 0x7f3a74104bb0 2024-04-21T23:16:18.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.256+0000 7f3a717fa640 1 -- 172.21.15.40:0/2349977229 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a64004e00 con 0x7f3a74104bb0 2024-04-21T23:16:18.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.256+0000 7f3a7a9c8640 1 -- 172.21.15.40:0/2349977229 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a741a2180 con 0x7f3a74104bb0 2024-04-21T23:16:18.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.257+0000 7f3a7a9c8640 1 -- 172.21.15.40:0/2349977229 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a740fd720 con 0x7f3a74104bb0 2024-04-21T23:16:18.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.258+0000 7f3a717fa640 1 -- 172.21.15.40:0/2349977229 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f3a64004fa0 con 0x7f3a74104bb0 2024-04-21T23:16:18.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.258+0000 7f3a717fa640 1 --2- 172.21.15.40:0/2349977229 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f3a600a2c10 0x7f3a600a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:18.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.259+0000 7f3a717fa640 1 -- 172.21.15.40:0/2349977229 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f3a640cef00 con 0x7f3a74104bb0 2024-04-21T23:16:18.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.259+0000 7f3a73fff640 1 --2- 172.21.15.40:0/2349977229 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f3a600a2c10 0x7f3a600a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:18.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.259+0000 7f3a73fff640 1 --2- 172.21.15.40:0/2349977229 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f3a600a2c10 0x7f3a600a50d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f3a5c02c7b0 tx=0x7f3a5c005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:18.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.262+0000 7f3a717fa640 1 -- 172.21.15.40:0/2349977229 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f3a6409b650 con 0x7f3a74104bb0 2024-04-21T23:16:18.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.447+0000 7f3a7a9c8640 1 -- 172.21.15.40:0/2349977229 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3a7402d370 con 0x7f3a74104bb0 2024-04-21T23:16:18.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.448+0000 7f3a717fa640 1 -- 172.21.15.40:0/2349977229 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24138 (secure 0 0 0) 0x7f3a6409f950 con 0x7f3a74104bb0 2024-04-21T23:16:18.448 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:16:18.448 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":90,"fsid":"d814e4dc-0033-11ef-bc93-c7b262605968","created":"2024-04-21T23:07:52.659207+0000","modified":"2024-04-21T23:16:00.780824+0000","last_up_change":"2024-04-21T23:15:05.308821+0000","last_in_change":"2024-04-21T23:14:45.447491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:11:14.824775+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T23:15:54.322571+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T23:15:55.164493+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T23:15:58.838266+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"0e8d692a-22fe-4a87-8d99-9151f96cbb02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6803","nonce":2009758784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6805","nonce":2009758784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6809","nonce":2009758784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6807","nonce":2009758784}]},"public_addr":"172.21.15.40:6803/2009758784","cluster_addr":"172.21.15.40:6805/2009758784","heartbeat_back_addr":"172.21.15.40:6809/2009758784","heartbeat_front_addr":"172.21.15.40:6807/2009758784","state":["exists","up"]},{"osd":1,"uuid":"fe5fe11b-0e84-4359-83a7-1ba45105755a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6801","nonce":1763903836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6803","nonce":1763903836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6807","nonce":1763903836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6805","nonce":1763903836}]},"public_addr":"172.21.15.111:6801/1763903836","cluster_addr":"172.21.15.111:6803/1763903836","heartbeat_back_addr":"172.21.15.111:6807/1763903836","heartbeat_front_addr":"172.21.15.111:6805/1763903836","state":["exists","up"]},{"osd":2,"uuid":"813ad735-5cc2-4da0-ae16-6db238395fbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6801","nonce":439695825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6803","nonce":439695825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6807","nonce":439695825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6805","nonce":439695825}]},"public_addr":"172.21.15.174:6801/439695825","cluster_addr":"172.21.15.174:6803/439695825","heartbeat_back_addr":"172.21.15.174:6807/439695825","heartbeat_front_addr":"172.21.15.174:6805/439695825","state":["exists","up"]},{"osd":3,"uuid":"09becb03-b7ce-4ffc-917d-b7900ec37e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6811","nonce":3454312436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6813","nonce":3454312436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6817","nonce":3454312436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6815","nonce":3454312436}]},"public_addr":"172.21.15.40:6811/3454312436","cluster_addr":"172.21.15.40:6813/3454312436","heartbeat_back_addr":"172.21.15.40:6817/3454312436","heartbeat_front_addr":"172.21.15.40:6815/3454312436","state":["exists","up"]},{"osd":4,"uuid":"46eabfd7-d88a-4efd-94eb-04354d3a9d7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6809","nonce":1810866070}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6811","nonce":1810866070}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6815","nonce":1810866070}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6813","nonce":1810866070}]},"public_addr":"172.21.15.111:6809/1810866070","cluster_addr":"172.21.15.111:6811/1810866070","heartbeat_back_addr":"172.21.15.111:6815/1810866070","heartbeat_front_addr":"172.21.15.111:6813/1810866070","state":["exists","up"]},{"osd":5,"uuid":"6f73b582-6c65-41db-aab0-9b360f83d5c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6809","nonce":3752986711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6811","nonce":3752986711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6815","nonce":3752986711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6813","nonce":3752986711}]},"public_addr":"172.21.15.174:6809/3752986711","cluster_addr":"172.21.15.174:6811/3752986711","heartbeat_back_addr":"172.21.15.174:6815/3752986711","heartbeat_front_addr":"172.21.15.174:6813/3752986711","state":["exists","up"]},{"osd":6,"uuid":"39f305f0-12d2-4732-aaa0-cd66700af538","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6819","nonce":1618855427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6821","nonce":1618855427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6825","nonce":1618855427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6823","nonce":1618855427}]},"public_addr":"172.21.15.40:6819/1618855427","cluster_addr":"172.21.15.40:6821/1618855427","heartbeat_back_addr":"172.21.15.40:6825/1618855427","heartbeat_front_addr":"172.21.15.40:6823/1618855427","state":["exists","up"]},{"osd":7,"uuid":"769f2468-5102-490b-89f8-d0444d3169a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6817","nonce":927054162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6819","nonce":927054162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6823","nonce":927054162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6821","nonce":927054162}]},"public_addr":"172.21.15.111:6817/927054162","cluster_addr":"172.21.15.111:6819/927054162","heartbeat_back_addr":"172.21.15.111:6823/927054162","heartbeat_front_addr":"172.21.15.111:6821/927054162","state":["exists","up"]},{"osd":8,"uuid":"1a444e80-007d-473b-b972-4c814a813c22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6817","nonce":3817848746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6819","nonce":3817848746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6823","nonce":3817848746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6821","nonce":3817848746}]},"public_addr":"172.21.15.174:6817/3817848746","cluster_addr":"172.21.15.174:6819/3817848746","heartbeat_back_addr":"172.21.15.174:6823/3817848746","heartbeat_front_addr":"172.21.15.174:6821/3817848746","state":["exists","up"]},{"osd":9,"uuid":"9a6b6a56-6ae5-49fe-9e5b-cb17d686e0dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6827","nonce":1649446531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6829","nonce":1649446531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6833","nonce":1649446531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6831","nonce":1649446531}]},"public_addr":"172.21.15.40:6827/1649446531","cluster_addr":"172.21.15.40:6829/1649446531","heartbeat_back_addr":"172.21.15.40:6833/1649446531","heartbeat_front_addr":"172.21.15.40:6831/1649446531","state":["exists","up"]},{"osd":10,"uuid":"a70038ba-e2ad-4170-85cc-ea0470a2edef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6825","nonce":1227538935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6827","nonce":1227538935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6831","nonce":1227538935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6829","nonce":1227538935}]},"public_addr":"172.21.15.111:6825/1227538935","cluster_addr":"172.21.15.111:6827/1227538935","heartbeat_back_addr":"172.21.15.111:6831/1227538935","heartbeat_front_addr":"172.21.15.111:6829/1227538935","state":["exists","up"]},{"osd":11,"uuid":"781532bb-7a5e-44e3-8eb7-b584b0054666","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6825","nonce":2045865553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6827","nonce":2045865553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6831","nonce":2045865553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6829","nonce":2045865553}]},"public_addr":"172.21.15.174:6825/2045865553","cluster_addr":"172.21.15.174:6827/2045865553","heartbeat_back_addr":"172.21.15.174:6831/2045865553","heartbeat_front_addr":"172.21.15.174:6829/2045865553","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:10:24.761290+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:10:48.565428+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:11:12.307818+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:11:37.951529+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:02.832523+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:27.568879+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:52.997707+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:13:18.937274+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:13:44.708657+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:14:10.833855+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:14:37.078272+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:15:03.878076+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:6801/526554319":"2024-04-22T23:08:50.305770+0000","172.21.15.40:6800/526554319":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/3178763471":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/4045911364":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/276300080":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/3689649814":"2024-04-22T23:08:23.743030+0000","172.21.15.40:0/2241116715":"2024-04-22T23:08:23.743030+0000","172.21.15.40:6801/1479040326":"2024-04-22T23:08:23.743030+0000","172.21.15.40:6800/1479040326":"2024-04-22T23:08:23.743030+0000","172.21.15.40:0/681298099":"2024-04-22T23:08:23.743030+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T23:16:18.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.450+0000 7f3a7a9c8640 1 -- 172.21.15.40:0/2349977229 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f3a600a2c10 msgr2=0x7f3a600a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:18.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.450+0000 7f3a7a9c8640 1 --2- 172.21.15.40:0/2349977229 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f3a600a2c10 0x7f3a600a50d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f3a5c02c7b0 tx=0x7f3a5c005fd0 comp rx=0 tx=0).stop 2024-04-21T23:16:18.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.450+0000 7f3a7a9c8640 1 -- 172.21.15.40:0/2349977229 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a74104bb0 msgr2=0x7f3a74100230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:18.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.450+0000 7f3a7a9c8640 1 --2- 172.21.15.40:0/2349977229 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a74104bb0 0x7f3a74100230 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f3a6400d740 tx=0x7f3a6400dc00 comp rx=0 tx=0).stop 2024-04-21T23:16:18.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.450+0000 7f3a7a9c8640 1 -- 172.21.15.40:0/2349977229 shutdown_connections 2024-04-21T23:16:18.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.450+0000 7f3a7a9c8640 1 --2- 172.21.15.40:0/2349977229 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f3a600a2c10 0x7f3a600a50d0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:18.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.450+0000 7f3a7a9c8640 1 --2- 172.21.15.40:0/2349977229 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a741074e0 0x7f3a74100770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:18.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.450+0000 7f3a7a9c8640 1 --2- 172.21.15.40:0/2349977229 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a74104bb0 0x7f3a74100230 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:18.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.450+0000 7f3a7a9c8640 1 --2- 172.21.15.40:0/2349977229 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3a74102280 0x7f3a74101bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:18.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.450+0000 7f3a7a9c8640 1 -- 172.21.15.40:0/2349977229 >> 172.21.15.40:0/2349977229 conn(0x7f3a740fbc30 msgr2=0x7f3a74102cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:18.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.450+0000 7f3a7a9c8640 1 -- 172.21.15.40:0/2349977229 shutdown_connections 2024-04-21T23:16:18.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.450+0000 7f3a7a9c8640 1 -- 172.21.15.40:0/2349977229 wait complete. 2024-04-21T23:16:18.462 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T23:16:18.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.564+0000 7f21f9c40640 1 -- 172.21.15.40:0/3731508553 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21f41029a0 msgr2=0x7f21f4102e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:18.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.564+0000 7f21f9c40640 1 --2- 172.21.15.40:0/3731508553 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21f41029a0 0x7f21f4102e20 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f21dc009900 tx=0x7f21dc02bc30 comp rx=0 tx=0).stop 2024-04-21T23:16:18.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.564+0000 7f21f9c40640 1 -- 172.21.15.40:0/3731508553 shutdown_connections 2024-04-21T23:16:18.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.564+0000 7f21f9c40640 1 --2- 172.21.15.40:0/3731508553 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f21f4103360 0x7f21f4109bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:18.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.564+0000 7f21f9c40640 1 --2- 172.21.15.40:0/3731508553 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21f41029a0 0x7f21f4102e20 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:18.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.564+0000 7f21f9c40640 1 --2- 172.21.15.40:0/3731508553 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f21f41017b0 0x7f21f4101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:18.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.564+0000 7f21f9c40640 1 -- 172.21.15.40:0/3731508553 >> 172.21.15.40:0/3731508553 conn(0x7f21f40fcd40 msgr2=0x7f21f40ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:18.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.564+0000 7f21f9c40640 1 -- 172.21.15.40:0/3731508553 shutdown_connections 2024-04-21T23:16:18.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.564+0000 7f21f9c40640 1 -- 172.21.15.40:0/3731508553 wait complete. 2024-04-21T23:16:18.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.565+0000 7f21f9c40640 1 Processor -- start 2024-04-21T23:16:18.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.565+0000 7f21f9c40640 1 -- start start 2024-04-21T23:16:18.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.565+0000 7f21f9c40640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f21f41017b0 0x7f21f4197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:18.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.565+0000 7f21f9c40640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f21f41029a0 0x7f21f4198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:18.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.565+0000 7f21f9c40640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21f4103360 0x7f21f419f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:18.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.565+0000 7f21f9c40640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f21f410d430 con 0x7f21f4103360 2024-04-21T23:16:18.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.565+0000 7f21f9c40640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f21f410d2b0 con 0x7f21f41017b0 2024-04-21T23:16:18.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.565+0000 7f21f9c40640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f21f410d5b0 con 0x7f21f41029a0 2024-04-21T23:16:18.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.565+0000 7f21f3fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21f4103360 0x7f21f419f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:18.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.565+0000 7f21f3fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21f4103360 0x7f21f419f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35528/0 (socket says 172.21.15.40:35528) 2024-04-21T23:16:18.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.565+0000 7f21f3fff640 1 -- 172.21.15.40:0/898173547 learned_addr learned my addr 172.21.15.40:0/898173547 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:16:18.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.565+0000 7f21f2ffd640 1 --2- 172.21.15.40:0/898173547 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f21f41029a0 0x7f21f4198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:18.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.565+0000 7f21f37fe640 1 --2- 172.21.15.40:0/898173547 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f21f41017b0 0x7f21f4197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:18.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.565+0000 7f21f3fff640 1 -- 172.21.15.40:0/898173547 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f21f41029a0 msgr2=0x7f21f4198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:18.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.565+0000 7f21f3fff640 1 --2- 172.21.15.40:0/898173547 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f21f41029a0 0x7f21f4198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:18.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.565+0000 7f21f3fff640 1 -- 172.21.15.40:0/898173547 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f21f41017b0 msgr2=0x7f21f4197b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:18.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.565+0000 7f21f3fff640 1 --2- 172.21.15.40:0/898173547 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f21f41017b0 0x7f21f4197b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:18.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.565+0000 7f21f3fff640 1 -- 172.21.15.40:0/898173547 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21f419f720 con 0x7f21f4103360 2024-04-21T23:16:18.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.566+0000 7f21f3fff640 1 --2- 172.21.15.40:0/898173547 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21f4103360 0x7f21f419f0b0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f21ec00cad0 tx=0x7f21ec00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:18.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.566+0000 7f21f0ff9640 1 -- 172.21.15.40:0/898173547 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21ec00ae20 con 0x7f21f4103360 2024-04-21T23:16:18.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.566+0000 7f21f9c40640 1 -- 172.21.15.40:0/898173547 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21f419fa10 con 0x7f21f4103360 2024-04-21T23:16:18.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.566+0000 7f21f0ff9640 1 -- 172.21.15.40:0/898173547 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f21ec005020 con 0x7f21f4103360 2024-04-21T23:16:18.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.566+0000 7f21f0ff9640 1 -- 172.21.15.40:0/898173547 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21ec00f6b0 con 0x7f21f4103360 2024-04-21T23:16:18.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.566+0000 7f21f9c40640 1 -- 172.21.15.40:0/898173547 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21f419ff80 con 0x7f21f4103360 2024-04-21T23:16:18.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.567+0000 7f21f9c40640 1 -- 172.21.15.40:0/898173547 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21f4108ad0 con 0x7f21f4103360 2024-04-21T23:16:18.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.568+0000 7f21f0ff9640 1 -- 172.21.15.40:0/898173547 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f21ec0051d0 con 0x7f21f4103360 2024-04-21T23:16:18.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.568+0000 7f21f0ff9640 1 --2- 172.21.15.40:0/898173547 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f21e00a2b90 0x7f21e00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:18.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.569+0000 7f21f37fe640 1 --2- 172.21.15.40:0/898173547 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f21e00a2b90 0x7f21e00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:18.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.569+0000 7f21f0ff9640 1 -- 172.21.15.40:0/898173547 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f21ec0cf670 con 0x7f21f4103360 2024-04-21T23:16:18.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.569+0000 7f21f37fe640 1 --2- 172.21.15.40:0/898173547 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f21e00a2b90 0x7f21e00a5050 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f21e4005fd0 tx=0x7f21e4005c70 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:18.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.573+0000 7f21f0ff9640 1 -- 172.21.15.40:0/898173547 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f21ec014030 con 0x7f21f4103360 2024-04-21T23:16:18.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.762+0000 7f21f9c40640 1 -- 172.21.15.40:0/898173547 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f21f4062340 con 0x7f21f4103360 2024-04-21T23:16:18.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.763+0000 7f21f0ff9640 1 -- 172.21.15.40:0/898173547 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24138 (secure 0 0 0) 0x7f21ec09c330 con 0x7f21f4103360 2024-04-21T23:16:18.763 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:16:18.763 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":90,"fsid":"d814e4dc-0033-11ef-bc93-c7b262605968","created":"2024-04-21T23:07:52.659207+0000","modified":"2024-04-21T23:16:00.780824+0000","last_up_change":"2024-04-21T23:15:05.308821+0000","last_in_change":"2024-04-21T23:14:45.447491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:11:14.824775+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T23:15:54.322571+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T23:15:55.164493+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T23:15:58.838266+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"0e8d692a-22fe-4a87-8d99-9151f96cbb02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6803","nonce":2009758784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6805","nonce":2009758784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6809","nonce":2009758784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6807","nonce":2009758784}]},"public_addr":"172.21.15.40:6803/2009758784","cluster_addr":"172.21.15.40:6805/2009758784","heartbeat_back_addr":"172.21.15.40:6809/2009758784","heartbeat_front_addr":"172.21.15.40:6807/2009758784","state":["exists","up"]},{"osd":1,"uuid":"fe5fe11b-0e84-4359-83a7-1ba45105755a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6801","nonce":1763903836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6803","nonce":1763903836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6807","nonce":1763903836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6805","nonce":1763903836}]},"public_addr":"172.21.15.111:6801/1763903836","cluster_addr":"172.21.15.111:6803/1763903836","heartbeat_back_addr":"172.21.15.111:6807/1763903836","heartbeat_front_addr":"172.21.15.111:6805/1763903836","state":["exists","up"]},{"osd":2,"uuid":"813ad735-5cc2-4da0-ae16-6db238395fbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6801","nonce":439695825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6803","nonce":439695825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6807","nonce":439695825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6805","nonce":439695825}]},"public_addr":"172.21.15.174:6801/439695825","cluster_addr":"172.21.15.174:6803/439695825","heartbeat_back_addr":"172.21.15.174:6807/439695825","heartbeat_front_addr":"172.21.15.174:6805/439695825","state":["exists","up"]},{"osd":3,"uuid":"09becb03-b7ce-4ffc-917d-b7900ec37e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6811","nonce":3454312436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6813","nonce":3454312436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6817","nonce":3454312436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6815","nonce":3454312436}]},"public_addr":"172.21.15.40:6811/3454312436","cluster_addr":"172.21.15.40:6813/3454312436","heartbeat_back_addr":"172.21.15.40:6817/3454312436","heartbeat_front_addr":"172.21.15.40:6815/3454312436","state":["exists","up"]},{"osd":4,"uuid":"46eabfd7-d88a-4efd-94eb-04354d3a9d7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6809","nonce":1810866070}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6811","nonce":1810866070}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6815","nonce":1810866070}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6813","nonce":1810866070}]},"public_addr":"172.21.15.111:6809/1810866070","cluster_addr":"172.21.15.111:6811/1810866070","heartbeat_back_addr":"172.21.15.111:6815/1810866070","heartbeat_front_addr":"172.21.15.111:6813/1810866070","state":["exists","up"]},{"osd":5,"uuid":"6f73b582-6c65-41db-aab0-9b360f83d5c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6809","nonce":3752986711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6811","nonce":3752986711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6815","nonce":3752986711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6813","nonce":3752986711}]},"public_addr":"172.21.15.174:6809/3752986711","cluster_addr":"172.21.15.174:6811/3752986711","heartbeat_back_addr":"172.21.15.174:6815/3752986711","heartbeat_front_addr":"172.21.15.174:6813/3752986711","state":["exists","up"]},{"osd":6,"uuid":"39f305f0-12d2-4732-aaa0-cd66700af538","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6819","nonce":1618855427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6821","nonce":1618855427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6825","nonce":1618855427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6823","nonce":1618855427}]},"public_addr":"172.21.15.40:6819/1618855427","cluster_addr":"172.21.15.40:6821/1618855427","heartbeat_back_addr":"172.21.15.40:6825/1618855427","heartbeat_front_addr":"172.21.15.40:6823/1618855427","state":["exists","up"]},{"osd":7,"uuid":"769f2468-5102-490b-89f8-d0444d3169a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6817","nonce":927054162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6819","nonce":927054162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6823","nonce":927054162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6821","nonce":927054162}]},"public_addr":"172.21.15.111:6817/927054162","cluster_addr":"172.21.15.111:6819/927054162","heartbeat_back_addr":"172.21.15.111:6823/927054162","heartbeat_front_addr":"172.21.15.111:6821/927054162","state":["exists","up"]},{"osd":8,"uuid":"1a444e80-007d-473b-b972-4c814a813c22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6817","nonce":3817848746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6819","nonce":3817848746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6823","nonce":3817848746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6821","nonce":3817848746}]},"public_addr":"172.21.15.174:6817/3817848746","cluster_addr":"172.21.15.174:6819/3817848746","heartbeat_back_addr":"172.21.15.174:6823/3817848746","heartbeat_front_addr":"172.21.15.174:6821/3817848746","state":["exists","up"]},{"osd":9,"uuid":"9a6b6a56-6ae5-49fe-9e5b-cb17d686e0dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6827","nonce":1649446531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6829","nonce":1649446531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6833","nonce":1649446531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6831","nonce":1649446531}]},"public_addr":"172.21.15.40:6827/1649446531","cluster_addr":"172.21.15.40:6829/1649446531","heartbeat_back_addr":"172.21.15.40:6833/1649446531","heartbeat_front_addr":"172.21.15.40:6831/1649446531","state":["exists","up"]},{"osd":10,"uuid":"a70038ba-e2ad-4170-85cc-ea0470a2edef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6825","nonce":1227538935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6827","nonce":1227538935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6831","nonce":1227538935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6829","nonce":1227538935}]},"public_addr":"172.21.15.111:6825/1227538935","cluster_addr":"172.21.15.111:6827/1227538935","heartbeat_back_addr":"172.21.15.111:6831/1227538935","heartbeat_front_addr":"172.21.15.111:6829/1227538935","state":["exists","up"]},{"osd":11,"uuid":"781532bb-7a5e-44e3-8eb7-b584b0054666","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6825","nonce":2045865553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6827","nonce":2045865553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6831","nonce":2045865553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6829","nonce":2045865553}]},"public_addr":"172.21.15.174:6825/2045865553","cluster_addr":"172.21.15.174:6827/2045865553","heartbeat_back_addr":"172.21.15.174:6831/2045865553","heartbeat_front_addr":"172.21.15.174:6829/2045865553","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:10:24.761290+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:10:48.565428+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:11:12.307818+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:11:37.951529+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:02.832523+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:27.568879+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:52.997707+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:13:18.937274+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:13:44.708657+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:14:10.833855+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:14:37.078272+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:15:03.878076+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:6801/526554319":"2024-04-22T23:08:50.305770+0000","172.21.15.40:6800/526554319":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/3178763471":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/4045911364":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/276300080":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/3689649814":"2024-04-22T23:08:23.743030+0000","172.21.15.40:0/2241116715":"2024-04-22T23:08:23.743030+0000","172.21.15.40:6801/1479040326":"2024-04-22T23:08:23.743030+0000","172.21.15.40:6800/1479040326":"2024-04-22T23:08:23.743030+0000","172.21.15.40:0/681298099":"2024-04-22T23:08:23.743030+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T23:16:18.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.766+0000 7f21f9c40640 1 -- 172.21.15.40:0/898173547 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f21e00a2b90 msgr2=0x7f21e00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:18.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.766+0000 7f21f9c40640 1 --2- 172.21.15.40:0/898173547 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f21e00a2b90 0x7f21e00a5050 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f21e4005fd0 tx=0x7f21e4005c70 comp rx=0 tx=0).stop 2024-04-21T23:16:18.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.766+0000 7f21f9c40640 1 -- 172.21.15.40:0/898173547 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21f4103360 msgr2=0x7f21f419f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:18.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.766+0000 7f21f9c40640 1 --2- 172.21.15.40:0/898173547 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21f4103360 0x7f21f419f0b0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f21ec00cad0 tx=0x7f21ec00cf90 comp rx=0 tx=0).stop 2024-04-21T23:16:18.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.766+0000 7f21f9c40640 1 -- 172.21.15.40:0/898173547 shutdown_connections 2024-04-21T23:16:18.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.766+0000 7f21f9c40640 1 --2- 172.21.15.40:0/898173547 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f21e00a2b90 0x7f21e00a5050 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:18.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.766+0000 7f21f9c40640 1 --2- 172.21.15.40:0/898173547 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21f4103360 0x7f21f419f0b0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:18.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.766+0000 7f21f9c40640 1 --2- 172.21.15.40:0/898173547 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f21f41029a0 0x7f21f4198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:18.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.766+0000 7f21f9c40640 1 --2- 172.21.15.40:0/898173547 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f21f41017b0 0x7f21f4197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:18.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.766+0000 7f21f9c40640 1 -- 172.21.15.40:0/898173547 >> 172.21.15.40:0/898173547 conn(0x7f21f40fcd40 msgr2=0x7f21f4105ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:18.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.766+0000 7f21f9c40640 1 -- 172.21.15.40:0/898173547 shutdown_connections 2024-04-21T23:16:18.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.766+0000 7f21f9c40640 1 -- 172.21.15.40:0/898173547 wait complete. 2024-04-21T23:16:18.775 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:18 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/2349977229' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:16:18.776 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-21T23:16:18.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.881+0000 7f96a44a7640 1 -- 172.21.15.40:0/1424481653 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f969c102250 msgr2=0x7f969c108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:18.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.881+0000 7f96a44a7640 1 --2- 172.21.15.40:0/1424481653 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f969c102250 0x7f969c108ae0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f9694009ca0 tx=0x7f969402c0f0 comp rx=0 tx=0).stop 2024-04-21T23:16:18.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.881+0000 7f96a44a7640 1 -- 172.21.15.40:0/1424481653 shutdown_connections 2024-04-21T23:16:18.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.881+0000 7f96a44a7640 1 --2- 172.21.15.40:0/1424481653 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f969c102250 0x7f969c108ae0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:18.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.881+0000 7f96a44a7640 1 --2- 172.21.15.40:0/1424481653 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f969c101890 0x7f969c101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:18.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.881+0000 7f96a44a7640 1 --2- 172.21.15.40:0/1424481653 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f969c1006a0 0x7f969c100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:18.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.881+0000 7f96a44a7640 1 -- 172.21.15.40:0/1424481653 >> 172.21.15.40:0/1424481653 conn(0x7f969c0fbc30 msgr2=0x7f969c0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:18.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.882+0000 7f96a44a7640 1 -- 172.21.15.40:0/1424481653 shutdown_connections 2024-04-21T23:16:18.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.882+0000 7f96a44a7640 1 -- 172.21.15.40:0/1424481653 wait complete. 2024-04-21T23:16:18.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.882+0000 7f96a44a7640 1 Processor -- start 2024-04-21T23:16:18.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.882+0000 7f96a44a7640 1 -- start start 2024-04-21T23:16:18.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.882+0000 7f96a44a7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f969c1006a0 0x7f969c06a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:18.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.882+0000 7f96a44a7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f969c101890 0x7f969c19eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:18.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.882+0000 7f96a44a7640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f969c102250 0x7f969c1a10c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:18.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.882+0000 7f96a44a7640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f969c10c300 con 0x7f969c1006a0 2024-04-21T23:16:18.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.882+0000 7f96a44a7640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f969c10c180 con 0x7f969c102250 2024-04-21T23:16:18.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.882+0000 7f96a44a7640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f969c10c480 con 0x7f969c101890 2024-04-21T23:16:18.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.882+0000 7f96a2243640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f969c1006a0 0x7f969c06a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:18.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.882+0000 7f96a2243640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f969c1006a0 0x7f969c06a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35556/0 (socket says 172.21.15.40:35556) 2024-04-21T23:16:18.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.882+0000 7f96a2243640 1 -- 172.21.15.40:0/187166450 learned_addr learned my addr 172.21.15.40:0/187166450 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:16:18.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.883+0000 7f96a1a42640 1 --2- 172.21.15.40:0/187166450 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f969c101890 0x7f969c19eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:18.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.883+0000 7f96a2243640 1 -- 172.21.15.40:0/187166450 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f969c101890 msgr2=0x7f969c19eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:18.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.883+0000 7f96a2243640 1 --2- 172.21.15.40:0/187166450 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f969c101890 0x7f969c19eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:18.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.883+0000 7f96a2243640 1 -- 172.21.15.40:0/187166450 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f969c102250 msgr2=0x7f969c1a10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:18.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.883+0000 7f96a2243640 1 --2- 172.21.15.40:0/187166450 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f969c102250 0x7f969c1a10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:18.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.883+0000 7f96a2243640 1 -- 172.21.15.40:0/187166450 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f969c1a18b0 con 0x7f969c1006a0 2024-04-21T23:16:18.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.883+0000 7f96a2243640 1 --2- 172.21.15.40:0/187166450 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f969c1006a0 0x7f969c06a8b0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f969800e9c0 tx=0x7f969800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:18.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.883+0000 7f96937fe640 1 -- 172.21.15.40:0/187166450 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f969800cd80 con 0x7f969c1006a0 2024-04-21T23:16:18.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.883+0000 7f96937fe640 1 -- 172.21.15.40:0/187166450 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f96980035f0 con 0x7f969c1006a0 2024-04-21T23:16:18.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.883+0000 7f96a44a7640 1 -- 172.21.15.40:0/187166450 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f969c1a1b40 con 0x7f969c1006a0 2024-04-21T23:16:18.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.883+0000 7f96937fe640 1 -- 172.21.15.40:0/187166450 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9698010690 con 0x7f969c1006a0 2024-04-21T23:16:18.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.883+0000 7f96a44a7640 1 -- 172.21.15.40:0/187166450 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f969c1a20e0 con 0x7f969c1006a0 2024-04-21T23:16:18.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.884+0000 7f96a44a7640 1 -- 172.21.15.40:0/187166450 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f969c1079c0 con 0x7f969c1006a0 2024-04-21T23:16:18.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.885+0000 7f96937fe640 1 -- 172.21.15.40:0/187166450 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f96980037a0 con 0x7f969c1006a0 2024-04-21T23:16:18.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.886+0000 7f96937fe640 1 --2- 172.21.15.40:0/187166450 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f96880a2c40 0x7f96880a5100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:18.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.886+0000 7f96a1a42640 1 --2- 172.21.15.40:0/187166450 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f96880a2c40 0x7f96880a5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:18.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.886+0000 7f96937fe640 1 -- 172.21.15.40:0/187166450 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f96980cf720 con 0x7f969c1006a0 2024-04-21T23:16:18.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.887+0000 7f96a1a42640 1 --2- 172.21.15.40:0/187166450 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f96880a2c40 0x7f96880a5100 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f969c06a620 tx=0x7f968c00a4c0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:18.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:18.891+0000 7f96937fe640 1 -- 172.21.15.40:0/187166450 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f969809c440 con 0x7f969c1006a0 2024-04-21T23:16:18.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:18 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/2349977229' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:16:18.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:18 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/2349977229' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:16:19.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:19.112+0000 7f96a44a7640 1 -- 172.21.15.40:0/187166450 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f969c02d170 con 0x7f969c1006a0 2024-04-21T23:16:19.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:19.463+0000 7f96937fe640 1 -- 172.21.15.40:0/187166450 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) ==== 118+0+0 (secure 0 0 0) 0x7f969809bd40 con 0x7f969c1006a0 2024-04-21T23:16:19.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:19.465+0000 7f96a44a7640 1 -- 172.21.15.40:0/187166450 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f96880a2c40 msgr2=0x7f96880a5100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:19.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:19.465+0000 7f96a44a7640 1 --2- 172.21.15.40:0/187166450 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f96880a2c40 0x7f96880a5100 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f969c06a620 tx=0x7f968c00a4c0 comp rx=0 tx=0).stop 2024-04-21T23:16:19.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:19.465+0000 7f96a44a7640 1 -- 172.21.15.40:0/187166450 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f969c1006a0 msgr2=0x7f969c06a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:19.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:19.465+0000 7f96a44a7640 1 --2- 172.21.15.40:0/187166450 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f969c1006a0 0x7f969c06a8b0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f969800e9c0 tx=0x7f969800ee80 comp rx=0 tx=0).stop 2024-04-21T23:16:19.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:19.465+0000 7f96a44a7640 1 -- 172.21.15.40:0/187166450 shutdown_connections 2024-04-21T23:16:19.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:19.465+0000 7f96a44a7640 1 --2- 172.21.15.40:0/187166450 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f96880a2c40 0x7f96880a5100 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:19.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:19.465+0000 7f96a44a7640 1 --2- 172.21.15.40:0/187166450 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f969c102250 0x7f969c1a10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:19.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:19.465+0000 7f96a44a7640 1 --2- 172.21.15.40:0/187166450 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f969c101890 0x7f969c19eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:19.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:19.465+0000 7f96a44a7640 1 --2- 172.21.15.40:0/187166450 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f969c1006a0 0x7f969c06a8b0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:19.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:19.465+0000 7f96a44a7640 1 -- 172.21.15.40:0/187166450 >> 172.21.15.40:0/187166450 conn(0x7f969c0fbc30 msgr2=0x7f969c104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:19.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:19.465+0000 7f96a44a7640 1 -- 172.21.15.40:0/187166450 shutdown_connections 2024-04-21T23:16:19.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:19.465+0000 7f96a44a7640 1 -- 172.21.15.40:0/187166450 wait complete. 2024-04-21T23:16:19.476 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2}} 2024-04-21T23:16:19.476 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2024-04-21T23:16:19.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:19.582+0000 7f25f601b640 1 -- 172.21.15.40:0/3393743313 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25f0107460 msgr2=0x7f25f01098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:19.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:19.582+0000 7f25f601b640 1 --2- 172.21.15.40:0/3393743313 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25f0107460 0x7f25f01098b0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f25e8009ca0 tx=0x7f25e802c0f0 comp rx=0 tx=0).stop 2024-04-21T23:16:19.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:19.582+0000 7f25f601b640 1 -- 172.21.15.40:0/3393743313 shutdown_connections 2024-04-21T23:16:19.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:19.582+0000 7f25f601b640 1 --2- 172.21.15.40:0/3393743313 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25f0107460 0x7f25f01098b0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:19.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:19.582+0000 7f25f601b640 1 --2- 172.21.15.40:0/3393743313 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25f0104b30 0x7f25f0106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:19.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:19.582+0000 7f25f601b640 1 --2- 172.21.15.40:0/3393743313 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25f0102200 0x7f25f01045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:19.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:19.582+0000 7f25f601b640 1 -- 172.21.15.40:0/3393743313 >> 172.21.15.40:0/3393743313 conn(0x7f25f00fbbf0 msgr2=0x7f25f00fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:19.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:19.582+0000 7f25f601b640 1 -- 172.21.15.40:0/3393743313 shutdown_connections 2024-04-21T23:16:19.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:19.582+0000 7f25f601b640 1 -- 172.21.15.40:0/3393743313 wait complete. 2024-04-21T23:16:19.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:19.583+0000 7f25f601b640 1 Processor -- start 2024-04-21T23:16:19.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:19.583+0000 7f25f601b640 1 -- start start 2024-04-21T23:16:19.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:19.583+0000 7f25f601b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25f0102200 0x7f25f0101b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:19.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:19.583+0000 7f25f601b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25f0104b30 0x7f25f01001b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:19.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:19.583+0000 7f25f601b640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25f0107460 0x7f25f01006f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:19.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:19.583+0000 7f25f601b640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f25f010d420 con 0x7f25f0102200 2024-04-21T23:16:19.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:19.583+0000 7f25f601b640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f25f010d2a0 con 0x7f25f0107460 2024-04-21T23:16:19.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:19.583+0000 7f25f601b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f25f010d5a0 con 0x7f25f0104b30 2024-04-21T23:16:19.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:19.583+0000 7f25ef7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25f0102200 0x7f25f0101b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:19.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:19.583+0000 7f25ef7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25f0102200 0x7f25f0101b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35574/0 (socket says 172.21.15.40:35574) 2024-04-21T23:16:19.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:19.583+0000 7f25ef7fe640 1 -- 172.21.15.40:0/2432078202 learned_addr learned my addr 172.21.15.40:0/2432078202 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:16:19.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:19.583+0000 7f25effff640 1 --2- 172.21.15.40:0/2432078202 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25f0107460 0x7f25f01006f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:19.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:19.584+0000 7f25eeffd640 1 --2- 172.21.15.40:0/2432078202 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25f0104b30 0x7f25f01001b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:19.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:19.584+0000 7f25ef7fe640 1 -- 172.21.15.40:0/2432078202 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25f0104b30 msgr2=0x7f25f01001b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:19.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:19.584+0000 7f25ef7fe640 1 --2- 172.21.15.40:0/2432078202 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25f0104b30 0x7f25f01001b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:19.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:19.584+0000 7f25ef7fe640 1 -- 172.21.15.40:0/2432078202 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25f0107460 msgr2=0x7f25f01006f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:19.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:19.584+0000 7f25ef7fe640 1 --2- 172.21.15.40:0/2432078202 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25f0107460 0x7f25f01006f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:19.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:19.584+0000 7f25ef7fe640 1 -- 172.21.15.40:0/2432078202 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25f0100f80 con 0x7f25f0102200 2024-04-21T23:16:19.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:19.584+0000 7f25eeffd640 1 --2- 172.21.15.40:0/2432078202 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25f0104b30 0x7f25f01001b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:16:19.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:19.584+0000 7f25ef7fe640 1 --2- 172.21.15.40:0/2432078202 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25f0102200 0x7f25f0101b70 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f25e000d950 tx=0x7f25e000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:19.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:19.584+0000 7f25ecff9640 1 -- 172.21.15.40:0/2432078202 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25e0017070 con 0x7f25f0102200 2024-04-21T23:16:19.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:19.584+0000 7f25ecff9640 1 -- 172.21.15.40:0/2432078202 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25e0009c30 con 0x7f25f0102200 2024-04-21T23:16:19.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:19.584+0000 7f25f601b640 1 -- 172.21.15.40:0/2432078202 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25f01a1be0 con 0x7f25f0102200 2024-04-21T23:16:19.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:19.584+0000 7f25ecff9640 1 -- 172.21.15.40:0/2432078202 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25e000f6b0 con 0x7f25f0102200 2024-04-21T23:16:19.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:19.584+0000 7f25f601b640 1 -- 172.21.15.40:0/2432078202 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25f01a2130 con 0x7f25f0102200 2024-04-21T23:16:19.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:19.585+0000 7f25f601b640 1 -- 172.21.15.40:0/2432078202 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25f00fd6e0 con 0x7f25f0102200 2024-04-21T23:16:19.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:19.586+0000 7f25ecff9640 1 -- 172.21.15.40:0/2432078202 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f25e00109d0 con 0x7f25f0102200 2024-04-21T23:16:19.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:19.587+0000 7f25ecff9640 1 --2- 172.21.15.40:0/2432078202 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f25d80a2b90 0x7f25d80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:19.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:19.587+0000 7f25eeffd640 1 --2- 172.21.15.40:0/2432078202 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f25d80a2b90 0x7f25d80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:19.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:19.587+0000 7f25ecff9640 1 -- 172.21.15.40:0/2432078202 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f25e00cef10 con 0x7f25f0102200 2024-04-21T23:16:19.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:19.587+0000 7f25eeffd640 1 --2- 172.21.15.40:0/2432078202 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f25d80a2b90 0x7f25d80a5050 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f25f01018e0 tx=0x7f25e4005e50 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:19.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:19.591+0000 7f25ecff9640 1 -- 172.21.15.40:0/2432078202 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f25e0009de0 con 0x7f25f0102200 2024-04-21T23:16:19.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:19.815+0000 7f25f601b640 1 -- 172.21.15.40:0/2432078202 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) -- 0x7f25f00623e0 con 0x7f25f0102200 2024-04-21T23:16:19.931 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:19 smithi040 ceph-mon[27984]: pgmap v304: 129 pgs: 129 active+clean; 581 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-21T23:16:19.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:19 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/898173547' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:16:19.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:19 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/187166450' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-21T23:16:19.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:19 smithi174 ceph-mon[31409]: pgmap v304: 129 pgs: 129 active+clean; 581 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-21T23:16:19.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:19 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/898173547' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:16:19.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:19 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/187166450' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-21T23:16:19.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:19 smithi111 ceph-mon[31085]: pgmap v304: 129 pgs: 129 active+clean; 581 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-21T23:16:19.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:19 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/898173547' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:16:19.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:19 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/187166450' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-21T23:16:20.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:20.469+0000 7f25ecff9640 1 -- 172.21.15.40:0/2432078202 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v19) ==== 105+0+0 (secure 0 0 0) 0x7f25e009bc30 con 0x7f25f0102200 2024-04-21T23:16:20.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:20.472+0000 7f25f601b640 1 -- 172.21.15.40:0/2432078202 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f25d80a2b90 msgr2=0x7f25d80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:20.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:20.472+0000 7f25f601b640 1 --2- 172.21.15.40:0/2432078202 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f25d80a2b90 0x7f25d80a5050 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f25f01018e0 tx=0x7f25e4005e50 comp rx=0 tx=0).stop 2024-04-21T23:16:20.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:20.472+0000 7f25f601b640 1 -- 172.21.15.40:0/2432078202 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25f0102200 msgr2=0x7f25f0101b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:20.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:20.472+0000 7f25f601b640 1 --2- 172.21.15.40:0/2432078202 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25f0102200 0x7f25f0101b70 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f25e000d950 tx=0x7f25e000de10 comp rx=0 tx=0).stop 2024-04-21T23:16:20.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:20.472+0000 7f25f601b640 1 -- 172.21.15.40:0/2432078202 shutdown_connections 2024-04-21T23:16:20.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:20.472+0000 7f25f601b640 1 --2- 172.21.15.40:0/2432078202 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f25d80a2b90 0x7f25d80a5050 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:20.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:20.472+0000 7f25f601b640 1 --2- 172.21.15.40:0/2432078202 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25f0107460 0x7f25f01006f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:20.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:20.472+0000 7f25f601b640 1 --2- 172.21.15.40:0/2432078202 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25f0104b30 0x7f25f01001b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:20.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:20.472+0000 7f25f601b640 1 --2- 172.21.15.40:0/2432078202 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25f0102200 0x7f25f0101b70 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:20.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:20.472+0000 7f25f601b640 1 -- 172.21.15.40:0/2432078202 >> 172.21.15.40:0/2432078202 conn(0x7f25f00fbbf0 msgr2=0x7f25f0102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:20.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:20.473+0000 7f25f601b640 1 -- 172.21.15.40:0/2432078202 shutdown_connections 2024-04-21T23:16:20.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:20.473+0000 7f25f601b640 1 -- 172.21.15.40:0/2432078202 wait complete. 2024-04-21T23:16:20.488 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-04-21T23:16:20.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:20.600+0000 7fd23f2a7640 1 -- 172.21.15.40:0/2286163505 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd238101890 msgr2=0x7fd238101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:20.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:20.600+0000 7fd23f2a7640 1 --2- 172.21.15.40:0/2286163505 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd238101890 0x7fd238101d10 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fd228009900 tx=0x7fd22802bc30 comp rx=0 tx=0).stop 2024-04-21T23:16:20.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:20.600+0000 7fd23f2a7640 1 -- 172.21.15.40:0/2286163505 shutdown_connections 2024-04-21T23:16:20.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:20.600+0000 7fd23f2a7640 1 --2- 172.21.15.40:0/2286163505 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd238102250 0x7fd238108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:20.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:20.600+0000 7fd23f2a7640 1 --2- 172.21.15.40:0/2286163505 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd238101890 0x7fd238101d10 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:20.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:20.600+0000 7fd23f2a7640 1 --2- 172.21.15.40:0/2286163505 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2381006a0 0x7fd238100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:20.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:20.600+0000 7fd23f2a7640 1 -- 172.21.15.40:0/2286163505 >> 172.21.15.40:0/2286163505 conn(0x7fd2380fbc30 msgr2=0x7fd2380fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:20.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:20.600+0000 7fd23f2a7640 1 -- 172.21.15.40:0/2286163505 shutdown_connections 2024-04-21T23:16:20.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:20.600+0000 7fd23f2a7640 1 -- 172.21.15.40:0/2286163505 wait complete. 2024-04-21T23:16:20.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:20.601+0000 7fd23f2a7640 1 Processor -- start 2024-04-21T23:16:20.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:20.601+0000 7fd23f2a7640 1 -- start start 2024-04-21T23:16:20.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:20.601+0000 7fd23f2a7640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2381006a0 0x7fd23810b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:20.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:20.601+0000 7fd23f2a7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd238101890 0x7fd238109660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:20.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:20.601+0000 7fd23f2a7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd238102250 0x7fd238109ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:20.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:20.601+0000 7fd23f2a7640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd23810d440 con 0x7fd238101890 2024-04-21T23:16:20.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:20.601+0000 7fd23f2a7640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd23810d2c0 con 0x7fd2381006a0 2024-04-21T23:16:20.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:20.601+0000 7fd23f2a7640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd23810d5c0 con 0x7fd238102250 2024-04-21T23:16:20.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:20.601+0000 7fd23c842640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd238101890 0x7fd238109660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:20.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:20.601+0000 7fd23c842640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd238101890 0x7fd238109660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35592/0 (socket says 172.21.15.40:35592) 2024-04-21T23:16:20.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:20.601+0000 7fd23c842640 1 -- 172.21.15.40:0/3229170185 learned_addr learned my addr 172.21.15.40:0/3229170185 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:16:20.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:20.601+0000 7fd23d043640 1 --2- 172.21.15.40:0/3229170185 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2381006a0 0x7fd23810b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:20.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:20.601+0000 7fd23d844640 1 --2- 172.21.15.40:0/3229170185 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd238102250 0x7fd238109ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:20.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:20.601+0000 7fd23c842640 1 -- 172.21.15.40:0/3229170185 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd238102250 msgr2=0x7fd238109ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:20.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:20.601+0000 7fd23c842640 1 --2- 172.21.15.40:0/3229170185 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd238102250 0x7fd238109ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:20.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:20.601+0000 7fd23c842640 1 -- 172.21.15.40:0/3229170185 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2381006a0 msgr2=0x7fd23810b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:20.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:20.601+0000 7fd23c842640 1 --2- 172.21.15.40:0/3229170185 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2381006a0 0x7fd23810b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:20.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:20.601+0000 7fd23c842640 1 -- 172.21.15.40:0/3229170185 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd23810a430 con 0x7fd238101890 2024-04-21T23:16:20.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:20.601+0000 7fd23d043640 1 --2- 172.21.15.40:0/3229170185 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2381006a0 0x7fd23810b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:16:20.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:20.601+0000 7fd23d844640 1 --2- 172.21.15.40:0/3229170185 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd238102250 0x7fd238109ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:16:20.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:20.601+0000 7fd23c842640 1 --2- 172.21.15.40:0/3229170185 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd238101890 0x7fd238109660 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fd228009230 tx=0x7fd228005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:20.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:20.602+0000 7fd22e7fc640 1 -- 172.21.15.40:0/3229170185 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd228041070 con 0x7fd238101890 2024-04-21T23:16:20.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:20.602+0000 7fd23f2a7640 1 -- 172.21.15.40:0/3229170185 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2381a1bb0 con 0x7fd238101890 2024-04-21T23:16:20.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:20.602+0000 7fd22e7fc640 1 -- 172.21.15.40:0/3229170185 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd2280054e0 con 0x7fd238101890 2024-04-21T23:16:20.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:20.602+0000 7fd22e7fc640 1 -- 172.21.15.40:0/3229170185 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd228034440 con 0x7fd238101890 2024-04-21T23:16:20.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:20.602+0000 7fd23f2a7640 1 -- 172.21.15.40:0/3229170185 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2381a2150 con 0x7fd238101890 2024-04-21T23:16:20.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:20.603+0000 7fd23f2a7640 1 -- 172.21.15.40:0/3229170185 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd238107a80 con 0x7fd238101890 2024-04-21T23:16:20.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:20.604+0000 7fd22e7fc640 1 -- 172.21.15.40:0/3229170185 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fd228044070 con 0x7fd238101890 2024-04-21T23:16:20.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:20.604+0000 7fd22e7fc640 1 --2- 172.21.15.40:0/3229170185 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fd2240a2b90 0x7fd2240a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:20.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:20.604+0000 7fd23d043640 1 --2- 172.21.15.40:0/3229170185 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fd2240a2b90 0x7fd2240a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:20.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:20.604+0000 7fd22e7fc640 1 -- 172.21.15.40:0/3229170185 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fd2280f1020 con 0x7fd238101890 2024-04-21T23:16:20.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:20.605+0000 7fd23d043640 1 --2- 172.21.15.40:0/3229170185 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fd2240a2b90 0x7fd2240a5050 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd2340058c0 tx=0x7fd234005850 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:20.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:20.609+0000 7fd22e7fc640 1 -- 172.21.15.40:0/3229170185 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fd2280bdd40 con 0x7fd238101890 2024-04-21T23:16:20.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:20.835+0000 7fd23f2a7640 1 -- 172.21.15.40:0/3229170185 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7fd238062340 con 0x7fd238101890 2024-04-21T23:16:20.931 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:20 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/187166450' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-21T23:16:20.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:20 smithi040 ceph-mon[27984]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T23:16:20.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:20 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/2432078202' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-21T23:16:20.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:20 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/187166450' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-21T23:16:20.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:20 smithi174 ceph-mon[31409]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T23:16:20.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:20 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/2432078202' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-21T23:16:20.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:20 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/187166450' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-21T23:16:20.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:20 smithi111 ceph-mon[31085]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T23:16:20.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:20 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/2432078202' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-21T23:16:21.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:21.478+0000 7fd22e7fc640 1 -- 172.21.15.40:0/3229170185 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v21) ==== 121+0+0 (secure 0 0 0) 0x7fd2280b9020 con 0x7fd238101890 2024-04-21T23:16:21.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:21.480+0000 7fd23f2a7640 1 -- 172.21.15.40:0/3229170185 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fd2240a2b90 msgr2=0x7fd2240a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:21.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:21.480+0000 7fd23f2a7640 1 --2- 172.21.15.40:0/3229170185 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fd2240a2b90 0x7fd2240a5050 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd2340058c0 tx=0x7fd234005850 comp rx=0 tx=0).stop 2024-04-21T23:16:21.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:21.481+0000 7fd23f2a7640 1 -- 172.21.15.40:0/3229170185 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd238101890 msgr2=0x7fd238109660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:21.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:21.481+0000 7fd23f2a7640 1 --2- 172.21.15.40:0/3229170185 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd238101890 0x7fd238109660 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fd228009230 tx=0x7fd228005dd0 comp rx=0 tx=0).stop 2024-04-21T23:16:21.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:21.481+0000 7fd23f2a7640 1 -- 172.21.15.40:0/3229170185 shutdown_connections 2024-04-21T23:16:21.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:21.481+0000 7fd23f2a7640 1 --2- 172.21.15.40:0/3229170185 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fd2240a2b90 0x7fd2240a5050 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:21.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:21.481+0000 7fd23f2a7640 1 --2- 172.21.15.40:0/3229170185 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd238102250 0x7fd238109ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:21.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:21.481+0000 7fd23f2a7640 1 --2- 172.21.15.40:0/3229170185 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd238101890 0x7fd238109660 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:21.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:21.481+0000 7fd23f2a7640 1 --2- 172.21.15.40:0/3229170185 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2381006a0 0x7fd23810b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:21.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:21.481+0000 7fd23f2a7640 1 -- 172.21.15.40:0/3229170185 >> 172.21.15.40:0/3229170185 conn(0x7fd2380fbc30 msgr2=0x7fd238104c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:21.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:21.481+0000 7fd23f2a7640 1 -- 172.21.15.40:0/3229170185 shutdown_connections 2024-04-21T23:16:21.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:21.481+0000 7fd23f2a7640 1 -- 172.21.15.40:0/3229170185 wait complete. 2024-04-21T23:16:21.496 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-04-21T23:16:21.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:21.604+0000 7f9a14696640 1 -- 172.21.15.40:0/1331985435 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a0c107460 msgr2=0x7f9a0c1098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:21.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:21.604+0000 7f9a14696640 1 --2- 172.21.15.40:0/1331985435 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a0c107460 0x7f9a0c1098b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9a08009be0 tx=0x7f9a0802bef0 comp rx=0 tx=0).stop 2024-04-21T23:16:21.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:21.605+0000 7f9a14696640 1 -- 172.21.15.40:0/1331985435 shutdown_connections 2024-04-21T23:16:21.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:21.605+0000 7f9a14696640 1 --2- 172.21.15.40:0/1331985435 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a0c107460 0x7f9a0c1098b0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:21.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:21.605+0000 7f9a14696640 1 --2- 172.21.15.40:0/1331985435 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9a0c104b30 0x7f9a0c106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:21.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:21.605+0000 7f9a14696640 1 --2- 172.21.15.40:0/1331985435 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a0c102200 0x7f9a0c1045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:21.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:21.605+0000 7f9a14696640 1 -- 172.21.15.40:0/1331985435 >> 172.21.15.40:0/1331985435 conn(0x7f9a0c0fbbf0 msgr2=0x7f9a0c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:21.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:21.605+0000 7f9a14696640 1 -- 172.21.15.40:0/1331985435 shutdown_connections 2024-04-21T23:16:21.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:21.605+0000 7f9a14696640 1 -- 172.21.15.40:0/1331985435 wait complete. 2024-04-21T23:16:21.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:21.605+0000 7f9a14696640 1 Processor -- start 2024-04-21T23:16:21.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:21.605+0000 7f9a14696640 1 -- start start 2024-04-21T23:16:21.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:21.605+0000 7f9a14696640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a0c102200 0x7f9a0c197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:21.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:21.605+0000 7f9a14696640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a0c104b30 0x7f9a0c197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:21.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:21.605+0000 7f9a14696640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9a0c107460 0x7f9a0c19f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:21.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:21.605+0000 7f9a14696640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9a0c10d400 con 0x7f9a0c104b30 2024-04-21T23:16:21.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:21.605+0000 7f9a14696640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f9a0c10d280 con 0x7f9a0c102200 2024-04-21T23:16:21.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:21.605+0000 7f9a14696640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9a0c10d580 con 0x7f9a0c107460 2024-04-21T23:16:21.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:21.605+0000 7f9a11c31640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a0c104b30 0x7f9a0c197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:21.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:21.605+0000 7f9a11c31640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a0c104b30 0x7f9a0c197fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35610/0 (socket says 172.21.15.40:35610) 2024-04-21T23:16:21.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:21.606+0000 7f9a11c31640 1 -- 172.21.15.40:0/512463810 learned_addr learned my addr 172.21.15.40:0/512463810 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:16:21.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:21.606+0000 7f9a12432640 1 --2- 172.21.15.40:0/512463810 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a0c102200 0x7f9a0c197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:21.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:21.606+0000 7f9a12c33640 1 --2- 172.21.15.40:0/512463810 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9a0c107460 0x7f9a0c19f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:21.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:21.606+0000 7f9a11c31640 1 -- 172.21.15.40:0/512463810 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9a0c107460 msgr2=0x7f9a0c19f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:21.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:21.606+0000 7f9a11c31640 1 --2- 172.21.15.40:0/512463810 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9a0c107460 0x7f9a0c19f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:21.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:21.606+0000 7f9a11c31640 1 -- 172.21.15.40:0/512463810 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a0c102200 msgr2=0x7f9a0c197a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:21.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:21.606+0000 7f9a11c31640 1 --2- 172.21.15.40:0/512463810 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a0c102200 0x7f9a0c197a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:21.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:21.606+0000 7f9a11c31640 1 -- 172.21.15.40:0/512463810 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a0c19f6a0 con 0x7f9a0c104b30 2024-04-21T23:16:21.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:21.606+0000 7f9a12c33640 1 --2- 172.21.15.40:0/512463810 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9a0c107460 0x7f9a0c19f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:16:21.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:21.606+0000 7f9a12432640 1 --2- 172.21.15.40:0/512463810 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a0c102200 0x7f9a0c197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:16:21.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:21.606+0000 7f9a11c31640 1 --2- 172.21.15.40:0/512463810 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a0c104b30 0x7f9a0c197fc0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f99fc00d770 tx=0x7f99fc00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:21.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:21.606+0000 7f9a037fe640 1 -- 172.21.15.40:0/512463810 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99fc017070 con 0x7f9a0c104b30 2024-04-21T23:16:21.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:21.606+0000 7f9a14696640 1 -- 172.21.15.40:0/512463810 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a0c19f990 con 0x7f9a0c104b30 2024-04-21T23:16:21.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:21.606+0000 7f9a037fe640 1 -- 172.21.15.40:0/512463810 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f99fc009c30 con 0x7f9a0c104b30 2024-04-21T23:16:21.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:21.606+0000 7f9a037fe640 1 -- 172.21.15.40:0/512463810 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99fc00f620 con 0x7f9a0c104b30 2024-04-21T23:16:21.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:21.606+0000 7f9a14696640 1 -- 172.21.15.40:0/512463810 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a0c19fee0 con 0x7f9a0c104b30 2024-04-21T23:16:21.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:21.607+0000 7f9a14696640 1 -- 172.21.15.40:0/512463810 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a0c0fd6e0 con 0x7f9a0c104b30 2024-04-21T23:16:21.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:21.608+0000 7f9a037fe640 1 -- 172.21.15.40:0/512463810 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f99fc01a070 con 0x7f9a0c104b30 2024-04-21T23:16:21.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:21.609+0000 7f9a037fe640 1 --2- 172.21.15.40:0/512463810 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f99f80a2b90 0x7f99f80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:21.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:21.609+0000 7f9a12432640 1 --2- 172.21.15.40:0/512463810 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f99f80a2b90 0x7f99f80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:21.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:21.609+0000 7f9a037fe640 1 -- 172.21.15.40:0/512463810 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f99fc0ced80 con 0x7f9a0c104b30 2024-04-21T23:16:21.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:21.609+0000 7f9a12432640 1 --2- 172.21.15.40:0/512463810 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f99f80a2b90 0x7f99f80a5050 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9a040096c0 tx=0x7f9a04009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:21.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:21.612+0000 7f9a037fe640 1 -- 172.21.15.40:0/512463810 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f99fc009de0 con 0x7f9a0c104b30 2024-04-21T23:16:21.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:21.839+0000 7f9a14696640 1 -- 172.21.15.40:0/512463810 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f9a0c02d170 con 0x7f9a0c104b30 2024-04-21T23:16:21.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:21 smithi040 ceph-mon[27984]: pgmap v305: 129 pgs: 129 active+clean; 581 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-21T23:16:21.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:21 smithi040 ceph-mon[27984]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-21T23:16:21.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:21 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/2432078202' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-21T23:16:21.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:21 smithi040 ceph-mon[27984]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-21T23:16:21.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:21 smithi040 ceph-mon[27984]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T23:16:21.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:21 smithi040 ceph-mon[27984]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-04-21T23:16:21.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:21 smithi040 ceph-mon[27984]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-04-21T23:16:21.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:21 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/3229170185' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-21T23:16:21.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:21 smithi174 ceph-mon[31409]: pgmap v305: 129 pgs: 129 active+clean; 581 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-21T23:16:21.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:21 smithi174 ceph-mon[31409]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-21T23:16:21.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:21 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/2432078202' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-21T23:16:21.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:21 smithi174 ceph-mon[31409]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-21T23:16:21.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:21 smithi174 ceph-mon[31409]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T23:16:21.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:21 smithi174 ceph-mon[31409]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-04-21T23:16:21.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:21 smithi174 ceph-mon[31409]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-04-21T23:16:21.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:21 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/3229170185' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-21T23:16:21.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:21 smithi111 ceph-mon[31085]: pgmap v305: 129 pgs: 129 active+clean; 581 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-21T23:16:21.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:21 smithi111 ceph-mon[31085]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-21T23:16:21.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:21 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/2432078202' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-21T23:16:21.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:21 smithi111 ceph-mon[31085]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-21T23:16:21.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:21 smithi111 ceph-mon[31085]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T23:16:21.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:21 smithi111 ceph-mon[31085]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-04-21T23:16:21.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:21 smithi111 ceph-mon[31085]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-04-21T23:16:21.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:21 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/3229170185' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-21T23:16:22.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.493+0000 7f9a037fe640 1 -- 172.21.15.40:0/512463810 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v23) ==== 115+0+0 (secure 0 0 0) 0x7f99fc09baa0 con 0x7f9a0c104b30 2024-04-21T23:16:22.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.496+0000 7f9a14696640 1 -- 172.21.15.40:0/512463810 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f99f80a2b90 msgr2=0x7f99f80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:22.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.496+0000 7f9a14696640 1 --2- 172.21.15.40:0/512463810 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f99f80a2b90 0x7f99f80a5050 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9a040096c0 tx=0x7f9a04009260 comp rx=0 tx=0).stop 2024-04-21T23:16:22.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.496+0000 7f9a14696640 1 -- 172.21.15.40:0/512463810 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a0c104b30 msgr2=0x7f9a0c197fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:22.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.496+0000 7f9a14696640 1 --2- 172.21.15.40:0/512463810 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a0c104b30 0x7f9a0c197fc0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f99fc00d770 tx=0x7f99fc00dc30 comp rx=0 tx=0).stop 2024-04-21T23:16:22.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.496+0000 7f9a14696640 1 -- 172.21.15.40:0/512463810 shutdown_connections 2024-04-21T23:16:22.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.496+0000 7f9a14696640 1 --2- 172.21.15.40:0/512463810 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f99f80a2b90 0x7f99f80a5050 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:22.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.496+0000 7f9a14696640 1 --2- 172.21.15.40:0/512463810 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9a0c107460 0x7f9a0c19f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:22.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.496+0000 7f9a14696640 1 --2- 172.21.15.40:0/512463810 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a0c104b30 0x7f9a0c197fc0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:22.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.496+0000 7f9a14696640 1 --2- 172.21.15.40:0/512463810 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a0c102200 0x7f9a0c197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:22.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.496+0000 7f9a14696640 1 -- 172.21.15.40:0/512463810 >> 172.21.15.40:0/512463810 conn(0x7f9a0c0fbbf0 msgr2=0x7f9a0c102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:22.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.496+0000 7f9a14696640 1 -- 172.21.15.40:0/512463810 shutdown_connections 2024-04-21T23:16:22.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.497+0000 7f9a14696640 1 -- 172.21.15.40:0/512463810 wait complete. 2024-04-21T23:16:22.510 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2} subvols for filesystem 'cephfs' 2024-04-21T23:16:22.510 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-04-21T23:16:22.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.622+0000 7fef34d1e640 1 -- 172.21.15.40:0/104247552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef301006a0 msgr2=0x7fef30100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:22.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.622+0000 7fef34d1e640 1 --2- 172.21.15.40:0/104247552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef301006a0 0x7fef30100ac0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fef20009900 tx=0x7fef2002bc30 comp rx=0 tx=0).stop 2024-04-21T23:16:22.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.622+0000 7fef34d1e640 1 -- 172.21.15.40:0/104247552 shutdown_connections 2024-04-21T23:16:22.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.622+0000 7fef34d1e640 1 --2- 172.21.15.40:0/104247552 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fef30102250 0x7fef30108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:22.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.622+0000 7fef34d1e640 1 --2- 172.21.15.40:0/104247552 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef30101890 0x7fef30101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:22.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.622+0000 7fef34d1e640 1 --2- 172.21.15.40:0/104247552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef301006a0 0x7fef30100ac0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:22.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.622+0000 7fef34d1e640 1 -- 172.21.15.40:0/104247552 >> 172.21.15.40:0/104247552 conn(0x7fef300fbc30 msgr2=0x7fef300fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:22.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.622+0000 7fef34d1e640 1 -- 172.21.15.40:0/104247552 shutdown_connections 2024-04-21T23:16:22.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.622+0000 7fef34d1e640 1 -- 172.21.15.40:0/104247552 wait complete. 2024-04-21T23:16:22.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.623+0000 7fef34d1e640 1 Processor -- start 2024-04-21T23:16:22.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.623+0000 7fef34d1e640 1 -- start start 2024-04-21T23:16:22.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.623+0000 7fef34d1e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef301006a0 0x7fef3006a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:22.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.623+0000 7fef34d1e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fef30101890 0x7fef3019c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:22.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.623+0000 7fef34d1e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef30102250 0x7fef3019ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:22.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.623+0000 7fef34d1e640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fef3010c320 con 0x7fef30102250 2024-04-21T23:16:22.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.623+0000 7fef34d1e640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fef3010c1a0 con 0x7fef30101890 2024-04-21T23:16:22.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.623+0000 7fef34d1e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fef3010c4a0 con 0x7fef301006a0 2024-04-21T23:16:22.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.623+0000 7fef2ed9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef30102250 0x7fef3019ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:22.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.623+0000 7fef2ed9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef30102250 0x7fef3019ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43382/0 (socket says 172.21.15.40:43382) 2024-04-21T23:16:22.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.623+0000 7fef2ed9d640 1 -- 172.21.15.40:0/776096642 learned_addr learned my addr 172.21.15.40:0/776096642 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:16:22.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.623+0000 7fef2e59c640 1 --2- 172.21.15.40:0/776096642 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef301006a0 0x7fef3006a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:22.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.623+0000 7fef2dd9b640 1 --2- 172.21.15.40:0/776096642 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fef30101890 0x7fef3019c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:22.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.623+0000 7fef2ed9d640 1 -- 172.21.15.40:0/776096642 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef301006a0 msgr2=0x7fef3006a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:22.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.623+0000 7fef2ed9d640 1 --2- 172.21.15.40:0/776096642 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef301006a0 0x7fef3006a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:22.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.623+0000 7fef2ed9d640 1 -- 172.21.15.40:0/776096642 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fef30101890 msgr2=0x7fef3019c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:22.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.623+0000 7fef2ed9d640 1 --2- 172.21.15.40:0/776096642 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fef30101890 0x7fef3019c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:22.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.623+0000 7fef2ed9d640 1 -- 172.21.15.40:0/776096642 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef3019f720 con 0x7fef30102250 2024-04-21T23:16:22.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.624+0000 7fef2ed9d640 1 --2- 172.21.15.40:0/776096642 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef30102250 0x7fef3019ee60 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fef2800eb10 tx=0x7fef2800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:22.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.624+0000 7fef1f7fe640 1 -- 172.21.15.40:0/776096642 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef28004b20 con 0x7fef30102250 2024-04-21T23:16:22.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.624+0000 7fef1f7fe640 1 -- 172.21.15.40:0/776096642 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fef28004cc0 con 0x7fef30102250 2024-04-21T23:16:22.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.624+0000 7fef34d1e640 1 -- 172.21.15.40:0/776096642 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef3019fa10 con 0x7fef30102250 2024-04-21T23:16:22.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.624+0000 7fef1f7fe640 1 -- 172.21.15.40:0/776096642 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef28010760 con 0x7fef30102250 2024-04-21T23:16:22.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.624+0000 7fef34d1e640 1 -- 172.21.15.40:0/776096642 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef3019ff10 con 0x7fef30102250 2024-04-21T23:16:22.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.625+0000 7fef34d1e640 1 -- 172.21.15.40:0/776096642 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef30107a60 con 0x7fef30102250 2024-04-21T23:16:22.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.626+0000 7fef1f7fe640 1 -- 172.21.15.40:0/776096642 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fef28005020 con 0x7fef30102250 2024-04-21T23:16:22.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.627+0000 7fef1f7fe640 1 --2- 172.21.15.40:0/776096642 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fef180a2b90 0x7fef180a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:22.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.627+0000 7fef2e59c640 1 --2- 172.21.15.40:0/776096642 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fef180a2b90 0x7fef180a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:22.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.627+0000 7fef1f7fe640 1 -- 172.21.15.40:0/776096642 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fef280cf830 con 0x7fef30102250 2024-04-21T23:16:22.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.628+0000 7fef2e59c640 1 --2- 172.21.15.40:0/776096642 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fef180a2b90 0x7fef180a5050 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fef20000c00 tx=0x7fef20005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:22.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.632+0000 7fef1f7fe640 1 -- 172.21.15.40:0/776096642 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fef2809c550 con 0x7fef30102250 2024-04-21T23:16:22.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.841+0000 7fef34d1e640 1 -- 172.21.15.40:0/776096642 --> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fef30061b80 con 0x7fef180a2b90 2024-04-21T23:16:22.847 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:22 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/3229170185' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-21T23:16:22.847 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:22 smithi040 ceph-mon[27984]: mds.? [v2:172.21.15.174:6838/2467509274,v1:172.21.15.174:6839/2467509274] up:active 2024-04-21T23:16:22.847 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:22 smithi040 ceph-mon[27984]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-21T23:16:22.847 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:22 smithi040 ceph-mon[27984]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-21T23:16:22.847 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:22 smithi040 ceph-mon[27984]: Cluster is now healthy 2024-04-21T23:16:22.847 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:22 smithi040 ceph-mon[27984]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-04-21T23:16:22.847 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:22 smithi040 ceph-mon[27984]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-04-21T23:16:22.847 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:22 smithi040 ceph-mon[27984]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-04-21T23:16:22.847 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:22 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/512463810' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-21T23:16:22.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.864+0000 7fef1f7fe640 1 -- 172.21.15.40:0/776096642 <== mgr.14148 v2:172.21.15.40:6800/2500760788 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fef30061b80 con 0x7fef180a2b90 2024-04-21T23:16:22.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.866+0000 7fef34d1e640 1 -- 172.21.15.40:0/776096642 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fef180a2b90 msgr2=0x7fef180a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:22.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.866+0000 7fef34d1e640 1 --2- 172.21.15.40:0/776096642 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fef180a2b90 0x7fef180a5050 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fef20000c00 tx=0x7fef20005fd0 comp rx=0 tx=0).stop 2024-04-21T23:16:22.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.866+0000 7fef34d1e640 1 -- 172.21.15.40:0/776096642 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef30102250 msgr2=0x7fef3019ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:22.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.866+0000 7fef34d1e640 1 --2- 172.21.15.40:0/776096642 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef30102250 0x7fef3019ee60 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fef2800eb10 tx=0x7fef2800c2d0 comp rx=0 tx=0).stop 2024-04-21T23:16:22.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.866+0000 7fef34d1e640 1 -- 172.21.15.40:0/776096642 shutdown_connections 2024-04-21T23:16:22.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.866+0000 7fef34d1e640 1 --2- 172.21.15.40:0/776096642 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fef180a2b90 0x7fef180a5050 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:22.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.866+0000 7fef34d1e640 1 --2- 172.21.15.40:0/776096642 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fef30101890 0x7fef3019c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:22.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.866+0000 7fef34d1e640 1 --2- 172.21.15.40:0/776096642 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef30102250 0x7fef3019ee60 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:22.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.866+0000 7fef34d1e640 1 --2- 172.21.15.40:0/776096642 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef301006a0 0x7fef3006a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:22.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.866+0000 7fef34d1e640 1 -- 172.21.15.40:0/776096642 >> 172.21.15.40:0/776096642 conn(0x7fef300fbc30 msgr2=0x7fef30104c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:22.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.866+0000 7fef34d1e640 1 -- 172.21.15.40:0/776096642 shutdown_connections 2024-04-21T23:16:22.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.866+0000 7fef34d1e640 1 -- 172.21.15.40:0/776096642 wait complete. 2024-04-21T23:16:22.876 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-04-21T23:16:22.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:22 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/3229170185' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-21T23:16:22.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:22 smithi174 ceph-mon[31409]: mds.? [v2:172.21.15.174:6838/2467509274,v1:172.21.15.174:6839/2467509274] up:active 2024-04-21T23:16:22.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:22 smithi174 ceph-mon[31409]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-21T23:16:22.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:22 smithi174 ceph-mon[31409]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-21T23:16:22.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:22 smithi174 ceph-mon[31409]: Cluster is now healthy 2024-04-21T23:16:22.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:22 smithi174 ceph-mon[31409]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-04-21T23:16:22.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:22 smithi174 ceph-mon[31409]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-04-21T23:16:22.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:22 smithi174 ceph-mon[31409]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-04-21T23:16:22.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:22 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/512463810' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-21T23:16:22.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:22 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/3229170185' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-21T23:16:22.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:22 smithi111 ceph-mon[31085]: mds.? [v2:172.21.15.174:6838/2467509274,v1:172.21.15.174:6839/2467509274] up:active 2024-04-21T23:16:22.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:22 smithi111 ceph-mon[31085]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-21T23:16:22.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:22 smithi111 ceph-mon[31085]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-21T23:16:22.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:22 smithi111 ceph-mon[31085]: Cluster is now healthy 2024-04-21T23:16:22.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:22 smithi111 ceph-mon[31085]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-04-21T23:16:22.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:22 smithi111 ceph-mon[31085]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-04-21T23:16:22.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:22 smithi111 ceph-mon[31085]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-04-21T23:16:22.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:22 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/512463810' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-21T23:16:22.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.983+0000 7fc7788a7640 1 -- 172.21.15.40:0/3659896139 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc774102250 msgr2=0x7fc774108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:22.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.983+0000 7fc7788a7640 1 --2- 172.21.15.40:0/3659896139 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc774102250 0x7fc774108ae0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fc76c009ca0 tx=0x7fc76c02c0f0 comp rx=0 tx=0).stop 2024-04-21T23:16:22.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.983+0000 7fc7788a7640 1 -- 172.21.15.40:0/3659896139 shutdown_connections 2024-04-21T23:16:22.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.983+0000 7fc7788a7640 1 --2- 172.21.15.40:0/3659896139 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc774102250 0x7fc774108ae0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:22.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.983+0000 7fc7788a7640 1 --2- 172.21.15.40:0/3659896139 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc774101890 0x7fc774101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:22.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.983+0000 7fc7788a7640 1 --2- 172.21.15.40:0/3659896139 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7741006a0 0x7fc774100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:22.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.983+0000 7fc7788a7640 1 -- 172.21.15.40:0/3659896139 >> 172.21.15.40:0/3659896139 conn(0x7fc7740fbc30 msgr2=0x7fc7740fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:22.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.983+0000 7fc7788a7640 1 -- 172.21.15.40:0/3659896139 shutdown_connections 2024-04-21T23:16:22.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.983+0000 7fc7788a7640 1 -- 172.21.15.40:0/3659896139 wait complete. 2024-04-21T23:16:22.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.984+0000 7fc7788a7640 1 Processor -- start 2024-04-21T23:16:22.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.984+0000 7fc7788a7640 1 -- start start 2024-04-21T23:16:22.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.984+0000 7fc7788a7640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7741006a0 0x7fc774197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:22.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.984+0000 7fc7788a7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc774101890 0x7fc774198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:22.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.984+0000 7fc7788a7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc774102250 0x7fc77419f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:22.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.984+0000 7fc7788a7640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc77410c320 con 0x7fc774101890 2024-04-21T23:16:22.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.984+0000 7fc7788a7640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc77410c1a0 con 0x7fc7741006a0 2024-04-21T23:16:22.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.984+0000 7fc7788a7640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc77410c4a0 con 0x7fc774102250 2024-04-21T23:16:22.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.984+0000 7fc769d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc774101890 0x7fc774198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:22.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.984+0000 7fc769d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc774101890 0x7fc774198060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43420/0 (socket says 172.21.15.40:43420) 2024-04-21T23:16:22.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.984+0000 7fc769d9b640 1 -- 172.21.15.40:0/2715730459 learned_addr learned my addr 172.21.15.40:0/2715730459 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:16:22.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.984+0000 7fc77259c640 1 --2- 172.21.15.40:0/2715730459 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7741006a0 0x7fc774197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:22.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.985+0000 7fc772d9d640 1 --2- 172.21.15.40:0/2715730459 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc774102250 0x7fc77419f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:22.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.985+0000 7fc769d9b640 1 -- 172.21.15.40:0/2715730459 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc774102250 msgr2=0x7fc77419f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:22.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.985+0000 7fc769d9b640 1 --2- 172.21.15.40:0/2715730459 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc774102250 0x7fc77419f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:22.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.985+0000 7fc769d9b640 1 -- 172.21.15.40:0/2715730459 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7741006a0 msgr2=0x7fc774197b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:22.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.985+0000 7fc769d9b640 1 --2- 172.21.15.40:0/2715730459 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7741006a0 0x7fc774197b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:22.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.985+0000 7fc769d9b640 1 -- 172.21.15.40:0/2715730459 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc77419f740 con 0x7fc774101890 2024-04-21T23:16:22.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.985+0000 7fc769d9b640 1 --2- 172.21.15.40:0/2715730459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc774101890 0x7fc774198060 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fc75c00d770 tx=0x7fc75c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:22.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.985+0000 7fc76bfff640 1 -- 172.21.15.40:0/2715730459 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc75c017070 con 0x7fc774101890 2024-04-21T23:16:22.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.985+0000 7fc76bfff640 1 -- 172.21.15.40:0/2715730459 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc75c005500 con 0x7fc774101890 2024-04-21T23:16:22.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.985+0000 7fc7788a7640 1 -- 172.21.15.40:0/2715730459 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc77419fa30 con 0x7fc774101890 2024-04-21T23:16:22.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.985+0000 7fc76bfff640 1 -- 172.21.15.40:0/2715730459 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc75c004e00 con 0x7fc774101890 2024-04-21T23:16:22.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.985+0000 7fc7788a7640 1 -- 172.21.15.40:0/2715730459 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc77419ff30 con 0x7fc774101890 2024-04-21T23:16:22.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.986+0000 7fc7788a7640 1 -- 172.21.15.40:0/2715730459 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7741a0100 con 0x7fc774101890 2024-04-21T23:16:22.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.987+0000 7fc76bfff640 1 -- 172.21.15.40:0/2715730459 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fc75c004fa0 con 0x7fc774101890 2024-04-21T23:16:22.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.988+0000 7fc76bfff640 1 --2- 172.21.15.40:0/2715730459 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fc7600a2b90 0x7fc7600a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:22.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.988+0000 7fc77259c640 1 --2- 172.21.15.40:0/2715730459 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fc7600a2b90 0x7fc7600a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:22.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.988+0000 7fc76bfff640 1 -- 172.21.15.40:0/2715730459 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fc75c0ceca0 con 0x7fc774101890 2024-04-21T23:16:22.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.989+0000 7fc77259c640 1 --2- 172.21.15.40:0/2715730459 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fc7600a2b90 0x7fc7600a5050 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fc764005fd0 tx=0x7fc76400b410 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:22.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:22.993+0000 7fc76bfff640 1 -- 172.21.15.40:0/2715730459 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fc75c09b9c0 con 0x7fc774101890 2024-04-21T23:16:23.182 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:16:22 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:16:22.847+0000 7f3a38ddd700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T23:16:23.182 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:16:22 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:16:22.847+0000 7f3a38ddd700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T23:16:23.182 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:16:22 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:16:22.847+0000 7f3a38ddd700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T23:16:23.182 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:16:22 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:16:22.847+0000 7f3a38ddd700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T23:16:23.182 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:16:22 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:16:22.847+0000 7f3a38ddd700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T23:16:23.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.204+0000 7fc7788a7640 1 -- 172.21.15.40:0/2715730459 --> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fc7740fddb0 con 0x7fc7600a2b90 2024-04-21T23:16:23.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.227+0000 7fc76bfff640 1 -- 172.21.15.40:0/2715730459 <== mgr.14148 v2:172.21.15.40:6800/2500760788 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fc7740fddb0 con 0x7fc7600a2b90 2024-04-21T23:16:23.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.229+0000 7fc7788a7640 1 -- 172.21.15.40:0/2715730459 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fc7600a2b90 msgr2=0x7fc7600a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:23.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.229+0000 7fc7788a7640 1 --2- 172.21.15.40:0/2715730459 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fc7600a2b90 0x7fc7600a5050 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fc764005fd0 tx=0x7fc76400b410 comp rx=0 tx=0).stop 2024-04-21T23:16:23.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.229+0000 7fc7788a7640 1 -- 172.21.15.40:0/2715730459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc774101890 msgr2=0x7fc774198060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:23.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.229+0000 7fc7788a7640 1 --2- 172.21.15.40:0/2715730459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc774101890 0x7fc774198060 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fc75c00d770 tx=0x7fc75c00dc30 comp rx=0 tx=0).stop 2024-04-21T23:16:23.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.229+0000 7fc7788a7640 1 -- 172.21.15.40:0/2715730459 shutdown_connections 2024-04-21T23:16:23.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.229+0000 7fc7788a7640 1 --2- 172.21.15.40:0/2715730459 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fc7600a2b90 0x7fc7600a5050 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:23.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.229+0000 7fc7788a7640 1 --2- 172.21.15.40:0/2715730459 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc774102250 0x7fc77419f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:23.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.229+0000 7fc7788a7640 1 --2- 172.21.15.40:0/2715730459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc774101890 0x7fc774198060 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:23.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.229+0000 7fc7788a7640 1 --2- 172.21.15.40:0/2715730459 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7741006a0 0x7fc774197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:23.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.229+0000 7fc7788a7640 1 -- 172.21.15.40:0/2715730459 >> 172.21.15.40:0/2715730459 conn(0x7fc7740fbc30 msgr2=0x7fc7740fd6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:23.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.229+0000 7fc7788a7640 1 -- 172.21.15.40:0/2715730459 shutdown_connections 2024-04-21T23:16:23.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.229+0000 7fc7788a7640 1 -- 172.21.15.40:0/2715730459 wait complete. 2024-04-21T23:16:23.239 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-04-21T23:16:23.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.347+0000 7f298eecd640 1 -- 172.21.15.40:0/1949880020 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2988101370 msgr2=0x7f29881017f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:23.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.347+0000 7f298eecd640 1 --2- 172.21.15.40:0/1949880020 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2988101370 0x7f29881017f0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f2980009ca0 tx=0x7f298002c0f0 comp rx=0 tx=0).stop 2024-04-21T23:16:23.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.347+0000 7f298eecd640 1 -- 172.21.15.40:0/1949880020 shutdown_connections 2024-04-21T23:16:23.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.347+0000 7f298eecd640 1 --2- 172.21.15.40:0/1949880020 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2988101370 0x7f29881017f0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:23.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.347+0000 7f298eecd640 1 --2- 172.21.15.40:0/1949880020 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f29881009d0 0x7f2988100e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:23.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.347+0000 7f298eecd640 1 --2- 172.21.15.40:0/1949880020 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2988100070 0x7f2988100490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:23.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.347+0000 7f298eecd640 1 -- 172.21.15.40:0/1949880020 >> 172.21.15.40:0/1949880020 conn(0x7f29880fbc30 msgr2=0x7f29880fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:23.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.347+0000 7f298eecd640 1 -- 172.21.15.40:0/1949880020 shutdown_connections 2024-04-21T23:16:23.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.347+0000 7f298eecd640 1 -- 172.21.15.40:0/1949880020 wait complete. 2024-04-21T23:16:23.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.348+0000 7f298eecd640 1 Processor -- start 2024-04-21T23:16:23.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.348+0000 7f298eecd640 1 -- start start 2024-04-21T23:16:23.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.348+0000 7f298eecd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2988100070 0x7f2988199c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:23.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.348+0000 7f298eecd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f29881009d0 0x7f298819a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:23.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.348+0000 7f298eecd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2988101370 0x7f29881a11f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:23.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.348+0000 7f298eecd640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f298810f610 con 0x7f2988100070 2024-04-21T23:16:23.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.348+0000 7f298eecd640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f298810f490 con 0x7f2988101370 2024-04-21T23:16:23.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.348+0000 7f298eecd640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f298810f790 con 0x7f29881009d0 2024-04-21T23:16:23.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.348+0000 7f298cc69640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2988100070 0x7f2988199c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:23.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.348+0000 7f298cc69640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2988100070 0x7f2988199c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43430/0 (socket says 172.21.15.40:43430) 2024-04-21T23:16:23.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.348+0000 7f298cc69640 1 -- 172.21.15.40:0/2668349782 learned_addr learned my addr 172.21.15.40:0/2668349782 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:16:23.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.348+0000 7f298d46a640 1 --2- 172.21.15.40:0/2668349782 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2988101370 0x7f29881a11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:23.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.348+0000 7f298cc69640 1 -- 172.21.15.40:0/2668349782 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f29881009d0 msgr2=0x7f298819a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:23.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.348+0000 7f2987fff640 1 --2- 172.21.15.40:0/2668349782 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f29881009d0 0x7f298819a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:23.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.348+0000 7f298cc69640 1 --2- 172.21.15.40:0/2668349782 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f29881009d0 0x7f298819a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:23.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.348+0000 7f298cc69640 1 -- 172.21.15.40:0/2668349782 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2988101370 msgr2=0x7f29881a11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:23.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.348+0000 7f298cc69640 1 --2- 172.21.15.40:0/2668349782 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2988101370 0x7f29881a11f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:23.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.348+0000 7f298cc69640 1 -- 172.21.15.40:0/2668349782 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29881a1890 con 0x7f2988100070 2024-04-21T23:16:23.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.349+0000 7f298cc69640 1 --2- 172.21.15.40:0/2668349782 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2988100070 0x7f2988199c80 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f297c00e9f0 tx=0x7f297c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:23.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.349+0000 7f2985ffb640 1 -- 172.21.15.40:0/2668349782 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f297c00cd80 con 0x7f2988100070 2024-04-21T23:16:23.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.349+0000 7f2985ffb640 1 -- 172.21.15.40:0/2668349782 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f297c010910 con 0x7f2988100070 2024-04-21T23:16:23.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.349+0000 7f2985ffb640 1 -- 172.21.15.40:0/2668349782 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f297c018990 con 0x7f2988100070 2024-04-21T23:16:23.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.349+0000 7f298eecd640 1 -- 172.21.15.40:0/2668349782 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29881a1b80 con 0x7f2988100070 2024-04-21T23:16:23.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.349+0000 7f298eecd640 1 -- 172.21.15.40:0/2668349782 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29881a2120 con 0x7f2988100070 2024-04-21T23:16:23.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.350+0000 7f298eecd640 1 -- 172.21.15.40:0/2668349782 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29880689b0 con 0x7f2988100070 2024-04-21T23:16:23.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.351+0000 7f2985ffb640 1 -- 172.21.15.40:0/2668349782 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f297c010ac0 con 0x7f2988100070 2024-04-21T23:16:23.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.352+0000 7f2985ffb640 1 --2- 172.21.15.40:0/2668349782 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f29780a2c10 0x7f29780a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:23.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.352+0000 7f2987fff640 1 --2- 172.21.15.40:0/2668349782 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f29780a2c10 0x7f29780a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:23.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.352+0000 7f2985ffb640 1 -- 172.21.15.40:0/2668349782 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f297c0cf630 con 0x7f2988100070 2024-04-21T23:16:23.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.352+0000 7f2987fff640 1 --2- 172.21.15.40:0/2668349782 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f29780a2c10 0x7f29780a50d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2974005fd0 tx=0x7f2974005850 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:23.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.356+0000 7f2985ffb640 1 -- 172.21.15.40:0/2668349782 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f297c09c350 con 0x7f2988100070 2024-04-21T23:16:23.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.571+0000 7f298eecd640 1 -- 172.21.15.40:0/2668349782 --> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f2988061c10 con 0x7f29780a2c10 2024-04-21T23:16:23.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.576+0000 7f2985ffb640 1 -- 172.21.15.40:0/2668349782 <== mgr.14148 v2:172.21.15.40:6800/2500760788 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f2988061c10 con 0x7f29780a2c10 2024-04-21T23:16:23.576 INFO:teuthology.orchestra.run.smithi040.stdout:/volumes/qa/sv_0/ae1d1b4f-233f-414e-bf03-1a15a95f9e2a 2024-04-21T23:16:23.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.579+0000 7f298eecd640 1 -- 172.21.15.40:0/2668349782 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f29780a2c10 msgr2=0x7f29780a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:23.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.579+0000 7f298eecd640 1 --2- 172.21.15.40:0/2668349782 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f29780a2c10 0x7f29780a50d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2974005fd0 tx=0x7f2974005850 comp rx=0 tx=0).stop 2024-04-21T23:16:23.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.579+0000 7f298eecd640 1 -- 172.21.15.40:0/2668349782 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2988100070 msgr2=0x7f2988199c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:23.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.579+0000 7f298eecd640 1 --2- 172.21.15.40:0/2668349782 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2988100070 0x7f2988199c80 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f297c00e9f0 tx=0x7f297c00eeb0 comp rx=0 tx=0).stop 2024-04-21T23:16:23.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.579+0000 7f298eecd640 1 -- 172.21.15.40:0/2668349782 shutdown_connections 2024-04-21T23:16:23.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.579+0000 7f298eecd640 1 --2- 172.21.15.40:0/2668349782 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f29780a2c10 0x7f29780a50d0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:23.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.579+0000 7f298eecd640 1 --2- 172.21.15.40:0/2668349782 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2988101370 0x7f29881a11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:23.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.579+0000 7f298eecd640 1 --2- 172.21.15.40:0/2668349782 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f29881009d0 0x7f298819a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:23.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.579+0000 7f298eecd640 1 --2- 172.21.15.40:0/2668349782 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2988100070 0x7f2988199c80 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:23.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.579+0000 7f298eecd640 1 -- 172.21.15.40:0/2668349782 >> 172.21.15.40:0/2668349782 conn(0x7f29880fbc30 msgr2=0x7f29880fd730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:23.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.579+0000 7f298eecd640 1 -- 172.21.15.40:0/2668349782 shutdown_connections 2024-04-21T23:16:23.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.579+0000 7f298eecd640 1 -- 172.21.15.40:0/2668349782 wait complete. 2024-04-21T23:16:23.589 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-04-21T23:16:23.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.696+0000 7f5daad02640 1 -- 172.21.15.40:0/2307665462 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5da4102250 msgr2=0x7f5da4108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:23.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.696+0000 7f5daad02640 1 --2- 172.21.15.40:0/2307665462 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5da4102250 0x7f5da4108ae0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f5d9c009ca0 tx=0x7f5d9c02c0f0 comp rx=0 tx=0).stop 2024-04-21T23:16:23.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.696+0000 7f5daad02640 1 -- 172.21.15.40:0/2307665462 shutdown_connections 2024-04-21T23:16:23.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.696+0000 7f5daad02640 1 --2- 172.21.15.40:0/2307665462 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5da4101890 0x7f5da4101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:23.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.696+0000 7f5daad02640 1 --2- 172.21.15.40:0/2307665462 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5da4102250 0x7f5da4108ae0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:23.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.696+0000 7f5daad02640 1 --2- 172.21.15.40:0/2307665462 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5da41006a0 0x7f5da4100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:23.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.696+0000 7f5daad02640 1 -- 172.21.15.40:0/2307665462 >> 172.21.15.40:0/2307665462 conn(0x7f5da40fbc30 msgr2=0x7f5da40fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:23.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.696+0000 7f5daad02640 1 -- 172.21.15.40:0/2307665462 shutdown_connections 2024-04-21T23:16:23.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.697+0000 7f5daad02640 1 -- 172.21.15.40:0/2307665462 wait complete. 2024-04-21T23:16:23.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.697+0000 7f5daad02640 1 Processor -- start 2024-04-21T23:16:23.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.697+0000 7f5daad02640 1 -- start start 2024-04-21T23:16:23.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.697+0000 7f5daad02640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5da41006a0 0x7f5da41046a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:23.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.697+0000 7f5daad02640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5da4101890 0x7f5da4102ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:23.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.697+0000 7f5daad02640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5da4102250 0x7f5da4103220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:23.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.697+0000 7f5daad02640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5da410c320 con 0x7f5da41006a0 2024-04-21T23:16:23.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.697+0000 7f5daad02640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f5da410c1a0 con 0x7f5da4102250 2024-04-21T23:16:23.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.697+0000 7f5daad02640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5da410c4a0 con 0x7f5da4101890 2024-04-21T23:16:23.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.697+0000 7f5da8a9e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5da41006a0 0x7f5da41046a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:23.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.697+0000 7f5da8a9e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5da41006a0 0x7f5da41046a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43452/0 (socket says 172.21.15.40:43452) 2024-04-21T23:16:23.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.697+0000 7f5da8a9e640 1 -- 172.21.15.40:0/3052969194 learned_addr learned my addr 172.21.15.40:0/3052969194 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:16:23.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.698+0000 7f5da3fff640 1 --2- 172.21.15.40:0/3052969194 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5da4101890 0x7f5da4102ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:23.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.698+0000 7f5da929f640 1 --2- 172.21.15.40:0/3052969194 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5da4102250 0x7f5da4103220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:23.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.698+0000 7f5da8a9e640 1 -- 172.21.15.40:0/3052969194 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5da4101890 msgr2=0x7f5da4102ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:23.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.698+0000 7f5da8a9e640 1 --2- 172.21.15.40:0/3052969194 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5da4101890 0x7f5da4102ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:23.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.698+0000 7f5da8a9e640 1 -- 172.21.15.40:0/3052969194 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5da4102250 msgr2=0x7f5da4103220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:23.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.698+0000 7f5da8a9e640 1 --2- 172.21.15.40:0/3052969194 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5da4102250 0x7f5da4103220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:23.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.698+0000 7f5da8a9e640 1 -- 172.21.15.40:0/3052969194 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5da4103ab0 con 0x7f5da41006a0 2024-04-21T23:16:23.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.698+0000 7f5da929f640 1 --2- 172.21.15.40:0/3052969194 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5da4102250 0x7f5da4103220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:16:23.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.698+0000 7f5da3fff640 1 --2- 172.21.15.40:0/3052969194 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5da4101890 0x7f5da4102ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:16:23.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.698+0000 7f5da8a9e640 1 --2- 172.21.15.40:0/3052969194 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5da41006a0 0x7f5da41046a0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f5d9400d950 tx=0x7f5d9400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:23.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.698+0000 7f5da1ffb640 1 -- 172.21.15.40:0/3052969194 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d94017070 con 0x7f5da41006a0 2024-04-21T23:16:23.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.698+0000 7f5da1ffb640 1 -- 172.21.15.40:0/3052969194 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d94009c30 con 0x7f5da41006a0 2024-04-21T23:16:23.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.698+0000 7f5daad02640 1 -- 172.21.15.40:0/3052969194 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5da41a1b90 con 0x7f5da41006a0 2024-04-21T23:16:23.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.698+0000 7f5da1ffb640 1 -- 172.21.15.40:0/3052969194 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d9400f600 con 0x7f5da41006a0 2024-04-21T23:16:23.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.698+0000 7f5daad02640 1 -- 172.21.15.40:0/3052969194 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5da41a2090 con 0x7f5da41006a0 2024-04-21T23:16:23.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.699+0000 7f5daad02640 1 -- 172.21.15.40:0/3052969194 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5da40689b0 con 0x7f5da41006a0 2024-04-21T23:16:23.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.700+0000 7f5da1ffb640 1 -- 172.21.15.40:0/3052969194 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f5d9401a070 con 0x7f5da41006a0 2024-04-21T23:16:23.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.701+0000 7f5da1ffb640 1 --2- 172.21.15.40:0/3052969194 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f5d8c0a2b90 0x7f5d8c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:23.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.701+0000 7f5da1ffb640 1 -- 172.21.15.40:0/3052969194 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f5d940cedc0 con 0x7f5da41006a0 2024-04-21T23:16:23.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.701+0000 7f5da3fff640 1 --2- 172.21.15.40:0/3052969194 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f5d8c0a2b90 0x7f5d8c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:23.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.702+0000 7f5da3fff640 1 --2- 172.21.15.40:0/3052969194 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f5d8c0a2b90 0x7f5d8c0a5050 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5da4104410 tx=0x7f5d9800b5f0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:23.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.705+0000 7f5da1ffb640 1 -- 172.21.15.40:0/3052969194 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f5d94009de0 con 0x7f5da41006a0 2024-04-21T23:16:23.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.917+0000 7f5daad02640 1 -- 172.21.15.40:0/3052969194 --> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f5da406a450 con 0x7f5d8c0a2b90 2024-04-21T23:16:23.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:23 smithi040 ceph-mon[27984]: pgmap v306: 129 pgs: 129 active+clean; 581 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 682 B/s wr, 0 op/s 2024-04-21T23:16:23.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:23 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/512463810' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-21T23:16:23.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:23 smithi040 ceph-mon[27984]: mds.? [v2:172.21.15.111:6836/2703764182,v1:172.21.15.111:6837/2703764182] up:active 2024-04-21T23:16:23.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:23 smithi040 ceph-mon[27984]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2024-04-21T23:16:23.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:23 smithi040 ceph-mon[27984]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-21T23:16:23.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:23 smithi040 ceph-mon[27984]: from='client.14886 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:16:23.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:23 smithi040 ceph-mon[27984]: from='client.14898 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:16:23.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.942+0000 7f5da1ffb640 1 -- 172.21.15.40:0/3052969194 <== mgr.14148 v2:172.21.15.40:6800/2500760788 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f5da406a450 con 0x7f5d8c0a2b90 2024-04-21T23:16:23.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.944+0000 7f5daad02640 1 -- 172.21.15.40:0/3052969194 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f5d8c0a2b90 msgr2=0x7f5d8c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:23.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.944+0000 7f5daad02640 1 --2- 172.21.15.40:0/3052969194 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f5d8c0a2b90 0x7f5d8c0a5050 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5da4104410 tx=0x7f5d9800b5f0 comp rx=0 tx=0).stop 2024-04-21T23:16:23.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.944+0000 7f5daad02640 1 -- 172.21.15.40:0/3052969194 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5da41006a0 msgr2=0x7f5da41046a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:23.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.944+0000 7f5daad02640 1 --2- 172.21.15.40:0/3052969194 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5da41006a0 0x7f5da41046a0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f5d9400d950 tx=0x7f5d9400de10 comp rx=0 tx=0).stop 2024-04-21T23:16:23.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.944+0000 7f5daad02640 1 -- 172.21.15.40:0/3052969194 shutdown_connections 2024-04-21T23:16:23.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.944+0000 7f5daad02640 1 --2- 172.21.15.40:0/3052969194 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f5d8c0a2b90 0x7f5d8c0a5050 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:23.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.944+0000 7f5daad02640 1 --2- 172.21.15.40:0/3052969194 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5da4102250 0x7f5da4103220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:23.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.944+0000 7f5daad02640 1 --2- 172.21.15.40:0/3052969194 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5da4101890 0x7f5da4102ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:23.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.944+0000 7f5daad02640 1 --2- 172.21.15.40:0/3052969194 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5da41006a0 0x7f5da41046a0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:23.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.944+0000 7f5daad02640 1 -- 172.21.15.40:0/3052969194 >> 172.21.15.40:0/3052969194 conn(0x7f5da40fbc30 msgr2=0x7f5da4069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:23.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.944+0000 7f5daad02640 1 -- 172.21.15.40:0/3052969194 shutdown_connections 2024-04-21T23:16:23.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:23.944+0000 7f5daad02640 1 -- 172.21.15.40:0/3052969194 wait complete. 2024-04-21T23:16:23.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:23 smithi111 ceph-mon[31085]: pgmap v306: 129 pgs: 129 active+clean; 581 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 682 B/s wr, 0 op/s 2024-04-21T23:16:23.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:23 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/512463810' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-21T23:16:23.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:23 smithi111 ceph-mon[31085]: mds.? [v2:172.21.15.111:6836/2703764182,v1:172.21.15.111:6837/2703764182] up:active 2024-04-21T23:16:23.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:23 smithi111 ceph-mon[31085]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2024-04-21T23:16:23.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:23 smithi111 ceph-mon[31085]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-21T23:16:23.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:23 smithi111 ceph-mon[31085]: from='client.14886 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:16:23.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:23 smithi111 ceph-mon[31085]: from='client.14898 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:16:23.945 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:23 smithi174 ceph-mon[31409]: pgmap v306: 129 pgs: 129 active+clean; 581 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 682 B/s wr, 0 op/s 2024-04-21T23:16:23.945 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:23 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/512463810' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-21T23:16:23.945 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:23 smithi174 ceph-mon[31409]: mds.? [v2:172.21.15.111:6836/2703764182,v1:172.21.15.111:6837/2703764182] up:active 2024-04-21T23:16:23.946 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:23 smithi174 ceph-mon[31409]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2024-04-21T23:16:23.946 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:23 smithi174 ceph-mon[31409]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-21T23:16:23.946 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:23 smithi174 ceph-mon[31409]: from='client.14886 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:16:23.946 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:23 smithi174 ceph-mon[31409]: from='client.14898 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:16:23.954 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-04-21T23:16:24.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.062+0000 7f1c99420640 1 -- 172.21.15.40:0/2678626089 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c941017a0 msgr2=0x7f1c94101bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:24.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.062+0000 7f1c99420640 1 --2- 172.21.15.40:0/2678626089 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c941017a0 0x7f1c94101bc0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f1c84009900 tx=0x7f1c8402bc30 comp rx=0 tx=0).stop 2024-04-21T23:16:24.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.062+0000 7f1c99420640 1 -- 172.21.15.40:0/2678626089 shutdown_connections 2024-04-21T23:16:24.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.062+0000 7f1c99420640 1 --2- 172.21.15.40:0/2678626089 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1c94102990 0x7f1c94102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:24.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.062+0000 7f1c99420640 1 --2- 172.21.15.40:0/2678626089 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1c94103350 0x7f1c94109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:24.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.062+0000 7f1c99420640 1 --2- 172.21.15.40:0/2678626089 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c941017a0 0x7f1c94101bc0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:24.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.062+0000 7f1c99420640 1 -- 172.21.15.40:0/2678626089 >> 172.21.15.40:0/2678626089 conn(0x7f1c940fcd10 msgr2=0x7f1c940ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:24.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.062+0000 7f1c99420640 1 -- 172.21.15.40:0/2678626089 shutdown_connections 2024-04-21T23:16:24.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.062+0000 7f1c99420640 1 -- 172.21.15.40:0/2678626089 wait complete. 2024-04-21T23:16:24.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.062+0000 7f1c99420640 1 Processor -- start 2024-04-21T23:16:24.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.063+0000 7f1c99420640 1 -- start start 2024-04-21T23:16:24.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.063+0000 7f1c99420640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c941017a0 0x7f1c94197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:24.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.063+0000 7f1c99420640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1c94102990 0x7f1c94198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:24.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.063+0000 7f1c99420640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1c94103350 0x7f1c9419f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:24.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.063+0000 7f1c99420640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1c9410d3f0 con 0x7f1c941017a0 2024-04-21T23:16:24.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.063+0000 7f1c99420640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f1c9410d270 con 0x7f1c94102990 2024-04-21T23:16:24.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.063+0000 7f1c99420640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1c9410d570 con 0x7f1c94103350 2024-04-21T23:16:24.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.063+0000 7f1c92ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c941017a0 0x7f1c94197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:24.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.063+0000 7f1c92ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c941017a0 0x7f1c94197ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43468/0 (socket says 172.21.15.40:43468) 2024-04-21T23:16:24.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.063+0000 7f1c92ffd640 1 -- 172.21.15.40:0/1461341861 learned_addr learned my addr 172.21.15.40:0/1461341861 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:16:24.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.063+0000 7f1c937fe640 1 --2- 172.21.15.40:0/1461341861 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1c94103350 0x7f1c9419f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:24.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.063+0000 7f1c92ffd640 1 -- 172.21.15.40:0/1461341861 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1c94103350 msgr2=0x7f1c9419f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:24.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.063+0000 7f1c92ffd640 1 --2- 172.21.15.40:0/1461341861 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1c94103350 0x7f1c9419f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:24.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.063+0000 7f1c92ffd640 1 -- 172.21.15.40:0/1461341861 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1c94102990 msgr2=0x7f1c94198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:24.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.063+0000 7f1c927fc640 1 --2- 172.21.15.40:0/1461341861 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1c94102990 0x7f1c94198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:24.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.063+0000 7f1c92ffd640 1 --2- 172.21.15.40:0/1461341861 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1c94102990 0x7f1c94198010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:24.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.063+0000 7f1c92ffd640 1 -- 172.21.15.40:0/1461341861 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c9419f6f0 con 0x7f1c941017a0 2024-04-21T23:16:24.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.063+0000 7f1c937fe640 1 --2- 172.21.15.40:0/1461341861 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1c94103350 0x7f1c9419f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:16:24.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.064+0000 7f1c92ffd640 1 --2- 172.21.15.40:0/1461341861 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c941017a0 0x7f1c94197ad0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f1c84000c00 tx=0x7f1c84005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:24.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.064+0000 7f1c7bfff640 1 -- 172.21.15.40:0/1461341861 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c84041070 con 0x7f1c941017a0 2024-04-21T23:16:24.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.064+0000 7f1c7bfff640 1 -- 172.21.15.40:0/1461341861 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c840044d0 con 0x7f1c941017a0 2024-04-21T23:16:24.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.064+0000 7f1c99420640 1 -- 172.21.15.40:0/1461341861 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c9419f980 con 0x7f1c941017a0 2024-04-21T23:16:24.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.064+0000 7f1c7bfff640 1 -- 172.21.15.40:0/1461341861 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c84034440 con 0x7f1c941017a0 2024-04-21T23:16:24.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.064+0000 7f1c99420640 1 -- 172.21.15.40:0/1461341861 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c9419fe70 con 0x7f1c941017a0 2024-04-21T23:16:24.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.065+0000 7f1c99420640 1 -- 172.21.15.40:0/1461341861 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c94108b30 con 0x7f1c941017a0 2024-04-21T23:16:24.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.066+0000 7f1c7bfff640 1 -- 172.21.15.40:0/1461341861 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f1c84044070 con 0x7f1c941017a0 2024-04-21T23:16:24.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.066+0000 7f1c7bfff640 1 --2- 172.21.15.40:0/1461341861 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f1c7c0a2b90 0x7f1c7c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:24.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.067+0000 7f1c927fc640 1 --2- 172.21.15.40:0/1461341861 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f1c7c0a2b90 0x7f1c7c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:24.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.067+0000 7f1c7bfff640 1 -- 172.21.15.40:0/1461341861 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f1c840f1110 con 0x7f1c941017a0 2024-04-21T23:16:24.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.067+0000 7f1c927fc640 1 --2- 172.21.15.40:0/1461341861 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f1c7c0a2b90 0x7f1c7c0a5050 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f1c94198fe0 tx=0x7f1c88005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:24.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.070+0000 7f1c7bfff640 1 -- 172.21.15.40:0/1461341861 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f1c840bddd0 con 0x7f1c941017a0 2024-04-21T23:16:24.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.282+0000 7f1c99420640 1 -- 172.21.15.40:0/1461341861 --> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f1c94061b80 con 0x7f1c7c0a2b90 2024-04-21T23:16:24.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.287+0000 7f1c7bfff640 1 -- 172.21.15.40:0/1461341861 <== mgr.14148 v2:172.21.15.40:6800/2500760788 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f1c94061b80 con 0x7f1c7c0a2b90 2024-04-21T23:16:24.286 INFO:teuthology.orchestra.run.smithi040.stdout:/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66 2024-04-21T23:16:24.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.289+0000 7f1c99420640 1 -- 172.21.15.40:0/1461341861 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f1c7c0a2b90 msgr2=0x7f1c7c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:24.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.289+0000 7f1c99420640 1 --2- 172.21.15.40:0/1461341861 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f1c7c0a2b90 0x7f1c7c0a5050 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f1c94198fe0 tx=0x7f1c88005f90 comp rx=0 tx=0).stop 2024-04-21T23:16:24.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.289+0000 7f1c99420640 1 -- 172.21.15.40:0/1461341861 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c941017a0 msgr2=0x7f1c94197ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:24.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.289+0000 7f1c99420640 1 --2- 172.21.15.40:0/1461341861 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c941017a0 0x7f1c94197ad0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f1c84000c00 tx=0x7f1c84005d50 comp rx=0 tx=0).stop 2024-04-21T23:16:24.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.289+0000 7f1c99420640 1 -- 172.21.15.40:0/1461341861 shutdown_connections 2024-04-21T23:16:24.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.289+0000 7f1c99420640 1 --2- 172.21.15.40:0/1461341861 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f1c7c0a2b90 0x7f1c7c0a5050 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:24.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.289+0000 7f1c99420640 1 --2- 172.21.15.40:0/1461341861 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1c94102990 0x7f1c94198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:24.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.289+0000 7f1c99420640 1 --2- 172.21.15.40:0/1461341861 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1c94103350 0x7f1c9419f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:24.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.289+0000 7f1c99420640 1 --2- 172.21.15.40:0/1461341861 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c941017a0 0x7f1c94197ad0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:24.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.289+0000 7f1c99420640 1 -- 172.21.15.40:0/1461341861 >> 172.21.15.40:0/1461341861 conn(0x7f1c940fcd10 msgr2=0x7f1c94105d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:24.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.290+0000 7f1c99420640 1 -- 172.21.15.40:0/1461341861 shutdown_connections 2024-04-21T23:16:24.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.290+0000 7f1c99420640 1 -- 172.21.15.40:0/1461341861 wait complete. 2024-04-21T23:16:24.300 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:16:24.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.403+0000 7ff1a3acf640 1 -- 172.21.15.40:0/1727196539 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff19c103360 msgr2=0x7ff19c109bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:24.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.403+0000 7ff1a3acf640 1 --2- 172.21.15.40:0/1727196539 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff19c103360 0x7ff19c109bf0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7ff18c009ca0 tx=0x7ff18c02c0f0 comp rx=0 tx=0).stop 2024-04-21T23:16:24.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.403+0000 7ff1a3acf640 1 -- 172.21.15.40:0/1727196539 shutdown_connections 2024-04-21T23:16:24.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.403+0000 7ff1a3acf640 1 --2- 172.21.15.40:0/1727196539 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff19c1029a0 0x7ff19c102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:24.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.403+0000 7ff1a3acf640 1 --2- 172.21.15.40:0/1727196539 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff19c103360 0x7ff19c109bf0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:24.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.403+0000 7ff1a3acf640 1 --2- 172.21.15.40:0/1727196539 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff19c1017b0 0x7ff19c101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:24.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.403+0000 7ff1a3acf640 1 -- 172.21.15.40:0/1727196539 >> 172.21.15.40:0/1727196539 conn(0x7ff19c0fcd20 msgr2=0x7ff19c0ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:24.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.403+0000 7ff1a3acf640 1 -- 172.21.15.40:0/1727196539 shutdown_connections 2024-04-21T23:16:24.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.403+0000 7ff1a3acf640 1 -- 172.21.15.40:0/1727196539 wait complete. 2024-04-21T23:16:24.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.403+0000 7ff1a3acf640 1 Processor -- start 2024-04-21T23:16:24.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.403+0000 7ff1a3acf640 1 -- start start 2024-04-21T23:16:24.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.404+0000 7ff1a3acf640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff19c1017b0 0x7ff19c197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:24.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.404+0000 7ff1a3acf640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff19c1029a0 0x7ff19c197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:24.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.404+0000 7ff1a3acf640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff19c103360 0x7ff19c19f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:24.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.404+0000 7ff1a3acf640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff19c10d410 con 0x7ff19c103360 2024-04-21T23:16:24.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.404+0000 7ff1a3acf640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff19c10d290 con 0x7ff19c1017b0 2024-04-21T23:16:24.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.404+0000 7ff1a3acf640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff19c10d590 con 0x7ff19c1029a0 2024-04-21T23:16:24.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.404+0000 7ff1a206c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff19c103360 0x7ff19c19f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:24.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.404+0000 7ff1a206c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff19c103360 0x7ff19c19f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43492/0 (socket says 172.21.15.40:43492) 2024-04-21T23:16:24.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.404+0000 7ff1a206c640 1 -- 172.21.15.40:0/3938550799 learned_addr learned my addr 172.21.15.40:0/3938550799 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:16:24.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.404+0000 7ff1a186b640 1 --2- 172.21.15.40:0/3938550799 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff19c1017b0 0x7ff19c197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:24.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.404+0000 7ff1a106a640 1 --2- 172.21.15.40:0/3938550799 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff19c1029a0 0x7ff19c197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:24.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.404+0000 7ff1a206c640 1 -- 172.21.15.40:0/3938550799 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff19c1029a0 msgr2=0x7ff19c197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:24.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.404+0000 7ff1a206c640 1 --2- 172.21.15.40:0/3938550799 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff19c1029a0 0x7ff19c197f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:24.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.404+0000 7ff1a206c640 1 -- 172.21.15.40:0/3938550799 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff19c1017b0 msgr2=0x7ff19c197a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:24.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.404+0000 7ff1a206c640 1 --2- 172.21.15.40:0/3938550799 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff19c1017b0 0x7ff19c197a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:24.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.404+0000 7ff1a206c640 1 -- 172.21.15.40:0/3938550799 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff19c19f670 con 0x7ff19c103360 2024-04-21T23:16:24.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.404+0000 7ff1a206c640 1 --2- 172.21.15.40:0/3938550799 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff19c103360 0x7ff19c19f000 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7ff18c037040 tx=0x7ff18c032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:24.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.404+0000 7ff19affd640 1 -- 172.21.15.40:0/3938550799 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff18c041020 con 0x7ff19c103360 2024-04-21T23:16:24.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.404+0000 7ff19affd640 1 -- 172.21.15.40:0/3938550799 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff18c034d90 con 0x7ff19c103360 2024-04-21T23:16:24.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.404+0000 7ff19affd640 1 -- 172.21.15.40:0/3938550799 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff18c039040 con 0x7ff19c103360 2024-04-21T23:16:24.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.404+0000 7ff1a3acf640 1 -- 172.21.15.40:0/3938550799 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff19c19f900 con 0x7ff19c103360 2024-04-21T23:16:24.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.404+0000 7ff1a3acf640 1 -- 172.21.15.40:0/3938550799 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff19c19fdf0 con 0x7ff19c103360 2024-04-21T23:16:24.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.405+0000 7ff1a3acf640 1 -- 172.21.15.40:0/3938550799 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff19c108ad0 con 0x7ff19c103360 2024-04-21T23:16:24.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.406+0000 7ff19affd640 1 -- 172.21.15.40:0/3938550799 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7ff18c044020 con 0x7ff19c103360 2024-04-21T23:16:24.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.407+0000 7ff19affd640 1 --2- 172.21.15.40:0/3938550799 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7ff1840a2b90 0x7ff1840a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:24.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.407+0000 7ff1a186b640 1 --2- 172.21.15.40:0/3938550799 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7ff1840a2b90 0x7ff1840a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:24.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.408+0000 7ff19affd640 1 -- 172.21.15.40:0/3938550799 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7ff18c0f1550 con 0x7ff19c103360 2024-04-21T23:16:24.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.408+0000 7ff1a186b640 1 --2- 172.21.15.40:0/3938550799 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7ff1840a2b90 0x7ff1840a5050 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ff19c102800 tx=0x7ff19400b390 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:24.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.411+0000 7ff19affd640 1 -- 172.21.15.40:0/3938550799 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7ff18c04b020 con 0x7ff19c103360 2024-04-21T23:16:24.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.510+0000 7ff19affd640 1 -- 172.21.15.40:0/3938550799 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7ff18c0ba1c0 con 0x7ff19c103360 2024-04-21T23:16:24.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.642+0000 7ff1a3acf640 1 -- 172.21.15.40:0/3938550799 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff19c02d290 con 0x7ff19c103360 2024-04-21T23:16:24.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.643+0000 7ff19affd640 1 -- 172.21.15.40:0/3938550799 <== mon.0 v2:172.21.15.40:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11882 (secure 0 0 0) 0x7ff18c0be250 con 0x7ff19c103360 2024-04-21T23:16:24.642 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:16:24.642 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/1571948830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1571948830},{"type":"v1","addr":"172.21.15.40:6835","nonce":1571948830}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/969356642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":969356642},{"type":"v1","addr":"172.21.15.40:6837","nonce":969356642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6839/3025057845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":3025057845},{"type":"v1","addr":"172.21.15.40:6839","nonce":3025057845}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6841/3350488804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":3350488804},{"type":"v1","addr":"172.21.15.40:6841","nonce":3350488804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/3438493203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":3438493203},{"type":"v1","addr":"172.21.15.174:6833","nonce":3438493203}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/3105752678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":3105752678},{"type":"v1","addr":"172.21.15.111:6833","nonce":3105752678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T23:15:55.742233+0000","modified":"2024-04-21T23:16:22.498043+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24521,"mds_1":24502,"mds_2":24494},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24461":{"gid":24461,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6835/2704462092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2704462092},{"type":"v1","addr":"172.21.15.111:6835","nonce":2704462092}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/3905733045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3905733045},{"type":"v1","addr":"172.21.15.174:6835","nonce":3905733045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6837/465579746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":465579746},{"type":"v1","addr":"172.21.15.174:6837","nonce":465579746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/2703764182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":2703764182},{"type":"v1","addr":"172.21.15.111:6837","nonce":2703764182}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24502":{"gid":24502,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.174:6839/2467509274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":2467509274},{"type":"v1","addr":"172.21.15.174:6839","nonce":2467509274}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3712235548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3712235548},{"type":"v1","addr":"172.21.15.111:6839","nonce":3712235548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24494,24502,24521]},"id":1}]} 2024-04-21T23:16:24.643 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 24 2024-04-21T23:16:24.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.645+0000 7ff1a3acf640 1 -- 172.21.15.40:0/3938550799 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7ff1840a2b90 msgr2=0x7ff1840a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:24.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.645+0000 7ff1a3acf640 1 --2- 172.21.15.40:0/3938550799 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7ff1840a2b90 0x7ff1840a5050 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ff19c102800 tx=0x7ff19400b390 comp rx=0 tx=0).stop 2024-04-21T23:16:24.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.645+0000 7ff1a3acf640 1 -- 172.21.15.40:0/3938550799 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff19c103360 msgr2=0x7ff19c19f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:24.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.645+0000 7ff1a3acf640 1 --2- 172.21.15.40:0/3938550799 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff19c103360 0x7ff19c19f000 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7ff18c037040 tx=0x7ff18c032f20 comp rx=0 tx=0).stop 2024-04-21T23:16:24.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.645+0000 7ff1a3acf640 1 -- 172.21.15.40:0/3938550799 shutdown_connections 2024-04-21T23:16:24.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.645+0000 7ff1a3acf640 1 --2- 172.21.15.40:0/3938550799 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7ff1840a2b90 0x7ff1840a5050 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:24.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.645+0000 7ff1a3acf640 1 --2- 172.21.15.40:0/3938550799 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff19c103360 0x7ff19c19f000 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:24.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.645+0000 7ff1a3acf640 1 --2- 172.21.15.40:0/3938550799 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff19c1029a0 0x7ff19c197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:24.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.645+0000 7ff1a3acf640 1 --2- 172.21.15.40:0/3938550799 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff19c1017b0 0x7ff19c197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:24.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.645+0000 7ff1a3acf640 1 -- 172.21.15.40:0/3938550799 >> 172.21.15.40:0/3938550799 conn(0x7ff19c0fcd20 msgr2=0x7ff19c105ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:24.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.645+0000 7ff1a3acf640 1 -- 172.21.15.40:0/3938550799 shutdown_connections 2024-04-21T23:16:24.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.645+0000 7ff1a3acf640 1 -- 172.21.15.40:0/3938550799 wait complete. 2024-04-21T23:16:24.654 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:24 smithi040 ceph-mon[27984]: from='client.14904 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:16:24.654 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:24 smithi040 ceph-mon[27984]: from='client.14910 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:16:24.654 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:24 smithi040 ceph-mon[27984]: from='client.14916 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:16:24.656 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T23:16:24.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.758+0000 7fc8d0fc9640 1 -- 172.21.15.40:0/3908253101 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc8cc1029a0 msgr2=0x7fc8cc102e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:24.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.758+0000 7fc8d0fc9640 1 --2- 172.21.15.40:0/3908253101 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc8cc1029a0 0x7fc8cc102e20 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fc8b4009900 tx=0x7fc8b402bc30 comp rx=0 tx=0).stop 2024-04-21T23:16:24.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.758+0000 7fc8d0fc9640 1 -- 172.21.15.40:0/3908253101 shutdown_connections 2024-04-21T23:16:24.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.758+0000 7fc8d0fc9640 1 --2- 172.21.15.40:0/3908253101 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc8cc103360 0x7fc8cc109bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:24.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.758+0000 7fc8d0fc9640 1 --2- 172.21.15.40:0/3908253101 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc8cc1029a0 0x7fc8cc102e20 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:24.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.758+0000 7fc8d0fc9640 1 --2- 172.21.15.40:0/3908253101 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8cc1017b0 0x7fc8cc101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:24.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.758+0000 7fc8d0fc9640 1 -- 172.21.15.40:0/3908253101 >> 172.21.15.40:0/3908253101 conn(0x7fc8cc0fcd40 msgr2=0x7fc8cc0ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:24.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.758+0000 7fc8d0fc9640 1 -- 172.21.15.40:0/3908253101 shutdown_connections 2024-04-21T23:16:24.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.758+0000 7fc8d0fc9640 1 -- 172.21.15.40:0/3908253101 wait complete. 2024-04-21T23:16:24.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.759+0000 7fc8d0fc9640 1 Processor -- start 2024-04-21T23:16:24.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.759+0000 7fc8d0fc9640 1 -- start start 2024-04-21T23:16:24.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.759+0000 7fc8d0fc9640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc8cc1017b0 0x7fc8cc0754f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:24.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.759+0000 7fc8d0fc9640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc8cc1029a0 0x7fc8cc073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:24.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.759+0000 7fc8d0fc9640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8cc103360 0x7fc8cc074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:24.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.759+0000 7fc8d0fc9640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc8cc10d430 con 0x7fc8cc1029a0 2024-04-21T23:16:24.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.759+0000 7fc8d0fc9640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc8cc10d2b0 con 0x7fc8cc103360 2024-04-21T23:16:24.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.759+0000 7fc8d0fc9640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc8cc10d5b0 con 0x7fc8cc1017b0 2024-04-21T23:16:24.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.759+0000 7fc8c9d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc8cc1029a0 0x7fc8cc073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:24.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.759+0000 7fc8c9d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc8cc1029a0 0x7fc8cc073b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43514/0 (socket says 172.21.15.40:43514) 2024-04-21T23:16:24.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.759+0000 7fc8c9d9b640 1 -- 172.21.15.40:0/3245186211 learned_addr learned my addr 172.21.15.40:0/3245186211 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:16:24.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.759+0000 7fc8cad9d640 1 --2- 172.21.15.40:0/3245186211 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8cc103360 0x7fc8cc074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:24.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.760+0000 7fc8ca59c640 1 --2- 172.21.15.40:0/3245186211 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc8cc1017b0 0x7fc8cc0754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:24.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.760+0000 7fc8c9d9b640 1 -- 172.21.15.40:0/3245186211 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc8cc1017b0 msgr2=0x7fc8cc0754f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:24.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.760+0000 7fc8c9d9b640 1 --2- 172.21.15.40:0/3245186211 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc8cc1017b0 0x7fc8cc0754f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:24.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.760+0000 7fc8c9d9b640 1 -- 172.21.15.40:0/3245186211 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8cc103360 msgr2=0x7fc8cc074070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:24.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.760+0000 7fc8c9d9b640 1 --2- 172.21.15.40:0/3245186211 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8cc103360 0x7fc8cc074070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:24.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.760+0000 7fc8c9d9b640 1 -- 172.21.15.40:0/3245186211 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8cc074900 con 0x7fc8cc1029a0 2024-04-21T23:16:24.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.760+0000 7fc8c9d9b640 1 --2- 172.21.15.40:0/3245186211 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc8cc1029a0 0x7fc8cc073b30 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fc8b4009230 tx=0x7fc8b4005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:24.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.760+0000 7fc8c37fe640 1 -- 172.21.15.40:0/3245186211 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8b4041070 con 0x7fc8cc1029a0 2024-04-21T23:16:24.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.760+0000 7fc8d0fc9640 1 -- 172.21.15.40:0/3245186211 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8cc1a1b40 con 0x7fc8cc1029a0 2024-04-21T23:16:24.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.760+0000 7fc8c37fe640 1 -- 172.21.15.40:0/3245186211 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc8b40054e0 con 0x7fc8cc1029a0 2024-04-21T23:16:24.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.760+0000 7fc8c37fe640 1 -- 172.21.15.40:0/3245186211 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8b4034440 con 0x7fc8cc1029a0 2024-04-21T23:16:24.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.760+0000 7fc8d0fc9640 1 -- 172.21.15.40:0/3245186211 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8cc1a20e0 con 0x7fc8cc1029a0 2024-04-21T23:16:24.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.761+0000 7fc8d0fc9640 1 -- 172.21.15.40:0/3245186211 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8cc108ad0 con 0x7fc8cc1029a0 2024-04-21T23:16:24.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.762+0000 7fc8c37fe640 1 -- 172.21.15.40:0/3245186211 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fc8b4044070 con 0x7fc8cc1029a0 2024-04-21T23:16:24.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.763+0000 7fc8c37fe640 1 --2- 172.21.15.40:0/3245186211 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fc8b80a2c20 0x7fc8b80a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:24.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.763+0000 7fc8c37fe640 1 -- 172.21.15.40:0/3245186211 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fc8b40f0ff0 con 0x7fc8cc1029a0 2024-04-21T23:16:24.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.763+0000 7fc8ca59c640 1 --2- 172.21.15.40:0/3245186211 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fc8b80a2c20 0x7fc8b80a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:24.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.763+0000 7fc8ca59c640 1 --2- 172.21.15.40:0/3245186211 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fc8b80a2c20 0x7fc8b80a50e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fc8c4005fd0 tx=0x7fc8c4005850 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:24.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.767+0000 7fc8c37fe640 1 -- 172.21.15.40:0/3245186211 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fc8b40bdcc0 con 0x7fc8cc1029a0 2024-04-21T23:16:24.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:24 smithi174 ceph-mon[31409]: from='client.14904 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:16:24.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:24 smithi174 ceph-mon[31409]: from='client.14910 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:16:24.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:24 smithi174 ceph-mon[31409]: from='client.14916 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:16:24.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:24 smithi111 ceph-mon[31085]: from='client.14904 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:16:24.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:24 smithi111 ceph-mon[31085]: from='client.14910 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:16:24.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:24 smithi111 ceph-mon[31085]: from='client.14916 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:16:24.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.952+0000 7fc8d0fc9640 1 -- 172.21.15.40:0/3245186211 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc8cc02d390 con 0x7fc8cc1029a0 2024-04-21T23:16:24.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.954+0000 7fc8c37fe640 1 -- 172.21.15.40:0/3245186211 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24138 (secure 0 0 0) 0x7fc8b40bd5c0 con 0x7fc8cc1029a0 2024-04-21T23:16:24.953 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:16:24.953 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":90,"fsid":"d814e4dc-0033-11ef-bc93-c7b262605968","created":"2024-04-21T23:07:52.659207+0000","modified":"2024-04-21T23:16:00.780824+0000","last_up_change":"2024-04-21T23:15:05.308821+0000","last_in_change":"2024-04-21T23:14:45.447491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:11:14.824775+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T23:15:54.322571+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T23:15:55.164493+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T23:15:58.838266+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"0e8d692a-22fe-4a87-8d99-9151f96cbb02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6803","nonce":2009758784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6805","nonce":2009758784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6809","nonce":2009758784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6807","nonce":2009758784}]},"public_addr":"172.21.15.40:6803/2009758784","cluster_addr":"172.21.15.40:6805/2009758784","heartbeat_back_addr":"172.21.15.40:6809/2009758784","heartbeat_front_addr":"172.21.15.40:6807/2009758784","state":["exists","up"]},{"osd":1,"uuid":"fe5fe11b-0e84-4359-83a7-1ba45105755a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6801","nonce":1763903836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6803","nonce":1763903836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6807","nonce":1763903836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6805","nonce":1763903836}]},"public_addr":"172.21.15.111:6801/1763903836","cluster_addr":"172.21.15.111:6803/1763903836","heartbeat_back_addr":"172.21.15.111:6807/1763903836","heartbeat_front_addr":"172.21.15.111:6805/1763903836","state":["exists","up"]},{"osd":2,"uuid":"813ad735-5cc2-4da0-ae16-6db238395fbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6801","nonce":439695825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6803","nonce":439695825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6807","nonce":439695825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6805","nonce":439695825}]},"public_addr":"172.21.15.174:6801/439695825","cluster_addr":"172.21.15.174:6803/439695825","heartbeat_back_addr":"172.21.15.174:6807/439695825","heartbeat_front_addr":"172.21.15.174:6805/439695825","state":["exists","up"]},{"osd":3,"uuid":"09becb03-b7ce-4ffc-917d-b7900ec37e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6811","nonce":3454312436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6813","nonce":3454312436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6817","nonce":3454312436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6815","nonce":3454312436}]},"public_addr":"172.21.15.40:6811/3454312436","cluster_addr":"172.21.15.40:6813/3454312436","heartbeat_back_addr":"172.21.15.40:6817/3454312436","heartbeat_front_addr":"172.21.15.40:6815/3454312436","state":["exists","up"]},{"osd":4,"uuid":"46eabfd7-d88a-4efd-94eb-04354d3a9d7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6809","nonce":1810866070}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6811","nonce":1810866070}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6815","nonce":1810866070}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6813","nonce":1810866070}]},"public_addr":"172.21.15.111:6809/1810866070","cluster_addr":"172.21.15.111:6811/1810866070","heartbeat_back_addr":"172.21.15.111:6815/1810866070","heartbeat_front_addr":"172.21.15.111:6813/1810866070","state":["exists","up"]},{"osd":5,"uuid":"6f73b582-6c65-41db-aab0-9b360f83d5c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6809","nonce":3752986711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6811","nonce":3752986711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6815","nonce":3752986711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6813","nonce":3752986711}]},"public_addr":"172.21.15.174:6809/3752986711","cluster_addr":"172.21.15.174:6811/3752986711","heartbeat_back_addr":"172.21.15.174:6815/3752986711","heartbeat_front_addr":"172.21.15.174:6813/3752986711","state":["exists","up"]},{"osd":6,"uuid":"39f305f0-12d2-4732-aaa0-cd66700af538","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6819","nonce":1618855427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6821","nonce":1618855427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6825","nonce":1618855427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6823","nonce":1618855427}]},"public_addr":"172.21.15.40:6819/1618855427","cluster_addr":"172.21.15.40:6821/1618855427","heartbeat_back_addr":"172.21.15.40:6825/1618855427","heartbeat_front_addr":"172.21.15.40:6823/1618855427","state":["exists","up"]},{"osd":7,"uuid":"769f2468-5102-490b-89f8-d0444d3169a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6817","nonce":927054162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6819","nonce":927054162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6823","nonce":927054162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6821","nonce":927054162}]},"public_addr":"172.21.15.111:6817/927054162","cluster_addr":"172.21.15.111:6819/927054162","heartbeat_back_addr":"172.21.15.111:6823/927054162","heartbeat_front_addr":"172.21.15.111:6821/927054162","state":["exists","up"]},{"osd":8,"uuid":"1a444e80-007d-473b-b972-4c814a813c22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6817","nonce":3817848746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6819","nonce":3817848746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6823","nonce":3817848746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6821","nonce":3817848746}]},"public_addr":"172.21.15.174:6817/3817848746","cluster_addr":"172.21.15.174:6819/3817848746","heartbeat_back_addr":"172.21.15.174:6823/3817848746","heartbeat_front_addr":"172.21.15.174:6821/3817848746","state":["exists","up"]},{"osd":9,"uuid":"9a6b6a56-6ae5-49fe-9e5b-cb17d686e0dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6827","nonce":1649446531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6829","nonce":1649446531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6833","nonce":1649446531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6831","nonce":1649446531}]},"public_addr":"172.21.15.40:6827/1649446531","cluster_addr":"172.21.15.40:6829/1649446531","heartbeat_back_addr":"172.21.15.40:6833/1649446531","heartbeat_front_addr":"172.21.15.40:6831/1649446531","state":["exists","up"]},{"osd":10,"uuid":"a70038ba-e2ad-4170-85cc-ea0470a2edef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6825","nonce":1227538935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6827","nonce":1227538935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6831","nonce":1227538935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6829","nonce":1227538935}]},"public_addr":"172.21.15.111:6825/1227538935","cluster_addr":"172.21.15.111:6827/1227538935","heartbeat_back_addr":"172.21.15.111:6831/1227538935","heartbeat_front_addr":"172.21.15.111:6829/1227538935","state":["exists","up"]},{"osd":11,"uuid":"781532bb-7a5e-44e3-8eb7-b584b0054666","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6825","nonce":2045865553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6827","nonce":2045865553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6831","nonce":2045865553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6829","nonce":2045865553}]},"public_addr":"172.21.15.174:6825/2045865553","cluster_addr":"172.21.15.174:6827/2045865553","heartbeat_back_addr":"172.21.15.174:6831/2045865553","heartbeat_front_addr":"172.21.15.174:6829/2045865553","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:10:24.761290+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:10:48.565428+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:11:12.307818+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:11:37.951529+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:02.832523+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:27.568879+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:52.997707+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:13:18.937274+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:13:44.708657+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:14:10.833855+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:14:37.078272+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:15:03.878076+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:6801/526554319":"2024-04-22T23:08:50.305770+0000","172.21.15.40:6800/526554319":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/3178763471":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/4045911364":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/276300080":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/3689649814":"2024-04-22T23:08:23.743030+0000","172.21.15.40:0/2241116715":"2024-04-22T23:08:23.743030+0000","172.21.15.40:6801/1479040326":"2024-04-22T23:08:23.743030+0000","172.21.15.40:6800/1479040326":"2024-04-22T23:08:23.743030+0000","172.21.15.40:0/681298099":"2024-04-22T23:08:23.743030+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T23:16:24.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.955+0000 7fc8d0fc9640 1 -- 172.21.15.40:0/3245186211 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fc8b80a2c20 msgr2=0x7fc8b80a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:24.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.955+0000 7fc8d0fc9640 1 --2- 172.21.15.40:0/3245186211 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fc8b80a2c20 0x7fc8b80a50e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fc8c4005fd0 tx=0x7fc8c4005850 comp rx=0 tx=0).stop 2024-04-21T23:16:24.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.955+0000 7fc8d0fc9640 1 -- 172.21.15.40:0/3245186211 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc8cc1029a0 msgr2=0x7fc8cc073b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:24.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.955+0000 7fc8d0fc9640 1 --2- 172.21.15.40:0/3245186211 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc8cc1029a0 0x7fc8cc073b30 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fc8b4009230 tx=0x7fc8b4005dd0 comp rx=0 tx=0).stop 2024-04-21T23:16:24.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.956+0000 7fc8d0fc9640 1 -- 172.21.15.40:0/3245186211 shutdown_connections 2024-04-21T23:16:24.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.956+0000 7fc8d0fc9640 1 --2- 172.21.15.40:0/3245186211 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fc8b80a2c20 0x7fc8b80a50e0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:24.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.956+0000 7fc8d0fc9640 1 --2- 172.21.15.40:0/3245186211 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8cc103360 0x7fc8cc074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:24.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.956+0000 7fc8d0fc9640 1 --2- 172.21.15.40:0/3245186211 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc8cc1029a0 0x7fc8cc073b30 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:24.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.956+0000 7fc8d0fc9640 1 --2- 172.21.15.40:0/3245186211 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc8cc1017b0 0x7fc8cc0754f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:24.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.956+0000 7fc8d0fc9640 1 -- 172.21.15.40:0/3245186211 >> 172.21.15.40:0/3245186211 conn(0x7fc8cc0fcd40 msgr2=0x7fc8cc105ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:24.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.956+0000 7fc8d0fc9640 1 -- 172.21.15.40:0/3245186211 shutdown_connections 2024-04-21T23:16:24.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:24.956+0000 7fc8d0fc9640 1 -- 172.21.15.40:0/3245186211 wait complete. 2024-04-21T23:16:24.966 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-21T23:16:24.966 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-21T23:16:25.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.069+0000 7f5ef7bd7640 1 -- 172.21.15.40:0/4112570842 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ef0101880 msgr2=0x7f5ef0101d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:25.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.069+0000 7f5ef7bd7640 1 --2- 172.21.15.40:0/4112570842 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ef0101880 0x7f5ef0101d00 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f5eec009900 tx=0x7f5eec02bc30 comp rx=0 tx=0).stop 2024-04-21T23:16:25.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.069+0000 7f5ef7bd7640 1 -- 172.21.15.40:0/4112570842 shutdown_connections 2024-04-21T23:16:25.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.069+0000 7f5ef7bd7640 1 --2- 172.21.15.40:0/4112570842 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5ef0102240 0x7f5ef0108ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:25.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.069+0000 7f5ef7bd7640 1 --2- 172.21.15.40:0/4112570842 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ef0101880 0x7f5ef0101d00 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:25.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.069+0000 7f5ef7bd7640 1 --2- 172.21.15.40:0/4112570842 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ef0100690 0x7f5ef0100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:25.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.069+0000 7f5ef7bd7640 1 -- 172.21.15.40:0/4112570842 >> 172.21.15.40:0/4112570842 conn(0x7f5ef00fbc40 msgr2=0x7f5ef00fe060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:25.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.069+0000 7f5ef7bd7640 1 -- 172.21.15.40:0/4112570842 shutdown_connections 2024-04-21T23:16:25.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.069+0000 7f5ef7bd7640 1 -- 172.21.15.40:0/4112570842 wait complete. 2024-04-21T23:16:25.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.070+0000 7f5ef7bd7640 1 Processor -- start 2024-04-21T23:16:25.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.070+0000 7f5ef7bd7640 1 -- start start 2024-04-21T23:16:25.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.070+0000 7f5ef7bd7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ef0100690 0x7f5ef010b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:25.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.070+0000 7f5ef7bd7640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ef0101880 0x7f5ef0109650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:25.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.070+0000 7f5ef7bd7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5ef0102240 0x7f5ef0109b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:25.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.070+0000 7f5ef7bd7640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5ef010d430 con 0x7f5ef0100690 2024-04-21T23:16:25.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.070+0000 7f5ef7bd7640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f5ef010d2b0 con 0x7f5ef0101880 2024-04-21T23:16:25.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.070+0000 7f5ef7bd7640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5ef010d5b0 con 0x7f5ef0102240 2024-04-21T23:16:25.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.070+0000 7f5ef5973640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ef0100690 0x7f5ef010b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:25.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.070+0000 7f5ef5973640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ef0100690 0x7f5ef010b010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43520/0 (socket says 172.21.15.40:43520) 2024-04-21T23:16:25.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.070+0000 7f5ef5973640 1 -- 172.21.15.40:0/3978198690 learned_addr learned my addr 172.21.15.40:0/3978198690 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:16:25.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.070+0000 7f5ef6174640 1 --2- 172.21.15.40:0/3978198690 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5ef0102240 0x7f5ef0109b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:25.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.070+0000 7f5ef5172640 1 --2- 172.21.15.40:0/3978198690 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ef0101880 0x7f5ef0109650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:25.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.070+0000 7f5ef5973640 1 -- 172.21.15.40:0/3978198690 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5ef0102240 msgr2=0x7f5ef0109b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:25.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.070+0000 7f5ef5973640 1 --2- 172.21.15.40:0/3978198690 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5ef0102240 0x7f5ef0109b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:25.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.070+0000 7f5ef5973640 1 -- 172.21.15.40:0/3978198690 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ef0101880 msgr2=0x7f5ef0109650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:25.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.070+0000 7f5ef5973640 1 --2- 172.21.15.40:0/3978198690 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ef0101880 0x7f5ef0109650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:25.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.070+0000 7f5ef5973640 1 -- 172.21.15.40:0/3978198690 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ef010a420 con 0x7f5ef0100690 2024-04-21T23:16:25.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.070+0000 7f5ef5172640 1 --2- 172.21.15.40:0/3978198690 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ef0101880 0x7f5ef0109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:16:25.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.070+0000 7f5ef5973640 1 --2- 172.21.15.40:0/3978198690 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ef0100690 0x7f5ef010b010 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f5ee000d950 tx=0x7f5ee000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:25.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.071+0000 7f5ee6ffd640 1 -- 172.21.15.40:0/3978198690 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ee0017020 con 0x7f5ef0100690 2024-04-21T23:16:25.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.071+0000 7f5ee6ffd640 1 -- 172.21.15.40:0/3978198690 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ee0009c30 con 0x7f5ef0100690 2024-04-21T23:16:25.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.071+0000 7f5ee6ffd640 1 -- 172.21.15.40:0/3978198690 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ee000f720 con 0x7f5ef0100690 2024-04-21T23:16:25.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.071+0000 7f5ef7bd7640 1 -- 172.21.15.40:0/3978198690 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ef01a1b20 con 0x7f5ef0100690 2024-04-21T23:16:25.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.071+0000 7f5ef7bd7640 1 -- 172.21.15.40:0/3978198690 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ef01a2020 con 0x7f5ef0100690 2024-04-21T23:16:25.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.072+0000 7f5ef7bd7640 1 -- 172.21.15.40:0/3978198690 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ef0107950 con 0x7f5ef0100690 2024-04-21T23:16:25.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.073+0000 7f5ee6ffd640 1 -- 172.21.15.40:0/3978198690 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f5ee001a020 con 0x7f5ef0100690 2024-04-21T23:16:25.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.073+0000 7f5ee6ffd640 1 --2- 172.21.15.40:0/3978198690 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f5edc0a2c20 0x7f5edc0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:25.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.074+0000 7f5ef5172640 1 --2- 172.21.15.40:0/3978198690 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f5edc0a2c20 0x7f5edc0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:25.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.074+0000 7f5ee6ffd640 1 -- 172.21.15.40:0/3978198690 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f5ee00cf490 con 0x7f5ef0100690 2024-04-21T23:16:25.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.074+0000 7f5ef5172640 1 --2- 172.21.15.40:0/3978198690 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f5edc0a2c20 0x7f5edc0a50e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f5eec036040 tx=0x7f5eec000bc0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:25.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.078+0000 7f5ee6ffd640 1 -- 172.21.15.40:0/3978198690 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f5ee0021020 con 0x7f5ef0100690 2024-04-21T23:16:25.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.262+0000 7f5ef7bd7640 1 -- 172.21.15.40:0/3978198690 --> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5ef0061b80 con 0x7f5edc0a2c20 2024-04-21T23:16:25.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.273+0000 7f5ee6ffd640 1 -- 172.21.15.40:0/3978198690 <== mgr.14148 v2:172.21.15.40:6800/2500760788 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340266 (secure 0 0 0) 0x7f5ef0061b80 con 0x7f5edc0a2c20 2024-04-21T23:16:25.276 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:16:25.278 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-21T23:16:25.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.276+0000 7f5ef7bd7640 1 -- 172.21.15.40:0/3978198690 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f5edc0a2c20 msgr2=0x7f5edc0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:25.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.276+0000 7f5ef7bd7640 1 --2- 172.21.15.40:0/3978198690 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f5edc0a2c20 0x7f5edc0a50e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f5eec036040 tx=0x7f5eec000bc0 comp rx=0 tx=0).stop 2024-04-21T23:16:25.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.276+0000 7f5ef7bd7640 1 -- 172.21.15.40:0/3978198690 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ef0100690 msgr2=0x7f5ef010b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:25.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.276+0000 7f5ef7bd7640 1 --2- 172.21.15.40:0/3978198690 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ef0100690 0x7f5ef010b010 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f5ee000d950 tx=0x7f5ee000de10 comp rx=0 tx=0).stop 2024-04-21T23:16:25.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.276+0000 7f5ef7bd7640 1 -- 172.21.15.40:0/3978198690 shutdown_connections 2024-04-21T23:16:25.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.276+0000 7f5ef7bd7640 1 --2- 172.21.15.40:0/3978198690 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f5edc0a2c20 0x7f5edc0a50e0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:25.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.276+0000 7f5ef7bd7640 1 --2- 172.21.15.40:0/3978198690 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ef0101880 0x7f5ef0109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:25.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.276+0000 7f5ef7bd7640 1 --2- 172.21.15.40:0/3978198690 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5ef0102240 0x7f5ef0109b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:25.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.276+0000 7f5ef7bd7640 1 --2- 172.21.15.40:0/3978198690 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ef0100690 0x7f5ef010b010 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:25.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.276+0000 7f5ef7bd7640 1 -- 172.21.15.40:0/3978198690 >> 172.21.15.40:0/3978198690 conn(0x7f5ef00fbc40 msgr2=0x7f5ef0104b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:25.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.276+0000 7f5ef7bd7640 1 -- 172.21.15.40:0/3978198690 shutdown_connections 2024-04-21T23:16:25.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.276+0000 7f5ef7bd7640 1 -- 172.21.15.40:0/3978198690 wait complete. 2024-04-21T23:16:25.286 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":307,"stamp":"2024-04-21T23:16:24.400700+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595232,"num_objects":41,"num_object_clones":0,"num_object_copies":123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":144,"num_read_kb":127,"num_write":269,"num_write_kb":2973,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":211,"ondisk_log_size":211,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":358576,"kb_used_data":13556,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124583248,"statfs":{"total":1151940427776,"available":1151573245952,"internally_reserved":0,"allocated":13881344,"data_stored":7593312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":353154362},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":365,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":41,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003481"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.414140+0000","last_change":"2024-04-21T23:15:56.349368+0000","last_active":"2024-04-21T23:16:01.414140+0000","last_peered":"2024-04-21T23:16:01.414140+0000","last_clean":"2024-04-21T23:16:01.414140+0000","last_became_active":"2024-04-21T23:15:56.345743+0000","last_became_peered":"2024-04-21T23:15:56.345743+0000","last_unstale":"2024-04-21T23:16:01.414140+0000","last_undegraded":"2024-04-21T23:16:01.414140+0000","last_fullsized":"2024-04-21T23:16:01.414140+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:57:21.574012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.480927+0000","last_change":"2024-04-21T23:15:56.347231+0000","last_active":"2024-04-21T23:16:02.480927+0000","last_peered":"2024-04-21T23:16:02.480927+0000","last_clean":"2024-04-21T23:16:02.480927+0000","last_became_active":"2024-04-21T23:15:56.346847+0000","last_became_peered":"2024-04-21T23:15:56.346847+0000","last_unstale":"2024-04-21T23:16:02.480927+0000","last_undegraded":"2024-04-21T23:16:02.480927+0000","last_fullsized":"2024-04-21T23:16:02.480927+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:20:15.884789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.480695+0000","last_change":"2024-04-21T23:15:56.354897+0000","last_active":"2024-04-21T23:16:02.480695+0000","last_peered":"2024-04-21T23:16:02.480695+0000","last_clean":"2024-04-21T23:16:02.480695+0000","last_became_active":"2024-04-21T23:15:56.354494+0000","last_became_peered":"2024-04-21T23:15:56.354494+0000","last_unstale":"2024-04-21T23:16:02.480695+0000","last_undegraded":"2024-04-21T23:16:02.480695+0000","last_fullsized":"2024-04-21T23:16:02.480695+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:51:11.865310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.480899+0000","last_change":"2024-04-21T23:15:56.355184+0000","last_active":"2024-04-21T23:16:02.480899+0000","last_peered":"2024-04-21T23:16:02.480899+0000","last_clean":"2024-04-21T23:16:02.480899+0000","last_became_active":"2024-04-21T23:15:56.354748+0000","last_became_peered":"2024-04-21T23:15:56.354748+0000","last_unstale":"2024-04-21T23:16:02.480899+0000","last_undegraded":"2024-04-21T23:16:02.480899+0000","last_fullsized":"2024-04-21T23:16:02.480899+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:46:24.867488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.480384+0000","last_change":"2024-04-21T23:15:56.349446+0000","last_active":"2024-04-21T23:16:02.480384+0000","last_peered":"2024-04-21T23:16:02.480384+0000","last_clean":"2024-04-21T23:16:02.480384+0000","last_became_active":"2024-04-21T23:15:56.349050+0000","last_became_peered":"2024-04-21T23:15:56.349050+0000","last_unstale":"2024-04-21T23:16:02.480384+0000","last_undegraded":"2024-04-21T23:16:02.480384+0000","last_fullsized":"2024-04-21T23:16:02.480384+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:15:25.489044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.480430+0000","last_change":"2024-04-21T23:15:56.356095+0000","last_active":"2024-04-21T23:16:02.480430+0000","last_peered":"2024-04-21T23:16:02.480430+0000","last_clean":"2024-04-21T23:16:02.480430+0000","last_became_active":"2024-04-21T23:15:56.355754+0000","last_became_peered":"2024-04-21T23:15:56.355754+0000","last_unstale":"2024-04-21T23:16:02.480430+0000","last_undegraded":"2024-04-21T23:16:02.480430+0000","last_fullsized":"2024-04-21T23:16:02.480430+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:00:27.296457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.413381+0000","last_change":"2024-04-21T23:15:56.349733+0000","last_active":"2024-04-21T23:16:01.413381+0000","last_peered":"2024-04-21T23:16:01.413381+0000","last_clean":"2024-04-21T23:16:01.413381+0000","last_became_active":"2024-04-21T23:15:56.347820+0000","last_became_peered":"2024-04-21T23:15:56.347820+0000","last_unstale":"2024-04-21T23:16:01.413381+0000","last_undegraded":"2024-04-21T23:16:01.413381+0000","last_fullsized":"2024-04-21T23:16:01.413381+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:31:34.453626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.749212+0000","last_change":"2024-04-21T23:15:56.359385+0000","last_active":"2024-04-21T23:16:01.749212+0000","last_peered":"2024-04-21T23:16:01.749212+0000","last_clean":"2024-04-21T23:16:01.749212+0000","last_became_active":"2024-04-21T23:15:56.359074+0000","last_became_peered":"2024-04-21T23:15:56.359074+0000","last_unstale":"2024-04-21T23:16:01.749212+0000","last_undegraded":"2024-04-21T23:16:01.749212+0000","last_fullsized":"2024-04-21T23:16:01.749212+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:40:23.904592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.129548+0000","last_change":"2024-04-21T23:15:56.350886+0000","last_active":"2024-04-21T23:16:02.129548+0000","last_peered":"2024-04-21T23:16:02.129548+0000","last_clean":"2024-04-21T23:16:02.129548+0000","last_became_active":"2024-04-21T23:15:56.350531+0000","last_became_peered":"2024-04-21T23:15:56.350531+0000","last_unstale":"2024-04-21T23:16:02.129548+0000","last_undegraded":"2024-04-21T23:16:02.129548+0000","last_fullsized":"2024-04-21T23:16:02.129548+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:13:12.354473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.481404+0000","last_change":"2024-04-21T23:15:56.347376+0000","last_active":"2024-04-21T23:16:02.481404+0000","last_peered":"2024-04-21T23:16:02.481404+0000","last_clean":"2024-04-21T23:16:02.481404+0000","last_became_active":"2024-04-21T23:15:56.347016+0000","last_became_peered":"2024-04-21T23:15:56.347016+0000","last_unstale":"2024-04-21T23:16:02.481404+0000","last_undegraded":"2024-04-21T23:16:02.481404+0000","last_fullsized":"2024-04-21T23:16:02.481404+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:49:46.722931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.413990+0000","last_change":"2024-04-21T23:15:56.351754+0000","last_active":"2024-04-21T23:16:01.413990+0000","last_peered":"2024-04-21T23:16:01.413990+0000","last_clean":"2024-04-21T23:16:01.413990+0000","last_became_active":"2024-04-21T23:15:56.349860+0000","last_became_peered":"2024-04-21T23:15:56.349860+0000","last_unstale":"2024-04-21T23:16:01.413990+0000","last_undegraded":"2024-04-21T23:16:01.413990+0000","last_fullsized":"2024-04-21T23:16:01.413990+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:48:41.469708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.481302+0000","last_change":"2024-04-21T23:15:56.345177+0000","last_active":"2024-04-21T23:16:02.481302+0000","last_peered":"2024-04-21T23:16:02.481302+0000","last_clean":"2024-04-21T23:16:02.481302+0000","last_became_active":"2024-04-21T23:15:56.344791+0000","last_became_peered":"2024-04-21T23:15:56.344791+0000","last_unstale":"2024-04-21T23:16:02.481302+0000","last_undegraded":"2024-04-21T23:16:02.481302+0000","last_fullsized":"2024-04-21T23:16:02.481302+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:33:33.418383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.414014+0000","last_change":"2024-04-21T23:15:56.334473+0000","last_active":"2024-04-21T23:16:01.414014+0000","last_peered":"2024-04-21T23:16:01.414014+0000","last_clean":"2024-04-21T23:16:01.414014+0000","last_became_active":"2024-04-21T23:15:56.334219+0000","last_became_peered":"2024-04-21T23:15:56.334219+0000","last_unstale":"2024-04-21T23:16:01.414014+0000","last_undegraded":"2024-04-21T23:16:01.414014+0000","last_fullsized":"2024-04-21T23:16:01.414014+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:22:16.547760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.480474+0000","last_change":"2024-04-21T23:15:56.354544+0000","last_active":"2024-04-21T23:16:02.480474+0000","last_peered":"2024-04-21T23:16:02.480474+0000","last_clean":"2024-04-21T23:16:02.480474+0000","last_became_active":"2024-04-21T23:15:56.354206+0000","last_became_peered":"2024-04-21T23:15:56.354206+0000","last_unstale":"2024-04-21T23:16:02.480474+0000","last_undegraded":"2024-04-21T23:16:02.480474+0000","last_fullsized":"2024-04-21T23:16:02.480474+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:37:19.985422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.480301+0000","last_change":"2024-04-21T23:15:56.357447+0000","last_active":"2024-04-21T23:16:02.480301+0000","last_peered":"2024-04-21T23:16:02.480301+0000","last_clean":"2024-04-21T23:16:02.480301+0000","last_became_active":"2024-04-21T23:15:56.355850+0000","last_became_peered":"2024-04-21T23:15:56.355850+0000","last_unstale":"2024-04-21T23:16:02.480301+0000","last_undegraded":"2024-04-21T23:16:02.480301+0000","last_fullsized":"2024-04-21T23:16:02.480301+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:48:56.648994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.128924+0000","last_change":"2024-04-21T23:15:56.353712+0000","last_active":"2024-04-21T23:16:02.128924+0000","last_peered":"2024-04-21T23:16:02.128924+0000","last_clean":"2024-04-21T23:16:02.128924+0000","last_became_active":"2024-04-21T23:15:56.353388+0000","last_became_peered":"2024-04-21T23:15:56.353388+0000","last_unstale":"2024-04-21T23:16:02.128924+0000","last_undegraded":"2024-04-21T23:16:02.128924+0000","last_fullsized":"2024-04-21T23:16:02.128924+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:43:33.581222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.413993+0000","last_change":"2024-04-21T23:15:56.347041+0000","last_active":"2024-04-21T23:16:01.413993+0000","last_peered":"2024-04-21T23:16:01.413993+0000","last_clean":"2024-04-21T23:16:01.413993+0000","last_became_active":"2024-04-21T23:15:56.346630+0000","last_became_peered":"2024-04-21T23:15:56.346630+0000","last_unstale":"2024-04-21T23:16:01.413993+0000","last_undegraded":"2024-04-21T23:16:01.413993+0000","last_fullsized":"2024-04-21T23:16:01.413993+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:57:42.553666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.480700+0000","last_change":"2024-04-21T23:15:56.353809+0000","last_active":"2024-04-21T23:16:02.480700+0000","last_peered":"2024-04-21T23:16:02.480700+0000","last_clean":"2024-04-21T23:16:02.480700+0000","last_became_active":"2024-04-21T23:15:56.353469+0000","last_became_peered":"2024-04-21T23:15:56.353469+0000","last_unstale":"2024-04-21T23:16:02.480700+0000","last_undegraded":"2024-04-21T23:16:02.480700+0000","last_fullsized":"2024-04-21T23:16:02.480700+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:11:14.136845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.805461+0000","last_change":"2024-04-21T23:15:56.354500+0000","last_active":"2024-04-21T23:16:01.805461+0000","last_peered":"2024-04-21T23:16:01.805461+0000","last_clean":"2024-04-21T23:16:01.805461+0000","last_became_active":"2024-04-21T23:15:56.353893+0000","last_became_peered":"2024-04-21T23:15:56.353893+0000","last_unstale":"2024-04-21T23:16:01.805461+0000","last_undegraded":"2024-04-21T23:16:01.805461+0000","last_fullsized":"2024-04-21T23:16:01.805461+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:14:38.121448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.805252+0000","last_change":"2024-04-21T23:16:00.438245+0000","last_active":"2024-04-21T23:16:01.805252+0000","last_peered":"2024-04-21T23:16:01.805252+0000","last_clean":"2024-04-21T23:16:01.805252+0000","last_became_active":"2024-04-21T23:16:00.437755+0000","last_became_peered":"2024-04-21T23:16:00.437755+0000","last_unstale":"2024-04-21T23:16:01.805252+0000","last_undegraded":"2024-04-21T23:16:01.805252+0000","last_fullsized":"2024-04-21T23:16:01.805252+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:33:37.273223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"90'9","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:21.496232+0000","last_change":"2024-04-21T23:15:55.402553+0000","last_active":"2024-04-21T23:16:21.496232+0000","last_peered":"2024-04-21T23:16:21.496232+0000","last_clean":"2024-04-21T23:16:21.496232+0000","last_became_active":"2024-04-21T23:15:55.402192+0000","last_became_peered":"2024-04-21T23:15:55.402192+0000","last_unstale":"2024-04-21T23:16:21.496232+0000","last_undegraded":"2024-04-21T23:16:21.496232+0000","last_fullsized":"2024-04-21T23:16:21.496232+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:24:57.467906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.749466+0000","last_change":"2024-04-21T23:15:56.358797+0000","last_active":"2024-04-21T23:16:01.749466+0000","last_peered":"2024-04-21T23:16:01.749466+0000","last_clean":"2024-04-21T23:16:01.749466+0000","last_became_active":"2024-04-21T23:15:56.357997+0000","last_became_peered":"2024-04-21T23:15:56.357997+0000","last_unstale":"2024-04-21T23:16:01.749466+0000","last_undegraded":"2024-04-21T23:16:01.749466+0000","last_fullsized":"2024-04-21T23:16:01.749466+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:20:50.891506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.480298+0000","last_change":"2024-04-21T23:16:00.443368+0000","last_active":"2024-04-21T23:16:02.480298+0000","last_peered":"2024-04-21T23:16:02.480298+0000","last_clean":"2024-04-21T23:16:02.480298+0000","last_became_active":"2024-04-21T23:16:00.442940+0000","last_became_peered":"2024-04-21T23:16:00.442940+0000","last_unstale":"2024-04-21T23:16:02.480298+0000","last_undegraded":"2024-04-21T23:16:02.480298+0000","last_fullsized":"2024-04-21T23:16:02.480298+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:17:15.706460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"85'1","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.749541+0000","last_change":"2024-04-21T23:15:55.405216+0000","last_active":"2024-04-21T23:16:01.749541+0000","last_peered":"2024-04-21T23:16:01.749541+0000","last_clean":"2024-04-21T23:16:01.749541+0000","last_became_active":"2024-04-21T23:15:55.404844+0000","last_became_peered":"2024-04-21T23:15:55.404844+0000","last_unstale":"2024-04-21T23:16:01.749541+0000","last_undegraded":"2024-04-21T23:16:01.749541+0000","last_fullsized":"2024-04-21T23:16:01.749541+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:56:28.691844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.480884+0000","last_change":"2024-04-21T23:15:56.355789+0000","last_active":"2024-04-21T23:16:02.480884+0000","last_peered":"2024-04-21T23:16:02.480884+0000","last_clean":"2024-04-21T23:16:02.480884+0000","last_became_active":"2024-04-21T23:15:56.354972+0000","last_became_peered":"2024-04-21T23:15:56.354972+0000","last_unstale":"2024-04-21T23:16:02.480884+0000","last_undegraded":"2024-04-21T23:16:02.480884+0000","last_fullsized":"2024-04-21T23:16:02.480884+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:11:36.831453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.480637+0000","last_change":"2024-04-21T23:16:00.443241+0000","last_active":"2024-04-21T23:16:02.480637+0000","last_peered":"2024-04-21T23:16:02.480637+0000","last_clean":"2024-04-21T23:16:02.480637+0000","last_became_active":"2024-04-21T23:16:00.442820+0000","last_became_peered":"2024-04-21T23:16:00.442820+0000","last_unstale":"2024-04-21T23:16:02.480637+0000","last_undegraded":"2024-04-21T23:16:02.480637+0000","last_fullsized":"2024-04-21T23:16:02.480637+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:28:04.032621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"90'4","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:21.492858+0000","last_change":"2024-04-21T23:15:55.403311+0000","last_active":"2024-04-21T23:16:21.492858+0000","last_peered":"2024-04-21T23:16:21.492858+0000","last_clean":"2024-04-21T23:16:21.492858+0000","last_became_active":"2024-04-21T23:15:55.402943+0000","last_became_peered":"2024-04-21T23:15:55.402943+0000","last_unstale":"2024-04-21T23:16:21.492858+0000","last_undegraded":"2024-04-21T23:16:21.492858+0000","last_fullsized":"2024-04-21T23:16:21.492858+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:16:25.961089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.129201+0000","last_change":"2024-04-21T23:15:56.354462+0000","last_active":"2024-04-21T23:16:02.129201+0000","last_peered":"2024-04-21T23:16:02.129201+0000","last_clean":"2024-04-21T23:16:02.129201+0000","last_became_active":"2024-04-21T23:15:56.354134+0000","last_became_peered":"2024-04-21T23:15:56.354134+0000","last_unstale":"2024-04-21T23:16:02.129201+0000","last_undegraded":"2024-04-21T23:16:02.129201+0000","last_fullsized":"2024-04-21T23:16:02.129201+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:27:40.809483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.480246+0000","last_change":"2024-04-21T23:16:00.427717+0000","last_active":"2024-04-21T23:16:02.480246+0000","last_peered":"2024-04-21T23:16:02.480246+0000","last_clean":"2024-04-21T23:16:02.480246+0000","last_became_active":"2024-04-21T23:16:00.427296+0000","last_became_peered":"2024-04-21T23:16:00.427296+0000","last_unstale":"2024-04-21T23:16:02.480246+0000","last_undegraded":"2024-04-21T23:16:02.480246+0000","last_fullsized":"2024-04-21T23:16:02.480246+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:03:41.650045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.480762+0000","last_change":"2024-04-21T23:15:55.398641+0000","last_active":"2024-04-21T23:16:02.480762+0000","last_peered":"2024-04-21T23:16:02.480762+0000","last_clean":"2024-04-21T23:16:02.480762+0000","last_became_active":"2024-04-21T23:15:55.397712+0000","last_became_peered":"2024-04-21T23:15:55.397712+0000","last_unstale":"2024-04-21T23:16:02.480762+0000","last_undegraded":"2024-04-21T23:16:02.480762+0000","last_fullsized":"2024-04-21T23:16:02.480762+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:28:09.266740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.259858+0000","last_change":"2024-04-21T23:15:56.345359+0000","last_active":"2024-04-21T23:16:02.259858+0000","last_peered":"2024-04-21T23:16:02.259858+0000","last_clean":"2024-04-21T23:16:02.259858+0000","last_became_active":"2024-04-21T23:15:56.343652+0000","last_became_peered":"2024-04-21T23:15:56.343652+0000","last_unstale":"2024-04-21T23:16:02.259858+0000","last_undegraded":"2024-04-21T23:16:02.259858+0000","last_fullsized":"2024-04-21T23:16:02.259858+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:18:10.057999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.805468+0000","last_change":"2024-04-21T23:16:00.430640+0000","last_active":"2024-04-21T23:16:01.805468+0000","last_peered":"2024-04-21T23:16:01.805468+0000","last_clean":"2024-04-21T23:16:01.805468+0000","last_became_active":"2024-04-21T23:16:00.430141+0000","last_became_peered":"2024-04-21T23:16:00.430141+0000","last_unstale":"2024-04-21T23:16:01.805468+0000","last_undegraded":"2024-04-21T23:16:01.805468+0000","last_fullsized":"2024-04-21T23:16:01.805468+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:41:13.749780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"85'1","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.259505+0000","last_change":"2024-04-21T23:15:55.399823+0000","last_active":"2024-04-21T23:16:02.259505+0000","last_peered":"2024-04-21T23:16:02.259505+0000","last_clean":"2024-04-21T23:16:02.259505+0000","last_became_active":"2024-04-21T23:15:55.399490+0000","last_became_peered":"2024-04-21T23:15:55.399490+0000","last_unstale":"2024-04-21T23:16:02.259505+0000","last_undegraded":"2024-04-21T23:16:02.259505+0000","last_fullsized":"2024-04-21T23:16:02.259505+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:54:38.124841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.480401+0000","last_change":"2024-04-21T23:15:56.346444+0000","last_active":"2024-04-21T23:16:02.480401+0000","last_peered":"2024-04-21T23:16:02.480401+0000","last_clean":"2024-04-21T23:16:02.480401+0000","last_became_active":"2024-04-21T23:15:56.346091+0000","last_became_peered":"2024-04-21T23:15:56.346091+0000","last_unstale":"2024-04-21T23:16:02.480401+0000","last_undegraded":"2024-04-21T23:16:02.480401+0000","last_fullsized":"2024-04-21T23:16:02.480401+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:41:11.144570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.805306+0000","last_change":"2024-04-21T23:16:00.438198+0000","last_active":"2024-04-21T23:16:01.805306+0000","last_peered":"2024-04-21T23:16:01.805306+0000","last_clean":"2024-04-21T23:16:01.805306+0000","last_became_active":"2024-04-21T23:16:00.437767+0000","last_became_peered":"2024-04-21T23:16:00.437767+0000","last_unstale":"2024-04-21T23:16:01.805306+0000","last_undegraded":"2024-04-21T23:16:01.805306+0000","last_fullsized":"2024-04-21T23:16:01.805306+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:35:11.356414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"85'1","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.480399+0000","last_change":"2024-04-21T23:15:55.403896+0000","last_active":"2024-04-21T23:16:02.480399+0000","last_peered":"2024-04-21T23:16:02.480399+0000","last_clean":"2024-04-21T23:16:02.480399+0000","last_became_active":"2024-04-21T23:15:55.403423+0000","last_became_peered":"2024-04-21T23:15:55.403423+0000","last_unstale":"2024-04-21T23:16:02.480399+0000","last_undegraded":"2024-04-21T23:16:02.480399+0000","last_fullsized":"2024-04-21T23:16:02.480399+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:34:44.252259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.259745+0000","last_change":"2024-04-21T23:15:56.351995+0000","last_active":"2024-04-21T23:16:02.259745+0000","last_peered":"2024-04-21T23:16:02.259745+0000","last_clean":"2024-04-21T23:16:02.259745+0000","last_became_active":"2024-04-21T23:15:56.351614+0000","last_became_peered":"2024-04-21T23:15:56.351614+0000","last_unstale":"2024-04-21T23:16:02.259745+0000","last_undegraded":"2024-04-21T23:16:02.259745+0000","last_fullsized":"2024-04-21T23:16:02.259745+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:15:13.774193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.414055+0000","last_change":"2024-04-21T23:16:00.434104+0000","last_active":"2024-04-21T23:16:01.414055+0000","last_peered":"2024-04-21T23:16:01.414055+0000","last_clean":"2024-04-21T23:16:01.414055+0000","last_became_active":"2024-04-21T23:16:00.433681+0000","last_became_peered":"2024-04-21T23:16:00.433681+0000","last_unstale":"2024-04-21T23:16:01.414055+0000","last_undegraded":"2024-04-21T23:16:01.414055+0000","last_fullsized":"2024-04-21T23:16:01.414055+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:10:17.986652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"90'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:21.494544+0000","last_change":"2024-04-21T23:15:55.397536+0000","last_active":"2024-04-21T23:16:21.494544+0000","last_peered":"2024-04-21T23:16:21.494544+0000","last_clean":"2024-04-21T23:16:21.494544+0000","last_became_active":"2024-04-21T23:15:55.397163+0000","last_became_peered":"2024-04-21T23:15:55.397163+0000","last_unstale":"2024-04-21T23:16:21.494544+0000","last_undegraded":"2024-04-21T23:16:21.494544+0000","last_fullsized":"2024-04-21T23:16:21.494544+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:05:27.777804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.748988+0000","last_change":"2024-04-21T23:15:56.351840+0000","last_active":"2024-04-21T23:16:01.748988+0000","last_peered":"2024-04-21T23:16:01.748988+0000","last_clean":"2024-04-21T23:16:01.748988+0000","last_became_active":"2024-04-21T23:15:56.351523+0000","last_became_peered":"2024-04-21T23:15:56.351523+0000","last_unstale":"2024-04-21T23:16:01.748988+0000","last_undegraded":"2024-04-21T23:16:01.748988+0000","last_fullsized":"2024-04-21T23:16:01.748988+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:57:11.714284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.480927+0000","last_change":"2024-04-21T23:16:00.437473+0000","last_active":"2024-04-21T23:16:02.480927+0000","last_peered":"2024-04-21T23:16:02.480927+0000","last_clean":"2024-04-21T23:16:02.480927+0000","last_became_active":"2024-04-21T23:16:00.437014+0000","last_became_peered":"2024-04-21T23:16:00.437014+0000","last_unstale":"2024-04-21T23:16:02.480927+0000","last_undegraded":"2024-04-21T23:16:02.480927+0000","last_fullsized":"2024-04-21T23:16:02.480927+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:37:14.241896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.259306+0000","last_change":"2024-04-21T23:15:55.399213+0000","last_active":"2024-04-21T23:16:02.259306+0000","last_peered":"2024-04-21T23:16:02.259306+0000","last_clean":"2024-04-21T23:16:02.259306+0000","last_became_active":"2024-04-21T23:15:55.398880+0000","last_became_peered":"2024-04-21T23:15:55.398880+0000","last_unstale":"2024-04-21T23:16:02.259306+0000","last_undegraded":"2024-04-21T23:16:02.259306+0000","last_fullsized":"2024-04-21T23:16:02.259306+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:56:32.236862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.481085+0000","last_change":"2024-04-21T23:15:56.349330+0000","last_active":"2024-04-21T23:16:02.481085+0000","last_peered":"2024-04-21T23:16:02.481085+0000","last_clean":"2024-04-21T23:16:02.481085+0000","last_became_active":"2024-04-21T23:15:56.348988+0000","last_became_peered":"2024-04-21T23:15:56.348988+0000","last_unstale":"2024-04-21T23:16:02.481085+0000","last_undegraded":"2024-04-21T23:16:02.481085+0000","last_fullsized":"2024-04-21T23:16:02.481085+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:11:19.650119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.749110+0000","last_change":"2024-04-21T23:16:00.431959+0000","last_active":"2024-04-21T23:16:01.749110+0000","last_peered":"2024-04-21T23:16:01.749110+0000","last_clean":"2024-04-21T23:16:01.749110+0000","last_became_active":"2024-04-21T23:16:00.431570+0000","last_became_peered":"2024-04-21T23:16:00.431570+0000","last_unstale":"2024-04-21T23:16:01.749110+0000","last_undegraded":"2024-04-21T23:16:01.749110+0000","last_fullsized":"2024-04-21T23:16:01.749110+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:02:54.057326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.481011+0000","last_change":"2024-04-21T23:15:55.404505+0000","last_active":"2024-04-21T23:16:02.481011+0000","last_peered":"2024-04-21T23:16:02.481011+0000","last_clean":"2024-04-21T23:16:02.481011+0000","last_became_active":"2024-04-21T23:15:55.403945+0000","last_became_peered":"2024-04-21T23:15:55.403945+0000","last_unstale":"2024-04-21T23:16:02.481011+0000","last_undegraded":"2024-04-21T23:16:02.481011+0000","last_fullsized":"2024-04-21T23:16:02.481011+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:54:37.647597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.129513+0000","last_change":"2024-04-21T23:15:56.356238+0000","last_active":"2024-04-21T23:16:02.129513+0000","last_peered":"2024-04-21T23:16:02.129513+0000","last_clean":"2024-04-21T23:16:02.129513+0000","last_became_active":"2024-04-21T23:15:56.355116+0000","last_became_peered":"2024-04-21T23:15:56.355116+0000","last_unstale":"2024-04-21T23:16:02.129513+0000","last_undegraded":"2024-04-21T23:16:02.129513+0000","last_fullsized":"2024-04-21T23:16:02.129513+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:43:20.468549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.479998+0000","last_change":"2024-04-21T23:16:00.439069+0000","last_active":"2024-04-21T23:16:02.479998+0000","last_peered":"2024-04-21T23:16:02.479998+0000","last_clean":"2024-04-21T23:16:02.479998+0000","last_became_active":"2024-04-21T23:16:00.438510+0000","last_became_peered":"2024-04-21T23:16:00.438510+0000","last_unstale":"2024-04-21T23:16:02.479998+0000","last_undegraded":"2024-04-21T23:16:02.479998+0000","last_fullsized":"2024-04-21T23:16:02.479998+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:41:50.814112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.413979+0000","last_change":"2024-04-21T23:15:55.395682+0000","last_active":"2024-04-21T23:16:01.413979+0000","last_peered":"2024-04-21T23:16:01.413979+0000","last_clean":"2024-04-21T23:16:01.413979+0000","last_became_active":"2024-04-21T23:15:55.395320+0000","last_became_peered":"2024-04-21T23:15:55.395320+0000","last_unstale":"2024-04-21T23:16:01.413979+0000","last_undegraded":"2024-04-21T23:16:01.413979+0000","last_fullsized":"2024-04-21T23:16:01.413979+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:53:24.394310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.480995+0000","last_change":"2024-04-21T23:15:56.357228+0000","last_active":"2024-04-21T23:16:02.480995+0000","last_peered":"2024-04-21T23:16:02.480995+0000","last_clean":"2024-04-21T23:16:02.480995+0000","last_became_active":"2024-04-21T23:15:56.356808+0000","last_became_peered":"2024-04-21T23:15:56.356808+0000","last_unstale":"2024-04-21T23:16:02.480995+0000","last_undegraded":"2024-04-21T23:16:02.480995+0000","last_fullsized":"2024-04-21T23:16:02.480995+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:15:10.728530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.481196+0000","last_change":"2024-04-21T23:16:00.443569+0000","last_active":"2024-04-21T23:16:02.481196+0000","last_peered":"2024-04-21T23:16:02.481196+0000","last_clean":"2024-04-21T23:16:02.481196+0000","last_became_active":"2024-04-21T23:16:00.442972+0000","last_became_peered":"2024-04-21T23:16:00.442972+0000","last_unstale":"2024-04-21T23:16:02.481196+0000","last_undegraded":"2024-04-21T23:16:02.481196+0000","last_fullsized":"2024-04-21T23:16:02.481196+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:35:21.471407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"85'3","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.259847+0000","last_change":"2024-04-21T23:15:55.400134+0000","last_active":"2024-04-21T23:16:02.259847+0000","last_peered":"2024-04-21T23:16:02.259847+0000","last_clean":"2024-04-21T23:16:02.259847+0000","last_became_active":"2024-04-21T23:15:55.399768+0000","last_became_peered":"2024-04-21T23:15:55.399768+0000","last_unstale":"2024-04-21T23:16:02.259847+0000","last_undegraded":"2024-04-21T23:16:02.259847+0000","last_fullsized":"2024-04-21T23:16:02.259847+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:15:09.323035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.129324+0000","last_change":"2024-04-21T23:15:56.357278+0000","last_active":"2024-04-21T23:16:02.129324+0000","last_peered":"2024-04-21T23:16:02.129324+0000","last_clean":"2024-04-21T23:16:02.129324+0000","last_became_active":"2024-04-21T23:15:56.356902+0000","last_became_peered":"2024-04-21T23:15:56.356902+0000","last_unstale":"2024-04-21T23:16:02.129324+0000","last_undegraded":"2024-04-21T23:16:02.129324+0000","last_fullsized":"2024-04-21T23:16:02.129324+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:19:05.269109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.480309+0000","last_change":"2024-04-21T23:16:00.439704+0000","last_active":"2024-04-21T23:16:02.480309+0000","last_peered":"2024-04-21T23:16:02.480309+0000","last_clean":"2024-04-21T23:16:02.480309+0000","last_became_active":"2024-04-21T23:16:00.439286+0000","last_became_peered":"2024-04-21T23:16:00.439286+0000","last_unstale":"2024-04-21T23:16:02.480309+0000","last_undegraded":"2024-04-21T23:16:02.480309+0000","last_fullsized":"2024-04-21T23:16:02.480309+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:08:59.194481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"90'10","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:11.976303+0000","last_change":"2024-04-21T23:15:55.403344+0000","last_active":"2024-04-21T23:16:11.976303+0000","last_peered":"2024-04-21T23:16:11.976303+0000","last_clean":"2024-04-21T23:16:11.976303+0000","last_became_active":"2024-04-21T23:15:55.402876+0000","last_became_peered":"2024-04-21T23:15:55.402876+0000","last_unstale":"2024-04-21T23:16:11.976303+0000","last_undegraded":"2024-04-21T23:16:11.976303+0000","last_fullsized":"2024-04-21T23:16:11.976303+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:55:59.082964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.480698+0000","last_change":"2024-04-21T23:15:56.343897+0000","last_active":"2024-04-21T23:16:02.480698+0000","last_peered":"2024-04-21T23:16:02.480698+0000","last_clean":"2024-04-21T23:16:02.480698+0000","last_became_active":"2024-04-21T23:15:56.343567+0000","last_became_peered":"2024-04-21T23:15:56.343567+0000","last_unstale":"2024-04-21T23:16:02.480698+0000","last_undegraded":"2024-04-21T23:16:02.480698+0000","last_fullsized":"2024-04-21T23:16:02.480698+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:57:07.955722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.260326+0000","last_change":"2024-04-21T23:16:00.432722+0000","last_active":"2024-04-21T23:16:02.260326+0000","last_peered":"2024-04-21T23:16:02.260326+0000","last_clean":"2024-04-21T23:16:02.260326+0000","last_became_active":"2024-04-21T23:16:00.432257+0000","last_became_peered":"2024-04-21T23:16:00.432257+0000","last_unstale":"2024-04-21T23:16:02.260326+0000","last_undegraded":"2024-04-21T23:16:02.260326+0000","last_fullsized":"2024-04-21T23:16:02.260326+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:10:14.804490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"85'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.805207+0000","last_change":"2024-04-21T23:15:55.401300+0000","last_active":"2024-04-21T23:16:01.805207+0000","last_peered":"2024-04-21T23:16:01.805207+0000","last_clean":"2024-04-21T23:16:01.805207+0000","last_became_active":"2024-04-21T23:15:55.400904+0000","last_became_peered":"2024-04-21T23:15:55.400904+0000","last_unstale":"2024-04-21T23:16:01.805207+0000","last_undegraded":"2024-04-21T23:16:01.805207+0000","last_fullsized":"2024-04-21T23:16:01.805207+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:29:20.439936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.413691+0000","last_change":"2024-04-21T23:15:56.351046+0000","last_active":"2024-04-21T23:16:01.413691+0000","last_peered":"2024-04-21T23:16:01.413691+0000","last_clean":"2024-04-21T23:16:01.413691+0000","last_became_active":"2024-04-21T23:15:56.350662+0000","last_became_peered":"2024-04-21T23:15:56.350662+0000","last_unstale":"2024-04-21T23:16:01.413691+0000","last_undegraded":"2024-04-21T23:16:01.413691+0000","last_fullsized":"2024-04-21T23:16:01.413691+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:40:16.482061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.413838+0000","last_change":"2024-04-21T23:16:00.439816+0000","last_active":"2024-04-21T23:16:01.413838+0000","last_peered":"2024-04-21T23:16:01.413838+0000","last_clean":"2024-04-21T23:16:01.413838+0000","last_became_active":"2024-04-21T23:16:00.439330+0000","last_became_peered":"2024-04-21T23:16:00.439330+0000","last_unstale":"2024-04-21T23:16:01.413838+0000","last_undegraded":"2024-04-21T23:16:01.413838+0000","last_fullsized":"2024-04-21T23:16:01.413838+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:36:24.472968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.259833+0000","last_change":"2024-04-21T23:15:55.398588+0000","last_active":"2024-04-21T23:16:02.259833+0000","last_peered":"2024-04-21T23:16:02.259833+0000","last_clean":"2024-04-21T23:16:02.259833+0000","last_became_active":"2024-04-21T23:15:55.398264+0000","last_became_peered":"2024-04-21T23:15:55.398264+0000","last_unstale":"2024-04-21T23:16:02.259833+0000","last_undegraded":"2024-04-21T23:16:02.259833+0000","last_fullsized":"2024-04-21T23:16:02.259833+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:48:09.293481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.413843+0000","last_change":"2024-04-21T23:15:56.359331+0000","last_active":"2024-04-21T23:16:01.413843+0000","last_peered":"2024-04-21T23:16:01.413843+0000","last_clean":"2024-04-21T23:16:01.413843+0000","last_became_active":"2024-04-21T23:15:56.358189+0000","last_became_peered":"2024-04-21T23:15:56.358189+0000","last_unstale":"2024-04-21T23:16:01.413843+0000","last_undegraded":"2024-04-21T23:16:01.413843+0000","last_fullsized":"2024-04-21T23:16:01.413843+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:57:07.343998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.480425+0000","last_change":"2024-04-21T23:16:00.438581+0000","last_active":"2024-04-21T23:16:02.480425+0000","last_peered":"2024-04-21T23:16:02.480425+0000","last_clean":"2024-04-21T23:16:02.480425+0000","last_became_active":"2024-04-21T23:16:00.438219+0000","last_became_peered":"2024-04-21T23:16:00.438219+0000","last_unstale":"2024-04-21T23:16:02.480425+0000","last_undegraded":"2024-04-21T23:16:02.480425+0000","last_fullsized":"2024-04-21T23:16:02.480425+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:47:23.259045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.480391+0000","last_change":"2024-04-21T23:15:55.397503+0000","last_active":"2024-04-21T23:16:02.480391+0000","last_peered":"2024-04-21T23:16:02.480391+0000","last_clean":"2024-04-21T23:16:02.480391+0000","last_became_active":"2024-04-21T23:15:55.397181+0000","last_became_peered":"2024-04-21T23:15:55.397181+0000","last_unstale":"2024-04-21T23:16:02.480391+0000","last_undegraded":"2024-04-21T23:16:02.480391+0000","last_fullsized":"2024-04-21T23:16:02.480391+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:47:08.775783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.413523+0000","last_change":"2024-04-21T23:15:56.350872+0000","last_active":"2024-04-21T23:16:01.413523+0000","last_peered":"2024-04-21T23:16:01.413523+0000","last_clean":"2024-04-21T23:16:01.413523+0000","last_became_active":"2024-04-21T23:15:56.350519+0000","last_became_peered":"2024-04-21T23:15:56.350519+0000","last_unstale":"2024-04-21T23:16:01.413523+0000","last_undegraded":"2024-04-21T23:16:01.413523+0000","last_fullsized":"2024-04-21T23:16:01.413523+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:23:37.753893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.414402+0000","last_change":"2024-04-21T23:16:00.436069+0000","last_active":"2024-04-21T23:16:01.414402+0000","last_peered":"2024-04-21T23:16:01.414402+0000","last_clean":"2024-04-21T23:16:01.414402+0000","last_became_active":"2024-04-21T23:16:00.435617+0000","last_became_peered":"2024-04-21T23:16:00.435617+0000","last_unstale":"2024-04-21T23:16:01.414402+0000","last_undegraded":"2024-04-21T23:16:01.414402+0000","last_fullsized":"2024-04-21T23:16:01.414402+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:21:44.256928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"90'15","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:21.495660+0000","last_change":"2024-04-21T23:15:55.400900+0000","last_active":"2024-04-21T23:16:21.495660+0000","last_peered":"2024-04-21T23:16:21.495660+0000","last_clean":"2024-04-21T23:16:21.495660+0000","last_became_active":"2024-04-21T23:15:55.400558+0000","last_became_peered":"2024-04-21T23:15:55.400558+0000","last_unstale":"2024-04-21T23:16:21.495660+0000","last_undegraded":"2024-04-21T23:16:21.495660+0000","last_fullsized":"2024-04-21T23:16:21.495660+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:30:35.452601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.259321+0000","last_change":"2024-04-21T23:15:56.344526+0000","last_active":"2024-04-21T23:16:02.259321+0000","last_peered":"2024-04-21T23:16:02.259321+0000","last_clean":"2024-04-21T23:16:02.259321+0000","last_became_active":"2024-04-21T23:15:56.344069+0000","last_became_peered":"2024-04-21T23:15:56.344069+0000","last_unstale":"2024-04-21T23:16:02.259321+0000","last_undegraded":"2024-04-21T23:16:02.259321+0000","last_fullsized":"2024-04-21T23:16:02.259321+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:17:04.488828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.259531+0000","last_change":"2024-04-21T23:16:00.433580+0000","last_active":"2024-04-21T23:16:02.259531+0000","last_peered":"2024-04-21T23:16:02.259531+0000","last_clean":"2024-04-21T23:16:02.259531+0000","last_became_active":"2024-04-21T23:16:00.433185+0000","last_became_peered":"2024-04-21T23:16:00.433185+0000","last_unstale":"2024-04-21T23:16:02.259531+0000","last_undegraded":"2024-04-21T23:16:02.259531+0000","last_fullsized":"2024-04-21T23:16:02.259531+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:17:12.317266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'1","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.480812+0000","last_change":"2024-04-21T23:15:55.399032+0000","last_active":"2024-04-21T23:16:02.480812+0000","last_peered":"2024-04-21T23:16:02.480812+0000","last_clean":"2024-04-21T23:16:02.480812+0000","last_became_active":"2024-04-21T23:15:55.398208+0000","last_became_peered":"2024-04-21T23:15:55.398208+0000","last_unstale":"2024-04-21T23:16:02.480812+0000","last_undegraded":"2024-04-21T23:16:02.480812+0000","last_fullsized":"2024-04-21T23:16:02.480812+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:14:59.520571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.413884+0000","last_change":"2024-04-21T23:15:56.355357+0000","last_active":"2024-04-21T23:16:01.413884+0000","last_peered":"2024-04-21T23:16:01.413884+0000","last_clean":"2024-04-21T23:16:01.413884+0000","last_became_active":"2024-04-21T23:15:56.354704+0000","last_became_peered":"2024-04-21T23:15:56.354704+0000","last_unstale":"2024-04-21T23:16:01.413884+0000","last_undegraded":"2024-04-21T23:16:01.413884+0000","last_fullsized":"2024-04-21T23:16:01.413884+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:14:02.368659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.480555+0000","last_change":"2024-04-21T23:16:00.427781+0000","last_active":"2024-04-21T23:16:02.480555+0000","last_peered":"2024-04-21T23:16:02.480555+0000","last_clean":"2024-04-21T23:16:02.480555+0000","last_became_active":"2024-04-21T23:16:00.427244+0000","last_became_peered":"2024-04-21T23:16:00.427244+0000","last_unstale":"2024-04-21T23:16:02.480555+0000","last_undegraded":"2024-04-21T23:16:02.480555+0000","last_fullsized":"2024-04-21T23:16:02.480555+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:21:30.548865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.480455+0000","last_change":"2024-04-21T23:15:55.405687+0000","last_active":"2024-04-21T23:16:02.480455+0000","last_peered":"2024-04-21T23:16:02.480455+0000","last_clean":"2024-04-21T23:16:02.480455+0000","last_became_active":"2024-04-21T23:15:55.405379+0000","last_became_peered":"2024-04-21T23:15:55.405379+0000","last_unstale":"2024-04-21T23:16:02.480455+0000","last_undegraded":"2024-04-21T23:16:02.480455+0000","last_fullsized":"2024-04-21T23:16:02.480455+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:12:44.192092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.481218+0000","last_change":"2024-04-21T23:15:56.348989+0000","last_active":"2024-04-21T23:16:02.481218+0000","last_peered":"2024-04-21T23:16:02.481218+0000","last_clean":"2024-04-21T23:16:02.481218+0000","last_became_active":"2024-04-21T23:15:56.348039+0000","last_became_peered":"2024-04-21T23:15:56.348039+0000","last_unstale":"2024-04-21T23:16:02.481218+0000","last_undegraded":"2024-04-21T23:16:02.481218+0000","last_fullsized":"2024-04-21T23:16:02.481218+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:39:15.577567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.413818+0000","last_change":"2024-04-21T23:16:00.434557+0000","last_active":"2024-04-21T23:16:01.413818+0000","last_peered":"2024-04-21T23:16:01.413818+0000","last_clean":"2024-04-21T23:16:01.413818+0000","last_became_active":"2024-04-21T23:16:00.433737+0000","last_became_peered":"2024-04-21T23:16:00.433737+0000","last_unstale":"2024-04-21T23:16:01.413818+0000","last_undegraded":"2024-04-21T23:16:01.413818+0000","last_fullsized":"2024-04-21T23:16:01.413818+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:38:00.295054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.413600+0000","last_change":"2024-04-21T23:15:55.396792+0000","last_active":"2024-04-21T23:16:01.413600+0000","last_peered":"2024-04-21T23:16:01.413600+0000","last_clean":"2024-04-21T23:16:01.413600+0000","last_became_active":"2024-04-21T23:15:55.395740+0000","last_became_peered":"2024-04-21T23:15:55.395740+0000","last_unstale":"2024-04-21T23:16:01.413600+0000","last_undegraded":"2024-04-21T23:16:01.413600+0000","last_fullsized":"2024-04-21T23:16:01.413600+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:39:22.366310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.480782+0000","last_change":"2024-04-21T23:15:56.346372+0000","last_active":"2024-04-21T23:16:02.480782+0000","last_peered":"2024-04-21T23:16:02.480782+0000","last_clean":"2024-04-21T23:16:02.480782+0000","last_became_active":"2024-04-21T23:15:56.346047+0000","last_became_peered":"2024-04-21T23:15:56.346047+0000","last_unstale":"2024-04-21T23:16:02.480782+0000","last_undegraded":"2024-04-21T23:16:02.480782+0000","last_fullsized":"2024-04-21T23:16:02.480782+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:32:29.684227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.414268+0000","last_change":"2024-04-21T23:16:00.434747+0000","last_active":"2024-04-21T23:16:01.414268+0000","last_peered":"2024-04-21T23:16:01.414268+0000","last_clean":"2024-04-21T23:16:01.414268+0000","last_became_active":"2024-04-21T23:16:00.433950+0000","last_became_peered":"2024-04-21T23:16:00.433950+0000","last_unstale":"2024-04-21T23:16:01.414268+0000","last_undegraded":"2024-04-21T23:16:01.414268+0000","last_fullsized":"2024-04-21T23:16:01.414268+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:02:10.562135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.259303+0000","last_change":"2024-04-21T23:15:56.347115+0000","last_active":"2024-04-21T23:16:02.259303+0000","last_peered":"2024-04-21T23:16:02.259303+0000","last_clean":"2024-04-21T23:16:02.259303+0000","last_became_active":"2024-04-21T23:15:56.346803+0000","last_became_peered":"2024-04-21T23:15:56.346803+0000","last_unstale":"2024-04-21T23:16:02.259303+0000","last_undegraded":"2024-04-21T23:16:02.259303+0000","last_fullsized":"2024-04-21T23:16:02.259303+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:55:11.261095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'4","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:22.505318+0000","last_change":"2024-04-21T23:15:55.394900+0000","last_active":"2024-04-21T23:16:22.505318+0000","last_peered":"2024-04-21T23:16:22.505318+0000","last_clean":"2024-04-21T23:16:22.505318+0000","last_became_active":"2024-04-21T23:15:55.393556+0000","last_became_peered":"2024-04-21T23:15:55.393556+0000","last_unstale":"2024-04-21T23:16:22.505318+0000","last_undegraded":"2024-04-21T23:16:22.505318+0000","last_fullsized":"2024-04-21T23:16:22.505318+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:16:24.691930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.413829+0000","last_change":"2024-04-21T23:16:00.443338+0000","last_active":"2024-04-21T23:16:01.413829+0000","last_peered":"2024-04-21T23:16:01.413829+0000","last_clean":"2024-04-21T23:16:01.413829+0000","last_became_active":"2024-04-21T23:16:00.442855+0000","last_became_peered":"2024-04-21T23:16:00.442855+0000","last_unstale":"2024-04-21T23:16:01.413829+0000","last_undegraded":"2024-04-21T23:16:01.413829+0000","last_fullsized":"2024-04-21T23:16:01.413829+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:04:36.763027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.749041+0000","last_change":"2024-04-21T23:15:56.356788+0000","last_active":"2024-04-21T23:16:01.749041+0000","last_peered":"2024-04-21T23:16:01.749041+0000","last_clean":"2024-04-21T23:16:01.749041+0000","last_became_active":"2024-04-21T23:15:56.356474+0000","last_became_peered":"2024-04-21T23:15:56.356474+0000","last_unstale":"2024-04-21T23:16:01.749041+0000","last_undegraded":"2024-04-21T23:16:01.749041+0000","last_fullsized":"2024-04-21T23:16:01.749041+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:34:33.700897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"90'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:20.489576+0000","last_change":"2024-04-21T23:15:55.395398+0000","last_active":"2024-04-21T23:16:20.489576+0000","last_peered":"2024-04-21T23:16:20.489576+0000","last_clean":"2024-04-21T23:16:20.489576+0000","last_became_active":"2024-04-21T23:15:55.394978+0000","last_became_peered":"2024-04-21T23:15:55.394978+0000","last_unstale":"2024-04-21T23:16:20.489576+0000","last_undegraded":"2024-04-21T23:16:20.489576+0000","last_fullsized":"2024-04-21T23:16:20.489576+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:07:01.266885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":357,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.413325+0000","last_change":"2024-04-21T23:14:42.877959+0000","last_active":"2024-04-21T23:16:01.413325+0000","last_peered":"2024-04-21T23:16:01.413325+0000","last_clean":"2024-04-21T23:16:01.413325+0000","last_became_active":"2024-04-21T23:14:42.877384+0000","last_became_peered":"2024-04-21T23:14:42.877384+0000","last_unstale":"2024-04-21T23:16:01.413325+0000","last_undegraded":"2024-04-21T23:16:01.413325+0000","last_fullsized":"2024-04-21T23:16:01.413325+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:11:15.009349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:11:15.009349+0000","last_clean_scrub_stamp":"2024-04-21T23:11:15.009349+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:18:55.621052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.260050+0000","last_change":"2024-04-21T23:16:00.444576+0000","last_active":"2024-04-21T23:16:02.260050+0000","last_peered":"2024-04-21T23:16:02.260050+0000","last_clean":"2024-04-21T23:16:02.260050+0000","last_became_active":"2024-04-21T23:16:00.444153+0000","last_became_peered":"2024-04-21T23:16:00.444153+0000","last_unstale":"2024-04-21T23:16:02.260050+0000","last_undegraded":"2024-04-21T23:16:02.260050+0000","last_fullsized":"2024-04-21T23:16:02.260050+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:11:48.040383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.413329+0000","last_change":"2024-04-21T23:15:56.349870+0000","last_active":"2024-04-21T23:16:01.413329+0000","last_peered":"2024-04-21T23:16:01.413329+0000","last_clean":"2024-04-21T23:16:01.413329+0000","last_became_active":"2024-04-21T23:15:56.349535+0000","last_became_peered":"2024-04-21T23:15:56.349535+0000","last_unstale":"2024-04-21T23:16:01.413329+0000","last_undegraded":"2024-04-21T23:16:01.413329+0000","last_fullsized":"2024-04-21T23:16:01.413329+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:12:50.702556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.413320+0000","last_change":"2024-04-21T23:15:55.402406+0000","last_active":"2024-04-21T23:16:01.413320+0000","last_peered":"2024-04-21T23:16:01.413320+0000","last_clean":"2024-04-21T23:16:01.413320+0000","last_became_active":"2024-04-21T23:15:55.402075+0000","last_became_peered":"2024-04-21T23:15:55.402075+0000","last_unstale":"2024-04-21T23:16:01.413320+0000","last_undegraded":"2024-04-21T23:16:01.413320+0000","last_fullsized":"2024-04-21T23:16:01.413320+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:44:18.013031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.413814+0000","last_change":"2024-04-21T23:16:00.439206+0000","last_active":"2024-04-21T23:16:01.413814+0000","last_peered":"2024-04-21T23:16:01.413814+0000","last_clean":"2024-04-21T23:16:01.413814+0000","last_became_active":"2024-04-21T23:16:00.438601+0000","last_became_peered":"2024-04-21T23:16:00.438601+0000","last_unstale":"2024-04-21T23:16:01.413814+0000","last_undegraded":"2024-04-21T23:16:01.413814+0000","last_fullsized":"2024-04-21T23:16:01.413814+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:58:46.771456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"90'1","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:20.487401+0000","last_change":"2024-04-21T23:15:55.401154+0000","last_active":"2024-04-21T23:16:20.487401+0000","last_peered":"2024-04-21T23:16:20.487401+0000","last_clean":"2024-04-21T23:16:20.487401+0000","last_became_active":"2024-04-21T23:15:55.400579+0000","last_became_peered":"2024-04-21T23:15:55.400579+0000","last_unstale":"2024-04-21T23:16:20.487401+0000","last_undegraded":"2024-04-21T23:16:20.487401+0000","last_fullsized":"2024-04-21T23:16:20.487401+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:20:03.879660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.413866+0000","last_change":"2024-04-21T23:15:56.349837+0000","last_active":"2024-04-21T23:16:01.413866+0000","last_peered":"2024-04-21T23:16:01.413866+0000","last_clean":"2024-04-21T23:16:01.413866+0000","last_became_active":"2024-04-21T23:15:56.349466+0000","last_became_peered":"2024-04-21T23:15:56.349466+0000","last_unstale":"2024-04-21T23:16:01.413866+0000","last_undegraded":"2024-04-21T23:16:01.413866+0000","last_fullsized":"2024-04-21T23:16:01.413866+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:15:32.675770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.749070+0000","last_change":"2024-04-21T23:16:00.432599+0000","last_active":"2024-04-21T23:16:01.749070+0000","last_peered":"2024-04-21T23:16:01.749070+0000","last_clean":"2024-04-21T23:16:01.749070+0000","last_became_active":"2024-04-21T23:16:00.432194+0000","last_became_peered":"2024-04-21T23:16:00.432194+0000","last_unstale":"2024-04-21T23:16:01.749070+0000","last_undegraded":"2024-04-21T23:16:01.749070+0000","last_fullsized":"2024-04-21T23:16:01.749070+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:10:21.354338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'3","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:21.494349+0000","last_change":"2024-04-21T23:15:55.400328+0000","last_active":"2024-04-21T23:16:21.494349+0000","last_peered":"2024-04-21T23:16:21.494349+0000","last_clean":"2024-04-21T23:16:21.494349+0000","last_became_active":"2024-04-21T23:15:55.399991+0000","last_became_peered":"2024-04-21T23:15:55.399991+0000","last_unstale":"2024-04-21T23:16:21.494349+0000","last_undegraded":"2024-04-21T23:16:21.494349+0000","last_fullsized":"2024-04-21T23:16:21.494349+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:58:54.480720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.749392+0000","last_change":"2024-04-21T23:15:56.358698+0000","last_active":"2024-04-21T23:16:01.749392+0000","last_peered":"2024-04-21T23:16:01.749392+0000","last_clean":"2024-04-21T23:16:01.749392+0000","last_became_active":"2024-04-21T23:15:56.358364+0000","last_became_peered":"2024-04-21T23:15:56.358364+0000","last_unstale":"2024-04-21T23:16:01.749392+0000","last_undegraded":"2024-04-21T23:16:01.749392+0000","last_fullsized":"2024-04-21T23:16:01.749392+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:34:15.281142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.804977+0000","last_change":"2024-04-21T23:16:00.432237+0000","last_active":"2024-04-21T23:16:01.804977+0000","last_peered":"2024-04-21T23:16:01.804977+0000","last_clean":"2024-04-21T23:16:01.804977+0000","last_became_active":"2024-04-21T23:16:00.431745+0000","last_became_peered":"2024-04-21T23:16:00.431745+0000","last_unstale":"2024-04-21T23:16:01.804977+0000","last_undegraded":"2024-04-21T23:16:01.804977+0000","last_fullsized":"2024-04-21T23:16:01.804977+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:25:00.477154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"90'4","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:20.489226+0000","last_change":"2024-04-21T23:15:55.398667+0000","last_active":"2024-04-21T23:16:20.489226+0000","last_peered":"2024-04-21T23:16:20.489226+0000","last_clean":"2024-04-21T23:16:20.489226+0000","last_became_active":"2024-04-21T23:15:55.398290+0000","last_became_peered":"2024-04-21T23:15:55.398290+0000","last_unstale":"2024-04-21T23:16:20.489226+0000","last_undegraded":"2024-04-21T23:16:20.489226+0000","last_fullsized":"2024-04-21T23:16:20.489226+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:57:58.992615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.480829+0000","last_change":"2024-04-21T23:15:56.359518+0000","last_active":"2024-04-21T23:16:02.480829+0000","last_peered":"2024-04-21T23:16:02.480829+0000","last_clean":"2024-04-21T23:16:02.480829+0000","last_became_active":"2024-04-21T23:15:56.358970+0000","last_became_peered":"2024-04-21T23:15:56.358970+0000","last_unstale":"2024-04-21T23:16:02.480829+0000","last_undegraded":"2024-04-21T23:16:02.480829+0000","last_fullsized":"2024-04-21T23:16:02.480829+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:41:22.113877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.480882+0000","last_change":"2024-04-21T23:16:00.443380+0000","last_active":"2024-04-21T23:16:02.480882+0000","last_peered":"2024-04-21T23:16:02.480882+0000","last_clean":"2024-04-21T23:16:02.480882+0000","last_became_active":"2024-04-21T23:16:00.442896+0000","last_became_peered":"2024-04-21T23:16:00.442896+0000","last_unstale":"2024-04-21T23:16:02.480882+0000","last_undegraded":"2024-04-21T23:16:02.480882+0000","last_fullsized":"2024-04-21T23:16:02.480882+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:51:00.089766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"90'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:20.488484+0000","last_change":"2024-04-21T23:15:55.395287+0000","last_active":"2024-04-21T23:16:20.488484+0000","last_peered":"2024-04-21T23:16:20.488484+0000","last_clean":"2024-04-21T23:16:20.488484+0000","last_became_active":"2024-04-21T23:15:55.394908+0000","last_became_peered":"2024-04-21T23:15:55.394908+0000","last_unstale":"2024-04-21T23:16:20.488484+0000","last_undegraded":"2024-04-21T23:16:20.488484+0000","last_fullsized":"2024-04-21T23:16:20.488484+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:57:29.915581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.480282+0000","last_change":"2024-04-21T23:15:56.348016+0000","last_active":"2024-04-21T23:16:02.480282+0000","last_peered":"2024-04-21T23:16:02.480282+0000","last_clean":"2024-04-21T23:16:02.480282+0000","last_became_active":"2024-04-21T23:15:56.347630+0000","last_became_peered":"2024-04-21T23:15:56.347630+0000","last_unstale":"2024-04-21T23:16:02.480282+0000","last_undegraded":"2024-04-21T23:16:02.480282+0000","last_fullsized":"2024-04-21T23:16:02.480282+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:17:27.372880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.480688+0000","last_change":"2024-04-21T23:16:00.442959+0000","last_active":"2024-04-21T23:16:02.480688+0000","last_peered":"2024-04-21T23:16:02.480688+0000","last_clean":"2024-04-21T23:16:02.480688+0000","last_became_active":"2024-04-21T23:16:00.442319+0000","last_became_peered":"2024-04-21T23:16:00.442319+0000","last_unstale":"2024-04-21T23:16:02.480688+0000","last_undegraded":"2024-04-21T23:16:02.480688+0000","last_fullsized":"2024-04-21T23:16:02.480688+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:59:39.868059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.749214+0000","last_change":"2024-04-21T23:15:55.400859+0000","last_active":"2024-04-21T23:16:01.749214+0000","last_peered":"2024-04-21T23:16:01.749214+0000","last_clean":"2024-04-21T23:16:01.749214+0000","last_became_active":"2024-04-21T23:15:55.400460+0000","last_became_peered":"2024-04-21T23:15:55.400460+0000","last_unstale":"2024-04-21T23:16:01.749214+0000","last_undegraded":"2024-04-21T23:16:01.749214+0000","last_fullsized":"2024-04-21T23:16:01.749214+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:31:14.113476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.413541+0000","last_change":"2024-04-21T23:15:56.350354+0000","last_active":"2024-04-21T23:16:01.413541+0000","last_peered":"2024-04-21T23:16:01.413541+0000","last_clean":"2024-04-21T23:16:01.413541+0000","last_became_active":"2024-04-21T23:15:56.349954+0000","last_became_peered":"2024-04-21T23:15:56.349954+0000","last_unstale":"2024-04-21T23:16:01.413541+0000","last_undegraded":"2024-04-21T23:16:01.413541+0000","last_fullsized":"2024-04-21T23:16:01.413541+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:42:59.165610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.480726+0000","last_change":"2024-04-21T23:16:00.444829+0000","last_active":"2024-04-21T23:16:02.480726+0000","last_peered":"2024-04-21T23:16:02.480726+0000","last_clean":"2024-04-21T23:16:02.480726+0000","last_became_active":"2024-04-21T23:16:00.444365+0000","last_became_peered":"2024-04-21T23:16:00.444365+0000","last_unstale":"2024-04-21T23:16:02.480726+0000","last_undegraded":"2024-04-21T23:16:02.480726+0000","last_fullsized":"2024-04-21T23:16:02.480726+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:23:16.747684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.805009+0000","last_change":"2024-04-21T23:15:55.397855+0000","last_active":"2024-04-21T23:16:01.805009+0000","last_peered":"2024-04-21T23:16:01.805009+0000","last_clean":"2024-04-21T23:16:01.805009+0000","last_became_active":"2024-04-21T23:15:55.397498+0000","last_became_peered":"2024-04-21T23:15:55.397498+0000","last_unstale":"2024-04-21T23:16:01.805009+0000","last_undegraded":"2024-04-21T23:16:01.805009+0000","last_fullsized":"2024-04-21T23:16:01.805009+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:45:44.680350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.749348+0000","last_change":"2024-04-21T23:15:56.353023+0000","last_active":"2024-04-21T23:16:01.749348+0000","last_peered":"2024-04-21T23:16:01.749348+0000","last_clean":"2024-04-21T23:16:01.749348+0000","last_became_active":"2024-04-21T23:15:56.352209+0000","last_became_peered":"2024-04-21T23:15:56.352209+0000","last_unstale":"2024-04-21T23:16:01.749348+0000","last_undegraded":"2024-04-21T23:16:01.749348+0000","last_fullsized":"2024-04-21T23:16:01.749348+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:20:58.075627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.480802+0000","last_change":"2024-04-21T23:16:00.429258+0000","last_active":"2024-04-21T23:16:02.480802+0000","last_peered":"2024-04-21T23:16:02.480802+0000","last_clean":"2024-04-21T23:16:02.480802+0000","last_became_active":"2024-04-21T23:16:00.428856+0000","last_became_peered":"2024-04-21T23:16:00.428856+0000","last_unstale":"2024-04-21T23:16:02.480802+0000","last_undegraded":"2024-04-21T23:16:02.480802+0000","last_fullsized":"2024-04-21T23:16:02.480802+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:07:44.608370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"90'4","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:20.489703+0000","last_change":"2024-04-21T23:15:55.399846+0000","last_active":"2024-04-21T23:16:20.489703+0000","last_peered":"2024-04-21T23:16:20.489703+0000","last_clean":"2024-04-21T23:16:20.489703+0000","last_became_active":"2024-04-21T23:15:55.399518+0000","last_became_peered":"2024-04-21T23:15:55.399518+0000","last_unstale":"2024-04-21T23:16:20.489703+0000","last_undegraded":"2024-04-21T23:16:20.489703+0000","last_fullsized":"2024-04-21T23:16:20.489703+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:51:08.293769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.413710+0000","last_change":"2024-04-21T23:15:56.352024+0000","last_active":"2024-04-21T23:16:01.413710+0000","last_peered":"2024-04-21T23:16:01.413710+0000","last_clean":"2024-04-21T23:16:01.413710+0000","last_became_active":"2024-04-21T23:15:56.351052+0000","last_became_peered":"2024-04-21T23:15:56.351052+0000","last_unstale":"2024-04-21T23:16:01.413710+0000","last_undegraded":"2024-04-21T23:16:01.413710+0000","last_fullsized":"2024-04-21T23:16:01.413710+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:36:42.777601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.129068+0000","last_change":"2024-04-21T23:16:00.432371+0000","last_active":"2024-04-21T23:16:02.129068+0000","last_peered":"2024-04-21T23:16:02.129068+0000","last_clean":"2024-04-21T23:16:02.129068+0000","last_became_active":"2024-04-21T23:16:00.431828+0000","last_became_peered":"2024-04-21T23:16:00.431828+0000","last_unstale":"2024-04-21T23:16:02.129068+0000","last_undegraded":"2024-04-21T23:16:02.129068+0000","last_fullsized":"2024-04-21T23:16:02.129068+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:02:06.500075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.805401+0000","last_change":"2024-04-21T23:15:55.407018+0000","last_active":"2024-04-21T23:16:01.805401+0000","last_peered":"2024-04-21T23:16:01.805401+0000","last_clean":"2024-04-21T23:16:01.805401+0000","last_became_active":"2024-04-21T23:15:55.406709+0000","last_became_peered":"2024-04-21T23:15:55.406709+0000","last_unstale":"2024-04-21T23:16:01.805401+0000","last_undegraded":"2024-04-21T23:16:01.805401+0000","last_fullsized":"2024-04-21T23:16:01.805401+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:10:40.138470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.413992+0000","last_change":"2024-04-21T23:15:56.358569+0000","last_active":"2024-04-21T23:16:01.413992+0000","last_peered":"2024-04-21T23:16:01.413992+0000","last_clean":"2024-04-21T23:16:01.413992+0000","last_became_active":"2024-04-21T23:15:56.357533+0000","last_became_peered":"2024-04-21T23:15:56.357533+0000","last_unstale":"2024-04-21T23:16:01.413992+0000","last_undegraded":"2024-04-21T23:16:01.413992+0000","last_fullsized":"2024-04-21T23:16:01.413992+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:59:58.525340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.480215+0000","last_change":"2024-04-21T23:16:00.443981+0000","last_active":"2024-04-21T23:16:02.480215+0000","last_peered":"2024-04-21T23:16:02.480215+0000","last_clean":"2024-04-21T23:16:02.480215+0000","last_became_active":"2024-04-21T23:16:00.443368+0000","last_became_peered":"2024-04-21T23:16:00.443368+0000","last_unstale":"2024-04-21T23:16:02.480215+0000","last_undegraded":"2024-04-21T23:16:02.480215+0000","last_fullsized":"2024-04-21T23:16:02.480215+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:56:48.776540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.260015+0000","last_change":"2024-04-21T23:15:55.397334+0000","last_active":"2024-04-21T23:16:02.260015+0000","last_peered":"2024-04-21T23:16:02.260015+0000","last_clean":"2024-04-21T23:16:02.260015+0000","last_became_active":"2024-04-21T23:15:55.396745+0000","last_became_peered":"2024-04-21T23:15:55.396745+0000","last_unstale":"2024-04-21T23:16:02.260015+0000","last_undegraded":"2024-04-21T23:16:02.260015+0000","last_fullsized":"2024-04-21T23:16:02.260015+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:56:21.042187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.413696+0000","last_change":"2024-04-21T23:15:56.351012+0000","last_active":"2024-04-21T23:16:01.413696+0000","last_peered":"2024-04-21T23:16:01.413696+0000","last_clean":"2024-04-21T23:16:01.413696+0000","last_became_active":"2024-04-21T23:15:56.350042+0000","last_became_peered":"2024-04-21T23:15:56.350042+0000","last_unstale":"2024-04-21T23:16:01.413696+0000","last_undegraded":"2024-04-21T23:16:01.413696+0000","last_fullsized":"2024-04-21T23:16:01.413696+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:47:02.629870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.259218+0000","last_change":"2024-04-21T23:16:00.428775+0000","last_active":"2024-04-21T23:16:02.259218+0000","last_peered":"2024-04-21T23:16:02.259218+0000","last_clean":"2024-04-21T23:16:02.259218+0000","last_became_active":"2024-04-21T23:16:00.428278+0000","last_became_peered":"2024-04-21T23:16:00.428278+0000","last_unstale":"2024-04-21T23:16:02.259218+0000","last_undegraded":"2024-04-21T23:16:02.259218+0000","last_fullsized":"2024-04-21T23:16:02.259218+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:24:04.397387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.414260+0000","last_change":"2024-04-21T23:15:55.395524+0000","last_active":"2024-04-21T23:16:01.414260+0000","last_peered":"2024-04-21T23:16:01.414260+0000","last_clean":"2024-04-21T23:16:01.414260+0000","last_became_active":"2024-04-21T23:15:55.395220+0000","last_became_peered":"2024-04-21T23:15:55.395220+0000","last_unstale":"2024-04-21T23:16:01.414260+0000","last_undegraded":"2024-04-21T23:16:01.414260+0000","last_fullsized":"2024-04-21T23:16:01.414260+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:01:37.517620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.414009+0000","last_change":"2024-04-21T23:15:56.352128+0000","last_active":"2024-04-21T23:16:01.414009+0000","last_peered":"2024-04-21T23:16:01.414009+0000","last_clean":"2024-04-21T23:16:01.414009+0000","last_became_active":"2024-04-21T23:15:56.349349+0000","last_became_peered":"2024-04-21T23:15:56.349349+0000","last_unstale":"2024-04-21T23:16:01.414009+0000","last_undegraded":"2024-04-21T23:16:01.414009+0000","last_fullsized":"2024-04-21T23:16:01.414009+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:12:23.247084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.413681+0000","last_change":"2024-04-21T23:15:56.350830+0000","last_active":"2024-04-21T23:16:01.413681+0000","last_peered":"2024-04-21T23:16:01.413681+0000","last_clean":"2024-04-21T23:16:01.413681+0000","last_became_active":"2024-04-21T23:15:56.350385+0000","last_became_peered":"2024-04-21T23:15:56.350385+0000","last_unstale":"2024-04-21T23:16:01.413681+0000","last_undegraded":"2024-04-21T23:16:01.413681+0000","last_fullsized":"2024-04-21T23:16:01.413681+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:18:48.242375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.805000+0000","last_change":"2024-04-21T23:15:56.350947+0000","last_active":"2024-04-21T23:16:01.805000+0000","last_peered":"2024-04-21T23:16:01.805000+0000","last_clean":"2024-04-21T23:16:01.805000+0000","last_became_active":"2024-04-21T23:15:56.350383+0000","last_became_peered":"2024-04-21T23:15:56.350383+0000","last_unstale":"2024-04-21T23:16:01.805000+0000","last_undegraded":"2024-04-21T23:16:01.805000+0000","last_fullsized":"2024-04-21T23:16:01.805000+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:07:28.879822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.805068+0000","last_change":"2024-04-21T23:15:56.354559+0000","last_active":"2024-04-21T23:16:01.805068+0000","last_peered":"2024-04-21T23:16:01.805068+0000","last_clean":"2024-04-21T23:16:01.805068+0000","last_became_active":"2024-04-21T23:15:56.354205+0000","last_became_peered":"2024-04-21T23:15:56.354205+0000","last_unstale":"2024-04-21T23:16:01.805068+0000","last_undegraded":"2024-04-21T23:16:01.805068+0000","last_fullsized":"2024-04-21T23:16:01.805068+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:33:50.157971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.480393+0000","last_change":"2024-04-21T23:15:56.358395+0000","last_active":"2024-04-21T23:16:02.480393+0000","last_peered":"2024-04-21T23:16:02.480393+0000","last_clean":"2024-04-21T23:16:02.480393+0000","last_became_active":"2024-04-21T23:15:56.358077+0000","last_became_peered":"2024-04-21T23:15:56.358077+0000","last_unstale":"2024-04-21T23:16:02.480393+0000","last_undegraded":"2024-04-21T23:16:02.480393+0000","last_fullsized":"2024-04-21T23:16:02.480393+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:12:43.124588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.413742+0000","last_change":"2024-04-21T23:15:56.347476+0000","last_active":"2024-04-21T23:16:01.413742+0000","last_peered":"2024-04-21T23:16:01.413742+0000","last_clean":"2024-04-21T23:16:01.413742+0000","last_became_active":"2024-04-21T23:15:56.347097+0000","last_became_peered":"2024-04-21T23:15:56.347097+0000","last_unstale":"2024-04-21T23:16:01.413742+0000","last_undegraded":"2024-04-21T23:16:01.413742+0000","last_fullsized":"2024-04-21T23:16:01.413742+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:11:58.715526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.414100+0000","last_change":"2024-04-21T23:15:56.346413+0000","last_active":"2024-04-21T23:16:01.414100+0000","last_peered":"2024-04-21T23:16:01.414100+0000","last_clean":"2024-04-21T23:16:01.414100+0000","last_became_active":"2024-04-21T23:15:56.346056+0000","last_became_peered":"2024-04-21T23:15:56.346056+0000","last_unstale":"2024-04-21T23:16:01.414100+0000","last_undegraded":"2024-04-21T23:16:01.414100+0000","last_fullsized":"2024-04-21T23:16:01.414100+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:47:16.340323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.805595+0000","last_change":"2024-04-21T23:15:56.354876+0000","last_active":"2024-04-21T23:16:01.805595+0000","last_peered":"2024-04-21T23:16:01.805595+0000","last_clean":"2024-04-21T23:16:01.805595+0000","last_became_active":"2024-04-21T23:15:56.354525+0000","last_became_peered":"2024-04-21T23:15:56.354525+0000","last_unstale":"2024-04-21T23:16:01.805595+0000","last_undegraded":"2024-04-21T23:16:01.805595+0000","last_fullsized":"2024-04-21T23:16:01.805595+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:09:23.662664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.749516+0000","last_change":"2024-04-21T23:15:56.358456+0000","last_active":"2024-04-21T23:16:01.749516+0000","last_peered":"2024-04-21T23:16:01.749516+0000","last_clean":"2024-04-21T23:16:01.749516+0000","last_became_active":"2024-04-21T23:15:56.358060+0000","last_became_peered":"2024-04-21T23:15:56.358060+0000","last_unstale":"2024-04-21T23:16:01.749516+0000","last_undegraded":"2024-04-21T23:16:01.749516+0000","last_fullsized":"2024-04-21T23:16:01.749516+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:47:58.928438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.259136+0000","last_change":"2024-04-21T23:15:56.352870+0000","last_active":"2024-04-21T23:16:02.259136+0000","last_peered":"2024-04-21T23:16:02.259136+0000","last_clean":"2024-04-21T23:16:02.259136+0000","last_became_active":"2024-04-21T23:15:56.352267+0000","last_became_peered":"2024-04-21T23:15:56.352267+0000","last_unstale":"2024-04-21T23:16:02.259136+0000","last_undegraded":"2024-04-21T23:16:02.259136+0000","last_fullsized":"2024-04-21T23:16:02.259136+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:32:22.244773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.749230+0000","last_change":"2024-04-21T23:15:56.357468+0000","last_active":"2024-04-21T23:16:01.749230+0000","last_peered":"2024-04-21T23:16:01.749230+0000","last_clean":"2024-04-21T23:16:01.749230+0000","last_became_active":"2024-04-21T23:15:56.357104+0000","last_became_peered":"2024-04-21T23:15:56.357104+0000","last_unstale":"2024-04-21T23:16:01.749230+0000","last_undegraded":"2024-04-21T23:16:01.749230+0000","last_fullsized":"2024-04-21T23:16:01.749230+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:37:36.884036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.129286+0000","last_change":"2024-04-21T23:15:56.352206+0000","last_active":"2024-04-21T23:16:02.129286+0000","last_peered":"2024-04-21T23:16:02.129286+0000","last_clean":"2024-04-21T23:16:02.129286+0000","last_became_active":"2024-04-21T23:15:56.351835+0000","last_became_peered":"2024-04-21T23:15:56.351835+0000","last_unstale":"2024-04-21T23:16:02.129286+0000","last_undegraded":"2024-04-21T23:16:02.129286+0000","last_fullsized":"2024-04-21T23:16:02.129286+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:13:25.363479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.413871+0000","last_change":"2024-04-21T23:15:56.351901+0000","last_active":"2024-04-21T23:16:01.413871+0000","last_peered":"2024-04-21T23:16:01.413871+0000","last_clean":"2024-04-21T23:16:01.413871+0000","last_became_active":"2024-04-21T23:15:56.351545+0000","last_became_peered":"2024-04-21T23:15:56.351545+0000","last_unstale":"2024-04-21T23:16:01.413871+0000","last_undegraded":"2024-04-21T23:16:01.413871+0000","last_fullsized":"2024-04-21T23:16:01.413871+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:37:20.855631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.413774+0000","last_change":"2024-04-21T23:15:56.355773+0000","last_active":"2024-04-21T23:16:01.413774+0000","last_peered":"2024-04-21T23:16:01.413774+0000","last_clean":"2024-04-21T23:16:01.413774+0000","last_became_active":"2024-04-21T23:15:56.355269+0000","last_became_peered":"2024-04-21T23:15:56.355269+0000","last_unstale":"2024-04-21T23:16:01.413774+0000","last_undegraded":"2024-04-21T23:16:01.413774+0000","last_fullsized":"2024-04-21T23:16:01.413774+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:04:54.540817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4864,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":88,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":15768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892350994,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":995328,"data_stored":483870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612631,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36912,"kb_used_data":1576,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708240,"statfs":{"total":95995035648,"available":95957237760,"internally_reserved":0,"allocated":1613824,"data_stored":1078274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808861,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32168,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712984,"statfs":{"total":95995035648,"available":95962095616,"internally_reserved":0,"allocated":1015808,"data_stored":484688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070498,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1015808,"data_stored":484608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332134,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28580,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716572,"statfs":{"total":95995035648,"available":95965769728,"internally_reserved":0,"allocated":1593344,"data_stored":1074820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495660,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":999424,"data_stored":483960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574917,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1007616,"data_stored":487712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738442,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1024000,"data_stored":485348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411392,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1011712,"data_stored":487746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182458,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":1003520,"data_stored":483974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920822,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":999424,"data_stored":483892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757296,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32744,"kb_used_data":1564,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712408,"statfs":{"total":95995035648,"available":95961505792,"internally_reserved":0,"allocated":1601536,"data_stored":1074420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-21T23:16:25.293 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-21T23:16:25.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.429+0000 7f830c415640 1 -- 172.21.15.40:0/3235625796 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8304102250 msgr2=0x7f8304108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:25.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.429+0000 7f830c415640 1 --2- 172.21.15.40:0/3235625796 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8304102250 0x7f8304108ae0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f82fc009ca0 tx=0x7f82fc02c0f0 comp rx=0 tx=0).stop 2024-04-21T23:16:25.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.429+0000 7f830c415640 1 -- 172.21.15.40:0/3235625796 shutdown_connections 2024-04-21T23:16:25.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.429+0000 7f830c415640 1 --2- 172.21.15.40:0/3235625796 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8304101890 0x7f8304101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:25.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.429+0000 7f830c415640 1 --2- 172.21.15.40:0/3235625796 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8304102250 0x7f8304108ae0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:25.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.429+0000 7f830c415640 1 --2- 172.21.15.40:0/3235625796 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f83041006a0 0x7f8304100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:25.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.429+0000 7f830c415640 1 -- 172.21.15.40:0/3235625796 >> 172.21.15.40:0/3235625796 conn(0x7f83040fbc30 msgr2=0x7f83040fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:25.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.429+0000 7f830c415640 1 -- 172.21.15.40:0/3235625796 shutdown_connections 2024-04-21T23:16:25.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.429+0000 7f830c415640 1 -- 172.21.15.40:0/3235625796 wait complete. 2024-04-21T23:16:25.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.430+0000 7f830c415640 1 Processor -- start 2024-04-21T23:16:25.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.430+0000 7f830c415640 1 -- start start 2024-04-21T23:16:25.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.430+0000 7f830c415640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f83041006a0 0x7f8304197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:25.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.430+0000 7f830c415640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8304101890 0x7f8304197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:25.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.430+0000 7f830c415640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8304102250 0x7f830419eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:25.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.430+0000 7f830c415640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f830410c320 con 0x7f83041006a0 2024-04-21T23:16:25.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.430+0000 7f830c415640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f830410c1a0 con 0x7f8304102250 2024-04-21T23:16:25.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.430+0000 7f830c415640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f830410c4a0 con 0x7f8304101890 2024-04-21T23:16:25.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.430+0000 7f830a1b1640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f83041006a0 0x7f8304197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:25.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.430+0000 7f830a1b1640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f83041006a0 0x7f8304197a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43536/0 (socket says 172.21.15.40:43536) 2024-04-21T23:16:25.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.430+0000 7f830a1b1640 1 -- 172.21.15.40:0/1241095681 learned_addr learned my addr 172.21.15.40:0/1241095681 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:16:25.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.430+0000 7f830a9b2640 1 --2- 172.21.15.40:0/1241095681 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8304102250 0x7f830419eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:25.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.430+0000 7f830a1b1640 1 -- 172.21.15.40:0/1241095681 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8304101890 msgr2=0x7f8304197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:25.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.430+0000 7f830a1b1640 1 --2- 172.21.15.40:0/1241095681 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8304101890 0x7f8304197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:25.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.430+0000 7f830a1b1640 1 -- 172.21.15.40:0/1241095681 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8304102250 msgr2=0x7f830419eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:25.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.430+0000 7f830a1b1640 1 --2- 172.21.15.40:0/1241095681 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8304102250 0x7f830419eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:25.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.430+0000 7f830a1b1640 1 -- 172.21.15.40:0/1241095681 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f830419f660 con 0x7f83041006a0 2024-04-21T23:16:25.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.431+0000 7f830a9b2640 1 --2- 172.21.15.40:0/1241095681 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8304102250 0x7f830419eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:16:25.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.431+0000 7f830a1b1640 1 --2- 172.21.15.40:0/1241095681 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f83041006a0 0x7f8304197a90 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f830000d950 tx=0x7f830000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:25.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.431+0000 7f82fb7fe640 1 -- 172.21.15.40:0/1241095681 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8300017070 con 0x7f83041006a0 2024-04-21T23:16:25.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.431+0000 7f82fb7fe640 1 -- 172.21.15.40:0/1241095681 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8300009c30 con 0x7f83041006a0 2024-04-21T23:16:25.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.431+0000 7f830c415640 1 -- 172.21.15.40:0/1241095681 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f830419f950 con 0x7f83041006a0 2024-04-21T23:16:25.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.431+0000 7f82fb7fe640 1 -- 172.21.15.40:0/1241095681 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f830000f600 con 0x7f83041006a0 2024-04-21T23:16:25.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.431+0000 7f830c415640 1 -- 172.21.15.40:0/1241095681 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f830419fe50 con 0x7f83041006a0 2024-04-21T23:16:25.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.432+0000 7f830c415640 1 -- 172.21.15.40:0/1241095681 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83041a0080 con 0x7f83041006a0 2024-04-21T23:16:25.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.433+0000 7f82fb7fe640 1 -- 172.21.15.40:0/1241095681 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f830001a070 con 0x7f83041006a0 2024-04-21T23:16:25.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.433+0000 7f82fb7fe640 1 --2- 172.21.15.40:0/1241095681 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f82f00a2c20 0x7f82f00a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:25.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.434+0000 7f82fb7fe640 1 -- 172.21.15.40:0/1241095681 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f83000cef60 con 0x7f83041006a0 2024-04-21T23:16:25.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.434+0000 7f83099b0640 1 --2- 172.21.15.40:0/1241095681 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f82f00a2c20 0x7f82f00a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:25.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.434+0000 7f83099b0640 1 --2- 172.21.15.40:0/1241095681 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f82f00a2c20 0x7f82f00a50e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f82f4005fd0 tx=0x7f82f4005c70 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:25.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.438+0000 7f82fb7fe640 1 -- 172.21.15.40:0/1241095681 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f8300009de0 con 0x7f83041006a0 2024-04-21T23:16:25.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.618+0000 7f830c415640 1 -- 172.21.15.40:0/1241095681 --> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f83040fde20 con 0x7f82f00a2c20 2024-04-21T23:16:25.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.627+0000 7f82fb7fe640 1 -- 172.21.15.40:0/1241095681 <== mgr.14148 v2:172.21.15.40:6800/2500760788 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340266 (secure 0 0 0) 0x7f83040fde20 con 0x7f82f00a2c20 2024-04-21T23:16:25.630 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:16:25.631 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-21T23:16:25.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.630+0000 7f830c415640 1 -- 172.21.15.40:0/1241095681 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f82f00a2c20 msgr2=0x7f82f00a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:25.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.630+0000 7f830c415640 1 --2- 172.21.15.40:0/1241095681 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f82f00a2c20 0x7f82f00a50e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f82f4005fd0 tx=0x7f82f4005c70 comp rx=0 tx=0).stop 2024-04-21T23:16:25.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.630+0000 7f830c415640 1 -- 172.21.15.40:0/1241095681 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f83041006a0 msgr2=0x7f8304197a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:25.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.630+0000 7f830c415640 1 --2- 172.21.15.40:0/1241095681 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f83041006a0 0x7f8304197a90 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f830000d950 tx=0x7f830000de10 comp rx=0 tx=0).stop 2024-04-21T23:16:25.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.630+0000 7f830c415640 1 -- 172.21.15.40:0/1241095681 shutdown_connections 2024-04-21T23:16:25.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.630+0000 7f830c415640 1 --2- 172.21.15.40:0/1241095681 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f82f00a2c20 0x7f82f00a50e0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:25.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.630+0000 7f830c415640 1 --2- 172.21.15.40:0/1241095681 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8304102250 0x7f830419eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:25.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.630+0000 7f830c415640 1 --2- 172.21.15.40:0/1241095681 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8304101890 0x7f8304197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:25.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.630+0000 7f830c415640 1 --2- 172.21.15.40:0/1241095681 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f83041006a0 0x7f8304197a90 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:25.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.630+0000 7f830c415640 1 -- 172.21.15.40:0/1241095681 >> 172.21.15.40:0/1241095681 conn(0x7f83040fbc30 msgr2=0x7f83040fd710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:25.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.630+0000 7f830c415640 1 -- 172.21.15.40:0/1241095681 shutdown_connections 2024-04-21T23:16:25.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:25.630+0000 7f830c415640 1 -- 172.21.15.40:0/1241095681 wait complete. 2024-04-21T23:16:25.639 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:25 smithi040 ceph-mon[27984]: pgmap v307: 129 pgs: 129 active+clean; 581 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 682 B/s rd, 1.5 KiB/s wr, 4 op/s 2024-04-21T23:16:25.639 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:25 smithi040 ceph-mon[27984]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-21T23:16:25.640 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:25 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/3938550799' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:16:25.640 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:25 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/3245186211' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:16:25.640 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:25 smithi040 ceph-mon[27984]: from='client.14934 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T23:16:25.641 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":307,"stamp":"2024-04-21T23:16:24.400700+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595232,"num_objects":41,"num_object_clones":0,"num_object_copies":123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":144,"num_read_kb":127,"num_write":269,"num_write_kb":2973,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":211,"ondisk_log_size":211,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":358576,"kb_used_data":13556,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124583248,"statfs":{"total":1151940427776,"available":1151573245952,"internally_reserved":0,"allocated":13881344,"data_stored":7593312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":353154362},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":365,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":41,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003481"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.414140+0000","last_change":"2024-04-21T23:15:56.349368+0000","last_active":"2024-04-21T23:16:01.414140+0000","last_peered":"2024-04-21T23:16:01.414140+0000","last_clean":"2024-04-21T23:16:01.414140+0000","last_became_active":"2024-04-21T23:15:56.345743+0000","last_became_peered":"2024-04-21T23:15:56.345743+0000","last_unstale":"2024-04-21T23:16:01.414140+0000","last_undegraded":"2024-04-21T23:16:01.414140+0000","last_fullsized":"2024-04-21T23:16:01.414140+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:57:21.574012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.480927+0000","last_change":"2024-04-21T23:15:56.347231+0000","last_active":"2024-04-21T23:16:02.480927+0000","last_peered":"2024-04-21T23:16:02.480927+0000","last_clean":"2024-04-21T23:16:02.480927+0000","last_became_active":"2024-04-21T23:15:56.346847+0000","last_became_peered":"2024-04-21T23:15:56.346847+0000","last_unstale":"2024-04-21T23:16:02.480927+0000","last_undegraded":"2024-04-21T23:16:02.480927+0000","last_fullsized":"2024-04-21T23:16:02.480927+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:20:15.884789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.480695+0000","last_change":"2024-04-21T23:15:56.354897+0000","last_active":"2024-04-21T23:16:02.480695+0000","last_peered":"2024-04-21T23:16:02.480695+0000","last_clean":"2024-04-21T23:16:02.480695+0000","last_became_active":"2024-04-21T23:15:56.354494+0000","last_became_peered":"2024-04-21T23:15:56.354494+0000","last_unstale":"2024-04-21T23:16:02.480695+0000","last_undegraded":"2024-04-21T23:16:02.480695+0000","last_fullsized":"2024-04-21T23:16:02.480695+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:51:11.865310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.480899+0000","last_change":"2024-04-21T23:15:56.355184+0000","last_active":"2024-04-21T23:16:02.480899+0000","last_peered":"2024-04-21T23:16:02.480899+0000","last_clean":"2024-04-21T23:16:02.480899+0000","last_became_active":"2024-04-21T23:15:56.354748+0000","last_became_peered":"2024-04-21T23:15:56.354748+0000","last_unstale":"2024-04-21T23:16:02.480899+0000","last_undegraded":"2024-04-21T23:16:02.480899+0000","last_fullsized":"2024-04-21T23:16:02.480899+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:46:24.867488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.480384+0000","last_change":"2024-04-21T23:15:56.349446+0000","last_active":"2024-04-21T23:16:02.480384+0000","last_peered":"2024-04-21T23:16:02.480384+0000","last_clean":"2024-04-21T23:16:02.480384+0000","last_became_active":"2024-04-21T23:15:56.349050+0000","last_became_peered":"2024-04-21T23:15:56.349050+0000","last_unstale":"2024-04-21T23:16:02.480384+0000","last_undegraded":"2024-04-21T23:16:02.480384+0000","last_fullsized":"2024-04-21T23:16:02.480384+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:15:25.489044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.480430+0000","last_change":"2024-04-21T23:15:56.356095+0000","last_active":"2024-04-21T23:16:02.480430+0000","last_peered":"2024-04-21T23:16:02.480430+0000","last_clean":"2024-04-21T23:16:02.480430+0000","last_became_active":"2024-04-21T23:15:56.355754+0000","last_became_peered":"2024-04-21T23:15:56.355754+0000","last_unstale":"2024-04-21T23:16:02.480430+0000","last_undegraded":"2024-04-21T23:16:02.480430+0000","last_fullsized":"2024-04-21T23:16:02.480430+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:00:27.296457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.413381+0000","last_change":"2024-04-21T23:15:56.349733+0000","last_active":"2024-04-21T23:16:01.413381+0000","last_peered":"2024-04-21T23:16:01.413381+0000","last_clean":"2024-04-21T23:16:01.413381+0000","last_became_active":"2024-04-21T23:15:56.347820+0000","last_became_peered":"2024-04-21T23:15:56.347820+0000","last_unstale":"2024-04-21T23:16:01.413381+0000","last_undegraded":"2024-04-21T23:16:01.413381+0000","last_fullsized":"2024-04-21T23:16:01.413381+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:31:34.453626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.749212+0000","last_change":"2024-04-21T23:15:56.359385+0000","last_active":"2024-04-21T23:16:01.749212+0000","last_peered":"2024-04-21T23:16:01.749212+0000","last_clean":"2024-04-21T23:16:01.749212+0000","last_became_active":"2024-04-21T23:15:56.359074+0000","last_became_peered":"2024-04-21T23:15:56.359074+0000","last_unstale":"2024-04-21T23:16:01.749212+0000","last_undegraded":"2024-04-21T23:16:01.749212+0000","last_fullsized":"2024-04-21T23:16:01.749212+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:40:23.904592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.129548+0000","last_change":"2024-04-21T23:15:56.350886+0000","last_active":"2024-04-21T23:16:02.129548+0000","last_peered":"2024-04-21T23:16:02.129548+0000","last_clean":"2024-04-21T23:16:02.129548+0000","last_became_active":"2024-04-21T23:15:56.350531+0000","last_became_peered":"2024-04-21T23:15:56.350531+0000","last_unstale":"2024-04-21T23:16:02.129548+0000","last_undegraded":"2024-04-21T23:16:02.129548+0000","last_fullsized":"2024-04-21T23:16:02.129548+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:13:12.354473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.481404+0000","last_change":"2024-04-21T23:15:56.347376+0000","last_active":"2024-04-21T23:16:02.481404+0000","last_peered":"2024-04-21T23:16:02.481404+0000","last_clean":"2024-04-21T23:16:02.481404+0000","last_became_active":"2024-04-21T23:15:56.347016+0000","last_became_peered":"2024-04-21T23:15:56.347016+0000","last_unstale":"2024-04-21T23:16:02.481404+0000","last_undegraded":"2024-04-21T23:16:02.481404+0000","last_fullsized":"2024-04-21T23:16:02.481404+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:49:46.722931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.413990+0000","last_change":"2024-04-21T23:15:56.351754+0000","last_active":"2024-04-21T23:16:01.413990+0000","last_peered":"2024-04-21T23:16:01.413990+0000","last_clean":"2024-04-21T23:16:01.413990+0000","last_became_active":"2024-04-21T23:15:56.349860+0000","last_became_peered":"2024-04-21T23:15:56.349860+0000","last_unstale":"2024-04-21T23:16:01.413990+0000","last_undegraded":"2024-04-21T23:16:01.413990+0000","last_fullsized":"2024-04-21T23:16:01.413990+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:48:41.469708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.481302+0000","last_change":"2024-04-21T23:15:56.345177+0000","last_active":"2024-04-21T23:16:02.481302+0000","last_peered":"2024-04-21T23:16:02.481302+0000","last_clean":"2024-04-21T23:16:02.481302+0000","last_became_active":"2024-04-21T23:15:56.344791+0000","last_became_peered":"2024-04-21T23:15:56.344791+0000","last_unstale":"2024-04-21T23:16:02.481302+0000","last_undegraded":"2024-04-21T23:16:02.481302+0000","last_fullsized":"2024-04-21T23:16:02.481302+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:33:33.418383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.414014+0000","last_change":"2024-04-21T23:15:56.334473+0000","last_active":"2024-04-21T23:16:01.414014+0000","last_peered":"2024-04-21T23:16:01.414014+0000","last_clean":"2024-04-21T23:16:01.414014+0000","last_became_active":"2024-04-21T23:15:56.334219+0000","last_became_peered":"2024-04-21T23:15:56.334219+0000","last_unstale":"2024-04-21T23:16:01.414014+0000","last_undegraded":"2024-04-21T23:16:01.414014+0000","last_fullsized":"2024-04-21T23:16:01.414014+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:22:16.547760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.480474+0000","last_change":"2024-04-21T23:15:56.354544+0000","last_active":"2024-04-21T23:16:02.480474+0000","last_peered":"2024-04-21T23:16:02.480474+0000","last_clean":"2024-04-21T23:16:02.480474+0000","last_became_active":"2024-04-21T23:15:56.354206+0000","last_became_peered":"2024-04-21T23:15:56.354206+0000","last_unstale":"2024-04-21T23:16:02.480474+0000","last_undegraded":"2024-04-21T23:16:02.480474+0000","last_fullsized":"2024-04-21T23:16:02.480474+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:37:19.985422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.480301+0000","last_change":"2024-04-21T23:15:56.357447+0000","last_active":"2024-04-21T23:16:02.480301+0000","last_peered":"2024-04-21T23:16:02.480301+0000","last_clean":"2024-04-21T23:16:02.480301+0000","last_became_active":"2024-04-21T23:15:56.355850+0000","last_became_peered":"2024-04-21T23:15:56.355850+0000","last_unstale":"2024-04-21T23:16:02.480301+0000","last_undegraded":"2024-04-21T23:16:02.480301+0000","last_fullsized":"2024-04-21T23:16:02.480301+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:48:56.648994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.128924+0000","last_change":"2024-04-21T23:15:56.353712+0000","last_active":"2024-04-21T23:16:02.128924+0000","last_peered":"2024-04-21T23:16:02.128924+0000","last_clean":"2024-04-21T23:16:02.128924+0000","last_became_active":"2024-04-21T23:15:56.353388+0000","last_became_peered":"2024-04-21T23:15:56.353388+0000","last_unstale":"2024-04-21T23:16:02.128924+0000","last_undegraded":"2024-04-21T23:16:02.128924+0000","last_fullsized":"2024-04-21T23:16:02.128924+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:43:33.581222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.413993+0000","last_change":"2024-04-21T23:15:56.347041+0000","last_active":"2024-04-21T23:16:01.413993+0000","last_peered":"2024-04-21T23:16:01.413993+0000","last_clean":"2024-04-21T23:16:01.413993+0000","last_became_active":"2024-04-21T23:15:56.346630+0000","last_became_peered":"2024-04-21T23:15:56.346630+0000","last_unstale":"2024-04-21T23:16:01.413993+0000","last_undegraded":"2024-04-21T23:16:01.413993+0000","last_fullsized":"2024-04-21T23:16:01.413993+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:57:42.553666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.480700+0000","last_change":"2024-04-21T23:15:56.353809+0000","last_active":"2024-04-21T23:16:02.480700+0000","last_peered":"2024-04-21T23:16:02.480700+0000","last_clean":"2024-04-21T23:16:02.480700+0000","last_became_active":"2024-04-21T23:15:56.353469+0000","last_became_peered":"2024-04-21T23:15:56.353469+0000","last_unstale":"2024-04-21T23:16:02.480700+0000","last_undegraded":"2024-04-21T23:16:02.480700+0000","last_fullsized":"2024-04-21T23:16:02.480700+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:11:14.136845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.805461+0000","last_change":"2024-04-21T23:15:56.354500+0000","last_active":"2024-04-21T23:16:01.805461+0000","last_peered":"2024-04-21T23:16:01.805461+0000","last_clean":"2024-04-21T23:16:01.805461+0000","last_became_active":"2024-04-21T23:15:56.353893+0000","last_became_peered":"2024-04-21T23:15:56.353893+0000","last_unstale":"2024-04-21T23:16:01.805461+0000","last_undegraded":"2024-04-21T23:16:01.805461+0000","last_fullsized":"2024-04-21T23:16:01.805461+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:14:38.121448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.805252+0000","last_change":"2024-04-21T23:16:00.438245+0000","last_active":"2024-04-21T23:16:01.805252+0000","last_peered":"2024-04-21T23:16:01.805252+0000","last_clean":"2024-04-21T23:16:01.805252+0000","last_became_active":"2024-04-21T23:16:00.437755+0000","last_became_peered":"2024-04-21T23:16:00.437755+0000","last_unstale":"2024-04-21T23:16:01.805252+0000","last_undegraded":"2024-04-21T23:16:01.805252+0000","last_fullsized":"2024-04-21T23:16:01.805252+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:33:37.273223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"90'9","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:21.496232+0000","last_change":"2024-04-21T23:15:55.402553+0000","last_active":"2024-04-21T23:16:21.496232+0000","last_peered":"2024-04-21T23:16:21.496232+0000","last_clean":"2024-04-21T23:16:21.496232+0000","last_became_active":"2024-04-21T23:15:55.402192+0000","last_became_peered":"2024-04-21T23:15:55.402192+0000","last_unstale":"2024-04-21T23:16:21.496232+0000","last_undegraded":"2024-04-21T23:16:21.496232+0000","last_fullsized":"2024-04-21T23:16:21.496232+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:24:57.467906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.749466+0000","last_change":"2024-04-21T23:15:56.358797+0000","last_active":"2024-04-21T23:16:01.749466+0000","last_peered":"2024-04-21T23:16:01.749466+0000","last_clean":"2024-04-21T23:16:01.749466+0000","last_became_active":"2024-04-21T23:15:56.357997+0000","last_became_peered":"2024-04-21T23:15:56.357997+0000","last_unstale":"2024-04-21T23:16:01.749466+0000","last_undegraded":"2024-04-21T23:16:01.749466+0000","last_fullsized":"2024-04-21T23:16:01.749466+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:20:50.891506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.480298+0000","last_change":"2024-04-21T23:16:00.443368+0000","last_active":"2024-04-21T23:16:02.480298+0000","last_peered":"2024-04-21T23:16:02.480298+0000","last_clean":"2024-04-21T23:16:02.480298+0000","last_became_active":"2024-04-21T23:16:00.442940+0000","last_became_peered":"2024-04-21T23:16:00.442940+0000","last_unstale":"2024-04-21T23:16:02.480298+0000","last_undegraded":"2024-04-21T23:16:02.480298+0000","last_fullsized":"2024-04-21T23:16:02.480298+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:17:15.706460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"85'1","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.749541+0000","last_change":"2024-04-21T23:15:55.405216+0000","last_active":"2024-04-21T23:16:01.749541+0000","last_peered":"2024-04-21T23:16:01.749541+0000","last_clean":"2024-04-21T23:16:01.749541+0000","last_became_active":"2024-04-21T23:15:55.404844+0000","last_became_peered":"2024-04-21T23:15:55.404844+0000","last_unstale":"2024-04-21T23:16:01.749541+0000","last_undegraded":"2024-04-21T23:16:01.749541+0000","last_fullsized":"2024-04-21T23:16:01.749541+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:56:28.691844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.480884+0000","last_change":"2024-04-21T23:15:56.355789+0000","last_active":"2024-04-21T23:16:02.480884+0000","last_peered":"2024-04-21T23:16:02.480884+0000","last_clean":"2024-04-21T23:16:02.480884+0000","last_became_active":"2024-04-21T23:15:56.354972+0000","last_became_peered":"2024-04-21T23:15:56.354972+0000","last_unstale":"2024-04-21T23:16:02.480884+0000","last_undegraded":"2024-04-21T23:16:02.480884+0000","last_fullsized":"2024-04-21T23:16:02.480884+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:11:36.831453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.480637+0000","last_change":"2024-04-21T23:16:00.443241+0000","last_active":"2024-04-21T23:16:02.480637+0000","last_peered":"2024-04-21T23:16:02.480637+0000","last_clean":"2024-04-21T23:16:02.480637+0000","last_became_active":"2024-04-21T23:16:00.442820+0000","last_became_peered":"2024-04-21T23:16:00.442820+0000","last_unstale":"2024-04-21T23:16:02.480637+0000","last_undegraded":"2024-04-21T23:16:02.480637+0000","last_fullsized":"2024-04-21T23:16:02.480637+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:28:04.032621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"90'4","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:21.492858+0000","last_change":"2024-04-21T23:15:55.403311+0000","last_active":"2024-04-21T23:16:21.492858+0000","last_peered":"2024-04-21T23:16:21.492858+0000","last_clean":"2024-04-21T23:16:21.492858+0000","last_became_active":"2024-04-21T23:15:55.402943+0000","last_became_peered":"2024-04-21T23:15:55.402943+0000","last_unstale":"2024-04-21T23:16:21.492858+0000","last_undegraded":"2024-04-21T23:16:21.492858+0000","last_fullsized":"2024-04-21T23:16:21.492858+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:16:25.961089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.129201+0000","last_change":"2024-04-21T23:15:56.354462+0000","last_active":"2024-04-21T23:16:02.129201+0000","last_peered":"2024-04-21T23:16:02.129201+0000","last_clean":"2024-04-21T23:16:02.129201+0000","last_became_active":"2024-04-21T23:15:56.354134+0000","last_became_peered":"2024-04-21T23:15:56.354134+0000","last_unstale":"2024-04-21T23:16:02.129201+0000","last_undegraded":"2024-04-21T23:16:02.129201+0000","last_fullsized":"2024-04-21T23:16:02.129201+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:27:40.809483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.480246+0000","last_change":"2024-04-21T23:16:00.427717+0000","last_active":"2024-04-21T23:16:02.480246+0000","last_peered":"2024-04-21T23:16:02.480246+0000","last_clean":"2024-04-21T23:16:02.480246+0000","last_became_active":"2024-04-21T23:16:00.427296+0000","last_became_peered":"2024-04-21T23:16:00.427296+0000","last_unstale":"2024-04-21T23:16:02.480246+0000","last_undegraded":"2024-04-21T23:16:02.480246+0000","last_fullsized":"2024-04-21T23:16:02.480246+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:03:41.650045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.480762+0000","last_change":"2024-04-21T23:15:55.398641+0000","last_active":"2024-04-21T23:16:02.480762+0000","last_peered":"2024-04-21T23:16:02.480762+0000","last_clean":"2024-04-21T23:16:02.480762+0000","last_became_active":"2024-04-21T23:15:55.397712+0000","last_became_peered":"2024-04-21T23:15:55.397712+0000","last_unstale":"2024-04-21T23:16:02.480762+0000","last_undegraded":"2024-04-21T23:16:02.480762+0000","last_fullsized":"2024-04-21T23:16:02.480762+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:28:09.266740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.259858+0000","last_change":"2024-04-21T23:15:56.345359+0000","last_active":"2024-04-21T23:16:02.259858+0000","last_peered":"2024-04-21T23:16:02.259858+0000","last_clean":"2024-04-21T23:16:02.259858+0000","last_became_active":"2024-04-21T23:15:56.343652+0000","last_became_peered":"2024-04-21T23:15:56.343652+0000","last_unstale":"2024-04-21T23:16:02.259858+0000","last_undegraded":"2024-04-21T23:16:02.259858+0000","last_fullsized":"2024-04-21T23:16:02.259858+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:18:10.057999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.805468+0000","last_change":"2024-04-21T23:16:00.430640+0000","last_active":"2024-04-21T23:16:01.805468+0000","last_peered":"2024-04-21T23:16:01.805468+0000","last_clean":"2024-04-21T23:16:01.805468+0000","last_became_active":"2024-04-21T23:16:00.430141+0000","last_became_peered":"2024-04-21T23:16:00.430141+0000","last_unstale":"2024-04-21T23:16:01.805468+0000","last_undegraded":"2024-04-21T23:16:01.805468+0000","last_fullsized":"2024-04-21T23:16:01.805468+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:41:13.749780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"85'1","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.259505+0000","last_change":"2024-04-21T23:15:55.399823+0000","last_active":"2024-04-21T23:16:02.259505+0000","last_peered":"2024-04-21T23:16:02.259505+0000","last_clean":"2024-04-21T23:16:02.259505+0000","last_became_active":"2024-04-21T23:15:55.399490+0000","last_became_peered":"2024-04-21T23:15:55.399490+0000","last_unstale":"2024-04-21T23:16:02.259505+0000","last_undegraded":"2024-04-21T23:16:02.259505+0000","last_fullsized":"2024-04-21T23:16:02.259505+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:54:38.124841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.480401+0000","last_change":"2024-04-21T23:15:56.346444+0000","last_active":"2024-04-21T23:16:02.480401+0000","last_peered":"2024-04-21T23:16:02.480401+0000","last_clean":"2024-04-21T23:16:02.480401+0000","last_became_active":"2024-04-21T23:15:56.346091+0000","last_became_peered":"2024-04-21T23:15:56.346091+0000","last_unstale":"2024-04-21T23:16:02.480401+0000","last_undegraded":"2024-04-21T23:16:02.480401+0000","last_fullsized":"2024-04-21T23:16:02.480401+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:41:11.144570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.805306+0000","last_change":"2024-04-21T23:16:00.438198+0000","last_active":"2024-04-21T23:16:01.805306+0000","last_peered":"2024-04-21T23:16:01.805306+0000","last_clean":"2024-04-21T23:16:01.805306+0000","last_became_active":"2024-04-21T23:16:00.437767+0000","last_became_peered":"2024-04-21T23:16:00.437767+0000","last_unstale":"2024-04-21T23:16:01.805306+0000","last_undegraded":"2024-04-21T23:16:01.805306+0000","last_fullsized":"2024-04-21T23:16:01.805306+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:35:11.356414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"85'1","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.480399+0000","last_change":"2024-04-21T23:15:55.403896+0000","last_active":"2024-04-21T23:16:02.480399+0000","last_peered":"2024-04-21T23:16:02.480399+0000","last_clean":"2024-04-21T23:16:02.480399+0000","last_became_active":"2024-04-21T23:15:55.403423+0000","last_became_peered":"2024-04-21T23:15:55.403423+0000","last_unstale":"2024-04-21T23:16:02.480399+0000","last_undegraded":"2024-04-21T23:16:02.480399+0000","last_fullsized":"2024-04-21T23:16:02.480399+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:34:44.252259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.259745+0000","last_change":"2024-04-21T23:15:56.351995+0000","last_active":"2024-04-21T23:16:02.259745+0000","last_peered":"2024-04-21T23:16:02.259745+0000","last_clean":"2024-04-21T23:16:02.259745+0000","last_became_active":"2024-04-21T23:15:56.351614+0000","last_became_peered":"2024-04-21T23:15:56.351614+0000","last_unstale":"2024-04-21T23:16:02.259745+0000","last_undegraded":"2024-04-21T23:16:02.259745+0000","last_fullsized":"2024-04-21T23:16:02.259745+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:15:13.774193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.414055+0000","last_change":"2024-04-21T23:16:00.434104+0000","last_active":"2024-04-21T23:16:01.414055+0000","last_peered":"2024-04-21T23:16:01.414055+0000","last_clean":"2024-04-21T23:16:01.414055+0000","last_became_active":"2024-04-21T23:16:00.433681+0000","last_became_peered":"2024-04-21T23:16:00.433681+0000","last_unstale":"2024-04-21T23:16:01.414055+0000","last_undegraded":"2024-04-21T23:16:01.414055+0000","last_fullsized":"2024-04-21T23:16:01.414055+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:10:17.986652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"90'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:21.494544+0000","last_change":"2024-04-21T23:15:55.397536+0000","last_active":"2024-04-21T23:16:21.494544+0000","last_peered":"2024-04-21T23:16:21.494544+0000","last_clean":"2024-04-21T23:16:21.494544+0000","last_became_active":"2024-04-21T23:15:55.397163+0000","last_became_peered":"2024-04-21T23:15:55.397163+0000","last_unstale":"2024-04-21T23:16:21.494544+0000","last_undegraded":"2024-04-21T23:16:21.494544+0000","last_fullsized":"2024-04-21T23:16:21.494544+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:05:27.777804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.748988+0000","last_change":"2024-04-21T23:15:56.351840+0000","last_active":"2024-04-21T23:16:01.748988+0000","last_peered":"2024-04-21T23:16:01.748988+0000","last_clean":"2024-04-21T23:16:01.748988+0000","last_became_active":"2024-04-21T23:15:56.351523+0000","last_became_peered":"2024-04-21T23:15:56.351523+0000","last_unstale":"2024-04-21T23:16:01.748988+0000","last_undegraded":"2024-04-21T23:16:01.748988+0000","last_fullsized":"2024-04-21T23:16:01.748988+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:57:11.714284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.480927+0000","last_change":"2024-04-21T23:16:00.437473+0000","last_active":"2024-04-21T23:16:02.480927+0000","last_peered":"2024-04-21T23:16:02.480927+0000","last_clean":"2024-04-21T23:16:02.480927+0000","last_became_active":"2024-04-21T23:16:00.437014+0000","last_became_peered":"2024-04-21T23:16:00.437014+0000","last_unstale":"2024-04-21T23:16:02.480927+0000","last_undegraded":"2024-04-21T23:16:02.480927+0000","last_fullsized":"2024-04-21T23:16:02.480927+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:37:14.241896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.259306+0000","last_change":"2024-04-21T23:15:55.399213+0000","last_active":"2024-04-21T23:16:02.259306+0000","last_peered":"2024-04-21T23:16:02.259306+0000","last_clean":"2024-04-21T23:16:02.259306+0000","last_became_active":"2024-04-21T23:15:55.398880+0000","last_became_peered":"2024-04-21T23:15:55.398880+0000","last_unstale":"2024-04-21T23:16:02.259306+0000","last_undegraded":"2024-04-21T23:16:02.259306+0000","last_fullsized":"2024-04-21T23:16:02.259306+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:56:32.236862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.481085+0000","last_change":"2024-04-21T23:15:56.349330+0000","last_active":"2024-04-21T23:16:02.481085+0000","last_peered":"2024-04-21T23:16:02.481085+0000","last_clean":"2024-04-21T23:16:02.481085+0000","last_became_active":"2024-04-21T23:15:56.348988+0000","last_became_peered":"2024-04-21T23:15:56.348988+0000","last_unstale":"2024-04-21T23:16:02.481085+0000","last_undegraded":"2024-04-21T23:16:02.481085+0000","last_fullsized":"2024-04-21T23:16:02.481085+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:11:19.650119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.749110+0000","last_change":"2024-04-21T23:16:00.431959+0000","last_active":"2024-04-21T23:16:01.749110+0000","last_peered":"2024-04-21T23:16:01.749110+0000","last_clean":"2024-04-21T23:16:01.749110+0000","last_became_active":"2024-04-21T23:16:00.431570+0000","last_became_peered":"2024-04-21T23:16:00.431570+0000","last_unstale":"2024-04-21T23:16:01.749110+0000","last_undegraded":"2024-04-21T23:16:01.749110+0000","last_fullsized":"2024-04-21T23:16:01.749110+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:02:54.057326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.481011+0000","last_change":"2024-04-21T23:15:55.404505+0000","last_active":"2024-04-21T23:16:02.481011+0000","last_peered":"2024-04-21T23:16:02.481011+0000","last_clean":"2024-04-21T23:16:02.481011+0000","last_became_active":"2024-04-21T23:15:55.403945+0000","last_became_peered":"2024-04-21T23:15:55.403945+0000","last_unstale":"2024-04-21T23:16:02.481011+0000","last_undegraded":"2024-04-21T23:16:02.481011+0000","last_fullsized":"2024-04-21T23:16:02.481011+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:54:37.647597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.129513+0000","last_change":"2024-04-21T23:15:56.356238+0000","last_active":"2024-04-21T23:16:02.129513+0000","last_peered":"2024-04-21T23:16:02.129513+0000","last_clean":"2024-04-21T23:16:02.129513+0000","last_became_active":"2024-04-21T23:15:56.355116+0000","last_became_peered":"2024-04-21T23:15:56.355116+0000","last_unstale":"2024-04-21T23:16:02.129513+0000","last_undegraded":"2024-04-21T23:16:02.129513+0000","last_fullsized":"2024-04-21T23:16:02.129513+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:43:20.468549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.479998+0000","last_change":"2024-04-21T23:16:00.439069+0000","last_active":"2024-04-21T23:16:02.479998+0000","last_peered":"2024-04-21T23:16:02.479998+0000","last_clean":"2024-04-21T23:16:02.479998+0000","last_became_active":"2024-04-21T23:16:00.438510+0000","last_became_peered":"2024-04-21T23:16:00.438510+0000","last_unstale":"2024-04-21T23:16:02.479998+0000","last_undegraded":"2024-04-21T23:16:02.479998+0000","last_fullsized":"2024-04-21T23:16:02.479998+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:41:50.814112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.413979+0000","last_change":"2024-04-21T23:15:55.395682+0000","last_active":"2024-04-21T23:16:01.413979+0000","last_peered":"2024-04-21T23:16:01.413979+0000","last_clean":"2024-04-21T23:16:01.413979+0000","last_became_active":"2024-04-21T23:15:55.395320+0000","last_became_peered":"2024-04-21T23:15:55.395320+0000","last_unstale":"2024-04-21T23:16:01.413979+0000","last_undegraded":"2024-04-21T23:16:01.413979+0000","last_fullsized":"2024-04-21T23:16:01.413979+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:53:24.394310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.480995+0000","last_change":"2024-04-21T23:15:56.357228+0000","last_active":"2024-04-21T23:16:02.480995+0000","last_peered":"2024-04-21T23:16:02.480995+0000","last_clean":"2024-04-21T23:16:02.480995+0000","last_became_active":"2024-04-21T23:15:56.356808+0000","last_became_peered":"2024-04-21T23:15:56.356808+0000","last_unstale":"2024-04-21T23:16:02.480995+0000","last_undegraded":"2024-04-21T23:16:02.480995+0000","last_fullsized":"2024-04-21T23:16:02.480995+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:15:10.728530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.481196+0000","last_change":"2024-04-21T23:16:00.443569+0000","last_active":"2024-04-21T23:16:02.481196+0000","last_peered":"2024-04-21T23:16:02.481196+0000","last_clean":"2024-04-21T23:16:02.481196+0000","last_became_active":"2024-04-21T23:16:00.442972+0000","last_became_peered":"2024-04-21T23:16:00.442972+0000","last_unstale":"2024-04-21T23:16:02.481196+0000","last_undegraded":"2024-04-21T23:16:02.481196+0000","last_fullsized":"2024-04-21T23:16:02.481196+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:35:21.471407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"85'3","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.259847+0000","last_change":"2024-04-21T23:15:55.400134+0000","last_active":"2024-04-21T23:16:02.259847+0000","last_peered":"2024-04-21T23:16:02.259847+0000","last_clean":"2024-04-21T23:16:02.259847+0000","last_became_active":"2024-04-21T23:15:55.399768+0000","last_became_peered":"2024-04-21T23:15:55.399768+0000","last_unstale":"2024-04-21T23:16:02.259847+0000","last_undegraded":"2024-04-21T23:16:02.259847+0000","last_fullsized":"2024-04-21T23:16:02.259847+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:15:09.323035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.129324+0000","last_change":"2024-04-21T23:15:56.357278+0000","last_active":"2024-04-21T23:16:02.129324+0000","last_peered":"2024-04-21T23:16:02.129324+0000","last_clean":"2024-04-21T23:16:02.129324+0000","last_became_active":"2024-04-21T23:15:56.356902+0000","last_became_peered":"2024-04-21T23:15:56.356902+0000","last_unstale":"2024-04-21T23:16:02.129324+0000","last_undegraded":"2024-04-21T23:16:02.129324+0000","last_fullsized":"2024-04-21T23:16:02.129324+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:19:05.269109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.480309+0000","last_change":"2024-04-21T23:16:00.439704+0000","last_active":"2024-04-21T23:16:02.480309+0000","last_peered":"2024-04-21T23:16:02.480309+0000","last_clean":"2024-04-21T23:16:02.480309+0000","last_became_active":"2024-04-21T23:16:00.439286+0000","last_became_peered":"2024-04-21T23:16:00.439286+0000","last_unstale":"2024-04-21T23:16:02.480309+0000","last_undegraded":"2024-04-21T23:16:02.480309+0000","last_fullsized":"2024-04-21T23:16:02.480309+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:08:59.194481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"90'10","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:11.976303+0000","last_change":"2024-04-21T23:15:55.403344+0000","last_active":"2024-04-21T23:16:11.976303+0000","last_peered":"2024-04-21T23:16:11.976303+0000","last_clean":"2024-04-21T23:16:11.976303+0000","last_became_active":"2024-04-21T23:15:55.402876+0000","last_became_peered":"2024-04-21T23:15:55.402876+0000","last_unstale":"2024-04-21T23:16:11.976303+0000","last_undegraded":"2024-04-21T23:16:11.976303+0000","last_fullsized":"2024-04-21T23:16:11.976303+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:55:59.082964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.480698+0000","last_change":"2024-04-21T23:15:56.343897+0000","last_active":"2024-04-21T23:16:02.480698+0000","last_peered":"2024-04-21T23:16:02.480698+0000","last_clean":"2024-04-21T23:16:02.480698+0000","last_became_active":"2024-04-21T23:15:56.343567+0000","last_became_peered":"2024-04-21T23:15:56.343567+0000","last_unstale":"2024-04-21T23:16:02.480698+0000","last_undegraded":"2024-04-21T23:16:02.480698+0000","last_fullsized":"2024-04-21T23:16:02.480698+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:57:07.955722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.260326+0000","last_change":"2024-04-21T23:16:00.432722+0000","last_active":"2024-04-21T23:16:02.260326+0000","last_peered":"2024-04-21T23:16:02.260326+0000","last_clean":"2024-04-21T23:16:02.260326+0000","last_became_active":"2024-04-21T23:16:00.432257+0000","last_became_peered":"2024-04-21T23:16:00.432257+0000","last_unstale":"2024-04-21T23:16:02.260326+0000","last_undegraded":"2024-04-21T23:16:02.260326+0000","last_fullsized":"2024-04-21T23:16:02.260326+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:10:14.804490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"85'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.805207+0000","last_change":"2024-04-21T23:15:55.401300+0000","last_active":"2024-04-21T23:16:01.805207+0000","last_peered":"2024-04-21T23:16:01.805207+0000","last_clean":"2024-04-21T23:16:01.805207+0000","last_became_active":"2024-04-21T23:15:55.400904+0000","last_became_peered":"2024-04-21T23:15:55.400904+0000","last_unstale":"2024-04-21T23:16:01.805207+0000","last_undegraded":"2024-04-21T23:16:01.805207+0000","last_fullsized":"2024-04-21T23:16:01.805207+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:29:20.439936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.413691+0000","last_change":"2024-04-21T23:15:56.351046+0000","last_active":"2024-04-21T23:16:01.413691+0000","last_peered":"2024-04-21T23:16:01.413691+0000","last_clean":"2024-04-21T23:16:01.413691+0000","last_became_active":"2024-04-21T23:15:56.350662+0000","last_became_peered":"2024-04-21T23:15:56.350662+0000","last_unstale":"2024-04-21T23:16:01.413691+0000","last_undegraded":"2024-04-21T23:16:01.413691+0000","last_fullsized":"2024-04-21T23:16:01.413691+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:40:16.482061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.413838+0000","last_change":"2024-04-21T23:16:00.439816+0000","last_active":"2024-04-21T23:16:01.413838+0000","last_peered":"2024-04-21T23:16:01.413838+0000","last_clean":"2024-04-21T23:16:01.413838+0000","last_became_active":"2024-04-21T23:16:00.439330+0000","last_became_peered":"2024-04-21T23:16:00.439330+0000","last_unstale":"2024-04-21T23:16:01.413838+0000","last_undegraded":"2024-04-21T23:16:01.413838+0000","last_fullsized":"2024-04-21T23:16:01.413838+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:36:24.472968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.259833+0000","last_change":"2024-04-21T23:15:55.398588+0000","last_active":"2024-04-21T23:16:02.259833+0000","last_peered":"2024-04-21T23:16:02.259833+0000","last_clean":"2024-04-21T23:16:02.259833+0000","last_became_active":"2024-04-21T23:15:55.398264+0000","last_became_peered":"2024-04-21T23:15:55.398264+0000","last_unstale":"2024-04-21T23:16:02.259833+0000","last_undegraded":"2024-04-21T23:16:02.259833+0000","last_fullsized":"2024-04-21T23:16:02.259833+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:48:09.293481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.413843+0000","last_change":"2024-04-21T23:15:56.359331+0000","last_active":"2024-04-21T23:16:01.413843+0000","last_peered":"2024-04-21T23:16:01.413843+0000","last_clean":"2024-04-21T23:16:01.413843+0000","last_became_active":"2024-04-21T23:15:56.358189+0000","last_became_peered":"2024-04-21T23:15:56.358189+0000","last_unstale":"2024-04-21T23:16:01.413843+0000","last_undegraded":"2024-04-21T23:16:01.413843+0000","last_fullsized":"2024-04-21T23:16:01.413843+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:57:07.343998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.480425+0000","last_change":"2024-04-21T23:16:00.438581+0000","last_active":"2024-04-21T23:16:02.480425+0000","last_peered":"2024-04-21T23:16:02.480425+0000","last_clean":"2024-04-21T23:16:02.480425+0000","last_became_active":"2024-04-21T23:16:00.438219+0000","last_became_peered":"2024-04-21T23:16:00.438219+0000","last_unstale":"2024-04-21T23:16:02.480425+0000","last_undegraded":"2024-04-21T23:16:02.480425+0000","last_fullsized":"2024-04-21T23:16:02.480425+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:47:23.259045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.480391+0000","last_change":"2024-04-21T23:15:55.397503+0000","last_active":"2024-04-21T23:16:02.480391+0000","last_peered":"2024-04-21T23:16:02.480391+0000","last_clean":"2024-04-21T23:16:02.480391+0000","last_became_active":"2024-04-21T23:15:55.397181+0000","last_became_peered":"2024-04-21T23:15:55.397181+0000","last_unstale":"2024-04-21T23:16:02.480391+0000","last_undegraded":"2024-04-21T23:16:02.480391+0000","last_fullsized":"2024-04-21T23:16:02.480391+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:47:08.775783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.413523+0000","last_change":"2024-04-21T23:15:56.350872+0000","last_active":"2024-04-21T23:16:01.413523+0000","last_peered":"2024-04-21T23:16:01.413523+0000","last_clean":"2024-04-21T23:16:01.413523+0000","last_became_active":"2024-04-21T23:15:56.350519+0000","last_became_peered":"2024-04-21T23:15:56.350519+0000","last_unstale":"2024-04-21T23:16:01.413523+0000","last_undegraded":"2024-04-21T23:16:01.413523+0000","last_fullsized":"2024-04-21T23:16:01.413523+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:23:37.753893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.414402+0000","last_change":"2024-04-21T23:16:00.436069+0000","last_active":"2024-04-21T23:16:01.414402+0000","last_peered":"2024-04-21T23:16:01.414402+0000","last_clean":"2024-04-21T23:16:01.414402+0000","last_became_active":"2024-04-21T23:16:00.435617+0000","last_became_peered":"2024-04-21T23:16:00.435617+0000","last_unstale":"2024-04-21T23:16:01.414402+0000","last_undegraded":"2024-04-21T23:16:01.414402+0000","last_fullsized":"2024-04-21T23:16:01.414402+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:21:44.256928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"90'15","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:21.495660+0000","last_change":"2024-04-21T23:15:55.400900+0000","last_active":"2024-04-21T23:16:21.495660+0000","last_peered":"2024-04-21T23:16:21.495660+0000","last_clean":"2024-04-21T23:16:21.495660+0000","last_became_active":"2024-04-21T23:15:55.400558+0000","last_became_peered":"2024-04-21T23:15:55.400558+0000","last_unstale":"2024-04-21T23:16:21.495660+0000","last_undegraded":"2024-04-21T23:16:21.495660+0000","last_fullsized":"2024-04-21T23:16:21.495660+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:30:35.452601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.259321+0000","last_change":"2024-04-21T23:15:56.344526+0000","last_active":"2024-04-21T23:16:02.259321+0000","last_peered":"2024-04-21T23:16:02.259321+0000","last_clean":"2024-04-21T23:16:02.259321+0000","last_became_active":"2024-04-21T23:15:56.344069+0000","last_became_peered":"2024-04-21T23:15:56.344069+0000","last_unstale":"2024-04-21T23:16:02.259321+0000","last_undegraded":"2024-04-21T23:16:02.259321+0000","last_fullsized":"2024-04-21T23:16:02.259321+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:17:04.488828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.259531+0000","last_change":"2024-04-21T23:16:00.433580+0000","last_active":"2024-04-21T23:16:02.259531+0000","last_peered":"2024-04-21T23:16:02.259531+0000","last_clean":"2024-04-21T23:16:02.259531+0000","last_became_active":"2024-04-21T23:16:00.433185+0000","last_became_peered":"2024-04-21T23:16:00.433185+0000","last_unstale":"2024-04-21T23:16:02.259531+0000","last_undegraded":"2024-04-21T23:16:02.259531+0000","last_fullsized":"2024-04-21T23:16:02.259531+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:17:12.317266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'1","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.480812+0000","last_change":"2024-04-21T23:15:55.399032+0000","last_active":"2024-04-21T23:16:02.480812+0000","last_peered":"2024-04-21T23:16:02.480812+0000","last_clean":"2024-04-21T23:16:02.480812+0000","last_became_active":"2024-04-21T23:15:55.398208+0000","last_became_peered":"2024-04-21T23:15:55.398208+0000","last_unstale":"2024-04-21T23:16:02.480812+0000","last_undegraded":"2024-04-21T23:16:02.480812+0000","last_fullsized":"2024-04-21T23:16:02.480812+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:14:59.520571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.413884+0000","last_change":"2024-04-21T23:15:56.355357+0000","last_active":"2024-04-21T23:16:01.413884+0000","last_peered":"2024-04-21T23:16:01.413884+0000","last_clean":"2024-04-21T23:16:01.413884+0000","last_became_active":"2024-04-21T23:15:56.354704+0000","last_became_peered":"2024-04-21T23:15:56.354704+0000","last_unstale":"2024-04-21T23:16:01.413884+0000","last_undegraded":"2024-04-21T23:16:01.413884+0000","last_fullsized":"2024-04-21T23:16:01.413884+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:14:02.368659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.480555+0000","last_change":"2024-04-21T23:16:00.427781+0000","last_active":"2024-04-21T23:16:02.480555+0000","last_peered":"2024-04-21T23:16:02.480555+0000","last_clean":"2024-04-21T23:16:02.480555+0000","last_became_active":"2024-04-21T23:16:00.427244+0000","last_became_peered":"2024-04-21T23:16:00.427244+0000","last_unstale":"2024-04-21T23:16:02.480555+0000","last_undegraded":"2024-04-21T23:16:02.480555+0000","last_fullsized":"2024-04-21T23:16:02.480555+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:21:30.548865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.480455+0000","last_change":"2024-04-21T23:15:55.405687+0000","last_active":"2024-04-21T23:16:02.480455+0000","last_peered":"2024-04-21T23:16:02.480455+0000","last_clean":"2024-04-21T23:16:02.480455+0000","last_became_active":"2024-04-21T23:15:55.405379+0000","last_became_peered":"2024-04-21T23:15:55.405379+0000","last_unstale":"2024-04-21T23:16:02.480455+0000","last_undegraded":"2024-04-21T23:16:02.480455+0000","last_fullsized":"2024-04-21T23:16:02.480455+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:12:44.192092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.481218+0000","last_change":"2024-04-21T23:15:56.348989+0000","last_active":"2024-04-21T23:16:02.481218+0000","last_peered":"2024-04-21T23:16:02.481218+0000","last_clean":"2024-04-21T23:16:02.481218+0000","last_became_active":"2024-04-21T23:15:56.348039+0000","last_became_peered":"2024-04-21T23:15:56.348039+0000","last_unstale":"2024-04-21T23:16:02.481218+0000","last_undegraded":"2024-04-21T23:16:02.481218+0000","last_fullsized":"2024-04-21T23:16:02.481218+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:39:15.577567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.413818+0000","last_change":"2024-04-21T23:16:00.434557+0000","last_active":"2024-04-21T23:16:01.413818+0000","last_peered":"2024-04-21T23:16:01.413818+0000","last_clean":"2024-04-21T23:16:01.413818+0000","last_became_active":"2024-04-21T23:16:00.433737+0000","last_became_peered":"2024-04-21T23:16:00.433737+0000","last_unstale":"2024-04-21T23:16:01.413818+0000","last_undegraded":"2024-04-21T23:16:01.413818+0000","last_fullsized":"2024-04-21T23:16:01.413818+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:38:00.295054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.413600+0000","last_change":"2024-04-21T23:15:55.396792+0000","last_active":"2024-04-21T23:16:01.413600+0000","last_peered":"2024-04-21T23:16:01.413600+0000","last_clean":"2024-04-21T23:16:01.413600+0000","last_became_active":"2024-04-21T23:15:55.395740+0000","last_became_peered":"2024-04-21T23:15:55.395740+0000","last_unstale":"2024-04-21T23:16:01.413600+0000","last_undegraded":"2024-04-21T23:16:01.413600+0000","last_fullsized":"2024-04-21T23:16:01.413600+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:39:22.366310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.480782+0000","last_change":"2024-04-21T23:15:56.346372+0000","last_active":"2024-04-21T23:16:02.480782+0000","last_peered":"2024-04-21T23:16:02.480782+0000","last_clean":"2024-04-21T23:16:02.480782+0000","last_became_active":"2024-04-21T23:15:56.346047+0000","last_became_peered":"2024-04-21T23:15:56.346047+0000","last_unstale":"2024-04-21T23:16:02.480782+0000","last_undegraded":"2024-04-21T23:16:02.480782+0000","last_fullsized":"2024-04-21T23:16:02.480782+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:32:29.684227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.414268+0000","last_change":"2024-04-21T23:16:00.434747+0000","last_active":"2024-04-21T23:16:01.414268+0000","last_peered":"2024-04-21T23:16:01.414268+0000","last_clean":"2024-04-21T23:16:01.414268+0000","last_became_active":"2024-04-21T23:16:00.433950+0000","last_became_peered":"2024-04-21T23:16:00.433950+0000","last_unstale":"2024-04-21T23:16:01.414268+0000","last_undegraded":"2024-04-21T23:16:01.414268+0000","last_fullsized":"2024-04-21T23:16:01.414268+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:02:10.562135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.259303+0000","last_change":"2024-04-21T23:15:56.347115+0000","last_active":"2024-04-21T23:16:02.259303+0000","last_peered":"2024-04-21T23:16:02.259303+0000","last_clean":"2024-04-21T23:16:02.259303+0000","last_became_active":"2024-04-21T23:15:56.346803+0000","last_became_peered":"2024-04-21T23:15:56.346803+0000","last_unstale":"2024-04-21T23:16:02.259303+0000","last_undegraded":"2024-04-21T23:16:02.259303+0000","last_fullsized":"2024-04-21T23:16:02.259303+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:55:11.261095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'4","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:22.505318+0000","last_change":"2024-04-21T23:15:55.394900+0000","last_active":"2024-04-21T23:16:22.505318+0000","last_peered":"2024-04-21T23:16:22.505318+0000","last_clean":"2024-04-21T23:16:22.505318+0000","last_became_active":"2024-04-21T23:15:55.393556+0000","last_became_peered":"2024-04-21T23:15:55.393556+0000","last_unstale":"2024-04-21T23:16:22.505318+0000","last_undegraded":"2024-04-21T23:16:22.505318+0000","last_fullsized":"2024-04-21T23:16:22.505318+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:16:24.691930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.413829+0000","last_change":"2024-04-21T23:16:00.443338+0000","last_active":"2024-04-21T23:16:01.413829+0000","last_peered":"2024-04-21T23:16:01.413829+0000","last_clean":"2024-04-21T23:16:01.413829+0000","last_became_active":"2024-04-21T23:16:00.442855+0000","last_became_peered":"2024-04-21T23:16:00.442855+0000","last_unstale":"2024-04-21T23:16:01.413829+0000","last_undegraded":"2024-04-21T23:16:01.413829+0000","last_fullsized":"2024-04-21T23:16:01.413829+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:04:36.763027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.749041+0000","last_change":"2024-04-21T23:15:56.356788+0000","last_active":"2024-04-21T23:16:01.749041+0000","last_peered":"2024-04-21T23:16:01.749041+0000","last_clean":"2024-04-21T23:16:01.749041+0000","last_became_active":"2024-04-21T23:15:56.356474+0000","last_became_peered":"2024-04-21T23:15:56.356474+0000","last_unstale":"2024-04-21T23:16:01.749041+0000","last_undegraded":"2024-04-21T23:16:01.749041+0000","last_fullsized":"2024-04-21T23:16:01.749041+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:34:33.700897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"90'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:20.489576+0000","last_change":"2024-04-21T23:15:55.395398+0000","last_active":"2024-04-21T23:16:20.489576+0000","last_peered":"2024-04-21T23:16:20.489576+0000","last_clean":"2024-04-21T23:16:20.489576+0000","last_became_active":"2024-04-21T23:15:55.394978+0000","last_became_peered":"2024-04-21T23:15:55.394978+0000","last_unstale":"2024-04-21T23:16:20.489576+0000","last_undegraded":"2024-04-21T23:16:20.489576+0000","last_fullsized":"2024-04-21T23:16:20.489576+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:07:01.266885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":357,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.413325+0000","last_change":"2024-04-21T23:14:42.877959+0000","last_active":"2024-04-21T23:16:01.413325+0000","last_peered":"2024-04-21T23:16:01.413325+0000","last_clean":"2024-04-21T23:16:01.413325+0000","last_became_active":"2024-04-21T23:14:42.877384+0000","last_became_peered":"2024-04-21T23:14:42.877384+0000","last_unstale":"2024-04-21T23:16:01.413325+0000","last_undegraded":"2024-04-21T23:16:01.413325+0000","last_fullsized":"2024-04-21T23:16:01.413325+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:11:15.009349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:11:15.009349+0000","last_clean_scrub_stamp":"2024-04-21T23:11:15.009349+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:18:55.621052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.260050+0000","last_change":"2024-04-21T23:16:00.444576+0000","last_active":"2024-04-21T23:16:02.260050+0000","last_peered":"2024-04-21T23:16:02.260050+0000","last_clean":"2024-04-21T23:16:02.260050+0000","last_became_active":"2024-04-21T23:16:00.444153+0000","last_became_peered":"2024-04-21T23:16:00.444153+0000","last_unstale":"2024-04-21T23:16:02.260050+0000","last_undegraded":"2024-04-21T23:16:02.260050+0000","last_fullsized":"2024-04-21T23:16:02.260050+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:11:48.040383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.413329+0000","last_change":"2024-04-21T23:15:56.349870+0000","last_active":"2024-04-21T23:16:01.413329+0000","last_peered":"2024-04-21T23:16:01.413329+0000","last_clean":"2024-04-21T23:16:01.413329+0000","last_became_active":"2024-04-21T23:15:56.349535+0000","last_became_peered":"2024-04-21T23:15:56.349535+0000","last_unstale":"2024-04-21T23:16:01.413329+0000","last_undegraded":"2024-04-21T23:16:01.413329+0000","last_fullsized":"2024-04-21T23:16:01.413329+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:12:50.702556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.413320+0000","last_change":"2024-04-21T23:15:55.402406+0000","last_active":"2024-04-21T23:16:01.413320+0000","last_peered":"2024-04-21T23:16:01.413320+0000","last_clean":"2024-04-21T23:16:01.413320+0000","last_became_active":"2024-04-21T23:15:55.402075+0000","last_became_peered":"2024-04-21T23:15:55.402075+0000","last_unstale":"2024-04-21T23:16:01.413320+0000","last_undegraded":"2024-04-21T23:16:01.413320+0000","last_fullsized":"2024-04-21T23:16:01.413320+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:44:18.013031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.413814+0000","last_change":"2024-04-21T23:16:00.439206+0000","last_active":"2024-04-21T23:16:01.413814+0000","last_peered":"2024-04-21T23:16:01.413814+0000","last_clean":"2024-04-21T23:16:01.413814+0000","last_became_active":"2024-04-21T23:16:00.438601+0000","last_became_peered":"2024-04-21T23:16:00.438601+0000","last_unstale":"2024-04-21T23:16:01.413814+0000","last_undegraded":"2024-04-21T23:16:01.413814+0000","last_fullsized":"2024-04-21T23:16:01.413814+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:58:46.771456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"90'1","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:20.487401+0000","last_change":"2024-04-21T23:15:55.401154+0000","last_active":"2024-04-21T23:16:20.487401+0000","last_peered":"2024-04-21T23:16:20.487401+0000","last_clean":"2024-04-21T23:16:20.487401+0000","last_became_active":"2024-04-21T23:15:55.400579+0000","last_became_peered":"2024-04-21T23:15:55.400579+0000","last_unstale":"2024-04-21T23:16:20.487401+0000","last_undegraded":"2024-04-21T23:16:20.487401+0000","last_fullsized":"2024-04-21T23:16:20.487401+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:20:03.879660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.413866+0000","last_change":"2024-04-21T23:15:56.349837+0000","last_active":"2024-04-21T23:16:01.413866+0000","last_peered":"2024-04-21T23:16:01.413866+0000","last_clean":"2024-04-21T23:16:01.413866+0000","last_became_active":"2024-04-21T23:15:56.349466+0000","last_became_peered":"2024-04-21T23:15:56.349466+0000","last_unstale":"2024-04-21T23:16:01.413866+0000","last_undegraded":"2024-04-21T23:16:01.413866+0000","last_fullsized":"2024-04-21T23:16:01.413866+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:15:32.675770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.749070+0000","last_change":"2024-04-21T23:16:00.432599+0000","last_active":"2024-04-21T23:16:01.749070+0000","last_peered":"2024-04-21T23:16:01.749070+0000","last_clean":"2024-04-21T23:16:01.749070+0000","last_became_active":"2024-04-21T23:16:00.432194+0000","last_became_peered":"2024-04-21T23:16:00.432194+0000","last_unstale":"2024-04-21T23:16:01.749070+0000","last_undegraded":"2024-04-21T23:16:01.749070+0000","last_fullsized":"2024-04-21T23:16:01.749070+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:10:21.354338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'3","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:21.494349+0000","last_change":"2024-04-21T23:15:55.400328+0000","last_active":"2024-04-21T23:16:21.494349+0000","last_peered":"2024-04-21T23:16:21.494349+0000","last_clean":"2024-04-21T23:16:21.494349+0000","last_became_active":"2024-04-21T23:15:55.399991+0000","last_became_peered":"2024-04-21T23:15:55.399991+0000","last_unstale":"2024-04-21T23:16:21.494349+0000","last_undegraded":"2024-04-21T23:16:21.494349+0000","last_fullsized":"2024-04-21T23:16:21.494349+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:58:54.480720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.749392+0000","last_change":"2024-04-21T23:15:56.358698+0000","last_active":"2024-04-21T23:16:01.749392+0000","last_peered":"2024-04-21T23:16:01.749392+0000","last_clean":"2024-04-21T23:16:01.749392+0000","last_became_active":"2024-04-21T23:15:56.358364+0000","last_became_peered":"2024-04-21T23:15:56.358364+0000","last_unstale":"2024-04-21T23:16:01.749392+0000","last_undegraded":"2024-04-21T23:16:01.749392+0000","last_fullsized":"2024-04-21T23:16:01.749392+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:34:15.281142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.804977+0000","last_change":"2024-04-21T23:16:00.432237+0000","last_active":"2024-04-21T23:16:01.804977+0000","last_peered":"2024-04-21T23:16:01.804977+0000","last_clean":"2024-04-21T23:16:01.804977+0000","last_became_active":"2024-04-21T23:16:00.431745+0000","last_became_peered":"2024-04-21T23:16:00.431745+0000","last_unstale":"2024-04-21T23:16:01.804977+0000","last_undegraded":"2024-04-21T23:16:01.804977+0000","last_fullsized":"2024-04-21T23:16:01.804977+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:25:00.477154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"90'4","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:20.489226+0000","last_change":"2024-04-21T23:15:55.398667+0000","last_active":"2024-04-21T23:16:20.489226+0000","last_peered":"2024-04-21T23:16:20.489226+0000","last_clean":"2024-04-21T23:16:20.489226+0000","last_became_active":"2024-04-21T23:15:55.398290+0000","last_became_peered":"2024-04-21T23:15:55.398290+0000","last_unstale":"2024-04-21T23:16:20.489226+0000","last_undegraded":"2024-04-21T23:16:20.489226+0000","last_fullsized":"2024-04-21T23:16:20.489226+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:57:58.992615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.480829+0000","last_change":"2024-04-21T23:15:56.359518+0000","last_active":"2024-04-21T23:16:02.480829+0000","last_peered":"2024-04-21T23:16:02.480829+0000","last_clean":"2024-04-21T23:16:02.480829+0000","last_became_active":"2024-04-21T23:15:56.358970+0000","last_became_peered":"2024-04-21T23:15:56.358970+0000","last_unstale":"2024-04-21T23:16:02.480829+0000","last_undegraded":"2024-04-21T23:16:02.480829+0000","last_fullsized":"2024-04-21T23:16:02.480829+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:41:22.113877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.480882+0000","last_change":"2024-04-21T23:16:00.443380+0000","last_active":"2024-04-21T23:16:02.480882+0000","last_peered":"2024-04-21T23:16:02.480882+0000","last_clean":"2024-04-21T23:16:02.480882+0000","last_became_active":"2024-04-21T23:16:00.442896+0000","last_became_peered":"2024-04-21T23:16:00.442896+0000","last_unstale":"2024-04-21T23:16:02.480882+0000","last_undegraded":"2024-04-21T23:16:02.480882+0000","last_fullsized":"2024-04-21T23:16:02.480882+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:51:00.089766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"90'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:20.488484+0000","last_change":"2024-04-21T23:15:55.395287+0000","last_active":"2024-04-21T23:16:20.488484+0000","last_peered":"2024-04-21T23:16:20.488484+0000","last_clean":"2024-04-21T23:16:20.488484+0000","last_became_active":"2024-04-21T23:15:55.394908+0000","last_became_peered":"2024-04-21T23:15:55.394908+0000","last_unstale":"2024-04-21T23:16:20.488484+0000","last_undegraded":"2024-04-21T23:16:20.488484+0000","last_fullsized":"2024-04-21T23:16:20.488484+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:57:29.915581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.480282+0000","last_change":"2024-04-21T23:15:56.348016+0000","last_active":"2024-04-21T23:16:02.480282+0000","last_peered":"2024-04-21T23:16:02.480282+0000","last_clean":"2024-04-21T23:16:02.480282+0000","last_became_active":"2024-04-21T23:15:56.347630+0000","last_became_peered":"2024-04-21T23:15:56.347630+0000","last_unstale":"2024-04-21T23:16:02.480282+0000","last_undegraded":"2024-04-21T23:16:02.480282+0000","last_fullsized":"2024-04-21T23:16:02.480282+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:17:27.372880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.480688+0000","last_change":"2024-04-21T23:16:00.442959+0000","last_active":"2024-04-21T23:16:02.480688+0000","last_peered":"2024-04-21T23:16:02.480688+0000","last_clean":"2024-04-21T23:16:02.480688+0000","last_became_active":"2024-04-21T23:16:00.442319+0000","last_became_peered":"2024-04-21T23:16:00.442319+0000","last_unstale":"2024-04-21T23:16:02.480688+0000","last_undegraded":"2024-04-21T23:16:02.480688+0000","last_fullsized":"2024-04-21T23:16:02.480688+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:59:39.868059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.749214+0000","last_change":"2024-04-21T23:15:55.400859+0000","last_active":"2024-04-21T23:16:01.749214+0000","last_peered":"2024-04-21T23:16:01.749214+0000","last_clean":"2024-04-21T23:16:01.749214+0000","last_became_active":"2024-04-21T23:15:55.400460+0000","last_became_peered":"2024-04-21T23:15:55.400460+0000","last_unstale":"2024-04-21T23:16:01.749214+0000","last_undegraded":"2024-04-21T23:16:01.749214+0000","last_fullsized":"2024-04-21T23:16:01.749214+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:31:14.113476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.413541+0000","last_change":"2024-04-21T23:15:56.350354+0000","last_active":"2024-04-21T23:16:01.413541+0000","last_peered":"2024-04-21T23:16:01.413541+0000","last_clean":"2024-04-21T23:16:01.413541+0000","last_became_active":"2024-04-21T23:15:56.349954+0000","last_became_peered":"2024-04-21T23:15:56.349954+0000","last_unstale":"2024-04-21T23:16:01.413541+0000","last_undegraded":"2024-04-21T23:16:01.413541+0000","last_fullsized":"2024-04-21T23:16:01.413541+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:42:59.165610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.480726+0000","last_change":"2024-04-21T23:16:00.444829+0000","last_active":"2024-04-21T23:16:02.480726+0000","last_peered":"2024-04-21T23:16:02.480726+0000","last_clean":"2024-04-21T23:16:02.480726+0000","last_became_active":"2024-04-21T23:16:00.444365+0000","last_became_peered":"2024-04-21T23:16:00.444365+0000","last_unstale":"2024-04-21T23:16:02.480726+0000","last_undegraded":"2024-04-21T23:16:02.480726+0000","last_fullsized":"2024-04-21T23:16:02.480726+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:23:16.747684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.805009+0000","last_change":"2024-04-21T23:15:55.397855+0000","last_active":"2024-04-21T23:16:01.805009+0000","last_peered":"2024-04-21T23:16:01.805009+0000","last_clean":"2024-04-21T23:16:01.805009+0000","last_became_active":"2024-04-21T23:15:55.397498+0000","last_became_peered":"2024-04-21T23:15:55.397498+0000","last_unstale":"2024-04-21T23:16:01.805009+0000","last_undegraded":"2024-04-21T23:16:01.805009+0000","last_fullsized":"2024-04-21T23:16:01.805009+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:45:44.680350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.749348+0000","last_change":"2024-04-21T23:15:56.353023+0000","last_active":"2024-04-21T23:16:01.749348+0000","last_peered":"2024-04-21T23:16:01.749348+0000","last_clean":"2024-04-21T23:16:01.749348+0000","last_became_active":"2024-04-21T23:15:56.352209+0000","last_became_peered":"2024-04-21T23:15:56.352209+0000","last_unstale":"2024-04-21T23:16:01.749348+0000","last_undegraded":"2024-04-21T23:16:01.749348+0000","last_fullsized":"2024-04-21T23:16:01.749348+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:20:58.075627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.480802+0000","last_change":"2024-04-21T23:16:00.429258+0000","last_active":"2024-04-21T23:16:02.480802+0000","last_peered":"2024-04-21T23:16:02.480802+0000","last_clean":"2024-04-21T23:16:02.480802+0000","last_became_active":"2024-04-21T23:16:00.428856+0000","last_became_peered":"2024-04-21T23:16:00.428856+0000","last_unstale":"2024-04-21T23:16:02.480802+0000","last_undegraded":"2024-04-21T23:16:02.480802+0000","last_fullsized":"2024-04-21T23:16:02.480802+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:07:44.608370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"90'4","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:20.489703+0000","last_change":"2024-04-21T23:15:55.399846+0000","last_active":"2024-04-21T23:16:20.489703+0000","last_peered":"2024-04-21T23:16:20.489703+0000","last_clean":"2024-04-21T23:16:20.489703+0000","last_became_active":"2024-04-21T23:15:55.399518+0000","last_became_peered":"2024-04-21T23:15:55.399518+0000","last_unstale":"2024-04-21T23:16:20.489703+0000","last_undegraded":"2024-04-21T23:16:20.489703+0000","last_fullsized":"2024-04-21T23:16:20.489703+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:51:08.293769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.413710+0000","last_change":"2024-04-21T23:15:56.352024+0000","last_active":"2024-04-21T23:16:01.413710+0000","last_peered":"2024-04-21T23:16:01.413710+0000","last_clean":"2024-04-21T23:16:01.413710+0000","last_became_active":"2024-04-21T23:15:56.351052+0000","last_became_peered":"2024-04-21T23:15:56.351052+0000","last_unstale":"2024-04-21T23:16:01.413710+0000","last_undegraded":"2024-04-21T23:16:01.413710+0000","last_fullsized":"2024-04-21T23:16:01.413710+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:36:42.777601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.129068+0000","last_change":"2024-04-21T23:16:00.432371+0000","last_active":"2024-04-21T23:16:02.129068+0000","last_peered":"2024-04-21T23:16:02.129068+0000","last_clean":"2024-04-21T23:16:02.129068+0000","last_became_active":"2024-04-21T23:16:00.431828+0000","last_became_peered":"2024-04-21T23:16:00.431828+0000","last_unstale":"2024-04-21T23:16:02.129068+0000","last_undegraded":"2024-04-21T23:16:02.129068+0000","last_fullsized":"2024-04-21T23:16:02.129068+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:02:06.500075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.805401+0000","last_change":"2024-04-21T23:15:55.407018+0000","last_active":"2024-04-21T23:16:01.805401+0000","last_peered":"2024-04-21T23:16:01.805401+0000","last_clean":"2024-04-21T23:16:01.805401+0000","last_became_active":"2024-04-21T23:15:55.406709+0000","last_became_peered":"2024-04-21T23:15:55.406709+0000","last_unstale":"2024-04-21T23:16:01.805401+0000","last_undegraded":"2024-04-21T23:16:01.805401+0000","last_fullsized":"2024-04-21T23:16:01.805401+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:10:40.138470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.413992+0000","last_change":"2024-04-21T23:15:56.358569+0000","last_active":"2024-04-21T23:16:01.413992+0000","last_peered":"2024-04-21T23:16:01.413992+0000","last_clean":"2024-04-21T23:16:01.413992+0000","last_became_active":"2024-04-21T23:15:56.357533+0000","last_became_peered":"2024-04-21T23:15:56.357533+0000","last_unstale":"2024-04-21T23:16:01.413992+0000","last_undegraded":"2024-04-21T23:16:01.413992+0000","last_fullsized":"2024-04-21T23:16:01.413992+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:59:58.525340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.480215+0000","last_change":"2024-04-21T23:16:00.443981+0000","last_active":"2024-04-21T23:16:02.480215+0000","last_peered":"2024-04-21T23:16:02.480215+0000","last_clean":"2024-04-21T23:16:02.480215+0000","last_became_active":"2024-04-21T23:16:00.443368+0000","last_became_peered":"2024-04-21T23:16:00.443368+0000","last_unstale":"2024-04-21T23:16:02.480215+0000","last_undegraded":"2024-04-21T23:16:02.480215+0000","last_fullsized":"2024-04-21T23:16:02.480215+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:56:48.776540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.260015+0000","last_change":"2024-04-21T23:15:55.397334+0000","last_active":"2024-04-21T23:16:02.260015+0000","last_peered":"2024-04-21T23:16:02.260015+0000","last_clean":"2024-04-21T23:16:02.260015+0000","last_became_active":"2024-04-21T23:15:55.396745+0000","last_became_peered":"2024-04-21T23:15:55.396745+0000","last_unstale":"2024-04-21T23:16:02.260015+0000","last_undegraded":"2024-04-21T23:16:02.260015+0000","last_fullsized":"2024-04-21T23:16:02.260015+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:56:21.042187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.413696+0000","last_change":"2024-04-21T23:15:56.351012+0000","last_active":"2024-04-21T23:16:01.413696+0000","last_peered":"2024-04-21T23:16:01.413696+0000","last_clean":"2024-04-21T23:16:01.413696+0000","last_became_active":"2024-04-21T23:15:56.350042+0000","last_became_peered":"2024-04-21T23:15:56.350042+0000","last_unstale":"2024-04-21T23:16:01.413696+0000","last_undegraded":"2024-04-21T23:16:01.413696+0000","last_fullsized":"2024-04-21T23:16:01.413696+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:47:02.629870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.259218+0000","last_change":"2024-04-21T23:16:00.428775+0000","last_active":"2024-04-21T23:16:02.259218+0000","last_peered":"2024-04-21T23:16:02.259218+0000","last_clean":"2024-04-21T23:16:02.259218+0000","last_became_active":"2024-04-21T23:16:00.428278+0000","last_became_peered":"2024-04-21T23:16:00.428278+0000","last_unstale":"2024-04-21T23:16:02.259218+0000","last_undegraded":"2024-04-21T23:16:02.259218+0000","last_fullsized":"2024-04-21T23:16:02.259218+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:24:04.397387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.414260+0000","last_change":"2024-04-21T23:15:55.395524+0000","last_active":"2024-04-21T23:16:01.414260+0000","last_peered":"2024-04-21T23:16:01.414260+0000","last_clean":"2024-04-21T23:16:01.414260+0000","last_became_active":"2024-04-21T23:15:55.395220+0000","last_became_peered":"2024-04-21T23:15:55.395220+0000","last_unstale":"2024-04-21T23:16:01.414260+0000","last_undegraded":"2024-04-21T23:16:01.414260+0000","last_fullsized":"2024-04-21T23:16:01.414260+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:01:37.517620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.414009+0000","last_change":"2024-04-21T23:15:56.352128+0000","last_active":"2024-04-21T23:16:01.414009+0000","last_peered":"2024-04-21T23:16:01.414009+0000","last_clean":"2024-04-21T23:16:01.414009+0000","last_became_active":"2024-04-21T23:15:56.349349+0000","last_became_peered":"2024-04-21T23:15:56.349349+0000","last_unstale":"2024-04-21T23:16:01.414009+0000","last_undegraded":"2024-04-21T23:16:01.414009+0000","last_fullsized":"2024-04-21T23:16:01.414009+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:12:23.247084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.413681+0000","last_change":"2024-04-21T23:15:56.350830+0000","last_active":"2024-04-21T23:16:01.413681+0000","last_peered":"2024-04-21T23:16:01.413681+0000","last_clean":"2024-04-21T23:16:01.413681+0000","last_became_active":"2024-04-21T23:15:56.350385+0000","last_became_peered":"2024-04-21T23:15:56.350385+0000","last_unstale":"2024-04-21T23:16:01.413681+0000","last_undegraded":"2024-04-21T23:16:01.413681+0000","last_fullsized":"2024-04-21T23:16:01.413681+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:18:48.242375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.805000+0000","last_change":"2024-04-21T23:15:56.350947+0000","last_active":"2024-04-21T23:16:01.805000+0000","last_peered":"2024-04-21T23:16:01.805000+0000","last_clean":"2024-04-21T23:16:01.805000+0000","last_became_active":"2024-04-21T23:15:56.350383+0000","last_became_peered":"2024-04-21T23:15:56.350383+0000","last_unstale":"2024-04-21T23:16:01.805000+0000","last_undegraded":"2024-04-21T23:16:01.805000+0000","last_fullsized":"2024-04-21T23:16:01.805000+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:07:28.879822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.805068+0000","last_change":"2024-04-21T23:15:56.354559+0000","last_active":"2024-04-21T23:16:01.805068+0000","last_peered":"2024-04-21T23:16:01.805068+0000","last_clean":"2024-04-21T23:16:01.805068+0000","last_became_active":"2024-04-21T23:15:56.354205+0000","last_became_peered":"2024-04-21T23:15:56.354205+0000","last_unstale":"2024-04-21T23:16:01.805068+0000","last_undegraded":"2024-04-21T23:16:01.805068+0000","last_fullsized":"2024-04-21T23:16:01.805068+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:33:50.157971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.480393+0000","last_change":"2024-04-21T23:15:56.358395+0000","last_active":"2024-04-21T23:16:02.480393+0000","last_peered":"2024-04-21T23:16:02.480393+0000","last_clean":"2024-04-21T23:16:02.480393+0000","last_became_active":"2024-04-21T23:15:56.358077+0000","last_became_peered":"2024-04-21T23:15:56.358077+0000","last_unstale":"2024-04-21T23:16:02.480393+0000","last_undegraded":"2024-04-21T23:16:02.480393+0000","last_fullsized":"2024-04-21T23:16:02.480393+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:12:43.124588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.413742+0000","last_change":"2024-04-21T23:15:56.347476+0000","last_active":"2024-04-21T23:16:01.413742+0000","last_peered":"2024-04-21T23:16:01.413742+0000","last_clean":"2024-04-21T23:16:01.413742+0000","last_became_active":"2024-04-21T23:15:56.347097+0000","last_became_peered":"2024-04-21T23:15:56.347097+0000","last_unstale":"2024-04-21T23:16:01.413742+0000","last_undegraded":"2024-04-21T23:16:01.413742+0000","last_fullsized":"2024-04-21T23:16:01.413742+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:11:58.715526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.414100+0000","last_change":"2024-04-21T23:15:56.346413+0000","last_active":"2024-04-21T23:16:01.414100+0000","last_peered":"2024-04-21T23:16:01.414100+0000","last_clean":"2024-04-21T23:16:01.414100+0000","last_became_active":"2024-04-21T23:15:56.346056+0000","last_became_peered":"2024-04-21T23:15:56.346056+0000","last_unstale":"2024-04-21T23:16:01.414100+0000","last_undegraded":"2024-04-21T23:16:01.414100+0000","last_fullsized":"2024-04-21T23:16:01.414100+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:47:16.340323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.805595+0000","last_change":"2024-04-21T23:15:56.354876+0000","last_active":"2024-04-21T23:16:01.805595+0000","last_peered":"2024-04-21T23:16:01.805595+0000","last_clean":"2024-04-21T23:16:01.805595+0000","last_became_active":"2024-04-21T23:15:56.354525+0000","last_became_peered":"2024-04-21T23:15:56.354525+0000","last_unstale":"2024-04-21T23:16:01.805595+0000","last_undegraded":"2024-04-21T23:16:01.805595+0000","last_fullsized":"2024-04-21T23:16:01.805595+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:09:23.662664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.749516+0000","last_change":"2024-04-21T23:15:56.358456+0000","last_active":"2024-04-21T23:16:01.749516+0000","last_peered":"2024-04-21T23:16:01.749516+0000","last_clean":"2024-04-21T23:16:01.749516+0000","last_became_active":"2024-04-21T23:15:56.358060+0000","last_became_peered":"2024-04-21T23:15:56.358060+0000","last_unstale":"2024-04-21T23:16:01.749516+0000","last_undegraded":"2024-04-21T23:16:01.749516+0000","last_fullsized":"2024-04-21T23:16:01.749516+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:47:58.928438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.259136+0000","last_change":"2024-04-21T23:15:56.352870+0000","last_active":"2024-04-21T23:16:02.259136+0000","last_peered":"2024-04-21T23:16:02.259136+0000","last_clean":"2024-04-21T23:16:02.259136+0000","last_became_active":"2024-04-21T23:15:56.352267+0000","last_became_peered":"2024-04-21T23:15:56.352267+0000","last_unstale":"2024-04-21T23:16:02.259136+0000","last_undegraded":"2024-04-21T23:16:02.259136+0000","last_fullsized":"2024-04-21T23:16:02.259136+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:32:22.244773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.749230+0000","last_change":"2024-04-21T23:15:56.357468+0000","last_active":"2024-04-21T23:16:01.749230+0000","last_peered":"2024-04-21T23:16:01.749230+0000","last_clean":"2024-04-21T23:16:01.749230+0000","last_became_active":"2024-04-21T23:15:56.357104+0000","last_became_peered":"2024-04-21T23:15:56.357104+0000","last_unstale":"2024-04-21T23:16:01.749230+0000","last_undegraded":"2024-04-21T23:16:01.749230+0000","last_fullsized":"2024-04-21T23:16:01.749230+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:37:36.884036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:02.129286+0000","last_change":"2024-04-21T23:15:56.352206+0000","last_active":"2024-04-21T23:16:02.129286+0000","last_peered":"2024-04-21T23:16:02.129286+0000","last_clean":"2024-04-21T23:16:02.129286+0000","last_became_active":"2024-04-21T23:15:56.351835+0000","last_became_peered":"2024-04-21T23:15:56.351835+0000","last_unstale":"2024-04-21T23:16:02.129286+0000","last_undegraded":"2024-04-21T23:16:02.129286+0000","last_fullsized":"2024-04-21T23:16:02.129286+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:13:25.363479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.413871+0000","last_change":"2024-04-21T23:15:56.351901+0000","last_active":"2024-04-21T23:16:01.413871+0000","last_peered":"2024-04-21T23:16:01.413871+0000","last_clean":"2024-04-21T23:16:01.413871+0000","last_became_active":"2024-04-21T23:15:56.351545+0000","last_became_peered":"2024-04-21T23:15:56.351545+0000","last_unstale":"2024-04-21T23:16:01.413871+0000","last_undegraded":"2024-04-21T23:16:01.413871+0000","last_fullsized":"2024-04-21T23:16:01.413871+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:37:20.855631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:16:01.413774+0000","last_change":"2024-04-21T23:15:56.355773+0000","last_active":"2024-04-21T23:16:01.413774+0000","last_peered":"2024-04-21T23:16:01.413774+0000","last_clean":"2024-04-21T23:16:01.413774+0000","last_became_active":"2024-04-21T23:15:56.355269+0000","last_became_peered":"2024-04-21T23:15:56.355269+0000","last_unstale":"2024-04-21T23:16:01.413774+0000","last_undegraded":"2024-04-21T23:16:01.413774+0000","last_fullsized":"2024-04-21T23:16:01.413774+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:04:54.540817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4864,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":88,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":15768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892350994,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":995328,"data_stored":483870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612631,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36912,"kb_used_data":1576,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708240,"statfs":{"total":95995035648,"available":95957237760,"internally_reserved":0,"allocated":1613824,"data_stored":1078274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808861,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32168,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712984,"statfs":{"total":95995035648,"available":95962095616,"internally_reserved":0,"allocated":1015808,"data_stored":484688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070498,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1015808,"data_stored":484608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332134,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28580,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716572,"statfs":{"total":95995035648,"available":95965769728,"internally_reserved":0,"allocated":1593344,"data_stored":1074820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495660,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":999424,"data_stored":483960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574917,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1007616,"data_stored":487712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738442,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1024000,"data_stored":485348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411392,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1011712,"data_stored":487746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182458,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":1003520,"data_stored":483974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920822,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":999424,"data_stored":483892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757296,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32744,"kb_used_data":1564,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712408,"statfs":{"total":95995035648,"available":95961505792,"internally_reserved":0,"allocated":1601536,"data_stored":1074420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-21T23:16:25.646 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-21T23:16:25.647 INFO:tasks.cephadm:Setting up client nodes... 2024-04-21T23:16:25.647 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-21T23:16:25.929 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:16:25.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:25 smithi174 ceph-mon[31409]: pgmap v307: 129 pgs: 129 active+clean; 581 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 682 B/s rd, 1.5 KiB/s wr, 4 op/s 2024-04-21T23:16:25.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:25 smithi174 ceph-mon[31409]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-21T23:16:25.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:25 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/3938550799' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:16:25.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:25 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/3245186211' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:16:25.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:25 smithi174 ceph-mon[31409]: from='client.14934 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T23:16:25.944 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 21 23:16:25 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-b[61419]: 2024-04-21T23:16:25.515+0000 7fccc5509700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-21T23:16:25.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:25 smithi111 ceph-mon[31085]: pgmap v307: 129 pgs: 129 active+clean; 581 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 682 B/s rd, 1.5 KiB/s wr, 4 op/s 2024-04-21T23:16:25.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:25 smithi111 ceph-mon[31085]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-21T23:16:25.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:25 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/3938550799' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:16:25.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:25 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/3245186211' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:16:25.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:25 smithi111 ceph-mon[31085]: from='client.14934 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T23:16:26.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:26 smithi040 ceph-mon[27984]: from='client.14940 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T23:16:26.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:26 smithi174 ceph-mon[31409]: from='client.14940 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T23:16:26.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:26 smithi111 ceph-mon[31085]: from='client.14940 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T23:16:27.807 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:27 smithi040 ceph-mon[27984]: pgmap v308: 129 pgs: 129 active+clean; 620 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 5.7 KiB/s wr, 10 op/s 2024-04-21T23:16:27.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:27.825+0000 7ff70faa2700 1 -- 172.21.15.40:0/4138562340 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7081022b0 msgr2=0x7ff7081026d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:27.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:27.825+0000 7ff70faa2700 1 --2- 172.21.15.40:0/4138562340 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7081022b0 0x7ff7081026d0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7ff7040099c0 tx=0x7ff704009cd0 comp rx=0 tx=0).stop 2024-04-21T23:16:27.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:27.825+0000 7ff70faa2700 1 -- 172.21.15.40:0/4138562340 shutdown_connections 2024-04-21T23:16:27.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:27.825+0000 7ff70faa2700 1 --2- 172.21.15.40:0/4138562340 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff708103e50 0x7ff70810a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:27.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:27.825+0000 7ff70faa2700 1 --2- 172.21.15.40:0/4138562340 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7081022b0 0x7ff7081026d0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:27.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:27.825+0000 7ff70faa2700 1 --2- 172.21.15.40:0/4138562340 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff7081034b0 0x7ff708103910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:27.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:27.825+0000 7ff70faa2700 1 -- 172.21.15.40:0/4138562340 >> 172.21.15.40:0/4138562340 conn(0x7ff7080fd850 msgr2=0x7ff7080ffc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:27.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:27.825+0000 7ff70faa2700 1 -- 172.21.15.40:0/4138562340 shutdown_connections 2024-04-21T23:16:27.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:27.825+0000 7ff70faa2700 1 -- 172.21.15.40:0/4138562340 wait complete. 2024-04-21T23:16:27.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:27.826+0000 7ff70faa2700 1 Processor -- start 2024-04-21T23:16:27.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:27.826+0000 7ff70faa2700 1 -- start start 2024-04-21T23:16:27.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:27.826+0000 7ff70faa2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7081022b0 0x7ff708075fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:27.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:27.826+0000 7ff70faa2700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7081034b0 0x7ff708074620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:27.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:27.826+0000 7ff70faa2700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff708103e50 0x7ff708074b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:27.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:27.826+0000 7ff70faa2700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff70810de90 con 0x7ff7081034b0 2024-04-21T23:16:27.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:27.826+0000 7ff70faa2700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff70810dd10 con 0x7ff7081022b0 2024-04-21T23:16:27.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:27.826+0000 7ff70faa2700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff70810e010 con 0x7ff708103e50 2024-04-21T23:16:27.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:27.826+0000 7ff70d03d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7081034b0 0x7ff708074620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:27.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:27.826+0000 7ff70d03d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7081034b0 0x7ff708074620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43554/0 (socket says 172.21.15.40:43554) 2024-04-21T23:16:27.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:27.826+0000 7ff70d03d700 1 -- 172.21.15.40:0/537398108 learned_addr learned my addr 172.21.15.40:0/537398108 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:16:27.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:27.826+0000 7ff70e03f700 1 --2- 172.21.15.40:0/537398108 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff708103e50 0x7ff708074b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:27.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:27.826+0000 7ff70d83e700 1 --2- 172.21.15.40:0/537398108 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7081022b0 0x7ff708075fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:27.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:27.826+0000 7ff70d03d700 1 -- 172.21.15.40:0/537398108 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff708103e50 msgr2=0x7ff708074b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:27.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:27.826+0000 7ff70d03d700 1 --2- 172.21.15.40:0/537398108 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff708103e50 0x7ff708074b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:27.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:27.826+0000 7ff70d03d700 1 -- 172.21.15.40:0/537398108 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7081022b0 msgr2=0x7ff708075fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:27.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:27.826+0000 7ff70d03d700 1 --2- 172.21.15.40:0/537398108 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7081022b0 0x7ff708075fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:27.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:27.826+0000 7ff70d03d700 1 -- 172.21.15.40:0/537398108 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff708075410 con 0x7ff7081034b0 2024-04-21T23:16:27.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:27.826+0000 7ff70d83e700 1 --2- 172.21.15.40:0/537398108 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7081022b0 0x7ff708075fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:16:27.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:27.826+0000 7ff70d03d700 1 --2- 172.21.15.40:0/537398108 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7081034b0 0x7ff708074620 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7ff6f800d980 tx=0x7ff6f800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:27.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:27.827+0000 7ff6feffd700 1 -- 172.21.15.40:0/537398108 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6f8017070 con 0x7ff7081034b0 2024-04-21T23:16:27.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:27.827+0000 7ff6feffd700 1 -- 172.21.15.40:0/537398108 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6f8005020 con 0x7ff7081034b0 2024-04-21T23:16:27.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:27.827+0000 7ff70faa2700 1 -- 172.21.15.40:0/537398108 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7081a2570 con 0x7ff7081034b0 2024-04-21T23:16:27.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:27.827+0000 7ff6feffd700 1 -- 172.21.15.40:0/537398108 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6f8009bc0 con 0x7ff7081034b0 2024-04-21T23:16:27.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:27.827+0000 7ff70faa2700 1 -- 172.21.15.40:0/537398108 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7081a2a30 con 0x7ff7081034b0 2024-04-21T23:16:27.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:27.828+0000 7ff70faa2700 1 -- 172.21.15.40:0/537398108 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7080507a0 con 0x7ff7081034b0 2024-04-21T23:16:27.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:27.829+0000 7ff6feffd700 1 -- 172.21.15.40:0/537398108 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7ff6f80051d0 con 0x7ff7081034b0 2024-04-21T23:16:27.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:27.830+0000 7ff6feffd700 1 --2- 172.21.15.40:0/537398108 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7ff6f40a2d60 0x7ff6f40a5220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:27.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:27.830+0000 7ff70d83e700 1 --2- 172.21.15.40:0/537398108 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7ff6f40a2d60 0x7ff6f40a5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:27.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:27.830+0000 7ff6feffd700 1 -- 172.21.15.40:0/537398108 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7ff6f80cf340 con 0x7ff7081034b0 2024-04-21T23:16:27.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:27.830+0000 7ff70d83e700 1 --2- 172.21.15.40:0/537398108 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7ff6f40a2d60 0x7ff6f40a5220 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ff7040039c0 tx=0x7ff704009510 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:27.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:27.834+0000 7ff6feffd700 1 -- 172.21.15.40:0/537398108 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7ff6f8020020 con 0x7ff7081034b0 2024-04-21T23:16:27.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:27 smithi174 ceph-mon[31409]: pgmap v308: 129 pgs: 129 active+clean; 620 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 5.7 KiB/s wr, 10 op/s 2024-04-21T23:16:27.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:27 smithi111 ceph-mon[31085]: pgmap v308: 129 pgs: 129 active+clean; 620 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 5.7 KiB/s wr, 10 op/s 2024-04-21T23:16:28.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:28.124+0000 7ff70faa2700 1 -- 172.21.15.40:0/537398108 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7ff708062e40 con 0x7ff7081034b0 2024-04-21T23:16:28.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:28.127+0000 7ff6feffd700 1 -- 172.21.15.40:0/537398108 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7ff6f809bf00 con 0x7ff7081034b0 2024-04-21T23:16:28.126 INFO:teuthology.orchestra.run.smithi040.stdout:[client.0] 2024-04-21T23:16:28.126 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQBMniVm4BKHBxAAx/QDeYGjqL1wxY894VZ+3A== 2024-04-21T23:16:28.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:28.130+0000 7ff70faa2700 1 -- 172.21.15.40:0/537398108 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7ff6f40a2d60 msgr2=0x7ff6f40a5220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:28.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:28.130+0000 7ff70faa2700 1 --2- 172.21.15.40:0/537398108 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7ff6f40a2d60 0x7ff6f40a5220 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ff7040039c0 tx=0x7ff704009510 comp rx=0 tx=0).stop 2024-04-21T23:16:28.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:28.130+0000 7ff70faa2700 1 -- 172.21.15.40:0/537398108 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7081034b0 msgr2=0x7ff708074620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:28.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:28.130+0000 7ff70faa2700 1 --2- 172.21.15.40:0/537398108 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7081034b0 0x7ff708074620 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7ff6f800d980 tx=0x7ff6f800dc90 comp rx=0 tx=0).stop 2024-04-21T23:16:28.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:28.130+0000 7ff70faa2700 1 -- 172.21.15.40:0/537398108 shutdown_connections 2024-04-21T23:16:28.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:28.130+0000 7ff70faa2700 1 --2- 172.21.15.40:0/537398108 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff708103e50 0x7ff708074b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:28.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:28.130+0000 7ff70faa2700 1 --2- 172.21.15.40:0/537398108 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7ff6f40a2d60 0x7ff6f40a5220 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:28.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:28.130+0000 7ff70faa2700 1 --2- 172.21.15.40:0/537398108 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7081022b0 0x7ff708075fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:28.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:28.130+0000 7ff70faa2700 1 --2- 172.21.15.40:0/537398108 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7081034b0 0x7ff708074620 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:28.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:28.130+0000 7ff70faa2700 1 -- 172.21.15.40:0/537398108 >> 172.21.15.40:0/537398108 conn(0x7ff7080fd850 msgr2=0x7ff7081067b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:28.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:28.130+0000 7ff70faa2700 1 -- 172.21.15.40:0/537398108 shutdown_connections 2024-04-21T23:16:28.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:28.130+0000 7ff70faa2700 1 -- 172.21.15.40:0/537398108 wait complete. 2024-04-21T23:16:28.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:28 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/537398108' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-21T23:16:28.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:28 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/537398108' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-21T23:16:28.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:28 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/537398108' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-21T23:16:28.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:28 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/537398108' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-21T23:16:28.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:28 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/537398108' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-21T23:16:28.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:28 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/537398108' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-21T23:16:29.099 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-21T23:16:29.099 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-21T23:16:29.099 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-21T23:16:29.137 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-21T23:16:29.137 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-21T23:16:29.137 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph mgr dump --format=json 2024-04-21T23:16:29.430 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:16:30.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:30 smithi040 ceph-mon[27984]: pgmap v309: 129 pgs: 129 active+clean; 620 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 5.8 KiB/s wr, 11 op/s 2024-04-21T23:16:30.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:30 smithi174 ceph-mon[31409]: pgmap v309: 129 pgs: 129 active+clean; 620 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 5.8 KiB/s wr, 11 op/s 2024-04-21T23:16:30.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:30 smithi111 ceph-mon[31085]: pgmap v309: 129 pgs: 129 active+clean; 620 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 5.8 KiB/s wr, 11 op/s 2024-04-21T23:16:31.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:31.395+0000 7fd2c99d9700 1 -- 172.21.15.40:0/3867798289 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd2c4102510 msgr2=0x7fd2c41028f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:31.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:31.395+0000 7fd2c99d9700 1 --2- 172.21.15.40:0/3867798289 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd2c4102510 0x7fd2c41028f0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fd2b4009990 tx=0x7fd2b4009ca0 comp rx=0 tx=0).stop 2024-04-21T23:16:31.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:31.395+0000 7fd2c99d9700 1 -- 172.21.15.40:0/3867798289 shutdown_connections 2024-04-21T23:16:31.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:31.395+0000 7fd2c99d9700 1 --2- 172.21.15.40:0/3867798289 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd2c4107590 0x7fd2c410f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:31.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:31.395+0000 7fd2c99d9700 1 --2- 172.21.15.40:0/3867798289 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd2c4102510 0x7fd2c41028f0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:31.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:31.395+0000 7fd2c99d9700 1 --2- 172.21.15.40:0/3867798289 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2c4102ec0 0x7fd2c4106eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:31.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:31.395+0000 7fd2c99d9700 1 -- 172.21.15.40:0/3867798289 >> 172.21.15.40:0/3867798289 conn(0x7fd2c40793a0 msgr2=0x7fd2c40797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:31.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:31.395+0000 7fd2c99d9700 1 -- 172.21.15.40:0/3867798289 shutdown_connections 2024-04-21T23:16:31.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:31.395+0000 7fd2c99d9700 1 -- 172.21.15.40:0/3867798289 wait complete. 2024-04-21T23:16:31.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:31.396+0000 7fd2c99d9700 1 Processor -- start 2024-04-21T23:16:31.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:31.396+0000 7fd2c99d9700 1 -- start start 2024-04-21T23:16:31.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:31.396+0000 7fd2c99d9700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd2c4102510 0x7fd2c419e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:31.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:31.396+0000 7fd2c99d9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2c4102ec0 0x7fd2c419eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:31.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:31.396+0000 7fd2c99d9700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd2c4107590 0x7fd2c41987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:31.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:31.397+0000 7fd2c99d9700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd2c4112580 con 0x7fd2c4102510 2024-04-21T23:16:31.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:31.397+0000 7fd2c99d9700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd2c4112400 con 0x7fd2c4102ec0 2024-04-21T23:16:31.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:31.397+0000 7fd2c99d9700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd2c4112700 con 0x7fd2c4107590 2024-04-21T23:16:31.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:31.397+0000 7fd2c2ffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd2c4102510 0x7fd2c419e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:31.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:31.397+0000 7fd2c2ffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd2c4102510 0x7fd2c419e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43582/0 (socket says 172.21.15.40:43582) 2024-04-21T23:16:31.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:31.397+0000 7fd2c2ffd700 1 -- 172.21.15.40:0/1502687060 learned_addr learned my addr 172.21.15.40:0/1502687060 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:16:31.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:31.397+0000 7fd2ba7fc700 1 --2- 172.21.15.40:0/1502687060 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2c4102ec0 0x7fd2c419eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:31.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:31.397+0000 7fd2c37fe700 1 --2- 172.21.15.40:0/1502687060 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd2c4107590 0x7fd2c41987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:31.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:31.397+0000 7fd2c2ffd700 1 -- 172.21.15.40:0/1502687060 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd2c4107590 msgr2=0x7fd2c41987a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:31.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:31.397+0000 7fd2c2ffd700 1 --2- 172.21.15.40:0/1502687060 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd2c4107590 0x7fd2c41987a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:31.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:31.397+0000 7fd2c2ffd700 1 -- 172.21.15.40:0/1502687060 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2c4102ec0 msgr2=0x7fd2c419eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:31.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:31.397+0000 7fd2c2ffd700 1 --2- 172.21.15.40:0/1502687060 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2c4102ec0 0x7fd2c419eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:31.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:31.397+0000 7fd2c2ffd700 1 -- 172.21.15.40:0/1502687060 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2c4198f80 con 0x7fd2c4102510 2024-04-21T23:16:31.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:31.397+0000 7fd2c2ffd700 1 --2- 172.21.15.40:0/1502687060 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd2c4102510 0x7fd2c419e620 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fd2b4000c00 tx=0x7fd2b40047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:31.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:31.397+0000 7fd2c89d7700 1 -- 172.21.15.40:0/1502687060 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2b4027070 con 0x7fd2c4102510 2024-04-21T23:16:31.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:31.397+0000 7fd2c99d9700 1 -- 172.21.15.40:0/1502687060 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2c4199210 con 0x7fd2c4102510 2024-04-21T23:16:31.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:31.397+0000 7fd2c89d7700 1 -- 172.21.15.40:0/1502687060 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd2b4003e40 con 0x7fd2c4102510 2024-04-21T23:16:31.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:31.397+0000 7fd2c89d7700 1 -- 172.21.15.40:0/1502687060 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2b4022d10 con 0x7fd2c4102510 2024-04-21T23:16:31.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:31.398+0000 7fd2c99d9700 1 -- 172.21.15.40:0/1502687060 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2c41a5480 con 0x7fd2c4102510 2024-04-21T23:16:31.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:31.398+0000 7fd2c99d9700 1 -- 172.21.15.40:0/1502687060 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2c4103990 con 0x7fd2c4102510 2024-04-21T23:16:31.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:31.400+0000 7fd2c89d7700 1 -- 172.21.15.40:0/1502687060 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fd2b402a070 con 0x7fd2c4102510 2024-04-21T23:16:31.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:31.400+0000 7fd2c89d7700 1 --2- 172.21.15.40:0/1502687060 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fd2b00a2cb0 0x7fd2b00a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:31.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:31.400+0000 7fd2ba7fc700 1 --2- 172.21.15.40:0/1502687060 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fd2b00a2cb0 0x7fd2b00a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:31.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:31.400+0000 7fd2c89d7700 1 -- 172.21.15.40:0/1502687060 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fd2b40d08f0 con 0x7fd2c4102510 2024-04-21T23:16:31.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:31.401+0000 7fd2ba7fc700 1 --2- 172.21.15.40:0/1502687060 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fd2b00a2cb0 0x7fd2b00a5170 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fd2c4107340 tx=0x7fd2ac00a380 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:31.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:31.404+0000 7fd2c89d7700 1 -- 172.21.15.40:0/1502687060 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fd2b4031070 con 0x7fd2c4102510 2024-04-21T23:16:31.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:31.670+0000 7fd2c99d9700 1 -- 172.21.15.40:0/1502687060 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fd2c4199cd0 con 0x7fd2c4102510 2024-04-21T23:16:31.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:31.675+0000 7fd2c89d7700 1 -- 172.21.15.40:0/1502687060 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+263201 (secure 0 0 0) 0x7fd2b4013070 con 0x7fd2c4102510 2024-04-21T23:16:31.677 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:16:31.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:31.677+0000 7fd2c99d9700 1 -- 172.21.15.40:0/1502687060 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fd2b00a2cb0 msgr2=0x7fd2b00a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:31.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:31.677+0000 7fd2c99d9700 1 --2- 172.21.15.40:0/1502687060 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fd2b00a2cb0 0x7fd2b00a5170 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fd2c4107340 tx=0x7fd2ac00a380 comp rx=0 tx=0).stop 2024-04-21T23:16:31.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:31.677+0000 7fd2c99d9700 1 -- 172.21.15.40:0/1502687060 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd2c4102510 msgr2=0x7fd2c419e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:31.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:31.677+0000 7fd2c99d9700 1 --2- 172.21.15.40:0/1502687060 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd2c4102510 0x7fd2c419e620 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fd2b4000c00 tx=0x7fd2b40047b0 comp rx=0 tx=0).stop 2024-04-21T23:16:31.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:31.678+0000 7fd2c99d9700 1 -- 172.21.15.40:0/1502687060 shutdown_connections 2024-04-21T23:16:31.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:31.678+0000 7fd2c99d9700 1 --2- 172.21.15.40:0/1502687060 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd2c4107590 0x7fd2c41987a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:31.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:31.678+0000 7fd2c99d9700 1 --2- 172.21.15.40:0/1502687060 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fd2b00a2cb0 0x7fd2b00a5170 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:31.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:31.678+0000 7fd2c99d9700 1 --2- 172.21.15.40:0/1502687060 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd2c4102510 0x7fd2c419e620 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:31.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:31.678+0000 7fd2c99d9700 1 --2- 172.21.15.40:0/1502687060 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2c4102ec0 0x7fd2c419eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:31.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:31.678+0000 7fd2c99d9700 1 -- 172.21.15.40:0/1502687060 >> 172.21.15.40:0/1502687060 conn(0x7fd2c40793a0 msgr2=0x7fd2c41002c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:31.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:31.678+0000 7fd2c99d9700 1 -- 172.21.15.40:0/1502687060 shutdown_connections 2024-04-21T23:16:31.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:31.678+0000 7fd2c99d9700 1 -- 172.21.15.40:0/1502687060 wait complete. 2024-04-21T23:16:32.416 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:32 smithi040 ceph-mon[27984]: pgmap v310: 129 pgs: 129 active+clean; 625 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.3 KiB/s wr, 13 op/s 2024-04-21T23:16:32.416 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:32 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/1502687060' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-21T23:16:32.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:32 smithi174 ceph-mon[31409]: pgmap v310: 129 pgs: 129 active+clean; 625 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.3 KiB/s wr, 13 op/s 2024-04-21T23:16:32.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:32 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/1502687060' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-21T23:16:32.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:32 smithi111 ceph-mon[31085]: pgmap v310: 129 pgs: 129 active+clean; 625 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.3 KiB/s wr, 13 op/s 2024-04-21T23:16:32.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:32 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/1502687060' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-21T23:16:32.454 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":17,"flags":0,"active_gid":14148,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2500760788},{"type":"v1","addr":"172.21.15.40:6801","nonce":2500760788}]},"active_addr":"172.21.15.40:6801/2500760788","active_change":"2024-04-21T23:08:50.305883+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24106,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24113,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.40:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.40:0","nonce":1406242364}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.40:0","nonce":1737227818}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.40:0","nonce":2742002992}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.40:0","nonce":2494672676}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.40:0","nonce":2121120590}]}]} 2024-04-21T23:16:32.459 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-21T23:16:32.459 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-21T23:16:32.459 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-21T23:16:32.715 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:16:34.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:34 smithi040 ceph-mon[27984]: pgmap v311: 129 pgs: 129 active+clean; 625 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.3 KiB/s wr, 14 op/s 2024-04-21T23:16:34.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:34 smithi174 ceph-mon[31409]: pgmap v311: 129 pgs: 129 active+clean; 625 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.3 KiB/s wr, 14 op/s 2024-04-21T23:16:34.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:34 smithi111 ceph-mon[31085]: pgmap v311: 129 pgs: 129 active+clean; 625 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.3 KiB/s wr, 14 op/s 2024-04-21T23:16:34.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:34.544+0000 7f521c96c700 1 -- 172.21.15.40:0/1058708030 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5218107570 msgr2=0x7f521810f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:34.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:34.544+0000 7f521c96c700 1 --2- 172.21.15.40:0/1058708030 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5218107570 0x7f521810f140 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f521000b0a0 tx=0x7f521000b3b0 comp rx=0 tx=0).stop 2024-04-21T23:16:34.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:34.545+0000 7f521c96c700 1 -- 172.21.15.40:0/1058708030 shutdown_connections 2024-04-21T23:16:34.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:34.545+0000 7f521c96c700 1 --2- 172.21.15.40:0/1058708030 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5218107570 0x7f521810f140 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:34.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:34.545+0000 7f521c96c700 1 --2- 172.21.15.40:0/1058708030 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52181024e0 0x7f52181028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:34.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:34.545+0000 7f521c96c700 1 --2- 172.21.15.40:0/1058708030 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5218102e90 0x7f5218106e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:34.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:34.545+0000 7f521c96c700 1 -- 172.21.15.40:0/1058708030 >> 172.21.15.40:0/1058708030 conn(0x7f5218079390 msgr2=0x7f52180797a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:34.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:34.545+0000 7f521c96c700 1 -- 172.21.15.40:0/1058708030 shutdown_connections 2024-04-21T23:16:34.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:34.545+0000 7f521c96c700 1 -- 172.21.15.40:0/1058708030 wait complete. 2024-04-21T23:16:34.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:34.545+0000 7f521c96c700 1 Processor -- start 2024-04-21T23:16:34.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:34.545+0000 7f521c96c700 1 -- start start 2024-04-21T23:16:34.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:34.546+0000 7f521c96c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52181024e0 0x7f521819e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:34.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:34.546+0000 7f521c96c700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5218102e90 0x7f521819eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:34.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:34.546+0000 7f521c96c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5218107570 0x7f52181987d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:34.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:34.546+0000 7f521c96c700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5218112550 con 0x7f5218107570 2024-04-21T23:16:34.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:34.546+0000 7f521c96c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f52181123d0 con 0x7f52181024e0 2024-04-21T23:16:34.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:34.546+0000 7f521c96c700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f52181126d0 con 0x7f5218102e90 2024-04-21T23:16:34.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:34.546+0000 7f5216d9d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5218107570 0x7f52181987d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:34.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:34.546+0000 7f5216d9d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5218107570 0x7f52181987d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49028/0 (socket says 172.21.15.40:49028) 2024-04-21T23:16:34.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:34.546+0000 7f5216d9d700 1 -- 172.21.15.40:0/1256249966 learned_addr learned my addr 172.21.15.40:0/1256249966 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:16:34.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:34.546+0000 7f521659c700 1 --2- 172.21.15.40:0/1256249966 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52181024e0 0x7f521819e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:34.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:34.546+0000 7f5216d9d700 1 -- 172.21.15.40:0/1256249966 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5218102e90 msgr2=0x7f521819eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:34.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:34.546+0000 7f5216d9d700 1 --2- 172.21.15.40:0/1256249966 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5218102e90 0x7f521819eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:34.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:34.546+0000 7f5216d9d700 1 -- 172.21.15.40:0/1256249966 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52181024e0 msgr2=0x7f521819e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:34.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:34.546+0000 7f5216d9d700 1 --2- 172.21.15.40:0/1256249966 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52181024e0 0x7f521819e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:34.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:34.546+0000 7f5216d9d700 1 -- 172.21.15.40:0/1256249966 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5218198fb0 con 0x7f5218107570 2024-04-21T23:16:34.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:34.546+0000 7f5216d9d700 1 --2- 172.21.15.40:0/1256249966 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5218107570 0x7f52181987d0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f52100062a0 tx=0x7f521000bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:34.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:34.546+0000 7f520f7fe700 1 -- 172.21.15.40:0/1256249966 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5210021070 con 0x7f5218107570 2024-04-21T23:16:34.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:34.547+0000 7f520f7fe700 1 -- 172.21.15.40:0/1256249966 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5210007db0 con 0x7f5218107570 2024-04-21T23:16:34.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:34.547+0000 7f521c96c700 1 -- 172.21.15.40:0/1256249966 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5218199240 con 0x7f5218107570 2024-04-21T23:16:34.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:34.547+0000 7f520f7fe700 1 -- 172.21.15.40:0/1256249966 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5210005460 con 0x7f5218107570 2024-04-21T23:16:34.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:34.547+0000 7f521c96c700 1 -- 172.21.15.40:0/1256249966 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52181a54b0 con 0x7f5218107570 2024-04-21T23:16:34.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:34.547+0000 7f521c96c700 1 -- 172.21.15.40:0/1256249966 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5218103970 con 0x7f5218107570 2024-04-21T23:16:34.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:34.549+0000 7f520f7fe700 1 -- 172.21.15.40:0/1256249966 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f5210024070 con 0x7f5218107570 2024-04-21T23:16:34.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:34.549+0000 7f520f7fe700 1 --2- 172.21.15.40:0/1256249966 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f52040a2cb0 0x7f52040a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:34.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:34.549+0000 7f520f7fe700 1 -- 172.21.15.40:0/1256249966 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f521000f020 con 0x7f5218107570 2024-04-21T23:16:34.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:34.549+0000 7f521659c700 1 --2- 172.21.15.40:0/1256249966 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f52040a2cb0 0x7f52040a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:34.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:34.550+0000 7f521659c700 1 --2- 172.21.15.40:0/1256249966 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f52040a2cb0 0x7f52040a5170 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f5208005fd0 tx=0x7f520800a560 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:34.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:34.553+0000 7f520f7fe700 1 -- 172.21.15.40:0/1256249966 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f521002b030 con 0x7f5218107570 2024-04-21T23:16:34.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:34.760+0000 7f521c96c700 1 -- 172.21.15.40:0/1256249966 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f521802db30 con 0x7f5218107570 2024-04-21T23:16:34.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:34.762+0000 7f520f7fe700 1 -- 172.21.15.40:0/1256249966 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24138 (secure 0 0 0) 0x7f521009d850 con 0x7f5218107570 2024-04-21T23:16:34.762 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:16:34.762 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":90,"fsid":"d814e4dc-0033-11ef-bc93-c7b262605968","created":"2024-04-21T23:07:52.659207+0000","modified":"2024-04-21T23:16:00.780824+0000","last_up_change":"2024-04-21T23:15:05.308821+0000","last_in_change":"2024-04-21T23:14:45.447491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:11:14.824775+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T23:15:54.322571+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T23:15:55.164493+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T23:15:58.838266+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"0e8d692a-22fe-4a87-8d99-9151f96cbb02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6803","nonce":2009758784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6805","nonce":2009758784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6809","nonce":2009758784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6807","nonce":2009758784}]},"public_addr":"172.21.15.40:6803/2009758784","cluster_addr":"172.21.15.40:6805/2009758784","heartbeat_back_addr":"172.21.15.40:6809/2009758784","heartbeat_front_addr":"172.21.15.40:6807/2009758784","state":["exists","up"]},{"osd":1,"uuid":"fe5fe11b-0e84-4359-83a7-1ba45105755a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6801","nonce":1763903836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6803","nonce":1763903836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6807","nonce":1763903836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6805","nonce":1763903836}]},"public_addr":"172.21.15.111:6801/1763903836","cluster_addr":"172.21.15.111:6803/1763903836","heartbeat_back_addr":"172.21.15.111:6807/1763903836","heartbeat_front_addr":"172.21.15.111:6805/1763903836","state":["exists","up"]},{"osd":2,"uuid":"813ad735-5cc2-4da0-ae16-6db238395fbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6801","nonce":439695825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6803","nonce":439695825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6807","nonce":439695825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6805","nonce":439695825}]},"public_addr":"172.21.15.174:6801/439695825","cluster_addr":"172.21.15.174:6803/439695825","heartbeat_back_addr":"172.21.15.174:6807/439695825","heartbeat_front_addr":"172.21.15.174:6805/439695825","state":["exists","up"]},{"osd":3,"uuid":"09becb03-b7ce-4ffc-917d-b7900ec37e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6811","nonce":3454312436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6813","nonce":3454312436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6817","nonce":3454312436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6815","nonce":3454312436}]},"public_addr":"172.21.15.40:6811/3454312436","cluster_addr":"172.21.15.40:6813/3454312436","heartbeat_back_addr":"172.21.15.40:6817/3454312436","heartbeat_front_addr":"172.21.15.40:6815/3454312436","state":["exists","up"]},{"osd":4,"uuid":"46eabfd7-d88a-4efd-94eb-04354d3a9d7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6809","nonce":1810866070}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6811","nonce":1810866070}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6815","nonce":1810866070}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6813","nonce":1810866070}]},"public_addr":"172.21.15.111:6809/1810866070","cluster_addr":"172.21.15.111:6811/1810866070","heartbeat_back_addr":"172.21.15.111:6815/1810866070","heartbeat_front_addr":"172.21.15.111:6813/1810866070","state":["exists","up"]},{"osd":5,"uuid":"6f73b582-6c65-41db-aab0-9b360f83d5c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6809","nonce":3752986711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6811","nonce":3752986711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6815","nonce":3752986711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6813","nonce":3752986711}]},"public_addr":"172.21.15.174:6809/3752986711","cluster_addr":"172.21.15.174:6811/3752986711","heartbeat_back_addr":"172.21.15.174:6815/3752986711","heartbeat_front_addr":"172.21.15.174:6813/3752986711","state":["exists","up"]},{"osd":6,"uuid":"39f305f0-12d2-4732-aaa0-cd66700af538","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6819","nonce":1618855427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6821","nonce":1618855427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6825","nonce":1618855427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6823","nonce":1618855427}]},"public_addr":"172.21.15.40:6819/1618855427","cluster_addr":"172.21.15.40:6821/1618855427","heartbeat_back_addr":"172.21.15.40:6825/1618855427","heartbeat_front_addr":"172.21.15.40:6823/1618855427","state":["exists","up"]},{"osd":7,"uuid":"769f2468-5102-490b-89f8-d0444d3169a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6817","nonce":927054162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6819","nonce":927054162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6823","nonce":927054162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6821","nonce":927054162}]},"public_addr":"172.21.15.111:6817/927054162","cluster_addr":"172.21.15.111:6819/927054162","heartbeat_back_addr":"172.21.15.111:6823/927054162","heartbeat_front_addr":"172.21.15.111:6821/927054162","state":["exists","up"]},{"osd":8,"uuid":"1a444e80-007d-473b-b972-4c814a813c22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6817","nonce":3817848746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6819","nonce":3817848746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6823","nonce":3817848746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6821","nonce":3817848746}]},"public_addr":"172.21.15.174:6817/3817848746","cluster_addr":"172.21.15.174:6819/3817848746","heartbeat_back_addr":"172.21.15.174:6823/3817848746","heartbeat_front_addr":"172.21.15.174:6821/3817848746","state":["exists","up"]},{"osd":9,"uuid":"9a6b6a56-6ae5-49fe-9e5b-cb17d686e0dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6827","nonce":1649446531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6829","nonce":1649446531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6833","nonce":1649446531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6831","nonce":1649446531}]},"public_addr":"172.21.15.40:6827/1649446531","cluster_addr":"172.21.15.40:6829/1649446531","heartbeat_back_addr":"172.21.15.40:6833/1649446531","heartbeat_front_addr":"172.21.15.40:6831/1649446531","state":["exists","up"]},{"osd":10,"uuid":"a70038ba-e2ad-4170-85cc-ea0470a2edef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6825","nonce":1227538935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6827","nonce":1227538935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6831","nonce":1227538935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6829","nonce":1227538935}]},"public_addr":"172.21.15.111:6825/1227538935","cluster_addr":"172.21.15.111:6827/1227538935","heartbeat_back_addr":"172.21.15.111:6831/1227538935","heartbeat_front_addr":"172.21.15.111:6829/1227538935","state":["exists","up"]},{"osd":11,"uuid":"781532bb-7a5e-44e3-8eb7-b584b0054666","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6825","nonce":2045865553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6827","nonce":2045865553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6831","nonce":2045865553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6829","nonce":2045865553}]},"public_addr":"172.21.15.174:6825/2045865553","cluster_addr":"172.21.15.174:6827/2045865553","heartbeat_back_addr":"172.21.15.174:6831/2045865553","heartbeat_front_addr":"172.21.15.174:6829/2045865553","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:10:24.761290+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:10:48.565428+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:11:12.307818+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:11:37.951529+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:02.832523+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:27.568879+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:52.997707+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:13:18.937274+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:13:44.708657+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:14:10.833855+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:14:37.078272+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:15:03.878076+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:6801/526554319":"2024-04-22T23:08:50.305770+0000","172.21.15.40:6800/526554319":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/3178763471":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/4045911364":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/276300080":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/3689649814":"2024-04-22T23:08:23.743030+0000","172.21.15.40:0/2241116715":"2024-04-22T23:08:23.743030+0000","172.21.15.40:6801/1479040326":"2024-04-22T23:08:23.743030+0000","172.21.15.40:6800/1479040326":"2024-04-22T23:08:23.743030+0000","172.21.15.40:0/681298099":"2024-04-22T23:08:23.743030+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T23:16:34.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:34.764+0000 7f521c96c700 1 -- 172.21.15.40:0/1256249966 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f52040a2cb0 msgr2=0x7f52040a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:34.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:34.764+0000 7f521c96c700 1 --2- 172.21.15.40:0/1256249966 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f52040a2cb0 0x7f52040a5170 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f5208005fd0 tx=0x7f520800a560 comp rx=0 tx=0).stop 2024-04-21T23:16:34.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:34.764+0000 7f521c96c700 1 -- 172.21.15.40:0/1256249966 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5218107570 msgr2=0x7f52181987d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:34.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:34.764+0000 7f521c96c700 1 --2- 172.21.15.40:0/1256249966 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5218107570 0x7f52181987d0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f52100062a0 tx=0x7f521000bf10 comp rx=0 tx=0).stop 2024-04-21T23:16:34.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:34.764+0000 7f521c96c700 1 -- 172.21.15.40:0/1256249966 shutdown_connections 2024-04-21T23:16:34.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:34.764+0000 7f521c96c700 1 --2- 172.21.15.40:0/1256249966 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f52040a2cb0 0x7f52040a5170 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:34.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:34.764+0000 7f521c96c700 1 --2- 172.21.15.40:0/1256249966 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5218107570 0x7f52181987d0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:34.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:34.764+0000 7f521c96c700 1 --2- 172.21.15.40:0/1256249966 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52181024e0 0x7f521819e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:34.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:34.764+0000 7f521c96c700 1 --2- 172.21.15.40:0/1256249966 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5218102e90 0x7f521819eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:34.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:34.764+0000 7f521c96c700 1 -- 172.21.15.40:0/1256249966 >> 172.21.15.40:0/1256249966 conn(0x7f5218079390 msgr2=0x7f5218100330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:34.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:34.764+0000 7f521c96c700 1 -- 172.21.15.40:0/1256249966 shutdown_connections 2024-04-21T23:16:34.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:34.764+0000 7f521c96c700 1 -- 172.21.15.40:0/1256249966 wait complete. 2024-04-21T23:16:35.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:35 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/1256249966' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:16:35.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:35 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/1256249966' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:16:35.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:35 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/1256249966' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:16:35.488 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-21T23:16:35.488 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-21T23:16:35.736 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:16:36.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:36 smithi040 ceph-mon[27984]: pgmap v312: 129 pgs: 129 active+clean; 625 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 5.8 KiB/s wr, 14 op/s 2024-04-21T23:16:36.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:36 smithi174 ceph-mon[31409]: pgmap v312: 129 pgs: 129 active+clean; 625 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 5.8 KiB/s wr, 14 op/s 2024-04-21T23:16:36.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:36 smithi111 ceph-mon[31085]: pgmap v312: 129 pgs: 129 active+clean; 625 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 5.8 KiB/s wr, 14 op/s 2024-04-21T23:16:37.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:37.614+0000 7ff196a64700 1 -- 172.21.15.40:0/54540202 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff19006b550 msgr2=0x7ff190106ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:37.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:37.614+0000 7ff196a64700 1 --2- 172.21.15.40:0/54540202 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff19006b550 0x7ff190106ed0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7ff17c0099c0 tx=0x7ff17c009cd0 comp rx=0 tx=0).stop 2024-04-21T23:16:37.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:37.614+0000 7ff196a64700 1 -- 172.21.15.40:0/54540202 shutdown_connections 2024-04-21T23:16:37.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:37.614+0000 7ff196a64700 1 --2- 172.21.15.40:0/54540202 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff190107600 0x7ff19010f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:37.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:37.614+0000 7ff196a64700 1 --2- 172.21.15.40:0/54540202 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff19006aba0 0x7ff19006af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:37.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:37.614+0000 7ff196a64700 1 --2- 172.21.15.40:0/54540202 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff19006b550 0x7ff190106ed0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:37.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:37.614+0000 7ff196a64700 1 -- 172.21.15.40:0/54540202 >> 172.21.15.40:0/54540202 conn(0x7ff190077e90 msgr2=0x7ff1900782a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:37.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:37.614+0000 7ff196a64700 1 -- 172.21.15.40:0/54540202 shutdown_connections 2024-04-21T23:16:37.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:37.614+0000 7ff196a64700 1 -- 172.21.15.40:0/54540202 wait complete. 2024-04-21T23:16:37.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:37.615+0000 7ff196a64700 1 Processor -- start 2024-04-21T23:16:37.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:37.615+0000 7ff196a64700 1 -- start start 2024-04-21T23:16:37.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:37.615+0000 7ff196a64700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff19006aba0 0x7ff19019e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:37.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:37.615+0000 7ff196a64700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff19006b550 0x7ff19019ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:37.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:37.615+0000 7ff196a64700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff190107600 0x7ff1901988b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:37.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:37.615+0000 7ff196a64700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff190112590 con 0x7ff19006aba0 2024-04-21T23:16:37.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:37.615+0000 7ff196a64700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff190112410 con 0x7ff19006b550 2024-04-21T23:16:37.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:37.615+0000 7ff196a64700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff190112710 con 0x7ff190107600 2024-04-21T23:16:37.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:37.615+0000 7ff194800700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff19006aba0 0x7ff19019e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:37.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:37.615+0000 7ff194800700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff19006aba0 0x7ff19019e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49046/0 (socket says 172.21.15.40:49046) 2024-04-21T23:16:37.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:37.615+0000 7ff194800700 1 -- 172.21.15.40:0/845209959 learned_addr learned my addr 172.21.15.40:0/845209959 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:16:37.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:37.615+0000 7ff18ffff700 1 --2- 172.21.15.40:0/845209959 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff19006b550 0x7ff19019ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:37.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:37.615+0000 7ff195001700 1 --2- 172.21.15.40:0/845209959 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff190107600 0x7ff1901988b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:37.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:37.616+0000 7ff194800700 1 -- 172.21.15.40:0/845209959 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff190107600 msgr2=0x7ff1901988b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:37.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:37.616+0000 7ff194800700 1 --2- 172.21.15.40:0/845209959 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff190107600 0x7ff1901988b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:37.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:37.616+0000 7ff194800700 1 -- 172.21.15.40:0/845209959 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff19006b550 msgr2=0x7ff19019ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:37.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:37.616+0000 7ff194800700 1 --2- 172.21.15.40:0/845209959 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff19006b550 0x7ff19019ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:37.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:37.616+0000 7ff194800700 1 -- 172.21.15.40:0/845209959 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff190199010 con 0x7ff19006aba0 2024-04-21T23:16:37.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:37.616+0000 7ff194800700 1 --2- 172.21.15.40:0/845209959 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff19006aba0 0x7ff19019e6f0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7ff18400e9a0 tx=0x7ff18400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:37.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:37.616+0000 7ff18dffb700 1 -- 172.21.15.40:0/845209959 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff18400ca10 con 0x7ff19006aba0 2024-04-21T23:16:37.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:37.616+0000 7ff196a64700 1 -- 172.21.15.40:0/845209959 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff190199300 con 0x7ff19006aba0 2024-04-21T23:16:37.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:37.616+0000 7ff18dffb700 1 -- 172.21.15.40:0/845209959 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff18400cbb0 con 0x7ff19006aba0 2024-04-21T23:16:37.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:37.616+0000 7ff18dffb700 1 -- 172.21.15.40:0/845209959 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff184018680 con 0x7ff19006aba0 2024-04-21T23:16:37.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:37.616+0000 7ff196a64700 1 -- 172.21.15.40:0/845209959 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1901a55c0 con 0x7ff19006aba0 2024-04-21T23:16:37.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:37.617+0000 7ff196a64700 1 -- 172.21.15.40:0/845209959 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff190068f20 con 0x7ff19006aba0 2024-04-21T23:16:37.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:37.618+0000 7ff18dffb700 1 -- 172.21.15.40:0/845209959 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7ff184010430 con 0x7ff19006aba0 2024-04-21T23:16:37.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:37.619+0000 7ff18dffb700 1 --2- 172.21.15.40:0/845209959 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7ff1800a2cb0 0x7ff1800a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:37.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:37.619+0000 7ff18ffff700 1 --2- 172.21.15.40:0/845209959 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7ff1800a2cb0 0x7ff1800a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:37.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:37.619+0000 7ff18dffb700 1 -- 172.21.15.40:0/845209959 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7ff1840cf650 con 0x7ff19006aba0 2024-04-21T23:16:37.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:37.620+0000 7ff18ffff700 1 --2- 172.21.15.40:0/845209959 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7ff1800a2cb0 0x7ff1800a5170 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff17c009990 tx=0x7ff17c005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:37.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:37.624+0000 7ff18dffb700 1 -- 172.21.15.40:0/845209959 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7ff18409c210 con 0x7ff19006aba0 2024-04-21T23:16:37.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:37.832+0000 7ff196a64700 1 -- 172.21.15.40:0/845209959 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff19002db30 con 0x7ff19006aba0 2024-04-21T23:16:37.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:37.834+0000 7ff18dffb700 1 -- 172.21.15.40:0/845209959 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24138 (secure 0 0 0) 0x7ff18409bb10 con 0x7ff19006aba0 2024-04-21T23:16:37.833 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:16:37.834 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":90,"fsid":"d814e4dc-0033-11ef-bc93-c7b262605968","created":"2024-04-21T23:07:52.659207+0000","modified":"2024-04-21T23:16:00.780824+0000","last_up_change":"2024-04-21T23:15:05.308821+0000","last_in_change":"2024-04-21T23:14:45.447491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:11:14.824775+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T23:15:54.322571+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T23:15:55.164493+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T23:15:58.838266+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"0e8d692a-22fe-4a87-8d99-9151f96cbb02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6803","nonce":2009758784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6805","nonce":2009758784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6809","nonce":2009758784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6807","nonce":2009758784}]},"public_addr":"172.21.15.40:6803/2009758784","cluster_addr":"172.21.15.40:6805/2009758784","heartbeat_back_addr":"172.21.15.40:6809/2009758784","heartbeat_front_addr":"172.21.15.40:6807/2009758784","state":["exists","up"]},{"osd":1,"uuid":"fe5fe11b-0e84-4359-83a7-1ba45105755a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6801","nonce":1763903836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6803","nonce":1763903836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6807","nonce":1763903836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6805","nonce":1763903836}]},"public_addr":"172.21.15.111:6801/1763903836","cluster_addr":"172.21.15.111:6803/1763903836","heartbeat_back_addr":"172.21.15.111:6807/1763903836","heartbeat_front_addr":"172.21.15.111:6805/1763903836","state":["exists","up"]},{"osd":2,"uuid":"813ad735-5cc2-4da0-ae16-6db238395fbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6801","nonce":439695825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6803","nonce":439695825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6807","nonce":439695825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6805","nonce":439695825}]},"public_addr":"172.21.15.174:6801/439695825","cluster_addr":"172.21.15.174:6803/439695825","heartbeat_back_addr":"172.21.15.174:6807/439695825","heartbeat_front_addr":"172.21.15.174:6805/439695825","state":["exists","up"]},{"osd":3,"uuid":"09becb03-b7ce-4ffc-917d-b7900ec37e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6811","nonce":3454312436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6813","nonce":3454312436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6817","nonce":3454312436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6815","nonce":3454312436}]},"public_addr":"172.21.15.40:6811/3454312436","cluster_addr":"172.21.15.40:6813/3454312436","heartbeat_back_addr":"172.21.15.40:6817/3454312436","heartbeat_front_addr":"172.21.15.40:6815/3454312436","state":["exists","up"]},{"osd":4,"uuid":"46eabfd7-d88a-4efd-94eb-04354d3a9d7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6809","nonce":1810866070}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6811","nonce":1810866070}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6815","nonce":1810866070}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6813","nonce":1810866070}]},"public_addr":"172.21.15.111:6809/1810866070","cluster_addr":"172.21.15.111:6811/1810866070","heartbeat_back_addr":"172.21.15.111:6815/1810866070","heartbeat_front_addr":"172.21.15.111:6813/1810866070","state":["exists","up"]},{"osd":5,"uuid":"6f73b582-6c65-41db-aab0-9b360f83d5c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6809","nonce":3752986711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6811","nonce":3752986711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6815","nonce":3752986711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6813","nonce":3752986711}]},"public_addr":"172.21.15.174:6809/3752986711","cluster_addr":"172.21.15.174:6811/3752986711","heartbeat_back_addr":"172.21.15.174:6815/3752986711","heartbeat_front_addr":"172.21.15.174:6813/3752986711","state":["exists","up"]},{"osd":6,"uuid":"39f305f0-12d2-4732-aaa0-cd66700af538","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6819","nonce":1618855427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6821","nonce":1618855427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6825","nonce":1618855427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6823","nonce":1618855427}]},"public_addr":"172.21.15.40:6819/1618855427","cluster_addr":"172.21.15.40:6821/1618855427","heartbeat_back_addr":"172.21.15.40:6825/1618855427","heartbeat_front_addr":"172.21.15.40:6823/1618855427","state":["exists","up"]},{"osd":7,"uuid":"769f2468-5102-490b-89f8-d0444d3169a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6817","nonce":927054162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6819","nonce":927054162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6823","nonce":927054162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6821","nonce":927054162}]},"public_addr":"172.21.15.111:6817/927054162","cluster_addr":"172.21.15.111:6819/927054162","heartbeat_back_addr":"172.21.15.111:6823/927054162","heartbeat_front_addr":"172.21.15.111:6821/927054162","state":["exists","up"]},{"osd":8,"uuid":"1a444e80-007d-473b-b972-4c814a813c22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6817","nonce":3817848746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6819","nonce":3817848746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6823","nonce":3817848746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6821","nonce":3817848746}]},"public_addr":"172.21.15.174:6817/3817848746","cluster_addr":"172.21.15.174:6819/3817848746","heartbeat_back_addr":"172.21.15.174:6823/3817848746","heartbeat_front_addr":"172.21.15.174:6821/3817848746","state":["exists","up"]},{"osd":9,"uuid":"9a6b6a56-6ae5-49fe-9e5b-cb17d686e0dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6827","nonce":1649446531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6829","nonce":1649446531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6833","nonce":1649446531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6831","nonce":1649446531}]},"public_addr":"172.21.15.40:6827/1649446531","cluster_addr":"172.21.15.40:6829/1649446531","heartbeat_back_addr":"172.21.15.40:6833/1649446531","heartbeat_front_addr":"172.21.15.40:6831/1649446531","state":["exists","up"]},{"osd":10,"uuid":"a70038ba-e2ad-4170-85cc-ea0470a2edef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6825","nonce":1227538935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6827","nonce":1227538935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6831","nonce":1227538935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6829","nonce":1227538935}]},"public_addr":"172.21.15.111:6825/1227538935","cluster_addr":"172.21.15.111:6827/1227538935","heartbeat_back_addr":"172.21.15.111:6831/1227538935","heartbeat_front_addr":"172.21.15.111:6829/1227538935","state":["exists","up"]},{"osd":11,"uuid":"781532bb-7a5e-44e3-8eb7-b584b0054666","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6825","nonce":2045865553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6827","nonce":2045865553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6831","nonce":2045865553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6829","nonce":2045865553}]},"public_addr":"172.21.15.174:6825/2045865553","cluster_addr":"172.21.15.174:6827/2045865553","heartbeat_back_addr":"172.21.15.174:6831/2045865553","heartbeat_front_addr":"172.21.15.174:6829/2045865553","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:10:24.761290+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:10:48.565428+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:11:12.307818+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:11:37.951529+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:02.832523+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:27.568879+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:52.997707+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:13:18.937274+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:13:44.708657+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:14:10.833855+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:14:37.078272+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:15:03.878076+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:6801/526554319":"2024-04-22T23:08:50.305770+0000","172.21.15.40:6800/526554319":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/3178763471":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/4045911364":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/276300080":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/3689649814":"2024-04-22T23:08:23.743030+0000","172.21.15.40:0/2241116715":"2024-04-22T23:08:23.743030+0000","172.21.15.40:6801/1479040326":"2024-04-22T23:08:23.743030+0000","172.21.15.40:6800/1479040326":"2024-04-22T23:08:23.743030+0000","172.21.15.40:0/681298099":"2024-04-22T23:08:23.743030+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T23:16:37.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:37.836+0000 7ff196a64700 1 -- 172.21.15.40:0/845209959 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7ff1800a2cb0 msgr2=0x7ff1800a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:37.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:37.836+0000 7ff196a64700 1 --2- 172.21.15.40:0/845209959 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7ff1800a2cb0 0x7ff1800a5170 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff17c009990 tx=0x7ff17c005fb0 comp rx=0 tx=0).stop 2024-04-21T23:16:37.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:37.836+0000 7ff196a64700 1 -- 172.21.15.40:0/845209959 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff19006aba0 msgr2=0x7ff19019e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:37.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:37.836+0000 7ff196a64700 1 --2- 172.21.15.40:0/845209959 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff19006aba0 0x7ff19019e6f0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7ff18400e9a0 tx=0x7ff18400ecb0 comp rx=0 tx=0).stop 2024-04-21T23:16:37.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:37.836+0000 7ff196a64700 1 -- 172.21.15.40:0/845209959 shutdown_connections 2024-04-21T23:16:37.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:37.836+0000 7ff196a64700 1 --2- 172.21.15.40:0/845209959 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff190107600 0x7ff1901988b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:37.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:37.836+0000 7ff196a64700 1 --2- 172.21.15.40:0/845209959 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7ff1800a2cb0 0x7ff1800a5170 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:37.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:37.836+0000 7ff196a64700 1 --2- 172.21.15.40:0/845209959 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff19006aba0 0x7ff19019e6f0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:37.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:37.836+0000 7ff196a64700 1 --2- 172.21.15.40:0/845209959 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff19006b550 0x7ff19019ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:37.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:37.836+0000 7ff196a64700 1 -- 172.21.15.40:0/845209959 >> 172.21.15.40:0/845209959 conn(0x7ff190077e90 msgr2=0x7ff1900ff710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:37.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:37.836+0000 7ff196a64700 1 -- 172.21.15.40:0/845209959 shutdown_connections 2024-04-21T23:16:37.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:37.837+0000 7ff196a64700 1 -- 172.21.15.40:0/845209959 wait complete. 2024-04-21T23:16:38.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:38 smithi040 ceph-mon[27984]: pgmap v313: 129 pgs: 129 active+clean; 640 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 6.5 KiB/s wr, 13 op/s 2024-04-21T23:16:38.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:38 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/845209959' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:16:38.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:38 smithi174 ceph-mon[31409]: pgmap v313: 129 pgs: 129 active+clean; 640 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 6.5 KiB/s wr, 13 op/s 2024-04-21T23:16:38.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:38 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/845209959' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:16:38.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:38 smithi111 ceph-mon[31085]: pgmap v313: 129 pgs: 129 active+clean; 640 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 6.5 KiB/s wr, 13 op/s 2024-04-21T23:16:38.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:38 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/845209959' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:16:38.662 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-21T23:16:38.663 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-21T23:16:38.663 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-21T23:16:38.664 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-21T23:16:38.664 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-21T23:16:38.664 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-21T23:16:38.665 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-21T23:16:38.665 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-21T23:16:38.666 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-04-21T23:16:38.666 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-04-21T23:16:38.666 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-04-21T23:16:38.667 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-04-21T23:16:39.395 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:16:39.519 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:16:39.524 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:16:39.524 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:16:39.524 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:16:39.525 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:16:39.762 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:16:39.763 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:16:39.764 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:16:39.765 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:16:39.766 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:16:40.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:40 smithi040 ceph-mon[27984]: pgmap v314: 129 pgs: 129 active+clean; 640 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 2.2 KiB/s wr, 6 op/s 2024-04-21T23:16:40.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:40 smithi174 ceph-mon[31409]: pgmap v314: 129 pgs: 129 active+clean; 640 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 2.2 KiB/s wr, 6 op/s 2024-04-21T23:16:40.443 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:40 smithi111 ceph-mon[31085]: pgmap v314: 129 pgs: 129 active+clean; 640 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 2.2 KiB/s wr, 6 op/s 2024-04-21T23:16:42.329 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:42 smithi040 ceph-mon[27984]: pgmap v315: 129 pgs: 129 active+clean; 640 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 2.0 KiB/s wr, 8 op/s 2024-04-21T23:16:42.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:42 smithi174 ceph-mon[31409]: pgmap v315: 129 pgs: 129 active+clean; 640 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 2.0 KiB/s wr, 8 op/s 2024-04-21T23:16:42.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:42 smithi111 ceph-mon[31085]: pgmap v315: 129 pgs: 129 active+clean; 640 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 2.0 KiB/s wr, 8 op/s 2024-04-21T23:16:43.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:43.202+0000 7f5310222700 1 -- 172.21.15.40:0/2936907899 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53081082b0 msgr2=0x7f5308108690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:43.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:43.202+0000 7f5310222700 1 --2- 172.21.15.40:0/2936907899 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53081082b0 0x7f5308108690 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f5304009990 tx=0x7f5304009ca0 comp rx=0 tx=0).stop 2024-04-21T23:16:43.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:43.202+0000 7f5310222700 1 -- 172.21.15.40:0/2936907899 shutdown_connections 2024-04-21T23:16:43.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:43.202+0000 7f5310222700 1 --2- 172.21.15.40:0/2936907899 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f530806a8e0 0x7f530810f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:43.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:43.202+0000 7f5310222700 1 --2- 172.21.15.40:0/2936907899 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53081082b0 0x7f5308108690 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:43.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:43.202+0000 7f5310222700 1 --2- 172.21.15.40:0/2936907899 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5308069f40 0x7f530806a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:43.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:43.202+0000 7f5310222700 1 -- 172.21.15.40:0/2936907899 >> 172.21.15.40:0/2936907899 conn(0x7f53080780e0 msgr2=0x7f53080784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:43.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:43.202+0000 7f5310222700 1 -- 172.21.15.40:0/2936907899 shutdown_connections 2024-04-21T23:16:43.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:43.202+0000 7f5310222700 1 -- 172.21.15.40:0/2936907899 wait complete. 2024-04-21T23:16:43.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:43.203+0000 7f5310222700 1 Processor -- start 2024-04-21T23:16:43.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:43.203+0000 7f5310222700 1 -- start start 2024-04-21T23:16:43.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:43.203+0000 7f5310222700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5308069f40 0x7f5308101200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:43.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:43.203+0000 7f5310222700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f530806a8e0 0x7f5308101740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:43.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:43.203+0000 7f5310222700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f53081082b0 0x7f5308101c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:43.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:43.203+0000 7f5310222700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5308112640 con 0x7f530806a8e0 2024-04-21T23:16:43.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:43.203+0000 7f5310222700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f53081124c0 con 0x7f5308069f40 2024-04-21T23:16:43.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:43.203+0000 7f5310222700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f53081127c0 con 0x7f53081082b0 2024-04-21T23:16:43.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:43.203+0000 7f530d7bd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f530806a8e0 0x7f5308101740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:43.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:43.203+0000 7f530d7bd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f530806a8e0 0x7f5308101740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40320/0 (socket says 172.21.15.40:40320) 2024-04-21T23:16:43.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:43.203+0000 7f530d7bd700 1 -- 172.21.15.40:0/2761462947 learned_addr learned my addr 172.21.15.40:0/2761462947 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:16:43.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:43.203+0000 7f530e7bf700 1 --2- 172.21.15.40:0/2761462947 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f53081082b0 0x7f5308101c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:43.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:43.203+0000 7f530dfbe700 1 --2- 172.21.15.40:0/2761462947 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5308069f40 0x7f5308101200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:43.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:43.204+0000 7f530d7bd700 1 -- 172.21.15.40:0/2761462947 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f53081082b0 msgr2=0x7f5308101c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:43.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:43.204+0000 7f530d7bd700 1 --2- 172.21.15.40:0/2761462947 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f53081082b0 0x7f5308101c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:43.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:43.204+0000 7f530d7bd700 1 -- 172.21.15.40:0/2761462947 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5308069f40 msgr2=0x7f5308101200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:43.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:43.204+0000 7f530d7bd700 1 --2- 172.21.15.40:0/2761462947 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5308069f40 0x7f5308101200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:43.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:43.204+0000 7f530d7bd700 1 -- 172.21.15.40:0/2761462947 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53081a70e0 con 0x7f530806a8e0 2024-04-21T23:16:43.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:43.204+0000 7f530d7bd700 1 --2- 172.21.15.40:0/2761462947 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f530806a8e0 0x7f5308101740 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f52f800b530 tx=0x7f52f800b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:43.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:43.204+0000 7f52feffd700 1 -- 172.21.15.40:0/2761462947 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52f8015020 con 0x7f530806a8e0 2024-04-21T23:16:43.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:43.204+0000 7f52feffd700 1 -- 172.21.15.40:0/2761462947 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f52f8011d10 con 0x7f530806a8e0 2024-04-21T23:16:43.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:43.204+0000 7f5310222700 1 -- 172.21.15.40:0/2761462947 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53081a73d0 con 0x7f530806a8e0 2024-04-21T23:16:43.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:43.204+0000 7f52feffd700 1 -- 172.21.15.40:0/2761462947 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52f800f2e0 con 0x7f530806a8e0 2024-04-21T23:16:43.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:43.204+0000 7f5310222700 1 -- 172.21.15.40:0/2761462947 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53081a7910 con 0x7f530806a8e0 2024-04-21T23:16:43.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:43.205+0000 7f5310222700 1 -- 172.21.15.40:0/2761462947 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5308062940 con 0x7f530806a8e0 2024-04-21T23:16:43.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:43.206+0000 7f52feffd700 1 -- 172.21.15.40:0/2761462947 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f52f8018020 con 0x7f530806a8e0 2024-04-21T23:16:43.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:43.207+0000 7f52feffd700 1 --2- 172.21.15.40:0/2761462947 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f52f40a2cb0 0x7f52f40a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:43.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:43.207+0000 7f530dfbe700 1 --2- 172.21.15.40:0/2761462947 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f52f40a2cb0 0x7f52f40a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:43.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:43.207+0000 7f52feffd700 1 -- 172.21.15.40:0/2761462947 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f52f80cefa0 con 0x7f530806a8e0 2024-04-21T23:16:43.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:43.207+0000 7f52feffd700 1 --2- 172.21.15.40:0/2761462947 >> [v2:172.21.15.174:6824/2045865553,v1:172.21.15.174:6825/2045865553] conn(0x7f52f40ad970 0x7f52f40afdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:43.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:43.207+0000 7f52feffd700 1 -- 172.21.15.40:0/2761462947 --> [v2:172.21.15.174:6824/2045865553,v1:172.21.15.174:6825/2045865553] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f52f8017e50 con 0x7f52f40ad970 2024-04-21T23:16:43.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:43.207+0000 7f52feffd700 1 -- 172.21.15.40:0/2761462947 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f52f80cf3f0 con 0x7f530806a8e0 2024-04-21T23:16:43.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:43.207+0000 7f530dfbe700 1 --2- 172.21.15.40:0/2761462947 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f52f40a2cb0 0x7f52f40a5170 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f5304009750 tx=0x7f5304005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:43.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:43.207+0000 7f530e7bf700 1 --2- 172.21.15.40:0/2761462947 >> [v2:172.21.15.174:6824/2045865553,v1:172.21.15.174:6825/2045865553] conn(0x7f52f40ad970 0x7f52f40afdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:43.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:43.208+0000 7f530e7bf700 1 --2- 172.21.15.40:0/2761462947 >> [v2:172.21.15.174:6824/2045865553,v1:172.21.15.174:6825/2045865553] conn(0x7f52f40ad970 0x7f52f40afdb0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:43.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:43.209+0000 7f52feffd700 1 -- 172.21.15.40:0/2761462947 <== osd.11 v2:172.21.15.174:6824/2045865553 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f52f8017e50 con 0x7f52f40ad970 2024-04-21T23:16:43.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:43.225+0000 7f5310222700 1 -- 172.21.15.40:0/2761462947 --> [v2:172.21.15.174:6824/2045865553,v1:172.21.15.174:6825/2045865553] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5308062dc0 con 0x7f52f40ad970 2024-04-21T23:16:43.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:43.226+0000 7f52feffd700 1 -- 172.21.15.40:0/2761462947 <== osd.11 v2:172.21.15.174:6824/2045865553 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f5308062dc0 con 0x7f52f40ad970 2024-04-21T23:16:43.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:43.226+0000 7f5310222700 1 -- 172.21.15.40:0/2761462947 >> [v2:172.21.15.174:6824/2045865553,v1:172.21.15.174:6825/2045865553] conn(0x7f52f40ad970 msgr2=0x7f52f40afdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:43.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:43.226+0000 7f5310222700 1 --2- 172.21.15.40:0/2761462947 >> [v2:172.21.15.174:6824/2045865553,v1:172.21.15.174:6825/2045865553] conn(0x7f52f40ad970 0x7f52f40afdb0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:43.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:43.226+0000 7f5310222700 1 -- 172.21.15.40:0/2761462947 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f52f40a2cb0 msgr2=0x7f52f40a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:43.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:43.226+0000 7f5310222700 1 --2- 172.21.15.40:0/2761462947 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f52f40a2cb0 0x7f52f40a5170 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f5304009750 tx=0x7f5304005fd0 comp rx=0 tx=0).stop 2024-04-21T23:16:43.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:43.226+0000 7f5310222700 1 -- 172.21.15.40:0/2761462947 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f530806a8e0 msgr2=0x7f5308101740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:43.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:43.226+0000 7f5310222700 1 --2- 172.21.15.40:0/2761462947 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f530806a8e0 0x7f5308101740 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f52f800b530 tx=0x7f52f800b840 comp rx=0 tx=0).stop 2024-04-21T23:16:43.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:43.226+0000 7f530e7bf700 1 -- 172.21.15.40:0/2761462947 reap_dead start 2024-04-21T23:16:43.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:43.226+0000 7f5310222700 1 -- 172.21.15.40:0/2761462947 shutdown_connections 2024-04-21T23:16:43.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:43.226+0000 7f5310222700 1 -- 172.21.15.40:0/2761462947 >> 172.21.15.40:0/2761462947 conn(0x7f53080780e0 msgr2=0x7f53080ffe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:43.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:43.226+0000 7f5310222700 1 -- 172.21.15.40:0/2761462947 shutdown_connections 2024-04-21T23:16:43.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:43.226+0000 7f5310222700 1 -- 172.21.15.40:0/2761462947 wait complete. 2024-04-21T23:16:44.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:44 smithi040 ceph-mon[27984]: pgmap v316: 129 pgs: 129 active+clean; 640 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-04-21T23:16:44.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:44 smithi174 ceph-mon[31409]: pgmap v316: 129 pgs: 129 active+clean; 640 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-04-21T23:16:44.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:44 smithi111 ceph-mon[31085]: pgmap v316: 129 pgs: 129 active+clean; 640 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-04-21T23:16:45.367 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:16:46.417 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:46 smithi040 ceph-mon[27984]: pgmap v317: 129 pgs: 129 active+clean; 640 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-04-21T23:16:46.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:46 smithi174 ceph-mon[31409]: pgmap v317: 129 pgs: 129 active+clean; 640 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-04-21T23:16:46.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:46 smithi111 ceph-mon[31085]: pgmap v317: 129 pgs: 129 active+clean; 640 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-04-21T23:16:48.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:48 smithi040 ceph-mon[27984]: pgmap v318: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-04-21T23:16:48.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:48 smithi174 ceph-mon[31409]: pgmap v318: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-04-21T23:16:48.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:48 smithi111 ceph-mon[31085]: pgmap v318: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-04-21T23:16:48.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.594+0000 7f0010ca9700 1 -- 172.21.15.40:0/2817057050 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f000c1082b0 msgr2=0x7f000c108690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:48.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.594+0000 7f0010ca9700 1 --2- 172.21.15.40:0/2817057050 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f000c1082b0 0x7f000c108690 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7efffc009990 tx=0x7efffc009ca0 comp rx=0 tx=0).stop 2024-04-21T23:16:48.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.594+0000 7f0010ca9700 1 -- 172.21.15.40:0/2817057050 shutdown_connections 2024-04-21T23:16:48.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.594+0000 7f0010ca9700 1 --2- 172.21.15.40:0/2817057050 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f000c06a8e0 0x7f000c10f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:48.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.594+0000 7f0010ca9700 1 --2- 172.21.15.40:0/2817057050 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f000c1082b0 0x7f000c108690 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:48.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.594+0000 7f0010ca9700 1 --2- 172.21.15.40:0/2817057050 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f000c069f40 0x7f000c06a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:48.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.594+0000 7f0010ca9700 1 -- 172.21.15.40:0/2817057050 >> 172.21.15.40:0/2817057050 conn(0x7f000c0780e0 msgr2=0x7f000c0784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:48.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.594+0000 7f0010ca9700 1 -- 172.21.15.40:0/2817057050 shutdown_connections 2024-04-21T23:16:48.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.594+0000 7f0010ca9700 1 -- 172.21.15.40:0/2817057050 wait complete. 2024-04-21T23:16:48.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.595+0000 7f0010ca9700 1 Processor -- start 2024-04-21T23:16:48.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.595+0000 7f0010ca9700 1 -- start start 2024-04-21T23:16:48.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.595+0000 7f0010ca9700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f000c069f40 0x7f000c19e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:48.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.595+0000 7f0010ca9700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f000c06a8e0 0x7f000c19ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:48.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.595+0000 7f0010ca9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f000c1082b0 0x7f000c1988f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:48.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.595+0000 7f0010ca9700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f000c112570 con 0x7f000c069f40 2024-04-21T23:16:48.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.595+0000 7f0010ca9700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f000c1123f0 con 0x7f000c1082b0 2024-04-21T23:16:48.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.595+0000 7f0010ca9700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f000c1126f0 con 0x7f000c06a8e0 2024-04-21T23:16:48.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.595+0000 7f000a59c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f000c069f40 0x7f000c19e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:48.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.595+0000 7f000a59c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f000c069f40 0x7f000c19e730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40340/0 (socket says 172.21.15.40:40340) 2024-04-21T23:16:48.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.595+0000 7f000a59c700 1 -- 172.21.15.40:0/2693098421 learned_addr learned my addr 172.21.15.40:0/2693098421 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:16:48.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.595+0000 7f0009d9b700 1 --2- 172.21.15.40:0/2693098421 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f000c06a8e0 0x7f000c19ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:48.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.595+0000 7f000ad9d700 1 --2- 172.21.15.40:0/2693098421 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f000c1082b0 0x7f000c1988f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:48.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.595+0000 7f000a59c700 1 -- 172.21.15.40:0/2693098421 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f000c06a8e0 msgr2=0x7f000c19ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:48.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.595+0000 7f000a59c700 1 --2- 172.21.15.40:0/2693098421 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f000c06a8e0 0x7f000c19ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:48.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.595+0000 7f000a59c700 1 -- 172.21.15.40:0/2693098421 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f000c1082b0 msgr2=0x7f000c1988f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:48.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.595+0000 7f000a59c700 1 --2- 172.21.15.40:0/2693098421 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f000c1082b0 0x7f000c1988f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:48.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.595+0000 7f000a59c700 1 -- 172.21.15.40:0/2693098421 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f000c199050 con 0x7f000c069f40 2024-04-21T23:16:48.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.595+0000 7f000ad9d700 1 --2- 172.21.15.40:0/2693098421 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f000c1082b0 0x7f000c1988f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:16:48.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.596+0000 7f000a59c700 1 --2- 172.21.15.40:0/2693098421 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f000c069f40 0x7f000c19e730 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7efffc0038b0 tx=0x7efffc009f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:48.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.596+0000 7efffb7fe700 1 -- 172.21.15.40:0/2693098421 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efffc027070 con 0x7f000c069f40 2024-04-21T23:16:48.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.596+0000 7efffb7fe700 1 -- 172.21.15.40:0/2693098421 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efffc004050 con 0x7f000c069f40 2024-04-21T23:16:48.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.596+0000 7f0010ca9700 1 -- 172.21.15.40:0/2693098421 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f000c1992e0 con 0x7f000c069f40 2024-04-21T23:16:48.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.596+0000 7efffb7fe700 1 -- 172.21.15.40:0/2693098421 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efffc022cd0 con 0x7f000c069f40 2024-04-21T23:16:48.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.596+0000 7f0010ca9700 1 -- 172.21.15.40:0/2693098421 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f000c1a5600 con 0x7f000c069f40 2024-04-21T23:16:48.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.597+0000 7f0010ca9700 1 -- 172.21.15.40:0/2693098421 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f000c062940 con 0x7f000c069f40 2024-04-21T23:16:48.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.598+0000 7efffb7fe700 1 -- 172.21.15.40:0/2693098421 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7efffc02a070 con 0x7f000c069f40 2024-04-21T23:16:48.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.599+0000 7efffb7fe700 1 --2- 172.21.15.40:0/2693098421 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7efff40a2d30 0x7efff40a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:48.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.599+0000 7f0009d9b700 1 --2- 172.21.15.40:0/2693098421 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7efff40a2d30 0x7efff40a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:48.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.599+0000 7efffb7fe700 1 -- 172.21.15.40:0/2693098421 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7efffc0d0d50 con 0x7f000c069f40 2024-04-21T23:16:48.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.599+0000 7efffb7fe700 1 --2- 172.21.15.40:0/2693098421 >> [v2:172.21.15.40:6826/1649446531,v1:172.21.15.40:6827/1649446531] conn(0x7efff40ad9f0 0x7efff40afe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:48.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.599+0000 7efffb7fe700 1 -- 172.21.15.40:0/2693098421 --> [v2:172.21.15.40:6826/1649446531,v1:172.21.15.40:6827/1649446531] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7efffc010ec0 con 0x7efff40ad9f0 2024-04-21T23:16:48.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.599+0000 7efffb7fe700 1 -- 172.21.15.40:0/2693098421 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7efffc0d1100 con 0x7f000c069f40 2024-04-21T23:16:48.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.599+0000 7f000ad9d700 1 --2- 172.21.15.40:0/2693098421 >> [v2:172.21.15.40:6826/1649446531,v1:172.21.15.40:6827/1649446531] conn(0x7efff40ad9f0 0x7efff40afe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:48.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.599+0000 7f0009d9b700 1 --2- 172.21.15.40:0/2693098421 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7efff40a2d30 0x7efff40a51f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f000c078800 tx=0x7f000000b410 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:48.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.600+0000 7f000ad9d700 1 --2- 172.21.15.40:0/2693098421 >> [v2:172.21.15.40:6826/1649446531,v1:172.21.15.40:6827/1649446531] conn(0x7efff40ad9f0 0x7efff40afe30 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:48.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.600+0000 7efffb7fe700 1 -- 172.21.15.40:0/2693098421 <== osd.9 v2:172.21.15.40:6826/1649446531 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7efffc010ec0 con 0x7efff40ad9f0 2024-04-21T23:16:48.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.617+0000 7f0010ca9700 1 -- 172.21.15.40:0/2693098421 --> [v2:172.21.15.40:6826/1649446531,v1:172.21.15.40:6827/1649446531] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f000c062d40 con 0x7efff40ad9f0 2024-04-21T23:16:48.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.617+0000 7efffb7fe700 1 -- 172.21.15.40:0/2693098421 <== osd.9 v2:172.21.15.40:6826/1649446531 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f000c062d40 con 0x7efff40ad9f0 2024-04-21T23:16:48.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.617+0000 7f0010ca9700 1 -- 172.21.15.40:0/2693098421 >> [v2:172.21.15.40:6826/1649446531,v1:172.21.15.40:6827/1649446531] conn(0x7efff40ad9f0 msgr2=0x7efff40afe30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:48.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.617+0000 7f0010ca9700 1 --2- 172.21.15.40:0/2693098421 >> [v2:172.21.15.40:6826/1649446531,v1:172.21.15.40:6827/1649446531] conn(0x7efff40ad9f0 0x7efff40afe30 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:48.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.617+0000 7f0010ca9700 1 -- 172.21.15.40:0/2693098421 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7efff40a2d30 msgr2=0x7efff40a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:48.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.617+0000 7f0010ca9700 1 --2- 172.21.15.40:0/2693098421 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7efff40a2d30 0x7efff40a51f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f000c078800 tx=0x7f000000b410 comp rx=0 tx=0).stop 2024-04-21T23:16:48.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.617+0000 7f0010ca9700 1 -- 172.21.15.40:0/2693098421 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f000c069f40 msgr2=0x7f000c19e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:48.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.617+0000 7f0010ca9700 1 --2- 172.21.15.40:0/2693098421 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f000c069f40 0x7f000c19e730 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7efffc0038b0 tx=0x7efffc009f50 comp rx=0 tx=0).stop 2024-04-21T23:16:48.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.617+0000 7f000ad9d700 1 -- 172.21.15.40:0/2693098421 reap_dead start 2024-04-21T23:16:48.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.617+0000 7f0010ca9700 1 -- 172.21.15.40:0/2693098421 shutdown_connections 2024-04-21T23:16:48.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.617+0000 7f0010ca9700 1 -- 172.21.15.40:0/2693098421 >> 172.21.15.40:0/2693098421 conn(0x7f000c0780e0 msgr2=0x7f000c0fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:48.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.617+0000 7f0010ca9700 1 -- 172.21.15.40:0/2693098421 shutdown_connections 2024-04-21T23:16:48.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.617+0000 7f0010ca9700 1 -- 172.21.15.40:0/2693098421 wait complete. 2024-04-21T23:16:49.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.799+0000 7f6691296700 1 -- 172.21.15.40:0/3565502733 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f668c1071a0 msgr2=0x7f668c107580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:49.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.799+0000 7f6691296700 1 --2- 172.21.15.40:0/3565502733 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f668c1071a0 0x7f668c107580 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f667c009990 tx=0x7f667c009ca0 comp rx=0 tx=0).stop 2024-04-21T23:16:49.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.799+0000 7f6691296700 1 -- 172.21.15.40:0/3565502733 shutdown_connections 2024-04-21T23:16:49.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.799+0000 7f6691296700 1 --2- 172.21.15.40:0/3565502733 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f668c101af0 0x7f668c10dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:49.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.799+0000 7f6691296700 1 --2- 172.21.15.40:0/3565502733 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f668c1071a0 0x7f668c107580 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:49.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.799+0000 7f6691296700 1 --2- 172.21.15.40:0/3565502733 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f668c101130 0x7f668c1015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:49.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.799+0000 7f6691296700 1 -- 172.21.15.40:0/3565502733 >> 172.21.15.40:0/3565502733 conn(0x7f668c0780e0 msgr2=0x7f668c0784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:49.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.800+0000 7f6691296700 1 -- 172.21.15.40:0/3565502733 shutdown_connections 2024-04-21T23:16:49.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.800+0000 7f6691296700 1 -- 172.21.15.40:0/3565502733 wait complete. 2024-04-21T23:16:49.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.800+0000 7f6691296700 1 Processor -- start 2024-04-21T23:16:49.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.800+0000 7f6691296700 1 -- start start 2024-04-21T23:16:49.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.800+0000 7f6691296700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f668c101130 0x7f668c073c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:49.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.800+0000 7f6691296700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f668c101af0 0x7f668c069f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:49.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.800+0000 7f6691296700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f668c1071a0 0x7f668c06a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:49.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.800+0000 7f6691296700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f668c111490 con 0x7f668c101af0 2024-04-21T23:16:49.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.800+0000 7f6691296700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f668c111310 con 0x7f668c1071a0 2024-04-21T23:16:49.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.800+0000 7f6691296700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f668c111610 con 0x7f668c101130 2024-04-21T23:16:49.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.801+0000 7f6683fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f668c101af0 0x7f668c069f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:49.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.801+0000 7f6683fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f668c101af0 0x7f668c069f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40352/0 (socket says 172.21.15.40:40352) 2024-04-21T23:16:49.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.801+0000 7f6683fff700 1 -- 172.21.15.40:0/190955459 learned_addr learned my addr 172.21.15.40:0/190955459 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:16:49.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.801+0000 7f668affd700 1 --2- 172.21.15.40:0/190955459 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f668c101130 0x7f668c073c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:49.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.801+0000 7f668b7fe700 1 --2- 172.21.15.40:0/190955459 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f668c1071a0 0x7f668c06a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:49.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.801+0000 7f6683fff700 1 -- 172.21.15.40:0/190955459 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f668c101130 msgr2=0x7f668c073c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:49.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.801+0000 7f6683fff700 1 --2- 172.21.15.40:0/190955459 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f668c101130 0x7f668c073c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:49.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.801+0000 7f6683fff700 1 -- 172.21.15.40:0/190955459 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f668c1071a0 msgr2=0x7f668c06a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:49.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.801+0000 7f6683fff700 1 --2- 172.21.15.40:0/190955459 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f668c1071a0 0x7f668c06a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:49.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.801+0000 7f6683fff700 1 -- 172.21.15.40:0/190955459 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f668c06ac90 con 0x7f668c101af0 2024-04-21T23:16:49.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.801+0000 7f668affd700 1 --2- 172.21.15.40:0/190955459 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f668c101130 0x7f668c073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:16:49.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.801+0000 7f6683fff700 1 --2- 172.21.15.40:0/190955459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f668c101af0 0x7f668c069f10 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f667400d980 tx=0x7f667400dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:49.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.801+0000 7f6688ff9700 1 -- 172.21.15.40:0/190955459 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6674016070 con 0x7f668c101af0 2024-04-21T23:16:49.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.801+0000 7f6688ff9700 1 -- 172.21.15.40:0/190955459 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6674004500 con 0x7f668c101af0 2024-04-21T23:16:49.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.801+0000 7f6691296700 1 -- 172.21.15.40:0/190955459 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f668c1afa20 con 0x7f668c101af0 2024-04-21T23:16:49.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.801+0000 7f6688ff9700 1 -- 172.21.15.40:0/190955459 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6674004d00 con 0x7f668c101af0 2024-04-21T23:16:49.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.801+0000 7f6691296700 1 -- 172.21.15.40:0/190955459 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f668c1aff60 con 0x7f668c101af0 2024-04-21T23:16:49.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.802+0000 7f6691296700 1 -- 172.21.15.40:0/190955459 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f668c0628c0 con 0x7f668c101af0 2024-04-21T23:16:49.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.803+0000 7f6688ff9700 1 -- 172.21.15.40:0/190955459 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f667400bbf0 con 0x7f668c101af0 2024-04-21T23:16:49.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.804+0000 7f6688ff9700 1 --2- 172.21.15.40:0/190955459 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f66780a2cb0 0x7f66780a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:49.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.804+0000 7f668affd700 1 --2- 172.21.15.40:0/190955459 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f66780a2cb0 0x7f66780a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:49.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.804+0000 7f6688ff9700 1 -- 172.21.15.40:0/190955459 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f66740cf940 con 0x7f668c101af0 2024-04-21T23:16:49.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.804+0000 7f6688ff9700 1 --2- 172.21.15.40:0/190955459 >> [v2:172.21.15.111:6816/927054162,v1:172.21.15.111:6817/927054162] conn(0x7f66780ad970 0x7f66780afdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:49.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.804+0000 7f6688ff9700 1 -- 172.21.15.40:0/190955459 --> [v2:172.21.15.111:6816/927054162,v1:172.21.15.111:6817/927054162] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6674015ec0 con 0x7f66780ad970 2024-04-21T23:16:49.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.804+0000 7f6688ff9700 1 -- 172.21.15.40:0/190955459 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f66740cfd90 con 0x7f668c101af0 2024-04-21T23:16:49.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.804+0000 7f668affd700 1 --2- 172.21.15.40:0/190955459 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f66780a2cb0 0x7f66780a5170 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f667c00b5c0 tx=0x7f667c005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:49.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.805+0000 7f668b7fe700 1 --2- 172.21.15.40:0/190955459 >> [v2:172.21.15.111:6816/927054162,v1:172.21.15.111:6817/927054162] conn(0x7f66780ad970 0x7f66780afdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:49.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.805+0000 7f668b7fe700 1 --2- 172.21.15.40:0/190955459 >> [v2:172.21.15.111:6816/927054162,v1:172.21.15.111:6817/927054162] conn(0x7f66780ad970 0x7f66780afdb0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:49.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.806+0000 7f6688ff9700 1 -- 172.21.15.40:0/190955459 <== osd.7 v2:172.21.15.111:6816/927054162 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6674015ec0 con 0x7f66780ad970 2024-04-21T23:16:49.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.823+0000 7f6691296700 1 -- 172.21.15.40:0/190955459 --> [v2:172.21.15.111:6816/927054162,v1:172.21.15.111:6817/927054162] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f668c062d40 con 0x7f66780ad970 2024-04-21T23:16:49.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.824+0000 7f6688ff9700 1 -- 172.21.15.40:0/190955459 <== osd.7 v2:172.21.15.111:6816/927054162 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f668c062d40 con 0x7f66780ad970 2024-04-21T23:16:49.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.824+0000 7f6691296700 1 -- 172.21.15.40:0/190955459 >> [v2:172.21.15.111:6816/927054162,v1:172.21.15.111:6817/927054162] conn(0x7f66780ad970 msgr2=0x7f66780afdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:49.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.824+0000 7f6691296700 1 --2- 172.21.15.40:0/190955459 >> [v2:172.21.15.111:6816/927054162,v1:172.21.15.111:6817/927054162] conn(0x7f66780ad970 0x7f66780afdb0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:49.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.824+0000 7f6691296700 1 -- 172.21.15.40:0/190955459 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f66780a2cb0 msgr2=0x7f66780a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:49.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.824+0000 7f6691296700 1 --2- 172.21.15.40:0/190955459 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f66780a2cb0 0x7f66780a5170 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f667c00b5c0 tx=0x7f667c005fd0 comp rx=0 tx=0).stop 2024-04-21T23:16:49.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.824+0000 7f6691296700 1 -- 172.21.15.40:0/190955459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f668c101af0 msgr2=0x7f668c069f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:49.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.824+0000 7f6691296700 1 --2- 172.21.15.40:0/190955459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f668c101af0 0x7f668c069f10 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f667400d980 tx=0x7f667400dd40 comp rx=0 tx=0).stop 2024-04-21T23:16:49.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.824+0000 7f668b7fe700 1 -- 172.21.15.40:0/190955459 reap_dead start 2024-04-21T23:16:49.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.825+0000 7f6691296700 1 -- 172.21.15.40:0/190955459 shutdown_connections 2024-04-21T23:16:49.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.825+0000 7f6691296700 1 -- 172.21.15.40:0/190955459 >> 172.21.15.40:0/190955459 conn(0x7f668c0780e0 msgr2=0x7f668c0ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:49.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.825+0000 7f6691296700 1 -- 172.21.15.40:0/190955459 shutdown_connections 2024-04-21T23:16:49.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:48.825+0000 7f6691296700 1 -- 172.21.15.40:0/190955459 wait complete. 2024-04-21T23:16:50.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:50 smithi040 ceph-mon[27984]: pgmap v319: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-21T23:16:50.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:50 smithi174 ceph-mon[31409]: pgmap v319: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-21T23:16:50.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:50 smithi111 ceph-mon[31085]: pgmap v319: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-21T23:16:51.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:51 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]: dispatch 2024-04-21T23:16:51.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:51 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]: dispatch 2024-04-21T23:16:51.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:51 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]: dispatch 2024-04-21T23:16:51.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:51 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 1]}]: dispatch 2024-04-21T23:16:51.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:51 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]: dispatch 2024-04-21T23:16:51.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:51 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 6]}]: dispatch 2024-04-21T23:16:51.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:51 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]: dispatch 2024-04-21T23:16:51.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:51 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9]}]: dispatch 2024-04-21T23:16:51.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:51 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2a", "id": [3, 8]}]: dispatch 2024-04-21T23:16:51.433 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:51 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]: dispatch 2024-04-21T23:16:51.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:51 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]: dispatch 2024-04-21T23:16:51.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:51 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]: dispatch 2024-04-21T23:16:51.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:51 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]: dispatch 2024-04-21T23:16:51.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:51 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 1]}]: dispatch 2024-04-21T23:16:51.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:51 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]: dispatch 2024-04-21T23:16:51.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:51 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 6]}]: dispatch 2024-04-21T23:16:51.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:51 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]: dispatch 2024-04-21T23:16:51.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:51 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9]}]: dispatch 2024-04-21T23:16:51.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:51 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2a", "id": [3, 8]}]: dispatch 2024-04-21T23:16:51.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:51 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]: dispatch 2024-04-21T23:16:51.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:51 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]: dispatch 2024-04-21T23:16:51.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:51 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]: dispatch 2024-04-21T23:16:51.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:51 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]: dispatch 2024-04-21T23:16:51.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:51 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 1]}]: dispatch 2024-04-21T23:16:51.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:51 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]: dispatch 2024-04-21T23:16:51.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:51 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 6]}]: dispatch 2024-04-21T23:16:51.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:51 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]: dispatch 2024-04-21T23:16:51.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:51 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9]}]: dispatch 2024-04-21T23:16:51.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:51 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2a", "id": [3, 8]}]: dispatch 2024-04-21T23:16:51.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:51 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]: dispatch 2024-04-21T23:16:51.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:51.850+0000 7f7ff8606700 1 -- 172.21.15.40:0/2070388242 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ff006a8e0 msgr2=0x7f7ff010f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:51.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:51.850+0000 7f7ff8606700 1 --2- 172.21.15.40:0/2070388242 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ff006a8e0 0x7f7ff010f0f0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f7fe8009990 tx=0x7f7fe8009ca0 comp rx=0 tx=0).stop 2024-04-21T23:16:51.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:51.850+0000 7f7ff8606700 1 -- 172.21.15.40:0/2070388242 shutdown_connections 2024-04-21T23:16:51.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:51.850+0000 7f7ff8606700 1 --2- 172.21.15.40:0/2070388242 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7ff01082b0 0x7f7ff0108690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:51.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:51.850+0000 7f7ff8606700 1 --2- 172.21.15.40:0/2070388242 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ff006a8e0 0x7f7ff010f0f0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:51.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:51.850+0000 7f7ff8606700 1 --2- 172.21.15.40:0/2070388242 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ff0069f40 0x7f7ff006a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:51.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:51.850+0000 7f7ff8606700 1 -- 172.21.15.40:0/2070388242 >> 172.21.15.40:0/2070388242 conn(0x7f7ff00780e0 msgr2=0x7f7ff00784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:51.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:51.850+0000 7f7ff8606700 1 -- 172.21.15.40:0/2070388242 shutdown_connections 2024-04-21T23:16:51.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:51.850+0000 7f7ff8606700 1 -- 172.21.15.40:0/2070388242 wait complete. 2024-04-21T23:16:51.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:51.851+0000 7f7ff8606700 1 Processor -- start 2024-04-21T23:16:51.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:51.851+0000 7f7ff8606700 1 -- start start 2024-04-21T23:16:51.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:51.851+0000 7f7ff8606700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ff0069f40 0x7f7ff019e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:51.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:51.851+0000 7f7ff8606700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ff006a8e0 0x7f7ff019ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:51.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:51.851+0000 7f7ff8606700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7ff01082b0 0x7f7ff01987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:51.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:51.851+0000 7f7ff8606700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7ff0112570 con 0x7f7ff006a8e0 2024-04-21T23:16:51.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:51.851+0000 7f7ff8606700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f7ff01123f0 con 0x7f7ff0069f40 2024-04-21T23:16:51.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:51.851+0000 7f7ff8606700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7ff01126f0 con 0x7f7ff01082b0 2024-04-21T23:16:51.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:51.851+0000 7f7ff5ba1700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ff006a8e0 0x7f7ff019ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:51.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:51.851+0000 7f7ff5ba1700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ff006a8e0 0x7f7ff019ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57672/0 (socket says 172.21.15.40:57672) 2024-04-21T23:16:51.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:51.851+0000 7f7ff5ba1700 1 -- 172.21.15.40:0/2478212842 learned_addr learned my addr 172.21.15.40:0/2478212842 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:16:51.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:51.852+0000 7f7ff63a2700 1 --2- 172.21.15.40:0/2478212842 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ff0069f40 0x7f7ff019e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:51.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:51.852+0000 7f7ff6ba3700 1 --2- 172.21.15.40:0/2478212842 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7ff01082b0 0x7f7ff01987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:51.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:51.852+0000 7f7ff5ba1700 1 -- 172.21.15.40:0/2478212842 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7ff01082b0 msgr2=0x7f7ff01987f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:51.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:51.852+0000 7f7ff5ba1700 1 --2- 172.21.15.40:0/2478212842 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7ff01082b0 0x7f7ff01987f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:51.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:51.852+0000 7f7ff5ba1700 1 -- 172.21.15.40:0/2478212842 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ff0069f40 msgr2=0x7f7ff019e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:51.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:51.852+0000 7f7ff5ba1700 1 --2- 172.21.15.40:0/2478212842 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ff0069f40 0x7f7ff019e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:51.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:51.852+0000 7f7ff5ba1700 1 -- 172.21.15.40:0/2478212842 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ff0198fd0 con 0x7f7ff006a8e0 2024-04-21T23:16:51.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:51.852+0000 7f7ff5ba1700 1 --2- 172.21.15.40:0/2478212842 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ff006a8e0 0x7f7ff019ebb0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f7fe400c920 tx=0x7f7fe400cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:51.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:51.852+0000 7f7fef7fe700 1 -- 172.21.15.40:0/2478212842 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fe4007b00 con 0x7f7ff006a8e0 2024-04-21T23:16:51.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:51.852+0000 7f7fef7fe700 1 -- 172.21.15.40:0/2478212842 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7fe4004d10 con 0x7f7ff006a8e0 2024-04-21T23:16:51.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:51.852+0000 7f7ff8606700 1 -- 172.21.15.40:0/2478212842 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ff01992c0 con 0x7f7ff006a8e0 2024-04-21T23:16:51.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:51.852+0000 7f7fef7fe700 1 -- 172.21.15.40:0/2478212842 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fe4005680 con 0x7f7ff006a8e0 2024-04-21T23:16:51.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:51.852+0000 7f7ff8606700 1 -- 172.21.15.40:0/2478212842 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ff01a54d0 con 0x7f7ff006a8e0 2024-04-21T23:16:51.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:51.853+0000 7f7ff8606700 1 -- 172.21.15.40:0/2478212842 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7ff00629c0 con 0x7f7ff006a8e0 2024-04-21T23:16:51.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:51.855+0000 7f7fef7fe700 1 -- 172.21.15.40:0/2478212842 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f7fe4004750 con 0x7f7ff006a8e0 2024-04-21T23:16:51.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:51.855+0000 7f7fef7fe700 1 --2- 172.21.15.40:0/2478212842 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f7fd80a2cb0 0x7f7fd80a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:51.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:51.856+0000 7f7ff63a2700 1 --2- 172.21.15.40:0/2478212842 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f7fd80a2cb0 0x7f7fd80a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:51.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:51.856+0000 7f7fef7fe700 1 -- 172.21.15.40:0/2478212842 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f7fe40cf4b0 con 0x7f7ff006a8e0 2024-04-21T23:16:51.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:51.856+0000 7f7fef7fe700 1 --2- 172.21.15.40:0/2478212842 >> [v2:172.21.15.111:6824/1227538935,v1:172.21.15.111:6825/1227538935] conn(0x7f7fd80ade10 0x7f7fd80b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:51.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:51.856+0000 7f7fef7fe700 1 -- 172.21.15.40:0/2478212842 --> [v2:172.21.15.111:6824/1227538935,v1:172.21.15.111:6825/1227538935] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7fd80b0980 con 0x7f7fd80ade10 2024-04-21T23:16:51.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:51.856+0000 7f7fef7fe700 1 -- 172.21.15.40:0/2478212842 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f7fe40cf900 con 0x7f7ff006a8e0 2024-04-21T23:16:51.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:51.856+0000 7f7ff63a2700 1 --2- 172.21.15.40:0/2478212842 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f7fd80a2cb0 0x7f7fd80a5170 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f7fe000a400 tx=0x7f7fe000d040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:51.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:51.856+0000 7f7ff6ba3700 1 --2- 172.21.15.40:0/2478212842 >> [v2:172.21.15.111:6824/1227538935,v1:172.21.15.111:6825/1227538935] conn(0x7f7fd80ade10 0x7f7fd80b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:51.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:51.857+0000 7f7ff6ba3700 1 --2- 172.21.15.40:0/2478212842 >> [v2:172.21.15.111:6824/1227538935,v1:172.21.15.111:6825/1227538935] conn(0x7f7fd80ade10 0x7f7fd80b02b0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:51.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:51.858+0000 7f7fef7fe700 1 -- 172.21.15.40:0/2478212842 <== osd.10 v2:172.21.15.111:6824/1227538935 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7fd80b0980 con 0x7f7fd80ade10 2024-04-21T23:16:51.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:51.874+0000 7f7ff8606700 1 -- 172.21.15.40:0/2478212842 --> [v2:172.21.15.111:6824/1227538935,v1:172.21.15.111:6825/1227538935] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7ff0062e40 con 0x7f7fd80ade10 2024-04-21T23:16:51.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:51.875+0000 7f7fef7fe700 1 -- 172.21.15.40:0/2478212842 <== osd.10 v2:172.21.15.111:6824/1227538935 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f7ff0062e40 con 0x7f7fd80ade10 2024-04-21T23:16:51.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:51.875+0000 7f7ff8606700 1 -- 172.21.15.40:0/2478212842 >> [v2:172.21.15.111:6824/1227538935,v1:172.21.15.111:6825/1227538935] conn(0x7f7fd80ade10 msgr2=0x7f7fd80b02b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:51.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:51.875+0000 7f7ff8606700 1 --2- 172.21.15.40:0/2478212842 >> [v2:172.21.15.111:6824/1227538935,v1:172.21.15.111:6825/1227538935] conn(0x7f7fd80ade10 0x7f7fd80b02b0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:51.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:51.875+0000 7f7ff8606700 1 -- 172.21.15.40:0/2478212842 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f7fd80a2cb0 msgr2=0x7f7fd80a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:51.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:51.875+0000 7f7ff8606700 1 --2- 172.21.15.40:0/2478212842 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f7fd80a2cb0 0x7f7fd80a5170 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f7fe000a400 tx=0x7f7fe000d040 comp rx=0 tx=0).stop 2024-04-21T23:16:51.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:51.875+0000 7f7ff8606700 1 -- 172.21.15.40:0/2478212842 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ff006a8e0 msgr2=0x7f7ff019ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:51.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:51.875+0000 7f7ff8606700 1 --2- 172.21.15.40:0/2478212842 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ff006a8e0 0x7f7ff019ebb0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f7fe400c920 tx=0x7f7fe400cc30 comp rx=0 tx=0).stop 2024-04-21T23:16:51.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:51.875+0000 7f7ff6ba3700 1 -- 172.21.15.40:0/2478212842 reap_dead start 2024-04-21T23:16:51.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:51.875+0000 7f7ff8606700 1 -- 172.21.15.40:0/2478212842 shutdown_connections 2024-04-21T23:16:51.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:51.875+0000 7f7ff8606700 1 -- 172.21.15.40:0/2478212842 >> 172.21.15.40:0/2478212842 conn(0x7f7ff00780e0 msgr2=0x7f7ff00ffd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:51.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:51.875+0000 7f7ff8606700 1 -- 172.21.15.40:0/2478212842 shutdown_connections 2024-04-21T23:16:51.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:51.875+0000 7f7ff8606700 1 -- 172.21.15.40:0/2478212842 wait complete. 2024-04-21T23:16:52.406 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:52 smithi040 ceph-mon[27984]: pgmap v320: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 7 op/s 2024-04-21T23:16:52.406 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:52 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]': finished 2024-04-21T23:16:52.407 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:52 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]': finished 2024-04-21T23:16:52.407 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:52 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2024-04-21T23:16:52.407 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:52 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 1]}]': finished 2024-04-21T23:16:52.407 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:52 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]': finished 2024-04-21T23:16:52.407 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:52 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 6]}]': finished 2024-04-21T23:16:52.407 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:52 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]': finished 2024-04-21T23:16:52.407 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:52 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9]}]': finished 2024-04-21T23:16:52.407 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:52 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2a", "id": [3, 8]}]': finished 2024-04-21T23:16:52.407 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:52 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]': finished 2024-04-21T23:16:52.407 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:52 smithi040 ceph-mon[27984]: osdmap e91: 12 total, 12 up, 12 in 2024-04-21T23:16:52.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:52 smithi174 ceph-mon[31409]: pgmap v320: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 7 op/s 2024-04-21T23:16:52.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:52 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]': finished 2024-04-21T23:16:52.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:52 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]': finished 2024-04-21T23:16:52.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:52 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2024-04-21T23:16:52.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:52 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 1]}]': finished 2024-04-21T23:16:52.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:52 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]': finished 2024-04-21T23:16:52.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:52 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 6]}]': finished 2024-04-21T23:16:52.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:52 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]': finished 2024-04-21T23:16:52.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:52 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9]}]': finished 2024-04-21T23:16:52.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:52 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2a", "id": [3, 8]}]': finished 2024-04-21T23:16:52.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:52 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]': finished 2024-04-21T23:16:52.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:52 smithi174 ceph-mon[31409]: osdmap e91: 12 total, 12 up, 12 in 2024-04-21T23:16:52.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:52 smithi111 ceph-mon[31085]: pgmap v320: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 7 op/s 2024-04-21T23:16:52.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:52 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]': finished 2024-04-21T23:16:52.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:52 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]': finished 2024-04-21T23:16:52.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:52 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2024-04-21T23:16:52.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:52 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 1]}]': finished 2024-04-21T23:16:52.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:52 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]': finished 2024-04-21T23:16:52.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:52 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 6]}]': finished 2024-04-21T23:16:52.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:52 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]': finished 2024-04-21T23:16:52.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:52 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9]}]': finished 2024-04-21T23:16:52.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:52 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2a", "id": [3, 8]}]': finished 2024-04-21T23:16:52.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:52 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]': finished 2024-04-21T23:16:52.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:52 smithi111 ceph-mon[31085]: osdmap e91: 12 total, 12 up, 12 in 2024-04-21T23:16:53.534 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:53 smithi040 ceph-mon[27984]: osdmap e92: 12 total, 12 up, 12 in 2024-04-21T23:16:53.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:53 smithi174 ceph-mon[31409]: osdmap e92: 12 total, 12 up, 12 in 2024-04-21T23:16:53.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:53 smithi111 ceph-mon[31085]: osdmap e92: 12 total, 12 up, 12 in 2024-04-21T23:16:54.369 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:54 smithi040 ceph-mon[27984]: pgmap v323: 129 pgs: 2 peering, 127 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-21T23:16:54.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:54.392+0000 7f048c9db700 1 -- 172.21.15.40:0/3131892519 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04881013c0 msgr2=0x7f04881017a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:54.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:54.392+0000 7f048c9db700 1 --2- 172.21.15.40:0/3131892519 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04881013c0 0x7f04881017a0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f04780099c0 tx=0x7f0478009cd0 comp rx=0 tx=0).stop 2024-04-21T23:16:54.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:54.392+0000 7f048c9db700 1 -- 172.21.15.40:0/3131892519 shutdown_connections 2024-04-21T23:16:54.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:54.392+0000 7f048c9db700 1 --2- 172.21.15.40:0/3131892519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0488106450 0x7f048810e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:54.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:54.392+0000 7f048c9db700 1 --2- 172.21.15.40:0/3131892519 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04881013c0 0x7f04881017a0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:54.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:54.392+0000 7f048c9db700 1 --2- 172.21.15.40:0/3131892519 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0488101d70 0x7f0488105d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:54.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:54.392+0000 7f048c9db700 1 -- 172.21.15.40:0/3131892519 >> 172.21.15.40:0/3131892519 conn(0x7f04880780e0 msgr2=0x7f04880784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:54.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:54.392+0000 7f048c9db700 1 -- 172.21.15.40:0/3131892519 shutdown_connections 2024-04-21T23:16:54.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:54.392+0000 7f048c9db700 1 -- 172.21.15.40:0/3131892519 wait complete. 2024-04-21T23:16:54.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:54.393+0000 7f048c9db700 1 Processor -- start 2024-04-21T23:16:54.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:54.393+0000 7f048c9db700 1 -- start start 2024-04-21T23:16:54.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:54.393+0000 7f048c9db700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04881013c0 0x7f048819e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:54.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:54.393+0000 7f048c9db700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0488101d70 0x7f048819eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:54.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:54.393+0000 7f048c9db700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0488106450 0x7f04881987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:54.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:54.393+0000 7f048c9db700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0488111430 con 0x7f04881013c0 2024-04-21T23:16:54.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:54.393+0000 7f048c9db700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f04881112b0 con 0x7f0488101d70 2024-04-21T23:16:54.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:54.393+0000 7f048c9db700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f04881115b0 con 0x7f0488106450 2024-04-21T23:16:54.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:54.393+0000 7f048659c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04881013c0 0x7f048819e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:54.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:54.393+0000 7f048659c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04881013c0 0x7f048819e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57678/0 (socket says 172.21.15.40:57678) 2024-04-21T23:16:54.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:54.393+0000 7f048659c700 1 -- 172.21.15.40:0/3457388488 learned_addr learned my addr 172.21.15.40:0/3457388488 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:16:54.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:54.393+0000 7f0485d9b700 1 --2- 172.21.15.40:0/3457388488 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0488101d70 0x7f048819eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:54.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:54.393+0000 7f0486d9d700 1 --2- 172.21.15.40:0/3457388488 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0488106450 0x7f04881987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:54.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:54.393+0000 7f048659c700 1 -- 172.21.15.40:0/3457388488 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0488106450 msgr2=0x7f04881987b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:54.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:54.393+0000 7f048659c700 1 --2- 172.21.15.40:0/3457388488 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0488106450 0x7f04881987b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:54.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:54.393+0000 7f048659c700 1 -- 172.21.15.40:0/3457388488 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0488101d70 msgr2=0x7f048819eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:54.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:54.393+0000 7f048659c700 1 --2- 172.21.15.40:0/3457388488 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0488101d70 0x7f048819eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:54.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:54.393+0000 7f048659c700 1 -- 172.21.15.40:0/3457388488 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0488198f90 con 0x7f04881013c0 2024-04-21T23:16:54.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:54.394+0000 7f048659c700 1 --2- 172.21.15.40:0/3457388488 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04881013c0 0x7f048819e630 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f04780059d0 tx=0x7f04780038b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:54.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:54.394+0000 7f04777fe700 1 -- 172.21.15.40:0/3457388488 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0478027070 con 0x7f04881013c0 2024-04-21T23:16:54.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:54.394+0000 7f04777fe700 1 -- 172.21.15.40:0/3457388488 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0478003e40 con 0x7f04881013c0 2024-04-21T23:16:54.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:54.394+0000 7f048c9db700 1 -- 172.21.15.40:0/3457388488 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0488199220 con 0x7f04881013c0 2024-04-21T23:16:54.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:54.394+0000 7f04777fe700 1 -- 172.21.15.40:0/3457388488 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0478022d00 con 0x7f04881013c0 2024-04-21T23:16:54.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:54.394+0000 7f048c9db700 1 -- 172.21.15.40:0/3457388488 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04881a5490 con 0x7f04881013c0 2024-04-21T23:16:54.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:54.395+0000 7f048c9db700 1 -- 172.21.15.40:0/3457388488 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f04880629c0 con 0x7f04881013c0 2024-04-21T23:16:54.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:54.396+0000 7f04777fe700 1 -- 172.21.15.40:0/3457388488 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f047802a070 con 0x7f04881013c0 2024-04-21T23:16:54.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:54.397+0000 7f04777fe700 1 --2- 172.21.15.40:0/3457388488 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f04700a2d30 0x7f04700a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:54.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:54.397+0000 7f0485d9b700 1 --2- 172.21.15.40:0/3457388488 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f04700a2d30 0x7f04700a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:54.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:54.397+0000 7f04777fe700 1 -- 172.21.15.40:0/3457388488 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f04780d0e60 con 0x7f04881013c0 2024-04-21T23:16:54.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:54.397+0000 7f04777fe700 1 --2- 172.21.15.40:0/3457388488 >> [v2:172.21.15.174:6816/3817848746,v1:172.21.15.174:6817/3817848746] conn(0x7f04700ade90 0x7f04700b0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:54.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:54.397+0000 7f04777fe700 1 -- 172.21.15.40:0/3457388488 --> [v2:172.21.15.174:6816/3817848746,v1:172.21.15.174:6817/3817848746] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f04780095d0 con 0x7f04700ade90 2024-04-21T23:16:54.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:54.397+0000 7f04777fe700 1 -- 172.21.15.40:0/3457388488 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7f04780d12b0 con 0x7f04881013c0 2024-04-21T23:16:54.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:54.397+0000 7f0485d9b700 1 --2- 172.21.15.40:0/3457388488 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f04700a2d30 0x7f04700a51f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f04880ffae0 tx=0x7f047c00a380 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:54.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:54.397+0000 7f0486d9d700 1 --2- 172.21.15.40:0/3457388488 >> [v2:172.21.15.174:6816/3817848746,v1:172.21.15.174:6817/3817848746] conn(0x7f04700ade90 0x7f04700b0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:54.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:54.398+0000 7f0486d9d700 1 --2- 172.21.15.40:0/3457388488 >> [v2:172.21.15.174:6816/3817848746,v1:172.21.15.174:6817/3817848746] conn(0x7f04700ade90 0x7f04700b0330 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:54.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:54.399+0000 7f04777fe700 1 -- 172.21.15.40:0/3457388488 <== osd.8 v2:172.21.15.174:6816/3817848746 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f04780095d0 con 0x7f04700ade90 2024-04-21T23:16:54.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:54.415+0000 7f048c9db700 1 -- 172.21.15.40:0/3457388488 --> [v2:172.21.15.174:6816/3817848746,v1:172.21.15.174:6817/3817848746] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0488062dc0 con 0x7f04700ade90 2024-04-21T23:16:54.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:54.416+0000 7f04777fe700 1 -- 172.21.15.40:0/3457388488 <== osd.8 v2:172.21.15.174:6816/3817848746 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f0488062dc0 con 0x7f04700ade90 2024-04-21T23:16:54.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:54.417+0000 7f048c9db700 1 -- 172.21.15.40:0/3457388488 >> [v2:172.21.15.174:6816/3817848746,v1:172.21.15.174:6817/3817848746] conn(0x7f04700ade90 msgr2=0x7f04700b0330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:54.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:54.417+0000 7f048c9db700 1 --2- 172.21.15.40:0/3457388488 >> [v2:172.21.15.174:6816/3817848746,v1:172.21.15.174:6817/3817848746] conn(0x7f04700ade90 0x7f04700b0330 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:54.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:54.417+0000 7f048c9db700 1 -- 172.21.15.40:0/3457388488 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f04700a2d30 msgr2=0x7f04700a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:54.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:54.417+0000 7f048c9db700 1 --2- 172.21.15.40:0/3457388488 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f04700a2d30 0x7f04700a51f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f04880ffae0 tx=0x7f047c00a380 comp rx=0 tx=0).stop 2024-04-21T23:16:54.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:54.417+0000 7f048c9db700 1 -- 172.21.15.40:0/3457388488 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04881013c0 msgr2=0x7f048819e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:54.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:54.417+0000 7f048c9db700 1 --2- 172.21.15.40:0/3457388488 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04881013c0 0x7f048819e630 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f04780059d0 tx=0x7f04780038b0 comp rx=0 tx=0).stop 2024-04-21T23:16:54.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:54.417+0000 7f0486d9d700 1 -- 172.21.15.40:0/3457388488 reap_dead start 2024-04-21T23:16:54.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:54.417+0000 7f048c9db700 1 -- 172.21.15.40:0/3457388488 shutdown_connections 2024-04-21T23:16:54.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:54.417+0000 7f048c9db700 1 -- 172.21.15.40:0/3457388488 >> 172.21.15.40:0/3457388488 conn(0x7f04880780e0 msgr2=0x7f0488105290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:54.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:54.417+0000 7f048c9db700 1 -- 172.21.15.40:0/3457388488 shutdown_connections 2024-04-21T23:16:54.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:54.417+0000 7f048c9db700 1 -- 172.21.15.40:0/3457388488 wait complete. 2024-04-21T23:16:54.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:54 smithi174 ceph-mon[31409]: pgmap v323: 129 pgs: 2 peering, 127 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-21T23:16:54.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:54 smithi111 ceph-mon[31085]: pgmap v323: 129 pgs: 2 peering, 127 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-21T23:16:54.830 INFO:teuthology.orchestra.run.smithi040.stdout:347892350999 2024-04-21T23:16:54.831 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-04-21T23:16:55.429 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:55 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:16:55.572 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:55 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:16:55.615 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:55 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:16:55.671 INFO:teuthology.orchestra.run.smithi040.stdout:219043332140 2024-04-21T23:16:55.672 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-21T23:16:56.477 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:56 smithi040 ceph-mon[27984]: pgmap v324: 129 pgs: 2 peering, 127 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-04-21T23:16:56.479 INFO:teuthology.orchestra.run.smithi040.stdout:313532612638 2024-04-21T23:16:56.479 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-04-21T23:16:56.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:56.671+0000 7f78c97e6700 1 -- 172.21.15.40:0/2165750355 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78c41011c0 msgr2=0x7f78c410f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:56.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:56.671+0000 7f78c97e6700 1 --2- 172.21.15.40:0/2165750355 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78c41011c0 0x7f78c410f100 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f78bc00b0a0 tx=0x7f78bc00b3b0 comp rx=0 tx=0).stop 2024-04-21T23:16:56.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:56.671+0000 7f78c97e6700 1 -- 172.21.15.40:0/2165750355 shutdown_connections 2024-04-21T23:16:56.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:56.671+0000 7f78c97e6700 1 --2- 172.21.15.40:0/2165750355 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78c41011c0 0x7f78c410f100 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:56.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:56.671+0000 7f78c97e6700 1 --2- 172.21.15.40:0/2165750355 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78c41082c0 0x7f78c41086a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:56.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:56.671+0000 7f78c97e6700 1 --2- 172.21.15.40:0/2165750355 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f78c4100820 0x7f78c4100c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:56.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:56.671+0000 7f78c97e6700 1 -- 172.21.15.40:0/2165750355 >> 172.21.15.40:0/2165750355 conn(0x7f78c40777a0 msgr2=0x7f78c40feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:56.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:56.671+0000 7f78c97e6700 1 -- 172.21.15.40:0/2165750355 shutdown_connections 2024-04-21T23:16:56.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:56.671+0000 7f78c97e6700 1 -- 172.21.15.40:0/2165750355 wait complete. 2024-04-21T23:16:56.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:56.671+0000 7f78c97e6700 1 Processor -- start 2024-04-21T23:16:56.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:56.671+0000 7f78c97e6700 1 -- start start 2024-04-21T23:16:56.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:56.672+0000 7f78c97e6700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78c4100820 0x7f78c419e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:56.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:56.672+0000 7f78c97e6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78c41011c0 0x7f78c419ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:56.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:56.672+0000 7f78c97e6700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f78c41082c0 0x7f78c4198820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:56.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:56.672+0000 7f78c97e6700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f78c4112540 con 0x7f78c4100820 2024-04-21T23:16:56.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:56.672+0000 7f78c97e6700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f78c41123c0 con 0x7f78c41011c0 2024-04-21T23:16:56.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:56.672+0000 7f78c97e6700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f78c41126c0 con 0x7f78c41082c0 2024-04-21T23:16:56.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:56.672+0000 7f78c2ffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78c4100820 0x7f78c419e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:56.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:56.672+0000 7f78c2ffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78c4100820 0x7f78c419e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57702/0 (socket says 172.21.15.40:57702) 2024-04-21T23:16:56.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:56.672+0000 7f78c2ffd700 1 -- 172.21.15.40:0/2732827460 learned_addr learned my addr 172.21.15.40:0/2732827460 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:16:56.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:56.672+0000 7f78c27fc700 1 --2- 172.21.15.40:0/2732827460 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78c41011c0 0x7f78c419ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:56.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:56.672+0000 7f78c37fe700 1 --2- 172.21.15.40:0/2732827460 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f78c41082c0 0x7f78c4198820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:56.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:56.672+0000 7f78c2ffd700 1 -- 172.21.15.40:0/2732827460 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f78c41082c0 msgr2=0x7f78c4198820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:56.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:56.672+0000 7f78c2ffd700 1 --2- 172.21.15.40:0/2732827460 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f78c41082c0 0x7f78c4198820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:56.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:56.672+0000 7f78c2ffd700 1 -- 172.21.15.40:0/2732827460 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78c41011c0 msgr2=0x7f78c419ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:56.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:56.672+0000 7f78c2ffd700 1 --2- 172.21.15.40:0/2732827460 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78c41011c0 0x7f78c419ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:56.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:56.672+0000 7f78c2ffd700 1 -- 172.21.15.40:0/2732827460 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78c4199000 con 0x7f78c4100820 2024-04-21T23:16:56.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:56.672+0000 7f78c2ffd700 1 --2- 172.21.15.40:0/2732827460 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78c4100820 0x7f78c419e6a0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f78b400d980 tx=0x7f78b400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:56.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:56.672+0000 7f78abfff700 1 -- 172.21.15.40:0/2732827460 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78b4016070 con 0x7f78c4100820 2024-04-21T23:16:56.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:56.673+0000 7f78c97e6700 1 -- 172.21.15.40:0/2732827460 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78c41992f0 con 0x7f78c4100820 2024-04-21T23:16:56.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:56.673+0000 7f78abfff700 1 -- 172.21.15.40:0/2732827460 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78b4004d10 con 0x7f78c4100820 2024-04-21T23:16:56.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:56.673+0000 7f78abfff700 1 -- 172.21.15.40:0/2732827460 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78b4005670 con 0x7f78c4100820 2024-04-21T23:16:56.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:56.673+0000 7f78c97e6700 1 -- 172.21.15.40:0/2732827460 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78c41a5500 con 0x7f78c4100820 2024-04-21T23:16:56.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:56.673+0000 7f78c97e6700 1 -- 172.21.15.40:0/2732827460 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f78c4062940 con 0x7f78c4100820 2024-04-21T23:16:56.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:56.675+0000 7f78abfff700 1 -- 172.21.15.40:0/2732827460 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f78b4004750 con 0x7f78c4100820 2024-04-21T23:16:56.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:56.675+0000 7f78abfff700 1 --2- 172.21.15.40:0/2732827460 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f78ac0a2d30 0x7f78ac0a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:56.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:56.675+0000 7f78c27fc700 1 --2- 172.21.15.40:0/2732827460 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f78ac0a2d30 0x7f78ac0a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:56.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:56.675+0000 7f78abfff700 1 -- 172.21.15.40:0/2732827460 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f78b40cf890 con 0x7f78c4100820 2024-04-21T23:16:56.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:56.676+0000 7f78abfff700 1 --2- 172.21.15.40:0/2732827460 >> [v2:172.21.15.40:6802/2009758784,v1:172.21.15.40:6803/2009758784] conn(0x7f78ac0ade90 0x7f78ac0b0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:56.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:56.676+0000 7f78abfff700 1 -- 172.21.15.40:0/2732827460 --> [v2:172.21.15.40:6802/2009758784,v1:172.21.15.40:6803/2009758784] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f78b4015ec0 con 0x7f78ac0ade90 2024-04-21T23:16:56.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:56.676+0000 7f78abfff700 1 -- 172.21.15.40:0/2732827460 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7f78b40cfce0 con 0x7f78c4100820 2024-04-21T23:16:56.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:56.676+0000 7f78c27fc700 1 --2- 172.21.15.40:0/2732827460 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f78ac0a2d30 0x7f78ac0a51f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f78c41079f0 tx=0x7f78b8009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:56.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:56.676+0000 7f78c37fe700 1 --2- 172.21.15.40:0/2732827460 >> [v2:172.21.15.40:6802/2009758784,v1:172.21.15.40:6803/2009758784] conn(0x7f78ac0ade90 0x7f78ac0b0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:56.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:56.676+0000 7f78c37fe700 1 --2- 172.21.15.40:0/2732827460 >> [v2:172.21.15.40:6802/2009758784,v1:172.21.15.40:6803/2009758784] conn(0x7f78ac0ade90 0x7f78ac0b0330 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:56.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:56.677+0000 7f78abfff700 1 -- 172.21.15.40:0/2732827460 <== osd.0 v2:172.21.15.40:6802/2009758784 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f78b4015ec0 con 0x7f78ac0ade90 2024-04-21T23:16:56.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:56.693+0000 7f78c97e6700 1 -- 172.21.15.40:0/2732827460 --> [v2:172.21.15.40:6802/2009758784,v1:172.21.15.40:6803/2009758784] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f78c4062d40 con 0x7f78ac0ade90 2024-04-21T23:16:56.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:56.694+0000 7f78abfff700 1 -- 172.21.15.40:0/2732827460 <== osd.0 v2:172.21.15.40:6802/2009758784 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f78c4062d40 con 0x7f78ac0ade90 2024-04-21T23:16:56.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:56.694+0000 7f78c97e6700 1 -- 172.21.15.40:0/2732827460 >> [v2:172.21.15.40:6802/2009758784,v1:172.21.15.40:6803/2009758784] conn(0x7f78ac0ade90 msgr2=0x7f78ac0b0330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:56.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:56.694+0000 7f78c97e6700 1 --2- 172.21.15.40:0/2732827460 >> [v2:172.21.15.40:6802/2009758784,v1:172.21.15.40:6803/2009758784] conn(0x7f78ac0ade90 0x7f78ac0b0330 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:56.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:56.694+0000 7f78c97e6700 1 -- 172.21.15.40:0/2732827460 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f78ac0a2d30 msgr2=0x7f78ac0a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:56.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:56.694+0000 7f78c97e6700 1 --2- 172.21.15.40:0/2732827460 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f78ac0a2d30 0x7f78ac0a51f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f78c41079f0 tx=0x7f78b8009260 comp rx=0 tx=0).stop 2024-04-21T23:16:56.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:56.694+0000 7f78c97e6700 1 -- 172.21.15.40:0/2732827460 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78c4100820 msgr2=0x7f78c419e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:56.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:56.694+0000 7f78c97e6700 1 --2- 172.21.15.40:0/2732827460 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78c4100820 0x7f78c419e6a0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f78b400d980 tx=0x7f78b400dc90 comp rx=0 tx=0).stop 2024-04-21T23:16:56.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:56.694+0000 7f78c37fe700 1 -- 172.21.15.40:0/2732827460 reap_dead start 2024-04-21T23:16:56.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:56.694+0000 7f78c97e6700 1 -- 172.21.15.40:0/2732827460 shutdown_connections 2024-04-21T23:16:56.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:56.694+0000 7f78c97e6700 1 -- 172.21.15.40:0/2732827460 >> 172.21.15.40:0/2732827460 conn(0x7f78c40777a0 msgr2=0x7f78c40fee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:56.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:56.694+0000 7f78c97e6700 1 -- 172.21.15.40:0/2732827460 shutdown_connections 2024-04-21T23:16:56.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:56.694+0000 7f78c97e6700 1 -- 172.21.15.40:0/2732827460 wait complete. 2024-04-21T23:16:56.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:56 smithi111 ceph-mon[31085]: pgmap v324: 129 pgs: 2 peering, 127 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-04-21T23:16:56.695 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:56 smithi174 ceph-mon[31409]: pgmap v324: 129 pgs: 2 peering, 127 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-04-21T23:16:57.734 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:57 smithi040 ceph-mon[27984]: pgmap v325: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s; 13 B/s, 1 keys/s, 1 objects/s recovering 2024-04-21T23:16:57.735 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:57 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:16:57.735 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:57 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:16:57.735 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:57 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:16:57.735 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:57 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:16:57.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:57 smithi174 ceph-mon[31409]: pgmap v325: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s; 13 B/s, 1 keys/s, 1 objects/s recovering 2024-04-21T23:16:57.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:57 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:16:57.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:57 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:16:57.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:57 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:16:57.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:57 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:16:57.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:57 smithi111 ceph-mon[31085]: pgmap v325: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s; 13 B/s, 1 keys/s, 1 objects/s recovering 2024-04-21T23:16:57.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:57 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:16:57.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:57 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:16:57.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:57 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:16:57.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:57 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:16:58.386 INFO:teuthology.orchestra.run.smithi040.stdout:34359738449 2024-04-21T23:16:58.387 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-21T23:16:58.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:58.905+0000 7ff0df88c700 1 -- 172.21.15.40:0/2699965532 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0d8107190 msgr2=0x7ff0d8107570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:58.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:58.905+0000 7ff0df88c700 1 --2- 172.21.15.40:0/2699965532 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0d8107190 0x7ff0d8107570 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7ff0d4009990 tx=0x7ff0d4009ca0 comp rx=0 tx=0).stop 2024-04-21T23:16:58.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:58.905+0000 7ff0df88c700 1 -- 172.21.15.40:0/2699965532 shutdown_connections 2024-04-21T23:16:58.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:58.905+0000 7ff0df88c700 1 --2- 172.21.15.40:0/2699965532 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff0d8101ae0 0x7ff0d810dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:58.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:58.905+0000 7ff0df88c700 1 --2- 172.21.15.40:0/2699965532 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0d8107190 0x7ff0d8107570 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:58.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:58.905+0000 7ff0df88c700 1 --2- 172.21.15.40:0/2699965532 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0d8101120 0x7ff0d81015a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:58.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:58.905+0000 7ff0df88c700 1 -- 172.21.15.40:0/2699965532 >> 172.21.15.40:0/2699965532 conn(0x7ff0d80780a0 msgr2=0x7ff0d80784b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:58.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:58.905+0000 7ff0df88c700 1 -- 172.21.15.40:0/2699965532 shutdown_connections 2024-04-21T23:16:58.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:58.905+0000 7ff0df88c700 1 -- 172.21.15.40:0/2699965532 wait complete. 2024-04-21T23:16:58.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:58.905+0000 7ff0df88c700 1 Processor -- start 2024-04-21T23:16:58.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:58.905+0000 7ff0df88c700 1 -- start start 2024-04-21T23:16:58.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:58.906+0000 7ff0df88c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0d8101120 0x7ff0d8102e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:58.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:58.906+0000 7ff0df88c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0d8101ae0 0x7ff0d8103340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:58.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:58.906+0000 7ff0df88c700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff0d8107190 0x7ff0d8103880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:58.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:58.906+0000 7ff0df88c700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff0d8111400 con 0x7ff0d8101ae0 2024-04-21T23:16:58.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:58.906+0000 7ff0df88c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff0d8111280 con 0x7ff0d8101120 2024-04-21T23:16:58.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:58.906+0000 7ff0df88c700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff0d8111580 con 0x7ff0d8107190 2024-04-21T23:16:58.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:58.906+0000 7ff0dd628700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0d8101120 0x7ff0d8102e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:58.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:58.906+0000 7ff0dd628700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0d8101120 0x7ff0d8102e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.40:35968/0 (socket says 172.21.15.40:35968) 2024-04-21T23:16:58.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:58.906+0000 7ff0dd628700 1 -- 172.21.15.40:0/3125723297 learned_addr learned my addr 172.21.15.40:0/3125723297 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:16:58.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:58.906+0000 7ff0dce27700 1 --2- 172.21.15.40:0/3125723297 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0d8101ae0 0x7ff0d8103340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:58.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:58.906+0000 7ff0dde29700 1 --2- 172.21.15.40:0/3125723297 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff0d8107190 0x7ff0d8103880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:58.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:58.906+0000 7ff0dd628700 1 -- 172.21.15.40:0/3125723297 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff0d8107190 msgr2=0x7ff0d8103880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:58.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:58.906+0000 7ff0dd628700 1 --2- 172.21.15.40:0/3125723297 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff0d8107190 0x7ff0d8103880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:58.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:58.906+0000 7ff0dd628700 1 -- 172.21.15.40:0/3125723297 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0d8101ae0 msgr2=0x7ff0d8103340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:58.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:58.906+0000 7ff0dd628700 1 --2- 172.21.15.40:0/3125723297 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0d8101ae0 0x7ff0d8103340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:58.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:58.906+0000 7ff0dd628700 1 -- 172.21.15.40:0/3125723297 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0d81a6f00 con 0x7ff0d8101120 2024-04-21T23:16:58.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:58.906+0000 7ff0dce27700 1 --2- 172.21.15.40:0/3125723297 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0d8101ae0 0x7ff0d8103340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:16:58.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:58.907+0000 7ff0dd628700 1 --2- 172.21.15.40:0/3125723297 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0d8101120 0x7ff0d8102e00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff0d400b5c0 tx=0x7ff0d40047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:58.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:58.907+0000 7ff0ce7fc700 1 -- 172.21.15.40:0/3125723297 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0d4027070 con 0x7ff0d8101120 2024-04-21T23:16:58.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:58.907+0000 7ff0ce7fc700 1 -- 172.21.15.40:0/3125723297 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff0d4004030 con 0x7ff0d8101120 2024-04-21T23:16:58.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:58.907+0000 7ff0df88c700 1 -- 172.21.15.40:0/3125723297 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0d81a7190 con 0x7ff0d8101120 2024-04-21T23:16:58.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:58.907+0000 7ff0ce7fc700 1 -- 172.21.15.40:0/3125723297 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0d4022db0 con 0x7ff0d8101120 2024-04-21T23:16:58.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:58.907+0000 7ff0df88c700 1 -- 172.21.15.40:0/3125723297 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0d81a75e0 con 0x7ff0d8101120 2024-04-21T23:16:58.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:58.908+0000 7ff0df88c700 1 -- 172.21.15.40:0/3125723297 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff0d81051f0 con 0x7ff0d8101120 2024-04-21T23:16:58.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:58.909+0000 7ff0ce7fc700 1 -- 172.21.15.40:0/3125723297 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7ff0d402a070 con 0x7ff0d8101120 2024-04-21T23:16:58.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:58.910+0000 7ff0ce7fc700 1 --2- 172.21.15.40:0/3125723297 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7ff0c40a2cb0 0x7ff0c40a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:58.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:58.910+0000 7ff0dce27700 1 --2- 172.21.15.40:0/3125723297 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7ff0c40a2cb0 0x7ff0c40a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:58.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:58.910+0000 7ff0ce7fc700 1 -- 172.21.15.40:0/3125723297 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7ff0d40d0ca0 con 0x7ff0d8101120 2024-04-21T23:16:58.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:58.910+0000 7ff0ce7fc700 1 --2- 172.21.15.40:0/3125723297 >> [v2:172.21.15.111:6800/1763903836,v1:172.21.15.111:6801/1763903836] conn(0x7ff0c40ade10 0x7ff0c40b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:58.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:58.910+0000 7ff0ce7fc700 1 -- 172.21.15.40:0/3125723297 --> [v2:172.21.15.111:6800/1763903836,v1:172.21.15.111:6801/1763903836] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff0d4024ec0 con 0x7ff0c40ade10 2024-04-21T23:16:58.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:58.910+0000 7ff0ce7fc700 1 -- 172.21.15.40:0/3125723297 <== mon.2 v2:172.21.15.111:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7ff0d40d1050 con 0x7ff0d8101120 2024-04-21T23:16:58.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:58.911+0000 7ff0dde29700 1 --2- 172.21.15.40:0/3125723297 >> [v2:172.21.15.111:6800/1763903836,v1:172.21.15.111:6801/1763903836] conn(0x7ff0c40ade10 0x7ff0c40b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:58.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:58.911+0000 7ff0dce27700 1 --2- 172.21.15.40:0/3125723297 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7ff0c40a2cb0 0x7ff0c40a5170 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7ff0d8078680 tx=0x7ff0c800a300 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:58.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:58.911+0000 7ff0dde29700 1 --2- 172.21.15.40:0/3125723297 >> [v2:172.21.15.111:6800/1763903836,v1:172.21.15.111:6801/1763903836] conn(0x7ff0c40ade10 0x7ff0c40b02b0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:58.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:58.912+0000 7ff0ce7fc700 1 -- 172.21.15.40:0/3125723297 <== osd.1 v2:172.21.15.111:6800/1763903836 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff0d4024ec0 con 0x7ff0c40ade10 2024-04-21T23:16:58.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:58.928+0000 7ff0df88c700 1 -- 172.21.15.40:0/3125723297 --> [v2:172.21.15.111:6800/1763903836,v1:172.21.15.111:6801/1763903836] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff0d8102840 con 0x7ff0c40ade10 2024-04-21T23:16:58.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:58.929+0000 7ff0ce7fc700 1 -- 172.21.15.40:0/3125723297 <== osd.1 v2:172.21.15.111:6800/1763903836 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7ff0d8102840 con 0x7ff0c40ade10 2024-04-21T23:16:58.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:58.929+0000 7ff0df88c700 1 -- 172.21.15.40:0/3125723297 >> [v2:172.21.15.111:6800/1763903836,v1:172.21.15.111:6801/1763903836] conn(0x7ff0c40ade10 msgr2=0x7ff0c40b02b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:58.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:58.929+0000 7ff0df88c700 1 --2- 172.21.15.40:0/3125723297 >> [v2:172.21.15.111:6800/1763903836,v1:172.21.15.111:6801/1763903836] conn(0x7ff0c40ade10 0x7ff0c40b02b0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:58.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:58.929+0000 7ff0df88c700 1 -- 172.21.15.40:0/3125723297 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7ff0c40a2cb0 msgr2=0x7ff0c40a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:58.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:58.929+0000 7ff0df88c700 1 --2- 172.21.15.40:0/3125723297 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7ff0c40a2cb0 0x7ff0c40a5170 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7ff0d8078680 tx=0x7ff0c800a300 comp rx=0 tx=0).stop 2024-04-21T23:16:58.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:58.930+0000 7ff0df88c700 1 -- 172.21.15.40:0/3125723297 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0d8101120 msgr2=0x7ff0d8102e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:58.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:58.930+0000 7ff0df88c700 1 --2- 172.21.15.40:0/3125723297 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0d8101120 0x7ff0d8102e00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff0d400b5c0 tx=0x7ff0d40047b0 comp rx=0 tx=0).stop 2024-04-21T23:16:58.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:58.930+0000 7ff0dde29700 1 -- 172.21.15.40:0/3125723297 reap_dead start 2024-04-21T23:16:58.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:58.930+0000 7ff0df88c700 1 -- 172.21.15.40:0/3125723297 shutdown_connections 2024-04-21T23:16:58.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:58.930+0000 7ff0df88c700 1 -- 172.21.15.40:0/3125723297 >> 172.21.15.40:0/3125723297 conn(0x7ff0d80780a0 msgr2=0x7ff0d81004b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:58.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:58.930+0000 7ff0df88c700 1 -- 172.21.15.40:0/3125723297 shutdown_connections 2024-04-21T23:16:58.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:16:58.930+0000 7ff0df88c700 1 -- 172.21.15.40:0/3125723297 wait complete. 2024-04-21T23:16:59.059 INFO:teuthology.orchestra.run.smithi040.stdout:287762808867 2024-04-21T23:16:59.059 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-04-21T23:16:59.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:16:59 smithi040 ceph-mon[27984]: pgmap v326: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s; 13 B/s, 1 keys/s, 1 objects/s recovering 2024-04-21T23:16:59.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:16:59 smithi174 ceph-mon[31409]: pgmap v326: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s; 13 B/s, 1 keys/s, 1 objects/s recovering 2024-04-21T23:16:59.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:16:59 smithi111 ceph-mon[31085]: pgmap v326: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s; 13 B/s, 1 keys/s, 1 objects/s recovering 2024-04-21T23:17:00.927 INFO:teuthology.orchestra.run.smithi040.stdout:55834574925 2024-04-21T23:17:00.927 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-21T23:17:01.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.129+0000 7fe88e7e9700 1 -- 172.21.15.40:0/844731457 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe888102540 msgr2=0x7fe888102920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:01.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.129+0000 7fe88e7e9700 1 --2- 172.21.15.40:0/844731457 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe888102540 0x7fe888102920 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fe878009990 tx=0x7fe878009ca0 comp rx=0 tx=0).stop 2024-04-21T23:17:01.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.129+0000 7fe88e7e9700 1 -- 172.21.15.40:0/844731457 shutdown_connections 2024-04-21T23:17:01.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.129+0000 7fe88e7e9700 1 --2- 172.21.15.40:0/844731457 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe8881075a0 0x7fe88810f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:01.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.129+0000 7fe88e7e9700 1 --2- 172.21.15.40:0/844731457 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe888102540 0x7fe888102920 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:01.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.129+0000 7fe88e7e9700 1 --2- 172.21.15.40:0/844731457 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe888102ef0 0x7fe888106ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:01.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.129+0000 7fe88e7e9700 1 -- 172.21.15.40:0/844731457 >> 172.21.15.40:0/844731457 conn(0x7fe8880792c0 msgr2=0x7fe8880796d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:01.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.129+0000 7fe88e7e9700 1 -- 172.21.15.40:0/844731457 shutdown_connections 2024-04-21T23:17:01.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.129+0000 7fe88e7e9700 1 -- 172.21.15.40:0/844731457 wait complete. 2024-04-21T23:17:01.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.129+0000 7fe88e7e9700 1 Processor -- start 2024-04-21T23:17:01.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.130+0000 7fe88e7e9700 1 -- start start 2024-04-21T23:17:01.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.130+0000 7fe88e7e9700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe888102540 0x7fe888074530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:01.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.130+0000 7fe88e7e9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe888102ef0 0x7fe888074a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:01.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.130+0000 7fe88e7e9700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8881075a0 0x7fe888074fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:01.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.130+0000 7fe88e7e9700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe888112590 con 0x7fe8881075a0 2024-04-21T23:17:01.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.130+0000 7fe88e7e9700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe888112410 con 0x7fe888102ef0 2024-04-21T23:17:01.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.130+0000 7fe88e7e9700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe888112710 con 0x7fe888102540 2024-04-21T23:17:01.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.130+0000 7fe88cd86700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8881075a0 0x7fe888074fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:01.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.130+0000 7fe88cd86700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8881075a0 0x7fe888074fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57734/0 (socket says 172.21.15.40:57734) 2024-04-21T23:17:01.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.130+0000 7fe88cd86700 1 -- 172.21.15.40:0/2579413326 learned_addr learned my addr 172.21.15.40:0/2579413326 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:17:01.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.130+0000 7fe887fff700 1 --2- 172.21.15.40:0/2579413326 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe888102540 0x7fe888074530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:01.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.130+0000 7fe8877fe700 1 --2- 172.21.15.40:0/2579413326 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe888102ef0 0x7fe888074a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:01.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.131+0000 7fe88cd86700 1 -- 172.21.15.40:0/2579413326 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe888102540 msgr2=0x7fe888074530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:01.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.131+0000 7fe88cd86700 1 --2- 172.21.15.40:0/2579413326 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe888102540 0x7fe888074530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:01.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.131+0000 7fe88cd86700 1 -- 172.21.15.40:0/2579413326 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe888102ef0 msgr2=0x7fe888074a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:01.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.131+0000 7fe88cd86700 1 --2- 172.21.15.40:0/2579413326 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe888102ef0 0x7fe888074a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:01.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.131+0000 7fe88cd86700 1 -- 172.21.15.40:0/2579413326 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8881a6f10 con 0x7fe8881075a0 2024-04-21T23:17:01.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.131+0000 7fe8877fe700 1 --2- 172.21.15.40:0/2579413326 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe888102ef0 0x7fe888074a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:17:01.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.131+0000 7fe88cd86700 1 --2- 172.21.15.40:0/2579413326 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8881075a0 0x7fe888074fb0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fe88000ef30 tx=0x7fe88000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:01.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.131+0000 7fe8857fa700 1 -- 172.21.15.40:0/2579413326 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8800092e0 con 0x7fe8881075a0 2024-04-21T23:17:01.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.131+0000 7fe88e7e9700 1 -- 172.21.15.40:0/2579413326 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8881a71a0 con 0x7fe8881075a0 2024-04-21T23:17:01.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.131+0000 7fe8857fa700 1 -- 172.21.15.40:0/2579413326 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe880004bb0 con 0x7fe8881075a0 2024-04-21T23:17:01.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.131+0000 7fe88e7e9700 1 -- 172.21.15.40:0/2579413326 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8881a76e0 con 0x7fe8881075a0 2024-04-21T23:17:01.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.131+0000 7fe8857fa700 1 -- 172.21.15.40:0/2579413326 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe880004280 con 0x7fe8881075a0 2024-04-21T23:17:01.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.132+0000 7fe88e7e9700 1 -- 172.21.15.40:0/2579413326 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe874001030 con 0x7fe8881075a0 2024-04-21T23:17:01.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.134+0000 7fe8857fa700 1 -- 172.21.15.40:0/2579413326 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fe88001b030 con 0x7fe8881075a0 2024-04-21T23:17:01.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.135+0000 7fe8857fa700 1 --2- 172.21.15.40:0/2579413326 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fe8700a2cb0 0x7fe8700a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:01.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.135+0000 7fe887fff700 1 --2- 172.21.15.40:0/2579413326 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fe8700a2cb0 0x7fe8700a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:01.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.135+0000 7fe8857fa700 1 -- 172.21.15.40:0/2579413326 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fe8800cf950 con 0x7fe8881075a0 2024-04-21T23:17:01.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.135+0000 7fe8857fa700 1 --2- 172.21.15.40:0/2579413326 >> [v2:172.21.15.111:6808/1810866070,v1:172.21.15.111:6809/1810866070] conn(0x7fe8700ade10 0x7fe8700b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:01.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.135+0000 7fe8857fa700 1 -- 172.21.15.40:0/2579413326 --> [v2:172.21.15.111:6808/1810866070,v1:172.21.15.111:6809/1810866070] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe88001fe50 con 0x7fe8700ade10 2024-04-21T23:17:01.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.135+0000 7fe8857fa700 1 -- 172.21.15.40:0/2579413326 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7fe8800cfd00 con 0x7fe8881075a0 2024-04-21T23:17:01.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.135+0000 7fe887fff700 1 --2- 172.21.15.40:0/2579413326 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fe8700a2cb0 0x7fe8700a5170 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fe87800b2e0 tx=0x7fe878005960 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:01.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.136+0000 7fe8877fe700 1 --2- 172.21.15.40:0/2579413326 >> [v2:172.21.15.111:6808/1810866070,v1:172.21.15.111:6809/1810866070] conn(0x7fe8700ade10 0x7fe8700b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:01.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.136+0000 7fe8877fe700 1 --2- 172.21.15.40:0/2579413326 >> [v2:172.21.15.111:6808/1810866070,v1:172.21.15.111:6809/1810866070] conn(0x7fe8700ade10 0x7fe8700b02b0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:01.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.137+0000 7fe8857fa700 1 -- 172.21.15.40:0/2579413326 <== osd.4 v2:172.21.15.111:6808/1810866070 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe88001fe50 con 0x7fe8700ade10 2024-04-21T23:17:01.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.154+0000 7fe88e7e9700 1 -- 172.21.15.40:0/2579413326 --> [v2:172.21.15.111:6808/1810866070,v1:172.21.15.111:6809/1810866070] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fe874002e60 con 0x7fe8700ade10 2024-04-21T23:17:01.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.154+0000 7fe8857fa700 1 -- 172.21.15.40:0/2579413326 <== osd.4 v2:172.21.15.111:6808/1810866070 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fe874002e60 con 0x7fe8700ade10 2024-04-21T23:17:01.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.154+0000 7fe88e7e9700 1 -- 172.21.15.40:0/2579413326 >> [v2:172.21.15.111:6808/1810866070,v1:172.21.15.111:6809/1810866070] conn(0x7fe8700ade10 msgr2=0x7fe8700b02b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:01.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.154+0000 7fe88e7e9700 1 --2- 172.21.15.40:0/2579413326 >> [v2:172.21.15.111:6808/1810866070,v1:172.21.15.111:6809/1810866070] conn(0x7fe8700ade10 0x7fe8700b02b0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:01.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.155+0000 7fe88e7e9700 1 -- 172.21.15.40:0/2579413326 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fe8700a2cb0 msgr2=0x7fe8700a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:01.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.155+0000 7fe88e7e9700 1 --2- 172.21.15.40:0/2579413326 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fe8700a2cb0 0x7fe8700a5170 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fe87800b2e0 tx=0x7fe878005960 comp rx=0 tx=0).stop 2024-04-21T23:17:01.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.155+0000 7fe88e7e9700 1 -- 172.21.15.40:0/2579413326 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8881075a0 msgr2=0x7fe888074fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:01.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.155+0000 7fe88e7e9700 1 --2- 172.21.15.40:0/2579413326 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8881075a0 0x7fe888074fb0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fe88000ef30 tx=0x7fe88000c3a0 comp rx=0 tx=0).stop 2024-04-21T23:17:01.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.155+0000 7fe88cd86700 1 -- 172.21.15.40:0/2579413326 reap_dead start 2024-04-21T23:17:01.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.155+0000 7fe88e7e9700 1 -- 172.21.15.40:0/2579413326 shutdown_connections 2024-04-21T23:17:01.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.155+0000 7fe88e7e9700 1 -- 172.21.15.40:0/2579413326 >> 172.21.15.40:0/2579413326 conn(0x7fe8880792c0 msgr2=0x7fe888100350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:01.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.155+0000 7fe88e7e9700 1 -- 172.21.15.40:0/2579413326 shutdown_connections 2024-04-21T23:17:01.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.155+0000 7fe88e7e9700 1 -- 172.21.15.40:0/2579413326 wait complete. 2024-04-21T23:17:01.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.340+0000 7f2c37260700 1 -- 172.21.15.40:0/781966574 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c30102ca0 msgr2=0x7f2c3010f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:01.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.340+0000 7f2c37260700 1 --2- 172.21.15.40:0/781966574 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c30102ca0 0x7f2c3010f110 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f2c2800b0d0 tx=0x7f2c2800b3e0 comp rx=0 tx=0).stop 2024-04-21T23:17:01.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.340+0000 7f2c37260700 1 -- 172.21.15.40:0/781966574 shutdown_connections 2024-04-21T23:17:01.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.340+0000 7f2c37260700 1 --2- 172.21.15.40:0/781966574 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c30102ca0 0x7f2c3010f110 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:01.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.340+0000 7f2c37260700 1 --2- 172.21.15.40:0/781966574 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c301082f0 0x7f2c301086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:01.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.340+0000 7f2c37260700 1 --2- 172.21.15.40:0/781966574 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2c301022e0 0x7f2c30102760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:01.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.340+0000 7f2c37260700 1 -- 172.21.15.40:0/781966574 >> 172.21.15.40:0/781966574 conn(0x7f2c30079390 msgr2=0x7f2c300797a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:01.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.340+0000 7f2c37260700 1 -- 172.21.15.40:0/781966574 shutdown_connections 2024-04-21T23:17:01.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.340+0000 7f2c37260700 1 -- 172.21.15.40:0/781966574 wait complete. 2024-04-21T23:17:01.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.341+0000 7f2c37260700 1 Processor -- start 2024-04-21T23:17:01.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.341+0000 7f2c37260700 1 -- start start 2024-04-21T23:17:01.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.341+0000 7f2c37260700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c301022e0 0x7f2c3019e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:01.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.341+0000 7f2c37260700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c30102ca0 0x7f2c3019eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:01.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.341+0000 7f2c37260700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2c301082f0 0x7f2c30198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:01.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.341+0000 7f2c37260700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2c301125a0 con 0x7f2c301022e0 2024-04-21T23:17:01.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.341+0000 7f2c37260700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f2c30112420 con 0x7f2c30102ca0 2024-04-21T23:17:01.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.341+0000 7f2c37260700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2c30112720 con 0x7f2c301082f0 2024-04-21T23:17:01.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.341+0000 7f2c34ffc700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c301022e0 0x7f2c3019e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:01.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.342+0000 7f2c34ffc700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c301022e0 0x7f2c3019e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57758/0 (socket says 172.21.15.40:57758) 2024-04-21T23:17:01.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.342+0000 7f2c34ffc700 1 -- 172.21.15.40:0/3777287287 learned_addr learned my addr 172.21.15.40:0/3777287287 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:17:01.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.342+0000 7f2c2ffff700 1 --2- 172.21.15.40:0/3777287287 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c30102ca0 0x7f2c3019eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:01.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.342+0000 7f2c357fd700 1 --2- 172.21.15.40:0/3777287287 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2c301082f0 0x7f2c30198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:01.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.342+0000 7f2c34ffc700 1 -- 172.21.15.40:0/3777287287 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2c301082f0 msgr2=0x7f2c30198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:01.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.342+0000 7f2c34ffc700 1 --2- 172.21.15.40:0/3777287287 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2c301082f0 0x7f2c30198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:01.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.342+0000 7f2c34ffc700 1 -- 172.21.15.40:0/3777287287 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c30102ca0 msgr2=0x7f2c3019eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:01.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.342+0000 7f2c34ffc700 1 --2- 172.21.15.40:0/3777287287 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c30102ca0 0x7f2c3019eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:01.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.342+0000 7f2c34ffc700 1 -- 172.21.15.40:0/3777287287 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c30198fe0 con 0x7f2c301022e0 2024-04-21T23:17:01.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.342+0000 7f2c34ffc700 1 --2- 172.21.15.40:0/3777287287 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c301022e0 0x7f2c3019e630 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f2c2400c9b0 tx=0x7f2c2400ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:01.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.342+0000 7f2c2dffb700 1 -- 172.21.15.40:0/3777287287 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c24007b00 con 0x7f2c301022e0 2024-04-21T23:17:01.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.342+0000 7f2c2dffb700 1 -- 172.21.15.40:0/3777287287 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c2400ce50 con 0x7f2c301022e0 2024-04-21T23:17:01.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.342+0000 7f2c37260700 1 -- 172.21.15.40:0/3777287287 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c301992d0 con 0x7f2c301022e0 2024-04-21T23:17:01.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.342+0000 7f2c2dffb700 1 -- 172.21.15.40:0/3777287287 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c24018660 con 0x7f2c301022e0 2024-04-21T23:17:01.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.342+0000 7f2c37260700 1 -- 172.21.15.40:0/3777287287 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c301a55b0 con 0x7f2c301022e0 2024-04-21T23:17:01.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.343+0000 7f2c37260700 1 -- 172.21.15.40:0/3777287287 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2c30101580 con 0x7f2c301022e0 2024-04-21T23:17:01.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.344+0000 7f2c2dffb700 1 -- 172.21.15.40:0/3777287287 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f2c2400f450 con 0x7f2c301022e0 2024-04-21T23:17:01.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.345+0000 7f2c2dffb700 1 --2- 172.21.15.40:0/3777287287 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f2c200a2d30 0x7f2c200a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:01.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.345+0000 7f2c2ffff700 1 --2- 172.21.15.40:0/3777287287 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f2c200a2d30 0x7f2c200a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:01.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.345+0000 7f2c2dffb700 1 -- 172.21.15.40:0/3777287287 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f2c240cf770 con 0x7f2c301022e0 2024-04-21T23:17:01.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.345+0000 7f2c2dffb700 1 --2- 172.21.15.40:0/3777287287 >> [v2:172.21.15.174:6808/3752986711,v1:172.21.15.174:6809/3752986711] conn(0x7f2c200ade90 0x7f2c200b0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:01.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.345+0000 7f2c2dffb700 1 -- 172.21.15.40:0/3777287287 --> [v2:172.21.15.174:6808/3752986711,v1:172.21.15.174:6809/3752986711] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2c2400eec0 con 0x7f2c200ade90 2024-04-21T23:17:01.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.345+0000 7f2c2dffb700 1 -- 172.21.15.40:0/3777287287 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7f2c240cfbc0 con 0x7f2c301022e0 2024-04-21T23:17:01.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.346+0000 7f2c2ffff700 1 --2- 172.21.15.40:0/3777287287 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f2c200a2d30 0x7f2c200a51f0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f2c30199ec0 tx=0x7f2c1c00b3c0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:01.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.346+0000 7f2c357fd700 1 --2- 172.21.15.40:0/3777287287 >> [v2:172.21.15.174:6808/3752986711,v1:172.21.15.174:6809/3752986711] conn(0x7f2c200ade90 0x7f2c200b0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:01.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.346+0000 7f2c357fd700 1 --2- 172.21.15.40:0/3777287287 >> [v2:172.21.15.174:6808/3752986711,v1:172.21.15.174:6809/3752986711] conn(0x7f2c200ade90 0x7f2c200b0330 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:01.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.347+0000 7f2c2dffb700 1 -- 172.21.15.40:0/3777287287 <== osd.5 v2:172.21.15.174:6808/3752986711 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2c2400eec0 con 0x7f2c200ade90 2024-04-21T23:17:01.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.363+0000 7f2c37260700 1 -- 172.21.15.40:0/3777287287 --> [v2:172.21.15.174:6808/3752986711,v1:172.21.15.174:6809/3752986711] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2c301039a0 con 0x7f2c200ade90 2024-04-21T23:17:01.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.364+0000 7f2c2dffb700 1 -- 172.21.15.40:0/3777287287 <== osd.5 v2:172.21.15.174:6808/3752986711 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f2c301039a0 con 0x7f2c200ade90 2024-04-21T23:17:01.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.364+0000 7f2c37260700 1 -- 172.21.15.40:0/3777287287 >> [v2:172.21.15.174:6808/3752986711,v1:172.21.15.174:6809/3752986711] conn(0x7f2c200ade90 msgr2=0x7f2c200b0330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:01.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.364+0000 7f2c37260700 1 --2- 172.21.15.40:0/3777287287 >> [v2:172.21.15.174:6808/3752986711,v1:172.21.15.174:6809/3752986711] conn(0x7f2c200ade90 0x7f2c200b0330 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:01.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.364+0000 7f2c37260700 1 -- 172.21.15.40:0/3777287287 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f2c200a2d30 msgr2=0x7f2c200a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:01.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.364+0000 7f2c37260700 1 --2- 172.21.15.40:0/3777287287 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f2c200a2d30 0x7f2c200a51f0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f2c30199ec0 tx=0x7f2c1c00b3c0 comp rx=0 tx=0).stop 2024-04-21T23:17:01.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.364+0000 7f2c37260700 1 -- 172.21.15.40:0/3777287287 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c301022e0 msgr2=0x7f2c3019e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:01.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.364+0000 7f2c37260700 1 --2- 172.21.15.40:0/3777287287 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c301022e0 0x7f2c3019e630 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f2c2400c9b0 tx=0x7f2c2400ccc0 comp rx=0 tx=0).stop 2024-04-21T23:17:01.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.364+0000 7f2c357fd700 1 -- 172.21.15.40:0/3777287287 reap_dead start 2024-04-21T23:17:01.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.364+0000 7f2c37260700 1 -- 172.21.15.40:0/3777287287 shutdown_connections 2024-04-21T23:17:01.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.364+0000 7f2c37260700 1 -- 172.21.15.40:0/3777287287 >> 172.21.15.40:0/3777287287 conn(0x7f2c30079390 msgr2=0x7f2c30100ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:01.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.364+0000 7f2c37260700 1 -- 172.21.15.40:0/3777287287 shutdown_connections 2024-04-21T23:17:01.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:01.364+0000 7f2c37260700 1 -- 172.21.15.40:0/3777287287 wait complete. 2024-04-21T23:17:01.681 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:01 smithi040 ceph-mon[27984]: pgmap v327: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 12 B/s, 1 keys/s, 1 objects/s recovering 2024-04-21T23:17:01.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:01 smithi174 ceph-mon[31409]: pgmap v327: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 12 B/s, 1 keys/s, 1 objects/s recovering 2024-04-21T23:17:01.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:01 smithi111 ceph-mon[31085]: pgmap v327: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 12 B/s, 1 keys/s, 1 objects/s recovering 2024-04-21T23:17:02.248 INFO:teuthology.orchestra.run.smithi040.stdout:141733920831 2024-04-21T23:17:02.248 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-21T23:17:02.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:02.452+0000 7f9ae817a700 1 -- 172.21.15.40:0/106535108 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ae006a7d0 msgr2=0x7f9ae006ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:02.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:02.452+0000 7f9ae817a700 1 --2- 172.21.15.40:0/106535108 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ae006a7d0 0x7f9ae006ac30 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f9ad0009990 tx=0x7f9ad0009ca0 comp rx=0 tx=0).stop 2024-04-21T23:17:02.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:02.453+0000 7f9ae817a700 1 -- 172.21.15.40:0/106535108 shutdown_connections 2024-04-21T23:17:02.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:02.453+0000 7f9ae817a700 1 --2- 172.21.15.40:0/106535108 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9ae006b170 0x7f9ae010f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:02.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:02.453+0000 7f9ae817a700 1 --2- 172.21.15.40:0/106535108 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ae01082a0 0x7f9ae0108680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:02.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:02.453+0000 7f9ae817a700 1 --2- 172.21.15.40:0/106535108 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ae006a7d0 0x7f9ae006ac30 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:02.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:02.453+0000 7f9ae817a700 1 -- 172.21.15.40:0/106535108 >> 172.21.15.40:0/106535108 conn(0x7f9ae00780a0 msgr2=0x7f9ae00784b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:02.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:02.453+0000 7f9ae817a700 1 -- 172.21.15.40:0/106535108 shutdown_connections 2024-04-21T23:17:02.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:02.453+0000 7f9ae817a700 1 -- 172.21.15.40:0/106535108 wait complete. 2024-04-21T23:17:02.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:02.453+0000 7f9ae817a700 1 Processor -- start 2024-04-21T23:17:02.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:02.454+0000 7f9ae817a700 1 -- start start 2024-04-21T23:17:02.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:02.454+0000 7f9ae817a700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ae006a7d0 0x7f9ae01011f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:02.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:02.454+0000 7f9ae817a700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9ae006b170 0x7f9ae0101730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:02.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:02.454+0000 7f9ae817a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ae01082a0 0x7f9ae0101c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:02.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:02.454+0000 7f9ae817a700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9ae0112520 con 0x7f9ae006a7d0 2024-04-21T23:17:02.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:02.454+0000 7f9ae817a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f9ae01123a0 con 0x7f9ae01082a0 2024-04-21T23:17:02.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:02.454+0000 7f9ae817a700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9ae01126a0 con 0x7f9ae006b170 2024-04-21T23:17:02.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:02.454+0000 7f9ae5f16700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ae006a7d0 0x7f9ae01011f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:02.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:02.454+0000 7f9ae5f16700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ae006a7d0 0x7f9ae01011f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45440/0 (socket says 172.21.15.40:45440) 2024-04-21T23:17:02.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:02.454+0000 7f9ae5f16700 1 -- 172.21.15.40:0/2253503719 learned_addr learned my addr 172.21.15.40:0/2253503719 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:17:02.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:02.454+0000 7f9ae5715700 1 --2- 172.21.15.40:0/2253503719 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9ae006b170 0x7f9ae0101730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:02.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:02.454+0000 7f9ae6717700 1 --2- 172.21.15.40:0/2253503719 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ae01082a0 0x7f9ae0101c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:02.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:02.455+0000 7f9ae5f16700 1 -- 172.21.15.40:0/2253503719 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9ae006b170 msgr2=0x7f9ae0101730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:02.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:02.455+0000 7f9ae5f16700 1 --2- 172.21.15.40:0/2253503719 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9ae006b170 0x7f9ae0101730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:02.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:02.455+0000 7f9ae5f16700 1 -- 172.21.15.40:0/2253503719 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ae01082a0 msgr2=0x7f9ae0101c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:02.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:02.455+0000 7f9ae5f16700 1 --2- 172.21.15.40:0/2253503719 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ae01082a0 0x7f9ae0101c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:02.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:02.455+0000 7f9ae5f16700 1 -- 172.21.15.40:0/2253503719 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ae01a6ee0 con 0x7f9ae006a7d0 2024-04-21T23:17:02.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:02.455+0000 7f9ae6717700 1 --2- 172.21.15.40:0/2253503719 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ae01082a0 0x7f9ae0101c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:17:02.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:02.455+0000 7f9ae5f16700 1 --2- 172.21.15.40:0/2253503719 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ae006a7d0 0x7f9ae01011f0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f9adc00ef10 tx=0x7f9adc00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:02.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:02.455+0000 7f9ad6ffd700 1 -- 172.21.15.40:0/2253503719 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9adc00c9c0 con 0x7f9ae006a7d0 2024-04-21T23:17:02.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:02.455+0000 7f9ad6ffd700 1 -- 172.21.15.40:0/2253503719 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9adc019070 con 0x7f9ae006a7d0 2024-04-21T23:17:02.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:02.455+0000 7f9ae817a700 1 -- 172.21.15.40:0/2253503719 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ae01a7170 con 0x7f9ae006a7d0 2024-04-21T23:17:02.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:02.455+0000 7f9ae817a700 1 -- 172.21.15.40:0/2253503719 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ae01a76d0 con 0x7f9ae006a7d0 2024-04-21T23:17:02.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:02.455+0000 7f9ad6ffd700 1 -- 172.21.15.40:0/2253503719 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9adc007e60 con 0x7f9ae006a7d0 2024-04-21T23:17:02.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:02.456+0000 7f9ae817a700 1 -- 172.21.15.40:0/2253503719 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9ac4001030 con 0x7f9ae006a7d0 2024-04-21T23:17:02.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:02.457+0000 7f9ad6ffd700 1 -- 172.21.15.40:0/2253503719 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f9adc012050 con 0x7f9ae006a7d0 2024-04-21T23:17:02.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:02.458+0000 7f9ad6ffd700 1 --2- 172.21.15.40:0/2253503719 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f9acc0a2d30 0x7f9acc0a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:02.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:02.459+0000 7f9ad6ffd700 1 -- 172.21.15.40:0/2253503719 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f9adc0cfc90 con 0x7f9ae006a7d0 2024-04-21T23:17:02.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:02.459+0000 7f9ae5715700 1 --2- 172.21.15.40:0/2253503719 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f9acc0a2d30 0x7f9acc0a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:02.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:02.459+0000 7f9ad6ffd700 1 --2- 172.21.15.40:0/2253503719 >> [v2:172.21.15.174:6800/439695825,v1:172.21.15.174:6801/439695825] conn(0x7f9acc0ade90 0x7f9acc0b0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:02.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:02.459+0000 7f9ad6ffd700 1 -- 172.21.15.40:0/2253503719 --> [v2:172.21.15.174:6800/439695825,v1:172.21.15.174:6801/439695825] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9acc0b0a00 con 0x7f9acc0ade90 2024-04-21T23:17:02.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:02.459+0000 7f9ad6ffd700 1 -- 172.21.15.40:0/2253503719 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7f9adc0d00e0 con 0x7f9ae006a7d0 2024-04-21T23:17:02.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:02.459+0000 7f9ae6717700 1 --2- 172.21.15.40:0/2253503719 >> [v2:172.21.15.174:6800/439695825,v1:172.21.15.174:6801/439695825] conn(0x7f9acc0ade90 0x7f9acc0b0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:02.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:02.459+0000 7f9ae5715700 1 --2- 172.21.15.40:0/2253503719 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f9acc0a2d30 0x7f9acc0a51f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f9ad000b2e0 tx=0x7f9ad0005960 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:02.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:02.460+0000 7f9ae6717700 1 --2- 172.21.15.40:0/2253503719 >> [v2:172.21.15.174:6800/439695825,v1:172.21.15.174:6801/439695825] conn(0x7f9acc0ade90 0x7f9acc0b0330 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:02.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:02.461+0000 7f9ad6ffd700 1 -- 172.21.15.40:0/2253503719 <== osd.2 v2:172.21.15.174:6800/439695825 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9acc0b0a00 con 0x7f9acc0ade90 2024-04-21T23:17:02.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:02.479+0000 7f9ae817a700 1 -- 172.21.15.40:0/2253503719 --> [v2:172.21.15.174:6800/439695825,v1:172.21.15.174:6801/439695825] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9ac4002e60 con 0x7f9acc0ade90 2024-04-21T23:17:02.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:02.480+0000 7f9ad6ffd700 1 -- 172.21.15.40:0/2253503719 <== osd.2 v2:172.21.15.174:6800/439695825 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f9ac4002e60 con 0x7f9acc0ade90 2024-04-21T23:17:02.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:02.480+0000 7f9ae817a700 1 -- 172.21.15.40:0/2253503719 >> [v2:172.21.15.174:6800/439695825,v1:172.21.15.174:6801/439695825] conn(0x7f9acc0ade90 msgr2=0x7f9acc0b0330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:02.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:02.480+0000 7f9ae817a700 1 --2- 172.21.15.40:0/2253503719 >> [v2:172.21.15.174:6800/439695825,v1:172.21.15.174:6801/439695825] conn(0x7f9acc0ade90 0x7f9acc0b0330 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:02.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:02.480+0000 7f9ae817a700 1 -- 172.21.15.40:0/2253503719 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f9acc0a2d30 msgr2=0x7f9acc0a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:02.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:02.480+0000 7f9ae817a700 1 --2- 172.21.15.40:0/2253503719 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f9acc0a2d30 0x7f9acc0a51f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f9ad000b2e0 tx=0x7f9ad0005960 comp rx=0 tx=0).stop 2024-04-21T23:17:02.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:02.480+0000 7f9ae817a700 1 -- 172.21.15.40:0/2253503719 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ae006a7d0 msgr2=0x7f9ae01011f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:02.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:02.480+0000 7f9ae817a700 1 --2- 172.21.15.40:0/2253503719 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ae006a7d0 0x7f9ae01011f0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f9adc00ef10 tx=0x7f9adc00c3a0 comp rx=0 tx=0).stop 2024-04-21T23:17:02.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:02.480+0000 7f9ae6717700 1 -- 172.21.15.40:0/2253503719 reap_dead start 2024-04-21T23:17:02.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:02.480+0000 7f9ae817a700 1 -- 172.21.15.40:0/2253503719 shutdown_connections 2024-04-21T23:17:02.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:02.480+0000 7f9ae817a700 1 -- 172.21.15.40:0/2253503719 >> 172.21.15.40:0/2253503719 conn(0x7f9ae00780a0 msgr2=0x7f9ae00ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:02.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:02.480+0000 7f9ae817a700 1 -- 172.21.15.40:0/2253503719 shutdown_connections 2024-04-21T23:17:02.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:02.480+0000 7f9ae817a700 1 -- 172.21.15.40:0/2253503719 wait complete. 2024-04-21T23:17:02.950 INFO:teuthology.orchestra.run.smithi040.stdout:163208757305 2024-04-21T23:17:02.951 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-21T23:17:03.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:03.289+0000 7fc5a2bce700 1 -- 172.21.15.40:0/342195982 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc59c106490 msgr2=0x7fc59c10e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:03.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:03.289+0000 7fc5a2bce700 1 --2- 172.21.15.40:0/342195982 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc59c106490 0x7fc59c10e060 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fc59400b0d0 tx=0x7fc59400b3e0 comp rx=0 tx=0).stop 2024-04-21T23:17:03.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:03.289+0000 7fc5a2bce700 1 -- 172.21.15.40:0/342195982 shutdown_connections 2024-04-21T23:17:03.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:03.289+0000 7fc5a2bce700 1 --2- 172.21.15.40:0/342195982 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc59c106490 0x7fc59c10e060 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:03.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:03.289+0000 7fc5a2bce700 1 --2- 172.21.15.40:0/342195982 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc59c1013b0 0x7fc59c101790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:03.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:03.289+0000 7fc5a2bce700 1 --2- 172.21.15.40:0/342195982 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc59c101d60 0x7fc59c105db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:03.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:03.289+0000 7fc5a2bce700 1 -- 172.21.15.40:0/342195982 >> 172.21.15.40:0/342195982 conn(0x7fc59c0781d0 msgr2=0x7fc59c0785e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:03.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:03.289+0000 7fc5a2bce700 1 -- 172.21.15.40:0/342195982 shutdown_connections 2024-04-21T23:17:03.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:03.289+0000 7fc5a2bce700 1 -- 172.21.15.40:0/342195982 wait complete. 2024-04-21T23:17:03.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:03.290+0000 7fc5a2bce700 1 Processor -- start 2024-04-21T23:17:03.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:03.290+0000 7fc5a2bce700 1 -- start start 2024-04-21T23:17:03.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:03.290+0000 7fc5a2bce700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc59c1013b0 0x7fc59c102e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:03.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:03.290+0000 7fc5a2bce700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc59c101d60 0x7fc59c103390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:03.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:03.290+0000 7fc5a2bce700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc59c106490 0x7fc59c1038d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:03.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:03.290+0000 7fc5a2bce700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc59c1114f0 con 0x7fc59c106490 2024-04-21T23:17:03.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:03.290+0000 7fc5a2bce700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc59c111370 con 0x7fc59c1013b0 2024-04-21T23:17:03.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:03.290+0000 7fc5a2bce700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc59c111670 con 0x7fc59c101d60 2024-04-21T23:17:03.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:03.290+0000 7fc5a116b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc59c106490 0x7fc59c1038d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:03.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:03.290+0000 7fc5a116b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc59c106490 0x7fc59c1038d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45448/0 (socket says 172.21.15.40:45448) 2024-04-21T23:17:03.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:03.290+0000 7fc5a116b700 1 -- 172.21.15.40:0/1127306646 learned_addr learned my addr 172.21.15.40:0/1127306646 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:17:03.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:03.290+0000 7fc5a096a700 1 --2- 172.21.15.40:0/1127306646 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc59c1013b0 0x7fc59c102e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:03.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:03.290+0000 7fc59bfff700 1 --2- 172.21.15.40:0/1127306646 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc59c101d60 0x7fc59c103390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:03.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:03.291+0000 7fc5a116b700 1 -- 172.21.15.40:0/1127306646 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc59c101d60 msgr2=0x7fc59c103390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:03.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:03.291+0000 7fc5a116b700 1 --2- 172.21.15.40:0/1127306646 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc59c101d60 0x7fc59c103390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:03.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:03.291+0000 7fc5a116b700 1 -- 172.21.15.40:0/1127306646 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc59c1013b0 msgr2=0x7fc59c102e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:03.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:03.291+0000 7fc5a116b700 1 --2- 172.21.15.40:0/1127306646 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc59c1013b0 0x7fc59c102e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:03.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:03.291+0000 7fc5a116b700 1 -- 172.21.15.40:0/1127306646 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc59c1a6f20 con 0x7fc59c106490 2024-04-21T23:17:03.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:03.291+0000 7fc5a096a700 1 --2- 172.21.15.40:0/1127306646 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc59c1013b0 0x7fc59c102e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:17:03.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:03.291+0000 7fc59bfff700 1 --2- 172.21.15.40:0/1127306646 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc59c101d60 0x7fc59c103390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:17:03.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:03.291+0000 7fc5a116b700 1 --2- 172.21.15.40:0/1127306646 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc59c106490 0x7fc59c1038d0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fc594006070 tx=0x7fc594004ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:03.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:03.291+0000 7fc599ffb700 1 -- 172.21.15.40:0/1127306646 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc594021070 con 0x7fc59c106490 2024-04-21T23:17:03.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:03.291+0000 7fc599ffb700 1 -- 172.21.15.40:0/1127306646 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc594007df0 con 0x7fc59c106490 2024-04-21T23:17:03.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:03.291+0000 7fc5a2bce700 1 -- 172.21.15.40:0/1127306646 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc59c1a71b0 con 0x7fc59c106490 2024-04-21T23:17:03.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:03.291+0000 7fc599ffb700 1 -- 172.21.15.40:0/1127306646 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc594005470 con 0x7fc59c106490 2024-04-21T23:17:03.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:03.291+0000 7fc5a2bce700 1 -- 172.21.15.40:0/1127306646 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc59c1a7660 con 0x7fc59c106490 2024-04-21T23:17:03.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:03.292+0000 7fc5a2bce700 1 -- 172.21.15.40:0/1127306646 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc59c062940 con 0x7fc59c106490 2024-04-21T23:17:03.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:03.293+0000 7fc599ffb700 1 -- 172.21.15.40:0/1127306646 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fc594024070 con 0x7fc59c106490 2024-04-21T23:17:03.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:03.294+0000 7fc599ffb700 1 --2- 172.21.15.40:0/1127306646 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fc5840a2cb0 0x7fc5840a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:03.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:03.294+0000 7fc599ffb700 1 -- 172.21.15.40:0/1127306646 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fc59400f020 con 0x7fc59c106490 2024-04-21T23:17:03.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:03.295+0000 7fc599ffb700 1 --2- 172.21.15.40:0/1127306646 >> [v2:172.21.15.40:6810/3454312436,v1:172.21.15.40:6811/3454312436] conn(0x7fc5840ade10 0x7fc5840b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:03.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:03.295+0000 7fc599ffb700 1 -- 172.21.15.40:0/1127306646 --> [v2:172.21.15.40:6810/3454312436,v1:172.21.15.40:6811/3454312436] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc5840b0980 con 0x7fc5840ade10 2024-04-21T23:17:03.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:03.295+0000 7fc599ffb700 1 -- 172.21.15.40:0/1127306646 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7fc5940d1340 con 0x7fc59c106490 2024-04-21T23:17:03.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:03.295+0000 7fc5a096a700 1 --2- 172.21.15.40:0/1127306646 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fc5840a2cb0 0x7fc5840a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:03.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:03.295+0000 7fc59bfff700 1 --2- 172.21.15.40:0/1127306646 >> [v2:172.21.15.40:6810/3454312436,v1:172.21.15.40:6811/3454312436] conn(0x7fc5840ade10 0x7fc5840b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:03.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:03.295+0000 7fc5a096a700 1 --2- 172.21.15.40:0/1127306646 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fc5840a2cb0 0x7fc5840a5170 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fc58c005950 tx=0x7fc58c005f50 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:03.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:03.296+0000 7fc59bfff700 1 --2- 172.21.15.40:0/1127306646 >> [v2:172.21.15.40:6810/3454312436,v1:172.21.15.40:6811/3454312436] conn(0x7fc5840ade10 0x7fc5840b02b0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:03.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:03.297+0000 7fc599ffb700 1 -- 172.21.15.40:0/1127306646 <== osd.3 v2:172.21.15.40:6810/3454312436 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc5840b0980 con 0x7fc5840ade10 2024-04-21T23:17:03.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:03.315+0000 7fc5a2bce700 1 -- 172.21.15.40:0/1127306646 --> [v2:172.21.15.40:6810/3454312436,v1:172.21.15.40:6811/3454312436] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc59c062dc0 con 0x7fc5840ade10 2024-04-21T23:17:03.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:03.315+0000 7fc599ffb700 1 -- 172.21.15.40:0/1127306646 <== osd.3 v2:172.21.15.40:6810/3454312436 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fc59c062dc0 con 0x7fc5840ade10 2024-04-21T23:17:03.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:03.315+0000 7fc5a2bce700 1 -- 172.21.15.40:0/1127306646 >> [v2:172.21.15.40:6810/3454312436,v1:172.21.15.40:6811/3454312436] conn(0x7fc5840ade10 msgr2=0x7fc5840b02b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:03.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:03.315+0000 7fc5a2bce700 1 --2- 172.21.15.40:0/1127306646 >> [v2:172.21.15.40:6810/3454312436,v1:172.21.15.40:6811/3454312436] conn(0x7fc5840ade10 0x7fc5840b02b0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:03.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:03.315+0000 7fc5a2bce700 1 -- 172.21.15.40:0/1127306646 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fc5840a2cb0 msgr2=0x7fc5840a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:03.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:03.315+0000 7fc5a2bce700 1 --2- 172.21.15.40:0/1127306646 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fc5840a2cb0 0x7fc5840a5170 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fc58c005950 tx=0x7fc58c005f50 comp rx=0 tx=0).stop 2024-04-21T23:17:03.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:03.315+0000 7fc5a2bce700 1 -- 172.21.15.40:0/1127306646 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc59c106490 msgr2=0x7fc59c1038d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:03.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:03.315+0000 7fc5a2bce700 1 --2- 172.21.15.40:0/1127306646 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc59c106490 0x7fc59c1038d0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fc594006070 tx=0x7fc594004ef0 comp rx=0 tx=0).stop 2024-04-21T23:17:03.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:03.315+0000 7fc5a116b700 1 -- 172.21.15.40:0/1127306646 reap_dead start 2024-04-21T23:17:03.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:03.316+0000 7fc5a2bce700 1 -- 172.21.15.40:0/1127306646 shutdown_connections 2024-04-21T23:17:03.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:03.316+0000 7fc5a2bce700 1 -- 172.21.15.40:0/1127306646 >> 172.21.15.40:0/1127306646 conn(0x7fc59c0781d0 msgr2=0x7fc59c06aba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:03.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:03.316+0000 7fc5a2bce700 1 -- 172.21.15.40:0/1127306646 shutdown_connections 2024-04-21T23:17:03.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:03.316+0000 7fc5a2bce700 1 -- 172.21.15.40:0/1127306646 wait complete. 2024-04-21T23:17:03.744 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:03 smithi040 ceph-mon[27984]: pgmap v328: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 5 op/s; 10 B/s, 0 keys/s, 1 objects/s recovering 2024-04-21T23:17:03.770 INFO:teuthology.orchestra.run.smithi040.stdout:253403070505 2024-04-21T23:17:03.771 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-04-21T23:17:03.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:03 smithi111 ceph-mon[31085]: pgmap v328: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 5 op/s; 10 B/s, 0 keys/s, 1 objects/s recovering 2024-04-21T23:17:03.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:03 smithi174 ceph-mon[31409]: pgmap v328: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 5 op/s; 10 B/s, 0 keys/s, 1 objects/s recovering 2024-04-21T23:17:04.403 INFO:teuthology.orchestra.run.smithi040.stdout:107374182467 2024-04-21T23:17:04.404 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-21T23:17:05.201 INFO:teuthology.orchestra.run.smithi040.stdout:77309411401 2024-04-21T23:17:05.202 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-21T23:17:05.221 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:17:05.221 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:17:05.229 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:17:05.746 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:05 smithi111 ceph-mon[31085]: pgmap v329: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s; 9 B/s, 0 keys/s, 1 objects/s recovering 2024-04-21T23:17:05.747 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:05 smithi174 ceph-mon[31409]: pgmap v329: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s; 9 B/s, 0 keys/s, 1 objects/s recovering 2024-04-21T23:17:05.836 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:05 smithi040 ceph-mon[27984]: pgmap v329: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s; 9 B/s, 0 keys/s, 1 objects/s recovering 2024-04-21T23:17:07.522 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:17:07.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:07.525+0000 7f59537fe700 1 -- 172.21.15.40:0/3880369960 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5940004230 con 0x7f5954109800 2024-04-21T23:17:07.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:07.525+0000 7f595add9700 1 -- 172.21.15.40:0/3880369960 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5954109800 msgr2=0x7f5954109c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:07.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:07.525+0000 7f595add9700 1 --2- 172.21.15.40:0/3880369960 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5954109800 0x7f5954109c80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5940009990 tx=0x7f5940009ca0 comp rx=0 tx=0).stop 2024-04-21T23:17:07.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:07.526+0000 7f595add9700 1 -- 172.21.15.40:0/3880369960 shutdown_connections 2024-04-21T23:17:07.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:07.526+0000 7f595add9700 1 --2- 172.21.15.40:0/3880369960 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5954117360 0x7f5954119750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:07.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:07.526+0000 7f595add9700 1 --2- 172.21.15.40:0/3880369960 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f595410c9b0 0x7f5954109230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:07.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:07.526+0000 7f595add9700 1 --2- 172.21.15.40:0/3880369960 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5954109800 0x7f5954109c80 secure :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5940009990 tx=0x7f5940009ca0 comp rx=0 tx=0).stop 2024-04-21T23:17:07.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:07.526+0000 7f595add9700 1 -- 172.21.15.40:0/3880369960 >> 172.21.15.40:0/3880369960 conn(0x7f595406d3a0 msgr2=0x7f595406d7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:07.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:07.526+0000 7f595add9700 1 -- 172.21.15.40:0/3880369960 shutdown_connections 2024-04-21T23:17:07.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:07.526+0000 7f595add9700 1 -- 172.21.15.40:0/3880369960 wait complete. 2024-04-21T23:17:07.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:07.527+0000 7f595add9700 1 Processor -- start 2024-04-21T23:17:07.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:07.527+0000 7f595add9700 1 -- start start 2024-04-21T23:17:07.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:07.527+0000 7f595add9700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f595410c9b0 0x7f5954113dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:07.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:07.527+0000 7f595add9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5954117360 0x7f5954114310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:07.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:07.527+0000 7f595add9700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5954116e30 0x7f5954114890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:07.529 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:17:07.529 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:17:07.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:07.528+0000 7f595add9700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f595411d160 con 0x7f5954116e30 2024-04-21T23:17:07.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:07.528+0000 7f595add9700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f595411cfe0 con 0x7f5954117360 2024-04-21T23:17:07.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:07.528+0000 7f595add9700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f595411d2e0 con 0x7f595410c9b0 2024-04-21T23:17:07.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:07.528+0000 7f5958b75700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f595410c9b0 0x7f5954113dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:07.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:07.528+0000 7f5958b75700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f595410c9b0 0x7f5954113dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.40:60524/0 (socket says 172.21.15.40:60524) 2024-04-21T23:17:07.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:07.528+0000 7f5958b75700 1 -- 172.21.15.40:0/1657949815 learned_addr learned my addr 172.21.15.40:0/1657949815 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:17:07.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:07.528+0000 7f5959376700 1 --2- 172.21.15.40:0/1657949815 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5954116e30 0x7f5954114890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:07.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:07.528+0000 7f5953fff700 1 --2- 172.21.15.40:0/1657949815 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5954117360 0x7f5954114310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:07.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:07.529+0000 7f5959376700 1 -- 172.21.15.40:0/1657949815 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f595410c9b0 msgr2=0x7f5954113dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:07.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:07.529+0000 7f5959376700 1 --2- 172.21.15.40:0/1657949815 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f595410c9b0 0x7f5954113dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:07.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:07.529+0000 7f5959376700 1 -- 172.21.15.40:0/1657949815 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5954117360 msgr2=0x7f5954114310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:07.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:07.529+0000 7f5959376700 1 --2- 172.21.15.40:0/1657949815 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5954117360 0x7f5954114310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:07.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:07.529+0000 7f5959376700 1 -- 172.21.15.40:0/1657949815 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59541a99f0 con 0x7f5954116e30 2024-04-21T23:17:07.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:07.529+0000 7f5953fff700 1 --2- 172.21.15.40:0/1657949815 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5954117360 0x7f5954114310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:17:07.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:07.529+0000 7f5959376700 1 --2- 172.21.15.40:0/1657949815 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5954116e30 0x7f5954114890 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f594c00be00 tx=0x7f594c00bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:07.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:07.530+0000 7f5951ffb700 1 -- 172.21.15.40:0/1657949815 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f594c019070 con 0x7f5954116e30 2024-04-21T23:17:07.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:07.530+0000 7f5951ffb700 1 -- 172.21.15.40:0/1657949815 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f594c00cc90 con 0x7f5954116e30 2024-04-21T23:17:07.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:07.530+0000 7f5951ffb700 1 -- 172.21.15.40:0/1657949815 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f594c0127a0 con 0x7f5954116e30 2024-04-21T23:17:07.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:07.530+0000 7f595add9700 1 -- 172.21.15.40:0/1657949815 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59541a9ce0 con 0x7f5954116e30 2024-04-21T23:17:07.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:07.530+0000 7f595add9700 1 -- 172.21.15.40:0/1657949815 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59541aa140 con 0x7f5954116e30 2024-04-21T23:17:07.532 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:17:07.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:07.532+0000 7f595add9700 1 -- 172.21.15.40:0/1657949815 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5938001030 con 0x7f5954116e30 2024-04-21T23:17:07.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:07.536+0000 7f5951ffb700 1 -- 172.21.15.40:0/1657949815 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f594c01c070 con 0x7f5954116e30 2024-04-21T23:17:07.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:07.540+0000 7f5951ffb700 1 --2- 172.21.15.40:0/1657949815 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f59440a2e80 0x7f59440a5340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:07.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:07.540+0000 7f5951ffb700 1 -- 172.21.15.40:0/1657949815 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f594c0ce8b0 con 0x7f5954116e30 2024-04-21T23:17:07.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:07.540+0000 7f5951ffb700 1 --2- 172.21.15.40:0/1657949815 >> [v2:172.21.15.40:6818/1618855427,v1:172.21.15.40:6819/1618855427] conn(0x7f59440adfe0 0x7f59440b0480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:07.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:07.540+0000 7f5951ffb700 1 -- 172.21.15.40:0/1657949815 --> [v2:172.21.15.40:6818/1618855427,v1:172.21.15.40:6819/1618855427] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f594c010ec0 con 0x7f59440adfe0 2024-04-21T23:17:07.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:07.540+0000 7f5951ffb700 1 -- 172.21.15.40:0/1657949815 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7f594c0d32a0 con 0x7f5954116e30 2024-04-21T23:17:07.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:07.541+0000 7f5953fff700 1 --2- 172.21.15.40:0/1657949815 >> [v2:172.21.15.40:6818/1618855427,v1:172.21.15.40:6819/1618855427] conn(0x7f59440adfe0 0x7f59440b0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:07.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:07.542+0000 7f5953fff700 1 --2- 172.21.15.40:0/1657949815 >> [v2:172.21.15.40:6818/1618855427,v1:172.21.15.40:6819/1618855427] conn(0x7f59440adfe0 0x7f59440b0480 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:07.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:07.543+0000 7f5958b75700 1 --2- 172.21.15.40:0/1657949815 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f59440a2e80 0x7f59440a5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:07.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:07.543+0000 7f5951ffb700 1 -- 172.21.15.40:0/1657949815 <== osd.6 v2:172.21.15.40:6818/1618855427 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f594c010ec0 con 0x7f59440adfe0 2024-04-21T23:17:07.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:07.544+0000 7f5958b75700 1 --2- 172.21.15.40:0/1657949815 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f59440a2e80 0x7f59440a5340 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f59480099d0 tx=0x7f5948008040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:07.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:07.578+0000 7f595add9700 1 -- 172.21.15.40:0/1657949815 --> [v2:172.21.15.40:6818/1618855427,v1:172.21.15.40:6819/1618855427] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5938002e60 con 0x7f59440adfe0 2024-04-21T23:17:07.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:07.580+0000 7f5951ffb700 1 -- 172.21.15.40:0/1657949815 <== osd.6 v2:172.21.15.40:6818/1618855427 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f5938002e60 con 0x7f59440adfe0 2024-04-21T23:17:07.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:07.582+0000 7f593f7fe700 1 -- 172.21.15.40:0/1657949815 >> [v2:172.21.15.40:6818/1618855427,v1:172.21.15.40:6819/1618855427] conn(0x7f59440adfe0 msgr2=0x7f59440b0480 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:07.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:07.582+0000 7f593f7fe700 1 --2- 172.21.15.40:0/1657949815 >> [v2:172.21.15.40:6818/1618855427,v1:172.21.15.40:6819/1618855427] conn(0x7f59440adfe0 0x7f59440b0480 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:07.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:07.582+0000 7f593f7fe700 1 -- 172.21.15.40:0/1657949815 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f59440a2e80 msgr2=0x7f59440a5340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:07.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:07.582+0000 7f593f7fe700 1 --2- 172.21.15.40:0/1657949815 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f59440a2e80 0x7f59440a5340 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f59480099d0 tx=0x7f5948008040 comp rx=0 tx=0).stop 2024-04-21T23:17:07.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:07.582+0000 7f593f7fe700 1 -- 172.21.15.40:0/1657949815 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5954116e30 msgr2=0x7f5954114890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:07.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:07.582+0000 7f593f7fe700 1 --2- 172.21.15.40:0/1657949815 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5954116e30 0x7f5954114890 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f594c00be00 tx=0x7f594c00bee0 comp rx=0 tx=0).stop 2024-04-21T23:17:07.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:07.583+0000 7f5959376700 1 -- 172.21.15.40:0/1657949815 reap_dead start 2024-04-21T23:17:07.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:07.583+0000 7f593f7fe700 1 -- 172.21.15.40:0/1657949815 shutdown_connections 2024-04-21T23:17:07.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:07.583+0000 7f593f7fe700 1 -- 172.21.15.40:0/1657949815 >> 172.21.15.40:0/1657949815 conn(0x7f595406d3a0 msgr2=0x7f5954118b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:07.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:07.585+0000 7f593f7fe700 1 -- 172.21.15.40:0/1657949815 shutdown_connections 2024-04-21T23:17:07.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:07.585+0000 7f593f7fe700 1 -- 172.21.15.40:0/1657949815 wait complete. 2024-04-21T23:17:07.676 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:07 smithi040 ceph-mon[27984]: pgmap v330: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s; 9 B/s, 0 objects/s recovering 2024-04-21T23:17:07.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:07 smithi111 ceph-mon[31085]: pgmap v330: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s; 9 B/s, 0 objects/s recovering 2024-04-21T23:17:07.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:07 smithi174 ceph-mon[31409]: pgmap v330: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s; 9 B/s, 0 objects/s recovering 2024-04-21T23:17:08.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:08.662+0000 7f02e85a3700 1 -- 172.21.15.40:0/3141061239 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02e0069f40 msgr2=0x7f02e006a3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:08.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:08.662+0000 7f02e85a3700 1 --2- 172.21.15.40:0/3141061239 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02e0069f40 0x7f02e006a3a0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f02d0009990 tx=0x7f02d0009ca0 comp rx=0 tx=0).stop 2024-04-21T23:17:08.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:08.663+0000 7f02e85a3700 1 -- 172.21.15.40:0/3141061239 shutdown_connections 2024-04-21T23:17:08.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:08.663+0000 7f02e85a3700 1 --2- 172.21.15.40:0/3141061239 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f02e01082b0 0x7f02e0108690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:08.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:08.663+0000 7f02e85a3700 1 --2- 172.21.15.40:0/3141061239 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02e006a8e0 0x7f02e010f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:08.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:08.663+0000 7f02e85a3700 1 --2- 172.21.15.40:0/3141061239 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02e0069f40 0x7f02e006a3a0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:08.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:08.663+0000 7f02e85a3700 1 -- 172.21.15.40:0/3141061239 >> 172.21.15.40:0/3141061239 conn(0x7f02e00780e0 msgr2=0x7f02e00784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:08.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:08.663+0000 7f02e85a3700 1 -- 172.21.15.40:0/3141061239 shutdown_connections 2024-04-21T23:17:08.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:08.663+0000 7f02e85a3700 1 -- 172.21.15.40:0/3141061239 wait complete. 2024-04-21T23:17:08.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:08.663+0000 7f02e85a3700 1 Processor -- start 2024-04-21T23:17:08.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:08.663+0000 7f02e85a3700 1 -- start start 2024-04-21T23:17:08.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:08.664+0000 7f02e85a3700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02e0069f40 0x7f02e019e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:08.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:08.664+0000 7f02e85a3700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f02e006a8e0 0x7f02e019ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:08.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:08.664+0000 7f02e85a3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02e01082b0 0x7f02e01988d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:08.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:08.664+0000 7f02e85a3700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f02e01125b0 con 0x7f02e0069f40 2024-04-21T23:17:08.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:08.664+0000 7f02e85a3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f02e0112430 con 0x7f02e01082b0 2024-04-21T23:17:08.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:08.664+0000 7f02e85a3700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f02e0112730 con 0x7f02e006a8e0 2024-04-21T23:17:08.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:08.664+0000 7f02e633f700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02e0069f40 0x7f02e019e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:08.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:08.664+0000 7f02e633f700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02e0069f40 0x7f02e019e6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45484/0 (socket says 172.21.15.40:45484) 2024-04-21T23:17:08.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:08.664+0000 7f02e633f700 1 -- 172.21.15.40:0/528653990 learned_addr learned my addr 172.21.15.40:0/528653990 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:17:08.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:08.664+0000 7f02e5b3e700 1 --2- 172.21.15.40:0/528653990 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f02e006a8e0 0x7f02e019ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:08.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:08.664+0000 7f02e633f700 1 -- 172.21.15.40:0/528653990 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f02e006a8e0 msgr2=0x7f02e019ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:08.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:08.664+0000 7f02e633f700 1 --2- 172.21.15.40:0/528653990 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f02e006a8e0 0x7f02e019ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:08.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:08.664+0000 7f02e633f700 1 -- 172.21.15.40:0/528653990 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02e01082b0 msgr2=0x7f02e01988d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:08.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:08.664+0000 7f02e6b40700 1 --2- 172.21.15.40:0/528653990 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02e01082b0 0x7f02e01988d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:08.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:08.664+0000 7f02e633f700 1 --2- 172.21.15.40:0/528653990 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02e01082b0 0x7f02e01988d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:08.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:08.664+0000 7f02e633f700 1 -- 172.21.15.40:0/528653990 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02e0199030 con 0x7f02e0069f40 2024-04-21T23:17:08.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:08.664+0000 7f02e6b40700 1 --2- 172.21.15.40:0/528653990 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02e01082b0 0x7f02e01988d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:17:08.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:08.664+0000 7f02e5b3e700 1 --2- 172.21.15.40:0/528653990 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f02e006a8e0 0x7f02e019ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:17:08.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:08.664+0000 7f02e633f700 1 --2- 172.21.15.40:0/528653990 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02e0069f40 0x7f02e019e6c0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f02dc00e9c0 tx=0x7f02dc00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:08.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:08.664+0000 7f02d77fe700 1 -- 172.21.15.40:0/528653990 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02dc00cbd0 con 0x7f02e0069f40 2024-04-21T23:17:08.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:08.665+0000 7f02e85a3700 1 -- 172.21.15.40:0/528653990 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02e0199320 con 0x7f02e0069f40 2024-04-21T23:17:08.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:08.665+0000 7f02d77fe700 1 -- 172.21.15.40:0/528653990 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02dc00cd70 con 0x7f02e0069f40 2024-04-21T23:17:08.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:08.665+0000 7f02e85a3700 1 -- 172.21.15.40:0/528653990 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02e01a5590 con 0x7f02e0069f40 2024-04-21T23:17:08.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:08.665+0000 7f02d77fe700 1 -- 172.21.15.40:0/528653990 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02dc010430 con 0x7f02e0069f40 2024-04-21T23:17:08.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:08.666+0000 7f02e85a3700 1 -- 172.21.15.40:0/528653990 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02e0103960 con 0x7f02e0069f40 2024-04-21T23:17:08.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:08.667+0000 7f02d77fe700 1 -- 172.21.15.40:0/528653990 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f02dc004750 con 0x7f02e0069f40 2024-04-21T23:17:08.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:08.668+0000 7f02d77fe700 1 --2- 172.21.15.40:0/528653990 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f02cc0a2cb0 0x7f02cc0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:08.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:08.668+0000 7f02d77fe700 1 -- 172.21.15.40:0/528653990 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f02dc0cf7e0 con 0x7f02e0069f40 2024-04-21T23:17:08.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:08.668+0000 7f02e5b3e700 1 --2- 172.21.15.40:0/528653990 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f02cc0a2cb0 0x7f02cc0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:08.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:08.669+0000 7f02e5b3e700 1 --2- 172.21.15.40:0/528653990 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f02cc0a2cb0 0x7f02cc0a5170 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f02d000b2e0 tx=0x7f02d0005960 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:08.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:08.674+0000 7f02d77fe700 1 -- 172.21.15.40:0/528653990 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f02dc09c280 con 0x7f02e0069f40 2024-04-21T23:17:08.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:08.883+0000 7f02e85a3700 1 -- 172.21.15.40:0/528653990 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7f02e0199db0 con 0x7f02e0069f40 2024-04-21T23:17:08.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:08.883+0000 7f02d77fe700 1 -- 172.21.15.40:0/528653990 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f02dc09bb80 con 0x7f02e0069f40 2024-04-21T23:17:08.883 INFO:teuthology.orchestra.run.smithi040.stdout:347892351004 2024-04-21T23:17:08.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:08.885+0000 7f02e85a3700 1 -- 172.21.15.40:0/528653990 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f02cc0a2cb0 msgr2=0x7f02cc0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:08.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:08.885+0000 7f02e85a3700 1 --2- 172.21.15.40:0/528653990 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f02cc0a2cb0 0x7f02cc0a5170 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f02d000b2e0 tx=0x7f02d0005960 comp rx=0 tx=0).stop 2024-04-21T23:17:08.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:08.885+0000 7f02e85a3700 1 -- 172.21.15.40:0/528653990 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02e0069f40 msgr2=0x7f02e019e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:08.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:08.885+0000 7f02e85a3700 1 --2- 172.21.15.40:0/528653990 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02e0069f40 0x7f02e019e6c0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f02dc00e9c0 tx=0x7f02dc00ecd0 comp rx=0 tx=0).stop 2024-04-21T23:17:08.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:08.886+0000 7f02e85a3700 1 -- 172.21.15.40:0/528653990 shutdown_connections 2024-04-21T23:17:08.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:08.886+0000 7f02e85a3700 1 --2- 172.21.15.40:0/528653990 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f02cc0a2cb0 0x7f02cc0a5170 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:08.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:08.886+0000 7f02e85a3700 1 --2- 172.21.15.40:0/528653990 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02e01082b0 0x7f02e01988d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:08.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:08.886+0000 7f02e85a3700 1 --2- 172.21.15.40:0/528653990 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02e0069f40 0x7f02e019e6c0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:08.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:08.886+0000 7f02e85a3700 1 --2- 172.21.15.40:0/528653990 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f02e006a8e0 0x7f02e019ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:08.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:08.886+0000 7f02e85a3700 1 -- 172.21.15.40:0/528653990 >> 172.21.15.40:0/528653990 conn(0x7f02e00780e0 msgr2=0x7f02e00ffd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:08.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:08.886+0000 7f02e85a3700 1 -- 172.21.15.40:0/528653990 shutdown_connections 2024-04-21T23:17:08.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:08.886+0000 7f02e85a3700 1 -- 172.21.15.40:0/528653990 wait complete. 2024-04-21T23:17:09.817 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:09 smithi040 ceph-mon[27984]: pgmap v331: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-21T23:17:09.817 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:09 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/528653990' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-21T23:17:09.845 INFO:teuthology.orchestra.run.smithi040.stdout:197568495669 2024-04-21T23:17:09.845 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-21T23:17:09.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:09 smithi174 ceph-mon[31409]: pgmap v331: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-21T23:17:09.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:09 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/528653990' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-21T23:17:09.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:09 smithi111 ceph-mon[31085]: pgmap v331: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-21T23:17:09.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:09 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/528653990' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-21T23:17:11.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:11 smithi040 ceph-mon[27984]: pgmap v332: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:17:11.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:11 smithi174 ceph-mon[31409]: pgmap v332: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:17:11.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:11 smithi111 ceph-mon[31085]: pgmap v332: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:17:12.541 INFO:tasks.cephadm.ceph_manager.ceph:need seq 347892350999 got 347892351004 for osd.11 2024-04-21T23:17:12.541 DEBUG:teuthology.parallel:result is None 2024-04-21T23:17:13.891 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:13 smithi040 ceph-mon[27984]: pgmap v333: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-21T23:17:13.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:13 smithi174 ceph-mon[31409]: pgmap v333: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-21T23:17:13.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:13 smithi111 ceph-mon[31085]: pgmap v333: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-21T23:17:14.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.113+0000 7f72e3d06700 1 -- 172.21.15.40:0/3272487488 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72dc100820 msgr2=0x7f72dc100ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:14.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.113+0000 7f72e3d06700 1 --2- 172.21.15.40:0/3272487488 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72dc100820 0x7f72dc100ca0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f72cc009990 tx=0x7f72cc009ca0 comp rx=0 tx=0).stop 2024-04-21T23:17:14.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.113+0000 7f72e3d06700 1 -- 172.21.15.40:0/3272487488 shutdown_connections 2024-04-21T23:17:14.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.113+0000 7f72e3d06700 1 --2- 172.21.15.40:0/3272487488 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f72dc103140 0x7f72dc103520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:14.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.113+0000 7f72e3d06700 1 --2- 172.21.15.40:0/3272487488 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72dc1012c0 0x7f72dc110fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:14.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.113+0000 7f72e3d06700 1 --2- 172.21.15.40:0/3272487488 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72dc100820 0x7f72dc100ca0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:14.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.113+0000 7f72e3d06700 1 -- 172.21.15.40:0/3272487488 >> 172.21.15.40:0/3272487488 conn(0x7f72dc0777a0 msgr2=0x7f72dc0feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:14.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.113+0000 7f72e3d06700 1 -- 172.21.15.40:0/3272487488 shutdown_connections 2024-04-21T23:17:14.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.113+0000 7f72e3d06700 1 -- 172.21.15.40:0/3272487488 wait complete. 2024-04-21T23:17:14.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.114+0000 7f72e3d06700 1 Processor -- start 2024-04-21T23:17:14.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.114+0000 7f72e3d06700 1 -- start start 2024-04-21T23:17:14.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.114+0000 7f72e3d06700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72dc100820 0x7f72dc1a0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:14.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.114+0000 7f72e3d06700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f72dc1012c0 0x7f72dc1a0d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:14.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.114+0000 7f72e3d06700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72dc103140 0x7f72dc19aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:14.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.114+0000 7f72e3d06700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f72dc114750 con 0x7f72dc100820 2024-04-21T23:17:14.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.114+0000 7f72e3d06700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f72dc1145d0 con 0x7f72dc103140 2024-04-21T23:17:14.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.114+0000 7f72e3d06700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f72dc1148d0 con 0x7f72dc1012c0 2024-04-21T23:17:14.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.114+0000 7f72e1aa2700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72dc100820 0x7f72dc1a0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:14.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.114+0000 7f72e1aa2700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72dc100820 0x7f72dc1a0850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59440/0 (socket says 172.21.15.40:59440) 2024-04-21T23:17:14.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.114+0000 7f72e1aa2700 1 -- 172.21.15.40:0/1036459290 learned_addr learned my addr 172.21.15.40:0/1036459290 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:17:14.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.115+0000 7f72e22a3700 1 --2- 172.21.15.40:0/1036459290 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72dc103140 0x7f72dc19aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:14.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.115+0000 7f72e12a1700 1 --2- 172.21.15.40:0/1036459290 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f72dc1012c0 0x7f72dc1a0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:14.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.115+0000 7f72e1aa2700 1 -- 172.21.15.40:0/1036459290 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f72dc1012c0 msgr2=0x7f72dc1a0d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:14.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.115+0000 7f72e1aa2700 1 --2- 172.21.15.40:0/1036459290 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f72dc1012c0 0x7f72dc1a0d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:14.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.115+0000 7f72e1aa2700 1 -- 172.21.15.40:0/1036459290 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72dc103140 msgr2=0x7f72dc19aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:14.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.115+0000 7f72e1aa2700 1 --2- 172.21.15.40:0/1036459290 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72dc103140 0x7f72dc19aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:14.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.115+0000 7f72e1aa2700 1 -- 172.21.15.40:0/1036459290 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72dc19b170 con 0x7f72dc100820 2024-04-21T23:17:14.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.115+0000 7f72e22a3700 1 --2- 172.21.15.40:0/1036459290 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72dc103140 0x7f72dc19aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:17:14.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.115+0000 7f72e1aa2700 1 --2- 172.21.15.40:0/1036459290 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72dc100820 0x7f72dc1a0850 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f72d800d980 tx=0x7f72d800dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:14.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.115+0000 7f72d2ffd700 1 -- 172.21.15.40:0/1036459290 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72d8017070 con 0x7f72dc100820 2024-04-21T23:17:14.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.115+0000 7f72e3d06700 1 -- 172.21.15.40:0/1036459290 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72dc19b460 con 0x7f72dc100820 2024-04-21T23:17:14.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.115+0000 7f72d2ffd700 1 -- 172.21.15.40:0/1036459290 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72d8004d10 con 0x7f72dc100820 2024-04-21T23:17:14.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.115+0000 7f72d2ffd700 1 -- 172.21.15.40:0/1036459290 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72d80055c0 con 0x7f72dc100820 2024-04-21T23:17:14.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.115+0000 7f72e3d06700 1 -- 172.21.15.40:0/1036459290 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72dc1a7720 con 0x7f72dc100820 2024-04-21T23:17:14.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.116+0000 7f72e3d06700 1 -- 172.21.15.40:0/1036459290 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72dc101830 con 0x7f72dc100820 2024-04-21T23:17:14.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.117+0000 7f72d2ffd700 1 -- 172.21.15.40:0/1036459290 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f72d8004750 con 0x7f72dc100820 2024-04-21T23:17:14.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.118+0000 7f72d2ffd700 1 --2- 172.21.15.40:0/1036459290 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f72c80a2cb0 0x7f72c80a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:14.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.118+0000 7f72e12a1700 1 --2- 172.21.15.40:0/1036459290 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f72c80a2cb0 0x7f72c80a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:14.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.118+0000 7f72d2ffd700 1 -- 172.21.15.40:0/1036459290 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f72d80cf460 con 0x7f72dc100820 2024-04-21T23:17:14.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.118+0000 7f72e12a1700 1 --2- 172.21.15.40:0/1036459290 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f72c80a2cb0 0x7f72c80a5170 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f72cc006010 tx=0x7f72cc00b540 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:14.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.122+0000 7f72d2ffd700 1 -- 172.21.15.40:0/1036459290 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f72d8020020 con 0x7f72dc100820 2024-04-21T23:17:14.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.334+0000 7f72e3d06700 1 -- 172.21.15.40:0/1036459290 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7f72dc02db30 con 0x7f72dc100820 2024-04-21T23:17:14.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.334+0000 7f72d2ffd700 1 -- 172.21.15.40:0/1036459290 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f72d809bf00 con 0x7f72dc100820 2024-04-21T23:17:14.334 INFO:teuthology.orchestra.run.smithi040.stdout:313532612642 2024-04-21T23:17:14.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.336+0000 7f72e3d06700 1 -- 172.21.15.40:0/1036459290 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f72c80a2cb0 msgr2=0x7f72c80a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:14.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.336+0000 7f72e3d06700 1 --2- 172.21.15.40:0/1036459290 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f72c80a2cb0 0x7f72c80a5170 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f72cc006010 tx=0x7f72cc00b540 comp rx=0 tx=0).stop 2024-04-21T23:17:14.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.336+0000 7f72e3d06700 1 -- 172.21.15.40:0/1036459290 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72dc100820 msgr2=0x7f72dc1a0850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:14.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.336+0000 7f72e3d06700 1 --2- 172.21.15.40:0/1036459290 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72dc100820 0x7f72dc1a0850 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f72d800d980 tx=0x7f72d800dd40 comp rx=0 tx=0).stop 2024-04-21T23:17:14.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.336+0000 7f72e3d06700 1 -- 172.21.15.40:0/1036459290 shutdown_connections 2024-04-21T23:17:14.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.337+0000 7f72e3d06700 1 --2- 172.21.15.40:0/1036459290 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f72c80a2cb0 0x7f72c80a5170 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:14.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.337+0000 7f72e3d06700 1 --2- 172.21.15.40:0/1036459290 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72dc103140 0x7f72dc19aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:14.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.337+0000 7f72e3d06700 1 --2- 172.21.15.40:0/1036459290 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72dc100820 0x7f72dc1a0850 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:14.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.337+0000 7f72e3d06700 1 --2- 172.21.15.40:0/1036459290 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f72dc1012c0 0x7f72dc1a0d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:14.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.337+0000 7f72e3d06700 1 -- 172.21.15.40:0/1036459290 >> 172.21.15.40:0/1036459290 conn(0x7f72dc0777a0 msgr2=0x7f72dc0fdfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:14.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.337+0000 7f72e3d06700 1 -- 172.21.15.40:0/1036459290 shutdown_connections 2024-04-21T23:17:14.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.337+0000 7f72e3d06700 1 -- 172.21.15.40:0/1036459290 wait complete. 2024-04-21T23:17:14.803 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:14 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/1036459290' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-21T23:17:14.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:14 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/1036459290' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-21T23:17:14.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:14 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/1036459290' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-21T23:17:15.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:15.183+0000 7f06617a1700 1 -- 172.21.15.40:0/426052551 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f065c101af0 msgr2=0x7f065c10dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:15.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:15.183+0000 7f06617a1700 1 --2- 172.21.15.40:0/426052551 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f065c101af0 0x7f065c10dfe0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f065400b0a0 tx=0x7f065400b3b0 comp rx=0 tx=0).stop 2024-04-21T23:17:15.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:15.183+0000 7f06617a1700 1 -- 172.21.15.40:0/426052551 shutdown_connections 2024-04-21T23:17:15.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:15.183+0000 7f06617a1700 1 --2- 172.21.15.40:0/426052551 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f065c1071a0 0x7f065c107580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:15.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:15.183+0000 7f06617a1700 1 --2- 172.21.15.40:0/426052551 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f065c101af0 0x7f065c10dfe0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:15.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:15.183+0000 7f06617a1700 1 --2- 172.21.15.40:0/426052551 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f065c101130 0x7f065c1015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:15.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:15.183+0000 7f06617a1700 1 -- 172.21.15.40:0/426052551 >> 172.21.15.40:0/426052551 conn(0x7f065c0780e0 msgr2=0x7f065c0784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:15.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:15.183+0000 7f06617a1700 1 -- 172.21.15.40:0/426052551 shutdown_connections 2024-04-21T23:17:15.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:15.183+0000 7f06617a1700 1 -- 172.21.15.40:0/426052551 wait complete. 2024-04-21T23:17:15.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:15.183+0000 7f06617a1700 1 Processor -- start 2024-04-21T23:17:15.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:15.183+0000 7f06617a1700 1 -- start start 2024-04-21T23:17:15.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:15.184+0000 7f06617a1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f065c101130 0x7f065c19e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:15.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:15.184+0000 7f06617a1700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f065c101af0 0x7f065c19eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:15.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:15.184+0000 7f06617a1700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f065c1071a0 0x7f065c1987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:15.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:15.184+0000 7f06617a1700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f065c111430 con 0x7f065c1071a0 2024-04-21T23:17:15.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:15.184+0000 7f06617a1700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f065c1112b0 con 0x7f065c101130 2024-04-21T23:17:15.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:15.184+0000 7f06617a1700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f065c1115b0 con 0x7f065c101af0 2024-04-21T23:17:15.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:15.184+0000 7f065b7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f065c1071a0 0x7f065c1987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:15.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:15.184+0000 7f065b7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f065c1071a0 0x7f065c1987a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59478/0 (socket says 172.21.15.40:59478) 2024-04-21T23:17:15.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:15.184+0000 7f065b7fe700 1 -- 172.21.15.40:0/1056339300 learned_addr learned my addr 172.21.15.40:0/1056339300 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:17:15.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:15.184+0000 7f065a7fc700 1 --2- 172.21.15.40:0/1056339300 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f065c101af0 0x7f065c19eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:15.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:15.184+0000 7f065affd700 1 --2- 172.21.15.40:0/1056339300 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f065c101130 0x7f065c19e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:15.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:15.184+0000 7f065b7fe700 1 -- 172.21.15.40:0/1056339300 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f065c101af0 msgr2=0x7f065c19eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:15.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:15.184+0000 7f065b7fe700 1 --2- 172.21.15.40:0/1056339300 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f065c101af0 0x7f065c19eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:15.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:15.184+0000 7f065b7fe700 1 -- 172.21.15.40:0/1056339300 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f065c101130 msgr2=0x7f065c19e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:15.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:15.184+0000 7f065b7fe700 1 --2- 172.21.15.40:0/1056339300 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f065c101130 0x7f065c19e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:15.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:15.184+0000 7f065b7fe700 1 -- 172.21.15.40:0/1056339300 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f065c198f80 con 0x7f065c1071a0 2024-04-21T23:17:15.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:15.184+0000 7f065affd700 1 --2- 172.21.15.40:0/1056339300 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f065c101130 0x7f065c19e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:17:15.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:15.184+0000 7f065b7fe700 1 --2- 172.21.15.40:0/1056339300 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f065c1071a0 0x7f065c1987a0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f06540062a0 tx=0x7f0654004d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:15.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:15.185+0000 7f0643fff700 1 -- 172.21.15.40:0/1056339300 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0654021070 con 0x7f065c1071a0 2024-04-21T23:17:15.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:15.185+0000 7f0643fff700 1 -- 172.21.15.40:0/1056339300 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0654007c00 con 0x7f065c1071a0 2024-04-21T23:17:15.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:15.185+0000 7f06617a1700 1 -- 172.21.15.40:0/1056339300 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f065c199210 con 0x7f065c1071a0 2024-04-21T23:17:15.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:15.185+0000 7f0643fff700 1 -- 172.21.15.40:0/1056339300 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f065401d450 con 0x7f065c1071a0 2024-04-21T23:17:15.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:15.185+0000 7f06617a1700 1 -- 172.21.15.40:0/1056339300 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f065c102d40 con 0x7f065c1071a0 2024-04-21T23:17:15.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:15.186+0000 7f06617a1700 1 -- 172.21.15.40:0/1056339300 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f065c102850 con 0x7f065c1071a0 2024-04-21T23:17:15.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:15.187+0000 7f0643fff700 1 -- 172.21.15.40:0/1056339300 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f0654004020 con 0x7f065c1071a0 2024-04-21T23:17:15.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:15.188+0000 7f0643fff700 1 --2- 172.21.15.40:0/1056339300 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f06440a2d30 0x7f06440a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:15.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:15.188+0000 7f065affd700 1 --2- 172.21.15.40:0/1056339300 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f06440a2d30 0x7f06440a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:15.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:15.188+0000 7f0643fff700 1 -- 172.21.15.40:0/1056339300 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f06540d0590 con 0x7f065c1071a0 2024-04-21T23:17:15.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:15.189+0000 7f065affd700 1 --2- 172.21.15.40:0/1056339300 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f06440a2d30 0x7f06440a51f0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f064c009b50 tx=0x7f064c009550 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:15.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:15.192+0000 7f0643fff700 1 -- 172.21.15.40:0/1056339300 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f065409d060 con 0x7f065c1071a0 2024-04-21T23:17:15.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:15.402+0000 7f06617a1700 1 -- 172.21.15.40:0/1056339300 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f065c062e40 con 0x7f065c1071a0 2024-04-21T23:17:15.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:15.403+0000 7f0643fff700 1 -- 172.21.15.40:0/1056339300 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f065409c960 con 0x7f065c1071a0 2024-04-21T23:17:15.402 INFO:teuthology.orchestra.run.smithi040.stdout:219043332145 2024-04-21T23:17:15.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:15.405+0000 7f06617a1700 1 -- 172.21.15.40:0/1056339300 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f06440a2d30 msgr2=0x7f06440a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:15.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:15.405+0000 7f06617a1700 1 --2- 172.21.15.40:0/1056339300 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f06440a2d30 0x7f06440a51f0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f064c009b50 tx=0x7f064c009550 comp rx=0 tx=0).stop 2024-04-21T23:17:15.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:15.405+0000 7f06617a1700 1 -- 172.21.15.40:0/1056339300 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f065c1071a0 msgr2=0x7f065c1987a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:15.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:15.405+0000 7f06617a1700 1 --2- 172.21.15.40:0/1056339300 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f065c1071a0 0x7f065c1987a0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f06540062a0 tx=0x7f0654004d70 comp rx=0 tx=0).stop 2024-04-21T23:17:15.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:15.406+0000 7f06617a1700 1 -- 172.21.15.40:0/1056339300 shutdown_connections 2024-04-21T23:17:15.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:15.406+0000 7f06617a1700 1 --2- 172.21.15.40:0/1056339300 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f06440a2d30 0x7f06440a51f0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:15.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:15.406+0000 7f06617a1700 1 --2- 172.21.15.40:0/1056339300 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f065c1071a0 0x7f065c1987a0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:15.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:15.406+0000 7f06617a1700 1 --2- 172.21.15.40:0/1056339300 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f065c101130 0x7f065c19e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:15.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:15.406+0000 7f06617a1700 1 --2- 172.21.15.40:0/1056339300 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f065c101af0 0x7f065c19eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:15.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:15.406+0000 7f06617a1700 1 -- 172.21.15.40:0/1056339300 >> 172.21.15.40:0/1056339300 conn(0x7f065c0780e0 msgr2=0x7f065c100360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:15.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:15.406+0000 7f06617a1700 1 -- 172.21.15.40:0/1056339300 shutdown_connections 2024-04-21T23:17:15.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:15.406+0000 7f06617a1700 1 -- 172.21.15.40:0/1056339300 wait complete. 2024-04-21T23:17:15.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.587+0000 7f5100f1d700 1 -- 172.21.15.40:0/665621431 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50fc101130 msgr2=0x7f50fc1015b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:15.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.587+0000 7f5100f1d700 1 --2- 172.21.15.40:0/665621431 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50fc101130 0x7f50fc1015b0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f50f0009990 tx=0x7f50f0009ca0 comp rx=0 tx=0).stop 2024-04-21T23:17:15.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.587+0000 7f5100f1d700 1 -- 172.21.15.40:0/665621431 shutdown_connections 2024-04-21T23:17:15.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.587+0000 7f5100f1d700 1 --2- 172.21.15.40:0/665621431 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50fc1071a0 0x7f50fc107580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:15.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.587+0000 7f5100f1d700 1 --2- 172.21.15.40:0/665621431 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50fc101af0 0x7f50fc10dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:15.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.587+0000 7f5100f1d700 1 --2- 172.21.15.40:0/665621431 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50fc101130 0x7f50fc1015b0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:15.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.587+0000 7f5100f1d700 1 -- 172.21.15.40:0/665621431 >> 172.21.15.40:0/665621431 conn(0x7f50fc0780e0 msgr2=0x7f50fc0784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:15.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.587+0000 7f5100f1d700 1 -- 172.21.15.40:0/665621431 shutdown_connections 2024-04-21T23:17:15.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.587+0000 7f5100f1d700 1 -- 172.21.15.40:0/665621431 wait complete. 2024-04-21T23:17:15.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.588+0000 7f5100f1d700 1 Processor -- start 2024-04-21T23:17:15.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.588+0000 7f5100f1d700 1 -- start start 2024-04-21T23:17:15.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.588+0000 7f5100f1d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50fc101130 0x7f50fc19e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:15.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.588+0000 7f5100f1d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50fc101af0 0x7f50fc19eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:15.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.588+0000 7f5100f1d700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50fc1071a0 0x7f50fc1987d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:15.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.588+0000 7f5100f1d700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f50fc111430 con 0x7f50fc101130 2024-04-21T23:17:15.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.588+0000 7f5100f1d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f50fc1112b0 con 0x7f50fc101af0 2024-04-21T23:17:15.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.588+0000 7f5100f1d700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f50fc1115b0 con 0x7f50fc1071a0 2024-04-21T23:17:15.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.588+0000 7f50fa59c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50fc101130 0x7f50fc19e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:15.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.588+0000 7f50fa59c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50fc101130 0x7f50fc19e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59458/0 (socket says 172.21.15.40:59458) 2024-04-21T23:17:15.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.588+0000 7f50fa59c700 1 -- 172.21.15.40:0/2833526149 learned_addr learned my addr 172.21.15.40:0/2833526149 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:17:15.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.588+0000 7f50f9d9b700 1 --2- 172.21.15.40:0/2833526149 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50fc101af0 0x7f50fc19eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:15.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.588+0000 7f50fad9d700 1 --2- 172.21.15.40:0/2833526149 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50fc1071a0 0x7f50fc1987d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:15.526 INFO:teuthology.orchestra.run.smithi040.stdout:55834574928 2024-04-21T23:17:15.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.589+0000 7f50fa59c700 1 -- 172.21.15.40:0/2833526149 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50fc1071a0 msgr2=0x7f50fc1987d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:15.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.589+0000 7f50fa59c700 1 --2- 172.21.15.40:0/2833526149 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50fc1071a0 0x7f50fc1987d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:15.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.589+0000 7f50fa59c700 1 -- 172.21.15.40:0/2833526149 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50fc101af0 msgr2=0x7f50fc19eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:15.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.589+0000 7f50fa59c700 1 --2- 172.21.15.40:0/2833526149 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50fc101af0 0x7f50fc19eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:15.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.589+0000 7f50fa59c700 1 -- 172.21.15.40:0/2833526149 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50fc198fb0 con 0x7f50fc101130 2024-04-21T23:17:15.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.589+0000 7f50fa59c700 1 --2- 172.21.15.40:0/2833526149 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50fc101130 0x7f50fc19e650 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f50ec00c970 tx=0x7f50ec00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:15.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.589+0000 7f50eb7fe700 1 -- 172.21.15.40:0/2833526149 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50ec0078d0 con 0x7f50fc101130 2024-04-21T23:17:15.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.589+0000 7f5100f1d700 1 -- 172.21.15.40:0/2833526149 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50fc1992a0 con 0x7f50fc101130 2024-04-21T23:17:15.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.589+0000 7f50eb7fe700 1 -- 172.21.15.40:0/2833526149 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50ec018480 con 0x7f50fc101130 2024-04-21T23:17:15.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.589+0000 7f50eb7fe700 1 -- 172.21.15.40:0/2833526149 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50ec00f6b0 con 0x7f50fc101130 2024-04-21T23:17:15.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.589+0000 7f5100f1d700 1 -- 172.21.15.40:0/2833526149 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50fc1a54b0 con 0x7f50fc101130 2024-04-21T23:17:15.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.590+0000 7f5100f1d700 1 -- 172.21.15.40:0/2833526149 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50fc102850 con 0x7f50fc101130 2024-04-21T23:17:15.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.591+0000 7f50eb7fe700 1 -- 172.21.15.40:0/2833526149 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f50ec018630 con 0x7f50fc101130 2024-04-21T23:17:15.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.592+0000 7f50eb7fe700 1 --2- 172.21.15.40:0/2833526149 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f50e40a2cb0 0x7f50e40a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:15.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.592+0000 7f50f9d9b700 1 --2- 172.21.15.40:0/2833526149 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f50e40a2cb0 0x7f50e40a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:15.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.592+0000 7f50eb7fe700 1 -- 172.21.15.40:0/2833526149 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f50ec0cf870 con 0x7f50fc101130 2024-04-21T23:17:15.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.593+0000 7f50f9d9b700 1 --2- 172.21.15.40:0/2833526149 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f50e40a2cb0 0x7f50e40a5170 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f50f000b2e0 tx=0x7f50f0005960 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:15.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.596+0000 7f50eb7fe700 1 -- 172.21.15.40:0/2833526149 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f50ec09c3d0 con 0x7f50fc101130 2024-04-21T23:17:15.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.803+0000 7f5100f1d700 1 -- 172.21.15.40:0/2833526149 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f50fc1a56e0 con 0x7f50fc101130 2024-04-21T23:17:15.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.804+0000 7f50eb7fe700 1 -- 172.21.15.40:0/2833526149 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f50ec09bcd0 con 0x7f50fc101130 2024-04-21T23:17:15.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.806+0000 7f5100f1d700 1 -- 172.21.15.40:0/2833526149 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f50e40a2cb0 msgr2=0x7f50e40a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:15.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.806+0000 7f5100f1d700 1 --2- 172.21.15.40:0/2833526149 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f50e40a2cb0 0x7f50e40a5170 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f50f000b2e0 tx=0x7f50f0005960 comp rx=0 tx=0).stop 2024-04-21T23:17:15.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.806+0000 7f5100f1d700 1 -- 172.21.15.40:0/2833526149 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50fc101130 msgr2=0x7f50fc19e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:15.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.806+0000 7f5100f1d700 1 --2- 172.21.15.40:0/2833526149 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50fc101130 0x7f50fc19e650 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f50ec00c970 tx=0x7f50ec00cc80 comp rx=0 tx=0).stop 2024-04-21T23:17:15.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.806+0000 7f5100f1d700 1 -- 172.21.15.40:0/2833526149 shutdown_connections 2024-04-21T23:17:15.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.806+0000 7f5100f1d700 1 --2- 172.21.15.40:0/2833526149 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50fc1071a0 0x7f50fc1987d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:15.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.806+0000 7f5100f1d700 1 --2- 172.21.15.40:0/2833526149 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f50e40a2cb0 0x7f50e40a5170 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:15.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.806+0000 7f5100f1d700 1 --2- 172.21.15.40:0/2833526149 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50fc101130 0x7f50fc19e650 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:15.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.806+0000 7f5100f1d700 1 --2- 172.21.15.40:0/2833526149 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50fc101af0 0x7f50fc19eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:15.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.806+0000 7f5100f1d700 1 -- 172.21.15.40:0/2833526149 >> 172.21.15.40:0/2833526149 conn(0x7f50fc0780e0 msgr2=0x7f50fc1003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:15.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.806+0000 7f5100f1d700 1 -- 172.21.15.40:0/2833526149 shutdown_connections 2024-04-21T23:17:15.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:14.806+0000 7f5100f1d700 1 -- 172.21.15.40:0/2833526149 wait complete. 2024-04-21T23:17:15.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:15 smithi040 ceph-mon[27984]: pgmap v334: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T23:17:15.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:15 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/2833526149' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-21T23:17:15.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:15 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/1056339300' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-21T23:17:15.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:15 smithi174 ceph-mon[31409]: pgmap v334: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T23:17:15.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:15 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/2833526149' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-21T23:17:15.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:15 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/1056339300' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-21T23:17:15.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:15 smithi111 ceph-mon[31085]: pgmap v334: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T23:17:15.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:15 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/2833526149' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-21T23:17:15.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:15 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/1056339300' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-21T23:17:16.194 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574925 got 55834574928 for osd.1 2024-04-21T23:17:16.194 DEBUG:teuthology.parallel:result is None 2024-04-21T23:17:16.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:16.376+0000 7fb299a88700 1 -- 172.21.15.40:0/933275346 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb294100820 msgr2=0x7fb294100c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:16.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:16.376+0000 7fb299a88700 1 --2- 172.21.15.40:0/933275346 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb294100820 0x7fb294100c80 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fb27c009990 tx=0x7fb27c009ca0 comp rx=0 tx=0).stop 2024-04-21T23:17:16.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:16.376+0000 7fb299a88700 1 -- 172.21.15.40:0/933275346 shutdown_connections 2024-04-21T23:17:16.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:16.376+0000 7fb299a88700 1 --2- 172.21.15.40:0/933275346 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb2941082c0 0x7fb2941086a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:16.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:16.376+0000 7fb299a88700 1 --2- 172.21.15.40:0/933275346 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2941011c0 0x7fb29410f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:16.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:16.376+0000 7fb299a88700 1 --2- 172.21.15.40:0/933275346 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb294100820 0x7fb294100c80 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:16.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:16.376+0000 7fb299a88700 1 -- 172.21.15.40:0/933275346 >> 172.21.15.40:0/933275346 conn(0x7fb2940777a0 msgr2=0x7fb2940feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:16.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:16.376+0000 7fb299a88700 1 -- 172.21.15.40:0/933275346 shutdown_connections 2024-04-21T23:17:16.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:16.376+0000 7fb299a88700 1 -- 172.21.15.40:0/933275346 wait complete. 2024-04-21T23:17:16.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:16.377+0000 7fb299a88700 1 Processor -- start 2024-04-21T23:17:16.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:16.377+0000 7fb299a88700 1 -- start start 2024-04-21T23:17:16.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:16.377+0000 7fb299a88700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb294100820 0x7fb29419e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:16.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:16.377+0000 7fb299a88700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2941011c0 0x7fb29419eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:16.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:16.377+0000 7fb299a88700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2941082c0 0x7fb2941987e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:16.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:16.377+0000 7fb299a88700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb294112520 con 0x7fb2941082c0 2024-04-21T23:17:16.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:16.377+0000 7fb299a88700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb2941123a0 con 0x7fb2941011c0 2024-04-21T23:17:16.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:16.377+0000 7fb299a88700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb2941126a0 con 0x7fb294100820 2024-04-21T23:17:16.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:16.378+0000 7fb292ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2941011c0 0x7fb29419eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:16.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:16.378+0000 7fb2937fe700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb294100820 0x7fb29419e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:16.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:16.378+0000 7fb292ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2941011c0 0x7fb29419eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.40:50028/0 (socket says 172.21.15.40:50028) 2024-04-21T23:17:16.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:16.378+0000 7fb292ffd700 1 -- 172.21.15.40:0/3721555885 learned_addr learned my addr 172.21.15.40:0/3721555885 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:17:16.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:16.378+0000 7fb293fff700 1 --2- 172.21.15.40:0/3721555885 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2941082c0 0x7fb2941987e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:16.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:16.378+0000 7fb292ffd700 1 -- 172.21.15.40:0/3721555885 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb294100820 msgr2=0x7fb29419e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:16.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:16.378+0000 7fb292ffd700 1 --2- 172.21.15.40:0/3721555885 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb294100820 0x7fb29419e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:16.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:16.378+0000 7fb292ffd700 1 -- 172.21.15.40:0/3721555885 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2941082c0 msgr2=0x7fb2941987e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:16.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:16.378+0000 7fb292ffd700 1 --2- 172.21.15.40:0/3721555885 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2941082c0 0x7fb2941987e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:16.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:16.378+0000 7fb292ffd700 1 -- 172.21.15.40:0/3721555885 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb294198fc0 con 0x7fb2941011c0 2024-04-21T23:17:16.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:16.378+0000 7fb292ffd700 1 --2- 172.21.15.40:0/3721555885 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2941011c0 0x7fb29419eb50 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb27c0038b0 tx=0x7fb27c009f50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:16.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:16.379+0000 7fb290ff9700 1 -- 172.21.15.40:0/3721555885 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb27c027070 con 0x7fb2941011c0 2024-04-21T23:17:16.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:16.379+0000 7fb299a88700 1 -- 172.21.15.40:0/3721555885 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb294199250 con 0x7fb2941011c0 2024-04-21T23:17:16.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:16.379+0000 7fb299a88700 1 -- 172.21.15.40:0/3721555885 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2941a5470 con 0x7fb2941011c0 2024-04-21T23:17:16.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:16.379+0000 7fb290ff9700 1 -- 172.21.15.40:0/3721555885 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb27c004050 con 0x7fb2941011c0 2024-04-21T23:17:16.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:16.379+0000 7fb290ff9700 1 -- 172.21.15.40:0/3721555885 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb27c022c00 con 0x7fb2941011c0 2024-04-21T23:17:16.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:16.380+0000 7fb299a88700 1 -- 172.21.15.40:0/3721555885 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb294103b40 con 0x7fb2941011c0 2024-04-21T23:17:16.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:16.381+0000 7fb290ff9700 1 -- 172.21.15.40:0/3721555885 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fb27c02a070 con 0x7fb2941011c0 2024-04-21T23:17:16.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:16.382+0000 7fb290ff9700 1 --2- 172.21.15.40:0/3721555885 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fb2800a2d30 0x7fb2800a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:16.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:16.382+0000 7fb290ff9700 1 -- 172.21.15.40:0/3721555885 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fb27c0d0be0 con 0x7fb2941011c0 2024-04-21T23:17:16.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:16.382+0000 7fb2937fe700 1 --2- 172.21.15.40:0/3721555885 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fb2800a2d30 0x7fb2800a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:16.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:16.382+0000 7fb2937fe700 1 --2- 172.21.15.40:0/3721555885 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fb2800a2d30 0x7fb2800a51f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb2840045b0 tx=0x7fb28400b3c0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:16.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:16.386+0000 7fb290ff9700 1 -- 172.21.15.40:0/3721555885 <== mon.2 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fb27c031020 con 0x7fb2941011c0 2024-04-21T23:17:16.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:16.596+0000 7fb299a88700 1 -- 172.21.15.40:0/3721555885 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7fb294199cc0 con 0x7fb2941011c0 2024-04-21T23:17:16.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:16.597+0000 7fb290ff9700 1 -- 172.21.15.40:0/3721555885 <== mon.2 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fb27c013070 con 0x7fb2941011c0 2024-04-21T23:17:16.596 INFO:teuthology.orchestra.run.smithi040.stdout:287762808872 2024-04-21T23:17:16.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:16.599+0000 7fb299a88700 1 -- 172.21.15.40:0/3721555885 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fb2800a2d30 msgr2=0x7fb2800a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:16.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:16.599+0000 7fb299a88700 1 --2- 172.21.15.40:0/3721555885 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fb2800a2d30 0x7fb2800a51f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb2840045b0 tx=0x7fb28400b3c0 comp rx=0 tx=0).stop 2024-04-21T23:17:16.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:16.599+0000 7fb299a88700 1 -- 172.21.15.40:0/3721555885 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2941011c0 msgr2=0x7fb29419eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:16.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:16.599+0000 7fb299a88700 1 --2- 172.21.15.40:0/3721555885 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2941011c0 0x7fb29419eb50 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb27c0038b0 tx=0x7fb27c009f50 comp rx=0 tx=0).stop 2024-04-21T23:17:16.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:16.599+0000 7fb299a88700 1 -- 172.21.15.40:0/3721555885 shutdown_connections 2024-04-21T23:17:16.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:16.599+0000 7fb299a88700 1 --2- 172.21.15.40:0/3721555885 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb294100820 0x7fb29419e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:16.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:16.599+0000 7fb299a88700 1 --2- 172.21.15.40:0/3721555885 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fb2800a2d30 0x7fb2800a51f0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:16.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:16.599+0000 7fb299a88700 1 --2- 172.21.15.40:0/3721555885 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2941082c0 0x7fb2941987e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:16.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:16.599+0000 7fb299a88700 1 --2- 172.21.15.40:0/3721555885 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2941011c0 0x7fb29419eb50 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:16.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:16.599+0000 7fb299a88700 1 -- 172.21.15.40:0/3721555885 >> 172.21.15.40:0/3721555885 conn(0x7fb2940777a0 msgr2=0x7fb2940feef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:16.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:16.599+0000 7fb299a88700 1 -- 172.21.15.40:0/3721555885 shutdown_connections 2024-04-21T23:17:16.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:16.599+0000 7fb299a88700 1 -- 172.21.15.40:0/3721555885 wait complete. 2024-04-21T23:17:17.817 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:17 smithi040 ceph-mon[27984]: pgmap v335: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T23:17:17.817 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:17 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/3721555885' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-21T23:17:17.851 INFO:tasks.cephadm.ceph_manager.ceph:need seq 313532612638 got 313532612642 for osd.10 2024-04-21T23:17:17.851 DEBUG:teuthology.parallel:result is None 2024-04-21T23:17:17.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:17 smithi174 ceph-mon[31409]: pgmap v335: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T23:17:17.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:17 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/3721555885' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-21T23:17:17.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:17 smithi111 ceph-mon[31085]: pgmap v335: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T23:17:17.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:17 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/3721555885' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-21T23:17:18.390 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:17:18.392 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:17:18.393 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:17:18.394 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:17:19.080 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332140 got 219043332145 for osd.7 2024-04-21T23:17:19.080 DEBUG:teuthology.parallel:result is None 2024-04-21T23:17:19.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:19 smithi040 ceph-mon[27984]: pgmap v336: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:17:19.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:19 smithi174 ceph-mon[31409]: pgmap v336: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:17:19.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:19 smithi111 ceph-mon[31085]: pgmap v336: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:17:20.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:20.726+0000 7fa3ad448700 1 -- 172.21.15.40:0/2529348230 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3a8107570 msgr2=0x7fa3a810f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:20.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:20.726+0000 7fa3ad448700 1 --2- 172.21.15.40:0/2529348230 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3a8107570 0x7fa3a810f140 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fa3a000b0a0 tx=0x7fa3a000b3b0 comp rx=0 tx=0).stop 2024-04-21T23:17:20.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:20.726+0000 7fa3ad448700 1 -- 172.21.15.40:0/2529348230 shutdown_connections 2024-04-21T23:17:20.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:20.726+0000 7fa3ad448700 1 --2- 172.21.15.40:0/2529348230 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa3a81024e0 0x7fa3a81028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:20.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:20.726+0000 7fa3ad448700 1 --2- 172.21.15.40:0/2529348230 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3a8107570 0x7fa3a810f140 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:20.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:20.726+0000 7fa3ad448700 1 --2- 172.21.15.40:0/2529348230 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa3a8102e90 0x7fa3a8106e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:20.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:20.726+0000 7fa3ad448700 1 -- 172.21.15.40:0/2529348230 >> 172.21.15.40:0/2529348230 conn(0x7fa3a8079390 msgr2=0x7fa3a80797a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:20.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:20.726+0000 7fa3ad448700 1 -- 172.21.15.40:0/2529348230 shutdown_connections 2024-04-21T23:17:20.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:20.726+0000 7fa3ad448700 1 -- 172.21.15.40:0/2529348230 wait complete. 2024-04-21T23:17:20.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:20.727+0000 7fa3ad448700 1 Processor -- start 2024-04-21T23:17:20.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:20.727+0000 7fa3ad448700 1 -- start start 2024-04-21T23:17:20.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:20.727+0000 7fa3ad448700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa3a81024e0 0x7fa3a8074530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:20.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:20.727+0000 7fa3ad448700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3a8102e90 0x7fa3a8074a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:20.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:20.727+0000 7fa3ad448700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa3a8107570 0x7fa3a8074fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:20.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:20.727+0000 7fa3ad448700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa3a81125a0 con 0x7fa3a8102e90 2024-04-21T23:17:20.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:20.727+0000 7fa3ad448700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fa3a8112420 con 0x7fa3a8107570 2024-04-21T23:17:20.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:20.727+0000 7fa3ad448700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa3a8112720 con 0x7fa3a81024e0 2024-04-21T23:17:20.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:20.727+0000 7fa3a67fc700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3a8102e90 0x7fa3a8074a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:20.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:20.727+0000 7fa3a67fc700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3a8102e90 0x7fa3a8074a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59520/0 (socket says 172.21.15.40:59520) 2024-04-21T23:17:20.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:20.727+0000 7fa3a67fc700 1 -- 172.21.15.40:0/1586893950 learned_addr learned my addr 172.21.15.40:0/1586893950 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:17:20.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:20.727+0000 7fa3a6ffd700 1 --2- 172.21.15.40:0/1586893950 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa3a81024e0 0x7fa3a8074530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:20.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:20.727+0000 7fa3a77fe700 1 --2- 172.21.15.40:0/1586893950 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa3a8107570 0x7fa3a8074fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:20.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:20.727+0000 7fa3a67fc700 1 -- 172.21.15.40:0/1586893950 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa3a81024e0 msgr2=0x7fa3a8074530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:20.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:20.727+0000 7fa3a67fc700 1 --2- 172.21.15.40:0/1586893950 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa3a81024e0 0x7fa3a8074530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:20.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:20.727+0000 7fa3a67fc700 1 -- 172.21.15.40:0/1586893950 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa3a8107570 msgr2=0x7fa3a8074fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:20.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:20.727+0000 7fa3a67fc700 1 --2- 172.21.15.40:0/1586893950 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa3a8107570 0x7fa3a8074fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:20.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:20.728+0000 7fa3a67fc700 1 -- 172.21.15.40:0/1586893950 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3a81a6f40 con 0x7fa3a8102e90 2024-04-21T23:17:20.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:20.728+0000 7fa3a77fe700 1 --2- 172.21.15.40:0/1586893950 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa3a8107570 0x7fa3a8074fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:17:20.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:20.728+0000 7fa3a67fc700 1 --2- 172.21.15.40:0/1586893950 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3a8102e90 0x7fa3a8074a70 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fa39c00e970 tx=0x7fa39c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:20.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:20.728+0000 7fa38ffff700 1 -- 172.21.15.40:0/1586893950 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa39c00cbd0 con 0x7fa3a8102e90 2024-04-21T23:17:20.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:20.728+0000 7fa38ffff700 1 -- 172.21.15.40:0/1586893950 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa39c00cd70 con 0x7fa3a8102e90 2024-04-21T23:17:20.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:20.728+0000 7fa3ad448700 1 -- 172.21.15.40:0/1586893950 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3a81a7230 con 0x7fa3a8102e90 2024-04-21T23:17:20.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:20.728+0000 7fa38ffff700 1 -- 172.21.15.40:0/1586893950 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa39c0056b0 con 0x7fa3a8102e90 2024-04-21T23:17:20.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:20.728+0000 7fa3ad448700 1 -- 172.21.15.40:0/1586893950 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3a81a7770 con 0x7fa3a8102e90 2024-04-21T23:17:20.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:20.729+0000 7fa3ad448700 1 -- 172.21.15.40:0/1586893950 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3a8068f20 con 0x7fa3a8102e90 2024-04-21T23:17:20.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:20.730+0000 7fa38ffff700 1 -- 172.21.15.40:0/1586893950 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fa39c004750 con 0x7fa3a8102e90 2024-04-21T23:17:20.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:20.731+0000 7fa38ffff700 1 --2- 172.21.15.40:0/1586893950 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fa3900a2cb0 0x7fa3900a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:20.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:20.731+0000 7fa3a6ffd700 1 --2- 172.21.15.40:0/1586893950 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fa3900a2cb0 0x7fa3900a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:20.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:20.731+0000 7fa38ffff700 1 -- 172.21.15.40:0/1586893950 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fa39c0cff80 con 0x7fa3a8102e90 2024-04-21T23:17:20.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:20.731+0000 7fa3a6ffd700 1 --2- 172.21.15.40:0/1586893950 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fa3900a2cb0 0x7fa3900a5170 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fa398004630 tx=0x7fa39800b3c0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:20.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:20.734+0000 7fa38ffff700 1 -- 172.21.15.40:0/1586893950 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fa39c09c9c0 con 0x7fa3a8102e90 2024-04-21T23:17:20.850 INFO:tasks.cephadm.ceph_manager.ceph:need seq 287762808867 got 287762808872 for osd.9 2024-04-21T23:17:20.850 DEBUG:teuthology.parallel:result is None 2024-04-21T23:17:20.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:20.945+0000 7fa3ad448700 1 -- 172.21.15.40:0/1586893950 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fa3a802db30 con 0x7fa3a8102e90 2024-04-21T23:17:20.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:20.945+0000 7fa38ffff700 1 -- 172.21.15.40:0/1586893950 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa39c09c2c0 con 0x7fa3a8102e90 2024-04-21T23:17:20.944 INFO:teuthology.orchestra.run.smithi040.stdout:34359738454 2024-04-21T23:17:20.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:20.947+0000 7fa3ad448700 1 -- 172.21.15.40:0/1586893950 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fa3900a2cb0 msgr2=0x7fa3900a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:20.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:20.947+0000 7fa3ad448700 1 --2- 172.21.15.40:0/1586893950 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fa3900a2cb0 0x7fa3900a5170 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fa398004630 tx=0x7fa39800b3c0 comp rx=0 tx=0).stop 2024-04-21T23:17:20.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:20.947+0000 7fa3ad448700 1 -- 172.21.15.40:0/1586893950 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3a8102e90 msgr2=0x7fa3a8074a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:20.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:20.947+0000 7fa3ad448700 1 --2- 172.21.15.40:0/1586893950 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3a8102e90 0x7fa3a8074a70 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fa39c00e970 tx=0x7fa39c00ec80 comp rx=0 tx=0).stop 2024-04-21T23:17:20.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:20.947+0000 7fa3ad448700 1 -- 172.21.15.40:0/1586893950 shutdown_connections 2024-04-21T23:17:20.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:20.947+0000 7fa3ad448700 1 --2- 172.21.15.40:0/1586893950 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa3a81024e0 0x7fa3a8074530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:20.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:20.947+0000 7fa3ad448700 1 --2- 172.21.15.40:0/1586893950 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fa3900a2cb0 0x7fa3900a5170 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:20.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:20.947+0000 7fa3ad448700 1 --2- 172.21.15.40:0/1586893950 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa3a8107570 0x7fa3a8074fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:20.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:20.947+0000 7fa3ad448700 1 --2- 172.21.15.40:0/1586893950 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3a8102e90 0x7fa3a8074a70 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:20.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:20.947+0000 7fa3ad448700 1 -- 172.21.15.40:0/1586893950 >> 172.21.15.40:0/1586893950 conn(0x7fa3a8079390 msgr2=0x7fa3a8100330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:20.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:20.947+0000 7fa3ad448700 1 -- 172.21.15.40:0/1586893950 shutdown_connections 2024-04-21T23:17:20.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:20.947+0000 7fa3ad448700 1 -- 172.21.15.40:0/1586893950 wait complete. 2024-04-21T23:17:21.472 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:17:21.596 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:21 smithi040 ceph-mon[27984]: pgmap v337: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-04-21T23:17:21.596 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:21 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/1586893950' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-21T23:17:21.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:21 smithi174 ceph-mon[31409]: pgmap v337: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-04-21T23:17:21.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:21 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/1586893950' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-21T23:17:21.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:21 smithi111 ceph-mon[31085]: pgmap v337: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-04-21T23:17:21.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:21 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/1586893950' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-21T23:17:23.867 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:23 smithi040 ceph-mon[27984]: pgmap v338: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-21T23:17:23.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:23 smithi174 ceph-mon[31409]: pgmap v338: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-21T23:17:23.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:23 smithi111 ceph-mon[31085]: pgmap v338: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-21T23:17:25.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:25.772+0000 7fc84efd2700 1 -- 172.21.15.40:0/2016211625 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc848102540 msgr2=0x7fc848102920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:25.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:25.772+0000 7fc84efd2700 1 --2- 172.21.15.40:0/2016211625 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc848102540 0x7fc848102920 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fc83c009990 tx=0x7fc83c009ca0 comp rx=0 tx=0).stop 2024-04-21T23:17:25.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:25.772+0000 7fc84efd2700 1 -- 172.21.15.40:0/2016211625 shutdown_connections 2024-04-21T23:17:25.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:25.772+0000 7fc84efd2700 1 --2- 172.21.15.40:0/2016211625 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc8481075a0 0x7fc84810f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:25.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:25.772+0000 7fc84efd2700 1 --2- 172.21.15.40:0/2016211625 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc848102540 0x7fc848102920 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:25.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:25.772+0000 7fc84efd2700 1 --2- 172.21.15.40:0/2016211625 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc848102ef0 0x7fc848106ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:25.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:25.772+0000 7fc84efd2700 1 -- 172.21.15.40:0/2016211625 >> 172.21.15.40:0/2016211625 conn(0x7fc848079310 msgr2=0x7fc848079720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:25.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:25.772+0000 7fc84efd2700 1 -- 172.21.15.40:0/2016211625 shutdown_connections 2024-04-21T23:17:25.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:25.772+0000 7fc84efd2700 1 -- 172.21.15.40:0/2016211625 wait complete. 2024-04-21T23:17:25.773 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:25 smithi040 ceph-mon[27984]: pgmap v339: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T23:17:25.773 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:25 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/3871303909' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-21T23:17:25.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:25.773+0000 7fc84efd2700 1 Processor -- start 2024-04-21T23:17:25.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:25.773+0000 7fc84efd2700 1 -- start start 2024-04-21T23:17:25.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:25.773+0000 7fc84efd2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc848102540 0x7fc848074530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:25.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:25.773+0000 7fc84efd2700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc848102ef0 0x7fc848074a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:25.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:25.773+0000 7fc84efd2700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc8481075a0 0x7fc848074fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:25.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:25.773+0000 7fc84efd2700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc848112590 con 0x7fc848102ef0 2024-04-21T23:17:25.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:25.773+0000 7fc84efd2700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc848112410 con 0x7fc848102540 2024-04-21T23:17:25.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:25.773+0000 7fc84efd2700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc848112710 con 0x7fc8481075a0 2024-04-21T23:17:25.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:25.773+0000 7fc847fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc848102ef0 0x7fc848074a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:25.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:25.773+0000 7fc84cd6e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc848102540 0x7fc848074530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:25.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:25.773+0000 7fc847fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc848102ef0 0x7fc848074a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41642/0 (socket says 172.21.15.40:41642) 2024-04-21T23:17:25.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:25.773+0000 7fc847fff700 1 -- 172.21.15.40:0/22088737 learned_addr learned my addr 172.21.15.40:0/22088737 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:17:25.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:25.774+0000 7fc84d56f700 1 --2- 172.21.15.40:0/22088737 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc8481075a0 0x7fc848074fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:25.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:25.774+0000 7fc84cd6e700 1 -- 172.21.15.40:0/22088737 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc8481075a0 msgr2=0x7fc848074fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:25.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:25.774+0000 7fc84cd6e700 1 --2- 172.21.15.40:0/22088737 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc8481075a0 0x7fc848074fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:25.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:25.774+0000 7fc84cd6e700 1 -- 172.21.15.40:0/22088737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc848102ef0 msgr2=0x7fc848074a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:25.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:25.774+0000 7fc84cd6e700 1 --2- 172.21.15.40:0/22088737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc848102ef0 0x7fc848074a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:25.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:25.774+0000 7fc84cd6e700 1 -- 172.21.15.40:0/22088737 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8481a6f50 con 0x7fc848102540 2024-04-21T23:17:25.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:25.774+0000 7fc847fff700 1 --2- 172.21.15.40:0/22088737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc848102ef0 0x7fc848074a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:17:25.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:25.774+0000 7fc84cd6e700 1 --2- 172.21.15.40:0/22088737 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc848102540 0x7fc848074530 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc83c0059d0 tx=0x7fc83c0047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:25.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:25.774+0000 7fc845ffb700 1 -- 172.21.15.40:0/22088737 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc83c027070 con 0x7fc848102540 2024-04-21T23:17:25.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:25.774+0000 7fc84efd2700 1 -- 172.21.15.40:0/22088737 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8481a71e0 con 0x7fc848102540 2024-04-21T23:17:25.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:25.774+0000 7fc845ffb700 1 -- 172.21.15.40:0/22088737 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc83c004050 con 0x7fc848102540 2024-04-21T23:17:25.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:25.775+0000 7fc845ffb700 1 -- 172.21.15.40:0/22088737 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc83c022c00 con 0x7fc848102540 2024-04-21T23:17:25.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:25.775+0000 7fc84efd2700 1 -- 172.21.15.40:0/22088737 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8481a7690 con 0x7fc848102540 2024-04-21T23:17:25.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:25.776+0000 7fc84efd2700 1 -- 172.21.15.40:0/22088737 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc848068f20 con 0x7fc848102540 2024-04-21T23:17:25.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:25.777+0000 7fc845ffb700 1 -- 172.21.15.40:0/22088737 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fc83c02a070 con 0x7fc848102540 2024-04-21T23:17:25.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:25.778+0000 7fc845ffb700 1 --2- 172.21.15.40:0/22088737 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fc8380a2cb0 0x7fc8380a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:25.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:25.778+0000 7fc847fff700 1 --2- 172.21.15.40:0/22088737 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fc8380a2cb0 0x7fc8380a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:25.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:25.778+0000 7fc845ffb700 1 -- 172.21.15.40:0/22088737 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fc83c0d0a30 con 0x7fc848102540 2024-04-21T23:17:25.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:25.778+0000 7fc847fff700 1 --2- 172.21.15.40:0/22088737 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fc8380a2cb0 0x7fc8380a5170 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fc848100d90 tx=0x7fc83400b480 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:25.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:25.782+0000 7fc845ffb700 1 -- 172.21.15.40:0/22088737 <== mon.2 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fc83c031070 con 0x7fc848102540 2024-04-21T23:17:25.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:25 smithi174 ceph-mon[31409]: pgmap v339: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T23:17:25.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:25 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/3871303909' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-21T23:17:25.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:25 smithi111 ceph-mon[31085]: pgmap v339: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T23:17:25.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:25 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/3871303909' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-21T23:17:25.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:25.993+0000 7fc84efd2700 1 -- 172.21.15.40:0/22088737 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fc84802db30 con 0x7fc848102540 2024-04-21T23:17:25.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:25.995+0000 7fc845ffb700 1 -- 172.21.15.40:0/22088737 <== mon.2 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fc83c013070 con 0x7fc848102540 2024-04-21T23:17:25.995 INFO:teuthology.orchestra.run.smithi040.stdout:141733920835 2024-04-21T23:17:25.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:25.997+0000 7fc84efd2700 1 -- 172.21.15.40:0/22088737 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fc8380a2cb0 msgr2=0x7fc8380a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:25.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:25.997+0000 7fc84efd2700 1 --2- 172.21.15.40:0/22088737 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fc8380a2cb0 0x7fc8380a5170 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fc848100d90 tx=0x7fc83400b480 comp rx=0 tx=0).stop 2024-04-21T23:17:25.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:25.997+0000 7fc84efd2700 1 -- 172.21.15.40:0/22088737 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc848102540 msgr2=0x7fc848074530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:25.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:25.997+0000 7fc84efd2700 1 --2- 172.21.15.40:0/22088737 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc848102540 0x7fc848074530 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc83c0059d0 tx=0x7fc83c0047b0 comp rx=0 tx=0).stop 2024-04-21T23:17:25.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:25.997+0000 7fc84efd2700 1 -- 172.21.15.40:0/22088737 shutdown_connections 2024-04-21T23:17:25.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:25.997+0000 7fc84efd2700 1 --2- 172.21.15.40:0/22088737 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc8481075a0 0x7fc848074fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:25.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:25.997+0000 7fc84efd2700 1 --2- 172.21.15.40:0/22088737 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fc8380a2cb0 0x7fc8380a5170 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:25.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:25.997+0000 7fc84efd2700 1 --2- 172.21.15.40:0/22088737 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc848102540 0x7fc848074530 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:25.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:25.997+0000 7fc84efd2700 1 --2- 172.21.15.40:0/22088737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc848102ef0 0x7fc848074a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:25.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:25.997+0000 7fc84efd2700 1 -- 172.21.15.40:0/22088737 >> 172.21.15.40:0/22088737 conn(0x7fc848079310 msgr2=0x7fc848101a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:25.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:25.997+0000 7fc84efd2700 1 -- 172.21.15.40:0/22088737 shutdown_connections 2024-04-21T23:17:25.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:25.997+0000 7fc84efd2700 1 -- 172.21.15.40:0/22088737 wait complete. 2024-04-21T23:17:26.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:26 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/22088737' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-21T23:17:26.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:26 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/22088737' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-21T23:17:26.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:26 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/22088737' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-21T23:17:27.068 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920831 got 141733920835 for osd.4 2024-04-21T23:17:27.069 DEBUG:teuthology.parallel:result is None 2024-04-21T23:17:27.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:24.756+0000 7f39a62b6700 1 -- 172.21.15.40:0/2116838019 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39a0101420 msgr2=0x7f39a0101800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:27.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:24.756+0000 7f39a62b6700 1 --2- 172.21.15.40:0/2116838019 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39a0101420 0x7f39a0101800 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f39900099c0 tx=0x7f3990009cd0 comp rx=0 tx=0).stop 2024-04-21T23:17:27.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:24.756+0000 7f39a62b6700 1 -- 172.21.15.40:0/2116838019 shutdown_connections 2024-04-21T23:17:27.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:24.756+0000 7f39a62b6700 1 --2- 172.21.15.40:0/2116838019 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39a0106470 0x7f39a010e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:27.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:24.756+0000 7f39a62b6700 1 --2- 172.21.15.40:0/2116838019 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39a0101420 0x7f39a0101800 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:27.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:24.756+0000 7f39a62b6700 1 --2- 172.21.15.40:0/2116838019 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f39a0101dd0 0x7f39a0105d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:27.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:24.756+0000 7f39a62b6700 1 -- 172.21.15.40:0/2116838019 >> 172.21.15.40:0/2116838019 conn(0x7f39a0077180 msgr2=0x7f39a0077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:27.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:24.756+0000 7f39a62b6700 1 -- 172.21.15.40:0/2116838019 shutdown_connections 2024-04-21T23:17:27.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:24.756+0000 7f39a62b6700 1 -- 172.21.15.40:0/2116838019 wait complete. 2024-04-21T23:17:27.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:24.756+0000 7f39a62b6700 1 Processor -- start 2024-04-21T23:17:27.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:24.756+0000 7f39a62b6700 1 -- start start 2024-04-21T23:17:27.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:24.757+0000 7f39a62b6700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f39a0101420 0x7f39a019e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:27.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:24.757+0000 7f39a62b6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39a0101dd0 0x7f39a019ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:27.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:24.757+0000 7f39a62b6700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39a0106470 0x7f39a0198810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:27.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:24.757+0000 7f39a62b6700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f39a0111470 con 0x7f39a0106470 2024-04-21T23:17:27.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:24.757+0000 7f39a62b6700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f39a01112f0 con 0x7f39a0101dd0 2024-04-21T23:17:27.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:24.757+0000 7f39a62b6700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f39a01115f0 con 0x7f39a0101420 2024-04-21T23:17:27.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:24.757+0000 7f39a4853700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39a0106470 0x7f39a0198810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:27.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:24.757+0000 7f39a4853700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39a0106470 0x7f39a0198810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41624/0 (socket says 172.21.15.40:41624) 2024-04-21T23:17:27.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:24.757+0000 7f39a4853700 1 -- 172.21.15.40:0/3871303909 learned_addr learned my addr 172.21.15.40:0/3871303909 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:17:27.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:24.757+0000 7f399f7fe700 1 --2- 172.21.15.40:0/3871303909 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39a0101dd0 0x7f39a019ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:27.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:24.757+0000 7f399ffff700 1 --2- 172.21.15.40:0/3871303909 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f39a0101420 0x7f39a019e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:27.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:24.757+0000 7f39a4853700 1 -- 172.21.15.40:0/3871303909 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f39a0101420 msgr2=0x7f39a019e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:27.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:24.757+0000 7f39a4853700 1 --2- 172.21.15.40:0/3871303909 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f39a0101420 0x7f39a019e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:27.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:24.757+0000 7f39a4853700 1 -- 172.21.15.40:0/3871303909 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39a0101dd0 msgr2=0x7f39a019ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:27.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:24.757+0000 7f39a4853700 1 --2- 172.21.15.40:0/3871303909 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39a0101dd0 0x7f39a019ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:27.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:24.757+0000 7f39a4853700 1 -- 172.21.15.40:0/3871303909 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39a0198ff0 con 0x7f39a0106470 2024-04-21T23:17:27.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:24.758+0000 7f39a4853700 1 --2- 172.21.15.40:0/3871303909 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39a0106470 0x7f39a0198810 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f3998007d70 tx=0x7f399800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:27.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:24.758+0000 7f399d7fa700 1 -- 172.21.15.40:0/3871303909 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3998016020 con 0x7f39a0106470 2024-04-21T23:17:27.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:24.758+0000 7f39a62b6700 1 -- 172.21.15.40:0/3871303909 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39a01992e0 con 0x7f39a0106470 2024-04-21T23:17:27.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:24.758+0000 7f399d7fa700 1 -- 172.21.15.40:0/3871303909 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f399800ac90 con 0x7f39a0106470 2024-04-21T23:17:27.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:24.758+0000 7f399d7fa700 1 -- 172.21.15.40:0/3871303909 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39980115c0 con 0x7f39a0106470 2024-04-21T23:17:27.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:24.758+0000 7f39a62b6700 1 -- 172.21.15.40:0/3871303909 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39a01a54f0 con 0x7f39a0106470 2024-04-21T23:17:27.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:24.759+0000 7f39a62b6700 1 -- 172.21.15.40:0/3871303909 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39a0102870 con 0x7f39a0106470 2024-04-21T23:17:27.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:24.760+0000 7f399d7fa700 1 -- 172.21.15.40:0/3871303909 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f3998019020 con 0x7f39a0106470 2024-04-21T23:17:27.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:24.761+0000 7f399d7fa700 1 --2- 172.21.15.40:0/3871303909 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f39880a2cb0 0x7f39880a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:27.682 INFO:teuthology.orchestra.run.smithi040.stdout:253403070511 2024-04-21T23:17:27.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:24.761+0000 7f399ffff700 1 --2- 172.21.15.40:0/3871303909 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f39880a2cb0 0x7f39880a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:27.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:24.761+0000 7f399d7fa700 1 -- 172.21.15.40:0/3871303909 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f39980cf7a0 con 0x7f39a0106470 2024-04-21T23:17:27.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:24.761+0000 7f399ffff700 1 --2- 172.21.15.40:0/3871303909 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f39880a2cb0 0x7f39880a5170 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f3990000c00 tx=0x7f3990005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:27.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:24.764+0000 7f399d7fa700 1 -- 172.21.15.40:0/3871303909 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f3998020020 con 0x7f39a0106470 2024-04-21T23:17:27.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:24.980+0000 7f39a62b6700 1 -- 172.21.15.40:0/3871303909 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7f39a002db30 con 0x7f39a0106470 2024-04-21T23:17:27.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:24.980+0000 7f399d7fa700 1 -- 172.21.15.40:0/3871303909 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f399809c2f0 con 0x7f39a0106470 2024-04-21T23:17:27.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:24.982+0000 7f39a62b6700 1 -- 172.21.15.40:0/3871303909 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f39880a2cb0 msgr2=0x7f39880a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:27.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:24.982+0000 7f39a62b6700 1 --2- 172.21.15.40:0/3871303909 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f39880a2cb0 0x7f39880a5170 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f3990000c00 tx=0x7f3990005fb0 comp rx=0 tx=0).stop 2024-04-21T23:17:27.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:24.982+0000 7f39a62b6700 1 -- 172.21.15.40:0/3871303909 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39a0106470 msgr2=0x7f39a0198810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:27.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:24.982+0000 7f39a62b6700 1 --2- 172.21.15.40:0/3871303909 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39a0106470 0x7f39a0198810 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f3998007d70 tx=0x7f399800a340 comp rx=0 tx=0).stop 2024-04-21T23:17:27.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:24.982+0000 7f39a62b6700 1 -- 172.21.15.40:0/3871303909 shutdown_connections 2024-04-21T23:17:27.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:24.982+0000 7f39a62b6700 1 --2- 172.21.15.40:0/3871303909 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f39a0101420 0x7f39a019e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:27.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:24.982+0000 7f39a62b6700 1 --2- 172.21.15.40:0/3871303909 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f39880a2cb0 0x7f39880a5170 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:27.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:24.982+0000 7f39a62b6700 1 --2- 172.21.15.40:0/3871303909 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39a0106470 0x7f39a0198810 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:27.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:24.982+0000 7f39a62b6700 1 --2- 172.21.15.40:0/3871303909 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39a0101dd0 0x7f39a019ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:27.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:24.982+0000 7f39a62b6700 1 -- 172.21.15.40:0/3871303909 >> 172.21.15.40:0/3871303909 conn(0x7f39a0077180 msgr2=0x7f39a01052b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:27.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:24.982+0000 7f39a62b6700 1 -- 172.21.15.40:0/3871303909 shutdown_connections 2024-04-21T23:17:27.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:24.982+0000 7f39a62b6700 1 -- 172.21.15.40:0/3871303909 wait complete. 2024-04-21T23:17:27.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:27 smithi040 ceph-mon[27984]: pgmap v340: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T23:17:27.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:27 smithi174 ceph-mon[31409]: pgmap v340: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T23:17:27.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:27 smithi111 ceph-mon[31085]: pgmap v340: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T23:17:28.336 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070505 got 253403070511 for osd.8 2024-04-21T23:17:28.337 DEBUG:teuthology.parallel:result is None 2024-04-21T23:17:28.756 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738449 got 34359738454 for osd.0 2024-04-21T23:17:28.756 DEBUG:teuthology.parallel:result is None 2024-04-21T23:17:29.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:29 smithi040 ceph-mon[27984]: pgmap v341: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:17:29.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:29 smithi174 ceph-mon[31409]: pgmap v341: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:17:29.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:29 smithi111 ceph-mon[31085]: pgmap v341: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:17:30.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.050+0000 7f3c39aad700 1 -- 172.21.15.40:0/252301653 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c341075a0 msgr2=0x7f3c3410f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:30.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.050+0000 7f3c39aad700 1 --2- 172.21.15.40:0/252301653 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c341075a0 0x7f3c3410f170 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f3c2c00b0a0 tx=0x7f3c2c00b3b0 comp rx=0 tx=0).stop 2024-04-21T23:17:30.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.050+0000 7f3c39aad700 1 -- 172.21.15.40:0/252301653 shutdown_connections 2024-04-21T23:17:30.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.050+0000 7f3c39aad700 1 --2- 172.21.15.40:0/252301653 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c34102540 0x7f3c34102920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:30.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.050+0000 7f3c39aad700 1 --2- 172.21.15.40:0/252301653 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c341075a0 0x7f3c3410f170 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:30.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.050+0000 7f3c39aad700 1 --2- 172.21.15.40:0/252301653 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c34102ef0 0x7f3c34106ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:30.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.050+0000 7f3c39aad700 1 -- 172.21.15.40:0/252301653 >> 172.21.15.40:0/252301653 conn(0x7f3c340792c0 msgr2=0x7f3c340796d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:30.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.050+0000 7f3c39aad700 1 -- 172.21.15.40:0/252301653 shutdown_connections 2024-04-21T23:17:30.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.050+0000 7f3c39aad700 1 -- 172.21.15.40:0/252301653 wait complete. 2024-04-21T23:17:30.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.051+0000 7f3c39aad700 1 Processor -- start 2024-04-21T23:17:30.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.051+0000 7f3c39aad700 1 -- start start 2024-04-21T23:17:30.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.051+0000 7f3c39aad700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c34102540 0x7f3c341a0740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:30.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.051+0000 7f3c39aad700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c34102ef0 0x7f3c341a0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:30.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.051+0000 7f3c39aad700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c341075a0 0x7f3c3419a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:30.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.051+0000 7f3c39aad700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3c34112580 con 0x7f3c34102ef0 2024-04-21T23:17:30.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.051+0000 7f3c39aad700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3c34112400 con 0x7f3c34102540 2024-04-21T23:17:30.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.051+0000 7f3c39aad700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3c34112700 con 0x7f3c341075a0 2024-04-21T23:17:30.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.051+0000 7f3c32ffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c34102ef0 0x7f3c341a0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:30.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.051+0000 7f3c32ffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c34102ef0 0x7f3c341a0c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41650/0 (socket says 172.21.15.40:41650) 2024-04-21T23:17:30.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.051+0000 7f3c32ffd700 1 -- 172.21.15.40:0/2731110732 learned_addr learned my addr 172.21.15.40:0/2731110732 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:17:30.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.052+0000 7f3c33fff700 1 --2- 172.21.15.40:0/2731110732 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c341075a0 0x7f3c3419a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:30.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.052+0000 7f3c337fe700 1 --2- 172.21.15.40:0/2731110732 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c34102540 0x7f3c341a0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:30.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.052+0000 7f3c32ffd700 1 -- 172.21.15.40:0/2731110732 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c341075a0 msgr2=0x7f3c3419a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:30.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.052+0000 7f3c32ffd700 1 --2- 172.21.15.40:0/2731110732 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c341075a0 0x7f3c3419a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:30.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.052+0000 7f3c32ffd700 1 -- 172.21.15.40:0/2731110732 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c34102540 msgr2=0x7f3c341a0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:30.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.052+0000 7f3c32ffd700 1 --2- 172.21.15.40:0/2731110732 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c34102540 0x7f3c341a0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:30.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.052+0000 7f3c32ffd700 1 -- 172.21.15.40:0/2731110732 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c3419b060 con 0x7f3c34102ef0 2024-04-21T23:17:30.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.052+0000 7f3c32ffd700 1 --2- 172.21.15.40:0/2731110732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c34102ef0 0x7f3c341a0c80 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f3c2400c9e0 tx=0x7f3c2400ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:30.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.052+0000 7f3c30ff9700 1 -- 172.21.15.40:0/2731110732 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c24007aa0 con 0x7f3c34102ef0 2024-04-21T23:17:30.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.052+0000 7f3c30ff9700 1 -- 172.21.15.40:0/2731110732 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c24007c40 con 0x7f3c34102ef0 2024-04-21T23:17:30.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.052+0000 7f3c39aad700 1 -- 172.21.15.40:0/2731110732 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c3419b350 con 0x7f3c34102ef0 2024-04-21T23:17:30.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.052+0000 7f3c30ff9700 1 -- 172.21.15.40:0/2731110732 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c24018660 con 0x7f3c34102ef0 2024-04-21T23:17:30.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.052+0000 7f3c39aad700 1 -- 172.21.15.40:0/2731110732 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c341a7610 con 0x7f3c34102ef0 2024-04-21T23:17:30.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.053+0000 7f3c39aad700 1 -- 172.21.15.40:0/2731110732 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c34068f20 con 0x7f3c34102ef0 2024-04-21T23:17:30.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.055+0000 7f3c30ff9700 1 -- 172.21.15.40:0/2731110732 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f3c2400f450 con 0x7f3c34102ef0 2024-04-21T23:17:30.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.055+0000 7f3c30ff9700 1 --2- 172.21.15.40:0/2731110732 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f3c1c0a2d30 0x7f3c1c0a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:30.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.056+0000 7f3c337fe700 1 --2- 172.21.15.40:0/2731110732 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f3c1c0a2d30 0x7f3c1c0a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:30.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.056+0000 7f3c30ff9700 1 -- 172.21.15.40:0/2731110732 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f3c240cf760 con 0x7f3c34102ef0 2024-04-21T23:17:30.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.056+0000 7f3c337fe700 1 --2- 172.21.15.40:0/2731110732 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f3c1c0a2d30 0x7f3c1c0a51f0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f3c3419ba40 tx=0x7f3c2800b3c0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:30.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.063+0000 7f3c30ff9700 1 -- 172.21.15.40:0/2731110732 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f3c24014030 con 0x7f3c34102ef0 2024-04-21T23:17:30.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.272+0000 7f3c39aad700 1 -- 172.21.15.40:0/2731110732 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f3c3419be30 con 0x7f3c34102ef0 2024-04-21T23:17:30.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.272+0000 7f3c30ff9700 1 -- 172.21.15.40:0/2731110732 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f3c3419be30 con 0x7f3c34102ef0 2024-04-21T23:17:30.271 INFO:teuthology.orchestra.run.smithi040.stdout:163208757310 2024-04-21T23:17:30.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.275+0000 7f3c39aad700 1 -- 172.21.15.40:0/2731110732 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f3c1c0a2d30 msgr2=0x7f3c1c0a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:30.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.275+0000 7f3c39aad700 1 --2- 172.21.15.40:0/2731110732 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f3c1c0a2d30 0x7f3c1c0a51f0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f3c3419ba40 tx=0x7f3c2800b3c0 comp rx=0 tx=0).stop 2024-04-21T23:17:30.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.275+0000 7f3c39aad700 1 -- 172.21.15.40:0/2731110732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c34102ef0 msgr2=0x7f3c341a0c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:30.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.275+0000 7f3c39aad700 1 --2- 172.21.15.40:0/2731110732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c34102ef0 0x7f3c341a0c80 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f3c2400c9e0 tx=0x7f3c2400ccf0 comp rx=0 tx=0).stop 2024-04-21T23:17:30.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.275+0000 7f3c39aad700 1 -- 172.21.15.40:0/2731110732 shutdown_connections 2024-04-21T23:17:30.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.275+0000 7f3c39aad700 1 --2- 172.21.15.40:0/2731110732 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c341075a0 0x7f3c3419a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:30.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.275+0000 7f3c39aad700 1 --2- 172.21.15.40:0/2731110732 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f3c1c0a2d30 0x7f3c1c0a51f0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:30.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.275+0000 7f3c39aad700 1 --2- 172.21.15.40:0/2731110732 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c34102540 0x7f3c341a0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:30.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.275+0000 7f3c39aad700 1 --2- 172.21.15.40:0/2731110732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c34102ef0 0x7f3c341a0c80 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:30.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.275+0000 7f3c39aad700 1 -- 172.21.15.40:0/2731110732 >> 172.21.15.40:0/2731110732 conn(0x7f3c340792c0 msgr2=0x7f3c341003d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:30.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.275+0000 7f3c39aad700 1 -- 172.21.15.40:0/2731110732 shutdown_connections 2024-04-21T23:17:30.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.275+0000 7f3c39aad700 1 -- 172.21.15.40:0/2731110732 wait complete. 2024-04-21T23:17:30.681 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:30 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/2731110732' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-21T23:17:30.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.938+0000 7f18d2979700 1 -- 172.21.15.40:0/3316175458 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18cc101130 msgr2=0x7f18cc1015b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:30.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.938+0000 7f18d2979700 1 --2- 172.21.15.40:0/3316175458 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18cc101130 0x7f18cc1015b0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f18b4009990 tx=0x7f18b4009ca0 comp rx=0 tx=0).stop 2024-04-21T23:17:30.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.938+0000 7f18d2979700 1 -- 172.21.15.40:0/3316175458 shutdown_connections 2024-04-21T23:17:30.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.938+0000 7f18d2979700 1 --2- 172.21.15.40:0/3316175458 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f18cc101af0 0x7f18cc10dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:30.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.938+0000 7f18d2979700 1 --2- 172.21.15.40:0/3316175458 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f18cc1071a0 0x7f18cc107580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:30.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.938+0000 7f18d2979700 1 --2- 172.21.15.40:0/3316175458 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18cc101130 0x7f18cc1015b0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:30.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.938+0000 7f18d2979700 1 -- 172.21.15.40:0/3316175458 >> 172.21.15.40:0/3316175458 conn(0x7f18cc0780e0 msgr2=0x7f18cc0784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:30.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.938+0000 7f18d2979700 1 -- 172.21.15.40:0/3316175458 shutdown_connections 2024-04-21T23:17:30.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.938+0000 7f18d2979700 1 -- 172.21.15.40:0/3316175458 wait complete. 2024-04-21T23:17:30.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.939+0000 7f18d2979700 1 Processor -- start 2024-04-21T23:17:30.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.939+0000 7f18d2979700 1 -- start start 2024-04-21T23:17:30.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.939+0000 7f18d2979700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f18cc101130 0x7f18cc102e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:30.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.939+0000 7f18d2979700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18cc101af0 0x7f18cc103350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:30.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.939+0000 7f18d2979700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f18cc1071a0 0x7f18cc103890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:30.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.939+0000 7f18d2979700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f18cc111510 con 0x7f18cc101af0 2024-04-21T23:17:30.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.939+0000 7f18d2979700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f18cc111390 con 0x7f18cc101130 2024-04-21T23:17:30.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.939+0000 7f18d2979700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f18cc111690 con 0x7f18cc1071a0 2024-04-21T23:17:30.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.939+0000 7f18cb7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18cc101af0 0x7f18cc103350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:30.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.939+0000 7f18cb7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18cc101af0 0x7f18cc103350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41672/0 (socket says 172.21.15.40:41672) 2024-04-21T23:17:30.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.939+0000 7f18cb7fe700 1 -- 172.21.15.40:0/135875694 learned_addr learned my addr 172.21.15.40:0/135875694 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:17:30.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.939+0000 7f18d0f16700 1 --2- 172.21.15.40:0/135875694 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f18cc1071a0 0x7f18cc103890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:30.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.940+0000 7f18cbfff700 1 --2- 172.21.15.40:0/135875694 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f18cc101130 0x7f18cc102e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:30.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.940+0000 7f18cb7fe700 1 -- 172.21.15.40:0/135875694 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f18cc1071a0 msgr2=0x7f18cc103890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:30.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.940+0000 7f18cb7fe700 1 --2- 172.21.15.40:0/135875694 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f18cc1071a0 0x7f18cc103890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:30.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.940+0000 7f18cb7fe700 1 -- 172.21.15.40:0/135875694 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f18cc101130 msgr2=0x7f18cc102e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:30.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.940+0000 7f18cb7fe700 1 --2- 172.21.15.40:0/135875694 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f18cc101130 0x7f18cc102e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:30.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.940+0000 7f18cb7fe700 1 -- 172.21.15.40:0/135875694 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18cc1a70c0 con 0x7f18cc101af0 2024-04-21T23:17:30.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.940+0000 7f18cb7fe700 1 --2- 172.21.15.40:0/135875694 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18cc101af0 0x7f18cc103350 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f18b4000c00 tx=0x7f18b4003680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:30.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.940+0000 7f18c97fa700 1 -- 172.21.15.40:0/135875694 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18b4027070 con 0x7f18cc101af0 2024-04-21T23:17:30.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.940+0000 7f18c97fa700 1 -- 172.21.15.40:0/135875694 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18b4004110 con 0x7f18cc101af0 2024-04-21T23:17:30.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.940+0000 7f18d2979700 1 -- 172.21.15.40:0/135875694 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18cc1a7350 con 0x7f18cc101af0 2024-04-21T23:17:30.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.940+0000 7f18c97fa700 1 -- 172.21.15.40:0/135875694 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18b400f460 con 0x7f18cc101af0 2024-04-21T23:17:30.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.940+0000 7f18d2979700 1 -- 172.21.15.40:0/135875694 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18cc1a7800 con 0x7f18cc101af0 2024-04-21T23:17:30.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.941+0000 7f18d2979700 1 -- 172.21.15.40:0/135875694 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18cc068f20 con 0x7f18cc101af0 2024-04-21T23:17:30.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.943+0000 7f18c97fa700 1 -- 172.21.15.40:0/135875694 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f18b402a070 con 0x7f18cc101af0 2024-04-21T23:17:30.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.943+0000 7f18c97fa700 1 --2- 172.21.15.40:0/135875694 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f18b80a2d30 0x7f18b80a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:30.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.943+0000 7f18cbfff700 1 --2- 172.21.15.40:0/135875694 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f18b80a2d30 0x7f18b80a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:30.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.943+0000 7f18c97fa700 1 -- 172.21.15.40:0/135875694 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f18b40d0880 con 0x7f18cc101af0 2024-04-21T23:17:30.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.944+0000 7f18cbfff700 1 --2- 172.21.15.40:0/135875694 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f18b80a2d30 0x7f18b80a51f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f18bc006fd0 tx=0x7f18bc008040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:30.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:30 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/2731110732' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-21T23:17:30.943 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:30 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/2731110732' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-21T23:17:30.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:30.947+0000 7f18c97fa700 1 -- 172.21.15.40:0/135875694 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f18b4031070 con 0x7f18cc101af0 2024-04-21T23:17:31.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:31.156+0000 7f18d2979700 1 -- 172.21.15.40:0/135875694 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f18cc1a7a60 con 0x7f18cc101af0 2024-04-21T23:17:31.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:31.157+0000 7f18c97fa700 1 -- 172.21.15.40:0/135875694 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f18b4013070 con 0x7f18cc101af0 2024-04-21T23:17:31.156 INFO:teuthology.orchestra.run.smithi040.stdout:197568495674 2024-04-21T23:17:31.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:31.159+0000 7f18d2979700 1 -- 172.21.15.40:0/135875694 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f18b80a2d30 msgr2=0x7f18b80a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:31.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:31.159+0000 7f18d2979700 1 --2- 172.21.15.40:0/135875694 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f18b80a2d30 0x7f18b80a51f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f18bc006fd0 tx=0x7f18bc008040 comp rx=0 tx=0).stop 2024-04-21T23:17:31.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:31.159+0000 7f18d2979700 1 -- 172.21.15.40:0/135875694 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18cc101af0 msgr2=0x7f18cc103350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:31.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:31.159+0000 7f18d2979700 1 --2- 172.21.15.40:0/135875694 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18cc101af0 0x7f18cc103350 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f18b4000c00 tx=0x7f18b4003680 comp rx=0 tx=0).stop 2024-04-21T23:17:31.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:31.159+0000 7f18d2979700 1 -- 172.21.15.40:0/135875694 shutdown_connections 2024-04-21T23:17:31.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:31.159+0000 7f18d2979700 1 --2- 172.21.15.40:0/135875694 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f18cc1071a0 0x7f18cc103890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:31.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:31.159+0000 7f18d2979700 1 --2- 172.21.15.40:0/135875694 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f18b80a2d30 0x7f18b80a51f0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:31.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:31.159+0000 7f18d2979700 1 --2- 172.21.15.40:0/135875694 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f18cc101130 0x7f18cc102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:31.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:31.159+0000 7f18d2979700 1 --2- 172.21.15.40:0/135875694 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18cc101af0 0x7f18cc103350 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:31.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:31.159+0000 7f18d2979700 1 -- 172.21.15.40:0/135875694 >> 172.21.15.40:0/135875694 conn(0x7f18cc0780e0 msgr2=0x7f18cc1004f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:31.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:31.159+0000 7f18d2979700 1 -- 172.21.15.40:0/135875694 shutdown_connections 2024-04-21T23:17:31.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:31.159+0000 7f18d2979700 1 -- 172.21.15.40:0/135875694 wait complete. 2024-04-21T23:17:31.795 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:31 smithi040 ceph-mon[27984]: pgmap v342: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:17:31.795 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:31 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/135875694' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-21T23:17:31.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:31 smithi174 ceph-mon[31409]: pgmap v342: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:17:31.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:31 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/135875694' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-21T23:17:31.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:31 smithi111 ceph-mon[31085]: pgmap v342: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:17:31.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:31 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/135875694' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-21T23:17:32.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.114+0000 7f1e6d047700 1 -- 172.21.15.40:0/2366248797 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e681013b0 msgr2=0x7f1e68106ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:32.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.114+0000 7f1e6d047700 1 --2- 172.21.15.40:0/2366248797 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e681013b0 0x7f1e68106ef0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f1e5c009990 tx=0x7f1e5c009ca0 comp rx=0 tx=0).stop 2024-04-21T23:17:32.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.114+0000 7f1e6d047700 1 -- 172.21.15.40:0/2366248797 shutdown_connections 2024-04-21T23:17:32.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.114+0000 7f1e6d047700 1 --2- 172.21.15.40:0/2366248797 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1e68100a00 0x7f1e68100de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:32.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.114+0000 7f1e6d047700 1 --2- 172.21.15.40:0/2366248797 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e68107620 0x7f1e6810f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:32.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.114+0000 7f1e6d047700 1 --2- 172.21.15.40:0/2366248797 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e681013b0 0x7f1e68106ef0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:32.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.114+0000 7f1e6d047700 1 -- 172.21.15.40:0/2366248797 >> 172.21.15.40:0/2366248797 conn(0x7f1e68077730 msgr2=0x7f1e680feab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:32.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.114+0000 7f1e6d047700 1 -- 172.21.15.40:0/2366248797 shutdown_connections 2024-04-21T23:17:32.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.114+0000 7f1e6d047700 1 -- 172.21.15.40:0/2366248797 wait complete. 2024-04-21T23:17:32.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.114+0000 7f1e6d047700 1 Processor -- start 2024-04-21T23:17:32.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.115+0000 7f1e6d047700 1 -- start start 2024-04-21T23:17:32.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.115+0000 7f1e6d047700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e68100a00 0x7f1e6819e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:32.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.115+0000 7f1e6d047700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1e681013b0 0x7f1e6819ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:32.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.115+0000 7f1e6d047700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e68107620 0x7f1e681988e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:32.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.115+0000 7f1e6d047700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1e681125c0 con 0x7f1e68107620 2024-04-21T23:17:32.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.115+0000 7f1e6d047700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f1e68112440 con 0x7f1e68100a00 2024-04-21T23:17:32.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.115+0000 7f1e6d047700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1e68112740 con 0x7f1e681013b0 2024-04-21T23:17:32.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.115+0000 7f1e6759e700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e68107620 0x7f1e681988e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:32.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.115+0000 7f1e66d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e68100a00 0x7f1e6819e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:32.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.115+0000 7f1e66d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e68100a00 0x7f1e6819e720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.40:35552/0 (socket says 172.21.15.40:35552) 2024-04-21T23:17:32.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.115+0000 7f1e66d9d700 1 -- 172.21.15.40:0/2447238287 learned_addr learned my addr 172.21.15.40:0/2447238287 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:17:32.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.115+0000 7f1e6659c700 1 --2- 172.21.15.40:0/2447238287 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1e681013b0 0x7f1e6819ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:32.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.115+0000 7f1e6759e700 1 -- 172.21.15.40:0/2447238287 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1e681013b0 msgr2=0x7f1e6819ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:32.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.115+0000 7f1e6759e700 1 --2- 172.21.15.40:0/2447238287 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1e681013b0 0x7f1e6819ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:32.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.115+0000 7f1e6759e700 1 -- 172.21.15.40:0/2447238287 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e68100a00 msgr2=0x7f1e6819e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:32.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.115+0000 7f1e6759e700 1 --2- 172.21.15.40:0/2447238287 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e68100a00 0x7f1e6819e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:32.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.115+0000 7f1e6759e700 1 -- 172.21.15.40:0/2447238287 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e68199040 con 0x7f1e68107620 2024-04-21T23:17:32.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.115+0000 7f1e66d9d700 1 --2- 172.21.15.40:0/2447238287 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e68100a00 0x7f1e6819e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:17:32.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.116+0000 7f1e6759e700 1 --2- 172.21.15.40:0/2447238287 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e68107620 0x7f1e681988e0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f1e60007f10 tx=0x7f1e6000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:32.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.116+0000 7f1e4ffff700 1 -- 172.21.15.40:0/2447238287 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e60004db0 con 0x7f1e68107620 2024-04-21T23:17:32.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.116+0000 7f1e6d047700 1 -- 172.21.15.40:0/2447238287 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e68199330 con 0x7f1e68107620 2024-04-21T23:17:32.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.116+0000 7f1e4ffff700 1 -- 172.21.15.40:0/2447238287 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e60009e20 con 0x7f1e68107620 2024-04-21T23:17:32.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.116+0000 7f1e4ffff700 1 -- 172.21.15.40:0/2447238287 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e60015400 con 0x7f1e68107620 2024-04-21T23:17:32.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.116+0000 7f1e6d047700 1 -- 172.21.15.40:0/2447238287 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e681a55f0 con 0x7f1e68107620 2024-04-21T23:17:32.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.117+0000 7f1e6d047700 1 -- 172.21.15.40:0/2447238287 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e68068f20 con 0x7f1e68107620 2024-04-21T23:17:32.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.118+0000 7f1e4ffff700 1 -- 172.21.15.40:0/2447238287 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f1e6000db20 con 0x7f1e68107620 2024-04-21T23:17:32.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.119+0000 7f1e4ffff700 1 --2- 172.21.15.40:0/2447238287 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f1e500a2cb0 0x7f1e500a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:32.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.119+0000 7f1e66d9d700 1 --2- 172.21.15.40:0/2447238287 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f1e500a2cb0 0x7f1e500a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:32.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.119+0000 7f1e4ffff700 1 -- 172.21.15.40:0/2447238287 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f1e600ced50 con 0x7f1e68107620 2024-04-21T23:17:32.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.119+0000 7f1e66d9d700 1 --2- 172.21.15.40:0/2447238287 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f1e500a2cb0 0x7f1e500a5170 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f1e58005950 tx=0x7f1e5800a3b0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:32.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.124+0000 7f1e4ffff700 1 -- 172.21.15.40:0/2447238287 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f1e60010070 con 0x7f1e68107620 2024-04-21T23:17:32.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.336+0000 7f1e6d047700 1 -- 172.21.15.40:0/2447238287 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f1e6802db30 con 0x7f1e68107620 2024-04-21T23:17:32.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.337+0000 7f1e4ffff700 1 -- 172.21.15.40:0/2447238287 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f1e6009b7f0 con 0x7f1e68107620 2024-04-21T23:17:32.336 INFO:teuthology.orchestra.run.smithi040.stdout:107374182473 2024-04-21T23:17:32.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.339+0000 7f1e6d047700 1 -- 172.21.15.40:0/2447238287 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f1e500a2cb0 msgr2=0x7f1e500a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:32.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.339+0000 7f1e6d047700 1 --2- 172.21.15.40:0/2447238287 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f1e500a2cb0 0x7f1e500a5170 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f1e58005950 tx=0x7f1e5800a3b0 comp rx=0 tx=0).stop 2024-04-21T23:17:32.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.339+0000 7f1e6d047700 1 -- 172.21.15.40:0/2447238287 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e68107620 msgr2=0x7f1e681988e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:32.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.339+0000 7f1e6d047700 1 --2- 172.21.15.40:0/2447238287 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e68107620 0x7f1e681988e0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f1e60007f10 tx=0x7f1e6000d3b0 comp rx=0 tx=0).stop 2024-04-21T23:17:32.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.339+0000 7f1e6d047700 1 -- 172.21.15.40:0/2447238287 shutdown_connections 2024-04-21T23:17:32.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.339+0000 7f1e6d047700 1 --2- 172.21.15.40:0/2447238287 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f1e500a2cb0 0x7f1e500a5170 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:32.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.339+0000 7f1e6d047700 1 --2- 172.21.15.40:0/2447238287 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e68107620 0x7f1e681988e0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:32.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.339+0000 7f1e6d047700 1 --2- 172.21.15.40:0/2447238287 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e68100a00 0x7f1e6819e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:32.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.339+0000 7f1e6d047700 1 --2- 172.21.15.40:0/2447238287 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1e681013b0 0x7f1e6819ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:32.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.339+0000 7f1e6d047700 1 -- 172.21.15.40:0/2447238287 >> 172.21.15.40:0/2447238287 conn(0x7f1e68077730 msgr2=0x7f1e680feab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:32.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.339+0000 7f1e6d047700 1 -- 172.21.15.40:0/2447238287 shutdown_connections 2024-04-21T23:17:32.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.339+0000 7f1e6d047700 1 -- 172.21.15.40:0/2447238287 wait complete. 2024-04-21T23:17:32.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.388+0000 7f261c675700 1 -- 172.21.15.40:0/4246505516 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2614107150 msgr2=0x7f261406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:32.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.388+0000 7f261c675700 1 --2- 172.21.15.40:0/4246505516 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2614107150 0x7f261406a7d0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f260c00b0a0 tx=0x7f260c00b3b0 comp rx=0 tx=0).stop 2024-04-21T23:17:32.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.388+0000 7f261c675700 1 -- 172.21.15.40:0/4246505516 shutdown_connections 2024-04-21T23:17:32.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.388+0000 7f261c675700 1 --2- 172.21.15.40:0/4246505516 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2614104c60 0x7f26141050f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:32.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.388+0000 7f261c675700 1 --2- 172.21.15.40:0/4246505516 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2614107150 0x7f261406a7d0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:32.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.388+0000 7f261c675700 1 --2- 172.21.15.40:0/4246505516 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f261406ad10 0x7f261406b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:32.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.388+0000 7f261c675700 1 -- 172.21.15.40:0/4246505516 >> 172.21.15.40:0/4246505516 conn(0x7f26140780a0 msgr2=0x7f26140784b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:32.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.388+0000 7f261c675700 1 -- 172.21.15.40:0/4246505516 shutdown_connections 2024-04-21T23:17:32.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.388+0000 7f261c675700 1 -- 172.21.15.40:0/4246505516 wait complete. 2024-04-21T23:17:32.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.389+0000 7f261c675700 1 Processor -- start 2024-04-21T23:17:32.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.389+0000 7f261c675700 1 -- start start 2024-04-21T23:17:32.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.389+0000 7f261c675700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f261406ad10 0x7f261419ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:32.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.389+0000 7f261c675700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2614104c60 0x7f261419f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:32.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.389+0000 7f261c675700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2614107150 0x7f26141a3810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:32.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.389+0000 7f261c675700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2614118af0 con 0x7f261406ad10 2024-04-21T23:17:32.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.389+0000 7f261c675700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f2614118970 con 0x7f2614104c60 2024-04-21T23:17:32.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.389+0000 7f261c675700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2614118c70 con 0x7f2614107150 2024-04-21T23:17:32.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.389+0000 7f261a411700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f261406ad10 0x7f261419ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:32.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.389+0000 7f261a411700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f261406ad10 0x7f261419ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42670/0 (socket says 172.21.15.40:42670) 2024-04-21T23:17:32.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.389+0000 7f261a411700 1 -- 172.21.15.40:0/1235497283 learned_addr learned my addr 172.21.15.40:0/1235497283 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:17:32.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.390+0000 7f261ac12700 1 --2- 172.21.15.40:0/1235497283 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2614107150 0x7f26141a3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:32.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.390+0000 7f2619c10700 1 --2- 172.21.15.40:0/1235497283 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2614104c60 0x7f261419f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:32.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.390+0000 7f261a411700 1 -- 172.21.15.40:0/1235497283 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2614107150 msgr2=0x7f26141a3810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:32.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.390+0000 7f261a411700 1 --2- 172.21.15.40:0/1235497283 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2614107150 0x7f26141a3810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:32.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.390+0000 7f261a411700 1 -- 172.21.15.40:0/1235497283 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2614104c60 msgr2=0x7f261419f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:32.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.390+0000 7f261a411700 1 --2- 172.21.15.40:0/1235497283 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2614104c60 0x7f261419f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:32.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.390+0000 7f261a411700 1 -- 172.21.15.40:0/1235497283 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26141a3f50 con 0x7f261406ad10 2024-04-21T23:17:32.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.390+0000 7f261a411700 1 --2- 172.21.15.40:0/1235497283 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f261406ad10 0x7f261419ef40 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f260c00b070 tx=0x7f260c004e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:32.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.390+0000 7f260b7fe700 1 -- 172.21.15.40:0/1235497283 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f260c021070 con 0x7f261406ad10 2024-04-21T23:17:32.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.390+0000 7f260b7fe700 1 -- 172.21.15.40:0/1235497283 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f260c007b80 con 0x7f261406ad10 2024-04-21T23:17:32.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.390+0000 7f261c675700 1 -- 172.21.15.40:0/1235497283 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26141a41e0 con 0x7f261406ad10 2024-04-21T23:17:32.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.390+0000 7f261c675700 1 -- 172.21.15.40:0/1235497283 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26141abac0 con 0x7f261406ad10 2024-04-21T23:17:32.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.390+0000 7f260b7fe700 1 -- 172.21.15.40:0/1235497283 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f260c0052a0 con 0x7f261406ad10 2024-04-21T23:17:32.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.391+0000 7f261c675700 1 -- 172.21.15.40:0/1235497283 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2614068f20 con 0x7f261406ad10 2024-04-21T23:17:32.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.392+0000 7f260b7fe700 1 -- 172.21.15.40:0/1235497283 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f260c024070 con 0x7f261406ad10 2024-04-21T23:17:32.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.393+0000 7f260b7fe700 1 --2- 172.21.15.40:0/1235497283 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f26000a2cb0 0x7f26000a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:32.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.393+0000 7f260b7fe700 1 -- 172.21.15.40:0/1235497283 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f260c00f020 con 0x7f261406ad10 2024-04-21T23:17:32.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.393+0000 7f2619c10700 1 --2- 172.21.15.40:0/1235497283 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f26000a2cb0 0x7f26000a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:32.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.394+0000 7f2619c10700 1 --2- 172.21.15.40:0/1235497283 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f26000a2cb0 0x7f26000a5170 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f261406ab10 tx=0x7f261000a380 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:32.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.397+0000 7f260b7fe700 1 -- 172.21.15.40:0/1235497283 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f260c02b030 con 0x7f261406ad10 2024-04-21T23:17:32.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.605+0000 7f261c675700 1 -- 172.21.15.40:0/1235497283 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f26141abcf0 con 0x7f261406ad10 2024-04-21T23:17:32.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.605+0000 7f260b7fe700 1 -- 172.21.15.40:0/1235497283 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f260c09dc00 con 0x7f261406ad10 2024-04-21T23:17:32.605 INFO:teuthology.orchestra.run.smithi040.stdout:77309411407 2024-04-21T23:17:32.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.608+0000 7f261c675700 1 -- 172.21.15.40:0/1235497283 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f26000a2cb0 msgr2=0x7f26000a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:32.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.608+0000 7f261c675700 1 --2- 172.21.15.40:0/1235497283 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f26000a2cb0 0x7f26000a5170 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f261406ab10 tx=0x7f261000a380 comp rx=0 tx=0).stop 2024-04-21T23:17:32.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.608+0000 7f261c675700 1 -- 172.21.15.40:0/1235497283 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f261406ad10 msgr2=0x7f261419ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:32.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.608+0000 7f261c675700 1 --2- 172.21.15.40:0/1235497283 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f261406ad10 0x7f261419ef40 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f260c00b070 tx=0x7f260c004e40 comp rx=0 tx=0).stop 2024-04-21T23:17:32.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.608+0000 7f261c675700 1 -- 172.21.15.40:0/1235497283 shutdown_connections 2024-04-21T23:17:32.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.608+0000 7f261c675700 1 --2- 172.21.15.40:0/1235497283 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2614107150 0x7f26141a3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:32.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.608+0000 7f261c675700 1 --2- 172.21.15.40:0/1235497283 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f26000a2cb0 0x7f26000a5170 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:32.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.608+0000 7f261c675700 1 --2- 172.21.15.40:0/1235497283 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f261406ad10 0x7f261419ef40 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:32.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.608+0000 7f261c675700 1 --2- 172.21.15.40:0/1235497283 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2614104c60 0x7f261419f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:32.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.608+0000 7f261c675700 1 -- 172.21.15.40:0/1235497283 >> 172.21.15.40:0/1235497283 conn(0x7f26140780a0 msgr2=0x7f2614100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:32.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.608+0000 7f261c675700 1 -- 172.21.15.40:0/1235497283 shutdown_connections 2024-04-21T23:17:32.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:32.608+0000 7f261c675700 1 -- 172.21.15.40:0/1235497283 wait complete. 2024-04-21T23:17:32.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:32 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/2447238287' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-21T23:17:32.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:32 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/2447238287' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-21T23:17:32.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:32 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/2447238287' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-21T23:17:33.054 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182467 got 107374182473 for osd.3 2024-04-21T23:17:33.054 DEBUG:teuthology.parallel:result is None 2024-04-21T23:17:33.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:33 smithi040 ceph-mon[27984]: pgmap v343: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-21T23:17:33.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:33 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/1235497283' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-21T23:17:33.781 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411401 got 77309411407 for osd.2 2024-04-21T23:17:33.781 DEBUG:teuthology.parallel:result is None 2024-04-21T23:17:33.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:33 smithi174 ceph-mon[31409]: pgmap v343: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-21T23:17:33.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:33 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/1235497283' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-21T23:17:33.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:33 smithi111 ceph-mon[31085]: pgmap v343: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-21T23:17:33.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:33 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/1235497283' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-21T23:17:34.067 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757305 got 163208757310 for osd.5 2024-04-21T23:17:34.068 DEBUG:teuthology.parallel:result is None 2024-04-21T23:17:34.526 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495669 got 197568495674 for osd.6 2024-04-21T23:17:34.527 DEBUG:teuthology.parallel:result is None 2024-04-21T23:17:34.527 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-21T23:17:34.527 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-21T23:17:34.799 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:17:35.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:35 smithi040 ceph-mon[27984]: pgmap v344: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T23:17:35.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:35 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:17:35.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:35 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:17:35.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:35 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:17:35.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:35 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:17:35.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:35 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:17:36.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:35 smithi174 ceph-mon[31409]: pgmap v344: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T23:17:36.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:35 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:17:36.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:35 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:17:36.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:35 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:17:36.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:35 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:17:36.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:35 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:17:36.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:35 smithi111 ceph-mon[31085]: pgmap v344: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T23:17:36.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:35 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:17:36.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:35 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:17:36.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:35 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:17:36.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:35 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:17:36.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:35 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' 2024-04-21T23:17:36.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:36.562+0000 7f11addcd700 1 -- 172.21.15.40:0/1340744064 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f11a8069f40 msgr2=0x7f11a806a3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:36.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:36.562+0000 7f11addcd700 1 --2- 172.21.15.40:0/1340744064 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f11a8069f40 0x7f11a806a3a0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f119c009990 tx=0x7f119c009ca0 comp rx=0 tx=0).stop 2024-04-21T23:17:36.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:36.562+0000 7f11addcd700 1 -- 172.21.15.40:0/1340744064 shutdown_connections 2024-04-21T23:17:36.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:36.562+0000 7f11addcd700 1 --2- 172.21.15.40:0/1340744064 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11a806a8e0 0x7f11a810f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:36.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:36.562+0000 7f11addcd700 1 --2- 172.21.15.40:0/1340744064 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11a81082b0 0x7f11a8108690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:36.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:36.562+0000 7f11addcd700 1 --2- 172.21.15.40:0/1340744064 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f11a8069f40 0x7f11a806a3a0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:36.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:36.562+0000 7f11addcd700 1 -- 172.21.15.40:0/1340744064 >> 172.21.15.40:0/1340744064 conn(0x7f11a80780e0 msgr2=0x7f11a80784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:36.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:36.562+0000 7f11addcd700 1 -- 172.21.15.40:0/1340744064 shutdown_connections 2024-04-21T23:17:36.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:36.562+0000 7f11addcd700 1 -- 172.21.15.40:0/1340744064 wait complete. 2024-04-21T23:17:36.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:36.563+0000 7f11addcd700 1 Processor -- start 2024-04-21T23:17:36.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:36.563+0000 7f11addcd700 1 -- start start 2024-04-21T23:17:36.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:36.563+0000 7f11addcd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f11a8069f40 0x7f11a819e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:36.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:36.563+0000 7f11addcd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11a806a8e0 0x7f11a819ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:36.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:36.563+0000 7f11addcd700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11a81082b0 0x7f11a8198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:36.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:36.563+0000 7f11addcd700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f11a8112530 con 0x7f11a8069f40 2024-04-21T23:17:36.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:36.563+0000 7f11addcd700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f11a81123b0 con 0x7f11a806a8e0 2024-04-21T23:17:36.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:36.563+0000 7f11addcd700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f11a81126b0 con 0x7f11a81082b0 2024-04-21T23:17:36.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:36.564+0000 7f11a77fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f11a8069f40 0x7f11a819e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:36.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:36.564+0000 7f11a77fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f11a8069f40 0x7f11a819e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42690/0 (socket says 172.21.15.40:42690) 2024-04-21T23:17:36.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:36.564+0000 7f11a77fe700 1 -- 172.21.15.40:0/2037076372 learned_addr learned my addr 172.21.15.40:0/2037076372 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:17:36.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:36.564+0000 7f11a6ffd700 1 --2- 172.21.15.40:0/2037076372 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11a806a8e0 0x7f11a819ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:36.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:36.564+0000 7f11a7fff700 1 --2- 172.21.15.40:0/2037076372 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11a81082b0 0x7f11a8198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:36.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:36.564+0000 7f11a77fe700 1 -- 172.21.15.40:0/2037076372 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11a81082b0 msgr2=0x7f11a8198770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:36.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:36.564+0000 7f11a77fe700 1 --2- 172.21.15.40:0/2037076372 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11a81082b0 0x7f11a8198770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:36.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:36.564+0000 7f11a77fe700 1 -- 172.21.15.40:0/2037076372 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11a806a8e0 msgr2=0x7f11a819ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:36.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:36.564+0000 7f11a77fe700 1 --2- 172.21.15.40:0/2037076372 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11a806a8e0 0x7f11a819ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:36.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:36.564+0000 7f11a77fe700 1 -- 172.21.15.40:0/2037076372 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11a8199050 con 0x7f11a8069f40 2024-04-21T23:17:36.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:36.564+0000 7f11a6ffd700 1 --2- 172.21.15.40:0/2037076372 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11a806a8e0 0x7f11a819ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:17:36.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:36.564+0000 7f11a7fff700 1 --2- 172.21.15.40:0/2037076372 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11a81082b0 0x7f11a8198770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:17:36.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:36.564+0000 7f11a77fe700 1 --2- 172.21.15.40:0/2037076372 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f11a8069f40 0x7f11a819e6a0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f119800c9b0 tx=0x7f119800cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:36.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:36.564+0000 7f11a4ff9700 1 -- 172.21.15.40:0/2037076372 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1198007b00 con 0x7f11a8069f40 2024-04-21T23:17:36.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:36.564+0000 7f11a4ff9700 1 -- 172.21.15.40:0/2037076372 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1198007ca0 con 0x7f11a8069f40 2024-04-21T23:17:36.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:36.564+0000 7f11a4ff9700 1 -- 172.21.15.40:0/2037076372 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1198018780 con 0x7f11a8069f40 2024-04-21T23:17:36.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:36.564+0000 7f11addcd700 1 -- 172.21.15.40:0/2037076372 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11a8199340 con 0x7f11a8069f40 2024-04-21T23:17:36.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:36.564+0000 7f11addcd700 1 -- 172.21.15.40:0/2037076372 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11a81a5620 con 0x7f11a8069f40 2024-04-21T23:17:36.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:36.565+0000 7f11addcd700 1 -- 172.21.15.40:0/2037076372 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11a8103960 con 0x7f11a8069f40 2024-04-21T23:17:36.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:36.567+0000 7f11a4ff9700 1 -- 172.21.15.40:0/2037076372 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f119800f450 con 0x7f11a8069f40 2024-04-21T23:17:36.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:36.568+0000 7f11a4ff9700 1 --2- 172.21.15.40:0/2037076372 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f11900a2cb0 0x7f11900a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:36.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:36.568+0000 7f11a6ffd700 1 --2- 172.21.15.40:0/2037076372 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f11900a2cb0 0x7f11900a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:36.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:36.568+0000 7f11a4ff9700 1 -- 172.21.15.40:0/2037076372 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f11980cf870 con 0x7f11a8069f40 2024-04-21T23:17:36.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:36.568+0000 7f11a6ffd700 1 --2- 172.21.15.40:0/2037076372 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f11900a2cb0 0x7f11900a5170 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f119c006010 tx=0x7f119c00b540 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:36.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:36.572+0000 7f11a4ff9700 1 -- 172.21.15.40:0/2037076372 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f1198014030 con 0x7f11a8069f40 2024-04-21T23:17:36.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:36.773+0000 7f11addcd700 1 -- 172.21.15.40:0/2037076372 --> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f11a8199ca0 con 0x7f11900a2cb0 2024-04-21T23:17:36.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:36.783+0000 7f11a4ff9700 1 -- 172.21.15.40:0/2037076372 <== mgr.14148 v2:172.21.15.40:6800/2500760788 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340368 (secure 0 0 0) 0x7f11a8199ca0 con 0x7f11900a2cb0 2024-04-21T23:17:36.786 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:17:36.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:36.787+0000 7f11addcd700 1 -- 172.21.15.40:0/2037076372 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f11900a2cb0 msgr2=0x7f11900a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:36.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:36.787+0000 7f11addcd700 1 --2- 172.21.15.40:0/2037076372 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f11900a2cb0 0x7f11900a5170 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f119c006010 tx=0x7f119c00b540 comp rx=0 tx=0).stop 2024-04-21T23:17:36.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:36.787+0000 7f11addcd700 1 -- 172.21.15.40:0/2037076372 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f11a8069f40 msgr2=0x7f11a819e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:36.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:36.787+0000 7f11addcd700 1 --2- 172.21.15.40:0/2037076372 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f11a8069f40 0x7f11a819e6a0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f119800c9b0 tx=0x7f119800cd70 comp rx=0 tx=0).stop 2024-04-21T23:17:36.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:36.787+0000 7f11addcd700 1 -- 172.21.15.40:0/2037076372 shutdown_connections 2024-04-21T23:17:36.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:36.787+0000 7f11addcd700 1 --2- 172.21.15.40:0/2037076372 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11a81082b0 0x7f11a8198770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:36.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:36.787+0000 7f11addcd700 1 --2- 172.21.15.40:0/2037076372 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f11900a2cb0 0x7f11900a5170 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:36.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:36.787+0000 7f11addcd700 1 --2- 172.21.15.40:0/2037076372 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f11a8069f40 0x7f11a819e6a0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:36.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:36.787+0000 7f11addcd700 1 --2- 172.21.15.40:0/2037076372 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11a806a8e0 0x7f11a819ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:36.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:36.787+0000 7f11addcd700 1 -- 172.21.15.40:0/2037076372 >> 172.21.15.40:0/2037076372 conn(0x7f11a80780e0 msgr2=0x7f11a80ffc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:36.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:36.787+0000 7f11addcd700 1 -- 172.21.15.40:0/2037076372 shutdown_connections 2024-04-21T23:17:36.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:36.787+0000 7f11addcd700 1 -- 172.21.15.40:0/2037076372 wait complete. 2024-04-21T23:17:36.790 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-21T23:17:37.571 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":345,"stamp":"2024-04-21T23:17:36.422068+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":656378,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":593,"num_read_kb":415,"num_write":342,"num_write_kb":3053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1180848,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":277,"ondisk_log_size":277,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":367600,"kb_used_data":14228,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124574224,"statfs":{"total":1151940427776,"available":1151564005376,"internally_reserved":0,"allocated":14569472,"data_stored":8034800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":361674042},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":80,"num_read_kb":40,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003392"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.199761+0000","last_change":"2024-04-21T23:15:56.349368+0000","last_active":"2024-04-21T23:16:52.199761+0000","last_peered":"2024-04-21T23:16:52.199761+0000","last_clean":"2024-04-21T23:16:52.199761+0000","last_became_active":"2024-04-21T23:15:56.345743+0000","last_became_peered":"2024-04-21T23:15:56.345743+0000","last_unstale":"2024-04-21T23:16:52.199761+0000","last_undegraded":"2024-04-21T23:16:52.199761+0000","last_fullsized":"2024-04-21T23:16:52.199761+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:57:21.574012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.203016+0000","last_change":"2024-04-21T23:15:56.347231+0000","last_active":"2024-04-21T23:16:52.203016+0000","last_peered":"2024-04-21T23:16:52.203016+0000","last_clean":"2024-04-21T23:16:52.203016+0000","last_became_active":"2024-04-21T23:15:56.346847+0000","last_became_peered":"2024-04-21T23:15:56.346847+0000","last_unstale":"2024-04-21T23:16:52.203016+0000","last_undegraded":"2024-04-21T23:16:52.203016+0000","last_fullsized":"2024-04-21T23:16:52.203016+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:20:15.884789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.202557+0000","last_change":"2024-04-21T23:15:56.354897+0000","last_active":"2024-04-21T23:16:52.202557+0000","last_peered":"2024-04-21T23:16:52.202557+0000","last_clean":"2024-04-21T23:16:52.202557+0000","last_became_active":"2024-04-21T23:15:56.354494+0000","last_became_peered":"2024-04-21T23:15:56.354494+0000","last_unstale":"2024-04-21T23:16:52.202557+0000","last_undegraded":"2024-04-21T23:16:52.202557+0000","last_fullsized":"2024-04-21T23:16:52.202557+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:51:11.865310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.196805+0000","last_change":"2024-04-21T23:15:56.355184+0000","last_active":"2024-04-21T23:16:52.196805+0000","last_peered":"2024-04-21T23:16:52.196805+0000","last_clean":"2024-04-21T23:16:52.196805+0000","last_became_active":"2024-04-21T23:15:56.354748+0000","last_became_peered":"2024-04-21T23:15:56.354748+0000","last_unstale":"2024-04-21T23:16:52.196805+0000","last_undegraded":"2024-04-21T23:16:52.196805+0000","last_fullsized":"2024-04-21T23:16:52.196805+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:46:24.867488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.200037+0000","last_change":"2024-04-21T23:15:56.349446+0000","last_active":"2024-04-21T23:16:52.200037+0000","last_peered":"2024-04-21T23:16:52.200037+0000","last_clean":"2024-04-21T23:16:52.200037+0000","last_became_active":"2024-04-21T23:15:56.349050+0000","last_became_peered":"2024-04-21T23:15:56.349050+0000","last_unstale":"2024-04-21T23:16:52.200037+0000","last_undegraded":"2024-04-21T23:16:52.200037+0000","last_fullsized":"2024-04-21T23:16:52.200037+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:15:25.489044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.554066+0000","last_change":"2024-04-21T23:15:56.356095+0000","last_active":"2024-04-21T23:16:52.554066+0000","last_peered":"2024-04-21T23:16:52.554066+0000","last_clean":"2024-04-21T23:16:52.554066+0000","last_became_active":"2024-04-21T23:15:56.355754+0000","last_became_peered":"2024-04-21T23:15:56.355754+0000","last_unstale":"2024-04-21T23:16:52.554066+0000","last_undegraded":"2024-04-21T23:16:52.554066+0000","last_fullsized":"2024-04-21T23:16:52.554066+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:00:27.296457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.199230+0000","last_change":"2024-04-21T23:15:56.349733+0000","last_active":"2024-04-21T23:16:52.199230+0000","last_peered":"2024-04-21T23:16:52.199230+0000","last_clean":"2024-04-21T23:16:52.199230+0000","last_became_active":"2024-04-21T23:15:56.347820+0000","last_became_peered":"2024-04-21T23:15:56.347820+0000","last_unstale":"2024-04-21T23:16:52.199230+0000","last_undegraded":"2024-04-21T23:16:52.199230+0000","last_fullsized":"2024-04-21T23:16:52.199230+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:31:34.453626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.347818+0000","last_change":"2024-04-21T23:15:56.359385+0000","last_active":"2024-04-21T23:16:52.347818+0000","last_peered":"2024-04-21T23:16:52.347818+0000","last_clean":"2024-04-21T23:16:52.347818+0000","last_became_active":"2024-04-21T23:15:56.359074+0000","last_became_peered":"2024-04-21T23:15:56.359074+0000","last_unstale":"2024-04-21T23:16:52.347818+0000","last_undegraded":"2024-04-21T23:16:52.347818+0000","last_fullsized":"2024-04-21T23:16:52.347818+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:40:23.904592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.200243+0000","last_change":"2024-04-21T23:15:56.350886+0000","last_active":"2024-04-21T23:16:52.200243+0000","last_peered":"2024-04-21T23:16:52.200243+0000","last_clean":"2024-04-21T23:16:52.200243+0000","last_became_active":"2024-04-21T23:15:56.350531+0000","last_became_peered":"2024-04-21T23:15:56.350531+0000","last_unstale":"2024-04-21T23:16:52.200243+0000","last_undegraded":"2024-04-21T23:16:52.200243+0000","last_fullsized":"2024-04-21T23:16:52.200243+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:13:12.354473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.554360+0000","last_change":"2024-04-21T23:15:56.347376+0000","last_active":"2024-04-21T23:16:52.554360+0000","last_peered":"2024-04-21T23:16:52.554360+0000","last_clean":"2024-04-21T23:16:52.554360+0000","last_became_active":"2024-04-21T23:15:56.347016+0000","last_became_peered":"2024-04-21T23:15:56.347016+0000","last_unstale":"2024-04-21T23:16:52.554360+0000","last_undegraded":"2024-04-21T23:16:52.554360+0000","last_fullsized":"2024-04-21T23:16:52.554360+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:49:46.722931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.200197+0000","last_change":"2024-04-21T23:15:56.351754+0000","last_active":"2024-04-21T23:16:52.200197+0000","last_peered":"2024-04-21T23:16:52.200197+0000","last_clean":"2024-04-21T23:16:52.200197+0000","last_became_active":"2024-04-21T23:15:56.349860+0000","last_became_peered":"2024-04-21T23:15:56.349860+0000","last_unstale":"2024-04-21T23:16:52.200197+0000","last_undegraded":"2024-04-21T23:16:52.200197+0000","last_fullsized":"2024-04-21T23:16:52.200197+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:48:41.469708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.196675+0000","last_change":"2024-04-21T23:15:56.345177+0000","last_active":"2024-04-21T23:16:52.196675+0000","last_peered":"2024-04-21T23:16:52.196675+0000","last_clean":"2024-04-21T23:16:52.196675+0000","last_became_active":"2024-04-21T23:15:56.344791+0000","last_became_peered":"2024-04-21T23:15:56.344791+0000","last_unstale":"2024-04-21T23:16:52.196675+0000","last_undegraded":"2024-04-21T23:16:52.196675+0000","last_fullsized":"2024-04-21T23:16:52.196675+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:33:33.418383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.556505+0000","last_change":"2024-04-21T23:16:52.556505+0000","last_active":"2024-04-21T23:16:52.556505+0000","last_peered":"2024-04-21T23:16:52.556505+0000","last_clean":"2024-04-21T23:16:52.556505+0000","last_became_active":"2024-04-21T23:16:52.556177+0000","last_became_peered":"2024-04-21T23:16:52.556177+0000","last_unstale":"2024-04-21T23:16:52.556505+0000","last_undegraded":"2024-04-21T23:16:52.556505+0000","last_fullsized":"2024-04-21T23:16:52.556505+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:17:13.546746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.202410+0000","last_change":"2024-04-21T23:15:56.354544+0000","last_active":"2024-04-21T23:16:52.202410+0000","last_peered":"2024-04-21T23:16:52.202410+0000","last_clean":"2024-04-21T23:16:52.202410+0000","last_became_active":"2024-04-21T23:15:56.354206+0000","last_became_peered":"2024-04-21T23:15:56.354206+0000","last_unstale":"2024-04-21T23:16:52.202410+0000","last_undegraded":"2024-04-21T23:16:52.202410+0000","last_fullsized":"2024-04-21T23:16:52.202410+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:37:19.985422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.202201+0000","last_change":"2024-04-21T23:15:56.357447+0000","last_active":"2024-04-21T23:16:52.202201+0000","last_peered":"2024-04-21T23:16:52.202201+0000","last_clean":"2024-04-21T23:16:52.202201+0000","last_became_active":"2024-04-21T23:15:56.355850+0000","last_became_peered":"2024-04-21T23:15:56.355850+0000","last_unstale":"2024-04-21T23:16:52.202201+0000","last_undegraded":"2024-04-21T23:16:52.202201+0000","last_fullsized":"2024-04-21T23:16:52.202201+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:48:56.648994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.198855+0000","last_change":"2024-04-21T23:15:56.353712+0000","last_active":"2024-04-21T23:16:52.198855+0000","last_peered":"2024-04-21T23:16:52.198855+0000","last_clean":"2024-04-21T23:16:52.198855+0000","last_became_active":"2024-04-21T23:15:56.353388+0000","last_became_peered":"2024-04-21T23:15:56.353388+0000","last_unstale":"2024-04-21T23:16:52.198855+0000","last_undegraded":"2024-04-21T23:16:52.198855+0000","last_fullsized":"2024-04-21T23:16:52.198855+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:43:33.581222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.199691+0000","last_change":"2024-04-21T23:15:56.347041+0000","last_active":"2024-04-21T23:16:52.199691+0000","last_peered":"2024-04-21T23:16:52.199691+0000","last_clean":"2024-04-21T23:16:52.199691+0000","last_became_active":"2024-04-21T23:15:56.346630+0000","last_became_peered":"2024-04-21T23:15:56.346630+0000","last_unstale":"2024-04-21T23:16:52.199691+0000","last_undegraded":"2024-04-21T23:16:52.199691+0000","last_fullsized":"2024-04-21T23:16:52.199691+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:57:42.553666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.195943+0000","last_change":"2024-04-21T23:15:56.353809+0000","last_active":"2024-04-21T23:16:52.195943+0000","last_peered":"2024-04-21T23:16:52.195943+0000","last_clean":"2024-04-21T23:16:52.195943+0000","last_became_active":"2024-04-21T23:15:56.353469+0000","last_became_peered":"2024-04-21T23:15:56.353469+0000","last_unstale":"2024-04-21T23:16:52.195943+0000","last_undegraded":"2024-04-21T23:16:52.195943+0000","last_fullsized":"2024-04-21T23:16:52.195943+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:11:14.136845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.204480+0000","last_change":"2024-04-21T23:15:56.354500+0000","last_active":"2024-04-21T23:16:52.204480+0000","last_peered":"2024-04-21T23:16:52.204480+0000","last_clean":"2024-04-21T23:16:52.204480+0000","last_became_active":"2024-04-21T23:15:56.353893+0000","last_became_peered":"2024-04-21T23:15:56.353893+0000","last_unstale":"2024-04-21T23:16:52.204480+0000","last_undegraded":"2024-04-21T23:16:52.204480+0000","last_fullsized":"2024-04-21T23:16:52.204480+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:14:38.121448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.203524+0000","last_change":"2024-04-21T23:16:00.438245+0000","last_active":"2024-04-21T23:16:52.203524+0000","last_peered":"2024-04-21T23:16:52.203524+0000","last_clean":"2024-04-21T23:16:52.203524+0000","last_became_active":"2024-04-21T23:16:00.437755+0000","last_became_peered":"2024-04-21T23:16:00.437755+0000","last_unstale":"2024-04-21T23:16:52.203524+0000","last_undegraded":"2024-04-21T23:16:52.203524+0000","last_fullsized":"2024-04-21T23:16:52.203524+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:33:37.273223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"90'9","reported_seq":37,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.196046+0000","last_change":"2024-04-21T23:15:55.402553+0000","last_active":"2024-04-21T23:16:52.196046+0000","last_peered":"2024-04-21T23:16:52.196046+0000","last_clean":"2024-04-21T23:16:52.196046+0000","last_became_active":"2024-04-21T23:15:55.402192+0000","last_became_peered":"2024-04-21T23:15:55.402192+0000","last_unstale":"2024-04-21T23:16:52.196046+0000","last_undegraded":"2024-04-21T23:16:52.196046+0000","last_fullsized":"2024-04-21T23:16:52.196046+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:24:57.467906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.348222+0000","last_change":"2024-04-21T23:15:56.358797+0000","last_active":"2024-04-21T23:16:52.348222+0000","last_peered":"2024-04-21T23:16:52.348222+0000","last_clean":"2024-04-21T23:16:52.348222+0000","last_became_active":"2024-04-21T23:15:56.357997+0000","last_became_peered":"2024-04-21T23:15:56.357997+0000","last_unstale":"2024-04-21T23:16:52.348222+0000","last_undegraded":"2024-04-21T23:16:52.348222+0000","last_fullsized":"2024-04-21T23:16:52.348222+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:20:50.891506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.196189+0000","last_change":"2024-04-21T23:16:00.443368+0000","last_active":"2024-04-21T23:16:52.196189+0000","last_peered":"2024-04-21T23:16:52.196189+0000","last_clean":"2024-04-21T23:16:52.196189+0000","last_became_active":"2024-04-21T23:16:00.442940+0000","last_became_peered":"2024-04-21T23:16:00.442940+0000","last_unstale":"2024-04-21T23:16:52.196189+0000","last_undegraded":"2024-04-21T23:16:52.196189+0000","last_fullsized":"2024-04-21T23:16:52.196189+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:17:15.706460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"90'2","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.348606+0000","last_change":"2024-04-21T23:15:55.405216+0000","last_active":"2024-04-21T23:16:52.348606+0000","last_peered":"2024-04-21T23:16:52.348606+0000","last_clean":"2024-04-21T23:16:52.348606+0000","last_became_active":"2024-04-21T23:15:55.404844+0000","last_became_peered":"2024-04-21T23:15:55.404844+0000","last_unstale":"2024-04-21T23:16:52.348606+0000","last_undegraded":"2024-04-21T23:16:52.348606+0000","last_fullsized":"2024-04-21T23:16:52.348606+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:56:28.691844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.202921+0000","last_change":"2024-04-21T23:15:56.355789+0000","last_active":"2024-04-21T23:16:52.202921+0000","last_peered":"2024-04-21T23:16:52.202921+0000","last_clean":"2024-04-21T23:16:52.202921+0000","last_became_active":"2024-04-21T23:15:56.354972+0000","last_became_peered":"2024-04-21T23:15:56.354972+0000","last_unstale":"2024-04-21T23:16:52.202921+0000","last_undegraded":"2024-04-21T23:16:52.202921+0000","last_fullsized":"2024-04-21T23:16:52.202921+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:11:36.831453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.196834+0000","last_change":"2024-04-21T23:16:00.443241+0000","last_active":"2024-04-21T23:16:52.196834+0000","last_peered":"2024-04-21T23:16:52.196834+0000","last_clean":"2024-04-21T23:16:52.196834+0000","last_became_active":"2024-04-21T23:16:00.442820+0000","last_became_peered":"2024-04-21T23:16:00.442820+0000","last_unstale":"2024-04-21T23:16:52.196834+0000","last_undegraded":"2024-04-21T23:16:52.196834+0000","last_fullsized":"2024-04-21T23:16:52.196834+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:28:04.032621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"90'4","reported_seq":43,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.212906+0000","last_change":"2024-04-21T23:16:52.212906+0000","last_active":"2024-04-21T23:16:52.212906+0000","last_peered":"2024-04-21T23:16:52.212906+0000","last_clean":"2024-04-21T23:16:52.212906+0000","last_became_active":"2024-04-21T23:16:52.205828+0000","last_became_peered":"2024-04-21T23:16:52.205828+0000","last_unstale":"2024-04-21T23:16:52.212906+0000","last_undegraded":"2024-04-21T23:16:52.212906+0000","last_fullsized":"2024-04-21T23:16:52.212906+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:20:01.936578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.199572+0000","last_change":"2024-04-21T23:15:56.354462+0000","last_active":"2024-04-21T23:16:52.199572+0000","last_peered":"2024-04-21T23:16:52.199572+0000","last_clean":"2024-04-21T23:16:52.199572+0000","last_became_active":"2024-04-21T23:15:56.354134+0000","last_became_peered":"2024-04-21T23:15:56.354134+0000","last_unstale":"2024-04-21T23:16:52.199572+0000","last_undegraded":"2024-04-21T23:16:52.199572+0000","last_fullsized":"2024-04-21T23:16:52.199572+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:27:40.809483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.196258+0000","last_change":"2024-04-21T23:16:00.427717+0000","last_active":"2024-04-21T23:16:52.196258+0000","last_peered":"2024-04-21T23:16:52.196258+0000","last_clean":"2024-04-21T23:16:52.196258+0000","last_became_active":"2024-04-21T23:16:00.427296+0000","last_became_peered":"2024-04-21T23:16:00.427296+0000","last_unstale":"2024-04-21T23:16:52.196258+0000","last_undegraded":"2024-04-21T23:16:52.196258+0000","last_fullsized":"2024-04-21T23:16:52.196258+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:03:41.650045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.201497+0000","last_change":"2024-04-21T23:15:55.398641+0000","last_active":"2024-04-21T23:16:52.201497+0000","last_peered":"2024-04-21T23:16:52.201497+0000","last_clean":"2024-04-21T23:16:52.201497+0000","last_became_active":"2024-04-21T23:15:55.397712+0000","last_became_peered":"2024-04-21T23:15:55.397712+0000","last_unstale":"2024-04-21T23:16:52.201497+0000","last_undegraded":"2024-04-21T23:16:52.201497+0000","last_fullsized":"2024-04-21T23:16:52.201497+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:28:09.266740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.197823+0000","last_change":"2024-04-21T23:15:56.345359+0000","last_active":"2024-04-21T23:16:52.197823+0000","last_peered":"2024-04-21T23:16:52.197823+0000","last_clean":"2024-04-21T23:16:52.197823+0000","last_became_active":"2024-04-21T23:15:56.343652+0000","last_became_peered":"2024-04-21T23:15:56.343652+0000","last_unstale":"2024-04-21T23:16:52.197823+0000","last_undegraded":"2024-04-21T23:16:52.197823+0000","last_fullsized":"2024-04-21T23:16:52.197823+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:18:10.057999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.204025+0000","last_change":"2024-04-21T23:16:00.430640+0000","last_active":"2024-04-21T23:16:52.204025+0000","last_peered":"2024-04-21T23:16:52.204025+0000","last_clean":"2024-04-21T23:16:52.204025+0000","last_became_active":"2024-04-21T23:16:00.430141+0000","last_became_peered":"2024-04-21T23:16:00.430141+0000","last_unstale":"2024-04-21T23:16:52.204025+0000","last_undegraded":"2024-04-21T23:16:52.204025+0000","last_fullsized":"2024-04-21T23:16:52.204025+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:41:13.749780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'5","reported_seq":106,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:17:32.614037+0000","last_change":"2024-04-21T23:15:55.399823+0000","last_active":"2024-04-21T23:17:32.614037+0000","last_peered":"2024-04-21T23:17:32.614037+0000","last_clean":"2024-04-21T23:17:32.614037+0000","last_became_active":"2024-04-21T23:15:55.399490+0000","last_became_peered":"2024-04-21T23:15:55.399490+0000","last_unstale":"2024-04-21T23:17:32.614037+0000","last_undegraded":"2024-04-21T23:17:32.614037+0000","last_fullsized":"2024-04-21T23:17:32.614037+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:54:38.124841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.200149+0000","last_change":"2024-04-21T23:15:56.346444+0000","last_active":"2024-04-21T23:16:52.200149+0000","last_peered":"2024-04-21T23:16:52.200149+0000","last_clean":"2024-04-21T23:16:52.200149+0000","last_became_active":"2024-04-21T23:15:56.346091+0000","last_became_peered":"2024-04-21T23:15:56.346091+0000","last_unstale":"2024-04-21T23:16:52.200149+0000","last_undegraded":"2024-04-21T23:16:52.200149+0000","last_fullsized":"2024-04-21T23:16:52.200149+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:41:11.144570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.203107+0000","last_change":"2024-04-21T23:16:00.438198+0000","last_active":"2024-04-21T23:16:52.203107+0000","last_peered":"2024-04-21T23:16:52.203107+0000","last_clean":"2024-04-21T23:16:52.203107+0000","last_became_active":"2024-04-21T23:16:00.437767+0000","last_became_peered":"2024-04-21T23:16:00.437767+0000","last_unstale":"2024-04-21T23:16:52.203107+0000","last_undegraded":"2024-04-21T23:16:52.203107+0000","last_fullsized":"2024-04-21T23:16:52.203107+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:35:11.356414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"90'3","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.554804+0000","last_change":"2024-04-21T23:15:55.403896+0000","last_active":"2024-04-21T23:16:52.554804+0000","last_peered":"2024-04-21T23:16:52.554804+0000","last_clean":"2024-04-21T23:16:52.554804+0000","last_became_active":"2024-04-21T23:15:55.403423+0000","last_became_peered":"2024-04-21T23:15:55.403423+0000","last_unstale":"2024-04-21T23:16:52.554804+0000","last_undegraded":"2024-04-21T23:16:52.554804+0000","last_fullsized":"2024-04-21T23:16:52.554804+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:34:44.252259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.349722+0000","last_change":"2024-04-21T23:16:52.349354+0000","last_active":"2024-04-21T23:16:52.349722+0000","last_peered":"2024-04-21T23:16:52.349722+0000","last_clean":"2024-04-21T23:16:52.349722+0000","last_became_active":"2024-04-21T23:16:52.348991+0000","last_became_peered":"2024-04-21T23:16:52.348991+0000","last_unstale":"2024-04-21T23:16:52.349722+0000","last_undegraded":"2024-04-21T23:16:52.349722+0000","last_fullsized":"2024-04-21T23:16:52.349722+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:42:57.975963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.199641+0000","last_change":"2024-04-21T23:16:00.434104+0000","last_active":"2024-04-21T23:16:52.199641+0000","last_peered":"2024-04-21T23:16:52.199641+0000","last_clean":"2024-04-21T23:16:52.199641+0000","last_became_active":"2024-04-21T23:16:00.433681+0000","last_became_peered":"2024-04-21T23:16:00.433681+0000","last_unstale":"2024-04-21T23:16:52.199641+0000","last_undegraded":"2024-04-21T23:16:52.199641+0000","last_fullsized":"2024-04-21T23:16:52.199641+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:10:17.986652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"90'2","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.198381+0000","last_change":"2024-04-21T23:15:55.397536+0000","last_active":"2024-04-21T23:16:52.198381+0000","last_peered":"2024-04-21T23:16:52.198381+0000","last_clean":"2024-04-21T23:16:52.198381+0000","last_became_active":"2024-04-21T23:15:55.397163+0000","last_became_peered":"2024-04-21T23:15:55.397163+0000","last_unstale":"2024-04-21T23:16:52.198381+0000","last_undegraded":"2024-04-21T23:16:52.198381+0000","last_fullsized":"2024-04-21T23:16:52.198381+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:05:27.777804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.347696+0000","last_change":"2024-04-21T23:15:56.351840+0000","last_active":"2024-04-21T23:16:52.347696+0000","last_peered":"2024-04-21T23:16:52.347696+0000","last_clean":"2024-04-21T23:16:52.347696+0000","last_became_active":"2024-04-21T23:15:56.351523+0000","last_became_peered":"2024-04-21T23:15:56.351523+0000","last_unstale":"2024-04-21T23:16:52.347696+0000","last_undegraded":"2024-04-21T23:16:52.347696+0000","last_fullsized":"2024-04-21T23:16:52.347696+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:57:11.714284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.202885+0000","last_change":"2024-04-21T23:16:00.437473+0000","last_active":"2024-04-21T23:16:52.202885+0000","last_peered":"2024-04-21T23:16:52.202885+0000","last_clean":"2024-04-21T23:16:52.202885+0000","last_became_active":"2024-04-21T23:16:00.437014+0000","last_became_peered":"2024-04-21T23:16:00.437014+0000","last_unstale":"2024-04-21T23:16:52.202885+0000","last_undegraded":"2024-04-21T23:16:52.202885+0000","last_fullsized":"2024-04-21T23:16:52.202885+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:37:14.241896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"90'1","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.196460+0000","last_change":"2024-04-21T23:15:55.399213+0000","last_active":"2024-04-21T23:16:52.196460+0000","last_peered":"2024-04-21T23:16:52.196460+0000","last_clean":"2024-04-21T23:16:52.196460+0000","last_became_active":"2024-04-21T23:15:55.398880+0000","last_became_peered":"2024-04-21T23:15:55.398880+0000","last_unstale":"2024-04-21T23:16:52.196460+0000","last_undegraded":"2024-04-21T23:16:52.196460+0000","last_fullsized":"2024-04-21T23:16:52.196460+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:56:32.236862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.195415+0000","last_change":"2024-04-21T23:15:56.349330+0000","last_active":"2024-04-21T23:16:52.195415+0000","last_peered":"2024-04-21T23:16:52.195415+0000","last_clean":"2024-04-21T23:16:52.195415+0000","last_became_active":"2024-04-21T23:15:56.348988+0000","last_became_peered":"2024-04-21T23:15:56.348988+0000","last_unstale":"2024-04-21T23:16:52.195415+0000","last_undegraded":"2024-04-21T23:16:52.195415+0000","last_fullsized":"2024-04-21T23:16:52.195415+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:11:19.650119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.347940+0000","last_change":"2024-04-21T23:16:00.431959+0000","last_active":"2024-04-21T23:16:52.347940+0000","last_peered":"2024-04-21T23:16:52.347940+0000","last_clean":"2024-04-21T23:16:52.347940+0000","last_became_active":"2024-04-21T23:16:00.431570+0000","last_became_peered":"2024-04-21T23:16:00.431570+0000","last_unstale":"2024-04-21T23:16:52.347940+0000","last_undegraded":"2024-04-21T23:16:52.347940+0000","last_fullsized":"2024-04-21T23:16:52.347940+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:02:54.057326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"90'3","reported_seq":102,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:17:32.613433+0000","last_change":"2024-04-21T23:15:55.404505+0000","last_active":"2024-04-21T23:17:32.613433+0000","last_peered":"2024-04-21T23:17:32.613433+0000","last_clean":"2024-04-21T23:17:32.613433+0000","last_became_active":"2024-04-21T23:15:55.403945+0000","last_became_peered":"2024-04-21T23:15:55.403945+0000","last_unstale":"2024-04-21T23:17:32.613433+0000","last_undegraded":"2024-04-21T23:17:32.613433+0000","last_fullsized":"2024-04-21T23:17:32.613433+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:54:37.647597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":72,"num_read_kb":72,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.200278+0000","last_change":"2024-04-21T23:15:56.356238+0000","last_active":"2024-04-21T23:16:52.200278+0000","last_peered":"2024-04-21T23:16:52.200278+0000","last_clean":"2024-04-21T23:16:52.200278+0000","last_became_active":"2024-04-21T23:15:56.355116+0000","last_became_peered":"2024-04-21T23:15:56.355116+0000","last_unstale":"2024-04-21T23:16:52.200278+0000","last_undegraded":"2024-04-21T23:16:52.200278+0000","last_fullsized":"2024-04-21T23:16:52.200278+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:43:20.468549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.195318+0000","last_change":"2024-04-21T23:16:00.439069+0000","last_active":"2024-04-21T23:16:52.195318+0000","last_peered":"2024-04-21T23:16:52.195318+0000","last_clean":"2024-04-21T23:16:52.195318+0000","last_became_active":"2024-04-21T23:16:00.438510+0000","last_became_peered":"2024-04-21T23:16:00.438510+0000","last_unstale":"2024-04-21T23:16:52.195318+0000","last_undegraded":"2024-04-21T23:16:52.195318+0000","last_fullsized":"2024-04-21T23:16:52.195318+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:41:50.814112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"90'2","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.206602+0000","last_change":"2024-04-21T23:16:52.206602+0000","last_active":"2024-04-21T23:16:52.206602+0000","last_peered":"2024-04-21T23:16:52.206602+0000","last_clean":"2024-04-21T23:16:52.206602+0000","last_became_active":"2024-04-21T23:16:52.202755+0000","last_became_peered":"2024-04-21T23:16:52.202755+0000","last_unstale":"2024-04-21T23:16:52.206602+0000","last_undegraded":"2024-04-21T23:16:52.206602+0000","last_fullsized":"2024-04-21T23:16:52.206602+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:01:02.437392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.203071+0000","last_change":"2024-04-21T23:15:56.357228+0000","last_active":"2024-04-21T23:16:52.203071+0000","last_peered":"2024-04-21T23:16:52.203071+0000","last_clean":"2024-04-21T23:16:52.203071+0000","last_became_active":"2024-04-21T23:15:56.356808+0000","last_became_peered":"2024-04-21T23:15:56.356808+0000","last_unstale":"2024-04-21T23:16:52.203071+0000","last_undegraded":"2024-04-21T23:16:52.203071+0000","last_fullsized":"2024-04-21T23:16:52.203071+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:15:10.728530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.197048+0000","last_change":"2024-04-21T23:16:00.443569+0000","last_active":"2024-04-21T23:16:52.197048+0000","last_peered":"2024-04-21T23:16:52.197048+0000","last_clean":"2024-04-21T23:16:52.197048+0000","last_became_active":"2024-04-21T23:16:00.442972+0000","last_became_peered":"2024-04-21T23:16:00.442972+0000","last_unstale":"2024-04-21T23:16:52.197048+0000","last_undegraded":"2024-04-21T23:16:52.197048+0000","last_fullsized":"2024-04-21T23:16:52.197048+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:35:21.471407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"90'4","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.197455+0000","last_change":"2024-04-21T23:15:55.400134+0000","last_active":"2024-04-21T23:16:52.197455+0000","last_peered":"2024-04-21T23:16:52.197455+0000","last_clean":"2024-04-21T23:16:52.197455+0000","last_became_active":"2024-04-21T23:15:55.399768+0000","last_became_peered":"2024-04-21T23:15:55.399768+0000","last_unstale":"2024-04-21T23:16:52.197455+0000","last_undegraded":"2024-04-21T23:16:52.197455+0000","last_fullsized":"2024-04-21T23:16:52.197455+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:15:09.323035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.199757+0000","last_change":"2024-04-21T23:15:56.357278+0000","last_active":"2024-04-21T23:16:52.199757+0000","last_peered":"2024-04-21T23:16:52.199757+0000","last_clean":"2024-04-21T23:16:52.199757+0000","last_became_active":"2024-04-21T23:15:56.356902+0000","last_became_peered":"2024-04-21T23:15:56.356902+0000","last_unstale":"2024-04-21T23:16:52.199757+0000","last_undegraded":"2024-04-21T23:16:52.199757+0000","last_fullsized":"2024-04-21T23:16:52.199757+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:19:05.269109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.195385+0000","last_change":"2024-04-21T23:16:00.439704+0000","last_active":"2024-04-21T23:16:52.195385+0000","last_peered":"2024-04-21T23:16:52.195385+0000","last_clean":"2024-04-21T23:16:52.195385+0000","last_became_active":"2024-04-21T23:16:00.439286+0000","last_became_peered":"2024-04-21T23:16:00.439286+0000","last_unstale":"2024-04-21T23:16:52.195385+0000","last_undegraded":"2024-04-21T23:16:52.195385+0000","last_fullsized":"2024-04-21T23:16:52.195385+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:08:59.194481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"90'31","reported_seq":209,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:17:33.614868+0000","last_change":"2024-04-21T23:15:55.403344+0000","last_active":"2024-04-21T23:17:33.614868+0000","last_peered":"2024-04-21T23:17:33.614868+0000","last_clean":"2024-04-21T23:17:33.614868+0000","last_became_active":"2024-04-21T23:15:55.402876+0000","last_became_peered":"2024-04-21T23:15:55.402876+0000","last_unstale":"2024-04-21T23:17:33.614868+0000","last_undegraded":"2024-04-21T23:17:33.614868+0000","last_fullsized":"2024-04-21T23:17:33.614868+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:55:59.082964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61775,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":151,"num_read_kb":136,"num_write":32,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.200472+0000","last_change":"2024-04-21T23:15:56.343897+0000","last_active":"2024-04-21T23:16:52.200472+0000","last_peered":"2024-04-21T23:16:52.200472+0000","last_clean":"2024-04-21T23:16:52.200472+0000","last_became_active":"2024-04-21T23:15:56.343567+0000","last_became_peered":"2024-04-21T23:15:56.343567+0000","last_unstale":"2024-04-21T23:16:52.200472+0000","last_undegraded":"2024-04-21T23:16:52.200472+0000","last_fullsized":"2024-04-21T23:16:52.200472+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:57:07.955722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.198235+0000","last_change":"2024-04-21T23:16:00.432722+0000","last_active":"2024-04-21T23:16:52.198235+0000","last_peered":"2024-04-21T23:16:52.198235+0000","last_clean":"2024-04-21T23:16:52.198235+0000","last_became_active":"2024-04-21T23:16:00.432257+0000","last_became_peered":"2024-04-21T23:16:00.432257+0000","last_unstale":"2024-04-21T23:16:52.198235+0000","last_undegraded":"2024-04-21T23:16:52.198235+0000","last_fullsized":"2024-04-21T23:16:52.198235+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:10:14.804490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"90'6","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.202441+0000","last_change":"2024-04-21T23:15:55.401300+0000","last_active":"2024-04-21T23:16:52.202441+0000","last_peered":"2024-04-21T23:16:52.202441+0000","last_clean":"2024-04-21T23:16:52.202441+0000","last_became_active":"2024-04-21T23:15:55.400904+0000","last_became_peered":"2024-04-21T23:15:55.400904+0000","last_unstale":"2024-04-21T23:16:52.202441+0000","last_undegraded":"2024-04-21T23:16:52.202441+0000","last_fullsized":"2024-04-21T23:16:52.202441+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:29:20.439936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.198540+0000","last_change":"2024-04-21T23:15:56.351046+0000","last_active":"2024-04-21T23:16:52.198540+0000","last_peered":"2024-04-21T23:16:52.198540+0000","last_clean":"2024-04-21T23:16:52.198540+0000","last_became_active":"2024-04-21T23:15:56.350662+0000","last_became_peered":"2024-04-21T23:15:56.350662+0000","last_unstale":"2024-04-21T23:16:52.198540+0000","last_undegraded":"2024-04-21T23:16:52.198540+0000","last_fullsized":"2024-04-21T23:16:52.198540+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:40:16.482061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.199200+0000","last_change":"2024-04-21T23:16:00.439816+0000","last_active":"2024-04-21T23:16:52.199200+0000","last_peered":"2024-04-21T23:16:52.199200+0000","last_clean":"2024-04-21T23:16:52.199200+0000","last_became_active":"2024-04-21T23:16:00.439330+0000","last_became_peered":"2024-04-21T23:16:00.439330+0000","last_unstale":"2024-04-21T23:16:52.199200+0000","last_undegraded":"2024-04-21T23:16:52.199200+0000","last_fullsized":"2024-04-21T23:16:52.199200+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:36:24.472968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"90'3","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.196986+0000","last_change":"2024-04-21T23:15:55.398588+0000","last_active":"2024-04-21T23:16:52.196986+0000","last_peered":"2024-04-21T23:16:52.196986+0000","last_clean":"2024-04-21T23:16:52.196986+0000","last_became_active":"2024-04-21T23:15:55.398264+0000","last_became_peered":"2024-04-21T23:15:55.398264+0000","last_unstale":"2024-04-21T23:16:52.196986+0000","last_undegraded":"2024-04-21T23:16:52.196986+0000","last_fullsized":"2024-04-21T23:16:52.196986+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:48:09.293481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.202037+0000","last_change":"2024-04-21T23:16:52.202037+0000","last_active":"2024-04-21T23:16:52.202037+0000","last_peered":"2024-04-21T23:16:52.202037+0000","last_clean":"2024-04-21T23:16:52.202037+0000","last_became_active":"2024-04-21T23:16:52.201244+0000","last_became_peered":"2024-04-21T23:16:52.201244+0000","last_unstale":"2024-04-21T23:16:52.202037+0000","last_undegraded":"2024-04-21T23:16:52.202037+0000","last_fullsized":"2024-04-21T23:16:52.202037+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:37:34.530110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.195827+0000","last_change":"2024-04-21T23:16:00.438581+0000","last_active":"2024-04-21T23:16:52.195827+0000","last_peered":"2024-04-21T23:16:52.195827+0000","last_clean":"2024-04-21T23:16:52.195827+0000","last_became_active":"2024-04-21T23:16:00.438219+0000","last_became_peered":"2024-04-21T23:16:00.438219+0000","last_unstale":"2024-04-21T23:16:52.195827+0000","last_undegraded":"2024-04-21T23:16:52.195827+0000","last_fullsized":"2024-04-21T23:16:52.195827+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:47:23.259045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"90'4","reported_seq":108,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:17:34.615994+0000","last_change":"2024-04-21T23:15:55.397503+0000","last_active":"2024-04-21T23:17:34.615994+0000","last_peered":"2024-04-21T23:17:34.615994+0000","last_clean":"2024-04-21T23:17:34.615994+0000","last_became_active":"2024-04-21T23:15:55.397181+0000","last_became_peered":"2024-04-21T23:15:55.397181+0000","last_unstale":"2024-04-21T23:17:34.615994+0000","last_undegraded":"2024-04-21T23:17:34.615994+0000","last_fullsized":"2024-04-21T23:17:34.615994+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:47:08.775783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":78,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.199048+0000","last_change":"2024-04-21T23:15:56.350872+0000","last_active":"2024-04-21T23:16:52.199048+0000","last_peered":"2024-04-21T23:16:52.199048+0000","last_clean":"2024-04-21T23:16:52.199048+0000","last_became_active":"2024-04-21T23:15:56.350519+0000","last_became_peered":"2024-04-21T23:15:56.350519+0000","last_unstale":"2024-04-21T23:16:52.199048+0000","last_undegraded":"2024-04-21T23:16:52.199048+0000","last_fullsized":"2024-04-21T23:16:52.199048+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:23:37.753893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.199048+0000","last_change":"2024-04-21T23:16:00.436069+0000","last_active":"2024-04-21T23:16:52.199048+0000","last_peered":"2024-04-21T23:16:52.199048+0000","last_clean":"2024-04-21T23:16:52.199048+0000","last_became_active":"2024-04-21T23:16:00.435617+0000","last_became_peered":"2024-04-21T23:16:00.435617+0000","last_unstale":"2024-04-21T23:16:52.199048+0000","last_undegraded":"2024-04-21T23:16:52.199048+0000","last_fullsized":"2024-04-21T23:16:52.199048+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:21:44.256928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"90'15","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.220742+0000","last_change":"2024-04-21T23:16:52.220742+0000","last_active":"2024-04-21T23:16:52.220742+0000","last_peered":"2024-04-21T23:16:52.220742+0000","last_clean":"2024-04-21T23:16:52.220742+0000","last_became_active":"2024-04-21T23:16:52.204787+0000","last_became_peered":"2024-04-21T23:16:52.204787+0000","last_unstale":"2024-04-21T23:16:52.220742+0000","last_undegraded":"2024-04-21T23:16:52.220742+0000","last_fullsized":"2024-04-21T23:16:52.220742+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:53:12.057014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.202341+0000","last_change":"2024-04-21T23:16:52.202341+0000","last_active":"2024-04-21T23:16:52.202341+0000","last_peered":"2024-04-21T23:16:52.202341+0000","last_clean":"2024-04-21T23:16:52.202341+0000","last_became_active":"2024-04-21T23:16:52.201528+0000","last_became_peered":"2024-04-21T23:16:52.201528+0000","last_unstale":"2024-04-21T23:16:52.202341+0000","last_undegraded":"2024-04-21T23:16:52.202341+0000","last_fullsized":"2024-04-21T23:16:52.202341+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:41:24.767379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.196494+0000","last_change":"2024-04-21T23:16:00.433580+0000","last_active":"2024-04-21T23:16:52.196494+0000","last_peered":"2024-04-21T23:16:52.196494+0000","last_clean":"2024-04-21T23:16:52.196494+0000","last_became_active":"2024-04-21T23:16:00.433185+0000","last_became_peered":"2024-04-21T23:16:00.433185+0000","last_unstale":"2024-04-21T23:16:52.196494+0000","last_undegraded":"2024-04-21T23:16:52.196494+0000","last_fullsized":"2024-04-21T23:16:52.196494+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:17:12.317266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"90'2","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.201190+0000","last_change":"2024-04-21T23:15:55.399032+0000","last_active":"2024-04-21T23:16:52.201190+0000","last_peered":"2024-04-21T23:16:52.201190+0000","last_clean":"2024-04-21T23:16:52.201190+0000","last_became_active":"2024-04-21T23:15:55.398208+0000","last_became_peered":"2024-04-21T23:15:55.398208+0000","last_unstale":"2024-04-21T23:16:52.201190+0000","last_undegraded":"2024-04-21T23:16:52.201190+0000","last_fullsized":"2024-04-21T23:16:52.201190+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:14:59.520571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.199490+0000","last_change":"2024-04-21T23:15:56.355357+0000","last_active":"2024-04-21T23:16:52.199490+0000","last_peered":"2024-04-21T23:16:52.199490+0000","last_clean":"2024-04-21T23:16:52.199490+0000","last_became_active":"2024-04-21T23:15:56.354704+0000","last_became_peered":"2024-04-21T23:15:56.354704+0000","last_unstale":"2024-04-21T23:16:52.199490+0000","last_undegraded":"2024-04-21T23:16:52.199490+0000","last_fullsized":"2024-04-21T23:16:52.199490+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:14:02.368659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.200338+0000","last_change":"2024-04-21T23:16:00.427781+0000","last_active":"2024-04-21T23:16:52.200338+0000","last_peered":"2024-04-21T23:16:52.200338+0000","last_clean":"2024-04-21T23:16:52.200338+0000","last_became_active":"2024-04-21T23:16:00.427244+0000","last_became_peered":"2024-04-21T23:16:00.427244+0000","last_unstale":"2024-04-21T23:16:52.200338+0000","last_undegraded":"2024-04-21T23:16:52.200338+0000","last_fullsized":"2024-04-21T23:16:52.200338+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:21:30.548865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.195660+0000","last_change":"2024-04-21T23:15:55.405687+0000","last_active":"2024-04-21T23:16:52.195660+0000","last_peered":"2024-04-21T23:16:52.195660+0000","last_clean":"2024-04-21T23:16:52.195660+0000","last_became_active":"2024-04-21T23:15:55.405379+0000","last_became_peered":"2024-04-21T23:15:55.405379+0000","last_unstale":"2024-04-21T23:16:52.195660+0000","last_undegraded":"2024-04-21T23:16:52.195660+0000","last_fullsized":"2024-04-21T23:16:52.195660+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:12:44.192092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.554132+0000","last_change":"2024-04-21T23:15:56.348989+0000","last_active":"2024-04-21T23:16:52.554132+0000","last_peered":"2024-04-21T23:16:52.554132+0000","last_clean":"2024-04-21T23:16:52.554132+0000","last_became_active":"2024-04-21T23:15:56.348039+0000","last_became_peered":"2024-04-21T23:15:56.348039+0000","last_unstale":"2024-04-21T23:16:52.554132+0000","last_undegraded":"2024-04-21T23:16:52.554132+0000","last_fullsized":"2024-04-21T23:16:52.554132+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:39:15.577567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.198945+0000","last_change":"2024-04-21T23:16:00.434557+0000","last_active":"2024-04-21T23:16:52.198945+0000","last_peered":"2024-04-21T23:16:52.198945+0000","last_clean":"2024-04-21T23:16:52.198945+0000","last_became_active":"2024-04-21T23:16:00.433737+0000","last_became_peered":"2024-04-21T23:16:00.433737+0000","last_unstale":"2024-04-21T23:16:52.198945+0000","last_undegraded":"2024-04-21T23:16:52.198945+0000","last_fullsized":"2024-04-21T23:16:52.198945+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:38:00.295054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"90'1","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.199699+0000","last_change":"2024-04-21T23:15:55.396792+0000","last_active":"2024-04-21T23:16:52.199699+0000","last_peered":"2024-04-21T23:16:52.199699+0000","last_clean":"2024-04-21T23:16:52.199699+0000","last_became_active":"2024-04-21T23:15:55.395740+0000","last_became_peered":"2024-04-21T23:15:55.395740+0000","last_unstale":"2024-04-21T23:16:52.199699+0000","last_undegraded":"2024-04-21T23:16:52.199699+0000","last_fullsized":"2024-04-21T23:16:52.199699+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:39:22.366310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.554076+0000","last_change":"2024-04-21T23:15:56.346372+0000","last_active":"2024-04-21T23:16:52.554076+0000","last_peered":"2024-04-21T23:16:52.554076+0000","last_clean":"2024-04-21T23:16:52.554076+0000","last_became_active":"2024-04-21T23:15:56.346047+0000","last_became_peered":"2024-04-21T23:15:56.346047+0000","last_unstale":"2024-04-21T23:16:52.554076+0000","last_undegraded":"2024-04-21T23:16:52.554076+0000","last_fullsized":"2024-04-21T23:16:52.554076+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:32:29.684227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.199049+0000","last_change":"2024-04-21T23:16:00.434747+0000","last_active":"2024-04-21T23:16:52.199049+0000","last_peered":"2024-04-21T23:16:52.199049+0000","last_clean":"2024-04-21T23:16:52.199049+0000","last_became_active":"2024-04-21T23:16:00.433950+0000","last_became_peered":"2024-04-21T23:16:00.433950+0000","last_unstale":"2024-04-21T23:16:52.199049+0000","last_undegraded":"2024-04-21T23:16:52.199049+0000","last_fullsized":"2024-04-21T23:16:52.199049+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:02:10.562135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.206022+0000","last_change":"2024-04-21T23:16:52.206022+0000","last_active":"2024-04-21T23:16:52.206022+0000","last_peered":"2024-04-21T23:16:52.206022+0000","last_clean":"2024-04-21T23:16:52.206022+0000","last_became_active":"2024-04-21T23:16:52.205469+0000","last_became_peered":"2024-04-21T23:16:52.205469+0000","last_unstale":"2024-04-21T23:16:52.206022+0000","last_undegraded":"2024-04-21T23:16:52.206022+0000","last_fullsized":"2024-04-21T23:16:52.206022+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:34:46.264003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,8],"acting":[5,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'4","reported_seq":103,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:17:32.612737+0000","last_change":"2024-04-21T23:15:55.394900+0000","last_active":"2024-04-21T23:17:32.612737+0000","last_peered":"2024-04-21T23:17:32.612737+0000","last_clean":"2024-04-21T23:17:32.612737+0000","last_became_active":"2024-04-21T23:15:55.393556+0000","last_became_peered":"2024-04-21T23:15:55.393556+0000","last_unstale":"2024-04-21T23:17:32.612737+0000","last_undegraded":"2024-04-21T23:17:32.612737+0000","last_fullsized":"2024-04-21T23:17:32.612737+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:16:24.691930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":72,"num_read_kb":72,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.199791+0000","last_change":"2024-04-21T23:16:00.443338+0000","last_active":"2024-04-21T23:16:52.199791+0000","last_peered":"2024-04-21T23:16:52.199791+0000","last_clean":"2024-04-21T23:16:52.199791+0000","last_became_active":"2024-04-21T23:16:00.442855+0000","last_became_peered":"2024-04-21T23:16:00.442855+0000","last_unstale":"2024-04-21T23:16:52.199791+0000","last_undegraded":"2024-04-21T23:16:52.199791+0000","last_fullsized":"2024-04-21T23:16:52.199791+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:04:36.763027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.347911+0000","last_change":"2024-04-21T23:15:56.356788+0000","last_active":"2024-04-21T23:16:52.347911+0000","last_peered":"2024-04-21T23:16:52.347911+0000","last_clean":"2024-04-21T23:16:52.347911+0000","last_became_active":"2024-04-21T23:15:56.356474+0000","last_became_peered":"2024-04-21T23:15:56.356474+0000","last_unstale":"2024-04-21T23:16:52.347911+0000","last_undegraded":"2024-04-21T23:16:52.347911+0000","last_fullsized":"2024-04-21T23:16:52.347911+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:34:33.700897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"90'3","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.197035+0000","last_change":"2024-04-21T23:15:55.395398+0000","last_active":"2024-04-21T23:16:52.197035+0000","last_peered":"2024-04-21T23:16:52.197035+0000","last_clean":"2024-04-21T23:16:52.197035+0000","last_became_active":"2024-04-21T23:15:55.394978+0000","last_became_peered":"2024-04-21T23:15:55.394978+0000","last_unstale":"2024-04-21T23:16:52.197035+0000","last_undegraded":"2024-04-21T23:16:52.197035+0000","last_fullsized":"2024-04-21T23:16:52.197035+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:07:01.266885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":362,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.198867+0000","last_change":"2024-04-21T23:14:42.877959+0000","last_active":"2024-04-21T23:16:52.198867+0000","last_peered":"2024-04-21T23:16:52.198867+0000","last_clean":"2024-04-21T23:16:52.198867+0000","last_became_active":"2024-04-21T23:14:42.877384+0000","last_became_peered":"2024-04-21T23:14:42.877384+0000","last_unstale":"2024-04-21T23:16:52.198867+0000","last_undegraded":"2024-04-21T23:16:52.198867+0000","last_fullsized":"2024-04-21T23:16:52.198867+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:11:15.009349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:11:15.009349+0000","last_clean_scrub_stamp":"2024-04-21T23:11:15.009349+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:18:55.621052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.196712+0000","last_change":"2024-04-21T23:16:00.444576+0000","last_active":"2024-04-21T23:16:52.196712+0000","last_peered":"2024-04-21T23:16:52.196712+0000","last_clean":"2024-04-21T23:16:52.196712+0000","last_became_active":"2024-04-21T23:16:00.444153+0000","last_became_peered":"2024-04-21T23:16:00.444153+0000","last_unstale":"2024-04-21T23:16:52.196712+0000","last_undegraded":"2024-04-21T23:16:52.196712+0000","last_fullsized":"2024-04-21T23:16:52.196712+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:11:48.040383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.198235+0000","last_change":"2024-04-21T23:15:56.349870+0000","last_active":"2024-04-21T23:16:52.198235+0000","last_peered":"2024-04-21T23:16:52.198235+0000","last_clean":"2024-04-21T23:16:52.198235+0000","last_became_active":"2024-04-21T23:15:56.349535+0000","last_became_peered":"2024-04-21T23:15:56.349535+0000","last_unstale":"2024-04-21T23:16:52.198235+0000","last_undegraded":"2024-04-21T23:16:52.198235+0000","last_fullsized":"2024-04-21T23:16:52.198235+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:12:50.702556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"90'3","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.198934+0000","last_change":"2024-04-21T23:15:55.402406+0000","last_active":"2024-04-21T23:16:52.198934+0000","last_peered":"2024-04-21T23:16:52.198934+0000","last_clean":"2024-04-21T23:16:52.198934+0000","last_became_active":"2024-04-21T23:15:55.402075+0000","last_became_peered":"2024-04-21T23:15:55.402075+0000","last_unstale":"2024-04-21T23:16:52.198934+0000","last_undegraded":"2024-04-21T23:16:52.198934+0000","last_fullsized":"2024-04-21T23:16:52.198934+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:44:18.013031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.199962+0000","last_change":"2024-04-21T23:16:00.439206+0000","last_active":"2024-04-21T23:16:52.199962+0000","last_peered":"2024-04-21T23:16:52.199962+0000","last_clean":"2024-04-21T23:16:52.199962+0000","last_became_active":"2024-04-21T23:16:00.438601+0000","last_became_peered":"2024-04-21T23:16:00.438601+0000","last_unstale":"2024-04-21T23:16:52.199962+0000","last_undegraded":"2024-04-21T23:16:52.199962+0000","last_fullsized":"2024-04-21T23:16:52.199962+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:58:46.771456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"90'1","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.200579+0000","last_change":"2024-04-21T23:15:55.401154+0000","last_active":"2024-04-21T23:16:52.200579+0000","last_peered":"2024-04-21T23:16:52.200579+0000","last_clean":"2024-04-21T23:16:52.200579+0000","last_became_active":"2024-04-21T23:15:55.400579+0000","last_became_peered":"2024-04-21T23:15:55.400579+0000","last_unstale":"2024-04-21T23:16:52.200579+0000","last_undegraded":"2024-04-21T23:16:52.200579+0000","last_fullsized":"2024-04-21T23:16:52.200579+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:20:03.879660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.199056+0000","last_change":"2024-04-21T23:15:56.349837+0000","last_active":"2024-04-21T23:16:52.199056+0000","last_peered":"2024-04-21T23:16:52.199056+0000","last_clean":"2024-04-21T23:16:52.199056+0000","last_became_active":"2024-04-21T23:15:56.349466+0000","last_became_peered":"2024-04-21T23:15:56.349466+0000","last_unstale":"2024-04-21T23:16:52.199056+0000","last_undegraded":"2024-04-21T23:16:52.199056+0000","last_fullsized":"2024-04-21T23:16:52.199056+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:15:32.675770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.347709+0000","last_change":"2024-04-21T23:16:00.432599+0000","last_active":"2024-04-21T23:16:52.347709+0000","last_peered":"2024-04-21T23:16:52.347709+0000","last_clean":"2024-04-21T23:16:52.347709+0000","last_became_active":"2024-04-21T23:16:00.432194+0000","last_became_peered":"2024-04-21T23:16:00.432194+0000","last_unstale":"2024-04-21T23:16:52.347709+0000","last_undegraded":"2024-04-21T23:16:52.347709+0000","last_fullsized":"2024-04-21T23:16:52.347709+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:10:21.354338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'8","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.195989+0000","last_change":"2024-04-21T23:15:55.400328+0000","last_active":"2024-04-21T23:16:52.195989+0000","last_peered":"2024-04-21T23:16:52.195989+0000","last_clean":"2024-04-21T23:16:52.195989+0000","last_became_active":"2024-04-21T23:15:55.399991+0000","last_became_peered":"2024-04-21T23:15:55.399991+0000","last_unstale":"2024-04-21T23:16:52.195989+0000","last_undegraded":"2024-04-21T23:16:52.195989+0000","last_fullsized":"2024-04-21T23:16:52.195989+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:58:54.480720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.348373+0000","last_change":"2024-04-21T23:15:56.358698+0000","last_active":"2024-04-21T23:16:52.348373+0000","last_peered":"2024-04-21T23:16:52.348373+0000","last_clean":"2024-04-21T23:16:52.348373+0000","last_became_active":"2024-04-21T23:15:56.358364+0000","last_became_peered":"2024-04-21T23:15:56.358364+0000","last_unstale":"2024-04-21T23:16:52.348373+0000","last_undegraded":"2024-04-21T23:16:52.348373+0000","last_fullsized":"2024-04-21T23:16:52.348373+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:34:15.281142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.202403+0000","last_change":"2024-04-21T23:16:00.432237+0000","last_active":"2024-04-21T23:16:52.202403+0000","last_peered":"2024-04-21T23:16:52.202403+0000","last_clean":"2024-04-21T23:16:52.202403+0000","last_became_active":"2024-04-21T23:16:00.431745+0000","last_became_peered":"2024-04-21T23:16:00.431745+0000","last_unstale":"2024-04-21T23:16:52.202403+0000","last_undegraded":"2024-04-21T23:16:52.202403+0000","last_fullsized":"2024-04-21T23:16:52.202403+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:25:00.477154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"90'4","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.196551+0000","last_change":"2024-04-21T23:15:55.398667+0000","last_active":"2024-04-21T23:16:52.196551+0000","last_peered":"2024-04-21T23:16:52.196551+0000","last_clean":"2024-04-21T23:16:52.196551+0000","last_became_active":"2024-04-21T23:15:55.398290+0000","last_became_peered":"2024-04-21T23:15:55.398290+0000","last_unstale":"2024-04-21T23:16:52.196551+0000","last_undegraded":"2024-04-21T23:16:52.196551+0000","last_fullsized":"2024-04-21T23:16:52.196551+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:57:58.992615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.202959+0000","last_change":"2024-04-21T23:15:56.359518+0000","last_active":"2024-04-21T23:16:52.202959+0000","last_peered":"2024-04-21T23:16:52.202959+0000","last_clean":"2024-04-21T23:16:52.202959+0000","last_became_active":"2024-04-21T23:15:56.358970+0000","last_became_peered":"2024-04-21T23:15:56.358970+0000","last_unstale":"2024-04-21T23:16:52.202959+0000","last_undegraded":"2024-04-21T23:16:52.202959+0000","last_fullsized":"2024-04-21T23:16:52.202959+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:41:22.113877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.196491+0000","last_change":"2024-04-21T23:16:00.443380+0000","last_active":"2024-04-21T23:16:52.196491+0000","last_peered":"2024-04-21T23:16:52.196491+0000","last_clean":"2024-04-21T23:16:52.196491+0000","last_became_active":"2024-04-21T23:16:00.442896+0000","last_became_peered":"2024-04-21T23:16:00.442896+0000","last_unstale":"2024-04-21T23:16:52.196491+0000","last_undegraded":"2024-04-21T23:16:52.196491+0000","last_fullsized":"2024-04-21T23:16:52.196491+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:51:00.089766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"90'2","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.198995+0000","last_change":"2024-04-21T23:15:55.395287+0000","last_active":"2024-04-21T23:16:52.198995+0000","last_peered":"2024-04-21T23:16:52.198995+0000","last_clean":"2024-04-21T23:16:52.198995+0000","last_became_active":"2024-04-21T23:15:55.394908+0000","last_became_peered":"2024-04-21T23:15:55.394908+0000","last_unstale":"2024-04-21T23:16:52.198995+0000","last_undegraded":"2024-04-21T23:16:52.198995+0000","last_fullsized":"2024-04-21T23:16:52.198995+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:57:29.915581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.199898+0000","last_change":"2024-04-21T23:15:56.348016+0000","last_active":"2024-04-21T23:16:52.199898+0000","last_peered":"2024-04-21T23:16:52.199898+0000","last_clean":"2024-04-21T23:16:52.199898+0000","last_became_active":"2024-04-21T23:15:56.347630+0000","last_became_peered":"2024-04-21T23:15:56.347630+0000","last_unstale":"2024-04-21T23:16:52.199898+0000","last_undegraded":"2024-04-21T23:16:52.199898+0000","last_fullsized":"2024-04-21T23:16:52.199898+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:17:27.372880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.195946+0000","last_change":"2024-04-21T23:16:00.442959+0000","last_active":"2024-04-21T23:16:52.195946+0000","last_peered":"2024-04-21T23:16:52.195946+0000","last_clean":"2024-04-21T23:16:52.195946+0000","last_became_active":"2024-04-21T23:16:00.442319+0000","last_became_peered":"2024-04-21T23:16:00.442319+0000","last_unstale":"2024-04-21T23:16:52.195946+0000","last_undegraded":"2024-04-21T23:16:52.195946+0000","last_fullsized":"2024-04-21T23:16:52.195946+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:59:39.868059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"90'3","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.348067+0000","last_change":"2024-04-21T23:15:55.400859+0000","last_active":"2024-04-21T23:16:52.348067+0000","last_peered":"2024-04-21T23:16:52.348067+0000","last_clean":"2024-04-21T23:16:52.348067+0000","last_became_active":"2024-04-21T23:15:55.400460+0000","last_became_peered":"2024-04-21T23:15:55.400460+0000","last_unstale":"2024-04-21T23:16:52.348067+0000","last_undegraded":"2024-04-21T23:16:52.348067+0000","last_fullsized":"2024-04-21T23:16:52.348067+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:31:14.113476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.198944+0000","last_change":"2024-04-21T23:15:56.350354+0000","last_active":"2024-04-21T23:16:52.198944+0000","last_peered":"2024-04-21T23:16:52.198944+0000","last_clean":"2024-04-21T23:16:52.198944+0000","last_became_active":"2024-04-21T23:15:56.349954+0000","last_became_peered":"2024-04-21T23:15:56.349954+0000","last_unstale":"2024-04-21T23:16:52.198944+0000","last_undegraded":"2024-04-21T23:16:52.198944+0000","last_fullsized":"2024-04-21T23:16:52.198944+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:42:59.165610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.196304+0000","last_change":"2024-04-21T23:16:00.444829+0000","last_active":"2024-04-21T23:16:52.196304+0000","last_peered":"2024-04-21T23:16:52.196304+0000","last_clean":"2024-04-21T23:16:52.196304+0000","last_became_active":"2024-04-21T23:16:00.444365+0000","last_became_peered":"2024-04-21T23:16:00.444365+0000","last_unstale":"2024-04-21T23:16:52.196304+0000","last_undegraded":"2024-04-21T23:16:52.196304+0000","last_fullsized":"2024-04-21T23:16:52.196304+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:23:16.747684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"90'1","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.202176+0000","last_change":"2024-04-21T23:15:55.397855+0000","last_active":"2024-04-21T23:16:52.202176+0000","last_peered":"2024-04-21T23:16:52.202176+0000","last_clean":"2024-04-21T23:16:52.202176+0000","last_became_active":"2024-04-21T23:15:55.397498+0000","last_became_peered":"2024-04-21T23:15:55.397498+0000","last_unstale":"2024-04-21T23:16:52.202176+0000","last_undegraded":"2024-04-21T23:16:52.202176+0000","last_fullsized":"2024-04-21T23:16:52.202176+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:45:44.680350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.348237+0000","last_change":"2024-04-21T23:15:56.353023+0000","last_active":"2024-04-21T23:16:52.348237+0000","last_peered":"2024-04-21T23:16:52.348237+0000","last_clean":"2024-04-21T23:16:52.348237+0000","last_became_active":"2024-04-21T23:15:56.352209+0000","last_became_peered":"2024-04-21T23:15:56.352209+0000","last_unstale":"2024-04-21T23:16:52.348237+0000","last_undegraded":"2024-04-21T23:16:52.348237+0000","last_fullsized":"2024-04-21T23:16:52.348237+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:20:58.075627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.196359+0000","last_change":"2024-04-21T23:16:00.429258+0000","last_active":"2024-04-21T23:16:52.196359+0000","last_peered":"2024-04-21T23:16:52.196359+0000","last_clean":"2024-04-21T23:16:52.196359+0000","last_became_active":"2024-04-21T23:16:00.428856+0000","last_became_peered":"2024-04-21T23:16:00.428856+0000","last_unstale":"2024-04-21T23:16:52.196359+0000","last_undegraded":"2024-04-21T23:16:52.196359+0000","last_fullsized":"2024-04-21T23:16:52.196359+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:07:44.608370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"90'4","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.209766+0000","last_change":"2024-04-21T23:16:52.209766+0000","last_active":"2024-04-21T23:16:52.209766+0000","last_peered":"2024-04-21T23:16:52.209766+0000","last_clean":"2024-04-21T23:16:52.209766+0000","last_became_active":"2024-04-21T23:16:52.204654+0000","last_became_peered":"2024-04-21T23:16:52.204654+0000","last_unstale":"2024-04-21T23:16:52.209766+0000","last_undegraded":"2024-04-21T23:16:52.209766+0000","last_fullsized":"2024-04-21T23:16:52.209766+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:40:41.230460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.199832+0000","last_change":"2024-04-21T23:15:56.352024+0000","last_active":"2024-04-21T23:16:52.199832+0000","last_peered":"2024-04-21T23:16:52.199832+0000","last_clean":"2024-04-21T23:16:52.199832+0000","last_became_active":"2024-04-21T23:15:56.351052+0000","last_became_peered":"2024-04-21T23:15:56.351052+0000","last_unstale":"2024-04-21T23:16:52.199832+0000","last_undegraded":"2024-04-21T23:16:52.199832+0000","last_fullsized":"2024-04-21T23:16:52.199832+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:36:42.777601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.199302+0000","last_change":"2024-04-21T23:16:00.432371+0000","last_active":"2024-04-21T23:16:52.199302+0000","last_peered":"2024-04-21T23:16:52.199302+0000","last_clean":"2024-04-21T23:16:52.199302+0000","last_became_active":"2024-04-21T23:16:00.431828+0000","last_became_peered":"2024-04-21T23:16:00.431828+0000","last_unstale":"2024-04-21T23:16:52.199302+0000","last_undegraded":"2024-04-21T23:16:52.199302+0000","last_fullsized":"2024-04-21T23:16:52.199302+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:02:06.500075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"90'3","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.203819+0000","last_change":"2024-04-21T23:15:55.407018+0000","last_active":"2024-04-21T23:16:52.203819+0000","last_peered":"2024-04-21T23:16:52.203819+0000","last_clean":"2024-04-21T23:16:52.203819+0000","last_became_active":"2024-04-21T23:15:55.406709+0000","last_became_peered":"2024-04-21T23:15:55.406709+0000","last_unstale":"2024-04-21T23:16:52.203819+0000","last_undegraded":"2024-04-21T23:16:52.203819+0000","last_fullsized":"2024-04-21T23:16:52.203819+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:10:40.138470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.200314+0000","last_change":"2024-04-21T23:15:56.358569+0000","last_active":"2024-04-21T23:16:52.200314+0000","last_peered":"2024-04-21T23:16:52.200314+0000","last_clean":"2024-04-21T23:16:52.200314+0000","last_became_active":"2024-04-21T23:15:56.357533+0000","last_became_peered":"2024-04-21T23:15:56.357533+0000","last_unstale":"2024-04-21T23:16:52.200314+0000","last_undegraded":"2024-04-21T23:16:52.200314+0000","last_fullsized":"2024-04-21T23:16:52.200314+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:59:58.525340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.202421+0000","last_change":"2024-04-21T23:16:00.443981+0000","last_active":"2024-04-21T23:16:52.202421+0000","last_peered":"2024-04-21T23:16:52.202421+0000","last_clean":"2024-04-21T23:16:52.202421+0000","last_became_active":"2024-04-21T23:16:00.443368+0000","last_became_peered":"2024-04-21T23:16:00.443368+0000","last_unstale":"2024-04-21T23:16:52.202421+0000","last_undegraded":"2024-04-21T23:16:52.202421+0000","last_fullsized":"2024-04-21T23:16:52.202421+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:56:48.776540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.197952+0000","last_change":"2024-04-21T23:15:55.397334+0000","last_active":"2024-04-21T23:16:52.197952+0000","last_peered":"2024-04-21T23:16:52.197952+0000","last_clean":"2024-04-21T23:16:52.197952+0000","last_became_active":"2024-04-21T23:15:55.396745+0000","last_became_peered":"2024-04-21T23:15:55.396745+0000","last_unstale":"2024-04-21T23:16:52.197952+0000","last_undegraded":"2024-04-21T23:16:52.197952+0000","last_fullsized":"2024-04-21T23:16:52.197952+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:56:21.042187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.198978+0000","last_change":"2024-04-21T23:15:56.351012+0000","last_active":"2024-04-21T23:16:52.198978+0000","last_peered":"2024-04-21T23:16:52.198978+0000","last_clean":"2024-04-21T23:16:52.198978+0000","last_became_active":"2024-04-21T23:15:56.350042+0000","last_became_peered":"2024-04-21T23:15:56.350042+0000","last_unstale":"2024-04-21T23:16:52.198978+0000","last_undegraded":"2024-04-21T23:16:52.198978+0000","last_fullsized":"2024-04-21T23:16:52.198978+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:47:02.629870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.196296+0000","last_change":"2024-04-21T23:16:00.428775+0000","last_active":"2024-04-21T23:16:52.196296+0000","last_peered":"2024-04-21T23:16:52.196296+0000","last_clean":"2024-04-21T23:16:52.196296+0000","last_became_active":"2024-04-21T23:16:00.428278+0000","last_became_peered":"2024-04-21T23:16:00.428278+0000","last_unstale":"2024-04-21T23:16:52.196296+0000","last_undegraded":"2024-04-21T23:16:52.196296+0000","last_fullsized":"2024-04-21T23:16:52.196296+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:24:04.397387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.199165+0000","last_change":"2024-04-21T23:15:55.395524+0000","last_active":"2024-04-21T23:16:52.199165+0000","last_peered":"2024-04-21T23:16:52.199165+0000","last_clean":"2024-04-21T23:16:52.199165+0000","last_became_active":"2024-04-21T23:15:55.395220+0000","last_became_peered":"2024-04-21T23:15:55.395220+0000","last_unstale":"2024-04-21T23:16:52.199165+0000","last_undegraded":"2024-04-21T23:16:52.199165+0000","last_fullsized":"2024-04-21T23:16:52.199165+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:01:37.517620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.200201+0000","last_change":"2024-04-21T23:15:56.352128+0000","last_active":"2024-04-21T23:16:52.200201+0000","last_peered":"2024-04-21T23:16:52.200201+0000","last_clean":"2024-04-21T23:16:52.200201+0000","last_became_active":"2024-04-21T23:15:56.349349+0000","last_became_peered":"2024-04-21T23:15:56.349349+0000","last_unstale":"2024-04-21T23:16:52.200201+0000","last_undegraded":"2024-04-21T23:16:52.200201+0000","last_fullsized":"2024-04-21T23:16:52.200201+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:12:23.247084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.199062+0000","last_change":"2024-04-21T23:15:56.350830+0000","last_active":"2024-04-21T23:16:52.199062+0000","last_peered":"2024-04-21T23:16:52.199062+0000","last_clean":"2024-04-21T23:16:52.199062+0000","last_became_active":"2024-04-21T23:15:56.350385+0000","last_became_peered":"2024-04-21T23:15:56.350385+0000","last_unstale":"2024-04-21T23:16:52.199062+0000","last_undegraded":"2024-04-21T23:16:52.199062+0000","last_fullsized":"2024-04-21T23:16:52.199062+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:18:48.242375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.203499+0000","last_change":"2024-04-21T23:15:56.350947+0000","last_active":"2024-04-21T23:16:52.203499+0000","last_peered":"2024-04-21T23:16:52.203499+0000","last_clean":"2024-04-21T23:16:52.203499+0000","last_became_active":"2024-04-21T23:15:56.350383+0000","last_became_peered":"2024-04-21T23:15:56.350383+0000","last_unstale":"2024-04-21T23:16:52.203499+0000","last_undegraded":"2024-04-21T23:16:52.203499+0000","last_fullsized":"2024-04-21T23:16:52.203499+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:07:28.879822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.202013+0000","last_change":"2024-04-21T23:15:56.354559+0000","last_active":"2024-04-21T23:16:52.202013+0000","last_peered":"2024-04-21T23:16:52.202013+0000","last_clean":"2024-04-21T23:16:52.202013+0000","last_became_active":"2024-04-21T23:15:56.354205+0000","last_became_peered":"2024-04-21T23:15:56.354205+0000","last_unstale":"2024-04-21T23:16:52.202013+0000","last_undegraded":"2024-04-21T23:16:52.202013+0000","last_fullsized":"2024-04-21T23:16:52.202013+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:33:50.157971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.202624+0000","last_change":"2024-04-21T23:15:56.358395+0000","last_active":"2024-04-21T23:16:52.202624+0000","last_peered":"2024-04-21T23:16:52.202624+0000","last_clean":"2024-04-21T23:16:52.202624+0000","last_became_active":"2024-04-21T23:15:56.358077+0000","last_became_peered":"2024-04-21T23:15:56.358077+0000","last_unstale":"2024-04-21T23:16:52.202624+0000","last_undegraded":"2024-04-21T23:16:52.202624+0000","last_fullsized":"2024-04-21T23:16:52.202624+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:12:43.124588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.199048+0000","last_change":"2024-04-21T23:15:56.347476+0000","last_active":"2024-04-21T23:16:52.199048+0000","last_peered":"2024-04-21T23:16:52.199048+0000","last_clean":"2024-04-21T23:16:52.199048+0000","last_became_active":"2024-04-21T23:15:56.347097+0000","last_became_peered":"2024-04-21T23:15:56.347097+0000","last_unstale":"2024-04-21T23:16:52.199048+0000","last_undegraded":"2024-04-21T23:16:52.199048+0000","last_fullsized":"2024-04-21T23:16:52.199048+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:11:58.715526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.199918+0000","last_change":"2024-04-21T23:15:56.346413+0000","last_active":"2024-04-21T23:16:52.199918+0000","last_peered":"2024-04-21T23:16:52.199918+0000","last_clean":"2024-04-21T23:16:52.199918+0000","last_became_active":"2024-04-21T23:15:56.346056+0000","last_became_peered":"2024-04-21T23:15:56.346056+0000","last_unstale":"2024-04-21T23:16:52.199918+0000","last_undegraded":"2024-04-21T23:16:52.199918+0000","last_fullsized":"2024-04-21T23:16:52.199918+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:47:16.340323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.204480+0000","last_change":"2024-04-21T23:15:56.354876+0000","last_active":"2024-04-21T23:16:52.204480+0000","last_peered":"2024-04-21T23:16:52.204480+0000","last_clean":"2024-04-21T23:16:52.204480+0000","last_became_active":"2024-04-21T23:15:56.354525+0000","last_became_peered":"2024-04-21T23:15:56.354525+0000","last_unstale":"2024-04-21T23:16:52.204480+0000","last_undegraded":"2024-04-21T23:16:52.204480+0000","last_fullsized":"2024-04-21T23:16:52.204480+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:09:23.662664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.348477+0000","last_change":"2024-04-21T23:15:56.358456+0000","last_active":"2024-04-21T23:16:52.348477+0000","last_peered":"2024-04-21T23:16:52.348477+0000","last_clean":"2024-04-21T23:16:52.348477+0000","last_became_active":"2024-04-21T23:15:56.358060+0000","last_became_peered":"2024-04-21T23:15:56.358060+0000","last_unstale":"2024-04-21T23:16:52.348477+0000","last_undegraded":"2024-04-21T23:16:52.348477+0000","last_fullsized":"2024-04-21T23:16:52.348477+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:47:58.928438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.196453+0000","last_change":"2024-04-21T23:15:56.352870+0000","last_active":"2024-04-21T23:16:52.196453+0000","last_peered":"2024-04-21T23:16:52.196453+0000","last_clean":"2024-04-21T23:16:52.196453+0000","last_became_active":"2024-04-21T23:15:56.352267+0000","last_became_peered":"2024-04-21T23:15:56.352267+0000","last_unstale":"2024-04-21T23:16:52.196453+0000","last_undegraded":"2024-04-21T23:16:52.196453+0000","last_fullsized":"2024-04-21T23:16:52.196453+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:32:22.244773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.348178+0000","last_change":"2024-04-21T23:15:56.357468+0000","last_active":"2024-04-21T23:16:52.348178+0000","last_peered":"2024-04-21T23:16:52.348178+0000","last_clean":"2024-04-21T23:16:52.348178+0000","last_became_active":"2024-04-21T23:15:56.357104+0000","last_became_peered":"2024-04-21T23:15:56.357104+0000","last_unstale":"2024-04-21T23:16:52.348178+0000","last_undegraded":"2024-04-21T23:16:52.348178+0000","last_fullsized":"2024-04-21T23:16:52.348178+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:37:36.884036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.203142+0000","last_change":"2024-04-21T23:16:52.203142+0000","last_active":"2024-04-21T23:16:52.203142+0000","last_peered":"2024-04-21T23:16:52.203142+0000","last_clean":"2024-04-21T23:16:52.203142+0000","last_became_active":"2024-04-21T23:16:52.202558+0000","last_became_peered":"2024-04-21T23:16:52.202558+0000","last_unstale":"2024-04-21T23:16:52.203142+0000","last_undegraded":"2024-04-21T23:16:52.203142+0000","last_fullsized":"2024-04-21T23:16:52.203142+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:29:05.428123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,8],"acting":[1,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.199231+0000","last_change":"2024-04-21T23:15:56.351901+0000","last_active":"2024-04-21T23:16:52.199231+0000","last_peered":"2024-04-21T23:16:52.199231+0000","last_clean":"2024-04-21T23:16:52.199231+0000","last_became_active":"2024-04-21T23:15:56.351545+0000","last_became_peered":"2024-04-21T23:15:56.351545+0000","last_unstale":"2024-04-21T23:16:52.199231+0000","last_undegraded":"2024-04-21T23:16:52.199231+0000","last_fullsized":"2024-04-21T23:16:52.199231+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:37:20.855631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.199333+0000","last_change":"2024-04-21T23:15:56.355773+0000","last_active":"2024-04-21T23:16:52.199333+0000","last_peered":"2024-04-21T23:16:52.199333+0000","last_clean":"2024-04-21T23:16:52.199333+0000","last_became_active":"2024-04-21T23:15:56.355269+0000","last_became_peered":"2024-04-21T23:15:56.355269+0000","last_unstale":"2024-04-21T23:16:52.199333+0000","last_undegraded":"2024-04-21T23:16:52.199333+0000","last_fullsized":"2024-04-21T23:16:52.199333+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:04:54.540817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":65776,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":457,"num_read_kb":296,"num_write":159,"num_write_kb":113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":112,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":197328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":138,"ondisk_log_size":138,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351010,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1044480,"data_stored":507590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612647,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37028,"kb_used_data":1684,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708124,"statfs":{"total":95995035648,"available":95957118976,"internally_reserved":0,"allocated":1724416,"data_stored":1156912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808876,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32212,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712940,"statfs":{"total":95995035648,"available":95962050560,"internally_reserved":0,"allocated":1056768,"data_stored":511810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070513,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1044480,"data_stored":507522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332150,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28628,"kb_used_data":1608,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716524,"statfs":{"total":95995035648,"available":95965720576,"internally_reserved":0,"allocated":1646592,"data_stored":1103543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495675,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1032192,"data_stored":502890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574933,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1097728,"data_stored":564575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738457,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1056768,"data_stored":504278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411408,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28116,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717036,"statfs":{"total":95995035648,"available":95966244864,"internally_reserved":0,"allocated":1122304,"data_stored":565925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182474,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36360,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708792,"statfs":{"total":95995035648,"available":95957803008,"internally_reserved":0,"allocated":1044480,"data_stored":503084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920838,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1044480,"data_stored":507931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757312,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36960,"kb_used_data":1616,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708192,"statfs":{"total":95995035648,"available":95957188608,"internally_reserved":0,"allocated":1654784,"data_stored":1098740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":61775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":63125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":63744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-21T23:17:37.576 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-21T23:17:37.830 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:17:37.931 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:37 smithi040 ceph-mon[27984]: pgmap v345: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T23:17:37.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:37 smithi040 ceph-mon[27984]: from='client.15111 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T23:17:38.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:37 smithi174 ceph-mon[31409]: pgmap v345: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T23:17:38.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:37 smithi174 ceph-mon[31409]: from='client.15111 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T23:17:38.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:37 smithi111 ceph-mon[31085]: pgmap v345: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T23:17:38.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:37 smithi111 ceph-mon[31085]: from='client.15111 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T23:17:39.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:39.630+0000 7f735f8f3700 1 -- 172.21.15.40:0/2554853146 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f735806ac40 msgr2=0x7f735806b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:39.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:39.630+0000 7f735f8f3700 1 --2- 172.21.15.40:0/2554853146 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f735806ac40 0x7f735806b020 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f73540099c0 tx=0x7f7354009cd0 comp rx=0 tx=0).stop 2024-04-21T23:17:39.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:39.630+0000 7f735f8f3700 1 -- 172.21.15.40:0/2554853146 shutdown_connections 2024-04-21T23:17:39.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:39.630+0000 7f735f8f3700 1 --2- 172.21.15.40:0/2554853146 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7358107590 0x7f735810f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:39.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:39.630+0000 7f735f8f3700 1 --2- 172.21.15.40:0/2554853146 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f735806ac40 0x7f735806b020 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:39.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:39.630+0000 7f735f8f3700 1 --2- 172.21.15.40:0/2554853146 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f735806b5f0 0x7f7358106eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:39.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:39.630+0000 7f735f8f3700 1 -- 172.21.15.40:0/2554853146 >> 172.21.15.40:0/2554853146 conn(0x7f73580781d0 msgr2=0x7f73580785e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:39.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:39.631+0000 7f735f8f3700 1 -- 172.21.15.40:0/2554853146 shutdown_connections 2024-04-21T23:17:39.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:39.631+0000 7f735f8f3700 1 -- 172.21.15.40:0/2554853146 wait complete. 2024-04-21T23:17:39.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:39.631+0000 7f735f8f3700 1 Processor -- start 2024-04-21T23:17:39.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:39.631+0000 7f735f8f3700 1 -- start start 2024-04-21T23:17:39.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:39.632+0000 7f735f8f3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f735806ac40 0x7f735819e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:39.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:39.632+0000 7f735f8f3700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f735806b5f0 0x7f735819ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:39.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:39.632+0000 7f735f8f3700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7358107590 0x7f7358198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:39.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:39.632+0000 7f735f8f3700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7358112670 con 0x7f735806b5f0 2024-04-21T23:17:39.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:39.632+0000 7f735f8f3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f73581124f0 con 0x7f735806ac40 2024-04-21T23:17:39.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:39.632+0000 7f735f8f3700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f73581127f0 con 0x7f7358107590 2024-04-21T23:17:39.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:39.632+0000 7f735ce8e700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f735806b5f0 0x7f735819ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:39.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:39.632+0000 7f735ce8e700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f735806b5f0 0x7f735819ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42710/0 (socket says 172.21.15.40:42710) 2024-04-21T23:17:39.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:39.632+0000 7f735ce8e700 1 -- 172.21.15.40:0/2743428193 learned_addr learned my addr 172.21.15.40:0/2743428193 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:17:39.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:39.632+0000 7f735d68f700 1 --2- 172.21.15.40:0/2743428193 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f735806ac40 0x7f735819e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:39.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:39.632+0000 7f735de90700 1 --2- 172.21.15.40:0/2743428193 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7358107590 0x7f7358198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:39.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:39.632+0000 7f735ce8e700 1 -- 172.21.15.40:0/2743428193 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7358107590 msgr2=0x7f7358198970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:39.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:39.632+0000 7f735ce8e700 1 --2- 172.21.15.40:0/2743428193 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7358107590 0x7f7358198970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:39.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:39.632+0000 7f735ce8e700 1 -- 172.21.15.40:0/2743428193 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f735806ac40 msgr2=0x7f735819e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:39.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:39.632+0000 7f735ce8e700 1 --2- 172.21.15.40:0/2743428193 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f735806ac40 0x7f735819e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:39.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:39.632+0000 7f735ce8e700 1 -- 172.21.15.40:0/2743428193 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73581990d0 con 0x7f735806b5f0 2024-04-21T23:17:39.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:39.632+0000 7f735ce8e700 1 --2- 172.21.15.40:0/2743428193 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f735806b5f0 0x7f735819ecf0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f734800b500 tx=0x7f734800b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:39.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:39.632+0000 7f734e7fc700 1 -- 172.21.15.40:0/2743428193 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7348015020 con 0x7f735806b5f0 2024-04-21T23:17:39.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:39.633+0000 7f735f8f3700 1 -- 172.21.15.40:0/2743428193 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73581993c0 con 0x7f735806b5f0 2024-04-21T23:17:39.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:39.633+0000 7f734e7fc700 1 -- 172.21.15.40:0/2743428193 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7348011d10 con 0x7f735806b5f0 2024-04-21T23:17:39.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:39.633+0000 7f734e7fc700 1 -- 172.21.15.40:0/2743428193 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f734800f2e0 con 0x7f735806b5f0 2024-04-21T23:17:39.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:39.633+0000 7f735f8f3700 1 -- 172.21.15.40:0/2743428193 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73581a5680 con 0x7f735806b5f0 2024-04-21T23:17:39.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:39.634+0000 7f735f8f3700 1 -- 172.21.15.40:0/2743428193 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7358068f20 con 0x7f735806b5f0 2024-04-21T23:17:39.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:39.635+0000 7f734e7fc700 1 -- 172.21.15.40:0/2743428193 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f7348018020 con 0x7f735806b5f0 2024-04-21T23:17:39.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:39.635+0000 7f734e7fc700 1 --2- 172.21.15.40:0/2743428193 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f73440a2cb0 0x7f73440a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:39.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:39.636+0000 7f735d68f700 1 --2- 172.21.15.40:0/2743428193 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f73440a2cb0 0x7f73440a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:39.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:39.636+0000 7f734e7fc700 1 -- 172.21.15.40:0/2743428193 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f73480cec30 con 0x7f735806b5f0 2024-04-21T23:17:39.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:39.636+0000 7f735d68f700 1 --2- 172.21.15.40:0/2743428193 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f73440a2cb0 0x7f73440a5170 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f7354009990 tx=0x7f7354005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:39.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:39.640+0000 7f734e7fc700 1 -- 172.21.15.40:0/2743428193 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f734801f020 con 0x7f735806b5f0 2024-04-21T23:17:39.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:39.843+0000 7f735f8f3700 1 -- 172.21.15.40:0/2743428193 --> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7358062670 con 0x7f73440a2cb0 2024-04-21T23:17:39.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:39.853+0000 7f734e7fc700 1 -- 172.21.15.40:0/2743428193 <== mgr.14148 v2:172.21.15.40:6800/2500760788 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340368 (secure 0 0 0) 0x7f7358062670 con 0x7f73440a2cb0 2024-04-21T23:17:39.856 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:17:39.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:39.857+0000 7f735f8f3700 1 -- 172.21.15.40:0/2743428193 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f73440a2cb0 msgr2=0x7f73440a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:39.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:39.857+0000 7f735f8f3700 1 --2- 172.21.15.40:0/2743428193 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f73440a2cb0 0x7f73440a5170 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f7354009990 tx=0x7f7354005fb0 comp rx=0 tx=0).stop 2024-04-21T23:17:39.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:39.857+0000 7f735f8f3700 1 -- 172.21.15.40:0/2743428193 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f735806b5f0 msgr2=0x7f735819ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:39.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:39.857+0000 7f735f8f3700 1 --2- 172.21.15.40:0/2743428193 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f735806b5f0 0x7f735819ecf0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f734800b500 tx=0x7f734800b810 comp rx=0 tx=0).stop 2024-04-21T23:17:39.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:39.857+0000 7f735f8f3700 1 -- 172.21.15.40:0/2743428193 shutdown_connections 2024-04-21T23:17:39.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:39.857+0000 7f735f8f3700 1 --2- 172.21.15.40:0/2743428193 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7358107590 0x7f7358198970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:39.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:39.857+0000 7f735f8f3700 1 --2- 172.21.15.40:0/2743428193 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f73440a2cb0 0x7f73440a5170 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:39.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:39.857+0000 7f735f8f3700 1 --2- 172.21.15.40:0/2743428193 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f735806ac40 0x7f735819e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:39.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:39.857+0000 7f735f8f3700 1 --2- 172.21.15.40:0/2743428193 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f735806b5f0 0x7f735819ecf0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:39.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:39.857+0000 7f735f8f3700 1 -- 172.21.15.40:0/2743428193 >> 172.21.15.40:0/2743428193 conn(0x7f73580781d0 msgr2=0x7f7358106720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:39.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:39.857+0000 7f735f8f3700 1 -- 172.21.15.40:0/2743428193 shutdown_connections 2024-04-21T23:17:39.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:39.857+0000 7f735f8f3700 1 -- 172.21.15.40:0/2743428193 wait complete. 2024-04-21T23:17:39.860 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-21T23:17:40.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:39 smithi040 ceph-mon[27984]: pgmap v346: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:17:40.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:39 smithi174 ceph-mon[31409]: pgmap v346: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:17:40.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:39 smithi111 ceph-mon[31085]: pgmap v346: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:17:40.789 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":346,"stamp":"2024-04-21T23:17:38.422647+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":656378,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":593,"num_read_kb":415,"num_write":342,"num_write_kb":3053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1180848,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":277,"ondisk_log_size":277,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":367600,"kb_used_data":14228,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124574224,"statfs":{"total":1151940427776,"available":1151564005376,"internally_reserved":0,"allocated":14569472,"data_stored":8034800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":361674042},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":60,"num_read_kb":30,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003426"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.199761+0000","last_change":"2024-04-21T23:15:56.349368+0000","last_active":"2024-04-21T23:16:52.199761+0000","last_peered":"2024-04-21T23:16:52.199761+0000","last_clean":"2024-04-21T23:16:52.199761+0000","last_became_active":"2024-04-21T23:15:56.345743+0000","last_became_peered":"2024-04-21T23:15:56.345743+0000","last_unstale":"2024-04-21T23:16:52.199761+0000","last_undegraded":"2024-04-21T23:16:52.199761+0000","last_fullsized":"2024-04-21T23:16:52.199761+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:57:21.574012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.203016+0000","last_change":"2024-04-21T23:15:56.347231+0000","last_active":"2024-04-21T23:16:52.203016+0000","last_peered":"2024-04-21T23:16:52.203016+0000","last_clean":"2024-04-21T23:16:52.203016+0000","last_became_active":"2024-04-21T23:15:56.346847+0000","last_became_peered":"2024-04-21T23:15:56.346847+0000","last_unstale":"2024-04-21T23:16:52.203016+0000","last_undegraded":"2024-04-21T23:16:52.203016+0000","last_fullsized":"2024-04-21T23:16:52.203016+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:20:15.884789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.202557+0000","last_change":"2024-04-21T23:15:56.354897+0000","last_active":"2024-04-21T23:16:52.202557+0000","last_peered":"2024-04-21T23:16:52.202557+0000","last_clean":"2024-04-21T23:16:52.202557+0000","last_became_active":"2024-04-21T23:15:56.354494+0000","last_became_peered":"2024-04-21T23:15:56.354494+0000","last_unstale":"2024-04-21T23:16:52.202557+0000","last_undegraded":"2024-04-21T23:16:52.202557+0000","last_fullsized":"2024-04-21T23:16:52.202557+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:51:11.865310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.196805+0000","last_change":"2024-04-21T23:15:56.355184+0000","last_active":"2024-04-21T23:16:52.196805+0000","last_peered":"2024-04-21T23:16:52.196805+0000","last_clean":"2024-04-21T23:16:52.196805+0000","last_became_active":"2024-04-21T23:15:56.354748+0000","last_became_peered":"2024-04-21T23:15:56.354748+0000","last_unstale":"2024-04-21T23:16:52.196805+0000","last_undegraded":"2024-04-21T23:16:52.196805+0000","last_fullsized":"2024-04-21T23:16:52.196805+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:46:24.867488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.200037+0000","last_change":"2024-04-21T23:15:56.349446+0000","last_active":"2024-04-21T23:16:52.200037+0000","last_peered":"2024-04-21T23:16:52.200037+0000","last_clean":"2024-04-21T23:16:52.200037+0000","last_became_active":"2024-04-21T23:15:56.349050+0000","last_became_peered":"2024-04-21T23:15:56.349050+0000","last_unstale":"2024-04-21T23:16:52.200037+0000","last_undegraded":"2024-04-21T23:16:52.200037+0000","last_fullsized":"2024-04-21T23:16:52.200037+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:15:25.489044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.554066+0000","last_change":"2024-04-21T23:15:56.356095+0000","last_active":"2024-04-21T23:16:52.554066+0000","last_peered":"2024-04-21T23:16:52.554066+0000","last_clean":"2024-04-21T23:16:52.554066+0000","last_became_active":"2024-04-21T23:15:56.355754+0000","last_became_peered":"2024-04-21T23:15:56.355754+0000","last_unstale":"2024-04-21T23:16:52.554066+0000","last_undegraded":"2024-04-21T23:16:52.554066+0000","last_fullsized":"2024-04-21T23:16:52.554066+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:00:27.296457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.199230+0000","last_change":"2024-04-21T23:15:56.349733+0000","last_active":"2024-04-21T23:16:52.199230+0000","last_peered":"2024-04-21T23:16:52.199230+0000","last_clean":"2024-04-21T23:16:52.199230+0000","last_became_active":"2024-04-21T23:15:56.347820+0000","last_became_peered":"2024-04-21T23:15:56.347820+0000","last_unstale":"2024-04-21T23:16:52.199230+0000","last_undegraded":"2024-04-21T23:16:52.199230+0000","last_fullsized":"2024-04-21T23:16:52.199230+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:31:34.453626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.347818+0000","last_change":"2024-04-21T23:15:56.359385+0000","last_active":"2024-04-21T23:16:52.347818+0000","last_peered":"2024-04-21T23:16:52.347818+0000","last_clean":"2024-04-21T23:16:52.347818+0000","last_became_active":"2024-04-21T23:15:56.359074+0000","last_became_peered":"2024-04-21T23:15:56.359074+0000","last_unstale":"2024-04-21T23:16:52.347818+0000","last_undegraded":"2024-04-21T23:16:52.347818+0000","last_fullsized":"2024-04-21T23:16:52.347818+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:40:23.904592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.200243+0000","last_change":"2024-04-21T23:15:56.350886+0000","last_active":"2024-04-21T23:16:52.200243+0000","last_peered":"2024-04-21T23:16:52.200243+0000","last_clean":"2024-04-21T23:16:52.200243+0000","last_became_active":"2024-04-21T23:15:56.350531+0000","last_became_peered":"2024-04-21T23:15:56.350531+0000","last_unstale":"2024-04-21T23:16:52.200243+0000","last_undegraded":"2024-04-21T23:16:52.200243+0000","last_fullsized":"2024-04-21T23:16:52.200243+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:13:12.354473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.554360+0000","last_change":"2024-04-21T23:15:56.347376+0000","last_active":"2024-04-21T23:16:52.554360+0000","last_peered":"2024-04-21T23:16:52.554360+0000","last_clean":"2024-04-21T23:16:52.554360+0000","last_became_active":"2024-04-21T23:15:56.347016+0000","last_became_peered":"2024-04-21T23:15:56.347016+0000","last_unstale":"2024-04-21T23:16:52.554360+0000","last_undegraded":"2024-04-21T23:16:52.554360+0000","last_fullsized":"2024-04-21T23:16:52.554360+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:49:46.722931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.200197+0000","last_change":"2024-04-21T23:15:56.351754+0000","last_active":"2024-04-21T23:16:52.200197+0000","last_peered":"2024-04-21T23:16:52.200197+0000","last_clean":"2024-04-21T23:16:52.200197+0000","last_became_active":"2024-04-21T23:15:56.349860+0000","last_became_peered":"2024-04-21T23:15:56.349860+0000","last_unstale":"2024-04-21T23:16:52.200197+0000","last_undegraded":"2024-04-21T23:16:52.200197+0000","last_fullsized":"2024-04-21T23:16:52.200197+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:48:41.469708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.196675+0000","last_change":"2024-04-21T23:15:56.345177+0000","last_active":"2024-04-21T23:16:52.196675+0000","last_peered":"2024-04-21T23:16:52.196675+0000","last_clean":"2024-04-21T23:16:52.196675+0000","last_became_active":"2024-04-21T23:15:56.344791+0000","last_became_peered":"2024-04-21T23:15:56.344791+0000","last_unstale":"2024-04-21T23:16:52.196675+0000","last_undegraded":"2024-04-21T23:16:52.196675+0000","last_fullsized":"2024-04-21T23:16:52.196675+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:33:33.418383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.556505+0000","last_change":"2024-04-21T23:16:52.556505+0000","last_active":"2024-04-21T23:16:52.556505+0000","last_peered":"2024-04-21T23:16:52.556505+0000","last_clean":"2024-04-21T23:16:52.556505+0000","last_became_active":"2024-04-21T23:16:52.556177+0000","last_became_peered":"2024-04-21T23:16:52.556177+0000","last_unstale":"2024-04-21T23:16:52.556505+0000","last_undegraded":"2024-04-21T23:16:52.556505+0000","last_fullsized":"2024-04-21T23:16:52.556505+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:17:13.546746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.202410+0000","last_change":"2024-04-21T23:15:56.354544+0000","last_active":"2024-04-21T23:16:52.202410+0000","last_peered":"2024-04-21T23:16:52.202410+0000","last_clean":"2024-04-21T23:16:52.202410+0000","last_became_active":"2024-04-21T23:15:56.354206+0000","last_became_peered":"2024-04-21T23:15:56.354206+0000","last_unstale":"2024-04-21T23:16:52.202410+0000","last_undegraded":"2024-04-21T23:16:52.202410+0000","last_fullsized":"2024-04-21T23:16:52.202410+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:37:19.985422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.202201+0000","last_change":"2024-04-21T23:15:56.357447+0000","last_active":"2024-04-21T23:16:52.202201+0000","last_peered":"2024-04-21T23:16:52.202201+0000","last_clean":"2024-04-21T23:16:52.202201+0000","last_became_active":"2024-04-21T23:15:56.355850+0000","last_became_peered":"2024-04-21T23:15:56.355850+0000","last_unstale":"2024-04-21T23:16:52.202201+0000","last_undegraded":"2024-04-21T23:16:52.202201+0000","last_fullsized":"2024-04-21T23:16:52.202201+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:48:56.648994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.198855+0000","last_change":"2024-04-21T23:15:56.353712+0000","last_active":"2024-04-21T23:16:52.198855+0000","last_peered":"2024-04-21T23:16:52.198855+0000","last_clean":"2024-04-21T23:16:52.198855+0000","last_became_active":"2024-04-21T23:15:56.353388+0000","last_became_peered":"2024-04-21T23:15:56.353388+0000","last_unstale":"2024-04-21T23:16:52.198855+0000","last_undegraded":"2024-04-21T23:16:52.198855+0000","last_fullsized":"2024-04-21T23:16:52.198855+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:43:33.581222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.199691+0000","last_change":"2024-04-21T23:15:56.347041+0000","last_active":"2024-04-21T23:16:52.199691+0000","last_peered":"2024-04-21T23:16:52.199691+0000","last_clean":"2024-04-21T23:16:52.199691+0000","last_became_active":"2024-04-21T23:15:56.346630+0000","last_became_peered":"2024-04-21T23:15:56.346630+0000","last_unstale":"2024-04-21T23:16:52.199691+0000","last_undegraded":"2024-04-21T23:16:52.199691+0000","last_fullsized":"2024-04-21T23:16:52.199691+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:57:42.553666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.195943+0000","last_change":"2024-04-21T23:15:56.353809+0000","last_active":"2024-04-21T23:16:52.195943+0000","last_peered":"2024-04-21T23:16:52.195943+0000","last_clean":"2024-04-21T23:16:52.195943+0000","last_became_active":"2024-04-21T23:15:56.353469+0000","last_became_peered":"2024-04-21T23:15:56.353469+0000","last_unstale":"2024-04-21T23:16:52.195943+0000","last_undegraded":"2024-04-21T23:16:52.195943+0000","last_fullsized":"2024-04-21T23:16:52.195943+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:11:14.136845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.204480+0000","last_change":"2024-04-21T23:15:56.354500+0000","last_active":"2024-04-21T23:16:52.204480+0000","last_peered":"2024-04-21T23:16:52.204480+0000","last_clean":"2024-04-21T23:16:52.204480+0000","last_became_active":"2024-04-21T23:15:56.353893+0000","last_became_peered":"2024-04-21T23:15:56.353893+0000","last_unstale":"2024-04-21T23:16:52.204480+0000","last_undegraded":"2024-04-21T23:16:52.204480+0000","last_fullsized":"2024-04-21T23:16:52.204480+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:14:38.121448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.203524+0000","last_change":"2024-04-21T23:16:00.438245+0000","last_active":"2024-04-21T23:16:52.203524+0000","last_peered":"2024-04-21T23:16:52.203524+0000","last_clean":"2024-04-21T23:16:52.203524+0000","last_became_active":"2024-04-21T23:16:00.437755+0000","last_became_peered":"2024-04-21T23:16:00.437755+0000","last_unstale":"2024-04-21T23:16:52.203524+0000","last_undegraded":"2024-04-21T23:16:52.203524+0000","last_fullsized":"2024-04-21T23:16:52.203524+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:33:37.273223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"90'9","reported_seq":37,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.196046+0000","last_change":"2024-04-21T23:15:55.402553+0000","last_active":"2024-04-21T23:16:52.196046+0000","last_peered":"2024-04-21T23:16:52.196046+0000","last_clean":"2024-04-21T23:16:52.196046+0000","last_became_active":"2024-04-21T23:15:55.402192+0000","last_became_peered":"2024-04-21T23:15:55.402192+0000","last_unstale":"2024-04-21T23:16:52.196046+0000","last_undegraded":"2024-04-21T23:16:52.196046+0000","last_fullsized":"2024-04-21T23:16:52.196046+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:24:57.467906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.348222+0000","last_change":"2024-04-21T23:15:56.358797+0000","last_active":"2024-04-21T23:16:52.348222+0000","last_peered":"2024-04-21T23:16:52.348222+0000","last_clean":"2024-04-21T23:16:52.348222+0000","last_became_active":"2024-04-21T23:15:56.357997+0000","last_became_peered":"2024-04-21T23:15:56.357997+0000","last_unstale":"2024-04-21T23:16:52.348222+0000","last_undegraded":"2024-04-21T23:16:52.348222+0000","last_fullsized":"2024-04-21T23:16:52.348222+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:20:50.891506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.196189+0000","last_change":"2024-04-21T23:16:00.443368+0000","last_active":"2024-04-21T23:16:52.196189+0000","last_peered":"2024-04-21T23:16:52.196189+0000","last_clean":"2024-04-21T23:16:52.196189+0000","last_became_active":"2024-04-21T23:16:00.442940+0000","last_became_peered":"2024-04-21T23:16:00.442940+0000","last_unstale":"2024-04-21T23:16:52.196189+0000","last_undegraded":"2024-04-21T23:16:52.196189+0000","last_fullsized":"2024-04-21T23:16:52.196189+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:17:15.706460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"90'2","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.348606+0000","last_change":"2024-04-21T23:15:55.405216+0000","last_active":"2024-04-21T23:16:52.348606+0000","last_peered":"2024-04-21T23:16:52.348606+0000","last_clean":"2024-04-21T23:16:52.348606+0000","last_became_active":"2024-04-21T23:15:55.404844+0000","last_became_peered":"2024-04-21T23:15:55.404844+0000","last_unstale":"2024-04-21T23:16:52.348606+0000","last_undegraded":"2024-04-21T23:16:52.348606+0000","last_fullsized":"2024-04-21T23:16:52.348606+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:56:28.691844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.202921+0000","last_change":"2024-04-21T23:15:56.355789+0000","last_active":"2024-04-21T23:16:52.202921+0000","last_peered":"2024-04-21T23:16:52.202921+0000","last_clean":"2024-04-21T23:16:52.202921+0000","last_became_active":"2024-04-21T23:15:56.354972+0000","last_became_peered":"2024-04-21T23:15:56.354972+0000","last_unstale":"2024-04-21T23:16:52.202921+0000","last_undegraded":"2024-04-21T23:16:52.202921+0000","last_fullsized":"2024-04-21T23:16:52.202921+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:11:36.831453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.196834+0000","last_change":"2024-04-21T23:16:00.443241+0000","last_active":"2024-04-21T23:16:52.196834+0000","last_peered":"2024-04-21T23:16:52.196834+0000","last_clean":"2024-04-21T23:16:52.196834+0000","last_became_active":"2024-04-21T23:16:00.442820+0000","last_became_peered":"2024-04-21T23:16:00.442820+0000","last_unstale":"2024-04-21T23:16:52.196834+0000","last_undegraded":"2024-04-21T23:16:52.196834+0000","last_fullsized":"2024-04-21T23:16:52.196834+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:28:04.032621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"90'4","reported_seq":43,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.212906+0000","last_change":"2024-04-21T23:16:52.212906+0000","last_active":"2024-04-21T23:16:52.212906+0000","last_peered":"2024-04-21T23:16:52.212906+0000","last_clean":"2024-04-21T23:16:52.212906+0000","last_became_active":"2024-04-21T23:16:52.205828+0000","last_became_peered":"2024-04-21T23:16:52.205828+0000","last_unstale":"2024-04-21T23:16:52.212906+0000","last_undegraded":"2024-04-21T23:16:52.212906+0000","last_fullsized":"2024-04-21T23:16:52.212906+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:20:01.936578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.199572+0000","last_change":"2024-04-21T23:15:56.354462+0000","last_active":"2024-04-21T23:16:52.199572+0000","last_peered":"2024-04-21T23:16:52.199572+0000","last_clean":"2024-04-21T23:16:52.199572+0000","last_became_active":"2024-04-21T23:15:56.354134+0000","last_became_peered":"2024-04-21T23:15:56.354134+0000","last_unstale":"2024-04-21T23:16:52.199572+0000","last_undegraded":"2024-04-21T23:16:52.199572+0000","last_fullsized":"2024-04-21T23:16:52.199572+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:27:40.809483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.196258+0000","last_change":"2024-04-21T23:16:00.427717+0000","last_active":"2024-04-21T23:16:52.196258+0000","last_peered":"2024-04-21T23:16:52.196258+0000","last_clean":"2024-04-21T23:16:52.196258+0000","last_became_active":"2024-04-21T23:16:00.427296+0000","last_became_peered":"2024-04-21T23:16:00.427296+0000","last_unstale":"2024-04-21T23:16:52.196258+0000","last_undegraded":"2024-04-21T23:16:52.196258+0000","last_fullsized":"2024-04-21T23:16:52.196258+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:03:41.650045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.201497+0000","last_change":"2024-04-21T23:15:55.398641+0000","last_active":"2024-04-21T23:16:52.201497+0000","last_peered":"2024-04-21T23:16:52.201497+0000","last_clean":"2024-04-21T23:16:52.201497+0000","last_became_active":"2024-04-21T23:15:55.397712+0000","last_became_peered":"2024-04-21T23:15:55.397712+0000","last_unstale":"2024-04-21T23:16:52.201497+0000","last_undegraded":"2024-04-21T23:16:52.201497+0000","last_fullsized":"2024-04-21T23:16:52.201497+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:28:09.266740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.197823+0000","last_change":"2024-04-21T23:15:56.345359+0000","last_active":"2024-04-21T23:16:52.197823+0000","last_peered":"2024-04-21T23:16:52.197823+0000","last_clean":"2024-04-21T23:16:52.197823+0000","last_became_active":"2024-04-21T23:15:56.343652+0000","last_became_peered":"2024-04-21T23:15:56.343652+0000","last_unstale":"2024-04-21T23:16:52.197823+0000","last_undegraded":"2024-04-21T23:16:52.197823+0000","last_fullsized":"2024-04-21T23:16:52.197823+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:18:10.057999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.204025+0000","last_change":"2024-04-21T23:16:00.430640+0000","last_active":"2024-04-21T23:16:52.204025+0000","last_peered":"2024-04-21T23:16:52.204025+0000","last_clean":"2024-04-21T23:16:52.204025+0000","last_became_active":"2024-04-21T23:16:00.430141+0000","last_became_peered":"2024-04-21T23:16:00.430141+0000","last_unstale":"2024-04-21T23:16:52.204025+0000","last_undegraded":"2024-04-21T23:16:52.204025+0000","last_fullsized":"2024-04-21T23:16:52.204025+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:41:13.749780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'5","reported_seq":106,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:17:32.614037+0000","last_change":"2024-04-21T23:15:55.399823+0000","last_active":"2024-04-21T23:17:32.614037+0000","last_peered":"2024-04-21T23:17:32.614037+0000","last_clean":"2024-04-21T23:17:32.614037+0000","last_became_active":"2024-04-21T23:15:55.399490+0000","last_became_peered":"2024-04-21T23:15:55.399490+0000","last_unstale":"2024-04-21T23:17:32.614037+0000","last_undegraded":"2024-04-21T23:17:32.614037+0000","last_fullsized":"2024-04-21T23:17:32.614037+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:54:38.124841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.200149+0000","last_change":"2024-04-21T23:15:56.346444+0000","last_active":"2024-04-21T23:16:52.200149+0000","last_peered":"2024-04-21T23:16:52.200149+0000","last_clean":"2024-04-21T23:16:52.200149+0000","last_became_active":"2024-04-21T23:15:56.346091+0000","last_became_peered":"2024-04-21T23:15:56.346091+0000","last_unstale":"2024-04-21T23:16:52.200149+0000","last_undegraded":"2024-04-21T23:16:52.200149+0000","last_fullsized":"2024-04-21T23:16:52.200149+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:41:11.144570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.203107+0000","last_change":"2024-04-21T23:16:00.438198+0000","last_active":"2024-04-21T23:16:52.203107+0000","last_peered":"2024-04-21T23:16:52.203107+0000","last_clean":"2024-04-21T23:16:52.203107+0000","last_became_active":"2024-04-21T23:16:00.437767+0000","last_became_peered":"2024-04-21T23:16:00.437767+0000","last_unstale":"2024-04-21T23:16:52.203107+0000","last_undegraded":"2024-04-21T23:16:52.203107+0000","last_fullsized":"2024-04-21T23:16:52.203107+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:35:11.356414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"90'3","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.554804+0000","last_change":"2024-04-21T23:15:55.403896+0000","last_active":"2024-04-21T23:16:52.554804+0000","last_peered":"2024-04-21T23:16:52.554804+0000","last_clean":"2024-04-21T23:16:52.554804+0000","last_became_active":"2024-04-21T23:15:55.403423+0000","last_became_peered":"2024-04-21T23:15:55.403423+0000","last_unstale":"2024-04-21T23:16:52.554804+0000","last_undegraded":"2024-04-21T23:16:52.554804+0000","last_fullsized":"2024-04-21T23:16:52.554804+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:34:44.252259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.349722+0000","last_change":"2024-04-21T23:16:52.349354+0000","last_active":"2024-04-21T23:16:52.349722+0000","last_peered":"2024-04-21T23:16:52.349722+0000","last_clean":"2024-04-21T23:16:52.349722+0000","last_became_active":"2024-04-21T23:16:52.348991+0000","last_became_peered":"2024-04-21T23:16:52.348991+0000","last_unstale":"2024-04-21T23:16:52.349722+0000","last_undegraded":"2024-04-21T23:16:52.349722+0000","last_fullsized":"2024-04-21T23:16:52.349722+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:42:57.975963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.199641+0000","last_change":"2024-04-21T23:16:00.434104+0000","last_active":"2024-04-21T23:16:52.199641+0000","last_peered":"2024-04-21T23:16:52.199641+0000","last_clean":"2024-04-21T23:16:52.199641+0000","last_became_active":"2024-04-21T23:16:00.433681+0000","last_became_peered":"2024-04-21T23:16:00.433681+0000","last_unstale":"2024-04-21T23:16:52.199641+0000","last_undegraded":"2024-04-21T23:16:52.199641+0000","last_fullsized":"2024-04-21T23:16:52.199641+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:10:17.986652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"90'2","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.198381+0000","last_change":"2024-04-21T23:15:55.397536+0000","last_active":"2024-04-21T23:16:52.198381+0000","last_peered":"2024-04-21T23:16:52.198381+0000","last_clean":"2024-04-21T23:16:52.198381+0000","last_became_active":"2024-04-21T23:15:55.397163+0000","last_became_peered":"2024-04-21T23:15:55.397163+0000","last_unstale":"2024-04-21T23:16:52.198381+0000","last_undegraded":"2024-04-21T23:16:52.198381+0000","last_fullsized":"2024-04-21T23:16:52.198381+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:05:27.777804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.347696+0000","last_change":"2024-04-21T23:15:56.351840+0000","last_active":"2024-04-21T23:16:52.347696+0000","last_peered":"2024-04-21T23:16:52.347696+0000","last_clean":"2024-04-21T23:16:52.347696+0000","last_became_active":"2024-04-21T23:15:56.351523+0000","last_became_peered":"2024-04-21T23:15:56.351523+0000","last_unstale":"2024-04-21T23:16:52.347696+0000","last_undegraded":"2024-04-21T23:16:52.347696+0000","last_fullsized":"2024-04-21T23:16:52.347696+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:57:11.714284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.202885+0000","last_change":"2024-04-21T23:16:00.437473+0000","last_active":"2024-04-21T23:16:52.202885+0000","last_peered":"2024-04-21T23:16:52.202885+0000","last_clean":"2024-04-21T23:16:52.202885+0000","last_became_active":"2024-04-21T23:16:00.437014+0000","last_became_peered":"2024-04-21T23:16:00.437014+0000","last_unstale":"2024-04-21T23:16:52.202885+0000","last_undegraded":"2024-04-21T23:16:52.202885+0000","last_fullsized":"2024-04-21T23:16:52.202885+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:37:14.241896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"90'1","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.196460+0000","last_change":"2024-04-21T23:15:55.399213+0000","last_active":"2024-04-21T23:16:52.196460+0000","last_peered":"2024-04-21T23:16:52.196460+0000","last_clean":"2024-04-21T23:16:52.196460+0000","last_became_active":"2024-04-21T23:15:55.398880+0000","last_became_peered":"2024-04-21T23:15:55.398880+0000","last_unstale":"2024-04-21T23:16:52.196460+0000","last_undegraded":"2024-04-21T23:16:52.196460+0000","last_fullsized":"2024-04-21T23:16:52.196460+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:56:32.236862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.195415+0000","last_change":"2024-04-21T23:15:56.349330+0000","last_active":"2024-04-21T23:16:52.195415+0000","last_peered":"2024-04-21T23:16:52.195415+0000","last_clean":"2024-04-21T23:16:52.195415+0000","last_became_active":"2024-04-21T23:15:56.348988+0000","last_became_peered":"2024-04-21T23:15:56.348988+0000","last_unstale":"2024-04-21T23:16:52.195415+0000","last_undegraded":"2024-04-21T23:16:52.195415+0000","last_fullsized":"2024-04-21T23:16:52.195415+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:11:19.650119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.347940+0000","last_change":"2024-04-21T23:16:00.431959+0000","last_active":"2024-04-21T23:16:52.347940+0000","last_peered":"2024-04-21T23:16:52.347940+0000","last_clean":"2024-04-21T23:16:52.347940+0000","last_became_active":"2024-04-21T23:16:00.431570+0000","last_became_peered":"2024-04-21T23:16:00.431570+0000","last_unstale":"2024-04-21T23:16:52.347940+0000","last_undegraded":"2024-04-21T23:16:52.347940+0000","last_fullsized":"2024-04-21T23:16:52.347940+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:02:54.057326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"90'3","reported_seq":102,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:17:32.613433+0000","last_change":"2024-04-21T23:15:55.404505+0000","last_active":"2024-04-21T23:17:32.613433+0000","last_peered":"2024-04-21T23:17:32.613433+0000","last_clean":"2024-04-21T23:17:32.613433+0000","last_became_active":"2024-04-21T23:15:55.403945+0000","last_became_peered":"2024-04-21T23:15:55.403945+0000","last_unstale":"2024-04-21T23:17:32.613433+0000","last_undegraded":"2024-04-21T23:17:32.613433+0000","last_fullsized":"2024-04-21T23:17:32.613433+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:54:37.647597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":72,"num_read_kb":72,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.200278+0000","last_change":"2024-04-21T23:15:56.356238+0000","last_active":"2024-04-21T23:16:52.200278+0000","last_peered":"2024-04-21T23:16:52.200278+0000","last_clean":"2024-04-21T23:16:52.200278+0000","last_became_active":"2024-04-21T23:15:56.355116+0000","last_became_peered":"2024-04-21T23:15:56.355116+0000","last_unstale":"2024-04-21T23:16:52.200278+0000","last_undegraded":"2024-04-21T23:16:52.200278+0000","last_fullsized":"2024-04-21T23:16:52.200278+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:43:20.468549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.195318+0000","last_change":"2024-04-21T23:16:00.439069+0000","last_active":"2024-04-21T23:16:52.195318+0000","last_peered":"2024-04-21T23:16:52.195318+0000","last_clean":"2024-04-21T23:16:52.195318+0000","last_became_active":"2024-04-21T23:16:00.438510+0000","last_became_peered":"2024-04-21T23:16:00.438510+0000","last_unstale":"2024-04-21T23:16:52.195318+0000","last_undegraded":"2024-04-21T23:16:52.195318+0000","last_fullsized":"2024-04-21T23:16:52.195318+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:41:50.814112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"90'2","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.206602+0000","last_change":"2024-04-21T23:16:52.206602+0000","last_active":"2024-04-21T23:16:52.206602+0000","last_peered":"2024-04-21T23:16:52.206602+0000","last_clean":"2024-04-21T23:16:52.206602+0000","last_became_active":"2024-04-21T23:16:52.202755+0000","last_became_peered":"2024-04-21T23:16:52.202755+0000","last_unstale":"2024-04-21T23:16:52.206602+0000","last_undegraded":"2024-04-21T23:16:52.206602+0000","last_fullsized":"2024-04-21T23:16:52.206602+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:01:02.437392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.203071+0000","last_change":"2024-04-21T23:15:56.357228+0000","last_active":"2024-04-21T23:16:52.203071+0000","last_peered":"2024-04-21T23:16:52.203071+0000","last_clean":"2024-04-21T23:16:52.203071+0000","last_became_active":"2024-04-21T23:15:56.356808+0000","last_became_peered":"2024-04-21T23:15:56.356808+0000","last_unstale":"2024-04-21T23:16:52.203071+0000","last_undegraded":"2024-04-21T23:16:52.203071+0000","last_fullsized":"2024-04-21T23:16:52.203071+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:15:10.728530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.197048+0000","last_change":"2024-04-21T23:16:00.443569+0000","last_active":"2024-04-21T23:16:52.197048+0000","last_peered":"2024-04-21T23:16:52.197048+0000","last_clean":"2024-04-21T23:16:52.197048+0000","last_became_active":"2024-04-21T23:16:00.442972+0000","last_became_peered":"2024-04-21T23:16:00.442972+0000","last_unstale":"2024-04-21T23:16:52.197048+0000","last_undegraded":"2024-04-21T23:16:52.197048+0000","last_fullsized":"2024-04-21T23:16:52.197048+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:35:21.471407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"90'4","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.197455+0000","last_change":"2024-04-21T23:15:55.400134+0000","last_active":"2024-04-21T23:16:52.197455+0000","last_peered":"2024-04-21T23:16:52.197455+0000","last_clean":"2024-04-21T23:16:52.197455+0000","last_became_active":"2024-04-21T23:15:55.399768+0000","last_became_peered":"2024-04-21T23:15:55.399768+0000","last_unstale":"2024-04-21T23:16:52.197455+0000","last_undegraded":"2024-04-21T23:16:52.197455+0000","last_fullsized":"2024-04-21T23:16:52.197455+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:15:09.323035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.199757+0000","last_change":"2024-04-21T23:15:56.357278+0000","last_active":"2024-04-21T23:16:52.199757+0000","last_peered":"2024-04-21T23:16:52.199757+0000","last_clean":"2024-04-21T23:16:52.199757+0000","last_became_active":"2024-04-21T23:15:56.356902+0000","last_became_peered":"2024-04-21T23:15:56.356902+0000","last_unstale":"2024-04-21T23:16:52.199757+0000","last_undegraded":"2024-04-21T23:16:52.199757+0000","last_fullsized":"2024-04-21T23:16:52.199757+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:19:05.269109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.195385+0000","last_change":"2024-04-21T23:16:00.439704+0000","last_active":"2024-04-21T23:16:52.195385+0000","last_peered":"2024-04-21T23:16:52.195385+0000","last_clean":"2024-04-21T23:16:52.195385+0000","last_became_active":"2024-04-21T23:16:00.439286+0000","last_became_peered":"2024-04-21T23:16:00.439286+0000","last_unstale":"2024-04-21T23:16:52.195385+0000","last_undegraded":"2024-04-21T23:16:52.195385+0000","last_fullsized":"2024-04-21T23:16:52.195385+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:08:59.194481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"90'31","reported_seq":209,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:17:33.614868+0000","last_change":"2024-04-21T23:15:55.403344+0000","last_active":"2024-04-21T23:17:33.614868+0000","last_peered":"2024-04-21T23:17:33.614868+0000","last_clean":"2024-04-21T23:17:33.614868+0000","last_became_active":"2024-04-21T23:15:55.402876+0000","last_became_peered":"2024-04-21T23:15:55.402876+0000","last_unstale":"2024-04-21T23:17:33.614868+0000","last_undegraded":"2024-04-21T23:17:33.614868+0000","last_fullsized":"2024-04-21T23:17:33.614868+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:55:59.082964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61775,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":151,"num_read_kb":136,"num_write":32,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.200472+0000","last_change":"2024-04-21T23:15:56.343897+0000","last_active":"2024-04-21T23:16:52.200472+0000","last_peered":"2024-04-21T23:16:52.200472+0000","last_clean":"2024-04-21T23:16:52.200472+0000","last_became_active":"2024-04-21T23:15:56.343567+0000","last_became_peered":"2024-04-21T23:15:56.343567+0000","last_unstale":"2024-04-21T23:16:52.200472+0000","last_undegraded":"2024-04-21T23:16:52.200472+0000","last_fullsized":"2024-04-21T23:16:52.200472+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:57:07.955722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.198235+0000","last_change":"2024-04-21T23:16:00.432722+0000","last_active":"2024-04-21T23:16:52.198235+0000","last_peered":"2024-04-21T23:16:52.198235+0000","last_clean":"2024-04-21T23:16:52.198235+0000","last_became_active":"2024-04-21T23:16:00.432257+0000","last_became_peered":"2024-04-21T23:16:00.432257+0000","last_unstale":"2024-04-21T23:16:52.198235+0000","last_undegraded":"2024-04-21T23:16:52.198235+0000","last_fullsized":"2024-04-21T23:16:52.198235+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:10:14.804490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"90'6","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.202441+0000","last_change":"2024-04-21T23:15:55.401300+0000","last_active":"2024-04-21T23:16:52.202441+0000","last_peered":"2024-04-21T23:16:52.202441+0000","last_clean":"2024-04-21T23:16:52.202441+0000","last_became_active":"2024-04-21T23:15:55.400904+0000","last_became_peered":"2024-04-21T23:15:55.400904+0000","last_unstale":"2024-04-21T23:16:52.202441+0000","last_undegraded":"2024-04-21T23:16:52.202441+0000","last_fullsized":"2024-04-21T23:16:52.202441+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:29:20.439936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.198540+0000","last_change":"2024-04-21T23:15:56.351046+0000","last_active":"2024-04-21T23:16:52.198540+0000","last_peered":"2024-04-21T23:16:52.198540+0000","last_clean":"2024-04-21T23:16:52.198540+0000","last_became_active":"2024-04-21T23:15:56.350662+0000","last_became_peered":"2024-04-21T23:15:56.350662+0000","last_unstale":"2024-04-21T23:16:52.198540+0000","last_undegraded":"2024-04-21T23:16:52.198540+0000","last_fullsized":"2024-04-21T23:16:52.198540+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:40:16.482061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.199200+0000","last_change":"2024-04-21T23:16:00.439816+0000","last_active":"2024-04-21T23:16:52.199200+0000","last_peered":"2024-04-21T23:16:52.199200+0000","last_clean":"2024-04-21T23:16:52.199200+0000","last_became_active":"2024-04-21T23:16:00.439330+0000","last_became_peered":"2024-04-21T23:16:00.439330+0000","last_unstale":"2024-04-21T23:16:52.199200+0000","last_undegraded":"2024-04-21T23:16:52.199200+0000","last_fullsized":"2024-04-21T23:16:52.199200+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:36:24.472968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"90'3","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.196986+0000","last_change":"2024-04-21T23:15:55.398588+0000","last_active":"2024-04-21T23:16:52.196986+0000","last_peered":"2024-04-21T23:16:52.196986+0000","last_clean":"2024-04-21T23:16:52.196986+0000","last_became_active":"2024-04-21T23:15:55.398264+0000","last_became_peered":"2024-04-21T23:15:55.398264+0000","last_unstale":"2024-04-21T23:16:52.196986+0000","last_undegraded":"2024-04-21T23:16:52.196986+0000","last_fullsized":"2024-04-21T23:16:52.196986+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:48:09.293481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.202037+0000","last_change":"2024-04-21T23:16:52.202037+0000","last_active":"2024-04-21T23:16:52.202037+0000","last_peered":"2024-04-21T23:16:52.202037+0000","last_clean":"2024-04-21T23:16:52.202037+0000","last_became_active":"2024-04-21T23:16:52.201244+0000","last_became_peered":"2024-04-21T23:16:52.201244+0000","last_unstale":"2024-04-21T23:16:52.202037+0000","last_undegraded":"2024-04-21T23:16:52.202037+0000","last_fullsized":"2024-04-21T23:16:52.202037+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:37:34.530110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.195827+0000","last_change":"2024-04-21T23:16:00.438581+0000","last_active":"2024-04-21T23:16:52.195827+0000","last_peered":"2024-04-21T23:16:52.195827+0000","last_clean":"2024-04-21T23:16:52.195827+0000","last_became_active":"2024-04-21T23:16:00.438219+0000","last_became_peered":"2024-04-21T23:16:00.438219+0000","last_unstale":"2024-04-21T23:16:52.195827+0000","last_undegraded":"2024-04-21T23:16:52.195827+0000","last_fullsized":"2024-04-21T23:16:52.195827+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:47:23.259045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"90'4","reported_seq":108,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:17:34.615994+0000","last_change":"2024-04-21T23:15:55.397503+0000","last_active":"2024-04-21T23:17:34.615994+0000","last_peered":"2024-04-21T23:17:34.615994+0000","last_clean":"2024-04-21T23:17:34.615994+0000","last_became_active":"2024-04-21T23:15:55.397181+0000","last_became_peered":"2024-04-21T23:15:55.397181+0000","last_unstale":"2024-04-21T23:17:34.615994+0000","last_undegraded":"2024-04-21T23:17:34.615994+0000","last_fullsized":"2024-04-21T23:17:34.615994+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:47:08.775783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":78,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.199048+0000","last_change":"2024-04-21T23:15:56.350872+0000","last_active":"2024-04-21T23:16:52.199048+0000","last_peered":"2024-04-21T23:16:52.199048+0000","last_clean":"2024-04-21T23:16:52.199048+0000","last_became_active":"2024-04-21T23:15:56.350519+0000","last_became_peered":"2024-04-21T23:15:56.350519+0000","last_unstale":"2024-04-21T23:16:52.199048+0000","last_undegraded":"2024-04-21T23:16:52.199048+0000","last_fullsized":"2024-04-21T23:16:52.199048+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:23:37.753893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.199048+0000","last_change":"2024-04-21T23:16:00.436069+0000","last_active":"2024-04-21T23:16:52.199048+0000","last_peered":"2024-04-21T23:16:52.199048+0000","last_clean":"2024-04-21T23:16:52.199048+0000","last_became_active":"2024-04-21T23:16:00.435617+0000","last_became_peered":"2024-04-21T23:16:00.435617+0000","last_unstale":"2024-04-21T23:16:52.199048+0000","last_undegraded":"2024-04-21T23:16:52.199048+0000","last_fullsized":"2024-04-21T23:16:52.199048+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:21:44.256928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"90'15","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.220742+0000","last_change":"2024-04-21T23:16:52.220742+0000","last_active":"2024-04-21T23:16:52.220742+0000","last_peered":"2024-04-21T23:16:52.220742+0000","last_clean":"2024-04-21T23:16:52.220742+0000","last_became_active":"2024-04-21T23:16:52.204787+0000","last_became_peered":"2024-04-21T23:16:52.204787+0000","last_unstale":"2024-04-21T23:16:52.220742+0000","last_undegraded":"2024-04-21T23:16:52.220742+0000","last_fullsized":"2024-04-21T23:16:52.220742+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:53:12.057014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.202341+0000","last_change":"2024-04-21T23:16:52.202341+0000","last_active":"2024-04-21T23:16:52.202341+0000","last_peered":"2024-04-21T23:16:52.202341+0000","last_clean":"2024-04-21T23:16:52.202341+0000","last_became_active":"2024-04-21T23:16:52.201528+0000","last_became_peered":"2024-04-21T23:16:52.201528+0000","last_unstale":"2024-04-21T23:16:52.202341+0000","last_undegraded":"2024-04-21T23:16:52.202341+0000","last_fullsized":"2024-04-21T23:16:52.202341+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:41:24.767379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.196494+0000","last_change":"2024-04-21T23:16:00.433580+0000","last_active":"2024-04-21T23:16:52.196494+0000","last_peered":"2024-04-21T23:16:52.196494+0000","last_clean":"2024-04-21T23:16:52.196494+0000","last_became_active":"2024-04-21T23:16:00.433185+0000","last_became_peered":"2024-04-21T23:16:00.433185+0000","last_unstale":"2024-04-21T23:16:52.196494+0000","last_undegraded":"2024-04-21T23:16:52.196494+0000","last_fullsized":"2024-04-21T23:16:52.196494+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:17:12.317266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"90'2","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.201190+0000","last_change":"2024-04-21T23:15:55.399032+0000","last_active":"2024-04-21T23:16:52.201190+0000","last_peered":"2024-04-21T23:16:52.201190+0000","last_clean":"2024-04-21T23:16:52.201190+0000","last_became_active":"2024-04-21T23:15:55.398208+0000","last_became_peered":"2024-04-21T23:15:55.398208+0000","last_unstale":"2024-04-21T23:16:52.201190+0000","last_undegraded":"2024-04-21T23:16:52.201190+0000","last_fullsized":"2024-04-21T23:16:52.201190+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:14:59.520571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.199490+0000","last_change":"2024-04-21T23:15:56.355357+0000","last_active":"2024-04-21T23:16:52.199490+0000","last_peered":"2024-04-21T23:16:52.199490+0000","last_clean":"2024-04-21T23:16:52.199490+0000","last_became_active":"2024-04-21T23:15:56.354704+0000","last_became_peered":"2024-04-21T23:15:56.354704+0000","last_unstale":"2024-04-21T23:16:52.199490+0000","last_undegraded":"2024-04-21T23:16:52.199490+0000","last_fullsized":"2024-04-21T23:16:52.199490+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:14:02.368659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.200338+0000","last_change":"2024-04-21T23:16:00.427781+0000","last_active":"2024-04-21T23:16:52.200338+0000","last_peered":"2024-04-21T23:16:52.200338+0000","last_clean":"2024-04-21T23:16:52.200338+0000","last_became_active":"2024-04-21T23:16:00.427244+0000","last_became_peered":"2024-04-21T23:16:00.427244+0000","last_unstale":"2024-04-21T23:16:52.200338+0000","last_undegraded":"2024-04-21T23:16:52.200338+0000","last_fullsized":"2024-04-21T23:16:52.200338+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:21:30.548865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.195660+0000","last_change":"2024-04-21T23:15:55.405687+0000","last_active":"2024-04-21T23:16:52.195660+0000","last_peered":"2024-04-21T23:16:52.195660+0000","last_clean":"2024-04-21T23:16:52.195660+0000","last_became_active":"2024-04-21T23:15:55.405379+0000","last_became_peered":"2024-04-21T23:15:55.405379+0000","last_unstale":"2024-04-21T23:16:52.195660+0000","last_undegraded":"2024-04-21T23:16:52.195660+0000","last_fullsized":"2024-04-21T23:16:52.195660+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:12:44.192092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.554132+0000","last_change":"2024-04-21T23:15:56.348989+0000","last_active":"2024-04-21T23:16:52.554132+0000","last_peered":"2024-04-21T23:16:52.554132+0000","last_clean":"2024-04-21T23:16:52.554132+0000","last_became_active":"2024-04-21T23:15:56.348039+0000","last_became_peered":"2024-04-21T23:15:56.348039+0000","last_unstale":"2024-04-21T23:16:52.554132+0000","last_undegraded":"2024-04-21T23:16:52.554132+0000","last_fullsized":"2024-04-21T23:16:52.554132+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:39:15.577567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.198945+0000","last_change":"2024-04-21T23:16:00.434557+0000","last_active":"2024-04-21T23:16:52.198945+0000","last_peered":"2024-04-21T23:16:52.198945+0000","last_clean":"2024-04-21T23:16:52.198945+0000","last_became_active":"2024-04-21T23:16:00.433737+0000","last_became_peered":"2024-04-21T23:16:00.433737+0000","last_unstale":"2024-04-21T23:16:52.198945+0000","last_undegraded":"2024-04-21T23:16:52.198945+0000","last_fullsized":"2024-04-21T23:16:52.198945+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:38:00.295054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"90'1","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.199699+0000","last_change":"2024-04-21T23:15:55.396792+0000","last_active":"2024-04-21T23:16:52.199699+0000","last_peered":"2024-04-21T23:16:52.199699+0000","last_clean":"2024-04-21T23:16:52.199699+0000","last_became_active":"2024-04-21T23:15:55.395740+0000","last_became_peered":"2024-04-21T23:15:55.395740+0000","last_unstale":"2024-04-21T23:16:52.199699+0000","last_undegraded":"2024-04-21T23:16:52.199699+0000","last_fullsized":"2024-04-21T23:16:52.199699+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:39:22.366310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.554076+0000","last_change":"2024-04-21T23:15:56.346372+0000","last_active":"2024-04-21T23:16:52.554076+0000","last_peered":"2024-04-21T23:16:52.554076+0000","last_clean":"2024-04-21T23:16:52.554076+0000","last_became_active":"2024-04-21T23:15:56.346047+0000","last_became_peered":"2024-04-21T23:15:56.346047+0000","last_unstale":"2024-04-21T23:16:52.554076+0000","last_undegraded":"2024-04-21T23:16:52.554076+0000","last_fullsized":"2024-04-21T23:16:52.554076+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:32:29.684227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.199049+0000","last_change":"2024-04-21T23:16:00.434747+0000","last_active":"2024-04-21T23:16:52.199049+0000","last_peered":"2024-04-21T23:16:52.199049+0000","last_clean":"2024-04-21T23:16:52.199049+0000","last_became_active":"2024-04-21T23:16:00.433950+0000","last_became_peered":"2024-04-21T23:16:00.433950+0000","last_unstale":"2024-04-21T23:16:52.199049+0000","last_undegraded":"2024-04-21T23:16:52.199049+0000","last_fullsized":"2024-04-21T23:16:52.199049+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:02:10.562135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.206022+0000","last_change":"2024-04-21T23:16:52.206022+0000","last_active":"2024-04-21T23:16:52.206022+0000","last_peered":"2024-04-21T23:16:52.206022+0000","last_clean":"2024-04-21T23:16:52.206022+0000","last_became_active":"2024-04-21T23:16:52.205469+0000","last_became_peered":"2024-04-21T23:16:52.205469+0000","last_unstale":"2024-04-21T23:16:52.206022+0000","last_undegraded":"2024-04-21T23:16:52.206022+0000","last_fullsized":"2024-04-21T23:16:52.206022+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:34:46.264003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,8],"acting":[5,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'4","reported_seq":103,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:17:32.612737+0000","last_change":"2024-04-21T23:15:55.394900+0000","last_active":"2024-04-21T23:17:32.612737+0000","last_peered":"2024-04-21T23:17:32.612737+0000","last_clean":"2024-04-21T23:17:32.612737+0000","last_became_active":"2024-04-21T23:15:55.393556+0000","last_became_peered":"2024-04-21T23:15:55.393556+0000","last_unstale":"2024-04-21T23:17:32.612737+0000","last_undegraded":"2024-04-21T23:17:32.612737+0000","last_fullsized":"2024-04-21T23:17:32.612737+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:16:24.691930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":72,"num_read_kb":72,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.199791+0000","last_change":"2024-04-21T23:16:00.443338+0000","last_active":"2024-04-21T23:16:52.199791+0000","last_peered":"2024-04-21T23:16:52.199791+0000","last_clean":"2024-04-21T23:16:52.199791+0000","last_became_active":"2024-04-21T23:16:00.442855+0000","last_became_peered":"2024-04-21T23:16:00.442855+0000","last_unstale":"2024-04-21T23:16:52.199791+0000","last_undegraded":"2024-04-21T23:16:52.199791+0000","last_fullsized":"2024-04-21T23:16:52.199791+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:04:36.763027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.347911+0000","last_change":"2024-04-21T23:15:56.356788+0000","last_active":"2024-04-21T23:16:52.347911+0000","last_peered":"2024-04-21T23:16:52.347911+0000","last_clean":"2024-04-21T23:16:52.347911+0000","last_became_active":"2024-04-21T23:15:56.356474+0000","last_became_peered":"2024-04-21T23:15:56.356474+0000","last_unstale":"2024-04-21T23:16:52.347911+0000","last_undegraded":"2024-04-21T23:16:52.347911+0000","last_fullsized":"2024-04-21T23:16:52.347911+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:34:33.700897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"90'3","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.197035+0000","last_change":"2024-04-21T23:15:55.395398+0000","last_active":"2024-04-21T23:16:52.197035+0000","last_peered":"2024-04-21T23:16:52.197035+0000","last_clean":"2024-04-21T23:16:52.197035+0000","last_became_active":"2024-04-21T23:15:55.394978+0000","last_became_peered":"2024-04-21T23:15:55.394978+0000","last_unstale":"2024-04-21T23:16:52.197035+0000","last_undegraded":"2024-04-21T23:16:52.197035+0000","last_fullsized":"2024-04-21T23:16:52.197035+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:07:01.266885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":362,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.198867+0000","last_change":"2024-04-21T23:14:42.877959+0000","last_active":"2024-04-21T23:16:52.198867+0000","last_peered":"2024-04-21T23:16:52.198867+0000","last_clean":"2024-04-21T23:16:52.198867+0000","last_became_active":"2024-04-21T23:14:42.877384+0000","last_became_peered":"2024-04-21T23:14:42.877384+0000","last_unstale":"2024-04-21T23:16:52.198867+0000","last_undegraded":"2024-04-21T23:16:52.198867+0000","last_fullsized":"2024-04-21T23:16:52.198867+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:11:15.009349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:11:15.009349+0000","last_clean_scrub_stamp":"2024-04-21T23:11:15.009349+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:18:55.621052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.196712+0000","last_change":"2024-04-21T23:16:00.444576+0000","last_active":"2024-04-21T23:16:52.196712+0000","last_peered":"2024-04-21T23:16:52.196712+0000","last_clean":"2024-04-21T23:16:52.196712+0000","last_became_active":"2024-04-21T23:16:00.444153+0000","last_became_peered":"2024-04-21T23:16:00.444153+0000","last_unstale":"2024-04-21T23:16:52.196712+0000","last_undegraded":"2024-04-21T23:16:52.196712+0000","last_fullsized":"2024-04-21T23:16:52.196712+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:11:48.040383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.198235+0000","last_change":"2024-04-21T23:15:56.349870+0000","last_active":"2024-04-21T23:16:52.198235+0000","last_peered":"2024-04-21T23:16:52.198235+0000","last_clean":"2024-04-21T23:16:52.198235+0000","last_became_active":"2024-04-21T23:15:56.349535+0000","last_became_peered":"2024-04-21T23:15:56.349535+0000","last_unstale":"2024-04-21T23:16:52.198235+0000","last_undegraded":"2024-04-21T23:16:52.198235+0000","last_fullsized":"2024-04-21T23:16:52.198235+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:12:50.702556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"90'3","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.198934+0000","last_change":"2024-04-21T23:15:55.402406+0000","last_active":"2024-04-21T23:16:52.198934+0000","last_peered":"2024-04-21T23:16:52.198934+0000","last_clean":"2024-04-21T23:16:52.198934+0000","last_became_active":"2024-04-21T23:15:55.402075+0000","last_became_peered":"2024-04-21T23:15:55.402075+0000","last_unstale":"2024-04-21T23:16:52.198934+0000","last_undegraded":"2024-04-21T23:16:52.198934+0000","last_fullsized":"2024-04-21T23:16:52.198934+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:44:18.013031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.199962+0000","last_change":"2024-04-21T23:16:00.439206+0000","last_active":"2024-04-21T23:16:52.199962+0000","last_peered":"2024-04-21T23:16:52.199962+0000","last_clean":"2024-04-21T23:16:52.199962+0000","last_became_active":"2024-04-21T23:16:00.438601+0000","last_became_peered":"2024-04-21T23:16:00.438601+0000","last_unstale":"2024-04-21T23:16:52.199962+0000","last_undegraded":"2024-04-21T23:16:52.199962+0000","last_fullsized":"2024-04-21T23:16:52.199962+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:58:46.771456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"90'1","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.200579+0000","last_change":"2024-04-21T23:15:55.401154+0000","last_active":"2024-04-21T23:16:52.200579+0000","last_peered":"2024-04-21T23:16:52.200579+0000","last_clean":"2024-04-21T23:16:52.200579+0000","last_became_active":"2024-04-21T23:15:55.400579+0000","last_became_peered":"2024-04-21T23:15:55.400579+0000","last_unstale":"2024-04-21T23:16:52.200579+0000","last_undegraded":"2024-04-21T23:16:52.200579+0000","last_fullsized":"2024-04-21T23:16:52.200579+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:20:03.879660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.199056+0000","last_change":"2024-04-21T23:15:56.349837+0000","last_active":"2024-04-21T23:16:52.199056+0000","last_peered":"2024-04-21T23:16:52.199056+0000","last_clean":"2024-04-21T23:16:52.199056+0000","last_became_active":"2024-04-21T23:15:56.349466+0000","last_became_peered":"2024-04-21T23:15:56.349466+0000","last_unstale":"2024-04-21T23:16:52.199056+0000","last_undegraded":"2024-04-21T23:16:52.199056+0000","last_fullsized":"2024-04-21T23:16:52.199056+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:15:32.675770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.347709+0000","last_change":"2024-04-21T23:16:00.432599+0000","last_active":"2024-04-21T23:16:52.347709+0000","last_peered":"2024-04-21T23:16:52.347709+0000","last_clean":"2024-04-21T23:16:52.347709+0000","last_became_active":"2024-04-21T23:16:00.432194+0000","last_became_peered":"2024-04-21T23:16:00.432194+0000","last_unstale":"2024-04-21T23:16:52.347709+0000","last_undegraded":"2024-04-21T23:16:52.347709+0000","last_fullsized":"2024-04-21T23:16:52.347709+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:10:21.354338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'8","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.195989+0000","last_change":"2024-04-21T23:15:55.400328+0000","last_active":"2024-04-21T23:16:52.195989+0000","last_peered":"2024-04-21T23:16:52.195989+0000","last_clean":"2024-04-21T23:16:52.195989+0000","last_became_active":"2024-04-21T23:15:55.399991+0000","last_became_peered":"2024-04-21T23:15:55.399991+0000","last_unstale":"2024-04-21T23:16:52.195989+0000","last_undegraded":"2024-04-21T23:16:52.195989+0000","last_fullsized":"2024-04-21T23:16:52.195989+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:58:54.480720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.348373+0000","last_change":"2024-04-21T23:15:56.358698+0000","last_active":"2024-04-21T23:16:52.348373+0000","last_peered":"2024-04-21T23:16:52.348373+0000","last_clean":"2024-04-21T23:16:52.348373+0000","last_became_active":"2024-04-21T23:15:56.358364+0000","last_became_peered":"2024-04-21T23:15:56.358364+0000","last_unstale":"2024-04-21T23:16:52.348373+0000","last_undegraded":"2024-04-21T23:16:52.348373+0000","last_fullsized":"2024-04-21T23:16:52.348373+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:34:15.281142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.202403+0000","last_change":"2024-04-21T23:16:00.432237+0000","last_active":"2024-04-21T23:16:52.202403+0000","last_peered":"2024-04-21T23:16:52.202403+0000","last_clean":"2024-04-21T23:16:52.202403+0000","last_became_active":"2024-04-21T23:16:00.431745+0000","last_became_peered":"2024-04-21T23:16:00.431745+0000","last_unstale":"2024-04-21T23:16:52.202403+0000","last_undegraded":"2024-04-21T23:16:52.202403+0000","last_fullsized":"2024-04-21T23:16:52.202403+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:25:00.477154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"90'4","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.196551+0000","last_change":"2024-04-21T23:15:55.398667+0000","last_active":"2024-04-21T23:16:52.196551+0000","last_peered":"2024-04-21T23:16:52.196551+0000","last_clean":"2024-04-21T23:16:52.196551+0000","last_became_active":"2024-04-21T23:15:55.398290+0000","last_became_peered":"2024-04-21T23:15:55.398290+0000","last_unstale":"2024-04-21T23:16:52.196551+0000","last_undegraded":"2024-04-21T23:16:52.196551+0000","last_fullsized":"2024-04-21T23:16:52.196551+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:57:58.992615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.202959+0000","last_change":"2024-04-21T23:15:56.359518+0000","last_active":"2024-04-21T23:16:52.202959+0000","last_peered":"2024-04-21T23:16:52.202959+0000","last_clean":"2024-04-21T23:16:52.202959+0000","last_became_active":"2024-04-21T23:15:56.358970+0000","last_became_peered":"2024-04-21T23:15:56.358970+0000","last_unstale":"2024-04-21T23:16:52.202959+0000","last_undegraded":"2024-04-21T23:16:52.202959+0000","last_fullsized":"2024-04-21T23:16:52.202959+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:41:22.113877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.196491+0000","last_change":"2024-04-21T23:16:00.443380+0000","last_active":"2024-04-21T23:16:52.196491+0000","last_peered":"2024-04-21T23:16:52.196491+0000","last_clean":"2024-04-21T23:16:52.196491+0000","last_became_active":"2024-04-21T23:16:00.442896+0000","last_became_peered":"2024-04-21T23:16:00.442896+0000","last_unstale":"2024-04-21T23:16:52.196491+0000","last_undegraded":"2024-04-21T23:16:52.196491+0000","last_fullsized":"2024-04-21T23:16:52.196491+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:51:00.089766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"90'2","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.198995+0000","last_change":"2024-04-21T23:15:55.395287+0000","last_active":"2024-04-21T23:16:52.198995+0000","last_peered":"2024-04-21T23:16:52.198995+0000","last_clean":"2024-04-21T23:16:52.198995+0000","last_became_active":"2024-04-21T23:15:55.394908+0000","last_became_peered":"2024-04-21T23:15:55.394908+0000","last_unstale":"2024-04-21T23:16:52.198995+0000","last_undegraded":"2024-04-21T23:16:52.198995+0000","last_fullsized":"2024-04-21T23:16:52.198995+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:57:29.915581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.199898+0000","last_change":"2024-04-21T23:15:56.348016+0000","last_active":"2024-04-21T23:16:52.199898+0000","last_peered":"2024-04-21T23:16:52.199898+0000","last_clean":"2024-04-21T23:16:52.199898+0000","last_became_active":"2024-04-21T23:15:56.347630+0000","last_became_peered":"2024-04-21T23:15:56.347630+0000","last_unstale":"2024-04-21T23:16:52.199898+0000","last_undegraded":"2024-04-21T23:16:52.199898+0000","last_fullsized":"2024-04-21T23:16:52.199898+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:17:27.372880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.195946+0000","last_change":"2024-04-21T23:16:00.442959+0000","last_active":"2024-04-21T23:16:52.195946+0000","last_peered":"2024-04-21T23:16:52.195946+0000","last_clean":"2024-04-21T23:16:52.195946+0000","last_became_active":"2024-04-21T23:16:00.442319+0000","last_became_peered":"2024-04-21T23:16:00.442319+0000","last_unstale":"2024-04-21T23:16:52.195946+0000","last_undegraded":"2024-04-21T23:16:52.195946+0000","last_fullsized":"2024-04-21T23:16:52.195946+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:59:39.868059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"90'3","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.348067+0000","last_change":"2024-04-21T23:15:55.400859+0000","last_active":"2024-04-21T23:16:52.348067+0000","last_peered":"2024-04-21T23:16:52.348067+0000","last_clean":"2024-04-21T23:16:52.348067+0000","last_became_active":"2024-04-21T23:15:55.400460+0000","last_became_peered":"2024-04-21T23:15:55.400460+0000","last_unstale":"2024-04-21T23:16:52.348067+0000","last_undegraded":"2024-04-21T23:16:52.348067+0000","last_fullsized":"2024-04-21T23:16:52.348067+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:31:14.113476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.198944+0000","last_change":"2024-04-21T23:15:56.350354+0000","last_active":"2024-04-21T23:16:52.198944+0000","last_peered":"2024-04-21T23:16:52.198944+0000","last_clean":"2024-04-21T23:16:52.198944+0000","last_became_active":"2024-04-21T23:15:56.349954+0000","last_became_peered":"2024-04-21T23:15:56.349954+0000","last_unstale":"2024-04-21T23:16:52.198944+0000","last_undegraded":"2024-04-21T23:16:52.198944+0000","last_fullsized":"2024-04-21T23:16:52.198944+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:42:59.165610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.196304+0000","last_change":"2024-04-21T23:16:00.444829+0000","last_active":"2024-04-21T23:16:52.196304+0000","last_peered":"2024-04-21T23:16:52.196304+0000","last_clean":"2024-04-21T23:16:52.196304+0000","last_became_active":"2024-04-21T23:16:00.444365+0000","last_became_peered":"2024-04-21T23:16:00.444365+0000","last_unstale":"2024-04-21T23:16:52.196304+0000","last_undegraded":"2024-04-21T23:16:52.196304+0000","last_fullsized":"2024-04-21T23:16:52.196304+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:23:16.747684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"90'1","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.202176+0000","last_change":"2024-04-21T23:15:55.397855+0000","last_active":"2024-04-21T23:16:52.202176+0000","last_peered":"2024-04-21T23:16:52.202176+0000","last_clean":"2024-04-21T23:16:52.202176+0000","last_became_active":"2024-04-21T23:15:55.397498+0000","last_became_peered":"2024-04-21T23:15:55.397498+0000","last_unstale":"2024-04-21T23:16:52.202176+0000","last_undegraded":"2024-04-21T23:16:52.202176+0000","last_fullsized":"2024-04-21T23:16:52.202176+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:45:44.680350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.348237+0000","last_change":"2024-04-21T23:15:56.353023+0000","last_active":"2024-04-21T23:16:52.348237+0000","last_peered":"2024-04-21T23:16:52.348237+0000","last_clean":"2024-04-21T23:16:52.348237+0000","last_became_active":"2024-04-21T23:15:56.352209+0000","last_became_peered":"2024-04-21T23:15:56.352209+0000","last_unstale":"2024-04-21T23:16:52.348237+0000","last_undegraded":"2024-04-21T23:16:52.348237+0000","last_fullsized":"2024-04-21T23:16:52.348237+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:20:58.075627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.196359+0000","last_change":"2024-04-21T23:16:00.429258+0000","last_active":"2024-04-21T23:16:52.196359+0000","last_peered":"2024-04-21T23:16:52.196359+0000","last_clean":"2024-04-21T23:16:52.196359+0000","last_became_active":"2024-04-21T23:16:00.428856+0000","last_became_peered":"2024-04-21T23:16:00.428856+0000","last_unstale":"2024-04-21T23:16:52.196359+0000","last_undegraded":"2024-04-21T23:16:52.196359+0000","last_fullsized":"2024-04-21T23:16:52.196359+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:07:44.608370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"90'4","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.209766+0000","last_change":"2024-04-21T23:16:52.209766+0000","last_active":"2024-04-21T23:16:52.209766+0000","last_peered":"2024-04-21T23:16:52.209766+0000","last_clean":"2024-04-21T23:16:52.209766+0000","last_became_active":"2024-04-21T23:16:52.204654+0000","last_became_peered":"2024-04-21T23:16:52.204654+0000","last_unstale":"2024-04-21T23:16:52.209766+0000","last_undegraded":"2024-04-21T23:16:52.209766+0000","last_fullsized":"2024-04-21T23:16:52.209766+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:40:41.230460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.199832+0000","last_change":"2024-04-21T23:15:56.352024+0000","last_active":"2024-04-21T23:16:52.199832+0000","last_peered":"2024-04-21T23:16:52.199832+0000","last_clean":"2024-04-21T23:16:52.199832+0000","last_became_active":"2024-04-21T23:15:56.351052+0000","last_became_peered":"2024-04-21T23:15:56.351052+0000","last_unstale":"2024-04-21T23:16:52.199832+0000","last_undegraded":"2024-04-21T23:16:52.199832+0000","last_fullsized":"2024-04-21T23:16:52.199832+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:36:42.777601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.199302+0000","last_change":"2024-04-21T23:16:00.432371+0000","last_active":"2024-04-21T23:16:52.199302+0000","last_peered":"2024-04-21T23:16:52.199302+0000","last_clean":"2024-04-21T23:16:52.199302+0000","last_became_active":"2024-04-21T23:16:00.431828+0000","last_became_peered":"2024-04-21T23:16:00.431828+0000","last_unstale":"2024-04-21T23:16:52.199302+0000","last_undegraded":"2024-04-21T23:16:52.199302+0000","last_fullsized":"2024-04-21T23:16:52.199302+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:02:06.500075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"90'3","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.203819+0000","last_change":"2024-04-21T23:15:55.407018+0000","last_active":"2024-04-21T23:16:52.203819+0000","last_peered":"2024-04-21T23:16:52.203819+0000","last_clean":"2024-04-21T23:16:52.203819+0000","last_became_active":"2024-04-21T23:15:55.406709+0000","last_became_peered":"2024-04-21T23:15:55.406709+0000","last_unstale":"2024-04-21T23:16:52.203819+0000","last_undegraded":"2024-04-21T23:16:52.203819+0000","last_fullsized":"2024-04-21T23:16:52.203819+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:10:40.138470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.200314+0000","last_change":"2024-04-21T23:15:56.358569+0000","last_active":"2024-04-21T23:16:52.200314+0000","last_peered":"2024-04-21T23:16:52.200314+0000","last_clean":"2024-04-21T23:16:52.200314+0000","last_became_active":"2024-04-21T23:15:56.357533+0000","last_became_peered":"2024-04-21T23:15:56.357533+0000","last_unstale":"2024-04-21T23:16:52.200314+0000","last_undegraded":"2024-04-21T23:16:52.200314+0000","last_fullsized":"2024-04-21T23:16:52.200314+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:59:58.525340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.202421+0000","last_change":"2024-04-21T23:16:00.443981+0000","last_active":"2024-04-21T23:16:52.202421+0000","last_peered":"2024-04-21T23:16:52.202421+0000","last_clean":"2024-04-21T23:16:52.202421+0000","last_became_active":"2024-04-21T23:16:00.443368+0000","last_became_peered":"2024-04-21T23:16:00.443368+0000","last_unstale":"2024-04-21T23:16:52.202421+0000","last_undegraded":"2024-04-21T23:16:52.202421+0000","last_fullsized":"2024-04-21T23:16:52.202421+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:56:48.776540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.197952+0000","last_change":"2024-04-21T23:15:55.397334+0000","last_active":"2024-04-21T23:16:52.197952+0000","last_peered":"2024-04-21T23:16:52.197952+0000","last_clean":"2024-04-21T23:16:52.197952+0000","last_became_active":"2024-04-21T23:15:55.396745+0000","last_became_peered":"2024-04-21T23:15:55.396745+0000","last_unstale":"2024-04-21T23:16:52.197952+0000","last_undegraded":"2024-04-21T23:16:52.197952+0000","last_fullsized":"2024-04-21T23:16:52.197952+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:56:21.042187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.198978+0000","last_change":"2024-04-21T23:15:56.351012+0000","last_active":"2024-04-21T23:16:52.198978+0000","last_peered":"2024-04-21T23:16:52.198978+0000","last_clean":"2024-04-21T23:16:52.198978+0000","last_became_active":"2024-04-21T23:15:56.350042+0000","last_became_peered":"2024-04-21T23:15:56.350042+0000","last_unstale":"2024-04-21T23:16:52.198978+0000","last_undegraded":"2024-04-21T23:16:52.198978+0000","last_fullsized":"2024-04-21T23:16:52.198978+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:47:02.629870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.196296+0000","last_change":"2024-04-21T23:16:00.428775+0000","last_active":"2024-04-21T23:16:52.196296+0000","last_peered":"2024-04-21T23:16:52.196296+0000","last_clean":"2024-04-21T23:16:52.196296+0000","last_became_active":"2024-04-21T23:16:00.428278+0000","last_became_peered":"2024-04-21T23:16:00.428278+0000","last_unstale":"2024-04-21T23:16:52.196296+0000","last_undegraded":"2024-04-21T23:16:52.196296+0000","last_fullsized":"2024-04-21T23:16:52.196296+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:24:04.397387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.199165+0000","last_change":"2024-04-21T23:15:55.395524+0000","last_active":"2024-04-21T23:16:52.199165+0000","last_peered":"2024-04-21T23:16:52.199165+0000","last_clean":"2024-04-21T23:16:52.199165+0000","last_became_active":"2024-04-21T23:15:55.395220+0000","last_became_peered":"2024-04-21T23:15:55.395220+0000","last_unstale":"2024-04-21T23:16:52.199165+0000","last_undegraded":"2024-04-21T23:16:52.199165+0000","last_fullsized":"2024-04-21T23:16:52.199165+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:01:37.517620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.200201+0000","last_change":"2024-04-21T23:15:56.352128+0000","last_active":"2024-04-21T23:16:52.200201+0000","last_peered":"2024-04-21T23:16:52.200201+0000","last_clean":"2024-04-21T23:16:52.200201+0000","last_became_active":"2024-04-21T23:15:56.349349+0000","last_became_peered":"2024-04-21T23:15:56.349349+0000","last_unstale":"2024-04-21T23:16:52.200201+0000","last_undegraded":"2024-04-21T23:16:52.200201+0000","last_fullsized":"2024-04-21T23:16:52.200201+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:12:23.247084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.199062+0000","last_change":"2024-04-21T23:15:56.350830+0000","last_active":"2024-04-21T23:16:52.199062+0000","last_peered":"2024-04-21T23:16:52.199062+0000","last_clean":"2024-04-21T23:16:52.199062+0000","last_became_active":"2024-04-21T23:15:56.350385+0000","last_became_peered":"2024-04-21T23:15:56.350385+0000","last_unstale":"2024-04-21T23:16:52.199062+0000","last_undegraded":"2024-04-21T23:16:52.199062+0000","last_fullsized":"2024-04-21T23:16:52.199062+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:18:48.242375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.203499+0000","last_change":"2024-04-21T23:15:56.350947+0000","last_active":"2024-04-21T23:16:52.203499+0000","last_peered":"2024-04-21T23:16:52.203499+0000","last_clean":"2024-04-21T23:16:52.203499+0000","last_became_active":"2024-04-21T23:15:56.350383+0000","last_became_peered":"2024-04-21T23:15:56.350383+0000","last_unstale":"2024-04-21T23:16:52.203499+0000","last_undegraded":"2024-04-21T23:16:52.203499+0000","last_fullsized":"2024-04-21T23:16:52.203499+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:07:28.879822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.202013+0000","last_change":"2024-04-21T23:15:56.354559+0000","last_active":"2024-04-21T23:16:52.202013+0000","last_peered":"2024-04-21T23:16:52.202013+0000","last_clean":"2024-04-21T23:16:52.202013+0000","last_became_active":"2024-04-21T23:15:56.354205+0000","last_became_peered":"2024-04-21T23:15:56.354205+0000","last_unstale":"2024-04-21T23:16:52.202013+0000","last_undegraded":"2024-04-21T23:16:52.202013+0000","last_fullsized":"2024-04-21T23:16:52.202013+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:33:50.157971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.202624+0000","last_change":"2024-04-21T23:15:56.358395+0000","last_active":"2024-04-21T23:16:52.202624+0000","last_peered":"2024-04-21T23:16:52.202624+0000","last_clean":"2024-04-21T23:16:52.202624+0000","last_became_active":"2024-04-21T23:15:56.358077+0000","last_became_peered":"2024-04-21T23:15:56.358077+0000","last_unstale":"2024-04-21T23:16:52.202624+0000","last_undegraded":"2024-04-21T23:16:52.202624+0000","last_fullsized":"2024-04-21T23:16:52.202624+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:12:43.124588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.199048+0000","last_change":"2024-04-21T23:15:56.347476+0000","last_active":"2024-04-21T23:16:52.199048+0000","last_peered":"2024-04-21T23:16:52.199048+0000","last_clean":"2024-04-21T23:16:52.199048+0000","last_became_active":"2024-04-21T23:15:56.347097+0000","last_became_peered":"2024-04-21T23:15:56.347097+0000","last_unstale":"2024-04-21T23:16:52.199048+0000","last_undegraded":"2024-04-21T23:16:52.199048+0000","last_fullsized":"2024-04-21T23:16:52.199048+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:11:58.715526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.199918+0000","last_change":"2024-04-21T23:15:56.346413+0000","last_active":"2024-04-21T23:16:52.199918+0000","last_peered":"2024-04-21T23:16:52.199918+0000","last_clean":"2024-04-21T23:16:52.199918+0000","last_became_active":"2024-04-21T23:15:56.346056+0000","last_became_peered":"2024-04-21T23:15:56.346056+0000","last_unstale":"2024-04-21T23:16:52.199918+0000","last_undegraded":"2024-04-21T23:16:52.199918+0000","last_fullsized":"2024-04-21T23:16:52.199918+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:47:16.340323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.204480+0000","last_change":"2024-04-21T23:15:56.354876+0000","last_active":"2024-04-21T23:16:52.204480+0000","last_peered":"2024-04-21T23:16:52.204480+0000","last_clean":"2024-04-21T23:16:52.204480+0000","last_became_active":"2024-04-21T23:15:56.354525+0000","last_became_peered":"2024-04-21T23:15:56.354525+0000","last_unstale":"2024-04-21T23:16:52.204480+0000","last_undegraded":"2024-04-21T23:16:52.204480+0000","last_fullsized":"2024-04-21T23:16:52.204480+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:09:23.662664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.348477+0000","last_change":"2024-04-21T23:15:56.358456+0000","last_active":"2024-04-21T23:16:52.348477+0000","last_peered":"2024-04-21T23:16:52.348477+0000","last_clean":"2024-04-21T23:16:52.348477+0000","last_became_active":"2024-04-21T23:15:56.358060+0000","last_became_peered":"2024-04-21T23:15:56.358060+0000","last_unstale":"2024-04-21T23:16:52.348477+0000","last_undegraded":"2024-04-21T23:16:52.348477+0000","last_fullsized":"2024-04-21T23:16:52.348477+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:47:58.928438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.196453+0000","last_change":"2024-04-21T23:15:56.352870+0000","last_active":"2024-04-21T23:16:52.196453+0000","last_peered":"2024-04-21T23:16:52.196453+0000","last_clean":"2024-04-21T23:16:52.196453+0000","last_became_active":"2024-04-21T23:15:56.352267+0000","last_became_peered":"2024-04-21T23:15:56.352267+0000","last_unstale":"2024-04-21T23:16:52.196453+0000","last_undegraded":"2024-04-21T23:16:52.196453+0000","last_fullsized":"2024-04-21T23:16:52.196453+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:32:22.244773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.348178+0000","last_change":"2024-04-21T23:15:56.357468+0000","last_active":"2024-04-21T23:16:52.348178+0000","last_peered":"2024-04-21T23:16:52.348178+0000","last_clean":"2024-04-21T23:16:52.348178+0000","last_became_active":"2024-04-21T23:15:56.357104+0000","last_became_peered":"2024-04-21T23:15:56.357104+0000","last_unstale":"2024-04-21T23:16:52.348178+0000","last_undegraded":"2024-04-21T23:16:52.348178+0000","last_fullsized":"2024-04-21T23:16:52.348178+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:37:36.884036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.203142+0000","last_change":"2024-04-21T23:16:52.203142+0000","last_active":"2024-04-21T23:16:52.203142+0000","last_peered":"2024-04-21T23:16:52.203142+0000","last_clean":"2024-04-21T23:16:52.203142+0000","last_became_active":"2024-04-21T23:16:52.202558+0000","last_became_peered":"2024-04-21T23:16:52.202558+0000","last_unstale":"2024-04-21T23:16:52.203142+0000","last_undegraded":"2024-04-21T23:16:52.203142+0000","last_fullsized":"2024-04-21T23:16:52.203142+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:29:05.428123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,8],"acting":[1,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.199231+0000","last_change":"2024-04-21T23:15:56.351901+0000","last_active":"2024-04-21T23:16:52.199231+0000","last_peered":"2024-04-21T23:16:52.199231+0000","last_clean":"2024-04-21T23:16:52.199231+0000","last_became_active":"2024-04-21T23:15:56.351545+0000","last_became_peered":"2024-04-21T23:15:56.351545+0000","last_unstale":"2024-04-21T23:16:52.199231+0000","last_undegraded":"2024-04-21T23:16:52.199231+0000","last_fullsized":"2024-04-21T23:16:52.199231+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:37:20.855631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:16:52.199333+0000","last_change":"2024-04-21T23:15:56.355773+0000","last_active":"2024-04-21T23:16:52.199333+0000","last_peered":"2024-04-21T23:16:52.199333+0000","last_clean":"2024-04-21T23:16:52.199333+0000","last_became_active":"2024-04-21T23:15:56.355269+0000","last_became_peered":"2024-04-21T23:15:56.355269+0000","last_unstale":"2024-04-21T23:16:52.199333+0000","last_undegraded":"2024-04-21T23:16:52.199333+0000","last_fullsized":"2024-04-21T23:16:52.199333+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:04:54.540817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":65776,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":457,"num_read_kb":296,"num_write":159,"num_write_kb":113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":112,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":197328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":138,"ondisk_log_size":138,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351010,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1044480,"data_stored":507590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612647,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37028,"kb_used_data":1684,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708124,"statfs":{"total":95995035648,"available":95957118976,"internally_reserved":0,"allocated":1724416,"data_stored":1156912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808877,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32212,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712940,"statfs":{"total":95995035648,"available":95962050560,"internally_reserved":0,"allocated":1056768,"data_stored":511810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070514,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1044480,"data_stored":507522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332150,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28628,"kb_used_data":1608,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716524,"statfs":{"total":95995035648,"available":95965720576,"internally_reserved":0,"allocated":1646592,"data_stored":1103543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495675,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1032192,"data_stored":502890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574933,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1097728,"data_stored":564575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738458,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1056768,"data_stored":504278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411408,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28116,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717036,"statfs":{"total":95995035648,"available":95966244864,"internally_reserved":0,"allocated":1122304,"data_stored":565925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182474,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36360,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708792,"statfs":{"total":95995035648,"available":95957803008,"internally_reserved":0,"allocated":1044480,"data_stored":503084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920838,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1044480,"data_stored":507931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757312,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36960,"kb_used_data":1616,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708192,"statfs":{"total":95995035648,"available":95957188608,"internally_reserved":0,"allocated":1654784,"data_stored":1098740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":61775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":63125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":63744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-21T23:17:40.794 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-21T23:17:40.794 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-21T23:17:40.794 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-21T23:17:40.794 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph health --format=json 2024-04-21T23:17:41.040 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:17:41.162 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:40 smithi040 ceph-mon[27984]: from='client.15117 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T23:17:41.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:40 smithi174 ceph-mon[31409]: from='client.15117 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T23:17:41.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:40 smithi111 ceph-mon[31085]: from='client.15117 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T23:17:42.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:41 smithi040 ceph-mon[27984]: pgmap v347: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:17:42.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:41 smithi174 ceph-mon[31409]: pgmap v347: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:17:42.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:41 smithi111 ceph-mon[31085]: pgmap v347: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:17:42.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:42.950+0000 7fa98ec34700 1 -- 172.21.15.40:0/4075347242 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa988102ec0 msgr2=0x7fa988106eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:42.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:42.950+0000 7fa98ec34700 1 --2- 172.21.15.40:0/4075347242 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa988102ec0 0x7fa988106eb0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fa974009990 tx=0x7fa974009ca0 comp rx=0 tx=0).stop 2024-04-21T23:17:42.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:42.950+0000 7fa98ec34700 1 -- 172.21.15.40:0/4075347242 shutdown_connections 2024-04-21T23:17:42.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:42.950+0000 7fa98ec34700 1 --2- 172.21.15.40:0/4075347242 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa988102510 0x7fa9881028f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:42.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:42.950+0000 7fa98ec34700 1 --2- 172.21.15.40:0/4075347242 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa988107590 0x7fa98810f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:42.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:42.950+0000 7fa98ec34700 1 --2- 172.21.15.40:0/4075347242 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa988102ec0 0x7fa988106eb0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:42.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:42.950+0000 7fa98ec34700 1 -- 172.21.15.40:0/4075347242 >> 172.21.15.40:0/4075347242 conn(0x7fa9880793a0 msgr2=0x7fa9880797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:42.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:42.950+0000 7fa98ec34700 1 -- 172.21.15.40:0/4075347242 shutdown_connections 2024-04-21T23:17:42.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:42.950+0000 7fa98ec34700 1 -- 172.21.15.40:0/4075347242 wait complete. 2024-04-21T23:17:42.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:42.951+0000 7fa98ec34700 1 Processor -- start 2024-04-21T23:17:42.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:42.951+0000 7fa98ec34700 1 -- start start 2024-04-21T23:17:42.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:42.951+0000 7fa98ec34700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa988102510 0x7fa988074530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:42.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:42.951+0000 7fa98ec34700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa988102ec0 0x7fa988074a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:42.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:42.951+0000 7fa98ec34700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa988107590 0x7fa988074fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:42.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:42.951+0000 7fa98ec34700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa988112630 con 0x7fa988102510 2024-04-21T23:17:42.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:42.951+0000 7fa98ec34700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fa9881124b0 con 0x7fa988107590 2024-04-21T23:17:42.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:42.951+0000 7fa98ec34700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa9881127b0 con 0x7fa988102ec0 2024-04-21T23:17:42.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:42.951+0000 7fa98c9d0700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa988102510 0x7fa988074530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:42.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:42.951+0000 7fa98c9d0700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa988102510 0x7fa988074530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38708/0 (socket says 172.21.15.40:38708) 2024-04-21T23:17:42.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:42.951+0000 7fa98c9d0700 1 -- 172.21.15.40:0/1676725573 learned_addr learned my addr 172.21.15.40:0/1676725573 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:17:42.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:42.951+0000 7fa987fff700 1 --2- 172.21.15.40:0/1676725573 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa988102ec0 0x7fa988074a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:42.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:42.951+0000 7fa98d1d1700 1 --2- 172.21.15.40:0/1676725573 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa988107590 0x7fa988074fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:42.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:42.951+0000 7fa98c9d0700 1 -- 172.21.15.40:0/1676725573 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa988102ec0 msgr2=0x7fa988074a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:42.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:42.951+0000 7fa98c9d0700 1 --2- 172.21.15.40:0/1676725573 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa988102ec0 0x7fa988074a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:42.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:42.951+0000 7fa98c9d0700 1 -- 172.21.15.40:0/1676725573 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa988107590 msgr2=0x7fa988074fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:42.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:42.951+0000 7fa98c9d0700 1 --2- 172.21.15.40:0/1676725573 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa988107590 0x7fa988074fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:42.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:42.951+0000 7fa98c9d0700 1 -- 172.21.15.40:0/1676725573 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9881a7050 con 0x7fa988102510 2024-04-21T23:17:42.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:42.951+0000 7fa98d1d1700 1 --2- 172.21.15.40:0/1676725573 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa988107590 0x7fa988074fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:17:42.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:42.952+0000 7fa98c9d0700 1 --2- 172.21.15.40:0/1676725573 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa988102510 0x7fa988074530 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fa97c00b530 tx=0x7fa97c00b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:42.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:42.952+0000 7fa985ffb700 1 -- 172.21.15.40:0/1676725573 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa97c015020 con 0x7fa988102510 2024-04-21T23:17:42.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:42.952+0000 7fa985ffb700 1 -- 172.21.15.40:0/1676725573 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa97c011d10 con 0x7fa988102510 2024-04-21T23:17:42.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:42.952+0000 7fa98ec34700 1 -- 172.21.15.40:0/1676725573 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9881a7340 con 0x7fa988102510 2024-04-21T23:17:42.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:42.952+0000 7fa985ffb700 1 -- 172.21.15.40:0/1676725573 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa97c00f2e0 con 0x7fa988102510 2024-04-21T23:17:42.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:42.952+0000 7fa98ec34700 1 -- 172.21.15.40:0/1676725573 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9881a7850 con 0x7fa988102510 2024-04-21T23:17:42.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:42.953+0000 7fa98ec34700 1 -- 172.21.15.40:0/1676725573 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa988103990 con 0x7fa988102510 2024-04-21T23:17:42.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:42.954+0000 7fa985ffb700 1 -- 172.21.15.40:0/1676725573 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fa97c018020 con 0x7fa988102510 2024-04-21T23:17:42.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:42.955+0000 7fa985ffb700 1 --2- 172.21.15.40:0/1676725573 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fa9780a2cb0 0x7fa9780a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:42.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:42.955+0000 7fa987fff700 1 --2- 172.21.15.40:0/1676725573 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fa9780a2cb0 0x7fa9780a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:42.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:42.955+0000 7fa985ffb700 1 -- 172.21.15.40:0/1676725573 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fa97c0cece0 con 0x7fa988102510 2024-04-21T23:17:42.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:42.955+0000 7fa987fff700 1 --2- 172.21.15.40:0/1676725573 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fa9780a2cb0 0x7fa9780a5170 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fa974006010 tx=0x7fa974005960 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:42.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:42.959+0000 7fa985ffb700 1 -- 172.21.15.40:0/1676725573 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fa97c01f020 con 0x7fa988102510 2024-04-21T23:17:43.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:43.214+0000 7fa98ec34700 1 -- 172.21.15.40:0/1676725573 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fa98802db30 con 0x7fa988102510 2024-04-21T23:17:43.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:43.214+0000 7fa985ffb700 1 -- 172.21.15.40:0/1676725573 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fa97c09b830 con 0x7fa988102510 2024-04-21T23:17:43.214 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:17:43.214 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-21T23:17:43.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:43.216+0000 7fa98ec34700 1 -- 172.21.15.40:0/1676725573 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fa9780a2cb0 msgr2=0x7fa9780a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:43.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:43.216+0000 7fa98ec34700 1 --2- 172.21.15.40:0/1676725573 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fa9780a2cb0 0x7fa9780a5170 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fa974006010 tx=0x7fa974005960 comp rx=0 tx=0).stop 2024-04-21T23:17:43.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:43.216+0000 7fa98ec34700 1 -- 172.21.15.40:0/1676725573 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa988102510 msgr2=0x7fa988074530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:43.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:43.216+0000 7fa98ec34700 1 --2- 172.21.15.40:0/1676725573 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa988102510 0x7fa988074530 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fa97c00b530 tx=0x7fa97c00b840 comp rx=0 tx=0).stop 2024-04-21T23:17:43.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:43.216+0000 7fa98ec34700 1 -- 172.21.15.40:0/1676725573 shutdown_connections 2024-04-21T23:17:43.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:43.216+0000 7fa98ec34700 1 --2- 172.21.15.40:0/1676725573 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fa9780a2cb0 0x7fa9780a5170 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:43.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:43.216+0000 7fa98ec34700 1 --2- 172.21.15.40:0/1676725573 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa988107590 0x7fa988074fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:43.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:43.216+0000 7fa98ec34700 1 --2- 172.21.15.40:0/1676725573 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa988102510 0x7fa988074530 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:43.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:43.216+0000 7fa98ec34700 1 --2- 172.21.15.40:0/1676725573 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa988102ec0 0x7fa988074a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:43.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:43.216+0000 7fa98ec34700 1 -- 172.21.15.40:0/1676725573 >> 172.21.15.40:0/1676725573 conn(0x7fa9880793a0 msgr2=0x7fa988100580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:43.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:43.217+0000 7fa98ec34700 1 -- 172.21.15.40:0/1676725573 shutdown_connections 2024-04-21T23:17:43.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:43.217+0000 7fa98ec34700 1 -- 172.21.15.40:0/1676725573 wait complete. 2024-04-21T23:17:43.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:43 smithi040 ceph-mon[27984]: pgmap v348: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-21T23:17:43.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:43 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/1676725573' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-21T23:17:44.060 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-21T23:17:44.061 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T23:17:44.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.173+0000 7f56ce355640 1 -- 172.21.15.40:0/1283880259 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56c8101880 msgr2=0x7f56c8101d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:44.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.173+0000 7f56ce355640 1 --2- 172.21.15.40:0/1283880259 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56c8101880 0x7f56c8101d00 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f56b0009900 tx=0x7f56b002bc30 comp rx=0 tx=0).stop 2024-04-21T23:17:44.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.174+0000 7f56ce355640 1 -- 172.21.15.40:0/1283880259 shutdown_connections 2024-04-21T23:17:44.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.174+0000 7f56ce355640 1 --2- 172.21.15.40:0/1283880259 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f56c8102240 0x7f56c8108ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:44.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.174+0000 7f56ce355640 1 --2- 172.21.15.40:0/1283880259 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56c8101880 0x7f56c8101d00 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:44.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.174+0000 7f56ce355640 1 --2- 172.21.15.40:0/1283880259 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f56c8100690 0x7f56c8100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:44.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.174+0000 7f56ce355640 1 -- 172.21.15.40:0/1283880259 >> 172.21.15.40:0/1283880259 conn(0x7f56c80fbc20 msgr2=0x7f56c80fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:44.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.174+0000 7f56ce355640 1 -- 172.21.15.40:0/1283880259 shutdown_connections 2024-04-21T23:17:44.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.174+0000 7f56ce355640 1 -- 172.21.15.40:0/1283880259 wait complete. 2024-04-21T23:17:44.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.174+0000 7f56ce355640 1 Processor -- start 2024-04-21T23:17:44.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.174+0000 7f56ce355640 1 -- start start 2024-04-21T23:17:44.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.174+0000 7f56ce355640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56c8100690 0x7f56c8197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:44.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.174+0000 7f56ce355640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f56c8101880 0x7f56c8198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:44.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.175+0000 7f56ce355640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f56c8102240 0x7f56c819f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:44.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.175+0000 7f56ce355640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f56c810c370 con 0x7f56c8100690 2024-04-21T23:17:44.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.175+0000 7f56ce355640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f56c810c1f0 con 0x7f56c8102240 2024-04-21T23:17:44.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.175+0000 7f56ce355640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f56c810c4f0 con 0x7f56c8101880 2024-04-21T23:17:44.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.175+0000 7f56c7fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56c8100690 0x7f56c8197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:44.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.175+0000 7f56c7fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56c8100690 0x7f56c8197b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38734/0 (socket says 172.21.15.40:38734) 2024-04-21T23:17:44.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.175+0000 7f56c7fff640 1 -- 172.21.15.40:0/1922020847 learned_addr learned my addr 172.21.15.40:0/1922020847 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:17:44.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.175+0000 7f56cc8f2640 1 --2- 172.21.15.40:0/1922020847 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f56c8102240 0x7f56c819f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:44.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.175+0000 7f56c77fe640 1 --2- 172.21.15.40:0/1922020847 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f56c8101880 0x7f56c8198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:44.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.175+0000 7f56c7fff640 1 -- 172.21.15.40:0/1922020847 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f56c8101880 msgr2=0x7f56c8198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:44.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.175+0000 7f56c7fff640 1 --2- 172.21.15.40:0/1922020847 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f56c8101880 0x7f56c8198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:44.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.175+0000 7f56c7fff640 1 -- 172.21.15.40:0/1922020847 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f56c8102240 msgr2=0x7f56c819f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:44.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.175+0000 7f56c7fff640 1 --2- 172.21.15.40:0/1922020847 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f56c8102240 0x7f56c819f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:44.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.175+0000 7f56c7fff640 1 -- 172.21.15.40:0/1922020847 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56c819f730 con 0x7f56c8100690 2024-04-21T23:17:44.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.175+0000 7f56cc8f2640 1 --2- 172.21.15.40:0/1922020847 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f56c8102240 0x7f56c819f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:17:44.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.175+0000 7f56c7fff640 1 --2- 172.21.15.40:0/1922020847 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56c8100690 0x7f56c8197b10 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f56b800d770 tx=0x7f56b800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:44.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.175+0000 7f56c57fa640 1 -- 172.21.15.40:0/1922020847 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56b8017070 con 0x7f56c8100690 2024-04-21T23:17:44.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.175+0000 7f56c57fa640 1 -- 172.21.15.40:0/1922020847 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f56b8009c30 con 0x7f56c8100690 2024-04-21T23:17:44.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.175+0000 7f56ce355640 1 -- 172.21.15.40:0/1922020847 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56c819fa20 con 0x7f56c8100690 2024-04-21T23:17:44.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.175+0000 7f56c57fa640 1 -- 172.21.15.40:0/1922020847 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56b8003dd0 con 0x7f56c8100690 2024-04-21T23:17:44.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.175+0000 7f56ce355640 1 -- 172.21.15.40:0/1922020847 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56c819ff40 con 0x7f56c8100690 2024-04-21T23:17:44.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.177+0000 7f56ce355640 1 -- 172.21.15.40:0/1922020847 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56c81a0110 con 0x7f56c8100690 2024-04-21T23:17:44.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.178+0000 7f56c57fa640 1 -- 172.21.15.40:0/1922020847 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f56b8003fb0 con 0x7f56c8100690 2024-04-21T23:17:44.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.178+0000 7f56c57fa640 1 --2- 172.21.15.40:0/1922020847 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f56b40a2c20 0x7f56b40a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:44.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.179+0000 7f56c77fe640 1 --2- 172.21.15.40:0/1922020847 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f56b40a2c20 0x7f56b40a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:44.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.179+0000 7f56c57fa640 1 -- 172.21.15.40:0/1922020847 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f56b80cefc0 con 0x7f56c8100690 2024-04-21T23:17:44.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.179+0000 7f56c77fe640 1 --2- 172.21.15.40:0/1922020847 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f56b40a2c20 0x7f56b40a50e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f56b0006010 tx=0x7f56b0000bc0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:44.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.183+0000 7f56c57fa640 1 -- 172.21.15.40:0/1922020847 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f56b809ba60 con 0x7f56c8100690 2024-04-21T23:17:44.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:43 smithi174 ceph-mon[31409]: pgmap v348: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-21T23:17:44.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:43 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/1676725573' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-21T23:17:44.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:43 smithi111 ceph-mon[31085]: pgmap v348: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-21T23:17:44.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:43 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/1676725573' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-21T23:17:44.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.375+0000 7f56ce355640 1 -- 172.21.15.40:0/1922020847 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f56c81a02e0 con 0x7f56c8100690 2024-04-21T23:17:44.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.376+0000 7f56c57fa640 1 -- 172.21.15.40:0/1922020847 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+24666 (secure 0 0 0) 0x7f56b8009de0 con 0x7f56c8100690 2024-04-21T23:17:44.376 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:17:44.376 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":92,"fsid":"d814e4dc-0033-11ef-bc93-c7b262605968","created":"2024-04-21T23:07:52.659207+0000","modified":"2024-04-21T23:16:52.188054+0000","last_up_change":"2024-04-21T23:15:05.308821+0000","last_in_change":"2024-04-21T23:14:45.447491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:11:14.824775+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T23:15:54.322571+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T23:15:55.164493+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T23:15:58.838266+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"0e8d692a-22fe-4a87-8d99-9151f96cbb02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6803","nonce":2009758784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6805","nonce":2009758784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6809","nonce":2009758784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6807","nonce":2009758784}]},"public_addr":"172.21.15.40:6803/2009758784","cluster_addr":"172.21.15.40:6805/2009758784","heartbeat_back_addr":"172.21.15.40:6809/2009758784","heartbeat_front_addr":"172.21.15.40:6807/2009758784","state":["exists","up"]},{"osd":1,"uuid":"fe5fe11b-0e84-4359-83a7-1ba45105755a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6801","nonce":1763903836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6803","nonce":1763903836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6807","nonce":1763903836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6805","nonce":1763903836}]},"public_addr":"172.21.15.111:6801/1763903836","cluster_addr":"172.21.15.111:6803/1763903836","heartbeat_back_addr":"172.21.15.111:6807/1763903836","heartbeat_front_addr":"172.21.15.111:6805/1763903836","state":["exists","up"]},{"osd":2,"uuid":"813ad735-5cc2-4da0-ae16-6db238395fbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6801","nonce":439695825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6803","nonce":439695825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6807","nonce":439695825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6805","nonce":439695825}]},"public_addr":"172.21.15.174:6801/439695825","cluster_addr":"172.21.15.174:6803/439695825","heartbeat_back_addr":"172.21.15.174:6807/439695825","heartbeat_front_addr":"172.21.15.174:6805/439695825","state":["exists","up"]},{"osd":3,"uuid":"09becb03-b7ce-4ffc-917d-b7900ec37e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6811","nonce":3454312436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6813","nonce":3454312436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6817","nonce":3454312436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6815","nonce":3454312436}]},"public_addr":"172.21.15.40:6811/3454312436","cluster_addr":"172.21.15.40:6813/3454312436","heartbeat_back_addr":"172.21.15.40:6817/3454312436","heartbeat_front_addr":"172.21.15.40:6815/3454312436","state":["exists","up"]},{"osd":4,"uuid":"46eabfd7-d88a-4efd-94eb-04354d3a9d7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6809","nonce":1810866070}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6811","nonce":1810866070}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6815","nonce":1810866070}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6813","nonce":1810866070}]},"public_addr":"172.21.15.111:6809/1810866070","cluster_addr":"172.21.15.111:6811/1810866070","heartbeat_back_addr":"172.21.15.111:6815/1810866070","heartbeat_front_addr":"172.21.15.111:6813/1810866070","state":["exists","up"]},{"osd":5,"uuid":"6f73b582-6c65-41db-aab0-9b360f83d5c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6809","nonce":3752986711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6811","nonce":3752986711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6815","nonce":3752986711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6813","nonce":3752986711}]},"public_addr":"172.21.15.174:6809/3752986711","cluster_addr":"172.21.15.174:6811/3752986711","heartbeat_back_addr":"172.21.15.174:6815/3752986711","heartbeat_front_addr":"172.21.15.174:6813/3752986711","state":["exists","up"]},{"osd":6,"uuid":"39f305f0-12d2-4732-aaa0-cd66700af538","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6819","nonce":1618855427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6821","nonce":1618855427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6825","nonce":1618855427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6823","nonce":1618855427}]},"public_addr":"172.21.15.40:6819/1618855427","cluster_addr":"172.21.15.40:6821/1618855427","heartbeat_back_addr":"172.21.15.40:6825/1618855427","heartbeat_front_addr":"172.21.15.40:6823/1618855427","state":["exists","up"]},{"osd":7,"uuid":"769f2468-5102-490b-89f8-d0444d3169a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6817","nonce":927054162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6819","nonce":927054162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6823","nonce":927054162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6821","nonce":927054162}]},"public_addr":"172.21.15.111:6817/927054162","cluster_addr":"172.21.15.111:6819/927054162","heartbeat_back_addr":"172.21.15.111:6823/927054162","heartbeat_front_addr":"172.21.15.111:6821/927054162","state":["exists","up"]},{"osd":8,"uuid":"1a444e80-007d-473b-b972-4c814a813c22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6817","nonce":3817848746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6819","nonce":3817848746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6823","nonce":3817848746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6821","nonce":3817848746}]},"public_addr":"172.21.15.174:6817/3817848746","cluster_addr":"172.21.15.174:6819/3817848746","heartbeat_back_addr":"172.21.15.174:6823/3817848746","heartbeat_front_addr":"172.21.15.174:6821/3817848746","state":["exists","up"]},{"osd":9,"uuid":"9a6b6a56-6ae5-49fe-9e5b-cb17d686e0dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6827","nonce":1649446531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6829","nonce":1649446531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6833","nonce":1649446531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6831","nonce":1649446531}]},"public_addr":"172.21.15.40:6827/1649446531","cluster_addr":"172.21.15.40:6829/1649446531","heartbeat_back_addr":"172.21.15.40:6833/1649446531","heartbeat_front_addr":"172.21.15.40:6831/1649446531","state":["exists","up"]},{"osd":10,"uuid":"a70038ba-e2ad-4170-85cc-ea0470a2edef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6825","nonce":1227538935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6827","nonce":1227538935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6831","nonce":1227538935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6829","nonce":1227538935}]},"public_addr":"172.21.15.111:6825/1227538935","cluster_addr":"172.21.15.111:6827/1227538935","heartbeat_back_addr":"172.21.15.111:6831/1227538935","heartbeat_front_addr":"172.21.15.111:6829/1227538935","state":["exists","up"]},{"osd":11,"uuid":"781532bb-7a5e-44e3-8eb7-b584b0054666","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6825","nonce":2045865553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6827","nonce":2045865553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6831","nonce":2045865553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6829","nonce":2045865553}]},"public_addr":"172.21.15.174:6825/2045865553","cluster_addr":"172.21.15.174:6827/2045865553","heartbeat_back_addr":"172.21.15.174:6831/2045865553","heartbeat_front_addr":"172.21.15.174:6829/2045865553","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:10:24.761290+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:10:48.565428+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:11:12.307818+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:11:37.951529+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:02.832523+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:27.568879+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:52.997707+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:13:18.937274+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:13:44.708657+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:14:10.833855+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:14:37.078272+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:15:03.878076+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":1}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.2a","mappings":[{"from":3,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:6801/526554319":"2024-04-22T23:08:50.305770+0000","172.21.15.40:6800/526554319":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/3178763471":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/4045911364":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/276300080":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/3689649814":"2024-04-22T23:08:23.743030+0000","172.21.15.40:0/2241116715":"2024-04-22T23:08:23.743030+0000","172.21.15.40:6801/1479040326":"2024-04-22T23:08:23.743030+0000","172.21.15.40:6800/1479040326":"2024-04-22T23:08:23.743030+0000","172.21.15.40:0/681298099":"2024-04-22T23:08:23.743030+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T23:17:44.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.379+0000 7f56ce355640 1 -- 172.21.15.40:0/1922020847 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f56b40a2c20 msgr2=0x7f56b40a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:44.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.379+0000 7f56ce355640 1 --2- 172.21.15.40:0/1922020847 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f56b40a2c20 0x7f56b40a50e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f56b0006010 tx=0x7f56b0000bc0 comp rx=0 tx=0).stop 2024-04-21T23:17:44.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.379+0000 7f56ce355640 1 -- 172.21.15.40:0/1922020847 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56c8100690 msgr2=0x7f56c8197b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:44.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.379+0000 7f56ce355640 1 --2- 172.21.15.40:0/1922020847 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56c8100690 0x7f56c8197b10 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f56b800d770 tx=0x7f56b800dc30 comp rx=0 tx=0).stop 2024-04-21T23:17:44.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.379+0000 7f56ce355640 1 -- 172.21.15.40:0/1922020847 shutdown_connections 2024-04-21T23:17:44.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.379+0000 7f56ce355640 1 --2- 172.21.15.40:0/1922020847 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f56b40a2c20 0x7f56b40a50e0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:44.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.379+0000 7f56ce355640 1 --2- 172.21.15.40:0/1922020847 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f56c8102240 0x7f56c819f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:44.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.379+0000 7f56ce355640 1 --2- 172.21.15.40:0/1922020847 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f56c8101880 0x7f56c8198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:44.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.379+0000 7f56ce355640 1 --2- 172.21.15.40:0/1922020847 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56c8100690 0x7f56c8197b10 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:44.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.379+0000 7f56ce355640 1 -- 172.21.15.40:0/1922020847 >> 172.21.15.40:0/1922020847 conn(0x7f56c80fbc20 msgr2=0x7f56c8069e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:44.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.379+0000 7f56ce355640 1 -- 172.21.15.40:0/1922020847 shutdown_connections 2024-04-21T23:17:44.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.379+0000 7f56ce355640 1 -- 172.21.15.40:0/1922020847 wait complete. 2024-04-21T23:17:44.390 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T23:11:14.824775+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T23:15:54.322571+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T23:15:55.164493+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T23:15:58.838266+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T23:17:44.390 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T23:17:44.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.499+0000 7f633140e640 1 -- 172.21.15.40:0/1597350058 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f632c103340 msgr2=0x7f632c109bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:44.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.499+0000 7f633140e640 1 --2- 172.21.15.40:0/1597350058 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f632c103340 0x7f632c109bd0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f6324009ca0 tx=0x7f632402c0f0 comp rx=0 tx=0).stop 2024-04-21T23:17:44.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.499+0000 7f633140e640 1 -- 172.21.15.40:0/1597350058 shutdown_connections 2024-04-21T23:17:44.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.499+0000 7f633140e640 1 --2- 172.21.15.40:0/1597350058 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f632c103340 0x7f632c109bd0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:44.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.499+0000 7f633140e640 1 --2- 172.21.15.40:0/1597350058 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f632c102980 0x7f632c102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:44.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.499+0000 7f633140e640 1 --2- 172.21.15.40:0/1597350058 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f632c101790 0x7f632c101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:44.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.499+0000 7f633140e640 1 -- 172.21.15.40:0/1597350058 >> 172.21.15.40:0/1597350058 conn(0x7f632c0fcd20 msgr2=0x7f632c0ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:44.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.499+0000 7f633140e640 1 -- 172.21.15.40:0/1597350058 shutdown_connections 2024-04-21T23:17:44.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.499+0000 7f633140e640 1 -- 172.21.15.40:0/1597350058 wait complete. 2024-04-21T23:17:44.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.499+0000 7f633140e640 1 Processor -- start 2024-04-21T23:17:44.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.499+0000 7f633140e640 1 -- start start 2024-04-21T23:17:44.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.499+0000 7f633140e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f632c101790 0x7f632c0754f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:44.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.500+0000 7f633140e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f632c102980 0x7f632c073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:44.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.500+0000 7f633140e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f632c103340 0x7f632c074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:44.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.500+0000 7f633140e640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f632c10d410 con 0x7f632c102980 2024-04-21T23:17:44.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.500+0000 7f633140e640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f632c10d290 con 0x7f632c101790 2024-04-21T23:17:44.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.500+0000 7f633140e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f632c10d590 con 0x7f632c103340 2024-04-21T23:17:44.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.500+0000 7f632a7fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f632c102980 0x7f632c073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:44.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.500+0000 7f632a7fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f632c102980 0x7f632c073b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38748/0 (socket says 172.21.15.40:38748) 2024-04-21T23:17:44.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.500+0000 7f632a7fc640 1 -- 172.21.15.40:0/1406323707 learned_addr learned my addr 172.21.15.40:0/1406323707 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:17:44.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.500+0000 7f632affd640 1 --2- 172.21.15.40:0/1406323707 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f632c101790 0x7f632c0754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:44.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.500+0000 7f632b7fe640 1 --2- 172.21.15.40:0/1406323707 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f632c103340 0x7f632c074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:44.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.500+0000 7f632a7fc640 1 -- 172.21.15.40:0/1406323707 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f632c103340 msgr2=0x7f632c074070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:44.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.500+0000 7f632a7fc640 1 --2- 172.21.15.40:0/1406323707 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f632c103340 0x7f632c074070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:44.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.500+0000 7f632a7fc640 1 -- 172.21.15.40:0/1406323707 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f632c101790 msgr2=0x7f632c0754f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:44.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.500+0000 7f632a7fc640 1 --2- 172.21.15.40:0/1406323707 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f632c101790 0x7f632c0754f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:44.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.500+0000 7f632a7fc640 1 -- 172.21.15.40:0/1406323707 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f632c074900 con 0x7f632c102980 2024-04-21T23:17:44.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.500+0000 7f632a7fc640 1 --2- 172.21.15.40:0/1406323707 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f632c102980 0x7f632c073b30 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f632000c970 tx=0x7f632000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:44.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.500+0000 7f6313fff640 1 -- 172.21.15.40:0/1406323707 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6320007bb0 con 0x7f632c102980 2024-04-21T23:17:44.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.500+0000 7f633140e640 1 -- 172.21.15.40:0/1406323707 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f632c1a1ba0 con 0x7f632c102980 2024-04-21T23:17:44.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.500+0000 7f6313fff640 1 -- 172.21.15.40:0/1406323707 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6320007d50 con 0x7f632c102980 2024-04-21T23:17:44.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.500+0000 7f6313fff640 1 -- 172.21.15.40:0/1406323707 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f632000f450 con 0x7f632c102980 2024-04-21T23:17:44.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.500+0000 7f633140e640 1 -- 172.21.15.40:0/1406323707 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f632c1a20f0 con 0x7f632c102980 2024-04-21T23:17:44.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.501+0000 7f633140e640 1 -- 172.21.15.40:0/1406323707 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f632c108ab0 con 0x7f632c102980 2024-04-21T23:17:44.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.503+0000 7f6313fff640 1 -- 172.21.15.40:0/1406323707 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f6320005020 con 0x7f632c102980 2024-04-21T23:17:44.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.503+0000 7f6313fff640 1 --2- 172.21.15.40:0/1406323707 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f63140a2c20 0x7f63140a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:44.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.503+0000 7f6313fff640 1 -- 172.21.15.40:0/1406323707 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f63200cf320 con 0x7f632c102980 2024-04-21T23:17:44.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.503+0000 7f632affd640 1 --2- 172.21.15.40:0/1406323707 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f63140a2c20 0x7f63140a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:44.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.504+0000 7f632affd640 1 --2- 172.21.15.40:0/1406323707 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f63140a2c20 0x7f63140a50e0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f631c005d00 tx=0x7f631c005c50 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:44.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.508+0000 7f6313fff640 1 -- 172.21.15.40:0/1406323707 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f6320014030 con 0x7f632c102980 2024-04-21T23:17:44.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.693+0000 7f633140e640 1 -- 172.21.15.40:0/1406323707 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f632c02d360 con 0x7f632c102980 2024-04-21T23:17:44.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.693+0000 7f6313fff640 1 -- 172.21.15.40:0/1406323707 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v92) ==== 93+0+10 (secure 0 0 0) 0x7f632009be60 con 0x7f632c102980 2024-04-21T23:17:44.692 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-04-21T23:17:44.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.695+0000 7f633140e640 1 -- 172.21.15.40:0/1406323707 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f63140a2c20 msgr2=0x7f63140a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:44.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.695+0000 7f633140e640 1 --2- 172.21.15.40:0/1406323707 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f63140a2c20 0x7f63140a50e0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f631c005d00 tx=0x7f631c005c50 comp rx=0 tx=0).stop 2024-04-21T23:17:44.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.695+0000 7f633140e640 1 -- 172.21.15.40:0/1406323707 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f632c102980 msgr2=0x7f632c073b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:44.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.695+0000 7f633140e640 1 --2- 172.21.15.40:0/1406323707 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f632c102980 0x7f632c073b30 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f632000c970 tx=0x7f632000ce30 comp rx=0 tx=0).stop 2024-04-21T23:17:44.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.695+0000 7f633140e640 1 -- 172.21.15.40:0/1406323707 shutdown_connections 2024-04-21T23:17:44.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.695+0000 7f633140e640 1 --2- 172.21.15.40:0/1406323707 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f63140a2c20 0x7f63140a50e0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:44.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.695+0000 7f633140e640 1 --2- 172.21.15.40:0/1406323707 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f632c103340 0x7f632c074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:44.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.695+0000 7f633140e640 1 --2- 172.21.15.40:0/1406323707 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f632c102980 0x7f632c073b30 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:44.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.695+0000 7f633140e640 1 --2- 172.21.15.40:0/1406323707 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f632c101790 0x7f632c0754f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:44.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.695+0000 7f633140e640 1 -- 172.21.15.40:0/1406323707 >> 172.21.15.40:0/1406323707 conn(0x7f632c0fcd20 msgr2=0x7f632c105c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:44.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.695+0000 7f633140e640 1 -- 172.21.15.40:0/1406323707 shutdown_connections 2024-04-21T23:17:44.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.695+0000 7f633140e640 1 -- 172.21.15.40:0/1406323707 wait complete. 2024-04-21T23:17:44.706 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T23:17:44.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.813+0000 7f199eb4f640 1 -- 172.21.15.40:0/4204948718 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1998102250 msgr2=0x7f1998108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:44.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.813+0000 7f199eb4f640 1 --2- 172.21.15.40:0/4204948718 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1998102250 0x7f1998108ae0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f1990009ca0 tx=0x7f199002c0f0 comp rx=0 tx=0).stop 2024-04-21T23:17:44.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.813+0000 7f199eb4f640 1 -- 172.21.15.40:0/4204948718 shutdown_connections 2024-04-21T23:17:44.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.813+0000 7f199eb4f640 1 --2- 172.21.15.40:0/4204948718 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1998101890 0x7f1998101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:44.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.813+0000 7f199eb4f640 1 --2- 172.21.15.40:0/4204948718 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1998102250 0x7f1998108ae0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:44.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.813+0000 7f199eb4f640 1 --2- 172.21.15.40:0/4204948718 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f19981006a0 0x7f1998100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:44.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.813+0000 7f199eb4f640 1 -- 172.21.15.40:0/4204948718 >> 172.21.15.40:0/4204948718 conn(0x7f19980fbc30 msgr2=0x7f19980fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:44.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.813+0000 7f199eb4f640 1 -- 172.21.15.40:0/4204948718 shutdown_connections 2024-04-21T23:17:44.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.813+0000 7f199eb4f640 1 -- 172.21.15.40:0/4204948718 wait complete. 2024-04-21T23:17:44.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.814+0000 7f199eb4f640 1 Processor -- start 2024-04-21T23:17:44.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.814+0000 7f199eb4f640 1 -- start start 2024-04-21T23:17:44.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.814+0000 7f199eb4f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f19981006a0 0x7f19981979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:44.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.814+0000 7f199eb4f640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1998101890 0x7f1998197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:44.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.814+0000 7f199eb4f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1998102250 0x7f199819efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:44.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.814+0000 7f199eb4f640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f199810c320 con 0x7f19981006a0 2024-04-21T23:17:44.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.814+0000 7f199eb4f640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f199810c1a0 con 0x7f1998101890 2024-04-21T23:17:44.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.814+0000 7f199eb4f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f199810c4a0 con 0x7f1998102250 2024-04-21T23:17:44.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.814+0000 7f199c8eb640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f19981006a0 0x7f19981979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:44.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.814+0000 7f199c8eb640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f19981006a0 0x7f19981979f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38754/0 (socket says 172.21.15.40:38754) 2024-04-21T23:17:44.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.814+0000 7f199c8eb640 1 -- 172.21.15.40:0/1692854403 learned_addr learned my addr 172.21.15.40:0/1692854403 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:17:44.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.814+0000 7f1997fff640 1 --2- 172.21.15.40:0/1692854403 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1998101890 0x7f1998197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:44.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.814+0000 7f199d0ec640 1 --2- 172.21.15.40:0/1692854403 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1998102250 0x7f199819efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:44.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.814+0000 7f199c8eb640 1 -- 172.21.15.40:0/1692854403 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1998102250 msgr2=0x7f199819efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:44.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.814+0000 7f199c8eb640 1 --2- 172.21.15.40:0/1692854403 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1998102250 0x7f199819efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:44.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.814+0000 7f199c8eb640 1 -- 172.21.15.40:0/1692854403 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1998101890 msgr2=0x7f1998197f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:44.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.814+0000 7f199c8eb640 1 --2- 172.21.15.40:0/1692854403 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1998101890 0x7f1998197f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:44.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.814+0000 7f199c8eb640 1 -- 172.21.15.40:0/1692854403 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f199819f610 con 0x7f19981006a0 2024-04-21T23:17:44.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.815+0000 7f199c8eb640 1 --2- 172.21.15.40:0/1692854403 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f19981006a0 0x7f19981979f0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f198c00d950 tx=0x7f198c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:44.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.815+0000 7f1995ffb640 1 -- 172.21.15.40:0/1692854403 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f198c017070 con 0x7f19981006a0 2024-04-21T23:17:44.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.815+0000 7f199eb4f640 1 -- 172.21.15.40:0/1692854403 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f199819f900 con 0x7f19981006a0 2024-04-21T23:17:44.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.815+0000 7f1995ffb640 1 -- 172.21.15.40:0/1692854403 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f198c009c30 con 0x7f19981006a0 2024-04-21T23:17:44.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.815+0000 7f1995ffb640 1 -- 172.21.15.40:0/1692854403 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f198c00f600 con 0x7f19981006a0 2024-04-21T23:17:44.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.815+0000 7f199eb4f640 1 -- 172.21.15.40:0/1692854403 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f199819fe00 con 0x7f19981006a0 2024-04-21T23:17:44.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.816+0000 7f199eb4f640 1 -- 172.21.15.40:0/1692854403 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19981a0030 con 0x7f19981006a0 2024-04-21T23:17:44.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.817+0000 7f1995ffb640 1 -- 172.21.15.40:0/1692854403 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f198c01a070 con 0x7f19981006a0 2024-04-21T23:17:44.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.818+0000 7f1995ffb640 1 --2- 172.21.15.40:0/1692854403 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f19880a2c20 0x7f19880a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:44.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.818+0000 7f1997fff640 1 --2- 172.21.15.40:0/1692854403 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f19880a2c20 0x7f19880a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:44.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.818+0000 7f1995ffb640 1 -- 172.21.15.40:0/1692854403 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f198c0cf0d0 con 0x7f19981006a0 2024-04-21T23:17:44.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.818+0000 7f1997fff640 1 --2- 172.21.15.40:0/1692854403 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f19880a2c20 0x7f19880a50e0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f1984005e10 tx=0x7f1984005da0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:44.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:44.822+0000 7f1995ffb640 1 -- 172.21.15.40:0/1692854403 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f198c009de0 con 0x7f19981006a0 2024-04-21T23:17:45.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.007+0000 7f199eb4f640 1 -- 172.21.15.40:0/1692854403 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f19981a0200 con 0x7f19981006a0 2024-04-21T23:17:45.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.008+0000 7f1995ffb640 1 -- 172.21.15.40:0/1692854403 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v92) ==== 104+0+11 (secure 0 0 0) 0x7f198c09bb10 con 0x7f19981006a0 2024-04-21T23:17:45.007 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-21T23:17:45.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.010+0000 7f199eb4f640 1 -- 172.21.15.40:0/1692854403 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f19880a2c20 msgr2=0x7f19880a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:45.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.010+0000 7f199eb4f640 1 --2- 172.21.15.40:0/1692854403 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f19880a2c20 0x7f19880a50e0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f1984005e10 tx=0x7f1984005da0 comp rx=0 tx=0).stop 2024-04-21T23:17:45.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.010+0000 7f199eb4f640 1 -- 172.21.15.40:0/1692854403 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f19981006a0 msgr2=0x7f19981979f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:45.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.010+0000 7f199eb4f640 1 --2- 172.21.15.40:0/1692854403 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f19981006a0 0x7f19981979f0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f198c00d950 tx=0x7f198c00de10 comp rx=0 tx=0).stop 2024-04-21T23:17:45.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.010+0000 7f199eb4f640 1 -- 172.21.15.40:0/1692854403 shutdown_connections 2024-04-21T23:17:45.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.010+0000 7f199eb4f640 1 --2- 172.21.15.40:0/1692854403 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f19880a2c20 0x7f19880a50e0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:45.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.010+0000 7f199eb4f640 1 --2- 172.21.15.40:0/1692854403 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1998101890 0x7f1998197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:45.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.010+0000 7f199eb4f640 1 --2- 172.21.15.40:0/1692854403 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1998102250 0x7f199819efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:45.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.010+0000 7f199eb4f640 1 --2- 172.21.15.40:0/1692854403 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f19981006a0 0x7f19981979f0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:45.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.010+0000 7f199eb4f640 1 -- 172.21.15.40:0/1692854403 >> 172.21.15.40:0/1692854403 conn(0x7f19980fbc30 msgr2=0x7f19980fd620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:45.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.010+0000 7f199eb4f640 1 -- 172.21.15.40:0/1692854403 shutdown_connections 2024-04-21T23:17:45.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.010+0000 7f199eb4f640 1 -- 172.21.15.40:0/1692854403 wait complete. 2024-04-21T23:17:45.019 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:44 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/1922020847' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:17:45.019 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:44 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/1406323707' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:17:45.020 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T23:17:45.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.126+0000 7f62672b5640 1 -- 172.21.15.40:0/3266338210 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6260102980 msgr2=0x7f6260102e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:45.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.126+0000 7f62672b5640 1 --2- 172.21.15.40:0/3266338210 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6260102980 0x7f6260102e00 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f6250009900 tx=0x7f625002bc30 comp rx=0 tx=0).stop 2024-04-21T23:17:45.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.126+0000 7f62672b5640 1 -- 172.21.15.40:0/3266338210 shutdown_connections 2024-04-21T23:17:45.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.126+0000 7f62672b5640 1 --2- 172.21.15.40:0/3266338210 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6260103340 0x7f6260109bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:45.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.126+0000 7f62672b5640 1 --2- 172.21.15.40:0/3266338210 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6260102980 0x7f6260102e00 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:45.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.126+0000 7f62672b5640 1 --2- 172.21.15.40:0/3266338210 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6260101790 0x7f6260101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:45.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.126+0000 7f62672b5640 1 -- 172.21.15.40:0/3266338210 >> 172.21.15.40:0/3266338210 conn(0x7f62600fcd20 msgr2=0x7f62600ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:45.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.126+0000 7f62672b5640 1 -- 172.21.15.40:0/3266338210 shutdown_connections 2024-04-21T23:17:45.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.126+0000 7f62672b5640 1 -- 172.21.15.40:0/3266338210 wait complete. 2024-04-21T23:17:45.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.127+0000 7f62672b5640 1 Processor -- start 2024-04-21T23:17:45.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.127+0000 7f62672b5640 1 -- start start 2024-04-21T23:17:45.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.127+0000 7f62672b5640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6260101790 0x7f6260197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:45.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.127+0000 7f62672b5640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6260102980 0x7f6260197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:45.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.127+0000 7f62672b5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6260103340 0x7f626019f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:45.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.127+0000 7f62672b5640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f626010d410 con 0x7f6260103340 2024-04-21T23:17:45.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.127+0000 7f62672b5640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f626010d290 con 0x7f6260102980 2024-04-21T23:17:45.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.127+0000 7f62672b5640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f626010d590 con 0x7f6260101790 2024-04-21T23:17:45.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.127+0000 7f6265852640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6260103340 0x7f626019f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:45.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.127+0000 7f6265852640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6260103340 0x7f626019f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38782/0 (socket says 172.21.15.40:38782) 2024-04-21T23:17:45.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.127+0000 7f6265852640 1 -- 172.21.15.40:0/4292112888 learned_addr learned my addr 172.21.15.40:0/4292112888 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:17:45.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.127+0000 7f6265051640 1 --2- 172.21.15.40:0/4292112888 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6260101790 0x7f6260197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:45.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.127+0000 7f6264850640 1 --2- 172.21.15.40:0/4292112888 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6260102980 0x7f6260197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:45.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.127+0000 7f6265852640 1 -- 172.21.15.40:0/4292112888 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6260101790 msgr2=0x7f6260197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:45.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.127+0000 7f6265852640 1 --2- 172.21.15.40:0/4292112888 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6260101790 0x7f6260197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:45.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.127+0000 7f6265852640 1 -- 172.21.15.40:0/4292112888 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6260102980 msgr2=0x7f6260197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:45.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.127+0000 7f6265852640 1 --2- 172.21.15.40:0/4292112888 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6260102980 0x7f6260197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:45.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.127+0000 7f6265852640 1 -- 172.21.15.40:0/4292112888 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f626019f6c0 con 0x7f6260103340 2024-04-21T23:17:45.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.127+0000 7f6264850640 1 --2- 172.21.15.40:0/4292112888 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6260102980 0x7f6260197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:17:45.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.128+0000 7f6265852640 1 --2- 172.21.15.40:0/4292112888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6260103340 0x7f626019f050 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f625800eb10 tx=0x7f625800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:45.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.128+0000 7f62567fc640 1 -- 172.21.15.40:0/4292112888 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6258004b20 con 0x7f6260103340 2024-04-21T23:17:45.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.128+0000 7f62672b5640 1 -- 172.21.15.40:0/4292112888 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f626019f9b0 con 0x7f6260103340 2024-04-21T23:17:45.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.128+0000 7f62567fc640 1 -- 172.21.15.40:0/4292112888 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6258004cc0 con 0x7f6260103340 2024-04-21T23:17:45.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.128+0000 7f62567fc640 1 -- 172.21.15.40:0/4292112888 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6258010760 con 0x7f6260103340 2024-04-21T23:17:45.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.128+0000 7f62672b5640 1 -- 172.21.15.40:0/4292112888 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f626019feb0 con 0x7f6260103340 2024-04-21T23:17:45.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.129+0000 7f62672b5640 1 -- 172.21.15.40:0/4292112888 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6260108b80 con 0x7f6260103340 2024-04-21T23:17:45.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.130+0000 7f62567fc640 1 -- 172.21.15.40:0/4292112888 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f6258005020 con 0x7f6260103340 2024-04-21T23:17:45.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.131+0000 7f62567fc640 1 --2- 172.21.15.40:0/4292112888 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f624c0a2c20 0x7f624c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:45.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.131+0000 7f62567fc640 1 -- 172.21.15.40:0/4292112888 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f62580cfb90 con 0x7f6260103340 2024-04-21T23:17:45.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.131+0000 7f6265051640 1 --2- 172.21.15.40:0/4292112888 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f624c0a2c20 0x7f624c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:45.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.131+0000 7f6265051640 1 --2- 172.21.15.40:0/4292112888 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f624c0a2c20 0x7f624c0a50e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f62601027e0 tx=0x7f625c005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:45.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.135+0000 7f62567fc640 1 -- 172.21.15.40:0/4292112888 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f625809c630 con 0x7f6260103340 2024-04-21T23:17:45.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:44 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/1922020847' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:17:45.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:44 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/1406323707' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:17:45.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:44 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/1922020847' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:17:45.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:44 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/1406323707' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:17:45.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.320+0000 7f62672b5640 1 -- 172.21.15.40:0/4292112888 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f6260062340 con 0x7f6260103340 2024-04-21T23:17:45.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.321+0000 7f62567fc640 1 -- 172.21.15.40:0/4292112888 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v92) ==== 100+0+11 (secure 0 0 0) 0x7f625809bf30 con 0x7f6260103340 2024-04-21T23:17:45.320 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-04-21T23:17:45.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.323+0000 7f62672b5640 1 -- 172.21.15.40:0/4292112888 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f624c0a2c20 msgr2=0x7f624c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:45.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.323+0000 7f62672b5640 1 --2- 172.21.15.40:0/4292112888 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f624c0a2c20 0x7f624c0a50e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f62601027e0 tx=0x7f625c005f90 comp rx=0 tx=0).stop 2024-04-21T23:17:45.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.323+0000 7f62672b5640 1 -- 172.21.15.40:0/4292112888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6260103340 msgr2=0x7f626019f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:45.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.323+0000 7f62672b5640 1 --2- 172.21.15.40:0/4292112888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6260103340 0x7f626019f050 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f625800eb10 tx=0x7f625800c2d0 comp rx=0 tx=0).stop 2024-04-21T23:17:45.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.323+0000 7f62672b5640 1 -- 172.21.15.40:0/4292112888 shutdown_connections 2024-04-21T23:17:45.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.323+0000 7f62672b5640 1 --2- 172.21.15.40:0/4292112888 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f624c0a2c20 0x7f624c0a50e0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:45.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.323+0000 7f62672b5640 1 --2- 172.21.15.40:0/4292112888 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6260102980 0x7f6260197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:45.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.323+0000 7f62672b5640 1 --2- 172.21.15.40:0/4292112888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6260103340 0x7f626019f050 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:45.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.323+0000 7f62672b5640 1 --2- 172.21.15.40:0/4292112888 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6260101790 0x7f6260197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:45.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.323+0000 7f62672b5640 1 -- 172.21.15.40:0/4292112888 >> 172.21.15.40:0/4292112888 conn(0x7f62600fcd20 msgr2=0x7f6260105d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:45.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.323+0000 7f62672b5640 1 -- 172.21.15.40:0/4292112888 shutdown_connections 2024-04-21T23:17:45.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.323+0000 7f62672b5640 1 -- 172.21.15.40:0/4292112888 wait complete. 2024-04-21T23:17:45.333 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T23:17:45.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.438+0000 7f1ab622e640 1 -- 172.21.15.40:0/876892091 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ab0102260 msgr2=0x7f1ab0104650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:45.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.438+0000 7f1ab622e640 1 --2- 172.21.15.40:0/876892091 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ab0102260 0x7f1ab0104650 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f1aa0009900 tx=0x7f1aa002bc30 comp rx=0 tx=0).stop 2024-04-21T23:17:45.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.439+0000 7f1ab622e640 1 -- 172.21.15.40:0/876892091 shutdown_connections 2024-04-21T23:17:45.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.439+0000 7f1ab622e640 1 --2- 172.21.15.40:0/876892091 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ab01074c0 0x7f1ab01098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:45.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.439+0000 7f1ab622e640 1 --2- 172.21.15.40:0/876892091 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1ab0104b90 0x7f1ab0106f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:45.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.439+0000 7f1ab622e640 1 --2- 172.21.15.40:0/876892091 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ab0102260 0x7f1ab0104650 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:45.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.439+0000 7f1ab622e640 1 -- 172.21.15.40:0/876892091 >> 172.21.15.40:0/876892091 conn(0x7f1ab00fbbf0 msgr2=0x7f1ab00fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:45.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.439+0000 7f1ab622e640 1 -- 172.21.15.40:0/876892091 shutdown_connections 2024-04-21T23:17:45.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.439+0000 7f1ab622e640 1 -- 172.21.15.40:0/876892091 wait complete. 2024-04-21T23:17:45.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.439+0000 7f1ab622e640 1 Processor -- start 2024-04-21T23:17:45.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.439+0000 7f1ab622e640 1 -- start start 2024-04-21T23:17:45.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.439+0000 7f1ab622e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1ab0102260 0x7f1ab0197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:45.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.439+0000 7f1ab622e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ab0104b90 0x7f1ab0198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:45.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.439+0000 7f1ab622e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ab01074c0 0x7f1ab019f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:45.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.439+0000 7f1ab622e640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1ab010d420 con 0x7f1ab01074c0 2024-04-21T23:17:45.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.439+0000 7f1ab622e640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f1ab010d2a0 con 0x7f1ab0104b90 2024-04-21T23:17:45.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.439+0000 7f1ab622e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1ab010d5a0 con 0x7f1ab0102260 2024-04-21T23:17:45.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.440+0000 7f1aaffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ab01074c0 0x7f1ab019f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:45.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.440+0000 7f1aaffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ab01074c0 0x7f1ab019f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38792/0 (socket says 172.21.15.40:38792) 2024-04-21T23:17:45.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.440+0000 7f1aaffff640 1 -- 172.21.15.40:0/2517505359 learned_addr learned my addr 172.21.15.40:0/2517505359 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:17:45.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.440+0000 7f1aaeffd640 1 --2- 172.21.15.40:0/2517505359 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ab0104b90 0x7f1ab0198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:45.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.440+0000 7f1aaf7fe640 1 --2- 172.21.15.40:0/2517505359 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1ab0102260 0x7f1ab0197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:45.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.440+0000 7f1aaffff640 1 -- 172.21.15.40:0/2517505359 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1ab0102260 msgr2=0x7f1ab0197b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:45.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.440+0000 7f1aaffff640 1 --2- 172.21.15.40:0/2517505359 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1ab0102260 0x7f1ab0197b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:45.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.440+0000 7f1aaffff640 1 -- 172.21.15.40:0/2517505359 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ab0104b90 msgr2=0x7f1ab0198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:45.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.440+0000 7f1aaffff640 1 --2- 172.21.15.40:0/2517505359 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ab0104b90 0x7f1ab0198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:45.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.440+0000 7f1aaffff640 1 -- 172.21.15.40:0/2517505359 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ab019f770 con 0x7f1ab01074c0 2024-04-21T23:17:45.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.440+0000 7f1aaf7fe640 1 --2- 172.21.15.40:0/2517505359 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1ab0102260 0x7f1ab0197b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:17:45.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.440+0000 7f1aaffff640 1 --2- 172.21.15.40:0/2517505359 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ab01074c0 0x7f1ab019f100 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f1aa8009fd0 tx=0x7f1aa800b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:45.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.440+0000 7f1aacff9640 1 -- 172.21.15.40:0/2517505359 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aa8017070 con 0x7f1ab01074c0 2024-04-21T23:17:45.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.440+0000 7f1ab622e640 1 -- 172.21.15.40:0/2517505359 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ab019fa60 con 0x7f1ab01074c0 2024-04-21T23:17:45.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.440+0000 7f1aacff9640 1 -- 172.21.15.40:0/2517505359 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1aa8003e50 con 0x7f1ab01074c0 2024-04-21T23:17:45.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.440+0000 7f1aacff9640 1 -- 172.21.15.40:0/2517505359 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aa80106c0 con 0x7f1ab01074c0 2024-04-21T23:17:45.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.440+0000 7f1ab622e640 1 -- 172.21.15.40:0/2517505359 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ab019ffb0 con 0x7f1ab01074c0 2024-04-21T23:17:45.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.441+0000 7f1ab622e640 1 -- 172.21.15.40:0/2517505359 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ab00fd6e0 con 0x7f1ab01074c0 2024-04-21T23:17:45.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.443+0000 7f1aacff9640 1 -- 172.21.15.40:0/2517505359 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f1aa8003970 con 0x7f1ab01074c0 2024-04-21T23:17:45.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.443+0000 7f1aacff9640 1 --2- 172.21.15.40:0/2517505359 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f1a980a2c20 0x7f1a980a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:45.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.443+0000 7f1aaf7fe640 1 --2- 172.21.15.40:0/2517505359 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f1a980a2c20 0x7f1a980a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:45.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.443+0000 7f1aacff9640 1 -- 172.21.15.40:0/2517505359 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f1aa80cf380 con 0x7f1ab01074c0 2024-04-21T23:17:45.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.444+0000 7f1aaf7fe640 1 --2- 172.21.15.40:0/2517505359 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f1a980a2c20 0x7f1a980a50e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f1aa00096c0 tx=0x7f1aa0000bc0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:45.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.448+0000 7f1aacff9640 1 -- 172.21.15.40:0/2517505359 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f1aa809be20 con 0x7f1ab01074c0 2024-04-21T23:17:45.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.632+0000 7f1ab622e640 1 -- 172.21.15.40:0/2517505359 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f1ab0062340 con 0x7f1ab01074c0 2024-04-21T23:17:45.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.632+0000 7f1aacff9640 1 -- 172.21.15.40:0/2517505359 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v92) ==== 103+0+11 (secure 0 0 0) 0x7f1aa809b720 con 0x7f1ab01074c0 2024-04-21T23:17:45.631 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-21T23:17:45.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.634+0000 7f1ab622e640 1 -- 172.21.15.40:0/2517505359 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f1a980a2c20 msgr2=0x7f1a980a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:45.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.634+0000 7f1ab622e640 1 --2- 172.21.15.40:0/2517505359 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f1a980a2c20 0x7f1a980a50e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f1aa00096c0 tx=0x7f1aa0000bc0 comp rx=0 tx=0).stop 2024-04-21T23:17:45.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.634+0000 7f1ab622e640 1 -- 172.21.15.40:0/2517505359 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ab01074c0 msgr2=0x7f1ab019f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:45.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.634+0000 7f1ab622e640 1 --2- 172.21.15.40:0/2517505359 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ab01074c0 0x7f1ab019f100 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f1aa8009fd0 tx=0x7f1aa800b260 comp rx=0 tx=0).stop 2024-04-21T23:17:45.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.634+0000 7f1ab622e640 1 -- 172.21.15.40:0/2517505359 shutdown_connections 2024-04-21T23:17:45.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.634+0000 7f1ab622e640 1 --2- 172.21.15.40:0/2517505359 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f1a980a2c20 0x7f1a980a50e0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:45.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.634+0000 7f1ab622e640 1 --2- 172.21.15.40:0/2517505359 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ab0104b90 0x7f1ab0198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:45.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.634+0000 7f1ab622e640 1 --2- 172.21.15.40:0/2517505359 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ab01074c0 0x7f1ab019f100 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:45.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.634+0000 7f1ab622e640 1 --2- 172.21.15.40:0/2517505359 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1ab0102260 0x7f1ab0197b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:45.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.634+0000 7f1ab622e640 1 -- 172.21.15.40:0/2517505359 >> 172.21.15.40:0/2517505359 conn(0x7f1ab00fbbf0 msgr2=0x7f1ab0102b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:45.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.634+0000 7f1ab622e640 1 -- 172.21.15.40:0/2517505359 shutdown_connections 2024-04-21T23:17:45.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.634+0000 7f1ab622e640 1 -- 172.21.15.40:0/2517505359 wait complete. 2024-04-21T23:17:45.645 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:17:45.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.747+0000 7f5fcb102640 1 -- 172.21.15.40:0/4028293149 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5fc4102250 msgr2=0x7f5fc4108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:45.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.747+0000 7f5fcb102640 1 --2- 172.21.15.40:0/4028293149 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5fc4102250 0x7f5fc4108ae0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f5fbc009ca0 tx=0x7f5fbc02c100 comp rx=0 tx=0).stop 2024-04-21T23:17:45.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.747+0000 7f5fcb102640 1 -- 172.21.15.40:0/4028293149 shutdown_connections 2024-04-21T23:17:45.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.747+0000 7f5fcb102640 1 --2- 172.21.15.40:0/4028293149 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5fc4102250 0x7f5fc4108ae0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:45.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.747+0000 7f5fcb102640 1 --2- 172.21.15.40:0/4028293149 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5fc4101890 0x7f5fc4101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:45.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.747+0000 7f5fcb102640 1 --2- 172.21.15.40:0/4028293149 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5fc41006a0 0x7f5fc4100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:45.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.747+0000 7f5fcb102640 1 -- 172.21.15.40:0/4028293149 >> 172.21.15.40:0/4028293149 conn(0x7f5fc40fbc30 msgr2=0x7f5fc40fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:45.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.747+0000 7f5fcb102640 1 -- 172.21.15.40:0/4028293149 shutdown_connections 2024-04-21T23:17:45.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.747+0000 7f5fcb102640 1 -- 172.21.15.40:0/4028293149 wait complete. 2024-04-21T23:17:45.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.747+0000 7f5fcb102640 1 Processor -- start 2024-04-21T23:17:45.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.747+0000 7f5fcb102640 1 -- start start 2024-04-21T23:17:45.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.748+0000 7f5fcb102640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5fc41006a0 0x7f5fc4197990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:45.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.748+0000 7f5fcb102640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5fc4101890 0x7f5fc4197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:45.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.748+0000 7f5fcb102640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5fc4102250 0x7f5fc419ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:45.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.748+0000 7f5fcb102640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5fc410c320 con 0x7f5fc4101890 2024-04-21T23:17:45.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.748+0000 7f5fcb102640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f5fc410c1a0 con 0x7f5fc4102250 2024-04-21T23:17:45.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.748+0000 7f5fcb102640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5fc410c4a0 con 0x7f5fc41006a0 2024-04-21T23:17:45.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.748+0000 7f5fc3fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5fc4101890 0x7f5fc4197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:45.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.748+0000 7f5fc3fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5fc4101890 0x7f5fc4197ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38818/0 (socket says 172.21.15.40:38818) 2024-04-21T23:17:45.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.748+0000 7f5fc3fff640 1 -- 172.21.15.40:0/3653615762 learned_addr learned my addr 172.21.15.40:0/3653615762 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:17:45.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.748+0000 7f5fc8e9e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5fc41006a0 0x7f5fc4197990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:45.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.748+0000 7f5fc969f640 1 --2- 172.21.15.40:0/3653615762 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5fc4102250 0x7f5fc419ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:45.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.748+0000 7f5fc3fff640 1 -- 172.21.15.40:0/3653615762 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5fc41006a0 msgr2=0x7f5fc4197990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:45.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.748+0000 7f5fc3fff640 1 --2- 172.21.15.40:0/3653615762 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5fc41006a0 0x7f5fc4197990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:45.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.748+0000 7f5fc3fff640 1 -- 172.21.15.40:0/3653615762 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5fc4102250 msgr2=0x7f5fc419ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:45.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.748+0000 7f5fc3fff640 1 --2- 172.21.15.40:0/3653615762 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5fc4102250 0x7f5fc419ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:45.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.748+0000 7f5fc3fff640 1 -- 172.21.15.40:0/3653615762 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fc419f5b0 con 0x7f5fc4101890 2024-04-21T23:17:45.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.748+0000 7f5fc8e9e640 1 --2- 172.21.15.40:0/3653615762 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5fc41006a0 0x7f5fc4197990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:17:45.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.748+0000 7f5fc3fff640 1 --2- 172.21.15.40:0/3653615762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5fc4101890 0x7f5fc4197ed0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f5fb800c970 tx=0x7f5fb800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:45.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.751+0000 7f5fc1ffb640 1 -- 172.21.15.40:0/3653615762 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fb8007bb0 con 0x7f5fc4101890 2024-04-21T23:17:45.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.751+0000 7f5fc1ffb640 1 -- 172.21.15.40:0/3653615762 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5fb8007d50 con 0x7f5fc4101890 2024-04-21T23:17:45.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.751+0000 7f5fc1ffb640 1 -- 172.21.15.40:0/3653615762 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fb800f450 con 0x7f5fc4101890 2024-04-21T23:17:45.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.751+0000 7f5fcb102640 1 -- 172.21.15.40:0/3653615762 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fc419f8a0 con 0x7f5fc4101890 2024-04-21T23:17:45.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.752+0000 7f5fcb102640 1 -- 172.21.15.40:0/3653615762 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fc419fdf0 con 0x7f5fc4101890 2024-04-21T23:17:45.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.753+0000 7f5fcb102640 1 -- 172.21.15.40:0/3653615762 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fc41a0050 con 0x7f5fc4101890 2024-04-21T23:17:45.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.754+0000 7f5fc1ffb640 1 -- 172.21.15.40:0/3653615762 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f5fb8005020 con 0x7f5fc4101890 2024-04-21T23:17:45.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.754+0000 7f5fc1ffb640 1 --2- 172.21.15.40:0/3653615762 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f5fac0a2ca0 0x7f5fac0a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:45.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.754+0000 7f5fc1ffb640 1 -- 172.21.15.40:0/3653615762 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f5fb80cf1e0 con 0x7f5fc4101890 2024-04-21T23:17:45.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.754+0000 7f5fc8e9e640 1 --2- 172.21.15.40:0/3653615762 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f5fac0a2ca0 0x7f5fac0a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:45.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.755+0000 7f5fc8e9e640 1 --2- 172.21.15.40:0/3653615762 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f5fac0a2ca0 0x7f5fac0a5160 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5fc41016f0 tx=0x7f5fb4005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:45.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.759+0000 7f5fc1ffb640 1 -- 172.21.15.40:0/3653615762 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f5fb8014030 con 0x7f5fc4101890 2024-04-21T23:17:45.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:45 smithi040 ceph-mon[27984]: pgmap v349: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T23:17:45.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:45 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/1692854403' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T23:17:45.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:45 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/4292112888' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T23:17:45.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:45 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/2517505359' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T23:17:45.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.983+0000 7f5fcb102640 1 -- 172.21.15.40:0/3653615762 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5fc41a0220 con 0x7f5fc4101890 2024-04-21T23:17:45.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.983+0000 7f5fc1ffb640 1 -- 172.21.15.40:0/3653615762 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11882 (secure 0 0 0) 0x7f5fb809b630 con 0x7f5fc4101890 2024-04-21T23:17:45.983 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:17:45.983 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/1571948830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1571948830},{"type":"v1","addr":"172.21.15.40:6835","nonce":1571948830}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/969356642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":969356642},{"type":"v1","addr":"172.21.15.40:6837","nonce":969356642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6839/3025057845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":3025057845},{"type":"v1","addr":"172.21.15.40:6839","nonce":3025057845}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6841/3350488804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":3350488804},{"type":"v1","addr":"172.21.15.40:6841","nonce":3350488804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/3438493203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":3438493203},{"type":"v1","addr":"172.21.15.174:6833","nonce":3438493203}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/3105752678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":3105752678},{"type":"v1","addr":"172.21.15.111:6833","nonce":3105752678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T23:15:55.742233+0000","modified":"2024-04-21T23:16:22.498043+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24521,"mds_1":24502,"mds_2":24494},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24461":{"gid":24461,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6835/2704462092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2704462092},{"type":"v1","addr":"172.21.15.111:6835","nonce":2704462092}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/3905733045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3905733045},{"type":"v1","addr":"172.21.15.174:6835","nonce":3905733045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6837/465579746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":465579746},{"type":"v1","addr":"172.21.15.174:6837","nonce":465579746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/2703764182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":2703764182},{"type":"v1","addr":"172.21.15.111:6837","nonce":2703764182}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24502":{"gid":24502,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.174:6839/2467509274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":2467509274},{"type":"v1","addr":"172.21.15.174:6839","nonce":2467509274}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3712235548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3712235548},{"type":"v1","addr":"172.21.15.111:6839","nonce":3712235548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24494,24502,24521]},"id":1}]} 2024-04-21T23:17:45.983 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 24 2024-04-21T23:17:45.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.986+0000 7f5fcb102640 1 -- 172.21.15.40:0/3653615762 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f5fac0a2ca0 msgr2=0x7f5fac0a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:45.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.986+0000 7f5fcb102640 1 --2- 172.21.15.40:0/3653615762 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f5fac0a2ca0 0x7f5fac0a5160 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5fc41016f0 tx=0x7f5fb4005f90 comp rx=0 tx=0).stop 2024-04-21T23:17:45.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.986+0000 7f5fcb102640 1 -- 172.21.15.40:0/3653615762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5fc4101890 msgr2=0x7f5fc4197ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:45.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.986+0000 7f5fcb102640 1 --2- 172.21.15.40:0/3653615762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5fc4101890 0x7f5fc4197ed0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f5fb800c970 tx=0x7f5fb800ce30 comp rx=0 tx=0).stop 2024-04-21T23:17:45.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.986+0000 7f5fcb102640 1 -- 172.21.15.40:0/3653615762 shutdown_connections 2024-04-21T23:17:45.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.986+0000 7f5fcb102640 1 --2- 172.21.15.40:0/3653615762 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f5fac0a2ca0 0x7f5fac0a5160 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:45.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.986+0000 7f5fcb102640 1 --2- 172.21.15.40:0/3653615762 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5fc4102250 0x7f5fc419ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:45.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.986+0000 7f5fcb102640 1 --2- 172.21.15.40:0/3653615762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5fc4101890 0x7f5fc4197ed0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:45.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.986+0000 7f5fcb102640 1 --2- 172.21.15.40:0/3653615762 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5fc41006a0 0x7f5fc4197990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:45.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.986+0000 7f5fcb102640 1 -- 172.21.15.40:0/3653615762 >> 172.21.15.40:0/3653615762 conn(0x7f5fc40fbc30 msgr2=0x7f5fc40fd5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:45.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.986+0000 7f5fcb102640 1 -- 172.21.15.40:0/3653615762 shutdown_connections 2024-04-21T23:17:45.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:45.986+0000 7f5fcb102640 1 -- 172.21.15.40:0/3653615762 wait complete. 2024-04-21T23:17:45.996 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T23:17:46.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.098+0000 7f530b0c6640 1 -- 172.21.15.40:0/1074047407 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5304100680 msgr2=0x7f5304100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:46.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.098+0000 7f530b0c6640 1 --2- 172.21.15.40:0/1074047407 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5304100680 0x7f5304100aa0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f52f4009900 tx=0x7f52f402bc30 comp rx=0 tx=0).stop 2024-04-21T23:17:46.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.099+0000 7f530b0c6640 1 -- 172.21.15.40:0/1074047407 shutdown_connections 2024-04-21T23:17:46.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.099+0000 7f530b0c6640 1 --2- 172.21.15.40:0/1074047407 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5304102230 0x7f5304108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:46.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.099+0000 7f530b0c6640 1 --2- 172.21.15.40:0/1074047407 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5304101870 0x7f5304101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:46.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.099+0000 7f530b0c6640 1 --2- 172.21.15.40:0/1074047407 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5304100680 0x7f5304100aa0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:46.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.099+0000 7f530b0c6640 1 -- 172.21.15.40:0/1074047407 >> 172.21.15.40:0/1074047407 conn(0x7f53040fbbf0 msgr2=0x7f53040fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:46.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.099+0000 7f530b0c6640 1 -- 172.21.15.40:0/1074047407 shutdown_connections 2024-04-21T23:17:46.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.099+0000 7f530b0c6640 1 -- 172.21.15.40:0/1074047407 wait complete. 2024-04-21T23:17:46.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.099+0000 7f530b0c6640 1 Processor -- start 2024-04-21T23:17:46.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.099+0000 7f530b0c6640 1 -- start start 2024-04-21T23:17:46.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.099+0000 7f530b0c6640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5304100680 0x7f5304069440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:46.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.099+0000 7f530b0c6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5304101870 0x7f530419c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:46.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.099+0000 7f530b0c6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5304102230 0x7f530419ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:46.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.099+0000 7f530b0c6640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f530410c300 con 0x7f5304101870 2024-04-21T23:17:46.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.099+0000 7f530b0c6640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f530410c180 con 0x7f5304100680 2024-04-21T23:17:46.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.099+0000 7f530b0c6640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f530410c480 con 0x7f5304102230 2024-04-21T23:17:46.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.099+0000 7f5303fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5304101870 0x7f530419c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:46.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.099+0000 7f5303fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5304101870 0x7f530419c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38828/0 (socket says 172.21.15.40:38828) 2024-04-21T23:17:46.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.099+0000 7f5303fff640 1 -- 172.21.15.40:0/421133995 learned_addr learned my addr 172.21.15.40:0/421133995 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:17:46.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.100+0000 7f5308e62640 1 --2- 172.21.15.40:0/421133995 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5304100680 0x7f5304069440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:46.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.100+0000 7f5303fff640 1 -- 172.21.15.40:0/421133995 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5304102230 msgr2=0x7f530419ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:46.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.100+0000 7f5303fff640 1 --2- 172.21.15.40:0/421133995 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5304102230 0x7f530419ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:46.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.100+0000 7f5303fff640 1 -- 172.21.15.40:0/421133995 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5304100680 msgr2=0x7f5304069440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:46.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.100+0000 7f5303fff640 1 --2- 172.21.15.40:0/421133995 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5304100680 0x7f5304069440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:46.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.100+0000 7f5303fff640 1 -- 172.21.15.40:0/421133995 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f530419f730 con 0x7f5304101870 2024-04-21T23:17:46.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.100+0000 7f5303fff640 1 --2- 172.21.15.40:0/421133995 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5304101870 0x7f530419c800 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f52f800d770 tx=0x7f52f800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:46.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.100+0000 7f5301ffb640 1 -- 172.21.15.40:0/421133995 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52f8017070 con 0x7f5304101870 2024-04-21T23:17:46.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.100+0000 7f5301ffb640 1 -- 172.21.15.40:0/421133995 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f52f8005500 con 0x7f5304101870 2024-04-21T23:17:46.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.100+0000 7f530b0c6640 1 -- 172.21.15.40:0/421133995 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f530419f9c0 con 0x7f5304101870 2024-04-21T23:17:46.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.100+0000 7f5301ffb640 1 -- 172.21.15.40:0/421133995 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52f8004e00 con 0x7f5304101870 2024-04-21T23:17:46.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.100+0000 7f530b0c6640 1 -- 172.21.15.40:0/421133995 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f530419ff10 con 0x7f5304101870 2024-04-21T23:17:46.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.101+0000 7f530b0c6640 1 -- 172.21.15.40:0/421133995 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53041079a0 con 0x7f5304101870 2024-04-21T23:17:46.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.102+0000 7f5301ffb640 1 -- 172.21.15.40:0/421133995 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f52f8004fa0 con 0x7f5304101870 2024-04-21T23:17:46.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.103+0000 7f5301ffb640 1 --2- 172.21.15.40:0/421133995 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f52ec0a2ca0 0x7f52ec0a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:46.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.103+0000 7f5308e62640 1 --2- 172.21.15.40:0/421133995 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f52ec0a2ca0 0x7f52ec0a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:46.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.103+0000 7f5301ffb640 1 -- 172.21.15.40:0/421133995 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f52f80cefb0 con 0x7f5304101870 2024-04-21T23:17:46.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.103+0000 7f5308e62640 1 --2- 172.21.15.40:0/421133995 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f52ec0a2ca0 0x7f52ec0a5160 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f52f402c7b0 tx=0x7f52f4005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:46.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.108+0000 7f5301ffb640 1 -- 172.21.15.40:0/421133995 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f52f809ba50 con 0x7f5304101870 2024-04-21T23:17:46.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:45 smithi174 ceph-mon[31409]: pgmap v349: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T23:17:46.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:45 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/1692854403' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T23:17:46.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:45 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/4292112888' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T23:17:46.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:45 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/2517505359' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T23:17:46.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:45 smithi111 ceph-mon[31085]: pgmap v349: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T23:17:46.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:45 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/1692854403' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T23:17:46.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:45 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/4292112888' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T23:17:46.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:45 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/2517505359' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T23:17:46.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.293+0000 7f530b0c6640 1 -- 172.21.15.40:0/421133995 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f530402d320 con 0x7f5304101870 2024-04-21T23:17:46.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.294+0000 7f5301ffb640 1 -- 172.21.15.40:0/421133995 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+24666 (secure 0 0 0) 0x7f52f809b350 con 0x7f5304101870 2024-04-21T23:17:46.294 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:17:46.294 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":92,"fsid":"d814e4dc-0033-11ef-bc93-c7b262605968","created":"2024-04-21T23:07:52.659207+0000","modified":"2024-04-21T23:16:52.188054+0000","last_up_change":"2024-04-21T23:15:05.308821+0000","last_in_change":"2024-04-21T23:14:45.447491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:11:14.824775+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T23:15:54.322571+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T23:15:55.164493+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T23:15:58.838266+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"0e8d692a-22fe-4a87-8d99-9151f96cbb02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6803","nonce":2009758784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6805","nonce":2009758784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6809","nonce":2009758784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6807","nonce":2009758784}]},"public_addr":"172.21.15.40:6803/2009758784","cluster_addr":"172.21.15.40:6805/2009758784","heartbeat_back_addr":"172.21.15.40:6809/2009758784","heartbeat_front_addr":"172.21.15.40:6807/2009758784","state":["exists","up"]},{"osd":1,"uuid":"fe5fe11b-0e84-4359-83a7-1ba45105755a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6801","nonce":1763903836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6803","nonce":1763903836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6807","nonce":1763903836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6805","nonce":1763903836}]},"public_addr":"172.21.15.111:6801/1763903836","cluster_addr":"172.21.15.111:6803/1763903836","heartbeat_back_addr":"172.21.15.111:6807/1763903836","heartbeat_front_addr":"172.21.15.111:6805/1763903836","state":["exists","up"]},{"osd":2,"uuid":"813ad735-5cc2-4da0-ae16-6db238395fbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6801","nonce":439695825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6803","nonce":439695825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6807","nonce":439695825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6805","nonce":439695825}]},"public_addr":"172.21.15.174:6801/439695825","cluster_addr":"172.21.15.174:6803/439695825","heartbeat_back_addr":"172.21.15.174:6807/439695825","heartbeat_front_addr":"172.21.15.174:6805/439695825","state":["exists","up"]},{"osd":3,"uuid":"09becb03-b7ce-4ffc-917d-b7900ec37e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6811","nonce":3454312436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6813","nonce":3454312436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6817","nonce":3454312436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6815","nonce":3454312436}]},"public_addr":"172.21.15.40:6811/3454312436","cluster_addr":"172.21.15.40:6813/3454312436","heartbeat_back_addr":"172.21.15.40:6817/3454312436","heartbeat_front_addr":"172.21.15.40:6815/3454312436","state":["exists","up"]},{"osd":4,"uuid":"46eabfd7-d88a-4efd-94eb-04354d3a9d7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6809","nonce":1810866070}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6811","nonce":1810866070}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6815","nonce":1810866070}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6813","nonce":1810866070}]},"public_addr":"172.21.15.111:6809/1810866070","cluster_addr":"172.21.15.111:6811/1810866070","heartbeat_back_addr":"172.21.15.111:6815/1810866070","heartbeat_front_addr":"172.21.15.111:6813/1810866070","state":["exists","up"]},{"osd":5,"uuid":"6f73b582-6c65-41db-aab0-9b360f83d5c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6809","nonce":3752986711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6811","nonce":3752986711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6815","nonce":3752986711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6813","nonce":3752986711}]},"public_addr":"172.21.15.174:6809/3752986711","cluster_addr":"172.21.15.174:6811/3752986711","heartbeat_back_addr":"172.21.15.174:6815/3752986711","heartbeat_front_addr":"172.21.15.174:6813/3752986711","state":["exists","up"]},{"osd":6,"uuid":"39f305f0-12d2-4732-aaa0-cd66700af538","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6819","nonce":1618855427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6821","nonce":1618855427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6825","nonce":1618855427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6823","nonce":1618855427}]},"public_addr":"172.21.15.40:6819/1618855427","cluster_addr":"172.21.15.40:6821/1618855427","heartbeat_back_addr":"172.21.15.40:6825/1618855427","heartbeat_front_addr":"172.21.15.40:6823/1618855427","state":["exists","up"]},{"osd":7,"uuid":"769f2468-5102-490b-89f8-d0444d3169a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6817","nonce":927054162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6819","nonce":927054162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6823","nonce":927054162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6821","nonce":927054162}]},"public_addr":"172.21.15.111:6817/927054162","cluster_addr":"172.21.15.111:6819/927054162","heartbeat_back_addr":"172.21.15.111:6823/927054162","heartbeat_front_addr":"172.21.15.111:6821/927054162","state":["exists","up"]},{"osd":8,"uuid":"1a444e80-007d-473b-b972-4c814a813c22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6817","nonce":3817848746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6819","nonce":3817848746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6823","nonce":3817848746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6821","nonce":3817848746}]},"public_addr":"172.21.15.174:6817/3817848746","cluster_addr":"172.21.15.174:6819/3817848746","heartbeat_back_addr":"172.21.15.174:6823/3817848746","heartbeat_front_addr":"172.21.15.174:6821/3817848746","state":["exists","up"]},{"osd":9,"uuid":"9a6b6a56-6ae5-49fe-9e5b-cb17d686e0dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6827","nonce":1649446531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6829","nonce":1649446531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6833","nonce":1649446531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6831","nonce":1649446531}]},"public_addr":"172.21.15.40:6827/1649446531","cluster_addr":"172.21.15.40:6829/1649446531","heartbeat_back_addr":"172.21.15.40:6833/1649446531","heartbeat_front_addr":"172.21.15.40:6831/1649446531","state":["exists","up"]},{"osd":10,"uuid":"a70038ba-e2ad-4170-85cc-ea0470a2edef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6825","nonce":1227538935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6827","nonce":1227538935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6831","nonce":1227538935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6829","nonce":1227538935}]},"public_addr":"172.21.15.111:6825/1227538935","cluster_addr":"172.21.15.111:6827/1227538935","heartbeat_back_addr":"172.21.15.111:6831/1227538935","heartbeat_front_addr":"172.21.15.111:6829/1227538935","state":["exists","up"]},{"osd":11,"uuid":"781532bb-7a5e-44e3-8eb7-b584b0054666","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6825","nonce":2045865553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6827","nonce":2045865553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6831","nonce":2045865553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6829","nonce":2045865553}]},"public_addr":"172.21.15.174:6825/2045865553","cluster_addr":"172.21.15.174:6827/2045865553","heartbeat_back_addr":"172.21.15.174:6831/2045865553","heartbeat_front_addr":"172.21.15.174:6829/2045865553","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:10:24.761290+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:10:48.565428+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:11:12.307818+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:11:37.951529+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:02.832523+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:27.568879+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:52.997707+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:13:18.937274+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:13:44.708657+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:14:10.833855+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:14:37.078272+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:15:03.878076+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":1}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.2a","mappings":[{"from":3,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:6801/526554319":"2024-04-22T23:08:50.305770+0000","172.21.15.40:6800/526554319":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/3178763471":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/4045911364":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/276300080":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/3689649814":"2024-04-22T23:08:23.743030+0000","172.21.15.40:0/2241116715":"2024-04-22T23:08:23.743030+0000","172.21.15.40:6801/1479040326":"2024-04-22T23:08:23.743030+0000","172.21.15.40:6800/1479040326":"2024-04-22T23:08:23.743030+0000","172.21.15.40:0/681298099":"2024-04-22T23:08:23.743030+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T23:17:46.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.297+0000 7f530b0c6640 1 -- 172.21.15.40:0/421133995 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f52ec0a2ca0 msgr2=0x7f52ec0a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:46.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.297+0000 7f530b0c6640 1 --2- 172.21.15.40:0/421133995 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f52ec0a2ca0 0x7f52ec0a5160 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f52f402c7b0 tx=0x7f52f4005fd0 comp rx=0 tx=0).stop 2024-04-21T23:17:46.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.297+0000 7f530b0c6640 1 -- 172.21.15.40:0/421133995 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5304101870 msgr2=0x7f530419c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:46.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.297+0000 7f530b0c6640 1 --2- 172.21.15.40:0/421133995 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5304101870 0x7f530419c800 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f52f800d770 tx=0x7f52f800dc30 comp rx=0 tx=0).stop 2024-04-21T23:17:46.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.297+0000 7f530b0c6640 1 -- 172.21.15.40:0/421133995 shutdown_connections 2024-04-21T23:17:46.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.297+0000 7f530b0c6640 1 --2- 172.21.15.40:0/421133995 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f52ec0a2ca0 0x7f52ec0a5160 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:46.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.297+0000 7f530b0c6640 1 --2- 172.21.15.40:0/421133995 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5304102230 0x7f530419ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:46.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.297+0000 7f530b0c6640 1 --2- 172.21.15.40:0/421133995 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5304101870 0x7f530419c800 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:46.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.297+0000 7f530b0c6640 1 --2- 172.21.15.40:0/421133995 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5304100680 0x7f5304069440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:46.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.297+0000 7f530b0c6640 1 -- 172.21.15.40:0/421133995 >> 172.21.15.40:0/421133995 conn(0x7f53040fbbf0 msgr2=0x7f5304104b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:46.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.297+0000 7f530b0c6640 1 -- 172.21.15.40:0/421133995 shutdown_connections 2024-04-21T23:17:46.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.297+0000 7f530b0c6640 1 -- 172.21.15.40:0/421133995 wait complete. 2024-04-21T23:17:46.308 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T23:11:14.824775+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T23:15:54.322571+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T23:15:55.164493+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T23:15:58.838266+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T23:17:46.308 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T23:17:46.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.414+0000 7f3e4ccdf640 1 -- 172.21.15.40:0/558543488 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e481006a0 msgr2=0x7f3e48100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:46.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.414+0000 7f3e4ccdf640 1 --2- 172.21.15.40:0/558543488 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e481006a0 0x7f3e48100ac0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f3e38009900 tx=0x7f3e3802bc30 comp rx=0 tx=0).stop 2024-04-21T23:17:46.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.414+0000 7f3e4ccdf640 1 -- 172.21.15.40:0/558543488 shutdown_connections 2024-04-21T23:17:46.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.414+0000 7f3e4ccdf640 1 --2- 172.21.15.40:0/558543488 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3e48101890 0x7f3e48101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:46.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.414+0000 7f3e4ccdf640 1 --2- 172.21.15.40:0/558543488 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3e48102250 0x7f3e48108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:46.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.414+0000 7f3e4ccdf640 1 --2- 172.21.15.40:0/558543488 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e481006a0 0x7f3e48100ac0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:46.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.414+0000 7f3e4ccdf640 1 -- 172.21.15.40:0/558543488 >> 172.21.15.40:0/558543488 conn(0x7f3e480fbc30 msgr2=0x7f3e480fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:46.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.414+0000 7f3e4ccdf640 1 -- 172.21.15.40:0/558543488 shutdown_connections 2024-04-21T23:17:46.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.414+0000 7f3e4ccdf640 1 -- 172.21.15.40:0/558543488 wait complete. 2024-04-21T23:17:46.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.415+0000 7f3e4ccdf640 1 Processor -- start 2024-04-21T23:17:46.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.415+0000 7f3e4ccdf640 1 -- start start 2024-04-21T23:17:46.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.415+0000 7f3e4ccdf640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3e481006a0 0x7f3e48197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:46.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.415+0000 7f3e4ccdf640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3e48101890 0x7f3e481980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:46.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.415+0000 7f3e4ccdf640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e48102250 0x7f3e4819f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:46.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.415+0000 7f3e4ccdf640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3e4810c370 con 0x7f3e48102250 2024-04-21T23:17:46.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.415+0000 7f3e4ccdf640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3e4810c1f0 con 0x7f3e48101890 2024-04-21T23:17:46.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.415+0000 7f3e4ccdf640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3e4810c4f0 con 0x7f3e481006a0 2024-04-21T23:17:46.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.415+0000 7f3e46d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e48102250 0x7f3e4819f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:46.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.415+0000 7f3e46d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e48102250 0x7f3e4819f0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38854/0 (socket says 172.21.15.40:38854) 2024-04-21T23:17:46.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.415+0000 7f3e46d9d640 1 -- 172.21.15.40:0/322859151 learned_addr learned my addr 172.21.15.40:0/322859151 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:17:46.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.415+0000 7f3e4659c640 1 --2- 172.21.15.40:0/322859151 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3e481006a0 0x7f3e48197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:46.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.415+0000 7f3e46d9d640 1 -- 172.21.15.40:0/322859151 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3e481006a0 msgr2=0x7f3e48197b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:46.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.415+0000 7f3e45d9b640 1 --2- 172.21.15.40:0/322859151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3e48101890 0x7f3e481980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:46.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.415+0000 7f3e46d9d640 1 --2- 172.21.15.40:0/322859151 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3e481006a0 0x7f3e48197b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:46.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.415+0000 7f3e46d9d640 1 -- 172.21.15.40:0/322859151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3e48101890 msgr2=0x7f3e481980b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:46.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.415+0000 7f3e46d9d640 1 --2- 172.21.15.40:0/322859151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3e48101890 0x7f3e481980b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:46.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.416+0000 7f3e46d9d640 1 -- 172.21.15.40:0/322859151 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e4819f740 con 0x7f3e48102250 2024-04-21T23:17:46.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.416+0000 7f3e4659c640 1 --2- 172.21.15.40:0/322859151 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3e481006a0 0x7f3e48197b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:17:46.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.416+0000 7f3e45d9b640 1 --2- 172.21.15.40:0/322859151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3e48101890 0x7f3e481980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:17:46.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.416+0000 7f3e46d9d640 1 --2- 172.21.15.40:0/322859151 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e48102250 0x7f3e4819f0d0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f3e4000eb10 tx=0x7f3e4000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:46.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.416+0000 7f3e377fe640 1 -- 172.21.15.40:0/322859151 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e40004b20 con 0x7f3e48102250 2024-04-21T23:17:46.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.416+0000 7f3e377fe640 1 -- 172.21.15.40:0/322859151 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e40004cc0 con 0x7f3e48102250 2024-04-21T23:17:46.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.416+0000 7f3e4ccdf640 1 -- 172.21.15.40:0/322859151 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e4819fa30 con 0x7f3e48102250 2024-04-21T23:17:46.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.416+0000 7f3e377fe640 1 -- 172.21.15.40:0/322859151 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e40010760 con 0x7f3e48102250 2024-04-21T23:17:46.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.416+0000 7f3e4ccdf640 1 -- 172.21.15.40:0/322859151 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e4819ff80 con 0x7f3e48102250 2024-04-21T23:17:46.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.417+0000 7f3e4ccdf640 1 -- 172.21.15.40:0/322859151 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e481a0270 con 0x7f3e48102250 2024-04-21T23:17:46.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.418+0000 7f3e377fe640 1 -- 172.21.15.40:0/322859151 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f3e40005020 con 0x7f3e48102250 2024-04-21T23:17:46.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.419+0000 7f3e377fe640 1 --2- 172.21.15.40:0/322859151 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f3e300a2c20 0x7f3e300a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:46.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.419+0000 7f3e4659c640 1 --2- 172.21.15.40:0/322859151 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f3e300a2c20 0x7f3e300a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:46.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.419+0000 7f3e377fe640 1 -- 172.21.15.40:0/322859151 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f3e400cfd20 con 0x7f3e48102250 2024-04-21T23:17:46.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.419+0000 7f3e4659c640 1 --2- 172.21.15.40:0/322859151 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f3e300a2c20 0x7f3e300a50e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3e380096c0 tx=0x7f3e3802df90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:46.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.423+0000 7f3e377fe640 1 -- 172.21.15.40:0/322859151 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f3e4009c7c0 con 0x7f3e48102250 2024-04-21T23:17:46.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.606+0000 7f3e4ccdf640 1 -- 172.21.15.40:0/322859151 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3e481a0440 con 0x7f3e48102250 2024-04-21T23:17:46.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.607+0000 7f3e377fe640 1 -- 172.21.15.40:0/322859151 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v92) ==== 93+0+10 (secure 0 0 0) 0x7f3e4009c0c0 con 0x7f3e48102250 2024-04-21T23:17:46.606 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-04-21T23:17:46.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.608+0000 7f3e4ccdf640 1 -- 172.21.15.40:0/322859151 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f3e300a2c20 msgr2=0x7f3e300a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:46.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.608+0000 7f3e4ccdf640 1 --2- 172.21.15.40:0/322859151 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f3e300a2c20 0x7f3e300a50e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3e380096c0 tx=0x7f3e3802df90 comp rx=0 tx=0).stop 2024-04-21T23:17:46.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.608+0000 7f3e4ccdf640 1 -- 172.21.15.40:0/322859151 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e48102250 msgr2=0x7f3e4819f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:46.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.608+0000 7f3e4ccdf640 1 --2- 172.21.15.40:0/322859151 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e48102250 0x7f3e4819f0d0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f3e4000eb10 tx=0x7f3e4000c2d0 comp rx=0 tx=0).stop 2024-04-21T23:17:46.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.609+0000 7f3e4ccdf640 1 -- 172.21.15.40:0/322859151 shutdown_connections 2024-04-21T23:17:46.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.609+0000 7f3e4ccdf640 1 --2- 172.21.15.40:0/322859151 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f3e300a2c20 0x7f3e300a50e0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:46.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.609+0000 7f3e4ccdf640 1 --2- 172.21.15.40:0/322859151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3e48101890 0x7f3e481980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:46.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.609+0000 7f3e4ccdf640 1 --2- 172.21.15.40:0/322859151 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e48102250 0x7f3e4819f0d0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:46.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.609+0000 7f3e4ccdf640 1 --2- 172.21.15.40:0/322859151 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3e481006a0 0x7f3e48197b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:46.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.609+0000 7f3e4ccdf640 1 -- 172.21.15.40:0/322859151 >> 172.21.15.40:0/322859151 conn(0x7f3e480fbc30 msgr2=0x7f3e480fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:46.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.609+0000 7f3e4ccdf640 1 -- 172.21.15.40:0/322859151 shutdown_connections 2024-04-21T23:17:46.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.609+0000 7f3e4ccdf640 1 -- 172.21.15.40:0/322859151 wait complete. 2024-04-21T23:17:46.619 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T23:17:46.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.725+0000 7f075253e640 1 -- 172.21.15.40:0/197071876 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f074c101870 msgr2=0x7f074c101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:46.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.725+0000 7f075253e640 1 --2- 172.21.15.40:0/197071876 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f074c101870 0x7f074c101cf0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f0740009900 tx=0x7f074002bc30 comp rx=0 tx=0).stop 2024-04-21T23:17:46.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.725+0000 7f075253e640 1 -- 172.21.15.40:0/197071876 shutdown_connections 2024-04-21T23:17:46.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.725+0000 7f075253e640 1 --2- 172.21.15.40:0/197071876 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f074c102230 0x7f074c108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:46.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.725+0000 7f075253e640 1 --2- 172.21.15.40:0/197071876 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f074c101870 0x7f074c101cf0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:46.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.725+0000 7f075253e640 1 --2- 172.21.15.40:0/197071876 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f074c100680 0x7f074c100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:46.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.725+0000 7f075253e640 1 -- 172.21.15.40:0/197071876 >> 172.21.15.40:0/197071876 conn(0x7f074c0fbbf0 msgr2=0x7f074c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:46.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.725+0000 7f075253e640 1 -- 172.21.15.40:0/197071876 shutdown_connections 2024-04-21T23:17:46.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.725+0000 7f075253e640 1 -- 172.21.15.40:0/197071876 wait complete. 2024-04-21T23:17:46.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.725+0000 7f075253e640 1 Processor -- start 2024-04-21T23:17:46.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.725+0000 7f075253e640 1 -- start start 2024-04-21T23:17:46.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.726+0000 7f075253e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f074c100680 0x7f074c197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:46.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.726+0000 7f075253e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f074c101870 0x7f074c198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:46.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.726+0000 7f075253e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f074c102230 0x7f074c19f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:46.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.726+0000 7f075253e640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f074c10c300 con 0x7f074c100680 2024-04-21T23:17:46.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.726+0000 7f075253e640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f074c10c180 con 0x7f074c101870 2024-04-21T23:17:46.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.726+0000 7f075253e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f074c10c480 con 0x7f074c102230 2024-04-21T23:17:46.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.726+0000 7f074bfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f074c100680 0x7f074c197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:46.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.726+0000 7f074bfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f074c100680 0x7f074c197ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38864/0 (socket says 172.21.15.40:38864) 2024-04-21T23:17:46.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.726+0000 7f074bfff640 1 -- 172.21.15.40:0/3971446728 learned_addr learned my addr 172.21.15.40:0/3971446728 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:17:46.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.726+0000 7f0750adb640 1 --2- 172.21.15.40:0/3971446728 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f074c102230 0x7f074c19f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:46.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.726+0000 7f074b7fe640 1 --2- 172.21.15.40:0/3971446728 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f074c101870 0x7f074c198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:46.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.726+0000 7f074bfff640 1 -- 172.21.15.40:0/3971446728 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f074c102230 msgr2=0x7f074c19f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:46.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.726+0000 7f074bfff640 1 --2- 172.21.15.40:0/3971446728 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f074c102230 0x7f074c19f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:46.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.726+0000 7f074bfff640 1 -- 172.21.15.40:0/3971446728 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f074c101870 msgr2=0x7f074c198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:46.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.726+0000 7f074bfff640 1 --2- 172.21.15.40:0/3971446728 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f074c101870 0x7f074c198010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:46.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.726+0000 7f074bfff640 1 -- 172.21.15.40:0/3971446728 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f074c19f6f0 con 0x7f074c100680 2024-04-21T23:17:46.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.726+0000 7f074bfff640 1 --2- 172.21.15.40:0/3971446728 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f074c100680 0x7f074c197ad0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f073c00e9c0 tx=0x7f073c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:46.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.726+0000 7f07497fa640 1 -- 172.21.15.40:0/3971446728 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f073c00cd80 con 0x7f074c100680 2024-04-21T23:17:46.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.726+0000 7f07497fa640 1 -- 172.21.15.40:0/3971446728 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f073c0035f0 con 0x7f074c100680 2024-04-21T23:17:46.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.726+0000 7f075253e640 1 -- 172.21.15.40:0/3971446728 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f074c19f9e0 con 0x7f074c100680 2024-04-21T23:17:46.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.726+0000 7f07497fa640 1 -- 172.21.15.40:0/3971446728 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f073c010690 con 0x7f074c100680 2024-04-21T23:17:46.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.727+0000 7f075253e640 1 -- 172.21.15.40:0/3971446728 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f074c19ff30 con 0x7f074c100680 2024-04-21T23:17:46.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.728+0000 7f075253e640 1 -- 172.21.15.40:0/3971446728 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f074c1a01c0 con 0x7f074c100680 2024-04-21T23:17:46.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.729+0000 7f07497fa640 1 -- 172.21.15.40:0/3971446728 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f073c0037a0 con 0x7f074c100680 2024-04-21T23:17:46.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.730+0000 7f07497fa640 1 --2- 172.21.15.40:0/3971446728 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f07340a2c20 0x7f07340a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:46.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.730+0000 7f074b7fe640 1 --2- 172.21.15.40:0/3971446728 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f07340a2c20 0x7f07340a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:46.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.730+0000 7f07497fa640 1 -- 172.21.15.40:0/3971446728 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f073c0cf9c0 con 0x7f074c100680 2024-04-21T23:17:46.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.730+0000 7f074b7fe640 1 --2- 172.21.15.40:0/3971446728 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f07340a2c20 0x7f07340a50e0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f0740036040 tx=0x7f0740005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:46.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.734+0000 7f07497fa640 1 -- 172.21.15.40:0/3971446728 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f073c09c460 con 0x7f074c100680 2024-04-21T23:17:46.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.918+0000 7f075253e640 1 -- 172.21.15.40:0/3971446728 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f074c1a0390 con 0x7f074c100680 2024-04-21T23:17:46.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.919+0000 7f07497fa640 1 -- 172.21.15.40:0/3971446728 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v92) ==== 104+0+11 (secure 0 0 0) 0x7f073c09bd60 con 0x7f074c100680 2024-04-21T23:17:46.918 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-21T23:17:46.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.920+0000 7f075253e640 1 -- 172.21.15.40:0/3971446728 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f07340a2c20 msgr2=0x7f07340a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:46.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.920+0000 7f075253e640 1 --2- 172.21.15.40:0/3971446728 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f07340a2c20 0x7f07340a50e0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f0740036040 tx=0x7f0740005fd0 comp rx=0 tx=0).stop 2024-04-21T23:17:46.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.920+0000 7f075253e640 1 -- 172.21.15.40:0/3971446728 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f074c100680 msgr2=0x7f074c197ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:46.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.920+0000 7f075253e640 1 --2- 172.21.15.40:0/3971446728 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f074c100680 0x7f074c197ad0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f073c00e9c0 tx=0x7f073c00ee80 comp rx=0 tx=0).stop 2024-04-21T23:17:46.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.921+0000 7f075253e640 1 -- 172.21.15.40:0/3971446728 shutdown_connections 2024-04-21T23:17:46.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.921+0000 7f075253e640 1 --2- 172.21.15.40:0/3971446728 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f07340a2c20 0x7f07340a50e0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:46.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.921+0000 7f075253e640 1 --2- 172.21.15.40:0/3971446728 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f074c101870 0x7f074c198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:46.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.921+0000 7f075253e640 1 --2- 172.21.15.40:0/3971446728 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f074c102230 0x7f074c19f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:46.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.921+0000 7f075253e640 1 --2- 172.21.15.40:0/3971446728 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f074c100680 0x7f074c197ad0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:46.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.921+0000 7f075253e640 1 -- 172.21.15.40:0/3971446728 >> 172.21.15.40:0/3971446728 conn(0x7f074c0fbbf0 msgr2=0x7f074c0fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:46.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.921+0000 7f075253e640 1 -- 172.21.15.40:0/3971446728 shutdown_connections 2024-04-21T23:17:46.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:46.921+0000 7f075253e640 1 -- 172.21.15.40:0/3971446728 wait complete. 2024-04-21T23:17:46.930 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:46 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/3653615762' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:17:46.931 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:46 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/421133995' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:17:46.931 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:46 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/322859151' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:17:46.931 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T23:17:47.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.037+0000 7faad5039640 1 -- 172.21.15.40:0/2290320175 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faad0107520 msgr2=0x7faad0109910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:47.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.037+0000 7faad5039640 1 --2- 172.21.15.40:0/2290320175 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faad0107520 0x7faad0109910 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7faac0009ca0 tx=0x7faac002c0f0 comp rx=0 tx=0).stop 2024-04-21T23:17:47.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.037+0000 7faad5039640 1 -- 172.21.15.40:0/2290320175 shutdown_connections 2024-04-21T23:17:47.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.037+0000 7faad5039640 1 --2- 172.21.15.40:0/2290320175 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faad0104bf0 0x7faad0106fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:47.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.037+0000 7faad5039640 1 --2- 172.21.15.40:0/2290320175 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faad0107520 0x7faad0109910 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:47.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.037+0000 7faad5039640 1 --2- 172.21.15.40:0/2290320175 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faad01022c0 0x7faad01046b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:47.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.037+0000 7faad5039640 1 -- 172.21.15.40:0/2290320175 >> 172.21.15.40:0/2290320175 conn(0x7faad00fbbf0 msgr2=0x7faad00fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:47.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.037+0000 7faad5039640 1 -- 172.21.15.40:0/2290320175 shutdown_connections 2024-04-21T23:17:47.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.037+0000 7faad5039640 1 -- 172.21.15.40:0/2290320175 wait complete. 2024-04-21T23:17:47.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.038+0000 7faad5039640 1 Processor -- start 2024-04-21T23:17:47.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.038+0000 7faad5039640 1 -- start start 2024-04-21T23:17:47.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.038+0000 7faad5039640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faad01022c0 0x7faad0197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:47.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.038+0000 7faad5039640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faad0104bf0 0x7faad0197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:47.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.038+0000 7faad5039640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faad0107520 0x7faad019f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:47.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.038+0000 7faad5039640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7faad010d480 con 0x7faad0107520 2024-04-21T23:17:47.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.038+0000 7faad5039640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7faad010d300 con 0x7faad0104bf0 2024-04-21T23:17:47.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.038+0000 7faad5039640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7faad010d600 con 0x7faad01022c0 2024-04-21T23:17:47.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.039+0000 7faacf59e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faad0107520 0x7faad019f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:47.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.039+0000 7faace59c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faad0104bf0 0x7faad0197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:47.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.039+0000 7faacf59e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faad0107520 0x7faad019f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38872/0 (socket says 172.21.15.40:38872) 2024-04-21T23:17:47.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.039+0000 7faacf59e640 1 -- 172.21.15.40:0/3063557335 learned_addr learned my addr 172.21.15.40:0/3063557335 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:17:47.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.039+0000 7faaced9d640 1 --2- 172.21.15.40:0/3063557335 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faad01022c0 0x7faad0197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:47.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.039+0000 7faacf59e640 1 -- 172.21.15.40:0/3063557335 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faad01022c0 msgr2=0x7faad0197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:47.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.039+0000 7faacf59e640 1 --2- 172.21.15.40:0/3063557335 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faad01022c0 0x7faad0197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:47.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.039+0000 7faacf59e640 1 -- 172.21.15.40:0/3063557335 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faad0104bf0 msgr2=0x7faad0197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:47.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.039+0000 7faacf59e640 1 --2- 172.21.15.40:0/3063557335 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faad0104bf0 0x7faad0197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:47.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.039+0000 7faacf59e640 1 -- 172.21.15.40:0/3063557335 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faad019f6b0 con 0x7faad0107520 2024-04-21T23:17:47.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.039+0000 7faacf59e640 1 --2- 172.21.15.40:0/3063557335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faad0107520 0x7faad019f040 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7faac0037040 tx=0x7faac0032f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:47.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.039+0000 7faab7fff640 1 -- 172.21.15.40:0/3063557335 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faac0039040 con 0x7faad0107520 2024-04-21T23:17:47.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.039+0000 7faab7fff640 1 -- 172.21.15.40:0/3063557335 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faac0044070 con 0x7faad0107520 2024-04-21T23:17:47.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.039+0000 7faad5039640 1 -- 172.21.15.40:0/3063557335 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faad019f940 con 0x7faad0107520 2024-04-21T23:17:47.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.039+0000 7faab7fff640 1 -- 172.21.15.40:0/3063557335 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faac0004260 con 0x7faad0107520 2024-04-21T23:17:47.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.039+0000 7faad5039640 1 -- 172.21.15.40:0/3063557335 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faad019fe30 con 0x7faad0107520 2024-04-21T23:17:47.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.040+0000 7faad5039640 1 -- 172.21.15.40:0/3063557335 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faad00fd6e0 con 0x7faad0107520 2024-04-21T23:17:47.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.041+0000 7faab7fff640 1 -- 172.21.15.40:0/3063557335 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7faac0003970 con 0x7faad0107520 2024-04-21T23:17:47.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.042+0000 7faab7fff640 1 --2- 172.21.15.40:0/3063557335 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7faab80a2c20 0x7faab80a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:47.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.042+0000 7faaced9d640 1 --2- 172.21.15.40:0/3063557335 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7faab80a2c20 0x7faab80a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:47.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.042+0000 7faab7fff640 1 -- 172.21.15.40:0/3063557335 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7faac00f1b70 con 0x7faad0107520 2024-04-21T23:17:47.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.043+0000 7faaced9d640 1 --2- 172.21.15.40:0/3063557335 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7faab80a2c20 0x7faab80a50e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7faac8005e10 tx=0x7faac8005da0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:47.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.047+0000 7faab7fff640 1 -- 172.21.15.40:0/3063557335 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7faac00be720 con 0x7faad0107520 2024-04-21T23:17:47.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:46 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/3653615762' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:17:47.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:46 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/421133995' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:17:47.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:46 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/322859151' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:17:47.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:46 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/3653615762' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:17:47.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:46 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/421133995' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:17:47.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:46 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/322859151' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:17:47.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.233+0000 7faad5039640 1 -- 172.21.15.40:0/3063557335 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7faad0062340 con 0x7faad0107520 2024-04-21T23:17:47.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.234+0000 7faab7fff640 1 -- 172.21.15.40:0/3063557335 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v92) ==== 100+0+11 (secure 0 0 0) 0x7faac00be020 con 0x7faad0107520 2024-04-21T23:17:47.233 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-04-21T23:17:47.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.235+0000 7faad5039640 1 -- 172.21.15.40:0/3063557335 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7faab80a2c20 msgr2=0x7faab80a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:47.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.235+0000 7faad5039640 1 --2- 172.21.15.40:0/3063557335 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7faab80a2c20 0x7faab80a50e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7faac8005e10 tx=0x7faac8005da0 comp rx=0 tx=0).stop 2024-04-21T23:17:47.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.235+0000 7faad5039640 1 -- 172.21.15.40:0/3063557335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faad0107520 msgr2=0x7faad019f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:47.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.235+0000 7faad5039640 1 --2- 172.21.15.40:0/3063557335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faad0107520 0x7faad019f040 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7faac0037040 tx=0x7faac0032f90 comp rx=0 tx=0).stop 2024-04-21T23:17:47.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.236+0000 7faad5039640 1 -- 172.21.15.40:0/3063557335 shutdown_connections 2024-04-21T23:17:47.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.236+0000 7faad5039640 1 --2- 172.21.15.40:0/3063557335 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7faab80a2c20 0x7faab80a50e0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:47.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.236+0000 7faad5039640 1 --2- 172.21.15.40:0/3063557335 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faad0104bf0 0x7faad0197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:47.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.236+0000 7faad5039640 1 --2- 172.21.15.40:0/3063557335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faad0107520 0x7faad019f040 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:47.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.236+0000 7faad5039640 1 --2- 172.21.15.40:0/3063557335 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faad01022c0 0x7faad0197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:47.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.236+0000 7faad5039640 1 -- 172.21.15.40:0/3063557335 >> 172.21.15.40:0/3063557335 conn(0x7faad00fbbf0 msgr2=0x7faad0102c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:47.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.236+0000 7faad5039640 1 -- 172.21.15.40:0/3063557335 shutdown_connections 2024-04-21T23:17:47.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.236+0000 7faad5039640 1 -- 172.21.15.40:0/3063557335 wait complete. 2024-04-21T23:17:47.246 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T23:17:47.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.351+0000 7f83c08ae640 1 -- 172.21.15.40:0/1083901572 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f83bc104bb0 msgr2=0x7f83bc106fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:47.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.351+0000 7f83c08ae640 1 --2- 172.21.15.40:0/1083901572 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f83bc104bb0 0x7f83bc106fa0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f83a4009900 tx=0x7f83a402bc30 comp rx=0 tx=0).stop 2024-04-21T23:17:47.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.351+0000 7f83c08ae640 1 -- 172.21.15.40:0/1083901572 shutdown_connections 2024-04-21T23:17:47.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.351+0000 7f83c08ae640 1 --2- 172.21.15.40:0/1083901572 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f83bc1074e0 0x7f83bc1098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:47.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.351+0000 7f83c08ae640 1 --2- 172.21.15.40:0/1083901572 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f83bc104bb0 0x7f83bc106fa0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:47.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.351+0000 7f83c08ae640 1 --2- 172.21.15.40:0/1083901572 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83bc102280 0x7f83bc104670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:47.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.351+0000 7f83c08ae640 1 -- 172.21.15.40:0/1083901572 >> 172.21.15.40:0/1083901572 conn(0x7f83bc0fbc30 msgr2=0x7f83bc0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:47.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.351+0000 7f83c08ae640 1 -- 172.21.15.40:0/1083901572 shutdown_connections 2024-04-21T23:17:47.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.351+0000 7f83c08ae640 1 -- 172.21.15.40:0/1083901572 wait complete. 2024-04-21T23:17:47.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.352+0000 7f83c08ae640 1 Processor -- start 2024-04-21T23:17:47.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.352+0000 7f83c08ae640 1 -- start start 2024-04-21T23:17:47.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.352+0000 7f83c08ae640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83bc102280 0x7f83bc197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:47.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.352+0000 7f83c08ae640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f83bc104bb0 0x7f83bc198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:47.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.352+0000 7f83c08ae640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f83bc1074e0 0x7f83bc19f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:47.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.352+0000 7f83c08ae640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f83bc10d3f0 con 0x7f83bc1074e0 2024-04-21T23:17:47.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.352+0000 7f83c08ae640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f83bc10d270 con 0x7f83bc102280 2024-04-21T23:17:47.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.352+0000 7f83c08ae640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f83bc10d570 con 0x7f83bc104bb0 2024-04-21T23:17:47.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.352+0000 7f83bad9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f83bc1074e0 0x7f83bc19f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:47.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.352+0000 7f83bad9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f83bc1074e0 0x7f83bc19f0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38886/0 (socket says 172.21.15.40:38886) 2024-04-21T23:17:47.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.352+0000 7f83bad9d640 1 -- 172.21.15.40:0/1585519627 learned_addr learned my addr 172.21.15.40:0/1585519627 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:17:47.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.352+0000 7f83b3fff640 1 --2- 172.21.15.40:0/1585519627 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f83bc104bb0 0x7f83bc198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:47.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.352+0000 7f83ba59c640 1 --2- 172.21.15.40:0/1585519627 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83bc102280 0x7f83bc197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:47.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.353+0000 7f83bad9d640 1 -- 172.21.15.40:0/1585519627 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f83bc104bb0 msgr2=0x7f83bc198070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:47.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.353+0000 7f83bad9d640 1 --2- 172.21.15.40:0/1585519627 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f83bc104bb0 0x7f83bc198070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:47.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.353+0000 7f83bad9d640 1 -- 172.21.15.40:0/1585519627 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83bc102280 msgr2=0x7f83bc197b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:47.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.353+0000 7f83bad9d640 1 --2- 172.21.15.40:0/1585519627 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83bc102280 0x7f83bc197b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:47.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.353+0000 7f83bad9d640 1 -- 172.21.15.40:0/1585519627 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83bc19f750 con 0x7f83bc1074e0 2024-04-21T23:17:47.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.353+0000 7f83b3fff640 1 --2- 172.21.15.40:0/1585519627 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f83bc104bb0 0x7f83bc198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:17:47.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.353+0000 7f83bad9d640 1 --2- 172.21.15.40:0/1585519627 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f83bc1074e0 0x7f83bc19f0e0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f83b4009fd0 tx=0x7f83b400b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:47.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.353+0000 7f83b37fe640 1 -- 172.21.15.40:0/1585519627 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83b4017070 con 0x7f83bc1074e0 2024-04-21T23:17:47.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.353+0000 7f83c08ae640 1 -- 172.21.15.40:0/1585519627 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83bc19fa40 con 0x7f83bc1074e0 2024-04-21T23:17:47.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.353+0000 7f83b37fe640 1 -- 172.21.15.40:0/1585519627 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83b4003e50 con 0x7f83bc1074e0 2024-04-21T23:17:47.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.353+0000 7f83b37fe640 1 -- 172.21.15.40:0/1585519627 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83b40106c0 con 0x7f83bc1074e0 2024-04-21T23:17:47.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.353+0000 7f83c08ae640 1 -- 172.21.15.40:0/1585519627 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83bc19ff10 con 0x7f83bc1074e0 2024-04-21T23:17:47.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.354+0000 7f83c08ae640 1 -- 172.21.15.40:0/1585519627 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83bc0fd720 con 0x7f83bc1074e0 2024-04-21T23:17:47.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.355+0000 7f83b37fe640 1 -- 172.21.15.40:0/1585519627 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f83b4003970 con 0x7f83bc1074e0 2024-04-21T23:17:47.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.356+0000 7f83b37fe640 1 --2- 172.21.15.40:0/1585519627 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f83a80a2c20 0x7f83a80a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:47.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.356+0000 7f83ba59c640 1 --2- 172.21.15.40:0/1585519627 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f83a80a2c20 0x7f83a80a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:47.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.356+0000 7f83b37fe640 1 -- 172.21.15.40:0/1585519627 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f83b40cf300 con 0x7f83bc1074e0 2024-04-21T23:17:47.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.356+0000 7f83ba59c640 1 --2- 172.21.15.40:0/1585519627 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f83a80a2c20 0x7f83a80a50e0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f83ac005fd0 tx=0x7f83ac005850 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:47.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.360+0000 7f83b37fe640 1 -- 172.21.15.40:0/1585519627 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f83b409be40 con 0x7f83bc1074e0 2024-04-21T23:17:47.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.546+0000 7f83c08ae640 1 -- 172.21.15.40:0/1585519627 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f83bc02d320 con 0x7f83bc1074e0 2024-04-21T23:17:47.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.547+0000 7f83b37fe640 1 -- 172.21.15.40:0/1585519627 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v92) ==== 103+0+11 (secure 0 0 0) 0x7f83b409b740 con 0x7f83bc1074e0 2024-04-21T23:17:47.546 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-21T23:17:47.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.548+0000 7f83c08ae640 1 -- 172.21.15.40:0/1585519627 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f83a80a2c20 msgr2=0x7f83a80a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:47.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.548+0000 7f83c08ae640 1 --2- 172.21.15.40:0/1585519627 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f83a80a2c20 0x7f83a80a50e0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f83ac005fd0 tx=0x7f83ac005850 comp rx=0 tx=0).stop 2024-04-21T23:17:47.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.548+0000 7f83c08ae640 1 -- 172.21.15.40:0/1585519627 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f83bc1074e0 msgr2=0x7f83bc19f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:47.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.548+0000 7f83c08ae640 1 --2- 172.21.15.40:0/1585519627 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f83bc1074e0 0x7f83bc19f0e0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f83b4009fd0 tx=0x7f83b400b260 comp rx=0 tx=0).stop 2024-04-21T23:17:47.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.549+0000 7f83c08ae640 1 -- 172.21.15.40:0/1585519627 shutdown_connections 2024-04-21T23:17:47.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.549+0000 7f83c08ae640 1 --2- 172.21.15.40:0/1585519627 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f83a80a2c20 0x7f83a80a50e0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:47.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.549+0000 7f83c08ae640 1 --2- 172.21.15.40:0/1585519627 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f83bc1074e0 0x7f83bc19f0e0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:47.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.549+0000 7f83c08ae640 1 --2- 172.21.15.40:0/1585519627 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f83bc104bb0 0x7f83bc198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:47.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.549+0000 7f83c08ae640 1 --2- 172.21.15.40:0/1585519627 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83bc102280 0x7f83bc197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:47.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.549+0000 7f83c08ae640 1 -- 172.21.15.40:0/1585519627 >> 172.21.15.40:0/1585519627 conn(0x7f83bc0fbc30 msgr2=0x7f83bc102be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:47.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.549+0000 7f83c08ae640 1 -- 172.21.15.40:0/1585519627 shutdown_connections 2024-04-21T23:17:47.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.549+0000 7f83c08ae640 1 -- 172.21.15.40:0/1585519627 wait complete. 2024-04-21T23:17:47.559 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:17:47.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.662+0000 7f28c5ecf640 1 -- 172.21.15.40:0/2939343607 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28c0101db0 msgr2=0x7f28c01021d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:47.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.662+0000 7f28c5ecf640 1 --2- 172.21.15.40:0/2939343607 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28c0101db0 0x7f28c01021d0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f28b0009900 tx=0x7f28b002bc30 comp rx=0 tx=0).stop 2024-04-21T23:17:47.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.662+0000 7f28c5ecf640 1 -- 172.21.15.40:0/2939343607 shutdown_connections 2024-04-21T23:17:47.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.662+0000 7f28c5ecf640 1 --2- 172.21.15.40:0/2939343607 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28c00ff8a0 0x7f28c00ffd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:47.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.662+0000 7f28c5ecf640 1 --2- 172.21.15.40:0/2939343607 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28c0100260 0x7f28c0100710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:47.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.662+0000 7f28c5ecf640 1 --2- 172.21.15.40:0/2939343607 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28c0101db0 0x7f28c01021d0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:47.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.662+0000 7f28c5ecf640 1 -- 172.21.15.40:0/2939343607 >> 172.21.15.40:0/2939343607 conn(0x7f28c00fb420 msgr2=0x7f28c00fd860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:47.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.662+0000 7f28c5ecf640 1 -- 172.21.15.40:0/2939343607 shutdown_connections 2024-04-21T23:17:47.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.662+0000 7f28c5ecf640 1 -- 172.21.15.40:0/2939343607 wait complete. 2024-04-21T23:17:47.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.662+0000 7f28c5ecf640 1 Processor -- start 2024-04-21T23:17:47.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.663+0000 7f28c5ecf640 1 -- start start 2024-04-21T23:17:47.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.663+0000 7f28c5ecf640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28c00ff8a0 0x7f28c0199d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:47.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.663+0000 7f28c5ecf640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28c0100260 0x7f28c019a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:47.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.663+0000 7f28c5ecf640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28c0101db0 0x7f28c01a1280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:47.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.663+0000 7f28c5ecf640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f28c010f6b0 con 0x7f28c0101db0 2024-04-21T23:17:47.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.663+0000 7f28c5ecf640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f28c010f530 con 0x7f28c0100260 2024-04-21T23:17:47.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.663+0000 7f28c5ecf640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f28c010f830 con 0x7f28c00ff8a0 2024-04-21T23:17:47.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.663+0000 7f28bffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28c0101db0 0x7f28c01a1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:47.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.663+0000 7f28bffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28c0101db0 0x7f28c01a1280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38908/0 (socket says 172.21.15.40:38908) 2024-04-21T23:17:47.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.663+0000 7f28bffff640 1 -- 172.21.15.40:0/1975489254 learned_addr learned my addr 172.21.15.40:0/1975489254 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:17:47.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.663+0000 7f28bf7fe640 1 --2- 172.21.15.40:0/1975489254 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28c00ff8a0 0x7f28c0199d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:47.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.663+0000 7f28beffd640 1 --2- 172.21.15.40:0/1975489254 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28c0100260 0x7f28c019a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:47.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.663+0000 7f28bffff640 1 -- 172.21.15.40:0/1975489254 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28c00ff8a0 msgr2=0x7f28c0199d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:47.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.663+0000 7f28bffff640 1 --2- 172.21.15.40:0/1975489254 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28c00ff8a0 0x7f28c0199d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:47.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.663+0000 7f28bffff640 1 -- 172.21.15.40:0/1975489254 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28c0100260 msgr2=0x7f28c019a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:47.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.663+0000 7f28bffff640 1 --2- 172.21.15.40:0/1975489254 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28c0100260 0x7f28c019a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:47.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.663+0000 7f28bffff640 1 -- 172.21.15.40:0/1975489254 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28c01a1920 con 0x7f28c0101db0 2024-04-21T23:17:47.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.663+0000 7f28bf7fe640 1 --2- 172.21.15.40:0/1975489254 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28c00ff8a0 0x7f28c0199d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:17:47.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.663+0000 7f28beffd640 1 --2- 172.21.15.40:0/1975489254 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28c0100260 0x7f28c019a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:17:47.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.663+0000 7f28bffff640 1 --2- 172.21.15.40:0/1975489254 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28c0101db0 0x7f28c01a1280 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f28b800cad0 tx=0x7f28b800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:47.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.663+0000 7f28bcff9640 1 -- 172.21.15.40:0/1975489254 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28b800ae20 con 0x7f28c0101db0 2024-04-21T23:17:47.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.664+0000 7f28c5ecf640 1 -- 172.21.15.40:0/1975489254 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28c01a1c10 con 0x7f28c0101db0 2024-04-21T23:17:47.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.664+0000 7f28bcff9640 1 -- 172.21.15.40:0/1975489254 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28b8005020 con 0x7f28c0101db0 2024-04-21T23:17:47.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.664+0000 7f28bcff9640 1 -- 172.21.15.40:0/1975489254 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28b800f6b0 con 0x7f28c0101db0 2024-04-21T23:17:47.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.664+0000 7f28c5ecf640 1 -- 172.21.15.40:0/1975489254 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28c01a2160 con 0x7f28c0101db0 2024-04-21T23:17:47.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.665+0000 7f28c5ecf640 1 -- 172.21.15.40:0/1975489254 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28c010b820 con 0x7f28c0101db0 2024-04-21T23:17:47.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.666+0000 7f28bcff9640 1 -- 172.21.15.40:0/1975489254 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f28b80051d0 con 0x7f28c0101db0 2024-04-21T23:17:47.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.666+0000 7f28bcff9640 1 --2- 172.21.15.40:0/1975489254 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f28a80a2ca0 0x7f28a80a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:47.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.667+0000 7f28bf7fe640 1 --2- 172.21.15.40:0/1975489254 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f28a80a2ca0 0x7f28a80a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:47.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.667+0000 7f28bcff9640 1 -- 172.21.15.40:0/1975489254 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f28b80cfa60 con 0x7f28c0101db0 2024-04-21T23:17:47.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.667+0000 7f28bf7fe640 1 --2- 172.21.15.40:0/1975489254 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f28a80a2ca0 0x7f28a80a5160 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f28b002c7b0 tx=0x7f28b002c8a0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:47.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.670+0000 7f28bcff9640 1 -- 172.21.15.40:0/1975489254 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f28b8014030 con 0x7f28c0101db0 2024-04-21T23:17:47.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.894+0000 7f28c5ecf640 1 -- 172.21.15.40:0/1975489254 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f28c002d170 con 0x7f28c0101db0 2024-04-21T23:17:47.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.894+0000 7f28bcff9640 1 -- 172.21.15.40:0/1975489254 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11882 (secure 0 0 0) 0x7f28b809c4a0 con 0x7f28c0101db0 2024-04-21T23:17:47.894 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:17:47.894 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/1571948830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1571948830},{"type":"v1","addr":"172.21.15.40:6835","nonce":1571948830}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/969356642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":969356642},{"type":"v1","addr":"172.21.15.40:6837","nonce":969356642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6839/3025057845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":3025057845},{"type":"v1","addr":"172.21.15.40:6839","nonce":3025057845}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6841/3350488804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":3350488804},{"type":"v1","addr":"172.21.15.40:6841","nonce":3350488804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/3438493203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":3438493203},{"type":"v1","addr":"172.21.15.174:6833","nonce":3438493203}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/3105752678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":3105752678},{"type":"v1","addr":"172.21.15.111:6833","nonce":3105752678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T23:15:55.742233+0000","modified":"2024-04-21T23:16:22.498043+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24521,"mds_1":24502,"mds_2":24494},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24461":{"gid":24461,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6835/2704462092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2704462092},{"type":"v1","addr":"172.21.15.111:6835","nonce":2704462092}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/3905733045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3905733045},{"type":"v1","addr":"172.21.15.174:6835","nonce":3905733045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6837/465579746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":465579746},{"type":"v1","addr":"172.21.15.174:6837","nonce":465579746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/2703764182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":2703764182},{"type":"v1","addr":"172.21.15.111:6837","nonce":2703764182}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24502":{"gid":24502,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.174:6839/2467509274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":2467509274},{"type":"v1","addr":"172.21.15.174:6839","nonce":2467509274}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3712235548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3712235548},{"type":"v1","addr":"172.21.15.111:6839","nonce":3712235548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24494,24502,24521]},"id":1}]} 2024-04-21T23:17:47.894 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 24 2024-04-21T23:17:47.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.897+0000 7f28c5ecf640 1 -- 172.21.15.40:0/1975489254 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f28a80a2ca0 msgr2=0x7f28a80a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:47.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.897+0000 7f28c5ecf640 1 --2- 172.21.15.40:0/1975489254 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f28a80a2ca0 0x7f28a80a5160 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f28b002c7b0 tx=0x7f28b002c8a0 comp rx=0 tx=0).stop 2024-04-21T23:17:47.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.897+0000 7f28c5ecf640 1 -- 172.21.15.40:0/1975489254 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28c0101db0 msgr2=0x7f28c01a1280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:47.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.897+0000 7f28c5ecf640 1 --2- 172.21.15.40:0/1975489254 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28c0101db0 0x7f28c01a1280 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f28b800cad0 tx=0x7f28b800cf90 comp rx=0 tx=0).stop 2024-04-21T23:17:47.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.897+0000 7f28c5ecf640 1 -- 172.21.15.40:0/1975489254 shutdown_connections 2024-04-21T23:17:47.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.897+0000 7f28c5ecf640 1 --2- 172.21.15.40:0/1975489254 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f28a80a2ca0 0x7f28a80a5160 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:47.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.897+0000 7f28c5ecf640 1 --2- 172.21.15.40:0/1975489254 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28c0100260 0x7f28c019a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:47.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.897+0000 7f28c5ecf640 1 --2- 172.21.15.40:0/1975489254 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28c0101db0 0x7f28c01a1280 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:47.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.897+0000 7f28c5ecf640 1 --2- 172.21.15.40:0/1975489254 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28c00ff8a0 0x7f28c0199d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:47.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.897+0000 7f28c5ecf640 1 -- 172.21.15.40:0/1975489254 >> 172.21.15.40:0/1975489254 conn(0x7f28c00fb420 msgr2=0x7f28c00fcb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:47.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.897+0000 7f28c5ecf640 1 -- 172.21.15.40:0/1975489254 shutdown_connections 2024-04-21T23:17:47.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:47.897+0000 7f28c5ecf640 1 -- 172.21.15.40:0/1975489254 wait complete. 2024-04-21T23:17:47.906 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:47 smithi040 ceph-mon[27984]: pgmap v350: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T23:17:47.906 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:47 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/3971446728' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T23:17:47.907 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:47 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/3063557335' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T23:17:47.907 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:47 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/1585519627' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T23:17:47.908 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T23:17:48.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.011+0000 7f29028b5640 1 -- 172.21.15.40:0/2309629146 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28fc1006a0 msgr2=0x7f28fc100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:48.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.011+0000 7f29028b5640 1 --2- 172.21.15.40:0/2309629146 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28fc1006a0 0x7f28fc100ac0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f28ec009900 tx=0x7f28ec02bc30 comp rx=0 tx=0).stop 2024-04-21T23:17:48.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.011+0000 7f29028b5640 1 -- 172.21.15.40:0/2309629146 shutdown_connections 2024-04-21T23:17:48.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.011+0000 7f29028b5640 1 --2- 172.21.15.40:0/2309629146 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28fc101890 0x7f28fc101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.011+0000 7f29028b5640 1 --2- 172.21.15.40:0/2309629146 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28fc102250 0x7f28fc108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.011+0000 7f29028b5640 1 --2- 172.21.15.40:0/2309629146 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28fc1006a0 0x7f28fc100ac0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.011+0000 7f29028b5640 1 -- 172.21.15.40:0/2309629146 >> 172.21.15.40:0/2309629146 conn(0x7f28fc0fbc30 msgr2=0x7f28fc0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:48.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.011+0000 7f29028b5640 1 -- 172.21.15.40:0/2309629146 shutdown_connections 2024-04-21T23:17:48.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.011+0000 7f29028b5640 1 -- 172.21.15.40:0/2309629146 wait complete. 2024-04-21T23:17:48.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.011+0000 7f29028b5640 1 Processor -- start 2024-04-21T23:17:48.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.011+0000 7f29028b5640 1 -- start start 2024-04-21T23:17:48.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.012+0000 7f29028b5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28fc1006a0 0x7f28fc197ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:48.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.012+0000 7f29028b5640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28fc101890 0x7f28fc1980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:48.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.012+0000 7f29028b5640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28fc102250 0x7f28fc19f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:48.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.012+0000 7f29028b5640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f28fc10c2f0 con 0x7f28fc1006a0 2024-04-21T23:17:48.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.012+0000 7f29028b5640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f28fc10c170 con 0x7f28fc101890 2024-04-21T23:17:48.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.012+0000 7f29028b5640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f28fc10c470 con 0x7f28fc102250 2024-04-21T23:17:48.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.012+0000 7f28fbfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28fc1006a0 0x7f28fc197ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:48.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.012+0000 7f28fbfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28fc1006a0 0x7f28fc197ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38916/0 (socket says 172.21.15.40:38916) 2024-04-21T23:17:48.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.012+0000 7f28fbfff640 1 -- 172.21.15.40:0/1080525295 learned_addr learned my addr 172.21.15.40:0/1080525295 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:17:48.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.012+0000 7f2900e52640 1 --2- 172.21.15.40:0/1080525295 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28fc102250 0x7f28fc19f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:48.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.012+0000 7f28fb7fe640 1 --2- 172.21.15.40:0/1080525295 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28fc101890 0x7f28fc1980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:48.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.012+0000 7f28fbfff640 1 -- 172.21.15.40:0/1080525295 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28fc102250 msgr2=0x7f28fc19f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:48.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.012+0000 7f28fbfff640 1 --2- 172.21.15.40:0/1080525295 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28fc102250 0x7f28fc19f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.012+0000 7f28fbfff640 1 -- 172.21.15.40:0/1080525295 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28fc101890 msgr2=0x7f28fc1980e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:48.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.012+0000 7f28fbfff640 1 --2- 172.21.15.40:0/1080525295 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28fc101890 0x7f28fc1980e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.012+0000 7f28fbfff640 1 -- 172.21.15.40:0/1080525295 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28fc19f770 con 0x7f28fc1006a0 2024-04-21T23:17:48.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.012+0000 7f2900e52640 1 --2- 172.21.15.40:0/1080525295 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28fc102250 0x7f28fc19f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:17:48.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.012+0000 7f28fb7fe640 1 --2- 172.21.15.40:0/1080525295 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28fc101890 0x7f28fc1980e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:17:48.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.012+0000 7f28fbfff640 1 --2- 172.21.15.40:0/1080525295 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28fc1006a0 0x7f28fc197ba0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f28ec005ab0 tx=0x7f28ec005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:48.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.013+0000 7f28f97fa640 1 -- 172.21.15.40:0/1080525295 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28ec041070 con 0x7f28fc1006a0 2024-04-21T23:17:48.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.013+0000 7f29028b5640 1 -- 172.21.15.40:0/1080525295 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28fc19fa00 con 0x7f28fc1006a0 2024-04-21T23:17:48.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.013+0000 7f28f97fa640 1 -- 172.21.15.40:0/1080525295 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28ec0054e0 con 0x7f28fc1006a0 2024-04-21T23:17:48.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.013+0000 7f28f97fa640 1 -- 172.21.15.40:0/1080525295 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28ec034440 con 0x7f28fc1006a0 2024-04-21T23:17:48.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.013+0000 7f29028b5640 1 -- 172.21.15.40:0/1080525295 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28fc19ff40 con 0x7f28fc1006a0 2024-04-21T23:17:48.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.014+0000 7f29028b5640 1 -- 172.21.15.40:0/1080525295 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28fc1a0170 con 0x7f28fc1006a0 2024-04-21T23:17:48.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.015+0000 7f28f97fa640 1 -- 172.21.15.40:0/1080525295 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f28ec044070 con 0x7f28fc1006a0 2024-04-21T23:17:48.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.016+0000 7f28f97fa640 1 --2- 172.21.15.40:0/1080525295 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f28e40a2c20 0x7f28e40a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:48.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.016+0000 7f28f97fa640 1 -- 172.21.15.40:0/1080525295 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f28ec0f1390 con 0x7f28fc1006a0 2024-04-21T23:17:48.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.016+0000 7f28fb7fe640 1 --2- 172.21.15.40:0/1080525295 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f28e40a2c20 0x7f28e40a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:48.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.017+0000 7f28fb7fe640 1 --2- 172.21.15.40:0/1080525295 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f28e40a2c20 0x7f28e40a50e0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f28fc199060 tx=0x7f28f0005e50 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:48.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.020+0000 7f28f97fa640 1 -- 172.21.15.40:0/1080525295 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f28ec0bdee0 con 0x7f28fc1006a0 2024-04-21T23:17:48.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:47 smithi174 ceph-mon[31409]: pgmap v350: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T23:17:48.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:47 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/3971446728' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T23:17:48.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:47 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/3063557335' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T23:17:48.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:47 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/1585519627' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T23:17:48.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:47 smithi111 ceph-mon[31085]: pgmap v350: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T23:17:48.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:47 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/3971446728' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T23:17:48.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:47 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/3063557335' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T23:17:48.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:47 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/1585519627' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T23:17:48.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.207+0000 7f29028b5640 1 -- 172.21.15.40:0/1080525295 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f28fc1a0340 con 0x7f28fc1006a0 2024-04-21T23:17:48.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.209+0000 7f28f97fa640 1 -- 172.21.15.40:0/1080525295 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+24666 (secure 0 0 0) 0x7f28ec0bd7e0 con 0x7f28fc1006a0 2024-04-21T23:17:48.208 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:17:48.208 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":92,"fsid":"d814e4dc-0033-11ef-bc93-c7b262605968","created":"2024-04-21T23:07:52.659207+0000","modified":"2024-04-21T23:16:52.188054+0000","last_up_change":"2024-04-21T23:15:05.308821+0000","last_in_change":"2024-04-21T23:14:45.447491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:11:14.824775+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T23:15:54.322571+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T23:15:55.164493+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T23:15:58.838266+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"0e8d692a-22fe-4a87-8d99-9151f96cbb02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6803","nonce":2009758784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6805","nonce":2009758784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6809","nonce":2009758784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6807","nonce":2009758784}]},"public_addr":"172.21.15.40:6803/2009758784","cluster_addr":"172.21.15.40:6805/2009758784","heartbeat_back_addr":"172.21.15.40:6809/2009758784","heartbeat_front_addr":"172.21.15.40:6807/2009758784","state":["exists","up"]},{"osd":1,"uuid":"fe5fe11b-0e84-4359-83a7-1ba45105755a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6801","nonce":1763903836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6803","nonce":1763903836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6807","nonce":1763903836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6805","nonce":1763903836}]},"public_addr":"172.21.15.111:6801/1763903836","cluster_addr":"172.21.15.111:6803/1763903836","heartbeat_back_addr":"172.21.15.111:6807/1763903836","heartbeat_front_addr":"172.21.15.111:6805/1763903836","state":["exists","up"]},{"osd":2,"uuid":"813ad735-5cc2-4da0-ae16-6db238395fbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6801","nonce":439695825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6803","nonce":439695825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6807","nonce":439695825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6805","nonce":439695825}]},"public_addr":"172.21.15.174:6801/439695825","cluster_addr":"172.21.15.174:6803/439695825","heartbeat_back_addr":"172.21.15.174:6807/439695825","heartbeat_front_addr":"172.21.15.174:6805/439695825","state":["exists","up"]},{"osd":3,"uuid":"09becb03-b7ce-4ffc-917d-b7900ec37e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6811","nonce":3454312436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6813","nonce":3454312436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6817","nonce":3454312436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6815","nonce":3454312436}]},"public_addr":"172.21.15.40:6811/3454312436","cluster_addr":"172.21.15.40:6813/3454312436","heartbeat_back_addr":"172.21.15.40:6817/3454312436","heartbeat_front_addr":"172.21.15.40:6815/3454312436","state":["exists","up"]},{"osd":4,"uuid":"46eabfd7-d88a-4efd-94eb-04354d3a9d7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6809","nonce":1810866070}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6811","nonce":1810866070}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6815","nonce":1810866070}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6813","nonce":1810866070}]},"public_addr":"172.21.15.111:6809/1810866070","cluster_addr":"172.21.15.111:6811/1810866070","heartbeat_back_addr":"172.21.15.111:6815/1810866070","heartbeat_front_addr":"172.21.15.111:6813/1810866070","state":["exists","up"]},{"osd":5,"uuid":"6f73b582-6c65-41db-aab0-9b360f83d5c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6809","nonce":3752986711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6811","nonce":3752986711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6815","nonce":3752986711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6813","nonce":3752986711}]},"public_addr":"172.21.15.174:6809/3752986711","cluster_addr":"172.21.15.174:6811/3752986711","heartbeat_back_addr":"172.21.15.174:6815/3752986711","heartbeat_front_addr":"172.21.15.174:6813/3752986711","state":["exists","up"]},{"osd":6,"uuid":"39f305f0-12d2-4732-aaa0-cd66700af538","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6819","nonce":1618855427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6821","nonce":1618855427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6825","nonce":1618855427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6823","nonce":1618855427}]},"public_addr":"172.21.15.40:6819/1618855427","cluster_addr":"172.21.15.40:6821/1618855427","heartbeat_back_addr":"172.21.15.40:6825/1618855427","heartbeat_front_addr":"172.21.15.40:6823/1618855427","state":["exists","up"]},{"osd":7,"uuid":"769f2468-5102-490b-89f8-d0444d3169a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6817","nonce":927054162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6819","nonce":927054162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6823","nonce":927054162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6821","nonce":927054162}]},"public_addr":"172.21.15.111:6817/927054162","cluster_addr":"172.21.15.111:6819/927054162","heartbeat_back_addr":"172.21.15.111:6823/927054162","heartbeat_front_addr":"172.21.15.111:6821/927054162","state":["exists","up"]},{"osd":8,"uuid":"1a444e80-007d-473b-b972-4c814a813c22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6817","nonce":3817848746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6819","nonce":3817848746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6823","nonce":3817848746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6821","nonce":3817848746}]},"public_addr":"172.21.15.174:6817/3817848746","cluster_addr":"172.21.15.174:6819/3817848746","heartbeat_back_addr":"172.21.15.174:6823/3817848746","heartbeat_front_addr":"172.21.15.174:6821/3817848746","state":["exists","up"]},{"osd":9,"uuid":"9a6b6a56-6ae5-49fe-9e5b-cb17d686e0dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6827","nonce":1649446531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6829","nonce":1649446531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6833","nonce":1649446531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6831","nonce":1649446531}]},"public_addr":"172.21.15.40:6827/1649446531","cluster_addr":"172.21.15.40:6829/1649446531","heartbeat_back_addr":"172.21.15.40:6833/1649446531","heartbeat_front_addr":"172.21.15.40:6831/1649446531","state":["exists","up"]},{"osd":10,"uuid":"a70038ba-e2ad-4170-85cc-ea0470a2edef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6825","nonce":1227538935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6827","nonce":1227538935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6831","nonce":1227538935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6829","nonce":1227538935}]},"public_addr":"172.21.15.111:6825/1227538935","cluster_addr":"172.21.15.111:6827/1227538935","heartbeat_back_addr":"172.21.15.111:6831/1227538935","heartbeat_front_addr":"172.21.15.111:6829/1227538935","state":["exists","up"]},{"osd":11,"uuid":"781532bb-7a5e-44e3-8eb7-b584b0054666","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6825","nonce":2045865553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6827","nonce":2045865553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6831","nonce":2045865553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6829","nonce":2045865553}]},"public_addr":"172.21.15.174:6825/2045865553","cluster_addr":"172.21.15.174:6827/2045865553","heartbeat_back_addr":"172.21.15.174:6831/2045865553","heartbeat_front_addr":"172.21.15.174:6829/2045865553","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:10:24.761290+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:10:48.565428+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:11:12.307818+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:11:37.951529+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:02.832523+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:27.568879+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:52.997707+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:13:18.937274+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:13:44.708657+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:14:10.833855+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:14:37.078272+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:15:03.878076+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":1}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.2a","mappings":[{"from":3,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:6801/526554319":"2024-04-22T23:08:50.305770+0000","172.21.15.40:6800/526554319":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/3178763471":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/4045911364":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/276300080":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/3689649814":"2024-04-22T23:08:23.743030+0000","172.21.15.40:0/2241116715":"2024-04-22T23:08:23.743030+0000","172.21.15.40:6801/1479040326":"2024-04-22T23:08:23.743030+0000","172.21.15.40:6800/1479040326":"2024-04-22T23:08:23.743030+0000","172.21.15.40:0/681298099":"2024-04-22T23:08:23.743030+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T23:17:48.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.211+0000 7f29028b5640 1 -- 172.21.15.40:0/1080525295 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f28e40a2c20 msgr2=0x7f28e40a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:48.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.211+0000 7f29028b5640 1 --2- 172.21.15.40:0/1080525295 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f28e40a2c20 0x7f28e40a50e0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f28fc199060 tx=0x7f28f0005e50 comp rx=0 tx=0).stop 2024-04-21T23:17:48.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.211+0000 7f29028b5640 1 -- 172.21.15.40:0/1080525295 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28fc1006a0 msgr2=0x7f28fc197ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:48.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.211+0000 7f29028b5640 1 --2- 172.21.15.40:0/1080525295 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28fc1006a0 0x7f28fc197ba0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f28ec005ab0 tx=0x7f28ec005dc0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.211+0000 7f29028b5640 1 -- 172.21.15.40:0/1080525295 shutdown_connections 2024-04-21T23:17:48.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.211+0000 7f29028b5640 1 --2- 172.21.15.40:0/1080525295 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f28e40a2c20 0x7f28e40a50e0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.211+0000 7f29028b5640 1 --2- 172.21.15.40:0/1080525295 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28fc101890 0x7f28fc1980e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.211+0000 7f29028b5640 1 --2- 172.21.15.40:0/1080525295 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28fc102250 0x7f28fc19f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.211+0000 7f29028b5640 1 --2- 172.21.15.40:0/1080525295 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28fc1006a0 0x7f28fc197ba0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.211+0000 7f29028b5640 1 -- 172.21.15.40:0/1080525295 >> 172.21.15.40:0/1080525295 conn(0x7f28fc0fbc30 msgr2=0x7f28fc069da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:48.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.211+0000 7f29028b5640 1 -- 172.21.15.40:0/1080525295 shutdown_connections 2024-04-21T23:17:48.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.211+0000 7f29028b5640 1 -- 172.21.15.40:0/1080525295 wait complete. 2024-04-21T23:17:48.221 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:17:48.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.324+0000 7f49a71be640 1 -- 172.21.15.40:0/3800862134 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f49a01006a0 msgr2=0x7f49a0100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:48.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.324+0000 7f49a71be640 1 --2- 172.21.15.40:0/3800862134 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f49a01006a0 0x7f49a0100ac0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f4994009900 tx=0x7f499402bc30 comp rx=0 tx=0).stop 2024-04-21T23:17:48.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.324+0000 7f49a71be640 1 -- 172.21.15.40:0/3800862134 shutdown_connections 2024-04-21T23:17:48.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.324+0000 7f49a71be640 1 --2- 172.21.15.40:0/3800862134 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49a0101890 0x7f49a0101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.324+0000 7f49a71be640 1 --2- 172.21.15.40:0/3800862134 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f49a0102250 0x7f49a0108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.324+0000 7f49a71be640 1 --2- 172.21.15.40:0/3800862134 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f49a01006a0 0x7f49a0100ac0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.324+0000 7f49a71be640 1 -- 172.21.15.40:0/3800862134 >> 172.21.15.40:0/3800862134 conn(0x7f49a00fbc30 msgr2=0x7f49a00fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:48.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.324+0000 7f49a71be640 1 -- 172.21.15.40:0/3800862134 shutdown_connections 2024-04-21T23:17:48.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.324+0000 7f49a71be640 1 -- 172.21.15.40:0/3800862134 wait complete. 2024-04-21T23:17:48.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.325+0000 7f49a71be640 1 Processor -- start 2024-04-21T23:17:48.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.325+0000 7f49a71be640 1 -- start start 2024-04-21T23:17:48.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.325+0000 7f49a71be640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49a01006a0 0x7f49a0197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:48.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.325+0000 7f49a71be640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f49a0101890 0x7f49a0198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:48.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.325+0000 7f49a71be640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f49a0102250 0x7f49a019f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:48.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.325+0000 7f49a71be640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f49a010c370 con 0x7f49a0102250 2024-04-21T23:17:48.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.325+0000 7f49a71be640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f49a010c1f0 con 0x7f49a01006a0 2024-04-21T23:17:48.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.325+0000 7f49a71be640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f49a010c4f0 con 0x7f49a0101890 2024-04-21T23:17:48.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.325+0000 7f49a575b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f49a0102250 0x7f49a019f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:48.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.325+0000 7f49a575b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f49a0102250 0x7f49a019f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38934/0 (socket says 172.21.15.40:38934) 2024-04-21T23:17:48.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.325+0000 7f49a575b640 1 -- 172.21.15.40:0/1769351160 learned_addr learned my addr 172.21.15.40:0/1769351160 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:17:48.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.325+0000 7f49a4f5a640 1 --2- 172.21.15.40:0/1769351160 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49a01006a0 0x7f49a0197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:48.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.325+0000 7f499ffff640 1 --2- 172.21.15.40:0/1769351160 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f49a0101890 0x7f49a0198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:48.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.325+0000 7f49a575b640 1 -- 172.21.15.40:0/1769351160 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f49a0101890 msgr2=0x7f49a0198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:48.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.325+0000 7f49a575b640 1 --2- 172.21.15.40:0/1769351160 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f49a0101890 0x7f49a0198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.325+0000 7f49a575b640 1 -- 172.21.15.40:0/1769351160 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49a01006a0 msgr2=0x7f49a0197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:48.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.325+0000 7f49a575b640 1 --2- 172.21.15.40:0/1769351160 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49a01006a0 0x7f49a0197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.325+0000 7f49a575b640 1 -- 172.21.15.40:0/1769351160 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49a019f6b0 con 0x7f49a0102250 2024-04-21T23:17:48.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.325+0000 7f49a575b640 1 --2- 172.21.15.40:0/1769351160 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f49a0102250 0x7f49a019f040 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f499800eb10 tx=0x7f499800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:48.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.326+0000 7f499dffb640 1 -- 172.21.15.40:0/1769351160 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4998004b20 con 0x7f49a0102250 2024-04-21T23:17:48.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.326+0000 7f499dffb640 1 -- 172.21.15.40:0/1769351160 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4998004cc0 con 0x7f49a0102250 2024-04-21T23:17:48.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.326+0000 7f49a71be640 1 -- 172.21.15.40:0/1769351160 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49a019f9a0 con 0x7f49a0102250 2024-04-21T23:17:48.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.326+0000 7f499dffb640 1 -- 172.21.15.40:0/1769351160 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4998010760 con 0x7f49a0102250 2024-04-21T23:17:48.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.326+0000 7f49a71be640 1 -- 172.21.15.40:0/1769351160 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49a019fef0 con 0x7f49a0102250 2024-04-21T23:17:48.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.327+0000 7f49a71be640 1 -- 172.21.15.40:0/1769351160 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49a01a0120 con 0x7f49a0102250 2024-04-21T23:17:48.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.328+0000 7f499dffb640 1 -- 172.21.15.40:0/1769351160 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f4998005020 con 0x7f49a0102250 2024-04-21T23:17:48.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.329+0000 7f499dffb640 1 --2- 172.21.15.40:0/1769351160 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f49900a2c20 0x7f49900a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:48.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.329+0000 7f49a4f5a640 1 --2- 172.21.15.40:0/1769351160 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f49900a2c20 0x7f49900a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:48.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.329+0000 7f499dffb640 1 -- 172.21.15.40:0/1769351160 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f49980cfd80 con 0x7f49a0102250 2024-04-21T23:17:48.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.329+0000 7f49a4f5a640 1 --2- 172.21.15.40:0/1769351160 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f49900a2c20 0x7f49900a50e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f499402c170 tx=0x7f499402df90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:48.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.333+0000 7f499dffb640 1 -- 172.21.15.40:0/1769351160 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f499809c7c0 con 0x7f49a0102250 2024-04-21T23:17:48.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.566+0000 7f49a71be640 1 -- 172.21.15.40:0/1769351160 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f49a01a02f0 con 0x7f49a0102250 2024-04-21T23:17:48.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.567+0000 7f499dffb640 1 -- 172.21.15.40:0/1769351160 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11882 (secure 0 0 0) 0x7f499809c0c0 con 0x7f49a0102250 2024-04-21T23:17:48.566 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:17:48.566 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/1571948830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1571948830},{"type":"v1","addr":"172.21.15.40:6835","nonce":1571948830}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/969356642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":969356642},{"type":"v1","addr":"172.21.15.40:6837","nonce":969356642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6839/3025057845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":3025057845},{"type":"v1","addr":"172.21.15.40:6839","nonce":3025057845}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6841/3350488804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":3350488804},{"type":"v1","addr":"172.21.15.40:6841","nonce":3350488804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/3438493203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":3438493203},{"type":"v1","addr":"172.21.15.174:6833","nonce":3438493203}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/3105752678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":3105752678},{"type":"v1","addr":"172.21.15.111:6833","nonce":3105752678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T23:15:55.742233+0000","modified":"2024-04-21T23:16:22.498043+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24521,"mds_1":24502,"mds_2":24494},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24461":{"gid":24461,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6835/2704462092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2704462092},{"type":"v1","addr":"172.21.15.111:6835","nonce":2704462092}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/3905733045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3905733045},{"type":"v1","addr":"172.21.15.174:6835","nonce":3905733045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6837/465579746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":465579746},{"type":"v1","addr":"172.21.15.174:6837","nonce":465579746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/2703764182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":2703764182},{"type":"v1","addr":"172.21.15.111:6837","nonce":2703764182}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24502":{"gid":24502,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.174:6839/2467509274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":2467509274},{"type":"v1","addr":"172.21.15.174:6839","nonce":2467509274}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3712235548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3712235548},{"type":"v1","addr":"172.21.15.111:6839","nonce":3712235548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24494,24502,24521]},"id":1}]} 2024-04-21T23:17:48.567 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 24 2024-04-21T23:17:48.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.569+0000 7f49a71be640 1 -- 172.21.15.40:0/1769351160 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f49900a2c20 msgr2=0x7f49900a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:48.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.569+0000 7f49a71be640 1 --2- 172.21.15.40:0/1769351160 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f49900a2c20 0x7f49900a50e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f499402c170 tx=0x7f499402df90 comp rx=0 tx=0).stop 2024-04-21T23:17:48.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.569+0000 7f49a71be640 1 -- 172.21.15.40:0/1769351160 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f49a0102250 msgr2=0x7f49a019f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:48.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.569+0000 7f49a71be640 1 --2- 172.21.15.40:0/1769351160 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f49a0102250 0x7f49a019f040 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f499800eb10 tx=0x7f499800c2d0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.569+0000 7f49a71be640 1 -- 172.21.15.40:0/1769351160 shutdown_connections 2024-04-21T23:17:48.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.569+0000 7f49a71be640 1 --2- 172.21.15.40:0/1769351160 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f49900a2c20 0x7f49900a50e0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.569+0000 7f49a71be640 1 --2- 172.21.15.40:0/1769351160 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f49a0102250 0x7f49a019f040 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.569+0000 7f49a71be640 1 --2- 172.21.15.40:0/1769351160 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f49a0101890 0x7f49a0198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.569+0000 7f49a71be640 1 --2- 172.21.15.40:0/1769351160 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49a01006a0 0x7f49a0197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.569+0000 7f49a71be640 1 -- 172.21.15.40:0/1769351160 >> 172.21.15.40:0/1769351160 conn(0x7f49a00fbc30 msgr2=0x7f49a00fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:48.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.569+0000 7f49a71be640 1 -- 172.21.15.40:0/1769351160 shutdown_connections 2024-04-21T23:17:48.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.569+0000 7f49a71be640 1 -- 172.21.15.40:0/1769351160 wait complete. 2024-04-21T23:17:48.580 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-21T23:15:55.742233+0000', 'modified': '2024-04-21T23:16:22.498043+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24521, 'mds_1': 24502, 'mds_2': 24494}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24461': {'gid': 24461, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.111:6835/2704462092', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6834', 'nonce': 2704462092}, {'type': 'v1', 'addr': '172.21.15.111:6835', 'nonce': 2704462092}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24466': {'gid': 24466, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.174:6835/3905733045', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6834', 'nonce': 3905733045}, {'type': 'v1', 'addr': '172.21.15.174:6835', 'nonce': 3905733045}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24484': {'gid': 24484, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.174:6837/465579746', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6836', 'nonce': 465579746}, {'type': 'v1', 'addr': '172.21.15.174:6837', 'nonce': 465579746}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24494': {'gid': 24494, 'name': 'h', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.111:6837/2703764182', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6836', 'nonce': 2703764182}, {'type': 'v1', 'addr': '172.21.15.111:6837', 'nonce': 2703764182}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24502': {'gid': 24502, 'name': 'i', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.174:6839/2467509274', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6838', 'nonce': 2467509274}, {'type': 'v1', 'addr': '172.21.15.174:6839', 'nonce': 2467509274}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24521': {'gid': 24521, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.111:6839/3712235548', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6838', 'nonce': 3712235548}, {'type': 'v1', 'addr': '172.21.15.111:6839', 'nonce': 3712235548}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24521, 'qdb_cluster': [24494, 24502, 24521]} 2024-04-21T23:17:48.580 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-21T23:17:48.580 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-04-21T23:17:48.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.683+0000 7f673472c640 1 -- 172.21.15.40:0/257735713 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f672c101870 msgr2=0x7f672c101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:48.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.683+0000 7f673472c640 1 --2- 172.21.15.40:0/257735713 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f672c101870 0x7f672c101cf0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f671c009900 tx=0x7f671c02bc30 comp rx=0 tx=0).stop 2024-04-21T23:17:48.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.683+0000 7f673472c640 1 -- 172.21.15.40:0/257735713 shutdown_connections 2024-04-21T23:17:48.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.683+0000 7f673472c640 1 --2- 172.21.15.40:0/257735713 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f672c102230 0x7f672c108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.683+0000 7f673472c640 1 --2- 172.21.15.40:0/257735713 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f672c101870 0x7f672c101cf0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.683+0000 7f673472c640 1 --2- 172.21.15.40:0/257735713 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f672c100680 0x7f672c100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.683+0000 7f673472c640 1 -- 172.21.15.40:0/257735713 >> 172.21.15.40:0/257735713 conn(0x7f672c0fbbf0 msgr2=0x7f672c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:48.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.683+0000 7f673472c640 1 -- 172.21.15.40:0/257735713 shutdown_connections 2024-04-21T23:17:48.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.683+0000 7f673472c640 1 -- 172.21.15.40:0/257735713 wait complete. 2024-04-21T23:17:48.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.684+0000 7f673472c640 1 Processor -- start 2024-04-21T23:17:48.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.684+0000 7f673472c640 1 -- start start 2024-04-21T23:17:48.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.684+0000 7f673472c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f672c100680 0x7f672c069440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:48.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.684+0000 7f673472c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f672c101870 0x7f672c19c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:48.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.684+0000 7f673472c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f672c102230 0x7f672c19ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:48.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.684+0000 7f673472c640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f672c10c300 con 0x7f672c101870 2024-04-21T23:17:48.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.684+0000 7f673472c640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f672c10c180 con 0x7f672c100680 2024-04-21T23:17:48.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.684+0000 7f673472c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f672c10c480 con 0x7f672c102230 2024-04-21T23:17:48.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.684+0000 7f6731cc7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f672c101870 0x7f672c19c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:48.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.684+0000 7f6731cc7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f672c101870 0x7f672c19c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38952/0 (socket says 172.21.15.40:38952) 2024-04-21T23:17:48.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.684+0000 7f6731cc7640 1 -- 172.21.15.40:0/4142280536 learned_addr learned my addr 172.21.15.40:0/4142280536 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:17:48.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.684+0000 7f67324c8640 1 --2- 172.21.15.40:0/4142280536 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f672c100680 0x7f672c069440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:48.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.684+0000 7f6732cc9640 1 --2- 172.21.15.40:0/4142280536 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f672c102230 0x7f672c19ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:48.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.685+0000 7f6731cc7640 1 -- 172.21.15.40:0/4142280536 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f672c102230 msgr2=0x7f672c19ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:48.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.685+0000 7f6731cc7640 1 --2- 172.21.15.40:0/4142280536 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f672c102230 0x7f672c19ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.685+0000 7f6731cc7640 1 -- 172.21.15.40:0/4142280536 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f672c100680 msgr2=0x7f672c069440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:48.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.685+0000 7f6731cc7640 1 --2- 172.21.15.40:0/4142280536 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f672c100680 0x7f672c069440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.685+0000 7f6731cc7640 1 -- 172.21.15.40:0/4142280536 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f672c19f720 con 0x7f672c101870 2024-04-21T23:17:48.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.685+0000 7f6731cc7640 1 --2- 172.21.15.40:0/4142280536 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f672c101870 0x7f672c19c7f0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f671c009230 tx=0x7f671c005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:48.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.685+0000 7f67237fe640 1 -- 172.21.15.40:0/4142280536 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f671c041070 con 0x7f672c101870 2024-04-21T23:17:48.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.685+0000 7f67237fe640 1 -- 172.21.15.40:0/4142280536 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f671c0044d0 con 0x7f672c101870 2024-04-21T23:17:48.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.685+0000 7f673472c640 1 -- 172.21.15.40:0/4142280536 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f672c19f950 con 0x7f672c101870 2024-04-21T23:17:48.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.685+0000 7f67237fe640 1 -- 172.21.15.40:0/4142280536 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f671c034440 con 0x7f672c101870 2024-04-21T23:17:48.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.685+0000 7f673472c640 1 -- 172.21.15.40:0/4142280536 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f672c19fe40 con 0x7f672c101870 2024-04-21T23:17:48.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.688+0000 7f67237fe640 1 -- 172.21.15.40:0/4142280536 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f671c044070 con 0x7f672c101870 2024-04-21T23:17:48.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.688+0000 7f67373f2740 1 Processor -- start 2024-04-21T23:17:48.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.688+0000 7f67373f2740 1 -- start start 2024-04-21T23:17:48.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.688+0000 7f67373f2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563098bc1070 0x563098bc1490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:48.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.688+0000 7f67373f2740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563098bc1a60 0x563098bc6560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:48.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.688+0000 7f67373f2740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563098bc6bd0 0x563098bc9060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:48.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.688+0000 7f67373f2740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x563098b25df0 con 0x563098bc1a60 2024-04-21T23:17:48.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.688+0000 7f67373f2740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x563098ace640 con 0x563098bc6bd0 2024-04-21T23:17:48.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.688+0000 7f67373f2740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x563098b21db0 con 0x563098bc1070 2024-04-21T23:17:48.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.688+0000 7f67324c8640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563098bc1a60 0x563098bc6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:48.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.688+0000 7f67324c8640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563098bc1a60 0x563098bc6560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38962/0 (socket says 172.21.15.40:38962) 2024-04-21T23:17:48.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.688+0000 7f67324c8640 1 -- 172.21.15.40:0/4288791617 learned_addr learned my addr 172.21.15.40:0/4288791617 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:17:48.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.689+0000 7f67237fe640 1 --2- 172.21.15.40:0/4142280536 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f67180a2c20 0x7f67180a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:48.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.689+0000 7f67324c8640 1 -- 172.21.15.40:0/4288791617 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563098bc1070 msgr2=0x563098bc1490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:17:48.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.689+0000 7f67324c8640 1 --2- 172.21.15.40:0/4288791617 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563098bc1070 0x563098bc1490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.689+0000 7f6731cc7640 1 --2- 172.21.15.40:0/4288791617 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563098bc6bd0 0x563098bc9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:48.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.689+0000 7f67324c8640 1 -- 172.21.15.40:0/4288791617 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563098bc6bd0 msgr2=0x563098bc9060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:17:48.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.689+0000 7f67324c8640 1 --2- 172.21.15.40:0/4288791617 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563098bc6bd0 0x563098bc9060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.689+0000 7f67324c8640 1 -- 172.21.15.40:0/4288791617 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56309883a2d0 con 0x563098bc1a60 2024-04-21T23:17:48.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.689+0000 7f6731cc7640 1 --2- 172.21.15.40:0/4288791617 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563098bc6bd0 0x563098bc9060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:17:48.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.689+0000 7f6732cc9640 1 --2- 172.21.15.40:0/4142280536 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f67180a2c20 0x7f67180a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:48.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.689+0000 7f67237fe640 1 -- 172.21.15.40:0/4142280536 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f671c0f1070 con 0x7f672c101870 2024-04-21T23:17:48.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.689+0000 7f67324c8640 1 --2- 172.21.15.40:0/4288791617 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563098bc1a60 0x563098bc6560 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f672800bcb0 tx=0x7f67280107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d12bea346f8df7b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:48.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.689+0000 7f6720ff9640 1 -- 172.21.15.40:0/4288791617 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6728004020 con 0x563098bc1a60 2024-04-21T23:17:48.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.689+0000 7f6720ff9640 1 -- 172.21.15.40:0/4288791617 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6728005500 con 0x563098bc1a60 2024-04-21T23:17:48.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.689+0000 7f6720ff9640 1 -- 172.21.15.40:0/4288791617 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6728004d80 con 0x563098bc1a60 2024-04-21T23:17:48.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.689+0000 7f6732cc9640 1 --2- 172.21.15.40:0/4142280536 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f67180a2c20 0x7f67180a50e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x563098bc22c0 tx=0x7f67240098e0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:48.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.690+0000 7f67373f2740 1 -- 172.21.15.40:0/4288791617 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563098bc1a60 msgr2=0x563098bc6560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:48.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.690+0000 7f67373f2740 1 --2- 172.21.15.40:0/4288791617 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563098bc1a60 0x563098bc6560 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f672800bcb0 tx=0x7f67280107a0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.690+0000 7f67373f2740 1 -- 172.21.15.40:0/4288791617 shutdown_connections 2024-04-21T23:17:48.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.690+0000 7f67373f2740 1 --2- 172.21.15.40:0/4288791617 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563098bc6bd0 0x563098bc9060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.690+0000 7f67373f2740 1 --2- 172.21.15.40:0/4288791617 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563098bc1a60 0x563098bc6560 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.690+0000 7f67373f2740 1 --2- 172.21.15.40:0/4288791617 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563098bc1070 0x563098bc1490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.690+0000 7f67373f2740 1 -- 172.21.15.40:0/4288791617 >> 172.21.15.40:0/4288791617 conn(0x563098bbc5e0 msgr2=0x563098bbc9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:48.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.690+0000 7f67373f2740 1 -- 172.21.15.40:0/4288791617 shutdown_connections 2024-04-21T23:17:48.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.690+0000 7f67373f2740 1 -- 172.21.15.40:0/4288791617 wait complete. 2024-04-21T23:17:48.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.690+0000 7f67373f2740 1 -- start start 2024-04-21T23:17:48.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.690+0000 7f67373f2740 1 Processor -- start 2024-04-21T23:17:48.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.691+0000 7f67373f2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563098bc1070 0x563098bcb560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:48.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.691+0000 7f67373f2740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563098bc1a60 0x563098bcbaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:48.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.691+0000 7f67373f2740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563098bc6bd0 0x563098bd2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:48.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.691+0000 7f67373f2740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x563098bc9b80 con 0x563098bc1a60 2024-04-21T23:17:48.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.691+0000 7f67373f2740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x563098ae1be0 con 0x563098bc6bd0 2024-04-21T23:17:48.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.691+0000 7f67373f2740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x563098bc9d00 con 0x563098bc1070 2024-04-21T23:17:48.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.691+0000 7f67373f2740 10 client.0 _ll_register_callbacks cb 0x563098ae15d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:17:48.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.691+0000 7f6731cc7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563098bc1a60 0x563098bcbaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:48.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.691+0000 7f6731cc7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563098bc1a60 0x563098bcbaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38966/0 (socket says 172.21.15.40:38966) 2024-04-21T23:17:48.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.691+0000 7f6731cc7640 1 -- 172.21.15.40:0/2646458515 learned_addr learned my addr 172.21.15.40:0/2646458515 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:17:48.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.691+0000 7f67324c8640 1 --2- 172.21.15.40:0/2646458515 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563098bc1070 0x563098bcb560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:48.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.691+0000 7f6731cc7640 1 -- 172.21.15.40:0/2646458515 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563098bc1070 msgr2=0x563098bcb560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:17:48.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.691+0000 7f6732cc9640 1 --2- 172.21.15.40:0/2646458515 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563098bc6bd0 0x563098bd2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:48.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.691+0000 7f6731cc7640 1 --2- 172.21.15.40:0/2646458515 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563098bc1070 0x563098bcb560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.691+0000 7f6731cc7640 1 -- 172.21.15.40:0/2646458515 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563098bc6bd0 msgr2=0x563098bd2ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:17:48.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.691+0000 7f6731cc7640 1 --2- 172.21.15.40:0/2646458515 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563098bc6bd0 0x563098bd2ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.691+0000 7f6731cc7640 1 -- 172.21.15.40:0/2646458515 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5630989046b0 con 0x563098bc1a60 2024-04-21T23:17:48.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.691+0000 7f6731cc7640 1 --2- 172.21.15.40:0/2646458515 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563098bc1a60 0x563098bcbaa0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f671c02c7b0 tx=0x7f671c0095e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7589176f20a3a96 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:48.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.691+0000 7f6715ffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-21T23:17:48.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.691+0000 7f6715ffb640 1 -- 172.21.15.40:0/2646458515 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f671c04b070 con 0x563098bc1a60 2024-04-21T23:17:48.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.691+0000 7f6715ffb640 1 -- 172.21.15.40:0/2646458515 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f671c0bd980 con 0x563098bc1a60 2024-04-21T23:17:48.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.691+0000 7f67373f2740 1 -- 172.21.15.40:0/2646458515 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6728005500 con 0x563098bc1a60 2024-04-21T23:17:48.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.692+0000 7f6715ffb640 1 -- 172.21.15.40:0/2646458515 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f671c0a6ab0 con 0x563098bc1a60 2024-04-21T23:17:48.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.692+0000 7f67373f2740 10 client.15219 fetch_fsmap learned FSMap version 24 2024-04-21T23:17:48.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.692+0000 7f67373f2740 1 -- 172.21.15.40:0/2646458515 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x563098bd5a30 con 0x563098bc1a60 2024-04-21T23:17:48.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.692+0000 7f6715ffb640 1 -- 172.21.15.40:0/2646458515 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6728005500 con 0x563098bc1a60 2024-04-21T23:17:48.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.692+0000 7f6715ffb640 1 -- 172.21.15.40:0/2646458515 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f671c0c5d00 con 0x563098bc1a60 2024-04-21T23:17:48.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.692+0000 7f67373f2740 10 client.15219 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T23:17:48.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.692+0000 7f67373f2740 10 client.15219 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-21T23:17:48.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.692+0000 7f67373f2740 20 client.15219 populate_metadata read hostname 'smithi040' 2024-04-21T23:17:48.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.692+0000 7f67373f2740 1 --2- 172.21.15.40:0/2646458515 >> [v2:172.21.15.111:6836/2703764182,v1:172.21.15.111:6837/2703764182] conn(0x563098bd8100 0x563098bd9570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:48.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.692+0000 7f67373f2740 4 client.15219 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:17:48.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.692+0000 7f67373f2740 1 -- 172.21.15.40:0/2646458515 --> [v2:172.21.15.111:6836/2703764182,v1:172.21.15.111:6837/2703764182] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563098b33330 con 0x563098bd8100 2024-04-21T23:17:48.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.692+0000 7f67324c8640 1 --2- 172.21.15.40:0/2646458515 >> [v2:172.21.15.111:6836/2703764182,v1:172.21.15.111:6837/2703764182] conn(0x563098bd8100 0x563098bd9570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:48.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.693+0000 7f6715ffb640 10 client.15219 ms_handle_connect on v2:172.21.15.111:6836/2703764182 2024-04-21T23:17:48.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.693+0000 7f67324c8640 1 --2- 172.21.15.40:0/2646458515 >> [v2:172.21.15.111:6836/2703764182,v1:172.21.15.111:6837/2703764182] conn(0x563098bd8100 0x563098bd9570 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=acbfc3a3108db481 server_cookie=cd0a6e831b945dad in_seq=0 out_seq=0 2024-04-21T23:17:48.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.694+0000 7f6715ffb640 1 -- 172.21.15.40:0/2646458515 <== mds.2 v2:172.21.15.111:6836/2703764182 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563098b33330 con 0x563098bd8100 2024-04-21T23:17:48.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.694+0000 7f6715ffb640 10 client.15219 handle_command_reply: tid=0 2024-04-21T23:17:48.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.694+0000 7f67373f2740 1 client.15219 shutdown 2024-04-21T23:17:48.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.694+0000 7f67373f2740 1 -- 172.21.15.40:0/2646458515 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563098bc1a60 msgr2=0x563098bcbaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:48.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.694+0000 7f67373f2740 1 --2- 172.21.15.40:0/2646458515 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563098bc1a60 0x563098bcbaa0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f671c02c7b0 tx=0x7f671c0095e0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.694+0000 7f67373f2740 1 -- 172.21.15.40:0/2646458515 shutdown_connections 2024-04-21T23:17:48.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.694+0000 7f67373f2740 1 --2- 172.21.15.40:0/2646458515 >> [v2:172.21.15.111:6836/2703764182,v1:172.21.15.111:6837/2703764182] conn(0x563098bd8100 0x563098bd9570 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.694+0000 7f67373f2740 1 --2- 172.21.15.40:0/2646458515 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563098bc6bd0 0x563098bd2ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.694+0000 7f67373f2740 1 --2- 172.21.15.40:0/2646458515 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563098bc1a60 0x563098bcbaa0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.694+0000 7f6715ffb640 0 client.15219 ms_handle_reset on v2:172.21.15.111:6836/2703764182 2024-04-21T23:17:48.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.694+0000 7f67373f2740 1 --2- 172.21.15.40:0/2646458515 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563098bc1070 0x563098bcb560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.694+0000 7f67373f2740 1 -- 172.21.15.40:0/2646458515 >> 172.21.15.40:0/2646458515 conn(0x563098bbc5e0 msgr2=0x563098bbb720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:48.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.694+0000 7f67373f2740 1 -- 172.21.15.40:0/2646458515 shutdown_connections 2024-04-21T23:17:48.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.694+0000 7f67373f2740 1 -- 172.21.15.40:0/2646458515 wait complete. 2024-04-21T23:17:48.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.694+0000 7f67373f2740 20 client.15219 trim_cache size 0 max 16384 2024-04-21T23:17:48.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.712+0000 7f67373f2740 1 Processor -- start 2024-04-21T23:17:48.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.712+0000 7f67373f2740 1 -- start start 2024-04-21T23:17:48.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.712+0000 7f67373f2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563098bc1070 0x563098bb7340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:48.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.712+0000 7f67373f2740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563098bc1a60 0x563098bc2ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:48.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.712+0000 7f67373f2740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563098bc6bd0 0x563098bc3400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:48.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.712+0000 7f67373f2740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x563098bd78b0 con 0x563098bc1a60 2024-04-21T23:17:48.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.712+0000 7f67373f2740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x563098bd7f80 con 0x563098bc6bd0 2024-04-21T23:17:48.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.712+0000 7f67373f2740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x563098bd8650 con 0x563098bc1070 2024-04-21T23:17:48.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.713+0000 7f6731cc7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563098bc1a60 0x563098bc2ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:48.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.713+0000 7f6731cc7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563098bc1a60 0x563098bc2ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38978/0 (socket says 172.21.15.40:38978) 2024-04-21T23:17:48.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.713+0000 7f6731cc7640 1 -- 172.21.15.40:0/782027840 learned_addr learned my addr 172.21.15.40:0/782027840 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:17:48.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.713+0000 7f6732cc9640 1 --2- 172.21.15.40:0/782027840 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563098bc6bd0 0x563098bc3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:48.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.713+0000 7f67324c8640 1 --2- 172.21.15.40:0/782027840 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563098bc1070 0x563098bb7340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:48.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.713+0000 7f6731cc7640 1 -- 172.21.15.40:0/782027840 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563098bc1070 msgr2=0x563098bb7340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:17:48.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.713+0000 7f6731cc7640 1 --2- 172.21.15.40:0/782027840 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563098bc1070 0x563098bb7340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.713+0000 7f6731cc7640 1 -- 172.21.15.40:0/782027840 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563098bc6bd0 msgr2=0x563098bc3400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:17:48.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.713+0000 7f6731cc7640 1 --2- 172.21.15.40:0/782027840 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563098bc6bd0 0x563098bc3400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.713+0000 7f6731cc7640 1 -- 172.21.15.40:0/782027840 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563098bbac20 con 0x563098bc1a60 2024-04-21T23:17:48.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.713+0000 7f6732cc9640 1 --2- 172.21.15.40:0/782027840 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563098bc6bd0 0x563098bc3400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:17:48.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.713+0000 7f6731cc7640 1 --2- 172.21.15.40:0/782027840 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563098bc1a60 0x563098bc2ec0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f671c0b9ef0 tx=0x7f671c02bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a57a2cde5a494919 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:48.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.713+0000 7f6717fff640 1 -- 172.21.15.40:0/782027840 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f671c0d5070 con 0x563098bc1a60 2024-04-21T23:17:48.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.713+0000 7f6717fff640 1 -- 172.21.15.40:0/782027840 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f671c0bdc20 con 0x563098bc1a60 2024-04-21T23:17:48.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.713+0000 7f6717fff640 1 -- 172.21.15.40:0/782027840 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f671c04b050 con 0x563098bc1a60 2024-04-21T23:17:48.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.714+0000 7f67373f2740 1 -- 172.21.15.40:0/782027840 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563098bc1a60 msgr2=0x563098bc2ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:48.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.714+0000 7f67373f2740 1 --2- 172.21.15.40:0/782027840 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563098bc1a60 0x563098bc2ec0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f671c0b9ef0 tx=0x7f671c02bc30 comp rx=0 tx=0).stop 2024-04-21T23:17:48.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.714+0000 7f67373f2740 1 -- 172.21.15.40:0/782027840 shutdown_connections 2024-04-21T23:17:48.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.714+0000 7f67373f2740 1 --2- 172.21.15.40:0/782027840 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563098bc6bd0 0x563098bc3400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.714+0000 7f67373f2740 1 --2- 172.21.15.40:0/782027840 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563098bc1a60 0x563098bc2ec0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.714+0000 7f67373f2740 1 --2- 172.21.15.40:0/782027840 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563098bc1070 0x563098bb7340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.714+0000 7f67373f2740 1 -- 172.21.15.40:0/782027840 >> 172.21.15.40:0/782027840 conn(0x563098bbc5e0 msgr2=0x563098bbc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:48.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.714+0000 7f67373f2740 1 -- 172.21.15.40:0/782027840 shutdown_connections 2024-04-21T23:17:48.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.714+0000 7f67373f2740 1 -- 172.21.15.40:0/782027840 wait complete. 2024-04-21T23:17:48.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.714+0000 7f67373f2740 1 -- start start 2024-04-21T23:17:48.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.714+0000 7f67373f2740 1 Processor -- start 2024-04-21T23:17:48.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.714+0000 7f67373f2740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563098bc1070 0x563098c3a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:48.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.714+0000 7f67373f2740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563098bc1a60 0x563098c3a820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:48.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.714+0000 7f67373f2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563098bc6bd0 0x563098c41200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:48.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.714+0000 7f67373f2740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x563098bd93f0 con 0x563098bc1070 2024-04-21T23:17:48.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.714+0000 7f67373f2740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x563098bd8d20 con 0x563098bc1a60 2024-04-21T23:17:48.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.714+0000 7f67373f2740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x563098bd9ac0 con 0x563098bc6bd0 2024-04-21T23:17:48.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.714+0000 7f67373f2740 10 client.0 _ll_register_callbacks cb 0x7f671c005b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:17:48.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.715+0000 7f67324c8640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563098bc1070 0x563098c3a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:48.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.715+0000 7f67324c8640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563098bc1070 0x563098c3a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38988/0 (socket says 172.21.15.40:38988) 2024-04-21T23:17:48.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.715+0000 7f67324c8640 1 -- 172.21.15.40:0/3368400071 learned_addr learned my addr 172.21.15.40:0/3368400071 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:17:48.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.715+0000 7f6731cc7640 1 --2- 172.21.15.40:0/3368400071 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563098bc1a60 0x563098c3a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:48.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.715+0000 7f67324c8640 1 -- 172.21.15.40:0/3368400071 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563098bc6bd0 msgr2=0x563098c41200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:17:48.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.715+0000 7f6732cc9640 1 --2- 172.21.15.40:0/3368400071 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563098bc6bd0 0x563098c41200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:48.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.715+0000 7f67324c8640 1 --2- 172.21.15.40:0/3368400071 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563098bc6bd0 0x563098c41200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.715+0000 7f67324c8640 1 -- 172.21.15.40:0/3368400071 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563098bc1a60 msgr2=0x563098c3a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:17:48.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.715+0000 7f67324c8640 1 --2- 172.21.15.40:0/3368400071 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563098bc1a60 0x563098c3a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.715+0000 7f67324c8640 1 -- 172.21.15.40:0/3368400071 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563098c41810 con 0x563098bc1070 2024-04-21T23:17:48.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.715+0000 7f6731cc7640 1 --2- 172.21.15.40:0/3368400071 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563098bc1a60 0x563098c3a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:17:48.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.715+0000 7f6732cc9640 1 --2- 172.21.15.40:0/3368400071 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563098bc6bd0 0x563098c41200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:17:48.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.715+0000 7f67324c8640 1 --2- 172.21.15.40:0/3368400071 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563098bc1070 0x563098c3a2e0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f672800c010 tx=0x7f6728005890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7cdf2b204bc072e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:48.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.715+0000 7f6715ffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-21T23:17:48.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.715+0000 7f6715ffb640 1 -- 172.21.15.40:0/3368400071 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6728004240 con 0x563098bc1070 2024-04-21T23:17:48.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.715+0000 7f6715ffb640 1 -- 172.21.15.40:0/3368400071 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f67280043e0 con 0x563098bc1070 2024-04-21T23:17:48.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.715+0000 7f67373f2740 1 -- 172.21.15.40:0/3368400071 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f671c0bdc20 con 0x563098bc1070 2024-04-21T23:17:48.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.715+0000 7f6715ffb640 1 -- 172.21.15.40:0/3368400071 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f6728037050 con 0x563098bc1070 2024-04-21T23:17:48.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.715+0000 7f67373f2740 10 client.15225 fetch_fsmap learned FSMap version 24 2024-04-21T23:17:48.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.715+0000 7f67373f2740 1 -- 172.21.15.40:0/3368400071 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x563098c42b80 con 0x563098bc1070 2024-04-21T23:17:48.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.715+0000 7f6715ffb640 1 -- 172.21.15.40:0/3368400071 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f671c0bdc20 con 0x563098bc1070 2024-04-21T23:17:48.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.716+0000 7f6715ffb640 1 -- 172.21.15.40:0/3368400071 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f6728032030 con 0x563098bc1070 2024-04-21T23:17:48.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.716+0000 7f67373f2740 10 client.15225 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T23:17:48.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.716+0000 7f67373f2740 10 client.15225 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-21T23:17:48.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.716+0000 7f67373f2740 20 client.15225 populate_metadata read hostname 'smithi040' 2024-04-21T23:17:48.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.716+0000 7f67373f2740 1 --2- 172.21.15.40:0/3368400071 >> [v2:172.21.15.111:6836/2703764182,v1:172.21.15.111:6837/2703764182] conn(0x563098c47490 0x563098c461a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:48.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.716+0000 7f67373f2740 4 client.15225 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-21T23:17:48.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.716+0000 7f67373f2740 1 -- 172.21.15.40:0/3368400071 --> [v2:172.21.15.111:6836/2703764182,v1:172.21.15.111:6837/2703764182] -- command(tid 0: {"prefix": "status"}) -- 0x563098b2d680 con 0x563098c47490 2024-04-21T23:17:48.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.716+0000 7f6731cc7640 1 --2- 172.21.15.40:0/3368400071 >> [v2:172.21.15.111:6836/2703764182,v1:172.21.15.111:6837/2703764182] conn(0x563098c47490 0x563098c461a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:48.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.716+0000 7f6731cc7640 1 --2- 172.21.15.40:0/3368400071 >> [v2:172.21.15.111:6836/2703764182,v1:172.21.15.111:6837/2703764182] conn(0x563098c47490 0x563098c461a0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ed4ef956026eda6a server_cookie=ae411df5b12a0965 in_seq=0 out_seq=0 2024-04-21T23:17:48.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.716+0000 7f6715ffb640 10 client.15225 ms_handle_connect on v2:172.21.15.111:6836/2703764182 2024-04-21T23:17:48.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.716+0000 7f6715ffb640 1 -- 172.21.15.40:0/3368400071 <== mds.2 v2:172.21.15.111:6836/2703764182 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x563098b2d680 con 0x563098c47490 2024-04-21T23:17:48.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.716+0000 7f6715ffb640 10 client.15225 handle_command_reply: tid=0 2024-04-21T23:17:48.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.717+0000 7f67373f2740 1 client.15225 shutdown 2024-04-21T23:17:48.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.717+0000 7f67373f2740 1 -- 172.21.15.40:0/3368400071 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563098bc1070 msgr2=0x563098c3a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:48.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.717+0000 7f67373f2740 1 --2- 172.21.15.40:0/3368400071 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563098bc1070 0x563098c3a2e0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f672800c010 tx=0x7f6728005890 comp rx=0 tx=0).stop 2024-04-21T23:17:48.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.717+0000 7f67373f2740 1 -- 172.21.15.40:0/3368400071 shutdown_connections 2024-04-21T23:17:48.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.717+0000 7f67373f2740 1 --2- 172.21.15.40:0/3368400071 >> [v2:172.21.15.111:6836/2703764182,v1:172.21.15.111:6837/2703764182] conn(0x563098c47490 0x563098c461a0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.717+0000 7f67373f2740 1 --2- 172.21.15.40:0/3368400071 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563098bc1a60 0x563098c3a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.717+0000 7f67373f2740 1 --2- 172.21.15.40:0/3368400071 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563098bc6bd0 0x563098c41200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.717+0000 7f67373f2740 1 --2- 172.21.15.40:0/3368400071 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563098bc1070 0x563098c3a2e0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.717+0000 7f67373f2740 1 -- 172.21.15.40:0/3368400071 >> 172.21.15.40:0/3368400071 conn(0x563098bbc5e0 msgr2=0x563098bbc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:48.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.717+0000 7f6715ffb640 0 client.15225 ms_handle_reset on v2:172.21.15.111:6836/2703764182 2024-04-21T23:17:48.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.717+0000 7f67373f2740 1 -- 172.21.15.40:0/3368400071 shutdown_connections 2024-04-21T23:17:48.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.717+0000 7f67373f2740 1 -- 172.21.15.40:0/3368400071 wait complete. 2024-04-21T23:17:48.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.717+0000 7f67373f2740 20 client.15225 trim_cache size 0 max 16384 2024-04-21T23:17:48.724 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-21T23:17:48.724 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "d814e4dc-0033-11ef-bc93-c7b262605968", 2024-04-21T23:17:48.724 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 2, 2024-04-21T23:17:48.724 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 24494, 2024-04-21T23:17:48.724 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-04-21T23:17:48.724 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-04-21T23:17:48.724 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-04-21T23:17:48.724 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 87.232645789000003, 2024-04-21T23:17:48.725 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 24, 2024-04-21T23:17:48.725 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 92, 2024-04-21T23:17:48.725 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 90, 2024-04-21T23:17:48.725 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 129.956037799 2024-04-21T23:17:48.725 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-21T23:17:48.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.717+0000 7f673472c640 1 -- 172.21.15.40:0/4142280536 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f67180a2c20 msgr2=0x7f67180a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:48.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.717+0000 7f673472c640 1 --2- 172.21.15.40:0/4142280536 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f67180a2c20 0x7f67180a50e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x563098bc22c0 tx=0x7f67240098e0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.717+0000 7f673472c640 1 -- 172.21.15.40:0/4142280536 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f672c101870 msgr2=0x7f672c19c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:48.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.717+0000 7f673472c640 1 --2- 172.21.15.40:0/4142280536 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f672c101870 0x7f672c19c7f0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f671c009230 tx=0x7f671c005d50 comp rx=0 tx=0).stop 2024-04-21T23:17:48.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.717+0000 7f673472c640 1 -- 172.21.15.40:0/4142280536 shutdown_connections 2024-04-21T23:17:48.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.717+0000 7f673472c640 1 --2- 172.21.15.40:0/4142280536 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f67180a2c20 0x7f67180a50e0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.717+0000 7f673472c640 1 --2- 172.21.15.40:0/4142280536 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f672c102230 0x7f672c19ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.717+0000 7f673472c640 1 --2- 172.21.15.40:0/4142280536 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f672c101870 0x7f672c19c7f0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.717+0000 7f673472c640 1 --2- 172.21.15.40:0/4142280536 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f672c100680 0x7f672c069440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.717+0000 7f673472c640 1 -- 172.21.15.40:0/4142280536 >> 172.21.15.40:0/4142280536 conn(0x7f672c0fbbf0 msgr2=0x7f672c104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:48.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.717+0000 7f673472c640 1 -- 172.21.15.40:0/4142280536 shutdown_connections 2024-04-21T23:17:48.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.717+0000 7f673472c640 1 -- 172.21.15.40:0/4142280536 wait complete. 2024-04-21T23:17:48.728 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-21T23:17:48.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.831+0000 7ff907878640 1 -- 172.21.15.40:0/2945874490 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff900104c10 msgr2=0x7ff900107000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:48.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.831+0000 7ff907878640 1 --2- 172.21.15.40:0/2945874490 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff900104c10 0x7ff900107000 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7ff8f0009900 tx=0x7ff8f002bc30 comp rx=0 tx=0).stop 2024-04-21T23:17:48.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.831+0000 7ff907878640 1 -- 172.21.15.40:0/2945874490 shutdown_connections 2024-04-21T23:17:48.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.831+0000 7ff907878640 1 --2- 172.21.15.40:0/2945874490 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff900107540 0x7ff900109930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.831+0000 7ff907878640 1 --2- 172.21.15.40:0/2945874490 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff900104c10 0x7ff900107000 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.831+0000 7ff907878640 1 --2- 172.21.15.40:0/2945874490 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff9001022e0 0x7ff9001046d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.831+0000 7ff907878640 1 -- 172.21.15.40:0/2945874490 >> 172.21.15.40:0/2945874490 conn(0x7ff9000fbc30 msgr2=0x7ff9000fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:48.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.831+0000 7ff907878640 1 -- 172.21.15.40:0/2945874490 shutdown_connections 2024-04-21T23:17:48.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.831+0000 7ff907878640 1 -- 172.21.15.40:0/2945874490 wait complete. 2024-04-21T23:17:48.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.832+0000 7ff907878640 1 Processor -- start 2024-04-21T23:17:48.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.832+0000 7ff907878640 1 -- start start 2024-04-21T23:17:48.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.832+0000 7ff907878640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff9001022e0 0x7ff900197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:48.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.832+0000 7ff907878640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff900104c10 0x7ff900197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:48.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.832+0000 7ff907878640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff900107540 0x7ff90019f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:48.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.832+0000 7ff907878640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff90010d430 con 0x7ff900104c10 2024-04-21T23:17:48.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.832+0000 7ff907878640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff90010d2b0 con 0x7ff9001022e0 2024-04-21T23:17:48.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.832+0000 7ff907878640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff90010d5b0 con 0x7ff900107540 2024-04-21T23:17:48.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.832+0000 7ff904e13640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff900104c10 0x7ff900197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:48.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.832+0000 7ff904e13640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff900104c10 0x7ff900197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39006/0 (socket says 172.21.15.40:39006) 2024-04-21T23:17:48.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.832+0000 7ff904e13640 1 -- 172.21.15.40:0/1779188821 learned_addr learned my addr 172.21.15.40:0/1779188821 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:17:48.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.832+0000 7ff905614640 1 --2- 172.21.15.40:0/1779188821 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff9001022e0 0x7ff900197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:48.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.832+0000 7ff905e15640 1 --2- 172.21.15.40:0/1779188821 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff900107540 0x7ff90019f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:48.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.832+0000 7ff904e13640 1 -- 172.21.15.40:0/1779188821 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff900107540 msgr2=0x7ff90019f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:48.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.832+0000 7ff904e13640 1 --2- 172.21.15.40:0/1779188821 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff900107540 0x7ff90019f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.832+0000 7ff904e13640 1 -- 172.21.15.40:0/1779188821 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff9001022e0 msgr2=0x7ff900197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:48.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.832+0000 7ff904e13640 1 --2- 172.21.15.40:0/1779188821 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff9001022e0 0x7ff900197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.832+0000 7ff904e13640 1 -- 172.21.15.40:0/1779188821 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff90019f6d0 con 0x7ff900104c10 2024-04-21T23:17:48.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.833+0000 7ff904e13640 1 --2- 172.21.15.40:0/1779188821 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff900104c10 0x7ff900197ff0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7ff8f0005ab0 tx=0x7ff8f0005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:48.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.833+0000 7ff8f67fc640 1 -- 172.21.15.40:0/1779188821 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8f0041070 con 0x7ff900104c10 2024-04-21T23:17:48.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.833+0000 7ff907878640 1 -- 172.21.15.40:0/1779188821 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff90019f960 con 0x7ff900104c10 2024-04-21T23:17:48.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.833+0000 7ff8f67fc640 1 -- 172.21.15.40:0/1779188821 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8f00054e0 con 0x7ff900104c10 2024-04-21T23:17:48.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.833+0000 7ff907878640 1 -- 172.21.15.40:0/1779188821 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff90019fea0 con 0x7ff900104c10 2024-04-21T23:17:48.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.833+0000 7ff8f67fc640 1 -- 172.21.15.40:0/1779188821 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8f004b060 con 0x7ff900104c10 2024-04-21T23:17:48.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.835+0000 7ff8f67fc640 1 -- 172.21.15.40:0/1779188821 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7ff8f0044070 con 0x7ff900104c10 2024-04-21T23:17:48.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.835+0000 7ff90a45c740 1 Processor -- start 2024-04-21T23:17:48.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.835+0000 7ff90a45c740 1 -- start start 2024-04-21T23:17:48.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.836+0000 7ff90a45c740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cfe719e070 0x55cfe719e490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:48.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.836+0000 7ff90a45c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cfe719ea60 0x55cfe71a3560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:48.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.836+0000 7ff90a45c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cfe71a3bd0 0x55cfe71a6060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:48.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.836+0000 7ff90a45c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55cfe7102df0 con 0x55cfe719ea60 2024-04-21T23:17:48.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.836+0000 7ff90a45c740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55cfe70ab640 con 0x55cfe719e070 2024-04-21T23:17:48.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.836+0000 7ff90a45c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55cfe70fedb0 con 0x55cfe71a3bd0 2024-04-21T23:17:48.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.836+0000 7ff905614640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cfe719ea60 0x55cfe71a3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:48.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.836+0000 7ff8f67fc640 1 --2- 172.21.15.40:0/1779188821 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7ff8ec0a2c20 0x7ff8ec0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:48.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.836+0000 7ff905614640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cfe719ea60 0x55cfe71a3560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39014/0 (socket says 172.21.15.40:39014) 2024-04-21T23:17:48.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.836+0000 7ff905614640 1 -- 172.21.15.40:0/1816261415 learned_addr learned my addr 172.21.15.40:0/1816261415 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:17:48.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.836+0000 7ff905e15640 1 --2- 172.21.15.40:0/1816261415 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cfe719e070 0x55cfe719e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:48.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.836+0000 7ff904e13640 1 --2- 172.21.15.40:0/1816261415 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cfe71a3bd0 0x55cfe71a6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:48.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.836+0000 7ff905e15640 1 --2- 172.21.15.40:0/1779188821 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7ff8ec0a2c20 0x7ff8ec0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:48.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.836+0000 7ff8f67fc640 1 -- 172.21.15.40:0/1779188821 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7ff8f00f14b0 con 0x7ff900104c10 2024-04-21T23:17:48.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.836+0000 7ff905614640 1 -- 172.21.15.40:0/1816261415 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cfe71a3bd0 msgr2=0x55cfe71a6060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:17:48.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.836+0000 7ff905614640 1 --2- 172.21.15.40:0/1816261415 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cfe71a3bd0 0x55cfe71a6060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.836+0000 7ff905614640 1 -- 172.21.15.40:0/1816261415 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cfe719e070 msgr2=0x55cfe719e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:17:48.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.836+0000 7ff905614640 1 --2- 172.21.15.40:0/1816261415 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cfe719e070 0x55cfe719e490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.836+0000 7ff905614640 1 -- 172.21.15.40:0/1816261415 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cfe6e172d0 con 0x55cfe719ea60 2024-04-21T23:17:48.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.836+0000 7ff905614640 1 --2- 172.21.15.40:0/1816261415 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cfe719ea60 0x55cfe71a3560 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7ff8fc00cc60 tx=0x7ff8fc00e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d361b419996a852f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:48.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.837+0000 7ff905e15640 1 --2- 172.21.15.40:0/1779188821 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7ff8ec0a2c20 0x7ff8ec0a50e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7ff8f8009fd0 tx=0x7ff8f8004a40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:48.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.837+0000 7ff8eb7fe640 1 -- 172.21.15.40:0/1816261415 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8fc014070 con 0x55cfe719ea60 2024-04-21T23:17:48.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.837+0000 7ff8eb7fe640 1 -- 172.21.15.40:0/1816261415 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8fc0047a0 con 0x55cfe719ea60 2024-04-21T23:17:48.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.837+0000 7ff8eb7fe640 1 -- 172.21.15.40:0/1816261415 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8fc010760 con 0x55cfe719ea60 2024-04-21T23:17:48.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.837+0000 7ff90a45c740 1 -- 172.21.15.40:0/1816261415 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cfe719ea60 msgr2=0x55cfe71a3560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:48.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.837+0000 7ff90a45c740 1 --2- 172.21.15.40:0/1816261415 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cfe719ea60 0x55cfe71a3560 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7ff8fc00cc60 tx=0x7ff8fc00e720 comp rx=0 tx=0).stop 2024-04-21T23:17:48.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.837+0000 7ff90a45c740 1 -- 172.21.15.40:0/1816261415 shutdown_connections 2024-04-21T23:17:48.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.837+0000 7ff90a45c740 1 --2- 172.21.15.40:0/1816261415 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cfe71a3bd0 0x55cfe71a6060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.837+0000 7ff90a45c740 1 --2- 172.21.15.40:0/1816261415 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cfe719ea60 0x55cfe71a3560 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.837+0000 7ff90a45c740 1 --2- 172.21.15.40:0/1816261415 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cfe719e070 0x55cfe719e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.837+0000 7ff90a45c740 1 -- 172.21.15.40:0/1816261415 >> 172.21.15.40:0/1816261415 conn(0x55cfe71995e0 msgr2=0x55cfe71999e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:48.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.837+0000 7ff90a45c740 1 -- 172.21.15.40:0/1816261415 shutdown_connections 2024-04-21T23:17:48.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.837+0000 7ff90a45c740 1 -- 172.21.15.40:0/1816261415 wait complete. 2024-04-21T23:17:48.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.838+0000 7ff90a45c740 1 -- start start 2024-04-21T23:17:48.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.838+0000 7ff90a45c740 1 Processor -- start 2024-04-21T23:17:48.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.838+0000 7ff90a45c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cfe719e070 0x55cfe71a85b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:48.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.838+0000 7ff90a45c740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cfe719ea60 0x55cfe71a8af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:48.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.838+0000 7ff90a45c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cfe71a3bd0 0x55cfe71afb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:48.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.838+0000 7ff90a45c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55cfe71a6b80 con 0x55cfe71a3bd0 2024-04-21T23:17:48.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.838+0000 7ff90a45c740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55cfe70bebe0 con 0x55cfe719ea60 2024-04-21T23:17:48.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.838+0000 7ff90a45c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55cfe71a6d00 con 0x55cfe719e070 2024-04-21T23:17:48.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.838+0000 7ff90a45c740 10 client.0 _ll_register_callbacks cb 0x55cfe70be5d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:17:48.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.838+0000 7ff905e15640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cfe71a3bd0 0x55cfe71afb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:48.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.838+0000 7ff905e15640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cfe71a3bd0 0x55cfe71afb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39018/0 (socket says 172.21.15.40:39018) 2024-04-21T23:17:48.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.838+0000 7ff905e15640 1 -- 172.21.15.40:0/724565554 learned_addr learned my addr 172.21.15.40:0/724565554 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:17:48.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.838+0000 7ff905614640 1 --2- 172.21.15.40:0/724565554 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cfe719e070 0x55cfe71a85b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:48.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.838+0000 7ff905e15640 1 -- 172.21.15.40:0/724565554 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cfe719e070 msgr2=0x55cfe71a85b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:17:48.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.838+0000 7ff905e15640 1 --2- 172.21.15.40:0/724565554 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cfe719e070 0x55cfe71a85b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.839+0000 7ff905e15640 1 -- 172.21.15.40:0/724565554 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cfe719ea60 msgr2=0x55cfe71a8af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:17:48.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.839+0000 7ff905e15640 1 --2- 172.21.15.40:0/724565554 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cfe719ea60 0x55cfe71a8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.839+0000 7ff905e15640 1 -- 172.21.15.40:0/724565554 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cfe6ee16b0 con 0x55cfe71a3bd0 2024-04-21T23:17:48.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.839+0000 7ff905e15640 1 --2- 172.21.15.40:0/724565554 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cfe71a3bd0 0x55cfe71afb30 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7ff8f80142f0 tx=0x7ff8f8004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2e5a9635c13f805 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:48.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.839+0000 7ff8e8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-21T23:17:48.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.839+0000 7ff8e8ff9640 1 -- 172.21.15.40:0/724565554 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8f8024030 con 0x55cfe71a3bd0 2024-04-21T23:17:48.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.839+0000 7ff8e8ff9640 1 -- 172.21.15.40:0/724565554 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7ff8f8011e00 con 0x55cfe71a3bd0 2024-04-21T23:17:48.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.839+0000 7ff90a45c740 1 -- 172.21.15.40:0/724565554 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff8fc0047a0 con 0x55cfe71a3bd0 2024-04-21T23:17:48.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.839+0000 7ff8e8ff9640 1 -- 172.21.15.40:0/724565554 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7ff8f8031050 con 0x55cfe71a3bd0 2024-04-21T23:17:48.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.839+0000 7ff90a45c740 10 client.15237 fetch_fsmap learned FSMap version 24 2024-04-21T23:17:48.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.839+0000 7ff90a45c740 1 -- 172.21.15.40:0/724565554 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55cfe71b2af0 con 0x55cfe71a3bd0 2024-04-21T23:17:48.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.839+0000 7ff8e8ff9640 1 -- 172.21.15.40:0/724565554 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8fc0047a0 con 0x55cfe71a3bd0 2024-04-21T23:17:48.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.839+0000 7ff8e8ff9640 1 -- 172.21.15.40:0/724565554 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7ff8f8013960 con 0x55cfe71a3bd0 2024-04-21T23:17:48.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.839+0000 7ff90a45c740 10 client.15237 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T23:17:48.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.840+0000 7ff90a45c740 10 client.15237 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-21T23:17:48.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.840+0000 7ff90a45c740 20 client.15237 populate_metadata read hostname 'smithi040' 2024-04-21T23:17:48.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.840+0000 7ff90a45c740 1 --2- 172.21.15.40:0/724565554 >> [v2:172.21.15.174:6838/2467509274,v1:172.21.15.174:6839/2467509274] conn(0x55cfe71b5260 0x55cfe71b66d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:48.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.840+0000 7ff90a45c740 4 client.15237 mds_command: new command op to 24502 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:17:48.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.840+0000 7ff90a45c740 1 -- 172.21.15.40:0/724565554 --> [v2:172.21.15.174:6838/2467509274,v1:172.21.15.174:6839/2467509274] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cfe7110330 con 0x55cfe71b5260 2024-04-21T23:17:48.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.840+0000 7ff905614640 1 --2- 172.21.15.40:0/724565554 >> [v2:172.21.15.174:6838/2467509274,v1:172.21.15.174:6839/2467509274] conn(0x55cfe71b5260 0x55cfe71b66d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:48.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.840+0000 7ff8e8ff9640 10 client.15237 ms_handle_connect on v2:172.21.15.174:6838/2467509274 2024-04-21T23:17:48.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.840+0000 7ff905614640 1 --2- 172.21.15.40:0/724565554 >> [v2:172.21.15.174:6838/2467509274,v1:172.21.15.174:6839/2467509274] conn(0x55cfe71b5260 0x55cfe71b66d0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b3d3c71bda1da7be server_cookie=25ce88559d77d34b in_seq=0 out_seq=0 2024-04-21T23:17:48.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.841+0000 7ff8e8ff9640 1 -- 172.21.15.40:0/724565554 <== mds.1 v2:172.21.15.174:6838/2467509274 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cfe7110330 con 0x55cfe71b5260 2024-04-21T23:17:48.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.841+0000 7ff8e8ff9640 10 client.15237 handle_command_reply: tid=0 2024-04-21T23:17:48.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.841+0000 7ff90a45c740 1 client.15237 shutdown 2024-04-21T23:17:48.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.841+0000 7ff90a45c740 1 -- 172.21.15.40:0/724565554 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cfe71a3bd0 msgr2=0x55cfe71afb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:48.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.841+0000 7ff90a45c740 1 --2- 172.21.15.40:0/724565554 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cfe71a3bd0 0x55cfe71afb30 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7ff8f80142f0 tx=0x7ff8f8004ab0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.841+0000 7ff90a45c740 1 -- 172.21.15.40:0/724565554 shutdown_connections 2024-04-21T23:17:48.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.841+0000 7ff90a45c740 1 --2- 172.21.15.40:0/724565554 >> [v2:172.21.15.174:6838/2467509274,v1:172.21.15.174:6839/2467509274] conn(0x55cfe71b5260 0x55cfe71b66d0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.841+0000 7ff90a45c740 1 --2- 172.21.15.40:0/724565554 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cfe719ea60 0x55cfe71a8af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.841+0000 7ff90a45c740 1 --2- 172.21.15.40:0/724565554 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cfe71a3bd0 0x55cfe71afb30 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.841+0000 7ff8e8ff9640 0 client.15237 ms_handle_reset on v2:172.21.15.174:6838/2467509274 2024-04-21T23:17:48.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.841+0000 7ff90a45c740 1 --2- 172.21.15.40:0/724565554 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cfe719e070 0x55cfe71a85b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.841+0000 7ff90a45c740 1 -- 172.21.15.40:0/724565554 >> 172.21.15.40:0/724565554 conn(0x55cfe71995e0 msgr2=0x55cfe7198720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:48.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.842+0000 7ff90a45c740 1 -- 172.21.15.40:0/724565554 shutdown_connections 2024-04-21T23:17:48.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.842+0000 7ff90a45c740 1 -- 172.21.15.40:0/724565554 wait complete. 2024-04-21T23:17:48.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.842+0000 7ff90a45c740 20 client.15237 trim_cache size 0 max 16384 2024-04-21T23:17:48.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.860+0000 7ff90a45c740 1 Processor -- start 2024-04-21T23:17:48.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.860+0000 7ff90a45c740 1 -- start start 2024-04-21T23:17:48.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.860+0000 7ff90a45c740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cfe719e070 0x55cfe7197e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:48.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.860+0000 7ff90a45c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cfe719ea60 0x55cfe71afaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:48.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.860+0000 7ff90a45c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cfe71a3bd0 0x55cfe71983d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:48.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.860+0000 7ff90a45c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55cfe71b4a10 con 0x55cfe71a3bd0 2024-04-21T23:17:48.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.860+0000 7ff90a45c740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55cfe71b50e0 con 0x55cfe719e070 2024-04-21T23:17:48.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.860+0000 7ff90a45c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55cfe71b57b0 con 0x55cfe719ea60 2024-04-21T23:17:48.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.860+0000 7ff905e15640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cfe71a3bd0 0x55cfe71983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:48.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.860+0000 7ff905e15640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cfe71a3bd0 0x55cfe71983d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39032/0 (socket says 172.21.15.40:39032) 2024-04-21T23:17:48.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.860+0000 7ff905e15640 1 -- 172.21.15.40:0/650941459 learned_addr learned my addr 172.21.15.40:0/650941459 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:17:48.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.860+0000 7ff905614640 1 --2- 172.21.15.40:0/650941459 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cfe719e070 0x55cfe7197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:48.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.860+0000 7ff904e13640 1 --2- 172.21.15.40:0/650941459 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cfe719ea60 0x55cfe71afaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:48.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.860+0000 7ff905e15640 1 -- 172.21.15.40:0/650941459 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cfe719ea60 msgr2=0x55cfe71afaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:17:48.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.860+0000 7ff905e15640 1 --2- 172.21.15.40:0/650941459 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cfe719ea60 0x55cfe71afaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.860+0000 7ff905e15640 1 -- 172.21.15.40:0/650941459 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cfe719e070 msgr2=0x55cfe7197e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:17:48.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.860+0000 7ff905e15640 1 --2- 172.21.15.40:0/650941459 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cfe719e070 0x55cfe7197e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.860+0000 7ff905e15640 1 -- 172.21.15.40:0/650941459 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cfe719a350 con 0x55cfe71a3bd0 2024-04-21T23:17:48.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.860+0000 7ff905614640 1 --2- 172.21.15.40:0/650941459 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cfe719e070 0x55cfe7197e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:17:48.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.861+0000 7ff905e15640 1 --2- 172.21.15.40:0/650941459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cfe71a3bd0 0x55cfe71983d0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7ff8f80142c0 tx=0x7ff8f8011f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b129c91597fce3a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:48.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.861+0000 7ff8eaffd640 1 -- 172.21.15.40:0/650941459 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8f8024040 con 0x55cfe71a3bd0 2024-04-21T23:17:48.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.861+0000 7ff8eaffd640 1 -- 172.21.15.40:0/650941459 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8f802b810 con 0x55cfe71a3bd0 2024-04-21T23:17:48.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.861+0000 7ff8eaffd640 1 -- 172.21.15.40:0/650941459 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8f800fcd0 con 0x55cfe71a3bd0 2024-04-21T23:17:48.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.861+0000 7ff90a45c740 1 -- 172.21.15.40:0/650941459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cfe71a3bd0 msgr2=0x55cfe71983d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:48.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.861+0000 7ff90a45c740 1 --2- 172.21.15.40:0/650941459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cfe71a3bd0 0x55cfe71983d0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7ff8f80142c0 tx=0x7ff8f8011f90 comp rx=0 tx=0).stop 2024-04-21T23:17:48.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.861+0000 7ff90a45c740 1 -- 172.21.15.40:0/650941459 shutdown_connections 2024-04-21T23:17:48.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.861+0000 7ff90a45c740 1 --2- 172.21.15.40:0/650941459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cfe71a3bd0 0x55cfe71983d0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.861+0000 7ff90a45c740 1 --2- 172.21.15.40:0/650941459 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cfe719ea60 0x55cfe71afaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.861+0000 7ff90a45c740 1 --2- 172.21.15.40:0/650941459 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cfe719e070 0x55cfe7197e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.861+0000 7ff90a45c740 1 -- 172.21.15.40:0/650941459 >> 172.21.15.40:0/650941459 conn(0x55cfe71995e0 msgr2=0x55cfe71a8570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:48.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.861+0000 7ff90a45c740 1 -- 172.21.15.40:0/650941459 shutdown_connections 2024-04-21T23:17:48.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.861+0000 7ff90a45c740 1 -- 172.21.15.40:0/650941459 wait complete. 2024-04-21T23:17:48.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.862+0000 7ff90a45c740 1 -- start start 2024-04-21T23:17:48.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.862+0000 7ff90a45c740 1 Processor -- start 2024-04-21T23:17:48.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.862+0000 7ff90a45c740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cfe719e070 0x55cfe71abbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:48.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.862+0000 7ff90a45c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cfe719ea60 0x55cfe71ac120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:48.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.862+0000 7ff90a45c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cfe71a3bd0 0x55cfe71af0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:48.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.862+0000 7ff90a45c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55cfe71b6550 con 0x55cfe719ea60 2024-04-21T23:17:48.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.862+0000 7ff90a45c740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55cfe71b5e80 con 0x55cfe719e070 2024-04-21T23:17:48.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.862+0000 7ff90a45c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55cfe71b6c20 con 0x55cfe71a3bd0 2024-04-21T23:17:48.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.862+0000 7ff90a45c740 10 client.0 _ll_register_callbacks cb 0x7ff8f8014780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:17:48.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.862+0000 7ff904e13640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cfe719ea60 0x55cfe71ac120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:48.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.862+0000 7ff904e13640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cfe719ea60 0x55cfe71ac120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39034/0 (socket says 172.21.15.40:39034) 2024-04-21T23:17:48.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.862+0000 7ff904e13640 1 -- 172.21.15.40:0/2318795727 learned_addr learned my addr 172.21.15.40:0/2318795727 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:17:48.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.862+0000 7ff905e15640 1 --2- 172.21.15.40:0/2318795727 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cfe71a3bd0 0x55cfe71af0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:48.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.862+0000 7ff905614640 1 --2- 172.21.15.40:0/2318795727 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cfe719e070 0x55cfe71abbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:48.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.863+0000 7ff904e13640 1 -- 172.21.15.40:0/2318795727 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cfe71a3bd0 msgr2=0x55cfe71af0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:17:48.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.863+0000 7ff904e13640 1 --2- 172.21.15.40:0/2318795727 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cfe71a3bd0 0x55cfe71af0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.863+0000 7ff904e13640 1 -- 172.21.15.40:0/2318795727 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cfe719e070 msgr2=0x55cfe71abbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:17:48.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.863+0000 7ff904e13640 1 --2- 172.21.15.40:0/2318795727 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cfe719e070 0x55cfe71abbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.863+0000 7ff904e13640 1 -- 172.21.15.40:0/2318795727 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cfe71af6e0 con 0x55cfe719ea60 2024-04-21T23:17:48.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.863+0000 7ff905e15640 1 --2- 172.21.15.40:0/2318795727 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cfe71a3bd0 0x55cfe71af0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:17:48.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.863+0000 7ff904e13640 1 --2- 172.21.15.40:0/2318795727 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cfe719ea60 0x55cfe71ac120 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7ff8f00b9dc0 tx=0x7ff8f0049000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aba3c3d3ea8ed488 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:48.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.863+0000 7ff8e8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-21T23:17:48.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.863+0000 7ff8e8ff9640 1 -- 172.21.15.40:0/2318795727 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8f00bd2c0 con 0x55cfe719ea60 2024-04-21T23:17:48.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.863+0000 7ff8e8ff9640 1 -- 172.21.15.40:0/2318795727 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7ff8f0003780 con 0x55cfe719ea60 2024-04-21T23:17:48.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.863+0000 7ff90a45c740 1 -- 172.21.15.40:0/2318795727 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff8f802b810 con 0x55cfe719ea60 2024-04-21T23:17:48.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.863+0000 7ff8e8ff9640 1 -- 172.21.15.40:0/2318795727 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7ff8f003b310 con 0x55cfe719ea60 2024-04-21T23:17:48.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.863+0000 7ff90a45c740 10 client.15243 fetch_fsmap learned FSMap version 24 2024-04-21T23:17:48.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.863+0000 7ff90a45c740 1 -- 172.21.15.40:0/2318795727 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55cfe721bb40 con 0x55cfe719ea60 2024-04-21T23:17:48.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.863+0000 7ff8e8ff9640 1 -- 172.21.15.40:0/2318795727 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8f802b810 con 0x55cfe719ea60 2024-04-21T23:17:48.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.863+0000 7ff8e8ff9640 1 -- 172.21.15.40:0/2318795727 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7ff8f003a2a0 con 0x55cfe719ea60 2024-04-21T23:17:48.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.863+0000 7ff90a45c740 10 client.15243 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T23:17:48.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.863+0000 7ff90a45c740 10 client.15243 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-21T23:17:48.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.863+0000 7ff90a45c740 20 client.15243 populate_metadata read hostname 'smithi040' 2024-04-21T23:17:48.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.863+0000 7ff90a45c740 1 --2- 172.21.15.40:0/2318795727 >> [v2:172.21.15.174:6838/2467509274,v1:172.21.15.174:6839/2467509274] conn(0x55cfe7220740 0x55cfe721f450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:48.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.863+0000 7ff90a45c740 4 client.15243 mds_command: new command op to 24502 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-21T23:17:48.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.863+0000 7ff90a45c740 1 -- 172.21.15.40:0/2318795727 --> [v2:172.21.15.174:6838/2467509274,v1:172.21.15.174:6839/2467509274] -- command(tid 0: {"prefix": "status"}) -- 0x55cfe710a680 con 0x55cfe7220740 2024-04-21T23:17:48.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.864+0000 7ff905614640 1 --2- 172.21.15.40:0/2318795727 >> [v2:172.21.15.174:6838/2467509274,v1:172.21.15.174:6839/2467509274] conn(0x55cfe7220740 0x55cfe721f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:48.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.864+0000 7ff8e8ff9640 10 client.15243 ms_handle_connect on v2:172.21.15.174:6838/2467509274 2024-04-21T23:17:48.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.864+0000 7ff905614640 1 --2- 172.21.15.40:0/2318795727 >> [v2:172.21.15.174:6838/2467509274,v1:172.21.15.174:6839/2467509274] conn(0x55cfe7220740 0x55cfe721f450 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=580e2b1234244bc5 server_cookie=121255e79b429f9b in_seq=0 out_seq=0 2024-04-21T23:17:48.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.864+0000 7ff8e8ff9640 1 -- 172.21.15.40:0/2318795727 <== mds.1 v2:172.21.15.174:6838/2467509274 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55cfe710a680 con 0x55cfe7220740 2024-04-21T23:17:48.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.864+0000 7ff8e8ff9640 10 client.15243 handle_command_reply: tid=0 2024-04-21T23:17:48.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.864+0000 7ff90a45c740 1 client.15243 shutdown 2024-04-21T23:17:48.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.865+0000 7ff90a45c740 1 -- 172.21.15.40:0/2318795727 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cfe719ea60 msgr2=0x55cfe71ac120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:48.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.865+0000 7ff90a45c740 1 --2- 172.21.15.40:0/2318795727 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cfe719ea60 0x55cfe71ac120 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7ff8f00b9dc0 tx=0x7ff8f0049000 comp rx=0 tx=0).stop 2024-04-21T23:17:48.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.865+0000 7ff90a45c740 1 -- 172.21.15.40:0/2318795727 shutdown_connections 2024-04-21T23:17:48.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.865+0000 7ff90a45c740 1 --2- 172.21.15.40:0/2318795727 >> [v2:172.21.15.174:6838/2467509274,v1:172.21.15.174:6839/2467509274] conn(0x55cfe7220740 0x55cfe721f450 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.865+0000 7ff90a45c740 1 --2- 172.21.15.40:0/2318795727 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cfe71a3bd0 0x55cfe71af0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.865+0000 7ff90a45c740 1 --2- 172.21.15.40:0/2318795727 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cfe719ea60 0x55cfe71ac120 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.865+0000 7ff8e8ff9640 0 client.15243 ms_handle_reset on v2:172.21.15.174:6838/2467509274 2024-04-21T23:17:48.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.865+0000 7ff90a45c740 1 --2- 172.21.15.40:0/2318795727 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cfe719e070 0x55cfe71abbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.865+0000 7ff90a45c740 1 -- 172.21.15.40:0/2318795727 >> 172.21.15.40:0/2318795727 conn(0x55cfe71995e0 msgr2=0x55cfe7197e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:48.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.865+0000 7ff90a45c740 1 -- 172.21.15.40:0/2318795727 shutdown_connections 2024-04-21T23:17:48.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.865+0000 7ff90a45c740 1 -- 172.21.15.40:0/2318795727 wait complete. 2024-04-21T23:17:48.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.865+0000 7ff90a45c740 20 client.15243 trim_cache size 0 max 16384 2024-04-21T23:17:48.874 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-21T23:17:48.874 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "d814e4dc-0033-11ef-bc93-c7b262605968", 2024-04-21T23:17:48.874 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 1, 2024-04-21T23:17:48.874 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 24502, 2024-04-21T23:17:48.874 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-04-21T23:17:48.874 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-04-21T23:17:48.874 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-04-21T23:17:48.874 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 88.386055511999999, 2024-04-21T23:17:48.875 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 24, 2024-04-21T23:17:48.875 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 92, 2024-04-21T23:17:48.875 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 90, 2024-04-21T23:17:48.875 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 127.337692918 2024-04-21T23:17:48.875 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-21T23:17:48.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.865+0000 7ff907878640 1 -- 172.21.15.40:0/1779188821 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7ff8ec0a2c20 msgr2=0x7ff8ec0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:48.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.865+0000 7ff907878640 1 --2- 172.21.15.40:0/1779188821 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7ff8ec0a2c20 0x7ff8ec0a50e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7ff8f8009fd0 tx=0x7ff8f8004a40 comp rx=0 tx=0).stop 2024-04-21T23:17:48.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.865+0000 7ff907878640 1 -- 172.21.15.40:0/1779188821 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff900104c10 msgr2=0x7ff900197ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:48.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.865+0000 7ff907878640 1 --2- 172.21.15.40:0/1779188821 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff900104c10 0x7ff900197ff0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7ff8f0005ab0 tx=0x7ff8f0005dc0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.865+0000 7ff907878640 1 -- 172.21.15.40:0/1779188821 shutdown_connections 2024-04-21T23:17:48.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.865+0000 7ff907878640 1 --2- 172.21.15.40:0/1779188821 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7ff8ec0a2c20 0x7ff8ec0a50e0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.865+0000 7ff907878640 1 --2- 172.21.15.40:0/1779188821 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff900107540 0x7ff90019f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.865+0000 7ff907878640 1 --2- 172.21.15.40:0/1779188821 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff900104c10 0x7ff900197ff0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.865+0000 7ff907878640 1 --2- 172.21.15.40:0/1779188821 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff9001022e0 0x7ff900197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.865+0000 7ff907878640 1 -- 172.21.15.40:0/1779188821 >> 172.21.15.40:0/1779188821 conn(0x7ff9000fbc30 msgr2=0x7ff900102c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:48.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.866+0000 7ff907878640 1 -- 172.21.15.40:0/1779188821 shutdown_connections 2024-04-21T23:17:48.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.866+0000 7ff907878640 1 -- 172.21.15.40:0/1779188821 wait complete. 2024-04-21T23:17:48.877 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-21T23:17:48.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.981+0000 7f45616fa640 1 -- 172.21.15.40:0/1224105093 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f455c101890 msgr2=0x7f455c101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:48.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.981+0000 7f45616fa640 1 --2- 172.21.15.40:0/1224105093 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f455c101890 0x7f455c101d10 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f4544009900 tx=0x7f454402bc30 comp rx=0 tx=0).stop 2024-04-21T23:17:48.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.981+0000 7f45616fa640 1 -- 172.21.15.40:0/1224105093 shutdown_connections 2024-04-21T23:17:48.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.981+0000 7f45616fa640 1 --2- 172.21.15.40:0/1224105093 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f455c102250 0x7f455c108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.981+0000 7f45616fa640 1 --2- 172.21.15.40:0/1224105093 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f455c101890 0x7f455c101d10 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.981+0000 7f45616fa640 1 --2- 172.21.15.40:0/1224105093 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f455c1006a0 0x7f455c100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.981+0000 7f45616fa640 1 -- 172.21.15.40:0/1224105093 >> 172.21.15.40:0/1224105093 conn(0x7f455c0fbc30 msgr2=0x7f455c0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:48.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.981+0000 7f45616fa640 1 -- 172.21.15.40:0/1224105093 shutdown_connections 2024-04-21T23:17:48.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.981+0000 7f45616fa640 1 -- 172.21.15.40:0/1224105093 wait complete. 2024-04-21T23:17:48.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.982+0000 7f45616fa640 1 Processor -- start 2024-04-21T23:17:48.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.982+0000 7f45616fa640 1 -- start start 2024-04-21T23:17:48.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.982+0000 7f45616fa640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f455c1006a0 0x7f455c197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:48.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.982+0000 7f45616fa640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f455c101890 0x7f455c197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:48.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.982+0000 7f45616fa640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f455c102250 0x7f455c19efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:48.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.982+0000 7f45616fa640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f455c10c300 con 0x7f455c101890 2024-04-21T23:17:48.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.982+0000 7f45616fa640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f455c10c180 con 0x7f455c1006a0 2024-04-21T23:17:48.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.982+0000 7f45616fa640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f455c10c480 con 0x7f455c102250 2024-04-21T23:17:48.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.982+0000 7f45527fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f455c101890 0x7f455c197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:48.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.982+0000 7f45527fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f455c101890 0x7f455c197f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39046/0 (socket says 172.21.15.40:39046) 2024-04-21T23:17:48.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.982+0000 7f45527fc640 1 -- 172.21.15.40:0/3571725266 learned_addr learned my addr 172.21.15.40:0/3571725266 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:17:48.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.983+0000 7f455b7fe640 1 --2- 172.21.15.40:0/3571725266 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f455c102250 0x7f455c19efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:48.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.983+0000 7f455affd640 1 --2- 172.21.15.40:0/3571725266 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f455c1006a0 0x7f455c197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:48.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.983+0000 7f45527fc640 1 -- 172.21.15.40:0/3571725266 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f455c102250 msgr2=0x7f455c19efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:48.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.983+0000 7f45527fc640 1 --2- 172.21.15.40:0/3571725266 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f455c102250 0x7f455c19efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.983+0000 7f45527fc640 1 -- 172.21.15.40:0/3571725266 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f455c1006a0 msgr2=0x7f455c197a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:48.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.983+0000 7f45527fc640 1 --2- 172.21.15.40:0/3571725266 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f455c1006a0 0x7f455c197a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.983+0000 7f45527fc640 1 -- 172.21.15.40:0/3571725266 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f455c19f630 con 0x7f455c101890 2024-04-21T23:17:48.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.983+0000 7f455affd640 1 --2- 172.21.15.40:0/3571725266 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f455c1006a0 0x7f455c197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:17:48.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.983+0000 7f45527fc640 1 --2- 172.21.15.40:0/3571725266 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f455c101890 0x7f455c197f50 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f4544009230 tx=0x7f4544005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:48.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.983+0000 7f4558ff9640 1 -- 172.21.15.40:0/3571725266 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4544041070 con 0x7f455c101890 2024-04-21T23:17:48.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.983+0000 7f45616fa640 1 -- 172.21.15.40:0/3571725266 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f455c19f8c0 con 0x7f455c101890 2024-04-21T23:17:48.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.983+0000 7f4558ff9640 1 -- 172.21.15.40:0/3571725266 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f45440054e0 con 0x7f455c101890 2024-04-21T23:17:48.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.983+0000 7f4558ff9640 1 -- 172.21.15.40:0/3571725266 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4544034440 con 0x7f455c101890 2024-04-21T23:17:48.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.983+0000 7f45616fa640 1 -- 172.21.15.40:0/3571725266 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f455c19fe00 con 0x7f455c101890 2024-04-21T23:17:48.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.985+0000 7f4558ff9640 1 -- 172.21.15.40:0/3571725266 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f4544044070 con 0x7f455c101890 2024-04-21T23:17:48.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.986+0000 7f45643b7740 1 Processor -- start 2024-04-21T23:17:48.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.986+0000 7f45643b7740 1 -- start start 2024-04-21T23:17:48.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.986+0000 7f4558ff9640 1 --2- 172.21.15.40:0/3571725266 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f45480a2c20 0x7f45480a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:48.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.986+0000 7f45643b7740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd39873070 0x55dd39873490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:48.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.986+0000 7f45643b7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd39873a60 0x55dd39878560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:48.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.986+0000 7f45643b7740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd39878bd0 0x55dd3987b060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:48.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.986+0000 7f45643b7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55dd397d7df0 con 0x55dd39873a60 2024-04-21T23:17:48.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.986+0000 7f45643b7740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55dd39780640 con 0x55dd39873070 2024-04-21T23:17:48.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.986+0000 7f45643b7740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55dd397d3db0 con 0x55dd39878bd0 2024-04-21T23:17:48.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.986+0000 7f45527fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd39873a60 0x55dd39878560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:48.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.986+0000 7f455b7fe640 1 --2- 172.21.15.40:0/3571725266 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f45480a2c20 0x7f45480a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:48.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.986+0000 7f45527fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd39873a60 0x55dd39878560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39062/0 (socket says 172.21.15.40:39062) 2024-04-21T23:17:48.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.986+0000 7f45527fc640 1 -- 172.21.15.40:0/24273284 learned_addr learned my addr 172.21.15.40:0/24273284 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:17:48.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.986+0000 7f4558ff9640 1 -- 172.21.15.40:0/3571725266 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f45440f11e0 con 0x7f455c101890 2024-04-21T23:17:48.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.986+0000 7f455affd640 1 --2- 172.21.15.40:0/24273284 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd39873070 0x55dd39873490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:48.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.986+0000 7f45527fc640 1 -- 172.21.15.40:0/24273284 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd39878bd0 msgr2=0x55dd3987b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:17:48.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.987+0000 7f45527fc640 1 --2- 172.21.15.40:0/24273284 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd39878bd0 0x55dd3987b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.987+0000 7f45527fc640 1 -- 172.21.15.40:0/24273284 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd39873070 msgr2=0x55dd39873490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:17:48.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.987+0000 7f45527fc640 1 --2- 172.21.15.40:0/24273284 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd39873070 0x55dd39873490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.987+0000 7f45527fc640 1 -- 172.21.15.40:0/24273284 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd394ec2d0 con 0x55dd39873a60 2024-04-21T23:17:48.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.987+0000 7f455b7fe640 1 --2- 172.21.15.40:0/3571725266 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f45480a2c20 0x7f45480a50e0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f4554005fd0 tx=0x7f45540098c0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:48.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.987+0000 7f45527fc640 1 --2- 172.21.15.40:0/24273284 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd39873a60 0x55dd39878560 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f4544009430 tx=0x7f4544049000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d8135b425d750c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:48.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.987+0000 7f45517fa640 1 -- 172.21.15.40:0/24273284 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f454404b070 con 0x55dd39873a60 2024-04-21T23:17:48.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.987+0000 7f45517fa640 1 -- 172.21.15.40:0/24273284 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4544005690 con 0x55dd39873a60 2024-04-21T23:17:48.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.987+0000 7f45517fa640 1 -- 172.21.15.40:0/24273284 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45440a80b0 con 0x55dd39873a60 2024-04-21T23:17:48.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.988+0000 7f45643b7740 1 -- 172.21.15.40:0/24273284 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd39873a60 msgr2=0x55dd39878560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:48.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.988+0000 7f45643b7740 1 --2- 172.21.15.40:0/24273284 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd39873a60 0x55dd39878560 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f4544009430 tx=0x7f4544049000 comp rx=0 tx=0).stop 2024-04-21T23:17:48.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.988+0000 7f45643b7740 1 -- 172.21.15.40:0/24273284 shutdown_connections 2024-04-21T23:17:48.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.988+0000 7f45643b7740 1 --2- 172.21.15.40:0/24273284 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd39878bd0 0x55dd3987b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.988+0000 7f45643b7740 1 --2- 172.21.15.40:0/24273284 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd39873a60 0x55dd39878560 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.988+0000 7f45643b7740 1 --2- 172.21.15.40:0/24273284 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd39873070 0x55dd39873490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.988+0000 7f45643b7740 1 -- 172.21.15.40:0/24273284 >> 172.21.15.40:0/24273284 conn(0x55dd3986e5e0 msgr2=0x55dd3986e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:48.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.988+0000 7f45643b7740 1 -- 172.21.15.40:0/24273284 shutdown_connections 2024-04-21T23:17:48.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.988+0000 7f45643b7740 1 -- 172.21.15.40:0/24273284 wait complete. 2024-04-21T23:17:48.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.988+0000 7f45643b7740 1 -- start start 2024-04-21T23:17:48.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.988+0000 7f45643b7740 1 Processor -- start 2024-04-21T23:17:48.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.988+0000 7f45643b7740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd39873070 0x55dd3987d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:48.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.988+0000 7f45643b7740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd39873a60 0x55dd3987d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:48.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.988+0000 7f45643b7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd39878bd0 0x55dd39884800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:48.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.988+0000 7f45643b7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55dd3987b940 con 0x55dd39878bd0 2024-04-21T23:17:48.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.988+0000 7f45643b7740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55dd39793be0 con 0x55dd39873070 2024-04-21T23:17:48.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.988+0000 7f45643b7740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55dd3987bac0 con 0x55dd39873a60 2024-04-21T23:17:48.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.988+0000 7f45643b7740 10 client.0 _ll_register_callbacks cb 0x55dd397935d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:17:48.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.988+0000 7f455b7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd39878bd0 0x55dd39884800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:48.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.988+0000 7f455b7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd39878bd0 0x55dd39884800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39076/0 (socket says 172.21.15.40:39076) 2024-04-21T23:17:48.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.988+0000 7f455b7fe640 1 -- 172.21.15.40:0/3816301032 learned_addr learned my addr 172.21.15.40:0/3816301032 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:17:48.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.988+0000 7f455affd640 1 --2- 172.21.15.40:0/3816301032 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd39873070 0x55dd3987d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:48.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.989+0000 7f45527fc640 1 --2- 172.21.15.40:0/3816301032 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd39873a60 0x55dd3987d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:48.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.989+0000 7f455b7fe640 1 -- 172.21.15.40:0/3816301032 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd39873a60 msgr2=0x55dd3987d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:17:48.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.989+0000 7f455b7fe640 1 --2- 172.21.15.40:0/3816301032 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd39873a60 0x55dd3987d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.989+0000 7f455b7fe640 1 -- 172.21.15.40:0/3816301032 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd39873070 msgr2=0x55dd3987d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:17:48.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.989+0000 7f455b7fe640 1 --2- 172.21.15.40:0/3816301032 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd39873070 0x55dd3987d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.989+0000 7f455b7fe640 1 -- 172.21.15.40:0/3816301032 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd395b66b0 con 0x55dd39878bd0 2024-04-21T23:17:48.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.989+0000 7f45527fc640 1 --2- 172.21.15.40:0/3816301032 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd39873a60 0x55dd3987d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:17:48.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.989+0000 7f455b7fe640 1 --2- 172.21.15.40:0/3816301032 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd39878bd0 0x55dd39884800 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f45540138d0 tx=0x7f4554004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54fd48052d12ffde server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:48.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.989+0000 7f45427fc640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-21T23:17:48.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.989+0000 7f45427fc640 1 -- 172.21.15.40:0/3816301032 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f455400fcf0 con 0x55dd39878bd0 2024-04-21T23:17:48.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.989+0000 7f45427fc640 1 -- 172.21.15.40:0/3816301032 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f4554020450 con 0x55dd39878bd0 2024-04-21T23:17:48.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.989+0000 7f45643b7740 1 -- 172.21.15.40:0/3816301032 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55dd397e5330 con 0x55dd39878bd0 2024-04-21T23:17:48.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.989+0000 7f45427fc640 1 -- 172.21.15.40:0/3816301032 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f455402e050 con 0x55dd39878bd0 2024-04-21T23:17:48.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.989+0000 7f45643b7740 10 client.15255 fetch_fsmap learned FSMap version 24 2024-04-21T23:17:48.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.989+0000 7f45643b7740 1 -- 172.21.15.40:0/3816301032 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55dd39887850 con 0x55dd39878bd0 2024-04-21T23:17:48.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.989+0000 7f45427fc640 1 -- 172.21.15.40:0/3816301032 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55dd397e5330 con 0x55dd39878bd0 2024-04-21T23:17:48.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.989+0000 7f45427fc640 1 -- 172.21.15.40:0/3816301032 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f455400f040 con 0x55dd39878bd0 2024-04-21T23:17:48.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.990+0000 7f45643b7740 10 client.15255 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T23:17:48.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.990+0000 7f45643b7740 10 client.15255 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-21T23:17:48.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.990+0000 7f45643b7740 20 client.15255 populate_metadata read hostname 'smithi040' 2024-04-21T23:17:48.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.990+0000 7f45643b7740 1 --2- 172.21.15.40:0/3816301032 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55dd39889fb0 0x55dd3988b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:48.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.990+0000 7f45643b7740 4 client.15255 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:17:48.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.990+0000 7f45643b7740 1 -- 172.21.15.40:0/3816301032 --> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dd397df680 con 0x55dd39889fb0 2024-04-21T23:17:48.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.990+0000 7f455affd640 1 --2- 172.21.15.40:0/3816301032 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55dd39889fb0 0x55dd3988b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:48.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.990+0000 7f455affd640 1 --2- 172.21.15.40:0/3816301032 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55dd39889fb0 0x55dd3988b420 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=751aa3fa7b9e5646 server_cookie=61a529913e04e351 in_seq=0 out_seq=0 2024-04-21T23:17:48.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.990+0000 7f45427fc640 10 client.15255 ms_handle_connect on v2:172.21.15.111:6838/3712235548 2024-04-21T23:17:48.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.991+0000 7f45427fc640 1 -- 172.21.15.40:0/3816301032 <== mds.0 v2:172.21.15.111:6838/3712235548 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55dd397df680 con 0x55dd39889fb0 2024-04-21T23:17:48.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.991+0000 7f45427fc640 10 client.15255 handle_command_reply: tid=0 2024-04-21T23:17:48.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.991+0000 7f45643b7740 1 client.15255 shutdown 2024-04-21T23:17:48.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.991+0000 7f45643b7740 1 -- 172.21.15.40:0/3816301032 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd39878bd0 msgr2=0x55dd39884800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:48.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.991+0000 7f45643b7740 1 --2- 172.21.15.40:0/3816301032 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd39878bd0 0x55dd39884800 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f45540138d0 tx=0x7f4554004ab0 comp rx=0 tx=0).stop 2024-04-21T23:17:48.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.991+0000 7f45643b7740 1 -- 172.21.15.40:0/3816301032 shutdown_connections 2024-04-21T23:17:48.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.991+0000 7f45643b7740 1 --2- 172.21.15.40:0/3816301032 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55dd39889fb0 0x55dd3988b420 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:49.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.991+0000 7f45643b7740 1 --2- 172.21.15.40:0/3816301032 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd39878bd0 0x55dd39884800 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:49.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.991+0000 7f45643b7740 1 --2- 172.21.15.40:0/3816301032 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd39873a60 0x55dd3987d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:49.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.991+0000 7f45643b7740 1 --2- 172.21.15.40:0/3816301032 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd39873070 0x55dd3987d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:49.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.991+0000 7f45427fc640 0 client.15255 ms_handle_reset on v2:172.21.15.111:6838/3712235548 2024-04-21T23:17:49.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.991+0000 7f45643b7740 1 -- 172.21.15.40:0/3816301032 >> 172.21.15.40:0/3816301032 conn(0x55dd3986e5e0 msgr2=0x55dd3986d720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:49.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.991+0000 7f45643b7740 1 -- 172.21.15.40:0/3816301032 shutdown_connections 2024-04-21T23:17:49.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.991+0000 7f45643b7740 1 -- 172.21.15.40:0/3816301032 wait complete. 2024-04-21T23:17:49.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:48.991+0000 7f45643b7740 20 client.15255 trim_cache size 0 max 16384 2024-04-21T23:17:49.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.009+0000 7f45643b7740 1 Processor -- start 2024-04-21T23:17:49.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.009+0000 7f45643b7740 1 -- start start 2024-04-21T23:17:49.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.010+0000 7f45643b7740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd39873070 0x55dd39869340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:49.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.010+0000 7f45643b7740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd39873a60 0x55dd398847c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:49.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.010+0000 7f45643b7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd39878bd0 0x55dd3987d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:49.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.010+0000 7f45643b7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55dd39889760 con 0x55dd39878bd0 2024-04-21T23:17:49.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.010+0000 7f45643b7740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55dd39889e30 con 0x55dd39873a60 2024-04-21T23:17:49.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.010+0000 7f45643b7740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55dd3988a500 con 0x55dd39873070 2024-04-21T23:17:49.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.010+0000 7f455b7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd39878bd0 0x55dd3987d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:49.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.010+0000 7f455b7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd39878bd0 0x55dd3987d280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39090/0 (socket says 172.21.15.40:39090) 2024-04-21T23:17:49.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.010+0000 7f455b7fe640 1 -- 172.21.15.40:0/2062318421 learned_addr learned my addr 172.21.15.40:0/2062318421 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:17:49.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.010+0000 7f45527fc640 1 --2- 172.21.15.40:0/2062318421 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd39873a60 0x55dd398847c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:49.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.010+0000 7f455affd640 1 --2- 172.21.15.40:0/2062318421 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd39873070 0x55dd39869340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:49.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.010+0000 7f455b7fe640 1 -- 172.21.15.40:0/2062318421 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd39873070 msgr2=0x55dd39869340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:17:49.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.010+0000 7f455b7fe640 1 --2- 172.21.15.40:0/2062318421 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd39873070 0x55dd39869340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:49.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.010+0000 7f455b7fe640 1 -- 172.21.15.40:0/2062318421 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd39873a60 msgr2=0x55dd398847c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:17:49.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.010+0000 7f455b7fe640 1 --2- 172.21.15.40:0/2062318421 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd39873a60 0x55dd398847c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:49.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.010+0000 7f455b7fe640 1 -- 172.21.15.40:0/2062318421 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd3986cc20 con 0x55dd39878bd0 2024-04-21T23:17:49.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.010+0000 7f455affd640 1 --2- 172.21.15.40:0/2062318421 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd39873070 0x55dd39869340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:17:49.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.010+0000 7f455b7fe640 1 --2- 172.21.15.40:0/2062318421 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd39878bd0 0x55dd3987d280 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f4554009d50 tx=0x7f4554009d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9ff9db9f19d87ed server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:49.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.011+0000 7f4550ff9640 1 -- 172.21.15.40:0/2062318421 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45540293f0 con 0x55dd39878bd0 2024-04-21T23:17:49.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.011+0000 7f4550ff9640 1 -- 172.21.15.40:0/2062318421 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4554036070 con 0x55dd39878bd0 2024-04-21T23:17:49.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.011+0000 7f4550ff9640 1 -- 172.21.15.40:0/2062318421 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45540206b0 con 0x55dd39878bd0 2024-04-21T23:17:49.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.011+0000 7f45643b7740 1 -- 172.21.15.40:0/2062318421 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd39878bd0 msgr2=0x55dd3987d280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:49.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.011+0000 7f45643b7740 1 --2- 172.21.15.40:0/2062318421 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd39878bd0 0x55dd3987d280 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f4554009d50 tx=0x7f4554009d80 comp rx=0 tx=0).stop 2024-04-21T23:17:49.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.011+0000 7f45643b7740 1 -- 172.21.15.40:0/2062318421 shutdown_connections 2024-04-21T23:17:49.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.011+0000 7f45643b7740 1 --2- 172.21.15.40:0/2062318421 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd39873a60 0x55dd398847c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:49.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.011+0000 7f45643b7740 1 --2- 172.21.15.40:0/2062318421 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd39878bd0 0x55dd3987d280 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:49.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.011+0000 7f45643b7740 1 --2- 172.21.15.40:0/2062318421 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd39873070 0x55dd39869340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:49.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.011+0000 7f45643b7740 1 -- 172.21.15.40:0/2062318421 >> 172.21.15.40:0/2062318421 conn(0x55dd3986e5e0 msgr2=0x55dd3986e9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:49.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.011+0000 7f45643b7740 1 -- 172.21.15.40:0/2062318421 shutdown_connections 2024-04-21T23:17:49.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.011+0000 7f45643b7740 1 -- 172.21.15.40:0/2062318421 wait complete. 2024-04-21T23:17:49.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.012+0000 7f45643b7740 1 -- start start 2024-04-21T23:17:49.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.012+0000 7f45643b7740 1 Processor -- start 2024-04-21T23:17:49.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.012+0000 7f45643b7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd39873070 0x55dd39880200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:49.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.012+0000 7f45643b7740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd39873a60 0x55dd39880740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:49.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.012+0000 7f45643b7740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd39878bd0 0x55dd39883710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:49.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.012+0000 7f45643b7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55dd3988b2a0 con 0x55dd39873070 2024-04-21T23:17:49.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.012+0000 7f45643b7740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55dd3988abd0 con 0x55dd39873a60 2024-04-21T23:17:49.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.012+0000 7f45643b7740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55dd3988b970 con 0x55dd39878bd0 2024-04-21T23:17:49.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.012+0000 7f45643b7740 10 client.0 _ll_register_callbacks cb 0x7f4554013490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:17:49.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.012+0000 7f455affd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd39873070 0x55dd39880200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:49.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.012+0000 7f455affd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd39873070 0x55dd39880200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39098/0 (socket says 172.21.15.40:39098) 2024-04-21T23:17:49.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.012+0000 7f455affd640 1 -- 172.21.15.40:0/837481405 learned_addr learned my addr 172.21.15.40:0/837481405 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:17:49.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.012+0000 7f455affd640 1 -- 172.21.15.40:0/837481405 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd39878bd0 msgr2=0x55dd39883710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:17:49.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.012+0000 7f455affd640 1 --2- 172.21.15.40:0/837481405 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd39878bd0 0x55dd39883710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:49.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.012+0000 7f455affd640 1 -- 172.21.15.40:0/837481405 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd39873a60 msgr2=0x55dd39880740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:17:49.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.012+0000 7f455affd640 1 --2- 172.21.15.40:0/837481405 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd39873a60 0x55dd39880740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:49.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.012+0000 7f455affd640 1 -- 172.21.15.40:0/837481405 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd39883d00 con 0x55dd39873070 2024-04-21T23:17:49.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.012+0000 7f455affd640 1 --2- 172.21.15.40:0/837481405 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd39873070 0x55dd39880200 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f454c00b950 tx=0x7f454c0039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a1c199ead3c83de server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:49.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.012+0000 7f45427fc640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-21T23:17:49.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.013+0000 7f45427fc640 1 -- 172.21.15.40:0/837481405 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f454c01c880 con 0x55dd39873070 2024-04-21T23:17:49.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.013+0000 7f45427fc640 1 -- 172.21.15.40:0/837481405 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f454c00f420 con 0x55dd39873070 2024-04-21T23:17:49.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.013+0000 7f45643b7740 1 -- 172.21.15.40:0/837481405 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4554036070 con 0x55dd39873070 2024-04-21T23:17:49.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.013+0000 7f45427fc640 1 -- 172.21.15.40:0/837481405 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f454c02a050 con 0x55dd39873070 2024-04-21T23:17:49.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.013+0000 7f45643b7740 10 client.15261 fetch_fsmap learned FSMap version 24 2024-04-21T23:17:49.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.013+0000 7f45643b7740 1 -- 172.21.15.40:0/837481405 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55dd398f02e0 con 0x55dd39873070 2024-04-21T23:17:49.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.013+0000 7f45427fc640 1 -- 172.21.15.40:0/837481405 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4554036070 con 0x55dd39873070 2024-04-21T23:17:49.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.013+0000 7f45427fc640 1 -- 172.21.15.40:0/837481405 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f454c01b570 con 0x55dd39873070 2024-04-21T23:17:49.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.013+0000 7f45643b7740 10 client.15261 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T23:17:49.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.013+0000 7f45643b7740 10 client.15261 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-21T23:17:49.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.013+0000 7f45643b7740 20 client.15261 populate_metadata read hostname 'smithi040' 2024-04-21T23:17:49.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.013+0000 7f45643b7740 1 --2- 172.21.15.40:0/837481405 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55dd398f4fe0 0x55dd398f3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:49.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.013+0000 7f45643b7740 4 client.15261 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-21T23:17:49.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.013+0000 7f45643b7740 1 -- 172.21.15.40:0/837481405 --> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] -- command(tid 0: {"prefix": "status"}) -- 0x55dd3975da70 con 0x55dd398f4fe0 2024-04-21T23:17:49.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.014+0000 7f45527fc640 1 --2- 172.21.15.40:0/837481405 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55dd398f4fe0 0x55dd398f3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:49.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.014+0000 7f45527fc640 1 --2- 172.21.15.40:0/837481405 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55dd398f4fe0 0x55dd398f3cf0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=597d5ac283af261d server_cookie=8f27d52ec39d0af9 in_seq=0 out_seq=0 2024-04-21T23:17:49.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.014+0000 7f45427fc640 10 client.15261 ms_handle_connect on v2:172.21.15.111:6838/3712235548 2024-04-21T23:17:49.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.014+0000 7f45427fc640 1 -- 172.21.15.40:0/837481405 <== mds.0 v2:172.21.15.111:6838/3712235548 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55dd3975da70 con 0x55dd398f4fe0 2024-04-21T23:17:49.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.014+0000 7f45427fc640 10 client.15261 handle_command_reply: tid=0 2024-04-21T23:17:49.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.014+0000 7f45643b7740 1 client.15261 shutdown 2024-04-21T23:17:49.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.014+0000 7f45643b7740 1 -- 172.21.15.40:0/837481405 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd39873070 msgr2=0x55dd39880200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:49.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.014+0000 7f45643b7740 1 --2- 172.21.15.40:0/837481405 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd39873070 0x55dd39880200 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f454c00b950 tx=0x7f454c0039c0 comp rx=0 tx=0).stop 2024-04-21T23:17:49.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.014+0000 7f45643b7740 1 -- 172.21.15.40:0/837481405 shutdown_connections 2024-04-21T23:17:49.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.014+0000 7f45643b7740 1 --2- 172.21.15.40:0/837481405 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55dd398f4fe0 0x55dd398f3cf0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:49.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.014+0000 7f45643b7740 1 --2- 172.21.15.40:0/837481405 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd39873a60 0x55dd39880740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:49.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.014+0000 7f45643b7740 1 --2- 172.21.15.40:0/837481405 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd39878bd0 0x55dd39883710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:49.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.014+0000 7f45427fc640 0 client.15261 ms_handle_reset on v2:172.21.15.111:6838/3712235548 2024-04-21T23:17:49.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.014+0000 7f45643b7740 1 --2- 172.21.15.40:0/837481405 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd39873070 0x55dd39880200 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:49.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.014+0000 7f45643b7740 1 -- 172.21.15.40:0/837481405 >> 172.21.15.40:0/837481405 conn(0x55dd3986e5e0 msgr2=0x55dd39869340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:49.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.015+0000 7f45643b7740 1 -- 172.21.15.40:0/837481405 shutdown_connections 2024-04-21T23:17:49.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.015+0000 7f45643b7740 1 -- 172.21.15.40:0/837481405 wait complete. 2024-04-21T23:17:49.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.015+0000 7f45643b7740 20 client.15261 trim_cache size 0 max 16384 2024-04-21T23:17:49.020 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-21T23:17:49.020 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "d814e4dc-0033-11ef-bc93-c7b262605968", 2024-04-21T23:17:49.020 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 0, 2024-04-21T23:17:49.020 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 24521, 2024-04-21T23:17:49.020 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-04-21T23:17:49.020 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-04-21T23:17:49.020 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-04-21T23:17:49.021 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 112.63134630899999, 2024-04-21T23:17:49.021 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 24, 2024-04-21T23:17:49.021 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 92, 2024-04-21T23:17:49.021 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 90, 2024-04-21T23:17:49.021 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 119.022702272 2024-04-21T23:17:49.021 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-21T23:17:49.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.015+0000 7f45616fa640 1 -- 172.21.15.40:0/3571725266 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f45480a2c20 msgr2=0x7f45480a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:49.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.015+0000 7f45616fa640 1 --2- 172.21.15.40:0/3571725266 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f45480a2c20 0x7f45480a50e0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f4554005fd0 tx=0x7f45540098c0 comp rx=0 tx=0).stop 2024-04-21T23:17:49.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.015+0000 7f45616fa640 1 -- 172.21.15.40:0/3571725266 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f455c101890 msgr2=0x7f455c197f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:49.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.015+0000 7f45616fa640 1 --2- 172.21.15.40:0/3571725266 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f455c101890 0x7f455c197f50 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f4544009230 tx=0x7f4544005dd0 comp rx=0 tx=0).stop 2024-04-21T23:17:49.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.015+0000 7f45616fa640 1 -- 172.21.15.40:0/3571725266 shutdown_connections 2024-04-21T23:17:49.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.015+0000 7f45616fa640 1 --2- 172.21.15.40:0/3571725266 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f45480a2c20 0x7f45480a50e0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:49.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.015+0000 7f45616fa640 1 --2- 172.21.15.40:0/3571725266 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f455c102250 0x7f455c19efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:49.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.015+0000 7f45616fa640 1 --2- 172.21.15.40:0/3571725266 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f455c101890 0x7f455c197f50 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:49.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.015+0000 7f45616fa640 1 --2- 172.21.15.40:0/3571725266 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f455c1006a0 0x7f455c197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:49.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.015+0000 7f45616fa640 1 -- 172.21.15.40:0/3571725266 >> 172.21.15.40:0/3571725266 conn(0x7f455c0fbc30 msgr2=0x7f455c0fd690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:49.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.015+0000 7f45616fa640 1 -- 172.21.15.40:0/3571725266 shutdown_connections 2024-04-21T23:17:49.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:49.015+0000 7f45616fa640 1 -- 172.21.15.40:0/3571725266 wait complete. 2024-04-21T23:17:49.026 INFO:tasks.cephadm:Setup complete, yielding 2024-04-21T23:17:49.026 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-21T23:17:49.037 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi040.front.sepia.ceph.com 2024-04-21T23:17:49.037 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- bash -c 'ceph orch status' 2024-04-21T23:17:49.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:48 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/1975489254' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:17:49.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:48 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/1080525295' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:17:49.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:48 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/1769351160' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:17:49.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:48 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/1975489254' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:17:49.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:48 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/1080525295' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:17:49.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:48 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/1769351160' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:17:49.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:48 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/1975489254' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:17:49.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:48 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/1080525295' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:17:49.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:48 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/1769351160' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:17:49.283 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:17:49.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:49 smithi040 ceph-mon[27984]: pgmap v351: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:17:50.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:49 smithi174 ceph-mon[31409]: pgmap v351: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:17:50.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:49 smithi111 ceph-mon[31085]: pgmap v351: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:17:51.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:51.138+0000 7fccc5162700 1 -- 172.21.15.40:0/2224309885 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccc0103e30 msgr2=0x7fccc010a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:51.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:51.138+0000 7fccc5162700 1 --2- 172.21.15.40:0/2224309885 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccc0103e30 0x7fccc010a6e0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fccb800b0d0 tx=0x7fccb800b3e0 comp rx=0 tx=0).stop 2024-04-21T23:17:51.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:51.138+0000 7fccc5162700 1 -- 172.21.15.40:0/2224309885 shutdown_connections 2024-04-21T23:17:51.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:51.138+0000 7fccc5162700 1 --2- 172.21.15.40:0/2224309885 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fccc0102290 0x7fccc01026b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:51.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:51.138+0000 7fccc5162700 1 --2- 172.21.15.40:0/2224309885 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccc0103e30 0x7fccc010a6e0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:51.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:51.138+0000 7fccc5162700 1 --2- 172.21.15.40:0/2224309885 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fccc0103490 0x7fccc01038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:51.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:51.138+0000 7fccc5162700 1 -- 172.21.15.40:0/2224309885 >> 172.21.15.40:0/2224309885 conn(0x7fccc00fd830 msgr2=0x7fccc00ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:51.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:51.138+0000 7fccc5162700 1 -- 172.21.15.40:0/2224309885 shutdown_connections 2024-04-21T23:17:51.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:51.138+0000 7fccc5162700 1 -- 172.21.15.40:0/2224309885 wait complete. 2024-04-21T23:17:51.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:51.138+0000 7fccc5162700 1 Processor -- start 2024-04-21T23:17:51.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:51.138+0000 7fccc5162700 1 -- start start 2024-04-21T23:17:51.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:51.139+0000 7fccc5162700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fccc0102290 0x7fccc01983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:51.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:51.139+0000 7fccc5162700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccc0103490 0x7fccc0198930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:51.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:51.139+0000 7fccc5162700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fccc0103e30 0x7fccc019f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:51.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:51.139+0000 7fccc5162700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fccc010de70 con 0x7fccc0103490 2024-04-21T23:17:51.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:51.139+0000 7fccc5162700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fccc010dcf0 con 0x7fccc0103e30 2024-04-21T23:17:51.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:51.139+0000 7fccc5162700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fccc010dff0 con 0x7fccc0102290 2024-04-21T23:17:51.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:51.139+0000 7fccb7fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccc0103490 0x7fccc0198930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:51.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:51.139+0000 7fccb7fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccc0103490 0x7fccc0198930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39122/0 (socket says 172.21.15.40:39122) 2024-04-21T23:17:51.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:51.139+0000 7fccb7fff700 1 -- 172.21.15.40:0/3781652324 learned_addr learned my addr 172.21.15.40:0/3781652324 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:17:51.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:51.139+0000 7fccbed9d700 1 --2- 172.21.15.40:0/3781652324 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fccc0102290 0x7fccc01983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:51.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:51.139+0000 7fccbf59e700 1 --2- 172.21.15.40:0/3781652324 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fccc0103e30 0x7fccc019f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:51.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:51.139+0000 7fccb7fff700 1 -- 172.21.15.40:0/3781652324 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fccc0102290 msgr2=0x7fccc01983f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:51.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:51.139+0000 7fccb7fff700 1 --2- 172.21.15.40:0/3781652324 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fccc0102290 0x7fccc01983f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:51.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:51.139+0000 7fccb7fff700 1 -- 172.21.15.40:0/3781652324 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fccc0103e30 msgr2=0x7fccc019f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:51.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:51.139+0000 7fccb7fff700 1 --2- 172.21.15.40:0/3781652324 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fccc0103e30 0x7fccc019f9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:51.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:51.139+0000 7fccb7fff700 1 -- 172.21.15.40:0/3781652324 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccc01a0050 con 0x7fccc0103490 2024-04-21T23:17:51.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:51.139+0000 7fccb7fff700 1 --2- 172.21.15.40:0/3781652324 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccc0103490 0x7fccc0198930 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fcca800c950 tx=0x7fcca800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:51.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:51.139+0000 7fccbcd99700 1 -- 172.21.15.40:0/3781652324 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcca80078d0 con 0x7fccc0103490 2024-04-21T23:17:51.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:51.139+0000 7fccbcd99700 1 -- 172.21.15.40:0/3781652324 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcca800f450 con 0x7fccc0103490 2024-04-21T23:17:51.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:51.139+0000 7fccc5162700 1 -- 172.21.15.40:0/3781652324 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccc01a0340 con 0x7fccc0103490 2024-04-21T23:17:51.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:51.140+0000 7fccbcd99700 1 -- 172.21.15.40:0/3781652324 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcca8018660 con 0x7fccc0103490 2024-04-21T23:17:51.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:51.140+0000 7fccc5162700 1 -- 172.21.15.40:0/3781652324 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccc01a0880 con 0x7fccc0103490 2024-04-21T23:17:51.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:51.140+0000 7fccc5162700 1 -- 172.21.15.40:0/3781652324 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fccc0068f20 con 0x7fccc0103490 2024-04-21T23:17:51.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:51.142+0000 7fccbcd99700 1 -- 172.21.15.40:0/3781652324 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fcca800f600 con 0x7fccc0103490 2024-04-21T23:17:51.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:51.142+0000 7fccbcd99700 1 --2- 172.21.15.40:0/3781652324 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fccac0a2c20 0x7fccac0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:51.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:51.142+0000 7fccbcd99700 1 -- 172.21.15.40:0/3781652324 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fcca80cf380 con 0x7fccc0103490 2024-04-21T23:17:51.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:51.142+0000 7fccbed9d700 1 --2- 172.21.15.40:0/3781652324 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fccac0a2c20 0x7fccac0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:51.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:51.143+0000 7fccbed9d700 1 --2- 172.21.15.40:0/3781652324 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fccac0a2c20 0x7fccac0a50e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fccc01032f0 tx=0x7fccb000b410 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:51.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:51.146+0000 7fccbcd99700 1 -- 172.21.15.40:0/3781652324 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fcca8014030 con 0x7fccc0103490 2024-04-21T23:17:51.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:51.369+0000 7fccc5162700 1 -- 172.21.15.40:0/3781652324 --> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7fccc0062670 con 0x7fccac0a2c20 2024-04-21T23:17:51.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:51.371+0000 7fccbcd99700 1 -- 172.21.15.40:0/3781652324 <== mgr.14148 v2:172.21.15.40:6800/2500760788 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7fccc0062670 con 0x7fccac0a2c20 2024-04-21T23:17:51.370 INFO:teuthology.orchestra.run.smithi040.stdout:Backend: cephadm 2024-04-21T23:17:51.370 INFO:teuthology.orchestra.run.smithi040.stdout:Available: Yes 2024-04-21T23:17:51.370 INFO:teuthology.orchestra.run.smithi040.stdout:Paused: No 2024-04-21T23:17:51.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:51.373+0000 7fccc5162700 1 -- 172.21.15.40:0/3781652324 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fccac0a2c20 msgr2=0x7fccac0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:51.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:51.373+0000 7fccc5162700 1 --2- 172.21.15.40:0/3781652324 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fccac0a2c20 0x7fccac0a50e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fccc01032f0 tx=0x7fccb000b410 comp rx=0 tx=0).stop 2024-04-21T23:17:51.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:51.373+0000 7fccc5162700 1 -- 172.21.15.40:0/3781652324 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccc0103490 msgr2=0x7fccc0198930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:51.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:51.373+0000 7fccc5162700 1 --2- 172.21.15.40:0/3781652324 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccc0103490 0x7fccc0198930 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fcca800c950 tx=0x7fcca800cc60 comp rx=0 tx=0).stop 2024-04-21T23:17:51.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:51.373+0000 7fccc5162700 1 -- 172.21.15.40:0/3781652324 shutdown_connections 2024-04-21T23:17:51.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:51.373+0000 7fccc5162700 1 --2- 172.21.15.40:0/3781652324 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fccc0102290 0x7fccc01983f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:51.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:51.373+0000 7fccc5162700 1 --2- 172.21.15.40:0/3781652324 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fccac0a2c20 0x7fccac0a50e0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:51.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:51.373+0000 7fccc5162700 1 --2- 172.21.15.40:0/3781652324 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fccc0103e30 0x7fccc019f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:51.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:51.373+0000 7fccc5162700 1 --2- 172.21.15.40:0/3781652324 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccc0103490 0x7fccc0198930 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:51.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:51.373+0000 7fccc5162700 1 -- 172.21.15.40:0/3781652324 >> 172.21.15.40:0/3781652324 conn(0x7fccc00fd830 msgr2=0x7fccc0106790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:51.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:51.373+0000 7fccc5162700 1 -- 172.21.15.40:0/3781652324 shutdown_connections 2024-04-21T23:17:51.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:51.373+0000 7fccc5162700 1 -- 172.21.15.40:0/3781652324 wait complete. 2024-04-21T23:17:52.066 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:51 smithi040 ceph-mon[27984]: pgmap v352: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:17:52.066 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:51 smithi040 ceph-mon[27984]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-04-21T23:17:52.066 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:51 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [5, 8]}]: dispatch 2024-04-21T23:17:52.066 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:51 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]: dispatch 2024-04-21T23:17:52.067 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:51 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 10]}]: dispatch 2024-04-21T23:17:52.067 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:51 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]}]: dispatch 2024-04-21T23:17:52.186 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ps' 2024-04-21T23:17:52.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:51 smithi174 ceph-mon[31409]: pgmap v352: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:17:52.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:51 smithi174 ceph-mon[31409]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-04-21T23:17:52.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:51 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [5, 8]}]: dispatch 2024-04-21T23:17:52.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:51 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]: dispatch 2024-04-21T23:17:52.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:51 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 10]}]: dispatch 2024-04-21T23:17:52.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:51 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]}]: dispatch 2024-04-21T23:17:52.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:51 smithi111 ceph-mon[31085]: pgmap v352: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:17:52.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:51 smithi111 ceph-mon[31085]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-04-21T23:17:52.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:51 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [5, 8]}]: dispatch 2024-04-21T23:17:52.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:51 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]: dispatch 2024-04-21T23:17:52.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:51 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 10]}]: dispatch 2024-04-21T23:17:52.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:51 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]}]: dispatch 2024-04-21T23:17:52.436 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:17:53.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:52 smithi040 ceph-mon[27984]: from='client.15267 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:17:53.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:52 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [5, 8]}]': finished 2024-04-21T23:17:53.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:52 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]': finished 2024-04-21T23:17:53.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:52 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 10]}]': finished 2024-04-21T23:17:53.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:52 smithi040 ceph-mon[27984]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]}]': finished 2024-04-21T23:17:53.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:52 smithi040 ceph-mon[27984]: osdmap e93: 12 total, 12 up, 12 in 2024-04-21T23:17:53.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:52 smithi174 ceph-mon[31409]: from='client.15267 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:17:53.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:52 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [5, 8]}]': finished 2024-04-21T23:17:53.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:52 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]': finished 2024-04-21T23:17:53.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:52 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 10]}]': finished 2024-04-21T23:17:53.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:52 smithi174 ceph-mon[31409]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]}]': finished 2024-04-21T23:17:53.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:52 smithi174 ceph-mon[31409]: osdmap e93: 12 total, 12 up, 12 in 2024-04-21T23:17:53.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:52 smithi111 ceph-mon[31085]: from='client.15267 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:17:53.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:52 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [5, 8]}]': finished 2024-04-21T23:17:53.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:52 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]': finished 2024-04-21T23:17:53.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:52 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 10]}]': finished 2024-04-21T23:17:53.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:52 smithi111 ceph-mon[31085]: from='mgr.14148 172.21.15.40:0/2347392262' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]}]': finished 2024-04-21T23:17:53.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:52 smithi111 ceph-mon[31085]: osdmap e93: 12 total, 12 up, 12 in 2024-04-21T23:17:54.175 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:53 smithi040 ceph-mon[27984]: pgmap v354: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-21T23:17:54.176 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:53 smithi040 ceph-mon[27984]: osdmap e94: 12 total, 12 up, 12 in 2024-04-21T23:17:54.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:53 smithi174 ceph-mon[31409]: pgmap v354: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-21T23:17:54.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:53 smithi174 ceph-mon[31409]: osdmap e94: 12 total, 12 up, 12 in 2024-04-21T23:17:54.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:53 smithi111 ceph-mon[31085]: pgmap v354: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-21T23:17:54.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:53 smithi111 ceph-mon[31085]: osdmap e94: 12 total, 12 up, 12 in 2024-04-21T23:17:54.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:54.437+0000 7feea58fb700 1 -- 172.21.15.40:0/3113566070 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feea0103e50 msgr2=0x7feea010a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:54.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:54.437+0000 7feea58fb700 1 --2- 172.21.15.40:0/3113566070 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feea0103e50 0x7feea010a700 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fee9800b0d0 tx=0x7fee9800b3e0 comp rx=0 tx=0).stop 2024-04-21T23:17:54.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:54.437+0000 7feea58fb700 1 -- 172.21.15.40:0/3113566070 shutdown_connections 2024-04-21T23:17:54.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:54.437+0000 7feea58fb700 1 --2- 172.21.15.40:0/3113566070 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feea0103e50 0x7feea010a700 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:54.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:54.437+0000 7feea58fb700 1 --2- 172.21.15.40:0/3113566070 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feea01022b0 0x7feea01026d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:54.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:54.437+0000 7feea58fb700 1 --2- 172.21.15.40:0/3113566070 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feea01034b0 0x7feea0103910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:54.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:54.437+0000 7feea58fb700 1 -- 172.21.15.40:0/3113566070 >> 172.21.15.40:0/3113566070 conn(0x7feea00fd850 msgr2=0x7feea00ffc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:54.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:54.437+0000 7feea58fb700 1 -- 172.21.15.40:0/3113566070 shutdown_connections 2024-04-21T23:17:54.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:54.437+0000 7feea58fb700 1 -- 172.21.15.40:0/3113566070 wait complete. 2024-04-21T23:17:54.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:54.438+0000 7feea58fb700 1 Processor -- start 2024-04-21T23:17:54.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:54.438+0000 7feea58fb700 1 -- start start 2024-04-21T23:17:54.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:54.438+0000 7feea58fb700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feea01022b0 0x7feea01983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:54.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:54.438+0000 7feea58fb700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feea01034b0 0x7feea0198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:54.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:54.438+0000 7feea58fb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feea0103e50 0x7feea019f990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:54.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:54.438+0000 7feea58fb700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7feea010de70 con 0x7feea01022b0 2024-04-21T23:17:54.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:54.438+0000 7feea58fb700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7feea010dcf0 con 0x7feea0103e50 2024-04-21T23:17:54.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:54.438+0000 7feea58fb700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7feea010dff0 con 0x7feea01034b0 2024-04-21T23:17:54.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:54.438+0000 7fee9effd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feea01022b0 0x7feea01983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:54.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:54.438+0000 7fee9effd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feea01022b0 0x7feea01983d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35260/0 (socket says 172.21.15.40:35260) 2024-04-21T23:17:54.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:54.438+0000 7fee9effd700 1 -- 172.21.15.40:0/3608764666 learned_addr learned my addr 172.21.15.40:0/3608764666 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:17:54.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:54.438+0000 7fee9e7fc700 1 --2- 172.21.15.40:0/3608764666 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feea01034b0 0x7feea0198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:54.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:54.438+0000 7fee9f7fe700 1 --2- 172.21.15.40:0/3608764666 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feea0103e50 0x7feea019f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:54.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:54.439+0000 7fee9effd700 1 -- 172.21.15.40:0/3608764666 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feea01034b0 msgr2=0x7feea0198910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:54.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:54.439+0000 7fee9effd700 1 --2- 172.21.15.40:0/3608764666 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feea01034b0 0x7feea0198910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:54.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:54.439+0000 7fee9effd700 1 -- 172.21.15.40:0/3608764666 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feea0103e50 msgr2=0x7feea019f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:54.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:54.439+0000 7fee9effd700 1 --2- 172.21.15.40:0/3608764666 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feea0103e50 0x7feea019f990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:54.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:54.439+0000 7fee9effd700 1 -- 172.21.15.40:0/3608764666 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feea01a0030 con 0x7feea01022b0 2024-04-21T23:17:54.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:54.439+0000 7fee9f7fe700 1 --2- 172.21.15.40:0/3608764666 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feea0103e50 0x7feea019f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:17:54.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:54.439+0000 7fee9effd700 1 --2- 172.21.15.40:0/3608764666 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feea01022b0 0x7feea01983d0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fee9000d9b0 tx=0x7fee9000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:54.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:54.439+0000 7feea48f9700 1 -- 172.21.15.40:0/3608764666 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee90017030 con 0x7feea01022b0 2024-04-21T23:17:54.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:54.439+0000 7feea58fb700 1 -- 172.21.15.40:0/3608764666 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feea01a0320 con 0x7feea01022b0 2024-04-21T23:17:54.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:54.439+0000 7feea48f9700 1 -- 172.21.15.40:0/3608764666 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee90004500 con 0x7feea01022b0 2024-04-21T23:17:54.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:54.439+0000 7feea48f9700 1 -- 172.21.15.40:0/3608764666 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee90003aa0 con 0x7feea01022b0 2024-04-21T23:17:54.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:54.439+0000 7feea58fb700 1 -- 172.21.15.40:0/3608764666 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feea01a08b0 con 0x7feea01022b0 2024-04-21T23:17:54.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:54.440+0000 7feea58fb700 1 -- 172.21.15.40:0/3608764666 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feea0068f20 con 0x7feea01022b0 2024-04-21T23:17:54.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:54.441+0000 7feea48f9700 1 -- 172.21.15.40:0/3608764666 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fee9001a070 con 0x7feea01022b0 2024-04-21T23:17:54.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:54.442+0000 7feea48f9700 1 --2- 172.21.15.40:0/3608764666 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fee880a2c20 0x7fee880a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:54.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:54.442+0000 7fee9e7fc700 1 --2- 172.21.15.40:0/3608764666 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fee880a2c20 0x7fee880a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:54.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:54.442+0000 7feea48f9700 1 -- 172.21.15.40:0/3608764666 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fee900ced70 con 0x7feea01022b0 2024-04-21T23:17:54.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:54.443+0000 7fee9e7fc700 1 --2- 172.21.15.40:0/3608764666 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fee880a2c20 0x7fee880a50e0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fee94005950 tx=0x7fee9400b410 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:54.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:54.447+0000 7feea48f9700 1 -- 172.21.15.40:0/3608764666 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fee900232f0 con 0x7feea01022b0 2024-04-21T23:17:54.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:54.662+0000 7feea58fb700 1 -- 172.21.15.40:0/3608764666 --> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7feea0062670 con 0x7fee880a2c20 2024-04-21T23:17:54.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:54.675+0000 7feea48f9700 1 -- 172.21.15.40:0/3608764666 <== mgr.14148 v2:172.21.15.40:6800/2500760788 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4092 (secure 0 0 0) 0x7feea0062670 con 0x7fee880a2c20 2024-04-21T23:17:54.675 INFO:teuthology.orchestra.run.smithi040.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-21T23:17:54.675 INFO:teuthology.orchestra.run.smithi040.stdout:mds.a smithi040 running (2m) 19s ago 2m 15.3M - 19.0.0-2405-g820a2f8c 6702148920ad dfb13ba672d1 2024-04-21T23:17:54.675 INFO:teuthology.orchestra.run.smithi040.stdout:mds.b smithi111 running (2m) 58s ago 2m 25.9M - 19.0.0-2405-g820a2f8c 6702148920ad 7eaf129c2e3c 2024-04-21T23:17:54.675 INFO:teuthology.orchestra.run.smithi040.stdout:mds.c smithi174 running (2m) 58s ago 2m 17.0M - 19.0.0-2405-g820a2f8c 6702148920ad 92e760e59bf0 2024-04-21T23:17:54.675 INFO:teuthology.orchestra.run.smithi040.stdout:mds.d smithi040 running (2m) 19s ago 2m 16.4M - 19.0.0-2405-g820a2f8c 6702148920ad 968c7523f37c 2024-04-21T23:17:54.675 INFO:teuthology.orchestra.run.smithi040.stdout:mds.e smithi111 running (2m) 58s ago 2m 15.0M - 19.0.0-2405-g820a2f8c 6702148920ad e9670c79289b 2024-04-21T23:17:54.675 INFO:teuthology.orchestra.run.smithi040.stdout:mds.f smithi174 running (2m) 58s ago 2m 15.9M - 19.0.0-2405-g820a2f8c 6702148920ad b1cc187a796d 2024-04-21T23:17:54.675 INFO:teuthology.orchestra.run.smithi040.stdout:mds.g smithi040 running (2m) 19s ago 2m 13.9M - 19.0.0-2405-g820a2f8c 6702148920ad 780ba2a634dc 2024-04-21T23:17:54.675 INFO:teuthology.orchestra.run.smithi040.stdout:mds.h smithi111 running (2m) 58s ago 2m 18.4M - 19.0.0-2405-g820a2f8c 6702148920ad c4271d3bd94d 2024-04-21T23:17:54.676 INFO:teuthology.orchestra.run.smithi040.stdout:mds.i smithi174 running (2m) 58s ago 2m 16.2M - 19.0.0-2405-g820a2f8c 6702148920ad 27bf130082a0 2024-04-21T23:17:54.676 INFO:teuthology.orchestra.run.smithi040.stdout:mds.j smithi040 running (2m) 19s ago 2m 14.1M - 19.0.0-2405-g820a2f8c 6702148920ad bf807801f427 2024-04-21T23:17:54.676 INFO:teuthology.orchestra.run.smithi040.stdout:mds.k smithi111 running (2m) 58s ago 2m 14.0M - 19.0.0-2405-g820a2f8c 6702148920ad 2fd4155c0bf0 2024-04-21T23:17:54.676 INFO:teuthology.orchestra.run.smithi040.stdout:mds.l smithi174 running (2m) 58s ago 2m 16.4M - 19.0.0-2405-g820a2f8c 6702148920ad 816c950d8d17 2024-04-21T23:17:54.676 INFO:teuthology.orchestra.run.smithi040.stdout:mgr.x smithi040 *:9283,8765 running (9m) 19s ago 9m 514M - 19.0.0-2405-g820a2f8c 6702148920ad 7b8d28516472 2024-04-21T23:17:54.676 INFO:teuthology.orchestra.run.smithi040.stdout:mgr.y smithi111 *:8443,8765 running (7m) 58s ago 7m 454M - 19.0.0-2405-g820a2f8c 6702148920ad 8f392e362b15 2024-04-21T23:17:54.676 INFO:teuthology.orchestra.run.smithi040.stdout:mgr.z smithi174 *:8443,8765 running (7m) 58s ago 7m 453M - 19.0.0-2405-g820a2f8c 6702148920ad a7eda681942d 2024-04-21T23:17:54.676 INFO:teuthology.orchestra.run.smithi040.stdout:mon.a smithi040 running (9m) 19s ago 10m 50.7M 2048M 19.0.0-2405-g820a2f8c 6702148920ad 5fb73014130c 2024-04-21T23:17:54.676 INFO:teuthology.orchestra.run.smithi040.stdout:mon.b smithi111 running (8m) 58s ago 8m 39.6M 2048M 19.0.0-2405-g820a2f8c 6702148920ad 7f9ded098550 2024-04-21T23:17:54.676 INFO:teuthology.orchestra.run.smithi040.stdout:mon.c smithi174 running (8m) 58s ago 8m 39.2M 2048M 19.0.0-2405-g820a2f8c 6702148920ad 2500b95b86b5 2024-04-21T23:17:54.676 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0 smithi040 running (7m) 19s ago 7m 48.1M 4273M 19.0.0-2405-g820a2f8c 6702148920ad bb691923bf7a 2024-04-21T23:17:54.677 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1 smithi111 running (7m) 58s ago 7m 50.3M 4272M 19.0.0-2405-g820a2f8c 6702148920ad 6838042f57c3 2024-04-21T23:17:54.677 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2 smithi174 running (6m) 58s ago 6m 49.6M 4272M 19.0.0-2405-g820a2f8c 6702148920ad f08f14961388 2024-04-21T23:17:54.677 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3 smithi040 running (6m) 19s ago 6m 52.9M 4273M 19.0.0-2405-g820a2f8c 6702148920ad 095368d70c2f 2024-04-21T23:17:54.677 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4 smithi111 running (5m) 58s ago 5m 42.7M 4272M 19.0.0-2405-g820a2f8c 6702148920ad 214e5d90d6d2 2024-04-21T23:17:54.677 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5 smithi174 running (5m) 58s ago 5m 52.0M 4272M 19.0.0-2405-g820a2f8c 6702148920ad 1752207a0d70 2024-04-21T23:17:54.677 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6 smithi040 running (5m) 19s ago 5m 41.5M 4273M 19.0.0-2405-g820a2f8c 6702148920ad 67014dd8f226 2024-04-21T23:17:54.677 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7 smithi111 running (4m) 58s ago 4m 44.8M 4272M 19.0.0-2405-g820a2f8c 6702148920ad 38069fd11dfd 2024-04-21T23:17:54.677 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8 smithi174 running (4m) 58s ago 4m 44.0M 4272M 19.0.0-2405-g820a2f8c 6702148920ad c129400dfb96 2024-04-21T23:17:54.677 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9 smithi040 running (3m) 19s ago 3m 50.7M 4273M 19.0.0-2405-g820a2f8c 6702148920ad 556ebc8fe58e 2024-04-21T23:17:54.677 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10 smithi111 running (3m) 58s ago 3m 50.4M 4272M 19.0.0-2405-g820a2f8c 6702148920ad 177f8972b44f 2024-04-21T23:17:54.677 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11 smithi174 running (2m) 58s ago 2m 41.4M 4272M 19.0.0-2405-g820a2f8c 6702148920ad 3fda8950fec4 2024-04-21T23:17:54.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:54.678+0000 7feea58fb700 1 -- 172.21.15.40:0/3608764666 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fee880a2c20 msgr2=0x7fee880a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:54.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:54.678+0000 7feea58fb700 1 --2- 172.21.15.40:0/3608764666 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fee880a2c20 0x7fee880a50e0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fee94005950 tx=0x7fee9400b410 comp rx=0 tx=0).stop 2024-04-21T23:17:54.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:54.678+0000 7feea58fb700 1 -- 172.21.15.40:0/3608764666 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feea01022b0 msgr2=0x7feea01983d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:54.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:54.678+0000 7feea58fb700 1 --2- 172.21.15.40:0/3608764666 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feea01022b0 0x7feea01983d0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fee9000d9b0 tx=0x7fee9000dcc0 comp rx=0 tx=0).stop 2024-04-21T23:17:54.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:54.678+0000 7feea58fb700 1 -- 172.21.15.40:0/3608764666 shutdown_connections 2024-04-21T23:17:54.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:54.678+0000 7feea58fb700 1 --2- 172.21.15.40:0/3608764666 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fee880a2c20 0x7fee880a50e0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:54.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:54.678+0000 7feea58fb700 1 --2- 172.21.15.40:0/3608764666 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feea0103e50 0x7feea019f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:54.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:54.678+0000 7feea58fb700 1 --2- 172.21.15.40:0/3608764666 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feea01022b0 0x7feea01983d0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:54.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:54.678+0000 7feea58fb700 1 --2- 172.21.15.40:0/3608764666 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feea01034b0 0x7feea0198910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:54.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:54.678+0000 7feea58fb700 1 -- 172.21.15.40:0/3608764666 >> 172.21.15.40:0/3608764666 conn(0x7feea00fd850 msgr2=0x7feea01066e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:54.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:54.678+0000 7feea58fb700 1 -- 172.21.15.40:0/3608764666 shutdown_connections 2024-04-21T23:17:54.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:54.678+0000 7feea58fb700 1 -- 172.21.15.40:0/3608764666 wait complete. 2024-04-21T23:17:55.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:54 smithi040 ceph-mon[27984]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-04-21T23:17:55.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:54 smithi174 ceph-mon[31409]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-04-21T23:17:55.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:54 smithi111 ceph-mon[31085]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-04-21T23:17:55.528 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ls' 2024-04-21T23:17:55.784 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:17:56.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:55 smithi040 ceph-mon[27984]: pgmap v356: 129 pgs: 2 peering, 127 active+clean; 641 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-21T23:17:56.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:55 smithi040 ceph-mon[27984]: from='client.15273 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:17:56.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:55 smithi174 ceph-mon[31409]: pgmap v356: 129 pgs: 2 peering, 127 active+clean; 641 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-21T23:17:56.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:55 smithi174 ceph-mon[31409]: from='client.15273 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:17:56.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:55 smithi111 ceph-mon[31085]: pgmap v356: 129 pgs: 2 peering, 127 active+clean; 641 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-21T23:17:56.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:55 smithi111 ceph-mon[31085]: from='client.15273 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:17:57.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:57.533+0000 7f973dd87700 1 -- 172.21.15.40:0/1128857814 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9738107f80 msgr2=0x7f973810a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:57.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:57.533+0000 7f973dd87700 1 --2- 172.21.15.40:0/1128857814 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9738107f80 0x7f973810a3f0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f973000b0a0 tx=0x7f973000b3b0 comp rx=0 tx=0).stop 2024-04-21T23:17:57.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:57.533+0000 7f973dd87700 1 -- 172.21.15.40:0/1128857814 shutdown_connections 2024-04-21T23:17:57.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:57.533+0000 7f973dd87700 1 --2- 172.21.15.40:0/1128857814 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9738102d20 0x7f9738105110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:57.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:57.533+0000 7f973dd87700 1 --2- 172.21.15.40:0/1128857814 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9738107f80 0x7f973810a3f0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:57.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:57.533+0000 7f973dd87700 1 --2- 172.21.15.40:0/1128857814 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9738105650 0x7f9738107a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:57.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:57.533+0000 7f973dd87700 1 -- 172.21.15.40:0/1128857814 >> 172.21.15.40:0/1128857814 conn(0x7f97380fc730 msgr2=0x7f97380feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:57.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:57.533+0000 7f973dd87700 1 -- 172.21.15.40:0/1128857814 shutdown_connections 2024-04-21T23:17:57.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:57.533+0000 7f973dd87700 1 -- 172.21.15.40:0/1128857814 wait complete. 2024-04-21T23:17:57.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:57.533+0000 7f973dd87700 1 Processor -- start 2024-04-21T23:17:57.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:57.534+0000 7f973dd87700 1 -- start start 2024-04-21T23:17:57.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:57.534+0000 7f973dd87700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9738102d20 0x7f973819a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:57.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:57.534+0000 7f973dd87700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9738105650 0x7f973819ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:57.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:57.534+0000 7f973dd87700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9738107f80 0x7f97381a1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:57.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:57.534+0000 7f973dd87700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f973810de50 con 0x7f9738107f80 2024-04-21T23:17:57.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:57.534+0000 7f973dd87700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f973810dcd0 con 0x7f9738105650 2024-04-21T23:17:57.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:57.534+0000 7f973dd87700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f973810dfd0 con 0x7f9738102d20 2024-04-21T23:17:57.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:57.534+0000 7f9737fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9738107f80 0x7f97381a1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:57.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:57.534+0000 7f9737fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9738107f80 0x7f97381a1bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35276/0 (socket says 172.21.15.40:35276) 2024-04-21T23:17:57.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:57.534+0000 7f9737fff700 1 -- 172.21.15.40:0/1607692689 learned_addr learned my addr 172.21.15.40:0/1607692689 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:17:57.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:57.534+0000 7f9736ffd700 1 --2- 172.21.15.40:0/1607692689 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9738105650 0x7f973819ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:57.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:57.534+0000 7f97377fe700 1 --2- 172.21.15.40:0/1607692689 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9738102d20 0x7f973819a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:57.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:57.534+0000 7f9737fff700 1 -- 172.21.15.40:0/1607692689 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9738102d20 msgr2=0x7f973819a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:57.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:57.534+0000 7f9737fff700 1 --2- 172.21.15.40:0/1607692689 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9738102d20 0x7f973819a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:57.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:57.534+0000 7f9737fff700 1 -- 172.21.15.40:0/1607692689 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9738105650 msgr2=0x7f973819ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:57.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:57.534+0000 7f9737fff700 1 --2- 172.21.15.40:0/1607692689 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9738105650 0x7f973819ab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:57.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:57.534+0000 7f9737fff700 1 -- 172.21.15.40:0/1607692689 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97381a2270 con 0x7f9738107f80 2024-04-21T23:17:57.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:57.535+0000 7f9737fff700 1 --2- 172.21.15.40:0/1607692689 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9738107f80 0x7f97381a1bd0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f973000d010 tx=0x7f9730004dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:57.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:57.535+0000 7f9734ff9700 1 -- 172.21.15.40:0/1607692689 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9730021070 con 0x7f9738107f80 2024-04-21T23:17:57.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:57.535+0000 7f973dd87700 1 -- 172.21.15.40:0/1607692689 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97381a2500 con 0x7f9738107f80 2024-04-21T23:17:57.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:57.535+0000 7f9734ff9700 1 -- 172.21.15.40:0/1607692689 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9730007b80 con 0x7f9738107f80 2024-04-21T23:17:57.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:57.535+0000 7f9734ff9700 1 -- 172.21.15.40:0/1607692689 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f973001d450 con 0x7f9738107f80 2024-04-21T23:17:57.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:57.535+0000 7f973dd87700 1 -- 172.21.15.40:0/1607692689 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97381a2a00 con 0x7f9738107f80 2024-04-21T23:17:57.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:57.536+0000 7f973dd87700 1 -- 172.21.15.40:0/1607692689 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9738068f20 con 0x7f9738107f80 2024-04-21T23:17:57.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:57.537+0000 7f9734ff9700 1 -- 172.21.15.40:0/1607692689 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f9730004000 con 0x7f9738107f80 2024-04-21T23:17:57.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:57.538+0000 7f9734ff9700 1 --2- 172.21.15.40:0/1607692689 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f97200a2c20 0x7f97200a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:57.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:57.538+0000 7f97377fe700 1 --2- 172.21.15.40:0/1607692689 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f97200a2c20 0x7f97200a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:57.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:57.538+0000 7f9734ff9700 1 -- 172.21.15.40:0/1607692689 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f973000f020 con 0x7f9738107f80 2024-04-21T23:17:57.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:57.538+0000 7f97377fe700 1 --2- 172.21.15.40:0/1607692689 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f97200a2c20 0x7f97200a50e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f972c00ac50 tx=0x7f972c00a380 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:57.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:57.542+0000 7f9734ff9700 1 -- 172.21.15.40:0/1607692689 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f973002a020 con 0x7f9738107f80 2024-04-21T23:17:57.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:57.759+0000 7f973dd87700 1 -- 172.21.15.40:0/1607692689 --> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7f9738062670 con 0x7f97200a2c20 2024-04-21T23:17:57.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:57.766+0000 7f9734ff9700 1 -- 172.21.15.40:0/1607692689 <== mgr.14148 v2:172.21.15.40:6800/2500760788 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7f9738062670 con 0x7f97200a2c20 2024-04-21T23:17:57.765 INFO:teuthology.orchestra.run.smithi040.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-21T23:17:57.765 INFO:teuthology.orchestra.run.smithi040.stdout:mds.all 12/12 61s ago 2m smithi040=a;smithi040=d;smithi040=g;smithi040=j;smithi111=b;smithi111=e;smithi111=h;smithi111=k;smithi174=c;smithi174=f;smithi174=i;smithi174=l;count:12 2024-04-21T23:17:57.765 INFO:teuthology.orchestra.run.smithi040.stdout:mgr 3/3 61s ago 8m smithi040=x;smithi111=y;smithi174=z;count:3 2024-04-21T23:17:57.765 INFO:teuthology.orchestra.run.smithi040.stdout:mon 3/3 61s ago 8m smithi040:172.21.15.40=a;smithi111:172.21.15.111=b;smithi174:172.21.15.174=c;count:3 2024-04-21T23:17:57.765 INFO:teuthology.orchestra.run.smithi040.stdout:osd 12 61s ago - 2024-04-21T23:17:57.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:57.768+0000 7f973dd87700 1 -- 172.21.15.40:0/1607692689 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f97200a2c20 msgr2=0x7f97200a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:57.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:57.768+0000 7f973dd87700 1 --2- 172.21.15.40:0/1607692689 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f97200a2c20 0x7f97200a50e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f972c00ac50 tx=0x7f972c00a380 comp rx=0 tx=0).stop 2024-04-21T23:17:57.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:57.768+0000 7f973dd87700 1 -- 172.21.15.40:0/1607692689 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9738107f80 msgr2=0x7f97381a1bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:57.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:57.768+0000 7f973dd87700 1 --2- 172.21.15.40:0/1607692689 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9738107f80 0x7f97381a1bd0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f973000d010 tx=0x7f9730004dd0 comp rx=0 tx=0).stop 2024-04-21T23:17:57.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:57.768+0000 7f973dd87700 1 -- 172.21.15.40:0/1607692689 shutdown_connections 2024-04-21T23:17:57.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:57.768+0000 7f973dd87700 1 --2- 172.21.15.40:0/1607692689 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9738102d20 0x7f973819a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:57.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:57.768+0000 7f973dd87700 1 --2- 172.21.15.40:0/1607692689 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f97200a2c20 0x7f97200a50e0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:57.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:57.768+0000 7f973dd87700 1 --2- 172.21.15.40:0/1607692689 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9738107f80 0x7f97381a1bd0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:57.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:57.768+0000 7f973dd87700 1 --2- 172.21.15.40:0/1607692689 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9738105650 0x7f973819ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:57.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:57.768+0000 7f973dd87700 1 -- 172.21.15.40:0/1607692689 >> 172.21.15.40:0/1607692689 conn(0x7f97380fc730 msgr2=0x7f97381023f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:57.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:57.768+0000 7f973dd87700 1 -- 172.21.15.40:0/1607692689 shutdown_connections 2024-04-21T23:17:57.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:17:57.768+0000 7f973dd87700 1 -- 172.21.15.40:0/1607692689 wait complete. 2024-04-21T23:17:58.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:57 smithi040 ceph-mon[27984]: pgmap v357: 129 pgs: 2 peering, 127 active+clean; 641 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 639 B/s wr, 8 op/s 2024-04-21T23:17:58.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:57 smithi174 ceph-mon[31409]: pgmap v357: 129 pgs: 2 peering, 127 active+clean; 641 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 639 B/s wr, 8 op/s 2024-04-21T23:17:58.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:57 smithi111 ceph-mon[31085]: pgmap v357: 129 pgs: 2 peering, 127 active+clean; 641 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 639 B/s wr, 8 op/s 2024-04-21T23:17:58.474 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- bash -c 'ceph orch host ls' 2024-04-21T23:17:58.725 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:17:59.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:58 smithi040 ceph-mon[27984]: from='client.15279 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:17:59.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:58 smithi174 ceph-mon[31409]: from='client.15279 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:17:59.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:58 smithi111 ceph-mon[31085]: from='client.15279 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:18:00.162 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:59 smithi040 ceph-mon[27984]: pgmap v358: 129 pgs: 129 active+clean; 641 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 639 B/s wr, 5 op/s 2024-04-21T23:18:00.162 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:59 smithi040 ceph-mon[27984]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-04-21T23:18:00.163 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:17:59 smithi040 ceph-mon[27984]: Cluster is now healthy 2024-04-21T23:18:00.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:00.404+0000 7f8a4c8ac700 1 -- 172.21.15.40:0/162717023 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a48101190 msgr2=0x7f8a481015b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:00.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:00.404+0000 7f8a4c8ac700 1 --2- 172.21.15.40:0/162717023 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a48101190 0x7f8a481015b0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c0099c0 tx=0x7f8a3c009cd0 comp rx=0 tx=0).stop 2024-04-21T23:18:00.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:00.404+0000 7f8a4c8ac700 1 -- 172.21.15.40:0/162717023 shutdown_connections 2024-04-21T23:18:00.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:00.404+0000 7f8a4c8ac700 1 --2- 172.21.15.40:0/162717023 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a48102d30 0x7f8a481095e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:00.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:00.404+0000 7f8a4c8ac700 1 --2- 172.21.15.40:0/162717023 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a48101190 0x7f8a481015b0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:00.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:00.404+0000 7f8a4c8ac700 1 --2- 172.21.15.40:0/162717023 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a48102390 0x7f8a481027f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:00.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:00.404+0000 7f8a4c8ac700 1 -- 172.21.15.40:0/162717023 >> 172.21.15.40:0/162717023 conn(0x7f8a480fc750 msgr2=0x7f8a480feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:00.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:00.404+0000 7f8a4c8ac700 1 -- 172.21.15.40:0/162717023 shutdown_connections 2024-04-21T23:18:00.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:00.404+0000 7f8a4c8ac700 1 -- 172.21.15.40:0/162717023 wait complete. 2024-04-21T23:18:00.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:00.405+0000 7f8a4c8ac700 1 Processor -- start 2024-04-21T23:18:00.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:00.405+0000 7f8a4c8ac700 1 -- start start 2024-04-21T23:18:00.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:00.405+0000 7f8a4c8ac700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a48101190 0x7f8a4806b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:00.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:00.405+0000 7f8a4c8ac700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a48102390 0x7f8a4819f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:00.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:00.405+0000 7f8a4c8ac700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a48102d30 0x7f8a481a1b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:00.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:00.405+0000 7f8a4c8ac700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8a4810cd70 con 0x7f8a48102d30 2024-04-21T23:18:00.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:00.405+0000 7f8a4c8ac700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f8a4810cbf0 con 0x7f8a48102390 2024-04-21T23:18:00.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:00.405+0000 7f8a4c8ac700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8a4810cef0 con 0x7f8a48101190 2024-04-21T23:18:00.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:00.405+0000 7f8a46d9d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a48102d30 0x7f8a481a1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:00.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:00.405+0000 7f8a46d9d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a48102d30 0x7f8a481a1b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35294/0 (socket says 172.21.15.40:35294) 2024-04-21T23:18:00.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:00.405+0000 7f8a46d9d700 1 -- 172.21.15.40:0/2772133304 learned_addr learned my addr 172.21.15.40:0/2772133304 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:00.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:00.405+0000 7f8a4659c700 1 --2- 172.21.15.40:0/2772133304 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a48101190 0x7f8a4806b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:00.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:00.405+0000 7f8a45d9b700 1 --2- 172.21.15.40:0/2772133304 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a48102390 0x7f8a4819f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:00.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:00.406+0000 7f8a46d9d700 1 -- 172.21.15.40:0/2772133304 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a48101190 msgr2=0x7f8a4806b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:00.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:00.406+0000 7f8a46d9d700 1 --2- 172.21.15.40:0/2772133304 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a48101190 0x7f8a4806b460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:00.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:00.406+0000 7f8a46d9d700 1 -- 172.21.15.40:0/2772133304 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a48102390 msgr2=0x7f8a4819f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:00.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:00.406+0000 7f8a46d9d700 1 --2- 172.21.15.40:0/2772133304 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a48102390 0x7f8a4819f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:00.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:00.406+0000 7f8a46d9d700 1 -- 172.21.15.40:0/2772133304 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a481a2340 con 0x7f8a48102d30 2024-04-21T23:18:00.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:00.406+0000 7f8a46d9d700 1 --2- 172.21.15.40:0/2772133304 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a48102d30 0x7f8a481a1b60 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f8a40007ee0 tx=0x7f8a4000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:00.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:00.406+0000 7f8a377fe700 1 -- 172.21.15.40:0/2772133304 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a4000dbd0 con 0x7f8a48102d30 2024-04-21T23:18:00.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:00.406+0000 7f8a4c8ac700 1 -- 172.21.15.40:0/2772133304 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a481a2630 con 0x7f8a48102d30 2024-04-21T23:18:00.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:00.406+0000 7f8a377fe700 1 -- 172.21.15.40:0/2772133304 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a4000dd70 con 0x7f8a48102d30 2024-04-21T23:18:00.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:00.406+0000 7f8a4c8ac700 1 -- 172.21.15.40:0/2772133304 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a481a2bc0 con 0x7f8a48102d30 2024-04-21T23:18:00.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:00.406+0000 7f8a377fe700 1 -- 172.21.15.40:0/2772133304 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a40005a60 con 0x7f8a48102d30 2024-04-21T23:18:00.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:00.407+0000 7f8a4c8ac700 1 -- 172.21.15.40:0/2772133304 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a48068f20 con 0x7f8a48102d30 2024-04-21T23:18:00.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:00.408+0000 7f8a377fe700 1 -- 172.21.15.40:0/2772133304 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f8a4001a020 con 0x7f8a48102d30 2024-04-21T23:18:00.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:00.409+0000 7f8a377fe700 1 --2- 172.21.15.40:0/2772133304 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f8a300a2c20 0x7f8a300a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:00.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:00.409+0000 7f8a4659c700 1 --2- 172.21.15.40:0/2772133304 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f8a300a2c20 0x7f8a300a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:00.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:00.409+0000 7f8a377fe700 1 -- 172.21.15.40:0/2772133304 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f8a400ceac0 con 0x7f8a48102d30 2024-04-21T23:18:00.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:00.409+0000 7f8a4659c700 1 --2- 172.21.15.40:0/2772133304 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f8a300a2c20 0x7f8a300a50e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c00b580 tx=0x7f8a3c005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:00.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:00.413+0000 7f8a377fe700 1 -- 172.21.15.40:0/2772133304 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f8a40010070 con 0x7f8a48102d30 2024-04-21T23:18:00.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:59 smithi174 ceph-mon[31409]: pgmap v358: 129 pgs: 129 active+clean; 641 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 639 B/s wr, 5 op/s 2024-04-21T23:18:00.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:59 smithi174 ceph-mon[31409]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-04-21T23:18:00.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:17:59 smithi174 ceph-mon[31409]: Cluster is now healthy 2024-04-21T23:18:00.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:59 smithi111 ceph-mon[31085]: pgmap v358: 129 pgs: 129 active+clean; 641 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 639 B/s wr, 5 op/s 2024-04-21T23:18:00.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:59 smithi111 ceph-mon[31085]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-04-21T23:18:00.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:17:59 smithi111 ceph-mon[31085]: Cluster is now healthy 2024-04-21T23:18:00.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:00.631+0000 7f8a4c8ac700 1 -- 172.21.15.40:0/2772133304 --> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7f8a48062670 con 0x7f8a300a2c20 2024-04-21T23:18:00.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:00.633+0000 7f8a377fe700 1 -- 172.21.15.40:0/2772133304 <== mgr.14148 v2:172.21.15.40:6800/2500760788 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7f8a48062670 con 0x7f8a300a2c20 2024-04-21T23:18:00.632 INFO:teuthology.orchestra.run.smithi040.stdout:HOST ADDR LABELS STATUS 2024-04-21T23:18:00.632 INFO:teuthology.orchestra.run.smithi040.stdout:smithi040 172.21.15.40 2024-04-21T23:18:00.632 INFO:teuthology.orchestra.run.smithi040.stdout:smithi111 172.21.15.111 2024-04-21T23:18:00.632 INFO:teuthology.orchestra.run.smithi040.stdout:smithi174 172.21.15.174 2024-04-21T23:18:00.632 INFO:teuthology.orchestra.run.smithi040.stdout:3 hosts in cluster 2024-04-21T23:18:00.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:00.635+0000 7f8a4c8ac700 1 -- 172.21.15.40:0/2772133304 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f8a300a2c20 msgr2=0x7f8a300a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:00.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:00.635+0000 7f8a4c8ac700 1 --2- 172.21.15.40:0/2772133304 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f8a300a2c20 0x7f8a300a50e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c00b580 tx=0x7f8a3c005fb0 comp rx=0 tx=0).stop 2024-04-21T23:18:00.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:00.635+0000 7f8a4c8ac700 1 -- 172.21.15.40:0/2772133304 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a48102d30 msgr2=0x7f8a481a1b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:00.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:00.635+0000 7f8a4c8ac700 1 --2- 172.21.15.40:0/2772133304 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a48102d30 0x7f8a481a1b60 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f8a40007ee0 tx=0x7f8a4000d3b0 comp rx=0 tx=0).stop 2024-04-21T23:18:00.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:00.635+0000 7f8a4c8ac700 1 -- 172.21.15.40:0/2772133304 shutdown_connections 2024-04-21T23:18:00.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:00.635+0000 7f8a4c8ac700 1 --2- 172.21.15.40:0/2772133304 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a48101190 0x7f8a4806b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:00.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:00.635+0000 7f8a4c8ac700 1 --2- 172.21.15.40:0/2772133304 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f8a300a2c20 0x7f8a300a50e0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:00.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:00.635+0000 7f8a4c8ac700 1 --2- 172.21.15.40:0/2772133304 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a48102d30 0x7f8a481a1b60 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:00.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:00.635+0000 7f8a4c8ac700 1 --2- 172.21.15.40:0/2772133304 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a48102390 0x7f8a4819f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:00.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:00.635+0000 7f8a4c8ac700 1 -- 172.21.15.40:0/2772133304 >> 172.21.15.40:0/2772133304 conn(0x7f8a480fc750 msgr2=0x7f8a48105760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:00.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:00.636+0000 7f8a4c8ac700 1 -- 172.21.15.40:0/2772133304 shutdown_connections 2024-04-21T23:18:00.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:00.636+0000 7f8a4c8ac700 1 -- 172.21.15.40:0/2772133304 wait complete. 2024-04-21T23:18:01.348 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- bash -c 'ceph orch device ls' 2024-04-21T23:18:01.611 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:18:02.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:01 smithi040 ceph-mon[27984]: pgmap v359: 129 pgs: 129 active+clean; 641 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 480 B/s wr, 7 op/s 2024-04-21T23:18:02.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:01 smithi040 ceph-mon[27984]: from='client.15285 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:18:02.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:01 smithi174 ceph-mon[31409]: pgmap v359: 129 pgs: 129 active+clean; 641 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 480 B/s wr, 7 op/s 2024-04-21T23:18:02.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:01 smithi174 ceph-mon[31409]: from='client.15285 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:18:02.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:01 smithi111 ceph-mon[31085]: pgmap v359: 129 pgs: 129 active+clean; 641 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 480 B/s wr, 7 op/s 2024-04-21T23:18:02.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:01 smithi111 ceph-mon[31085]: from='client.15285 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:18:03.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:03.439+0000 7f3d1c6ed700 1 -- 172.21.15.40:0/3953661547 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d14102d50 msgr2=0x7f3d14109600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:03.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:03.439+0000 7f3d1c6ed700 1 --2- 172.21.15.40:0/3953661547 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d14102d50 0x7f3d14109600 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f3d0c00b0d0 tx=0x7f3d0c00b3e0 comp rx=0 tx=0).stop 2024-04-21T23:18:03.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:03.439+0000 7f3d1c6ed700 1 -- 172.21.15.40:0/3953661547 shutdown_connections 2024-04-21T23:18:03.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:03.439+0000 7f3d1c6ed700 1 --2- 172.21.15.40:0/3953661547 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d14102d50 0x7f3d14109600 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:03.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:03.439+0000 7f3d1c6ed700 1 --2- 172.21.15.40:0/3953661547 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d141011b0 0x7f3d141015d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:03.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:03.439+0000 7f3d1c6ed700 1 --2- 172.21.15.40:0/3953661547 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3d141023b0 0x7f3d14102810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:03.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:03.440+0000 7f3d1c6ed700 1 -- 172.21.15.40:0/3953661547 >> 172.21.15.40:0/3953661547 conn(0x7f3d140fc750 msgr2=0x7f3d140feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:03.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:03.440+0000 7f3d1c6ed700 1 -- 172.21.15.40:0/3953661547 shutdown_connections 2024-04-21T23:18:03.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:03.440+0000 7f3d1c6ed700 1 -- 172.21.15.40:0/3953661547 wait complete. 2024-04-21T23:18:03.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:03.440+0000 7f3d1c6ed700 1 Processor -- start 2024-04-21T23:18:03.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:03.440+0000 7f3d1c6ed700 1 -- start start 2024-04-21T23:18:03.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:03.440+0000 7f3d1c6ed700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d141011b0 0x7f3d141984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:03.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:03.440+0000 7f3d1c6ed700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d141023b0 0x7f3d141989e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:03.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:03.441+0000 7f3d1c6ed700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3d14102d50 0x7f3d1419fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:03.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:03.441+0000 7f3d1c6ed700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3d1410ce30 con 0x7f3d141011b0 2024-04-21T23:18:03.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:03.441+0000 7f3d1c6ed700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3d1410ccb0 con 0x7f3d141023b0 2024-04-21T23:18:03.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:03.441+0000 7f3d1c6ed700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3d1410cfb0 con 0x7f3d14102d50 2024-04-21T23:18:03.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:03.441+0000 7f3d1a489700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d141011b0 0x7f3d141984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:03.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:03.441+0000 7f3d1a489700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d141011b0 0x7f3d141984a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46460/0 (socket says 172.21.15.40:46460) 2024-04-21T23:18:03.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:03.441+0000 7f3d1a489700 1 -- 172.21.15.40:0/3856398183 learned_addr learned my addr 172.21.15.40:0/3856398183 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:03.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:03.441+0000 7f3d19c88700 1 --2- 172.21.15.40:0/3856398183 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d141023b0 0x7f3d141989e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:03.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:03.441+0000 7f3d1ac8a700 1 --2- 172.21.15.40:0/3856398183 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3d14102d50 0x7f3d1419fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:03.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:03.441+0000 7f3d1a489700 1 -- 172.21.15.40:0/3856398183 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3d14102d50 msgr2=0x7f3d1419fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:03.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:03.441+0000 7f3d1a489700 1 --2- 172.21.15.40:0/3856398183 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3d14102d50 0x7f3d1419fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:03.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:03.441+0000 7f3d1a489700 1 -- 172.21.15.40:0/3856398183 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d141023b0 msgr2=0x7f3d141989e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:03.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:03.441+0000 7f3d1a489700 1 --2- 172.21.15.40:0/3856398183 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d141023b0 0x7f3d141989e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:03.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:03.441+0000 7f3d1a489700 1 -- 172.21.15.40:0/3856398183 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d141a0100 con 0x7f3d141011b0 2024-04-21T23:18:03.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:03.441+0000 7f3d1a489700 1 --2- 172.21.15.40:0/3856398183 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d141011b0 0x7f3d141984a0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f3d0800b530 tx=0x7f3d0800b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:03.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:03.441+0000 7f3d077fe700 1 -- 172.21.15.40:0/3856398183 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d08015020 con 0x7f3d141011b0 2024-04-21T23:18:03.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:03.442+0000 7f3d077fe700 1 -- 172.21.15.40:0/3856398183 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d08011d10 con 0x7f3d141011b0 2024-04-21T23:18:03.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:03.442+0000 7f3d1c6ed700 1 -- 172.21.15.40:0/3856398183 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d141a03f0 con 0x7f3d141011b0 2024-04-21T23:18:03.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:03.442+0000 7f3d1c6ed700 1 -- 172.21.15.40:0/3856398183 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d141a09b0 con 0x7f3d141011b0 2024-04-21T23:18:03.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:03.442+0000 7f3d077fe700 1 -- 172.21.15.40:0/3856398183 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d08010d90 con 0x7f3d141011b0 2024-04-21T23:18:03.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:03.443+0000 7f3d1c6ed700 1 -- 172.21.15.40:0/3856398183 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3cf8005360 con 0x7f3d141011b0 2024-04-21T23:18:03.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:03.444+0000 7f3d077fe700 1 -- 172.21.15.40:0/3856398183 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f3d08018020 con 0x7f3d141011b0 2024-04-21T23:18:03.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:03.445+0000 7f3d077fe700 1 --2- 172.21.15.40:0/3856398183 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f3d000a2c20 0x7f3d000a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:03.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:03.445+0000 7f3d19c88700 1 --2- 172.21.15.40:0/3856398183 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f3d000a2c20 0x7f3d000a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:03.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:03.445+0000 7f3d077fe700 1 -- 172.21.15.40:0/3856398183 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f3d080ce9a0 con 0x7f3d141011b0 2024-04-21T23:18:03.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:03.446+0000 7f3d19c88700 1 --2- 172.21.15.40:0/3856398183 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f3d000a2c20 0x7f3d000a50e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f3d10004630 tx=0x7f3d1000b3c0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:03.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:03.450+0000 7f3d077fe700 1 -- 172.21.15.40:0/3856398183 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f3d0801f020 con 0x7f3d141011b0 2024-04-21T23:18:03.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:03.664+0000 7f3d1c6ed700 1 -- 172.21.15.40:0/3856398183 --> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7f3cf8000bf0 con 0x7f3d000a2c20 2024-04-21T23:18:03.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:03.665+0000 7f3d077fe700 1 -- 172.21.15.40:0/3856398183 <== mgr.14148 v2:172.21.15.40:6800/2500760788 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7f3cf8000bf0 con 0x7f3d000a2c20 2024-04-21T23:18:03.665 INFO:teuthology.orchestra.run.smithi040.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-04-21T23:18:03.665 INFO:teuthology.orchestra.run.smithi040.stdout:smithi040 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT5330009J400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-04-21T23:18:03.665 INFO:teuthology.orchestra.run.smithi040.stdout:smithi111 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400VZ400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-04-21T23:18:03.665 INFO:teuthology.orchestra.run.smithi040.stdout:smithi174 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT62050074400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-04-21T23:18:03.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:03.667+0000 7f3d1c6ed700 1 -- 172.21.15.40:0/3856398183 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f3d000a2c20 msgr2=0x7f3d000a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:03.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:03.667+0000 7f3d1c6ed700 1 --2- 172.21.15.40:0/3856398183 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f3d000a2c20 0x7f3d000a50e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f3d10004630 tx=0x7f3d1000b3c0 comp rx=0 tx=0).stop 2024-04-21T23:18:03.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:03.667+0000 7f3d1c6ed700 1 -- 172.21.15.40:0/3856398183 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d141011b0 msgr2=0x7f3d141984a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:03.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:03.667+0000 7f3d1c6ed700 1 --2- 172.21.15.40:0/3856398183 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d141011b0 0x7f3d141984a0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f3d0800b530 tx=0x7f3d0800b840 comp rx=0 tx=0).stop 2024-04-21T23:18:03.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:03.667+0000 7f3d1c6ed700 1 -- 172.21.15.40:0/3856398183 shutdown_connections 2024-04-21T23:18:03.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:03.667+0000 7f3d1c6ed700 1 --2- 172.21.15.40:0/3856398183 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3d14102d50 0x7f3d1419fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:03.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:03.667+0000 7f3d1c6ed700 1 --2- 172.21.15.40:0/3856398183 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f3d000a2c20 0x7f3d000a50e0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:03.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:03.667+0000 7f3d1c6ed700 1 --2- 172.21.15.40:0/3856398183 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d141011b0 0x7f3d141984a0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:03.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:03.667+0000 7f3d1c6ed700 1 --2- 172.21.15.40:0/3856398183 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d141023b0 0x7f3d141989e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:03.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:03.667+0000 7f3d1c6ed700 1 -- 172.21.15.40:0/3856398183 >> 172.21.15.40:0/3856398183 conn(0x7f3d140fc750 msgr2=0x7f3d140fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:03.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:03.668+0000 7f3d1c6ed700 1 -- 172.21.15.40:0/3856398183 shutdown_connections 2024-04-21T23:18:03.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:03.668+0000 7f3d1c6ed700 1 -- 172.21.15.40:0/3856398183 wait complete. 2024-04-21T23:18:04.354 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:03 smithi040 ceph-mon[27984]: pgmap v360: 129 pgs: 129 active+clean; 641 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 409 B/s wr, 6 op/s 2024-04-21T23:18:04.388 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-21T23:18:04.399 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi040.front.sepia.ceph.com 2024-04-21T23:18:04.399 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-21T23:18:04.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:03 smithi174 ceph-mon[31409]: pgmap v360: 129 pgs: 129 active+clean; 641 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 409 B/s wr, 6 op/s 2024-04-21T23:18:04.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:03 smithi111 ceph-mon[31085]: pgmap v360: 129 pgs: 129 active+clean; 641 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 409 B/s wr, 6 op/s 2024-04-21T23:18:04.652 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:18:05.049 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:04 smithi040 ceph-mon[27984]: from='client.15291 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:18:05.345 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:04 smithi174 ceph-mon[31409]: from='client.15291 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:18:05.345 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:04 smithi111 ceph-mon[31085]: from='client.15291 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:18:06.158 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:05 smithi040 ceph-mon[27984]: pgmap v361: 129 pgs: 129 active+clean; 641 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 355 B/s wr, 6 op/s 2024-04-21T23:18:06.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:06.416+0000 7f06d2e67700 1 -- 172.21.15.40:0/1537240143 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06cc101190 msgr2=0x7f06cc1015b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:06.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:06.416+0000 7f06d2e67700 1 --2- 172.21.15.40:0/1537240143 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06cc101190 0x7f06cc1015b0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f06c00099c0 tx=0x7f06c0009cd0 comp rx=0 tx=0).stop 2024-04-21T23:18:06.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:06.416+0000 7f06d2e67700 1 -- 172.21.15.40:0/1537240143 shutdown_connections 2024-04-21T23:18:06.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:06.416+0000 7f06d2e67700 1 --2- 172.21.15.40:0/1537240143 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06cc102d30 0x7f06cc1095e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:06.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:06.416+0000 7f06d2e67700 1 --2- 172.21.15.40:0/1537240143 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06cc101190 0x7f06cc1015b0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:06.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:06.416+0000 7f06d2e67700 1 --2- 172.21.15.40:0/1537240143 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06cc102390 0x7f06cc1027f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:06.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:06.416+0000 7f06d2e67700 1 -- 172.21.15.40:0/1537240143 >> 172.21.15.40:0/1537240143 conn(0x7f06cc0fc730 msgr2=0x7f06cc0feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:06.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:06.416+0000 7f06d2e67700 1 -- 172.21.15.40:0/1537240143 shutdown_connections 2024-04-21T23:18:06.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:06.416+0000 7f06d2e67700 1 -- 172.21.15.40:0/1537240143 wait complete. 2024-04-21T23:18:06.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:06.417+0000 7f06d2e67700 1 Processor -- start 2024-04-21T23:18:06.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:06.417+0000 7f06d2e67700 1 -- start start 2024-04-21T23:18:06.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:06.417+0000 7f06d2e67700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06cc101190 0x7f06cc1050d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:06.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:06.417+0000 7f06d2e67700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06cc102390 0x7f06cc103770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:06.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:06.417+0000 7f06d2e67700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06cc102d30 0x7f06cc103cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:06.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:06.417+0000 7f06d2e67700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f06cc10cde0 con 0x7f06cc102390 2024-04-21T23:18:06.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:06.417+0000 7f06d2e67700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f06cc10cc60 con 0x7f06cc102d30 2024-04-21T23:18:06.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:06.417+0000 7f06d2e67700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f06cc10cf60 con 0x7f06cc101190 2024-04-21T23:18:06.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:06.417+0000 7f06cbfff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06cc102390 0x7f06cc103770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:06.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:06.418+0000 7f06cbfff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06cc102390 0x7f06cc103770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46478/0 (socket says 172.21.15.40:46478) 2024-04-21T23:18:06.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:06.418+0000 7f06cbfff700 1 -- 172.21.15.40:0/993957525 learned_addr learned my addr 172.21.15.40:0/993957525 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:06.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:06.418+0000 7f06d1404700 1 --2- 172.21.15.40:0/993957525 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06cc102d30 0x7f06cc103cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:06.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:06.418+0000 7f06d0c03700 1 --2- 172.21.15.40:0/993957525 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06cc101190 0x7f06cc1050d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:06.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:06.418+0000 7f06cbfff700 1 -- 172.21.15.40:0/993957525 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06cc101190 msgr2=0x7f06cc1050d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:06.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:06.418+0000 7f06cbfff700 1 --2- 172.21.15.40:0/993957525 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06cc101190 0x7f06cc1050d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:06.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:06.418+0000 7f06cbfff700 1 -- 172.21.15.40:0/993957525 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06cc102d30 msgr2=0x7f06cc103cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:06.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:06.418+0000 7f06cbfff700 1 --2- 172.21.15.40:0/993957525 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06cc102d30 0x7f06cc103cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:06.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:06.418+0000 7f06cbfff700 1 -- 172.21.15.40:0/993957525 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06cc104580 con 0x7f06cc102390 2024-04-21T23:18:06.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:06.418+0000 7f06cbfff700 1 --2- 172.21.15.40:0/993957525 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06cc102390 0x7f06cc103770 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f06b800e9c0 tx=0x7f06b800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:06.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:06.418+0000 7f06c9ffb700 1 -- 172.21.15.40:0/993957525 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06b800cbd0 con 0x7f06cc102390 2024-04-21T23:18:06.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:06.418+0000 7f06c9ffb700 1 -- 172.21.15.40:0/993957525 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06b800ee60 con 0x7f06cc102390 2024-04-21T23:18:06.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:06.418+0000 7f06c9ffb700 1 -- 172.21.15.40:0/993957525 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06b8005630 con 0x7f06cc102390 2024-04-21T23:18:06.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:06.418+0000 7f06d2e67700 1 -- 172.21.15.40:0/993957525 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06cc1a2720 con 0x7f06cc102390 2024-04-21T23:18:06.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:06.418+0000 7f06d2e67700 1 -- 172.21.15.40:0/993957525 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06cc1a2be0 con 0x7f06cc102390 2024-04-21T23:18:06.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:06.419+0000 7f06d2e67700 1 -- 172.21.15.40:0/993957525 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06cc068f20 con 0x7f06cc102390 2024-04-21T23:18:06.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:06.420+0000 7f06c9ffb700 1 -- 172.21.15.40:0/993957525 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f06b8004750 con 0x7f06cc102390 2024-04-21T23:18:06.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:06.421+0000 7f06c9ffb700 1 --2- 172.21.15.40:0/993957525 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f06bc0a2c20 0x7f06bc0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:06.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:06.421+0000 7f06d0c03700 1 --2- 172.21.15.40:0/993957525 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f06bc0a2c20 0x7f06bc0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:06.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:06.421+0000 7f06c9ffb700 1 -- 172.21.15.40:0/993957525 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f06b80cf710 con 0x7f06cc102390 2024-04-21T23:18:06.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:06.422+0000 7f06d0c03700 1 --2- 172.21.15.40:0/993957525 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f06bc0a2c20 0x7f06bc0a50e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f06c0000c00 tx=0x7f06c0005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:06.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:06.426+0000 7f06c9ffb700 1 -- 172.21.15.40:0/993957525 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f06b809c2a0 con 0x7f06cc102390 2024-04-21T23:18:06.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:05 smithi174 ceph-mon[31409]: pgmap v361: 129 pgs: 129 active+clean; 641 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 355 B/s wr, 6 op/s 2024-04-21T23:18:06.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:05 smithi111 ceph-mon[31085]: pgmap v361: 129 pgs: 129 active+clean; 641 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 355 B/s wr, 6 op/s 2024-04-21T23:18:06.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:06.677+0000 7f06d2e67700 1 -- 172.21.15.40:0/993957525 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f06cc1a3460 con 0x7f06cc102390 2024-04-21T23:18:06.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:06.678+0000 7f06c9ffb700 1 -- 172.21.15.40:0/993957525 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 24 v24) ==== 76+0+3058 (secure 0 0 0) 0x7f06b809bba0 con 0x7f06cc102390 2024-04-21T23:18:06.677 INFO:teuthology.orchestra.run.smithi040.stdout:e24 2024-04-21T23:18:06.677 INFO:teuthology.orchestra.run.smithi040.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-21T23:18:06.677 INFO:teuthology.orchestra.run.smithi040.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-21T23:18:06.677 INFO:teuthology.orchestra.run.smithi040.stdout:legacy client fscid: 1 2024-04-21T23:18:06.677 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:18:06.677 INFO:teuthology.orchestra.run.smithi040.stdout:Filesystem 'cephfs' (1) 2024-04-21T23:18:06.677 INFO:teuthology.orchestra.run.smithi040.stdout:fs_name cephfs 2024-04-21T23:18:06.677 INFO:teuthology.orchestra.run.smithi040.stdout:epoch 24 2024-04-21T23:18:06.677 INFO:teuthology.orchestra.run.smithi040.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-21T23:18:06.677 INFO:teuthology.orchestra.run.smithi040.stdout:created 2024-04-21T23:15:55.742233+0000 2024-04-21T23:18:06.678 INFO:teuthology.orchestra.run.smithi040.stdout:modified 2024-04-21T23:16:22.498043+0000 2024-04-21T23:18:06.678 INFO:teuthology.orchestra.run.smithi040.stdout:tableserver 0 2024-04-21T23:18:06.678 INFO:teuthology.orchestra.run.smithi040.stdout:root 0 2024-04-21T23:18:06.678 INFO:teuthology.orchestra.run.smithi040.stdout:session_timeout 300 2024-04-21T23:18:06.678 INFO:teuthology.orchestra.run.smithi040.stdout:session_autoclose 300 2024-04-21T23:18:06.678 INFO:teuthology.orchestra.run.smithi040.stdout:max_file_size 1099511627776 2024-04-21T23:18:06.678 INFO:teuthology.orchestra.run.smithi040.stdout:max_xattr_size 65536 2024-04-21T23:18:06.678 INFO:teuthology.orchestra.run.smithi040.stdout:required_client_features {} 2024-04-21T23:18:06.678 INFO:teuthology.orchestra.run.smithi040.stdout:last_failure 0 2024-04-21T23:18:06.678 INFO:teuthology.orchestra.run.smithi040.stdout:last_failure_osd_epoch 0 2024-04-21T23:18:06.678 INFO:teuthology.orchestra.run.smithi040.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-21T23:18:06.678 INFO:teuthology.orchestra.run.smithi040.stdout:max_mds 3 2024-04-21T23:18:06.678 INFO:teuthology.orchestra.run.smithi040.stdout:in 0,1,2 2024-04-21T23:18:06.679 INFO:teuthology.orchestra.run.smithi040.stdout:up {0=24521,1=24502,2=24494} 2024-04-21T23:18:06.679 INFO:teuthology.orchestra.run.smithi040.stdout:failed 2024-04-21T23:18:06.679 INFO:teuthology.orchestra.run.smithi040.stdout:damaged 2024-04-21T23:18:06.679 INFO:teuthology.orchestra.run.smithi040.stdout:stopped 2024-04-21T23:18:06.679 INFO:teuthology.orchestra.run.smithi040.stdout:data_pools [3,4] 2024-04-21T23:18:06.679 INFO:teuthology.orchestra.run.smithi040.stdout:metadata_pool 2 2024-04-21T23:18:06.679 INFO:teuthology.orchestra.run.smithi040.stdout:inline_data disabled 2024-04-21T23:18:06.679 INFO:teuthology.orchestra.run.smithi040.stdout:balancer 2024-04-21T23:18:06.679 INFO:teuthology.orchestra.run.smithi040.stdout:bal_rank_mask -1 2024-04-21T23:18:06.679 INFO:teuthology.orchestra.run.smithi040.stdout:standby_count_wanted 0 2024-04-21T23:18:06.679 INFO:teuthology.orchestra.run.smithi040.stdout:qdb_cluster leader: 24521 members: 24494,24502,24521 2024-04-21T23:18:06.679 INFO:teuthology.orchestra.run.smithi040.stdout:[mds.b{0:24521} state up:active seq 3 addr [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] compat {c=[1],r=[1],i=[1fff]}] 2024-04-21T23:18:06.679 INFO:teuthology.orchestra.run.smithi040.stdout:[mds.c{0:24484} state up:standby-replay seq 1 addr [v2:172.21.15.174:6836/465579746,v1:172.21.15.174:6837/465579746] compat {c=[1],r=[1],i=[1fff]}] 2024-04-21T23:18:06.679 INFO:teuthology.orchestra.run.smithi040.stdout:[mds.i{1:24502} state up:active seq 11 addr [v2:172.21.15.174:6838/2467509274,v1:172.21.15.174:6839/2467509274] compat {c=[1],r=[1],i=[1fff]}] 2024-04-21T23:18:06.680 INFO:teuthology.orchestra.run.smithi040.stdout:[mds.f{1:24466} state up:standby-replay seq 1 addr [v2:172.21.15.174:6834/3905733045,v1:172.21.15.174:6835/3905733045] compat {c=[1],r=[1],i=[1fff]}] 2024-04-21T23:18:06.680 INFO:teuthology.orchestra.run.smithi040.stdout:[mds.h{2:24494} state up:active seq 12 addr [v2:172.21.15.111:6836/2703764182,v1:172.21.15.111:6837/2703764182] compat {c=[1],r=[1],i=[1fff]}] 2024-04-21T23:18:06.680 INFO:teuthology.orchestra.run.smithi040.stdout:[mds.e{2:24461} state up:standby-replay seq 1 addr [v2:172.21.15.111:6834/2704462092,v1:172.21.15.111:6835/2704462092] compat {c=[1],r=[1],i=[1fff]}] 2024-04-21T23:18:06.680 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:18:06.680 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:18:06.680 INFO:teuthology.orchestra.run.smithi040.stdout:Standby daemons: 2024-04-21T23:18:06.680 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:18:06.680 INFO:teuthology.orchestra.run.smithi040.stdout:[mds.j{-1:14535} state up:standby seq 1 addr [v2:172.21.15.40:6834/1571948830,v1:172.21.15.40:6835/1571948830] compat {c=[1],r=[1],i=[1fff]}] 2024-04-21T23:18:06.680 INFO:teuthology.orchestra.run.smithi040.stdout:[mds.d{-1:14547} state up:standby seq 1 addr [v2:172.21.15.40:6836/969356642,v1:172.21.15.40:6837/969356642] compat {c=[1],r=[1],i=[1fff]}] 2024-04-21T23:18:06.680 INFO:teuthology.orchestra.run.smithi040.stdout:[mds.g{-1:14616} state up:standby seq 1 addr [v2:172.21.15.40:6838/3025057845,v1:172.21.15.40:6839/3025057845] compat {c=[1],r=[1],i=[1fff]}] 2024-04-21T23:18:06.680 INFO:teuthology.orchestra.run.smithi040.stdout:[mds.a{-1:14628} state up:standby seq 1 addr [v2:172.21.15.40:6840/3350488804,v1:172.21.15.40:6841/3350488804] compat {c=[1],r=[1],i=[1fff]}] 2024-04-21T23:18:06.680 INFO:teuthology.orchestra.run.smithi040.stdout:[mds.l{-1:24415} state up:standby seq 1 addr [v2:172.21.15.174:6832/3438493203,v1:172.21.15.174:6833/3438493203] compat {c=[1],r=[1],i=[1fff]}] 2024-04-21T23:18:06.680 INFO:teuthology.orchestra.run.smithi040.stdout:[mds.k{-1:24428} state up:standby seq 1 addr [v2:172.21.15.111:6832/3105752678,v1:172.21.15.111:6833/3105752678] compat {c=[1],r=[1],i=[1fff]}] 2024-04-21T23:18:06.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:06.680+0000 7f06d2e67700 1 -- 172.21.15.40:0/993957525 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f06bc0a2c20 msgr2=0x7f06bc0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:06.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:06.680+0000 7f06d2e67700 1 --2- 172.21.15.40:0/993957525 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f06bc0a2c20 0x7f06bc0a50e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f06c0000c00 tx=0x7f06c0005fb0 comp rx=0 tx=0).stop 2024-04-21T23:18:06.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:06.680+0000 7f06d2e67700 1 -- 172.21.15.40:0/993957525 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06cc102390 msgr2=0x7f06cc103770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:06.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:06.680+0000 7f06d2e67700 1 --2- 172.21.15.40:0/993957525 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06cc102390 0x7f06cc103770 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f06b800e9c0 tx=0x7f06b800ecd0 comp rx=0 tx=0).stop 2024-04-21T23:18:06.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:06.680+0000 7f06d2e67700 1 -- 172.21.15.40:0/993957525 shutdown_connections 2024-04-21T23:18:06.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:06.680+0000 7f06d2e67700 1 --2- 172.21.15.40:0/993957525 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06cc101190 0x7f06cc1050d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:06.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:06.680+0000 7f06d2e67700 1 --2- 172.21.15.40:0/993957525 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f06bc0a2c20 0x7f06bc0a50e0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:06.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:06.680+0000 7f06d2e67700 1 --2- 172.21.15.40:0/993957525 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06cc102d30 0x7f06cc103cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:06.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:06.680+0000 7f06d2e67700 1 --2- 172.21.15.40:0/993957525 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06cc102390 0x7f06cc103770 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:06.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:06.680+0000 7f06d2e67700 1 -- 172.21.15.40:0/993957525 >> 172.21.15.40:0/993957525 conn(0x7f06cc0fc730 msgr2=0x7f06cc0fe240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:06.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:06.680+0000 7f06d2e67700 1 -- 172.21.15.40:0/993957525 shutdown_connections 2024-04-21T23:18:06.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:06.680+0000 7f06d2e67700 1 -- 172.21.15.40:0/993957525 wait complete. 2024-04-21T23:18:06.682 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 24 2024-04-21T23:18:07.355 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:06 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/993957525' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-21T23:18:07.385 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- bash -c 'ceph osd dump' 2024-04-21T23:18:07.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:07 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/993957525' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-21T23:18:07.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:07 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/993957525' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-21T23:18:07.632 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:18:08.050 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:08 smithi040 ceph-mon[27984]: pgmap v362: 129 pgs: 129 active+clean; 641 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-21T23:18:08.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:08 smithi174 ceph-mon[31409]: pgmap v362: 129 pgs: 129 active+clean; 641 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-21T23:18:08.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:08 smithi111 ceph-mon[31085]: pgmap v362: 129 pgs: 129 active+clean; 641 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-21T23:18:09.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:09.399+0000 7f3ade882700 1 -- 172.21.15.40:0/3249784470 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ad81022b0 msgr2=0x7f3ad81026d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:09.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:09.399+0000 7f3ade882700 1 --2- 172.21.15.40:0/3249784470 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ad81022b0 0x7f3ad81026d0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f3ac80099c0 tx=0x7f3ac8009cd0 comp rx=0 tx=0).stop 2024-04-21T23:18:09.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:09.399+0000 7f3ade882700 1 -- 172.21.15.40:0/3249784470 shutdown_connections 2024-04-21T23:18:09.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:09.399+0000 7f3ade882700 1 --2- 172.21.15.40:0/3249784470 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ad8103e50 0x7f3ad810a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:09.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:09.399+0000 7f3ade882700 1 --2- 172.21.15.40:0/3249784470 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ad81022b0 0x7f3ad81026d0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:09.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:09.399+0000 7f3ade882700 1 --2- 172.21.15.40:0/3249784470 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3ad81034b0 0x7f3ad8103910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:09.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:09.399+0000 7f3ade882700 1 -- 172.21.15.40:0/3249784470 >> 172.21.15.40:0/3249784470 conn(0x7f3ad80fd850 msgr2=0x7f3ad80ffc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:09.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:09.399+0000 7f3ade882700 1 -- 172.21.15.40:0/3249784470 shutdown_connections 2024-04-21T23:18:09.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:09.399+0000 7f3ade882700 1 -- 172.21.15.40:0/3249784470 wait complete. 2024-04-21T23:18:09.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:09.400+0000 7f3ade882700 1 Processor -- start 2024-04-21T23:18:09.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:09.400+0000 7f3ade882700 1 -- start start 2024-04-21T23:18:09.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:09.401+0000 7f3ade882700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ad81022b0 0x7f3ad8075fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:09.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:09.401+0000 7f3ade882700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ad81034b0 0x7f3ad8074620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:09.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:09.401+0000 7f3ade882700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3ad8103e50 0x7f3ad8074b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:09.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:09.401+0000 7f3ade882700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3ad810de70 con 0x7f3ad81034b0 2024-04-21T23:18:09.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:09.401+0000 7f3ade882700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3ad810dcf0 con 0x7f3ad81022b0 2024-04-21T23:18:09.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:09.401+0000 7f3ade882700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3ad810dff0 con 0x7f3ad8103e50 2024-04-21T23:18:09.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:09.401+0000 7f3ad77fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ad81034b0 0x7f3ad8074620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:09.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:09.401+0000 7f3ad77fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ad81034b0 0x7f3ad8074620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46500/0 (socket says 172.21.15.40:46500) 2024-04-21T23:18:09.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:09.401+0000 7f3ad77fe700 1 -- 172.21.15.40:0/2640391364 learned_addr learned my addr 172.21.15.40:0/2640391364 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:09.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:09.401+0000 7f3adce1f700 1 --2- 172.21.15.40:0/2640391364 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3ad8103e50 0x7f3ad8074b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:09.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:09.401+0000 7f3ad7fff700 1 --2- 172.21.15.40:0/2640391364 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ad81022b0 0x7f3ad8075fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:09.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:09.401+0000 7f3ad77fe700 1 -- 172.21.15.40:0/2640391364 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3ad8103e50 msgr2=0x7f3ad8074b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:09.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:09.401+0000 7f3ad77fe700 1 --2- 172.21.15.40:0/2640391364 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3ad8103e50 0x7f3ad8074b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:09.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:09.401+0000 7f3ad77fe700 1 -- 172.21.15.40:0/2640391364 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ad81022b0 msgr2=0x7f3ad8075fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:09.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:09.401+0000 7f3ad77fe700 1 --2- 172.21.15.40:0/2640391364 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ad81022b0 0x7f3ad8075fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:09.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:09.401+0000 7f3ad77fe700 1 -- 172.21.15.40:0/2640391364 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ad8075410 con 0x7f3ad81034b0 2024-04-21T23:18:09.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:09.401+0000 7f3ad7fff700 1 --2- 172.21.15.40:0/2640391364 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ad81022b0 0x7f3ad8075fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:18:09.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:09.401+0000 7f3ad77fe700 1 --2- 172.21.15.40:0/2640391364 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ad81034b0 0x7f3ad8074620 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f3acc00c9b0 tx=0x7f3acc00ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:09.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:09.401+0000 7f3ad57fa700 1 -- 172.21.15.40:0/2640391364 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3acc007b00 con 0x7f3ad81034b0 2024-04-21T23:18:09.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:09.402+0000 7f3ad57fa700 1 -- 172.21.15.40:0/2640391364 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3acc007ca0 con 0x7f3ad81034b0 2024-04-21T23:18:09.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:09.402+0000 7f3ad57fa700 1 -- 172.21.15.40:0/2640391364 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3acc018730 con 0x7f3ad81034b0 2024-04-21T23:18:09.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:09.402+0000 7f3ade882700 1 -- 172.21.15.40:0/2640391364 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ad81a25f0 con 0x7f3ad81034b0 2024-04-21T23:18:09.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:09.402+0000 7f3ade882700 1 -- 172.21.15.40:0/2640391364 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ad81a2ab0 con 0x7f3ad81034b0 2024-04-21T23:18:09.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:09.403+0000 7f3ade882700 1 -- 172.21.15.40:0/2640391364 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ad8068f20 con 0x7f3ad81034b0 2024-04-21T23:18:09.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:09.405+0000 7f3ad57fa700 1 -- 172.21.15.40:0/2640391364 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f3acc0188d0 con 0x7f3ad81034b0 2024-04-21T23:18:09.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:09.405+0000 7f3ad57fa700 1 --2- 172.21.15.40:0/2640391364 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f3ac00a2c20 0x7f3ac00a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:09.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:09.405+0000 7f3ad7fff700 1 --2- 172.21.15.40:0/2640391364 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f3ac00a2c20 0x7f3ac00a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:09.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:09.405+0000 7f3ad57fa700 1 -- 172.21.15.40:0/2640391364 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f3acc0cf670 con 0x7f3ad81034b0 2024-04-21T23:18:09.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:09.406+0000 7f3ad7fff700 1 --2- 172.21.15.40:0/2640391364 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f3ac00a2c20 0x7f3ac00a50e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f3ac8003890 tx=0x7f3ac8005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:09.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:09.409+0000 7f3ad57fa700 1 -- 172.21.15.40:0/2640391364 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f3acc014030 con 0x7f3ad81034b0 2024-04-21T23:18:09.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:09.616+0000 7f3ade882700 1 -- 172.21.15.40:0/2640391364 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f3ad802db30 con 0x7f3ad81034b0 2024-04-21T23:18:09.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:09.617+0000 7f3ad57fa700 1 -- 172.21.15.40:0/2640391364 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v94) ==== 56+0+5708 (secure 0 0 0) 0x7f3acc09c290 con 0x7f3ad81034b0 2024-04-21T23:18:09.616 INFO:teuthology.orchestra.run.smithi040.stdout:epoch 94 2024-04-21T23:18:09.616 INFO:teuthology.orchestra.run.smithi040.stdout:fsid d814e4dc-0033-11ef-bc93-c7b262605968 2024-04-21T23:18:09.616 INFO:teuthology.orchestra.run.smithi040.stdout:created 2024-04-21T23:07:52.659207+0000 2024-04-21T23:18:09.616 INFO:teuthology.orchestra.run.smithi040.stdout:modified 2024-04-21T23:17:52.907466+0000 2024-04-21T23:18:09.616 INFO:teuthology.orchestra.run.smithi040.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-21T23:18:09.616 INFO:teuthology.orchestra.run.smithi040.stdout:crush_version 27 2024-04-21T23:18:09.616 INFO:teuthology.orchestra.run.smithi040.stdout:full_ratio 0.95 2024-04-21T23:18:09.616 INFO:teuthology.orchestra.run.smithi040.stdout:backfillfull_ratio 0.9 2024-04-21T23:18:09.616 INFO:teuthology.orchestra.run.smithi040.stdout:nearfull_ratio 0.85 2024-04-21T23:18:09.616 INFO:teuthology.orchestra.run.smithi040.stdout:require_min_compat_client luminous 2024-04-21T23:18:09.616 INFO:teuthology.orchestra.run.smithi040.stdout:min_compat_client luminous 2024-04-21T23:18:09.617 INFO:teuthology.orchestra.run.smithi040.stdout:require_osd_release squid 2024-04-21T23:18:09.617 INFO:teuthology.orchestra.run.smithi040.stdout:stretch_mode_enabled false 2024-04-21T23:18:09.617 INFO:teuthology.orchestra.run.smithi040.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-04-21T23:18:09.617 INFO:teuthology.orchestra.run.smithi040.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-04-21T23:18:09.617 INFO:teuthology.orchestra.run.smithi040.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 86 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-04-21T23:18:09.617 INFO:teuthology.orchestra.run.smithi040.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 90 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-04-21T23:18:09.617 INFO:teuthology.orchestra.run.smithi040.stdout:max_osd 12 2024-04-21T23:18:09.617 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0 up in weight 1 up_from 8 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.40:6802/2009758784,v1:172.21.15.40:6803/2009758784] [v2:172.21.15.40:6804/2009758784,v1:172.21.15.40:6805/2009758784] exists,up 0e8d692a-22fe-4a87-8d99-9151f96cbb02 2024-04-21T23:18:09.617 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1 up in weight 1 up_from 13 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.111:6800/1763903836,v1:172.21.15.111:6801/1763903836] [v2:172.21.15.111:6802/1763903836,v1:172.21.15.111:6803/1763903836] exists,up fe5fe11b-0e84-4359-83a7-1ba45105755a 2024-04-21T23:18:09.617 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2 up in weight 1 up_from 18 up_thru 93 down_at 0 last_clean_interval [0,0) [v2:172.21.15.174:6800/439695825,v1:172.21.15.174:6801/439695825] [v2:172.21.15.174:6802/439695825,v1:172.21.15.174:6803/439695825] exists,up 813ad735-5cc2-4da0-ae16-6db238395fbe 2024-04-21T23:18:09.617 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3 up in weight 1 up_from 25 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.40:6810/3454312436,v1:172.21.15.40:6811/3454312436] [v2:172.21.15.40:6812/3454312436,v1:172.21.15.40:6813/3454312436] exists,up 09becb03-b7ce-4ffc-917d-b7900ec37e9c 2024-04-21T23:18:09.617 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4 up in weight 1 up_from 33 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.111:6808/1810866070,v1:172.21.15.111:6809/1810866070] [v2:172.21.15.111:6810/1810866070,v1:172.21.15.111:6811/1810866070] exists,up 46eabfd7-d88a-4efd-94eb-04354d3a9d7e 2024-04-21T23:18:09.617 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5 up in weight 1 up_from 38 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.174:6808/3752986711,v1:172.21.15.174:6809/3752986711] [v2:172.21.15.174:6810/3752986711,v1:172.21.15.174:6811/3752986711] exists,up 6f73b582-6c65-41db-aab0-9b360f83d5c6 2024-04-21T23:18:09.618 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6 up in weight 1 up_from 46 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.40:6818/1618855427,v1:172.21.15.40:6819/1618855427] [v2:172.21.15.40:6820/1618855427,v1:172.21.15.40:6821/1618855427] exists,up 39f305f0-12d2-4732-aaa0-cd66700af538 2024-04-21T23:18:09.618 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7 up in weight 1 up_from 51 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.111:6816/927054162,v1:172.21.15.111:6817/927054162] [v2:172.21.15.111:6818/927054162,v1:172.21.15.111:6819/927054162] exists,up 769f2468-5102-490b-89f8-d0444d3169a1 2024-04-21T23:18:09.618 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8 up in weight 1 up_from 59 up_thru 93 down_at 0 last_clean_interval [0,0) [v2:172.21.15.174:6816/3817848746,v1:172.21.15.174:6817/3817848746] [v2:172.21.15.174:6818/3817848746,v1:172.21.15.174:6819/3817848746] exists,up 1a444e80-007d-473b-b972-4c814a813c22 2024-04-21T23:18:09.618 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9 up in weight 1 up_from 67 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.40:6826/1649446531,v1:172.21.15.40:6827/1649446531] [v2:172.21.15.40:6828/1649446531,v1:172.21.15.40:6829/1649446531] exists,up 9a6b6a56-6ae5-49fe-9e5b-cb17d686e0dd 2024-04-21T23:18:09.618 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10 up in weight 1 up_from 73 up_thru 93 down_at 0 last_clean_interval [0,0) [v2:172.21.15.111:6824/1227538935,v1:172.21.15.111:6825/1227538935] [v2:172.21.15.111:6826/1227538935,v1:172.21.15.111:6827/1227538935] exists,up a70038ba-e2ad-4170-85cc-ea0470a2edef 2024-04-21T23:18:09.618 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11 up in weight 1 up_from 81 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.174:6824/2045865553,v1:172.21.15.174:6825/2045865553] [v2:172.21.15.174:6826/2045865553,v1:172.21.15.174:6827/2045865553] exists,up 781532bb-7a5e-44e3-8eb7-b584b0054666 2024-04-21T23:18:09.618 INFO:teuthology.orchestra.run.smithi040.stdout:pg_upmap_items 2.b [8,11] 2024-04-21T23:18:09.618 INFO:teuthology.orchestra.run.smithi040.stdout:pg_upmap_items 2.10 [8,11] 2024-04-21T23:18:09.618 INFO:teuthology.orchestra.run.smithi040.stdout:pg_upmap_items 2.16 [2,4] 2024-04-21T23:18:09.618 INFO:teuthology.orchestra.run.smithi040.stdout:pg_upmap_items 2.1d [2,1] 2024-04-21T23:18:09.618 INFO:teuthology.orchestra.run.smithi040.stdout:pg_upmap_items 3.0 [4,8] 2024-04-21T23:18:09.618 INFO:teuthology.orchestra.run.smithi040.stdout:pg_upmap_items 3.11 [4,6] 2024-04-21T23:18:09.618 INFO:teuthology.orchestra.run.smithi040.stdout:pg_upmap_items 3.13 [4,9] 2024-04-21T23:18:09.619 INFO:teuthology.orchestra.run.smithi040.stdout:pg_upmap_items 3.1b [5,9] 2024-04-21T23:18:09.619 INFO:teuthology.orchestra.run.smithi040.stdout:pg_upmap_items 3.2a [3,8] 2024-04-21T23:18:09.619 INFO:teuthology.orchestra.run.smithi040.stdout:pg_upmap_items 3.33 [5,9] 2024-04-21T23:18:09.619 INFO:teuthology.orchestra.run.smithi040.stdout:pg_upmap_items 4.9 [5,8] 2024-04-21T23:18:09.619 INFO:teuthology.orchestra.run.smithi040.stdout:pg_upmap_items 4.f [9,8] 2024-04-21T23:18:09.619 INFO:teuthology.orchestra.run.smithi040.stdout:pg_upmap_items 4.17 [9,10] 2024-04-21T23:18:09.619 INFO:teuthology.orchestra.run.smithi040.stdout:pg_upmap_items 4.19 [9,8] 2024-04-21T23:18:09.619 INFO:teuthology.orchestra.run.smithi040.stdout:blocklist 172.21.15.40:6801/526554319 expires 2024-04-22T23:08:50.305770+0000 2024-04-21T23:18:09.619 INFO:teuthology.orchestra.run.smithi040.stdout:blocklist 172.21.15.40:6800/526554319 expires 2024-04-22T23:08:50.305770+0000 2024-04-21T23:18:09.619 INFO:teuthology.orchestra.run.smithi040.stdout:blocklist 172.21.15.40:0/3178763471 expires 2024-04-22T23:08:50.305770+0000 2024-04-21T23:18:09.619 INFO:teuthology.orchestra.run.smithi040.stdout:blocklist 172.21.15.40:0/4045911364 expires 2024-04-22T23:08:50.305770+0000 2024-04-21T23:18:09.619 INFO:teuthology.orchestra.run.smithi040.stdout:blocklist 172.21.15.40:0/276300080 expires 2024-04-22T23:08:50.305770+0000 2024-04-21T23:18:09.619 INFO:teuthology.orchestra.run.smithi040.stdout:blocklist 172.21.15.40:0/3689649814 expires 2024-04-22T23:08:23.743030+0000 2024-04-21T23:18:09.619 INFO:teuthology.orchestra.run.smithi040.stdout:blocklist 172.21.15.40:0/2241116715 expires 2024-04-22T23:08:23.743030+0000 2024-04-21T23:18:09.619 INFO:teuthology.orchestra.run.smithi040.stdout:blocklist 172.21.15.40:6801/1479040326 expires 2024-04-22T23:08:23.743030+0000 2024-04-21T23:18:09.620 INFO:teuthology.orchestra.run.smithi040.stdout:blocklist 172.21.15.40:6800/1479040326 expires 2024-04-22T23:08:23.743030+0000 2024-04-21T23:18:09.620 INFO:teuthology.orchestra.run.smithi040.stdout:blocklist 172.21.15.40:0/681298099 expires 2024-04-22T23:08:23.743030+0000 2024-04-21T23:18:09.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:09.619+0000 7f3ade882700 1 -- 172.21.15.40:0/2640391364 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f3ac00a2c20 msgr2=0x7f3ac00a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:09.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:09.619+0000 7f3ade882700 1 --2- 172.21.15.40:0/2640391364 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f3ac00a2c20 0x7f3ac00a50e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f3ac8003890 tx=0x7f3ac8005fd0 comp rx=0 tx=0).stop 2024-04-21T23:18:09.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:09.619+0000 7f3ade882700 1 -- 172.21.15.40:0/2640391364 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ad81034b0 msgr2=0x7f3ad8074620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:09.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:09.619+0000 7f3ade882700 1 --2- 172.21.15.40:0/2640391364 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ad81034b0 0x7f3ad8074620 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f3acc00c9b0 tx=0x7f3acc00ccc0 comp rx=0 tx=0).stop 2024-04-21T23:18:09.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:09.619+0000 7f3ade882700 1 -- 172.21.15.40:0/2640391364 shutdown_connections 2024-04-21T23:18:09.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:09.619+0000 7f3ade882700 1 --2- 172.21.15.40:0/2640391364 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3ad8103e50 0x7f3ad8074b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:09.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:09.619+0000 7f3ade882700 1 --2- 172.21.15.40:0/2640391364 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f3ac00a2c20 0x7f3ac00a50e0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:09.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:09.619+0000 7f3ade882700 1 --2- 172.21.15.40:0/2640391364 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ad81022b0 0x7f3ad8075fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:09.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:09.619+0000 7f3ade882700 1 --2- 172.21.15.40:0/2640391364 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ad81034b0 0x7f3ad8074620 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:09.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:09.619+0000 7f3ade882700 1 -- 172.21.15.40:0/2640391364 >> 172.21.15.40:0/2640391364 conn(0x7f3ad80fd850 msgr2=0x7f3ad81067b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:09.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:09.619+0000 7f3ade882700 1 -- 172.21.15.40:0/2640391364 shutdown_connections 2024-04-21T23:18:09.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:09.619+0000 7f3ade882700 1 -- 172.21.15.40:0/2640391364 wait complete. 2024-04-21T23:18:10.398 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:10 smithi040 ceph-mon[27984]: pgmap v363: 129 pgs: 129 active+clean; 641 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:18:10.398 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:10 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/2640391364' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-21T23:18:10.434 INFO:teuthology.run_tasks:Running task fs.ready... 2024-04-21T23:18:10.445 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T23:18:10.446 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:10 smithi111 ceph-mon[31085]: pgmap v363: 129 pgs: 129 active+clean; 641 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:18:10.446 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:10 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/2640391364' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-21T23:18:10.446 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:10 smithi174 ceph-mon[31409]: pgmap v363: 129 pgs: 129 active+clean; 641 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:18:10.446 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:10 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/2640391364' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-21T23:18:10.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.559+0000 7f71a9d9a640 1 -- 172.21.15.40:0/705681479 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71a4101870 msgr2=0x7f71a4101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:10.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.559+0000 7f71a9d9a640 1 --2- 172.21.15.40:0/705681479 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71a4101870 0x7f71a4101cf0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f7198009900 tx=0x7f719802bc30 comp rx=0 tx=0).stop 2024-04-21T23:18:10.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.559+0000 7f71a9d9a640 1 -- 172.21.15.40:0/705681479 shutdown_connections 2024-04-21T23:18:10.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.559+0000 7f71a9d9a640 1 --2- 172.21.15.40:0/705681479 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f71a4102230 0x7f71a4108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:10.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.559+0000 7f71a9d9a640 1 --2- 172.21.15.40:0/705681479 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71a4101870 0x7f71a4101cf0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:10.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.559+0000 7f71a9d9a640 1 --2- 172.21.15.40:0/705681479 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f71a4100680 0x7f71a4100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:10.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.559+0000 7f71a9d9a640 1 -- 172.21.15.40:0/705681479 >> 172.21.15.40:0/705681479 conn(0x7f71a40fbbf0 msgr2=0x7f71a40fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:10.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.559+0000 7f71a9d9a640 1 -- 172.21.15.40:0/705681479 shutdown_connections 2024-04-21T23:18:10.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.559+0000 7f71a9d9a640 1 -- 172.21.15.40:0/705681479 wait complete. 2024-04-21T23:18:10.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.560+0000 7f71a9d9a640 1 Processor -- start 2024-04-21T23:18:10.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.560+0000 7f71a9d9a640 1 -- start start 2024-04-21T23:18:10.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.560+0000 7f71a9d9a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71a4100680 0x7f71a4196920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:10.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.560+0000 7f71a9d9a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f71a4101870 0x7f71a4196e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:10.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.560+0000 7f71a9d9a640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f71a4102230 0x7f71a419ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:10.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.560+0000 7f71a9d9a640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f71a410c300 con 0x7f71a4100680 2024-04-21T23:18:10.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.560+0000 7f71a9d9a640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f71a410c180 con 0x7f71a4102230 2024-04-21T23:18:10.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.560+0000 7f71a9d9a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f71a410c480 con 0x7f71a4101870 2024-04-21T23:18:10.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.560+0000 7f71a37fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71a4100680 0x7f71a4196920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:10.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.560+0000 7f71a37fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71a4100680 0x7f71a4196920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46510/0 (socket says 172.21.15.40:46510) 2024-04-21T23:18:10.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.560+0000 7f71a37fe640 1 -- 172.21.15.40:0/3186851544 learned_addr learned my addr 172.21.15.40:0/3186851544 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:10.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.560+0000 7f71a2ffd640 1 --2- 172.21.15.40:0/3186851544 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f71a4101870 0x7f71a4196e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:10.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.560+0000 7f71a3fff640 1 --2- 172.21.15.40:0/3186851544 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f71a4102230 0x7f71a419ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:10.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.560+0000 7f71a37fe640 1 -- 172.21.15.40:0/3186851544 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f71a4101870 msgr2=0x7f71a4196e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:10.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.560+0000 7f71a37fe640 1 --2- 172.21.15.40:0/3186851544 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f71a4101870 0x7f71a4196e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:10.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.560+0000 7f71a37fe640 1 -- 172.21.15.40:0/3186851544 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f71a4102230 msgr2=0x7f71a419ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:10.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.560+0000 7f71a37fe640 1 --2- 172.21.15.40:0/3186851544 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f71a4102230 0x7f71a419ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:10.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.560+0000 7f71a37fe640 1 -- 172.21.15.40:0/3186851544 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71a419e540 con 0x7f71a4100680 2024-04-21T23:18:10.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.560+0000 7f71a2ffd640 1 --2- 172.21.15.40:0/3186851544 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f71a4101870 0x7f71a4196e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:18:10.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.560+0000 7f71a3fff640 1 --2- 172.21.15.40:0/3186851544 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f71a4102230 0x7f71a419ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:18:10.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.561+0000 7f71a37fe640 1 --2- 172.21.15.40:0/3186851544 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71a4100680 0x7f71a4196920 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f719400c970 tx=0x7f719400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:10.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.561+0000 7f71a0ff9640 1 -- 172.21.15.40:0/3186851544 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7194007c10 con 0x7f71a4100680 2024-04-21T23:18:10.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.561+0000 7f71a0ff9640 1 -- 172.21.15.40:0/3186851544 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7194007db0 con 0x7f71a4100680 2024-04-21T23:18:10.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.561+0000 7f71a9d9a640 1 -- 172.21.15.40:0/3186851544 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71a419e830 con 0x7f71a4100680 2024-04-21T23:18:10.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.561+0000 7f71a0ff9640 1 -- 172.21.15.40:0/3186851544 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f719400f940 con 0x7f71a4100680 2024-04-21T23:18:10.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.561+0000 7f71a9d9a640 1 -- 172.21.15.40:0/3186851544 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71a419ed80 con 0x7f71a4100680 2024-04-21T23:18:10.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.562+0000 7f71a9d9a640 1 -- 172.21.15.40:0/3186851544 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71a41079a0 con 0x7f71a4100680 2024-04-21T23:18:10.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.563+0000 7f71a0ff9640 1 -- 172.21.15.40:0/3186851544 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f719400fae0 con 0x7f71a4100680 2024-04-21T23:18:10.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.564+0000 7f71a0ff9640 1 --2- 172.21.15.40:0/3186851544 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f718c0a2b90 0x7f718c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:10.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.564+0000 7f71a2ffd640 1 --2- 172.21.15.40:0/3186851544 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f718c0a2b90 0x7f718c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:10.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.564+0000 7f71a0ff9640 1 -- 172.21.15.40:0/3186851544 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f71940cf920 con 0x7f71a4100680 2024-04-21T23:18:10.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.564+0000 7f71a2ffd640 1 --2- 172.21.15.40:0/3186851544 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f718c0a2b90 0x7f718c0a5050 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f7198036040 tx=0x7f7198000bc0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:10.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.568+0000 7f71a0ff9640 1 -- 172.21.15.40:0/3186851544 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f719409c4e0 con 0x7f71a4100680 2024-04-21T23:18:10.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.753+0000 7f71a9d9a640 1 -- 172.21.15.40:0/3186851544 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f71a4062340 con 0x7f71a4100680 2024-04-21T23:18:10.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.755+0000 7f71a0ff9640 1 -- 172.21.15.40:0/3186851544 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+24853 (secure 0 0 0) 0x7f719409bde0 con 0x7f71a4100680 2024-04-21T23:18:10.755 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:18:10.755 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":94,"fsid":"d814e4dc-0033-11ef-bc93-c7b262605968","created":"2024-04-21T23:07:52.659207+0000","modified":"2024-04-21T23:17:52.907466+0000","last_up_change":"2024-04-21T23:15:05.308821+0000","last_in_change":"2024-04-21T23:14:45.447491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:11:14.824775+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T23:15:54.322571+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T23:15:55.164493+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T23:15:58.838266+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"0e8d692a-22fe-4a87-8d99-9151f96cbb02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6803","nonce":2009758784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6805","nonce":2009758784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6809","nonce":2009758784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6807","nonce":2009758784}]},"public_addr":"172.21.15.40:6803/2009758784","cluster_addr":"172.21.15.40:6805/2009758784","heartbeat_back_addr":"172.21.15.40:6809/2009758784","heartbeat_front_addr":"172.21.15.40:6807/2009758784","state":["exists","up"]},{"osd":1,"uuid":"fe5fe11b-0e84-4359-83a7-1ba45105755a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6801","nonce":1763903836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6803","nonce":1763903836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6807","nonce":1763903836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6805","nonce":1763903836}]},"public_addr":"172.21.15.111:6801/1763903836","cluster_addr":"172.21.15.111:6803/1763903836","heartbeat_back_addr":"172.21.15.111:6807/1763903836","heartbeat_front_addr":"172.21.15.111:6805/1763903836","state":["exists","up"]},{"osd":2,"uuid":"813ad735-5cc2-4da0-ae16-6db238395fbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6801","nonce":439695825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6803","nonce":439695825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6807","nonce":439695825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6805","nonce":439695825}]},"public_addr":"172.21.15.174:6801/439695825","cluster_addr":"172.21.15.174:6803/439695825","heartbeat_back_addr":"172.21.15.174:6807/439695825","heartbeat_front_addr":"172.21.15.174:6805/439695825","state":["exists","up"]},{"osd":3,"uuid":"09becb03-b7ce-4ffc-917d-b7900ec37e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6811","nonce":3454312436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6813","nonce":3454312436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6817","nonce":3454312436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6815","nonce":3454312436}]},"public_addr":"172.21.15.40:6811/3454312436","cluster_addr":"172.21.15.40:6813/3454312436","heartbeat_back_addr":"172.21.15.40:6817/3454312436","heartbeat_front_addr":"172.21.15.40:6815/3454312436","state":["exists","up"]},{"osd":4,"uuid":"46eabfd7-d88a-4efd-94eb-04354d3a9d7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6809","nonce":1810866070}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6811","nonce":1810866070}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6815","nonce":1810866070}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6813","nonce":1810866070}]},"public_addr":"172.21.15.111:6809/1810866070","cluster_addr":"172.21.15.111:6811/1810866070","heartbeat_back_addr":"172.21.15.111:6815/1810866070","heartbeat_front_addr":"172.21.15.111:6813/1810866070","state":["exists","up"]},{"osd":5,"uuid":"6f73b582-6c65-41db-aab0-9b360f83d5c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6809","nonce":3752986711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6811","nonce":3752986711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6815","nonce":3752986711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6813","nonce":3752986711}]},"public_addr":"172.21.15.174:6809/3752986711","cluster_addr":"172.21.15.174:6811/3752986711","heartbeat_back_addr":"172.21.15.174:6815/3752986711","heartbeat_front_addr":"172.21.15.174:6813/3752986711","state":["exists","up"]},{"osd":6,"uuid":"39f305f0-12d2-4732-aaa0-cd66700af538","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6819","nonce":1618855427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6821","nonce":1618855427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6825","nonce":1618855427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6823","nonce":1618855427}]},"public_addr":"172.21.15.40:6819/1618855427","cluster_addr":"172.21.15.40:6821/1618855427","heartbeat_back_addr":"172.21.15.40:6825/1618855427","heartbeat_front_addr":"172.21.15.40:6823/1618855427","state":["exists","up"]},{"osd":7,"uuid":"769f2468-5102-490b-89f8-d0444d3169a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6817","nonce":927054162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6819","nonce":927054162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6823","nonce":927054162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6821","nonce":927054162}]},"public_addr":"172.21.15.111:6817/927054162","cluster_addr":"172.21.15.111:6819/927054162","heartbeat_back_addr":"172.21.15.111:6823/927054162","heartbeat_front_addr":"172.21.15.111:6821/927054162","state":["exists","up"]},{"osd":8,"uuid":"1a444e80-007d-473b-b972-4c814a813c22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6817","nonce":3817848746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6819","nonce":3817848746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6823","nonce":3817848746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6821","nonce":3817848746}]},"public_addr":"172.21.15.174:6817/3817848746","cluster_addr":"172.21.15.174:6819/3817848746","heartbeat_back_addr":"172.21.15.174:6823/3817848746","heartbeat_front_addr":"172.21.15.174:6821/3817848746","state":["exists","up"]},{"osd":9,"uuid":"9a6b6a56-6ae5-49fe-9e5b-cb17d686e0dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6827","nonce":1649446531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6829","nonce":1649446531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6833","nonce":1649446531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6831","nonce":1649446531}]},"public_addr":"172.21.15.40:6827/1649446531","cluster_addr":"172.21.15.40:6829/1649446531","heartbeat_back_addr":"172.21.15.40:6833/1649446531","heartbeat_front_addr":"172.21.15.40:6831/1649446531","state":["exists","up"]},{"osd":10,"uuid":"a70038ba-e2ad-4170-85cc-ea0470a2edef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6825","nonce":1227538935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6827","nonce":1227538935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6831","nonce":1227538935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6829","nonce":1227538935}]},"public_addr":"172.21.15.111:6825/1227538935","cluster_addr":"172.21.15.111:6827/1227538935","heartbeat_back_addr":"172.21.15.111:6831/1227538935","heartbeat_front_addr":"172.21.15.111:6829/1227538935","state":["exists","up"]},{"osd":11,"uuid":"781532bb-7a5e-44e3-8eb7-b584b0054666","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6825","nonce":2045865553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6827","nonce":2045865553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6831","nonce":2045865553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6829","nonce":2045865553}]},"public_addr":"172.21.15.174:6825/2045865553","cluster_addr":"172.21.15.174:6827/2045865553","heartbeat_back_addr":"172.21.15.174:6831/2045865553","heartbeat_front_addr":"172.21.15.174:6829/2045865553","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:10:24.761290+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:10:48.565428+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:11:12.307818+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:11:37.951529+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:02.832523+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:27.568879+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:52.997707+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:13:18.937274+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:13:44.708657+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:14:10.833855+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:14:37.078272+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:15:03.878076+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":1}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.2a","mappings":[{"from":3,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"4.9","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":10}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:6801/526554319":"2024-04-22T23:08:50.305770+0000","172.21.15.40:6800/526554319":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/3178763471":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/4045911364":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/276300080":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/3689649814":"2024-04-22T23:08:23.743030+0000","172.21.15.40:0/2241116715":"2024-04-22T23:08:23.743030+0000","172.21.15.40:6801/1479040326":"2024-04-22T23:08:23.743030+0000","172.21.15.40:6800/1479040326":"2024-04-22T23:08:23.743030+0000","172.21.15.40:0/681298099":"2024-04-22T23:08:23.743030+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T23:18:10.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.757+0000 7f71a9d9a640 1 -- 172.21.15.40:0/3186851544 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f718c0a2b90 msgr2=0x7f718c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:10.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.757+0000 7f71a9d9a640 1 --2- 172.21.15.40:0/3186851544 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f718c0a2b90 0x7f718c0a5050 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f7198036040 tx=0x7f7198000bc0 comp rx=0 tx=0).stop 2024-04-21T23:18:10.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.757+0000 7f71a9d9a640 1 -- 172.21.15.40:0/3186851544 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71a4100680 msgr2=0x7f71a4196920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:10.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.757+0000 7f71a9d9a640 1 --2- 172.21.15.40:0/3186851544 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71a4100680 0x7f71a4196920 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f719400c970 tx=0x7f719400ce30 comp rx=0 tx=0).stop 2024-04-21T23:18:10.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.757+0000 7f71a9d9a640 1 -- 172.21.15.40:0/3186851544 shutdown_connections 2024-04-21T23:18:10.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.757+0000 7f71a9d9a640 1 --2- 172.21.15.40:0/3186851544 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f718c0a2b90 0x7f718c0a5050 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:10.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.757+0000 7f71a9d9a640 1 --2- 172.21.15.40:0/3186851544 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f71a4102230 0x7f71a419ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:10.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.757+0000 7f71a9d9a640 1 --2- 172.21.15.40:0/3186851544 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f71a4101870 0x7f71a4196e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:10.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.757+0000 7f71a9d9a640 1 --2- 172.21.15.40:0/3186851544 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71a4100680 0x7f71a4196920 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:10.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.757+0000 7f71a9d9a640 1 -- 172.21.15.40:0/3186851544 >> 172.21.15.40:0/3186851544 conn(0x7f71a40fbbf0 msgr2=0x7f71a4104b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:10.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.757+0000 7f71a9d9a640 1 -- 172.21.15.40:0/3186851544 shutdown_connections 2024-04-21T23:18:10.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.757+0000 7f71a9d9a640 1 -- 172.21.15.40:0/3186851544 wait complete. 2024-04-21T23:18:10.768 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T23:11:14.824775+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T23:15:54.322571+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T23:15:55.164493+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T23:15:58.838266+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T23:18:10.768 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T23:18:10.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.875+0000 7f8e7d8fb640 1 -- 172.21.15.40:0/762315661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e78102250 msgr2=0x7f8e78108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:10.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.875+0000 7f8e7d8fb640 1 --2- 172.21.15.40:0/762315661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e78102250 0x7f8e78108ae0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f8e70009ca0 tx=0x7f8e7002c0f0 comp rx=0 tx=0).stop 2024-04-21T23:18:10.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.875+0000 7f8e7d8fb640 1 -- 172.21.15.40:0/762315661 shutdown_connections 2024-04-21T23:18:10.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.875+0000 7f8e7d8fb640 1 --2- 172.21.15.40:0/762315661 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e78101890 0x7f8e78101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:10.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.875+0000 7f8e7d8fb640 1 --2- 172.21.15.40:0/762315661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e78102250 0x7f8e78108ae0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:10.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.875+0000 7f8e7d8fb640 1 --2- 172.21.15.40:0/762315661 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8e781006a0 0x7f8e78100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:10.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.875+0000 7f8e7d8fb640 1 -- 172.21.15.40:0/762315661 >> 172.21.15.40:0/762315661 conn(0x7f8e780fbc30 msgr2=0x7f8e780fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:10.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.875+0000 7f8e7d8fb640 1 -- 172.21.15.40:0/762315661 shutdown_connections 2024-04-21T23:18:10.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.875+0000 7f8e7d8fb640 1 -- 172.21.15.40:0/762315661 wait complete. 2024-04-21T23:18:10.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.875+0000 7f8e7d8fb640 1 Processor -- start 2024-04-21T23:18:10.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.876+0000 7f8e7d8fb640 1 -- start start 2024-04-21T23:18:10.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.876+0000 7f8e7d8fb640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e781006a0 0x7f8e78197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:10.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.876+0000 7f8e7d8fb640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e78101890 0x7f8e78197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:10.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.876+0000 7f8e7d8fb640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8e78102250 0x7f8e7819f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:10.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.876+0000 7f8e7d8fb640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8e7810c320 con 0x7f8e781006a0 2024-04-21T23:18:10.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.876+0000 7f8e7d8fb640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f8e7810c1a0 con 0x7f8e78101890 2024-04-21T23:18:10.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.876+0000 7f8e7d8fb640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8e7810c4a0 con 0x7f8e78102250 2024-04-21T23:18:10.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.876+0000 7f8e76ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e781006a0 0x7f8e78197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:10.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.876+0000 7f8e76ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e781006a0 0x7f8e78197aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46528/0 (socket says 172.21.15.40:46528) 2024-04-21T23:18:10.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.876+0000 7f8e76ffd640 1 -- 172.21.15.40:0/2433129222 learned_addr learned my addr 172.21.15.40:0/2433129222 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:10.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.876+0000 7f8e777fe640 1 --2- 172.21.15.40:0/2433129222 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8e78102250 0x7f8e7819f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:10.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.876+0000 7f8e767fc640 1 --2- 172.21.15.40:0/2433129222 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e78101890 0x7f8e78197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:10.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.876+0000 7f8e76ffd640 1 -- 172.21.15.40:0/2433129222 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8e78102250 msgr2=0x7f8e7819f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:10.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.876+0000 7f8e76ffd640 1 --2- 172.21.15.40:0/2433129222 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8e78102250 0x7f8e7819f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:10.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.876+0000 7f8e76ffd640 1 -- 172.21.15.40:0/2433129222 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e78101890 msgr2=0x7f8e78197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:10.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.876+0000 7f8e76ffd640 1 --2- 172.21.15.40:0/2433129222 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e78101890 0x7f8e78197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:10.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.876+0000 7f8e76ffd640 1 -- 172.21.15.40:0/2433129222 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e7819f6c0 con 0x7f8e781006a0 2024-04-21T23:18:10.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.877+0000 7f8e76ffd640 1 --2- 172.21.15.40:0/2433129222 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e781006a0 0x7f8e78197aa0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f8e6800d950 tx=0x7f8e6800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:10.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.877+0000 7f8e7c8f9640 1 -- 172.21.15.40:0/2433129222 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e68017070 con 0x7f8e781006a0 2024-04-21T23:18:10.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.877+0000 7f8e7c8f9640 1 -- 172.21.15.40:0/2433129222 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e68009c30 con 0x7f8e781006a0 2024-04-21T23:18:10.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.877+0000 7f8e7d8fb640 1 -- 172.21.15.40:0/2433129222 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e7819f9b0 con 0x7f8e781006a0 2024-04-21T23:18:10.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.877+0000 7f8e7c8f9640 1 -- 172.21.15.40:0/2433129222 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e6800f600 con 0x7f8e781006a0 2024-04-21T23:18:10.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.877+0000 7f8e7d8fb640 1 -- 172.21.15.40:0/2433129222 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e7819feb0 con 0x7f8e781006a0 2024-04-21T23:18:10.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.878+0000 7f8e7d8fb640 1 -- 172.21.15.40:0/2433129222 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e781a00e0 con 0x7f8e781006a0 2024-04-21T23:18:10.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.879+0000 7f8e7c8f9640 1 -- 172.21.15.40:0/2433129222 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f8e6801a070 con 0x7f8e781006a0 2024-04-21T23:18:10.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.880+0000 7f8e7c8f9640 1 --2- 172.21.15.40:0/2433129222 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f8e600a2b90 0x7f8e600a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:10.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.880+0000 7f8e767fc640 1 --2- 172.21.15.40:0/2433129222 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f8e600a2b90 0x7f8e600a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:10.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.880+0000 7f8e7c8f9640 1 -- 172.21.15.40:0/2433129222 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f8e680ceea0 con 0x7f8e781006a0 2024-04-21T23:18:10.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.880+0000 7f8e767fc640 1 --2- 172.21.15.40:0/2433129222 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f8e600a2b90 0x7f8e600a5050 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c005fd0 tx=0x7f8e6c005850 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:10.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:10.884+0000 7f8e7c8f9640 1 -- 172.21.15.40:0/2433129222 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f8e68009de0 con 0x7f8e781006a0 2024-04-21T23:18:11.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.067+0000 7f8e7d8fb640 1 -- 172.21.15.40:0/2433129222 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f8e781a0470 con 0x7f8e781006a0 2024-04-21T23:18:11.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.068+0000 7f8e7c8f9640 1 -- 172.21.15.40:0/2433129222 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7f8e6809ba30 con 0x7f8e781006a0 2024-04-21T23:18:11.067 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-04-21T23:18:11.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.070+0000 7f8e7d8fb640 1 -- 172.21.15.40:0/2433129222 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f8e600a2b90 msgr2=0x7f8e600a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:11.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.070+0000 7f8e7d8fb640 1 --2- 172.21.15.40:0/2433129222 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f8e600a2b90 0x7f8e600a5050 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c005fd0 tx=0x7f8e6c005850 comp rx=0 tx=0).stop 2024-04-21T23:18:11.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.070+0000 7f8e7d8fb640 1 -- 172.21.15.40:0/2433129222 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e781006a0 msgr2=0x7f8e78197aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:11.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.070+0000 7f8e7d8fb640 1 --2- 172.21.15.40:0/2433129222 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e781006a0 0x7f8e78197aa0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f8e6800d950 tx=0x7f8e6800de10 comp rx=0 tx=0).stop 2024-04-21T23:18:11.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.070+0000 7f8e7d8fb640 1 -- 172.21.15.40:0/2433129222 shutdown_connections 2024-04-21T23:18:11.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.070+0000 7f8e7d8fb640 1 --2- 172.21.15.40:0/2433129222 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f8e600a2b90 0x7f8e600a5050 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:11.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.070+0000 7f8e7d8fb640 1 --2- 172.21.15.40:0/2433129222 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e78101890 0x7f8e78197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:11.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.070+0000 7f8e7d8fb640 1 --2- 172.21.15.40:0/2433129222 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8e78102250 0x7f8e7819f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:11.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.070+0000 7f8e7d8fb640 1 --2- 172.21.15.40:0/2433129222 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e781006a0 0x7f8e78197aa0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:11.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.070+0000 7f8e7d8fb640 1 -- 172.21.15.40:0/2433129222 >> 172.21.15.40:0/2433129222 conn(0x7f8e780fbc30 msgr2=0x7f8e78069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:11.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.070+0000 7f8e7d8fb640 1 -- 172.21.15.40:0/2433129222 shutdown_connections 2024-04-21T23:18:11.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.070+0000 7f8e7d8fb640 1 -- 172.21.15.40:0/2433129222 wait complete. 2024-04-21T23:18:11.079 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:11 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/3186851544' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:18:11.080 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T23:18:11.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.186+0000 7fe5bf5d7640 1 -- 172.21.15.40:0/1544740236 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe5b80ffe20 msgr2=0x7fe5b8100240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:11.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.186+0000 7fe5bf5d7640 1 --2- 172.21.15.40:0/1544740236 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe5b80ffe20 0x7fe5b8100240 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fe5b4009900 tx=0x7fe5b402bc30 comp rx=0 tx=0).stop 2024-04-21T23:18:11.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.186+0000 7fe5bf5d7640 1 -- 172.21.15.40:0/1544740236 shutdown_connections 2024-04-21T23:18:11.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.186+0000 7fe5bf5d7640 1 --2- 172.21.15.40:0/1544740236 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5b8100f70 0x7fe5b81013f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:11.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.186+0000 7fe5bf5d7640 1 --2- 172.21.15.40:0/1544740236 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe5b8101930 0x7fe5b8108250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:11.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.186+0000 7fe5bf5d7640 1 --2- 172.21.15.40:0/1544740236 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe5b80ffe20 0x7fe5b8100240 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:11.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.186+0000 7fe5bf5d7640 1 -- 172.21.15.40:0/1544740236 >> 172.21.15.40:0/1544740236 conn(0x7fe5b80fb370 msgr2=0x7fe5b80fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:11.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.186+0000 7fe5bf5d7640 1 -- 172.21.15.40:0/1544740236 shutdown_connections 2024-04-21T23:18:11.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.186+0000 7fe5bf5d7640 1 -- 172.21.15.40:0/1544740236 wait complete. 2024-04-21T23:18:11.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.187+0000 7fe5bf5d7640 1 Processor -- start 2024-04-21T23:18:11.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.187+0000 7fe5bf5d7640 1 -- start start 2024-04-21T23:18:11.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.187+0000 7fe5bf5d7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe5b80ffe20 0x7fe5b81969b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:11.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.187+0000 7fe5bf5d7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe5b8100f70 0x7fe5b8196ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:11.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.187+0000 7fe5bf5d7640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5b8101930 0x7fe5b819df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:11.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.187+0000 7fe5bf5d7640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe5b810c370 con 0x7fe5b80ffe20 2024-04-21T23:18:11.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.187+0000 7fe5bf5d7640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe5b810c1f0 con 0x7fe5b8101930 2024-04-21T23:18:11.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.187+0000 7fe5bf5d7640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe5b810c4f0 con 0x7fe5b8100f70 2024-04-21T23:18:11.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.187+0000 7fe5bd373640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe5b80ffe20 0x7fe5b81969b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:11.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.187+0000 7fe5bd373640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe5b80ffe20 0x7fe5b81969b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46542/0 (socket says 172.21.15.40:46542) 2024-04-21T23:18:11.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.187+0000 7fe5bd373640 1 -- 172.21.15.40:0/374431911 learned_addr learned my addr 172.21.15.40:0/374431911 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:11.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.187+0000 7fe5bcb72640 1 --2- 172.21.15.40:0/374431911 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe5b8100f70 0x7fe5b8196ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:11.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.187+0000 7fe5bd373640 1 -- 172.21.15.40:0/374431911 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe5b8100f70 msgr2=0x7fe5b8196ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:11.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.187+0000 7fe5bd373640 1 --2- 172.21.15.40:0/374431911 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe5b8100f70 0x7fe5b8196ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:11.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.187+0000 7fe5bd373640 1 -- 172.21.15.40:0/374431911 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5b8101930 msgr2=0x7fe5b819df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:11.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.187+0000 7fe5bd373640 1 --2- 172.21.15.40:0/374431911 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5b8101930 0x7fe5b819df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:11.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.187+0000 7fe5bd373640 1 -- 172.21.15.40:0/374431911 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5b819e5d0 con 0x7fe5b80ffe20 2024-04-21T23:18:11.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.188+0000 7fe5bd373640 1 --2- 172.21.15.40:0/374431911 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe5b80ffe20 0x7fe5b81969b0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fe5b4000c00 tx=0x7fe5b4005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:11.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.188+0000 7fe5ae7fc640 1 -- 172.21.15.40:0/374431911 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5b4043020 con 0x7fe5b80ffe20 2024-04-21T23:18:11.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.188+0000 7fe5ae7fc640 1 -- 172.21.15.40:0/374431911 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe5b40054e0 con 0x7fe5b80ffe20 2024-04-21T23:18:11.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.188+0000 7fe5bf5d7640 1 -- 172.21.15.40:0/374431911 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5b819e860 con 0x7fe5b80ffe20 2024-04-21T23:18:11.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.188+0000 7fe5ae7fc640 1 -- 172.21.15.40:0/374431911 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5b4003e60 con 0x7fe5b80ffe20 2024-04-21T23:18:11.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.188+0000 7fe5bf5d7640 1 -- 172.21.15.40:0/374431911 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5b819edb0 con 0x7fe5b80ffe20 2024-04-21T23:18:11.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.189+0000 7fe5bf5d7640 1 -- 172.21.15.40:0/374431911 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5b8107130 con 0x7fe5b80ffe20 2024-04-21T23:18:11.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.190+0000 7fe5ae7fc640 1 -- 172.21.15.40:0/374431911 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fe5b4046020 con 0x7fe5b80ffe20 2024-04-21T23:18:11.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.190+0000 7fe5ae7fc640 1 --2- 172.21.15.40:0/374431911 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fe5a40a2b90 0x7fe5a40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:11.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.191+0000 7fe5ae7fc640 1 -- 172.21.15.40:0/374431911 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fe5b40f1740 con 0x7fe5b80ffe20 2024-04-21T23:18:11.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.191+0000 7fe5bcb72640 1 --2- 172.21.15.40:0/374431911 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fe5a40a2b90 0x7fe5a40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:11.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.191+0000 7fe5bcb72640 1 --2- 172.21.15.40:0/374431911 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fe5a40a2b90 0x7fe5a40a5050 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fe5a8005d00 tx=0x7fe5a8005c50 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:11.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.195+0000 7fe5ae7fc640 1 -- 172.21.15.40:0/374431911 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fe5b404d020 con 0x7fe5b80ffe20 2024-04-21T23:18:11.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.381+0000 7fe5bf5d7640 1 -- 172.21.15.40:0/374431911 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fe5b8062370 con 0x7fe5b80ffe20 2024-04-21T23:18:11.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.381+0000 7fe5ae7fc640 1 -- 172.21.15.40:0/374431911 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7fe5b403c350 con 0x7fe5b80ffe20 2024-04-21T23:18:11.381 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-21T23:18:11.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.383+0000 7fe5bf5d7640 1 -- 172.21.15.40:0/374431911 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fe5a40a2b90 msgr2=0x7fe5a40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:11.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.383+0000 7fe5bf5d7640 1 --2- 172.21.15.40:0/374431911 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fe5a40a2b90 0x7fe5a40a5050 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fe5a8005d00 tx=0x7fe5a8005c50 comp rx=0 tx=0).stop 2024-04-21T23:18:11.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.383+0000 7fe5bf5d7640 1 -- 172.21.15.40:0/374431911 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe5b80ffe20 msgr2=0x7fe5b81969b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:11.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.383+0000 7fe5bf5d7640 1 --2- 172.21.15.40:0/374431911 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe5b80ffe20 0x7fe5b81969b0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fe5b4000c00 tx=0x7fe5b4005dd0 comp rx=0 tx=0).stop 2024-04-21T23:18:11.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.383+0000 7fe5bf5d7640 1 -- 172.21.15.40:0/374431911 shutdown_connections 2024-04-21T23:18:11.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.383+0000 7fe5bf5d7640 1 --2- 172.21.15.40:0/374431911 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fe5a40a2b90 0x7fe5a40a5050 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:11.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.383+0000 7fe5bf5d7640 1 --2- 172.21.15.40:0/374431911 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5b8101930 0x7fe5b819df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:11.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.383+0000 7fe5bf5d7640 1 --2- 172.21.15.40:0/374431911 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe5b8100f70 0x7fe5b8196ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:11.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.383+0000 7fe5bf5d7640 1 --2- 172.21.15.40:0/374431911 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe5b80ffe20 0x7fe5b81969b0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:11.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.383+0000 7fe5bf5d7640 1 -- 172.21.15.40:0/374431911 >> 172.21.15.40:0/374431911 conn(0x7fe5b80fb370 msgr2=0x7fe5b8104300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:11.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.383+0000 7fe5bf5d7640 1 -- 172.21.15.40:0/374431911 shutdown_connections 2024-04-21T23:18:11.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.383+0000 7fe5bf5d7640 1 -- 172.21.15.40:0/374431911 wait complete. 2024-04-21T23:18:11.394 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T23:18:11.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:11 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/3186851544' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:18:11.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:11 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/3186851544' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:18:11.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.500+0000 7f6002f85640 1 -- 172.21.15.40:0/2241600656 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ffc101c80 msgr2=0x7f5ffc1020a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:11.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.500+0000 7f6002f85640 1 --2- 172.21.15.40:0/2241600656 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ffc101c80 0x7f5ffc1020a0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f5ff0009930 tx=0x7f5ff002bc30 comp rx=0 tx=0).stop 2024-04-21T23:18:11.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.500+0000 7f6002f85640 1 -- 172.21.15.40:0/2241600656 shutdown_connections 2024-04-21T23:18:11.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.500+0000 7f6002f85640 1 --2- 172.21.15.40:0/2241600656 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ffc1012e0 0x7f5ffc108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:11.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.500+0000 7f6002f85640 1 --2- 172.21.15.40:0/2241600656 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5ffc100920 0x7f5ffc100da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:11.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.500+0000 7f6002f85640 1 --2- 172.21.15.40:0/2241600656 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ffc101c80 0x7f5ffc1020a0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:11.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.500+0000 7f6002f85640 1 -- 172.21.15.40:0/2241600656 >> 172.21.15.40:0/2241600656 conn(0x7f5ffc0fb370 msgr2=0x7f5ffc0fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:11.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.500+0000 7f6002f85640 1 -- 172.21.15.40:0/2241600656 shutdown_connections 2024-04-21T23:18:11.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.500+0000 7f6002f85640 1 -- 172.21.15.40:0/2241600656 wait complete. 2024-04-21T23:18:11.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.501+0000 7f6002f85640 1 Processor -- start 2024-04-21T23:18:11.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.501+0000 7f6002f85640 1 -- start start 2024-04-21T23:18:11.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.501+0000 7f6002f85640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ffc100920 0x7f5ffc1046d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:11.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.501+0000 7f6002f85640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ffc1012e0 0x7f5ffc102d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:11.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.501+0000 7f6002f85640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5ffc101c80 0x7f5ffc103250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:11.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.501+0000 7f6002f85640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5ffc10c310 con 0x7f5ffc100920 2024-04-21T23:18:11.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.501+0000 7f6002f85640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f5ffc10c190 con 0x7f5ffc1012e0 2024-04-21T23:18:11.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.501+0000 7f6002f85640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5ffc10c490 con 0x7f5ffc101c80 2024-04-21T23:18:11.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.501+0000 7f6000d21640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ffc100920 0x7f5ffc1046d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:11.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.501+0000 7f6000d21640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ffc100920 0x7f5ffc1046d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46560/0 (socket says 172.21.15.40:46560) 2024-04-21T23:18:11.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.501+0000 7f6000d21640 1 -- 172.21.15.40:0/320094489 learned_addr learned my addr 172.21.15.40:0/320094489 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:11.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.501+0000 7f6001522640 1 --2- 172.21.15.40:0/320094489 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5ffc101c80 0x7f5ffc103250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:11.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.501+0000 7f5ffbfff640 1 --2- 172.21.15.40:0/320094489 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ffc1012e0 0x7f5ffc102d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:11.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.501+0000 7f6000d21640 1 -- 172.21.15.40:0/320094489 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5ffc101c80 msgr2=0x7f5ffc103250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:11.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.501+0000 7f6000d21640 1 --2- 172.21.15.40:0/320094489 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5ffc101c80 0x7f5ffc103250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:11.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.501+0000 7f6000d21640 1 -- 172.21.15.40:0/320094489 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ffc1012e0 msgr2=0x7f5ffc102d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:11.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.501+0000 7f6000d21640 1 --2- 172.21.15.40:0/320094489 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ffc1012e0 0x7f5ffc102d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:11.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.501+0000 7f6000d21640 1 -- 172.21.15.40:0/320094489 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ffc103ae0 con 0x7f5ffc100920 2024-04-21T23:18:11.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.501+0000 7f5ffbfff640 1 --2- 172.21.15.40:0/320094489 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ffc1012e0 0x7f5ffc102d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:18:11.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.501+0000 7f6000d21640 1 --2- 172.21.15.40:0/320094489 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ffc100920 0x7f5ffc1046d0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f5ff002c7b0 tx=0x7f5ff0005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:11.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.502+0000 7f5ff9ffb640 1 -- 172.21.15.40:0/320094489 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ff0041070 con 0x7f5ffc100920 2024-04-21T23:18:11.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.502+0000 7f5ff9ffb640 1 -- 172.21.15.40:0/320094489 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ff00054e0 con 0x7f5ffc100920 2024-04-21T23:18:11.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.502+0000 7f6002f85640 1 -- 172.21.15.40:0/320094489 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ffc1a1b60 con 0x7f5ffc100920 2024-04-21T23:18:11.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.502+0000 7f5ff9ffb640 1 -- 172.21.15.40:0/320094489 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ff0034440 con 0x7f5ffc100920 2024-04-21T23:18:11.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.502+0000 7f6002f85640 1 -- 172.21.15.40:0/320094489 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ffc1a2100 con 0x7f5ffc100920 2024-04-21T23:18:11.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.503+0000 7f6002f85640 1 -- 172.21.15.40:0/320094489 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ffc1a22d0 con 0x7f5ffc100920 2024-04-21T23:18:11.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.504+0000 7f5ff9ffb640 1 -- 172.21.15.40:0/320094489 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f5ff0044070 con 0x7f5ffc100920 2024-04-21T23:18:11.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.505+0000 7f5ff9ffb640 1 --2- 172.21.15.40:0/320094489 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f5fec0a2c10 0x7f5fec0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:11.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.505+0000 7f5ffbfff640 1 --2- 172.21.15.40:0/320094489 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f5fec0a2c10 0x7f5fec0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:11.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.505+0000 7f5ff9ffb640 1 -- 172.21.15.40:0/320094489 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f5ff00f10a0 con 0x7f5ffc100920 2024-04-21T23:18:11.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.505+0000 7f5ffbfff640 1 --2- 172.21.15.40:0/320094489 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f5fec0a2c10 0x7f5fec0a50d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc104440 tx=0x7f5fe8005d90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:11.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.510+0000 7f5ff9ffb640 1 -- 172.21.15.40:0/320094489 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f5ff00bdc30 con 0x7f5ffc100920 2024-04-21T23:18:11.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.696+0000 7f6002f85640 1 -- 172.21.15.40:0/320094489 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f5ffc1a24a0 con 0x7f5ffc100920 2024-04-21T23:18:11.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.696+0000 7f5ff9ffb640 1 -- 172.21.15.40:0/320094489 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7f5ff00bd530 con 0x7f5ffc100920 2024-04-21T23:18:11.695 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-04-21T23:18:11.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.698+0000 7f6002f85640 1 -- 172.21.15.40:0/320094489 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f5fec0a2c10 msgr2=0x7f5fec0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:11.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.698+0000 7f6002f85640 1 --2- 172.21.15.40:0/320094489 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f5fec0a2c10 0x7f5fec0a50d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc104440 tx=0x7f5fe8005d90 comp rx=0 tx=0).stop 2024-04-21T23:18:11.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.698+0000 7f6002f85640 1 -- 172.21.15.40:0/320094489 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ffc100920 msgr2=0x7f5ffc1046d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:11.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.698+0000 7f6002f85640 1 --2- 172.21.15.40:0/320094489 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ffc100920 0x7f5ffc1046d0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f5ff002c7b0 tx=0x7f5ff0005dd0 comp rx=0 tx=0).stop 2024-04-21T23:18:11.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.698+0000 7f6002f85640 1 -- 172.21.15.40:0/320094489 shutdown_connections 2024-04-21T23:18:11.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.698+0000 7f6002f85640 1 --2- 172.21.15.40:0/320094489 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f5fec0a2c10 0x7f5fec0a50d0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:11.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.698+0000 7f6002f85640 1 --2- 172.21.15.40:0/320094489 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ffc1012e0 0x7f5ffc102d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:11.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.698+0000 7f6002f85640 1 --2- 172.21.15.40:0/320094489 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5ffc101c80 0x7f5ffc103250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:11.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.698+0000 7f6002f85640 1 --2- 172.21.15.40:0/320094489 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ffc100920 0x7f5ffc1046d0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:11.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.698+0000 7f6002f85640 1 -- 172.21.15.40:0/320094489 >> 172.21.15.40:0/320094489 conn(0x7f5ffc0fb370 msgr2=0x7f5ffc0fce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:11.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.698+0000 7f6002f85640 1 -- 172.21.15.40:0/320094489 shutdown_connections 2024-04-21T23:18:11.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.698+0000 7f6002f85640 1 -- 172.21.15.40:0/320094489 wait complete. 2024-04-21T23:18:11.708 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T23:18:11.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.814+0000 7f434d51e640 1 -- 172.21.15.40:0/1354720523 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4348101890 msgr2=0x7f4348101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:11.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.814+0000 7f434d51e640 1 --2- 172.21.15.40:0/1354720523 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4348101890 0x7f4348101d10 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f4330009900 tx=0x7f433002bc30 comp rx=0 tx=0).stop 2024-04-21T23:18:11.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.814+0000 7f434d51e640 1 -- 172.21.15.40:0/1354720523 shutdown_connections 2024-04-21T23:18:11.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.814+0000 7f434d51e640 1 --2- 172.21.15.40:0/1354720523 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4348102250 0x7f4348108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:11.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.814+0000 7f434d51e640 1 --2- 172.21.15.40:0/1354720523 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4348101890 0x7f4348101d10 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:11.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.814+0000 7f434d51e640 1 --2- 172.21.15.40:0/1354720523 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43481006a0 0x7f4348100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:11.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.814+0000 7f434d51e640 1 -- 172.21.15.40:0/1354720523 >> 172.21.15.40:0/1354720523 conn(0x7f43480fbc30 msgr2=0x7f43480fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:11.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.814+0000 7f434d51e640 1 -- 172.21.15.40:0/1354720523 shutdown_connections 2024-04-21T23:18:11.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.814+0000 7f434d51e640 1 -- 172.21.15.40:0/1354720523 wait complete. 2024-04-21T23:18:11.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.815+0000 7f434d51e640 1 Processor -- start 2024-04-21T23:18:11.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.815+0000 7f434d51e640 1 -- start start 2024-04-21T23:18:11.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.815+0000 7f434d51e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43481006a0 0x7f43481969f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:11.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.815+0000 7f434d51e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4348101890 0x7f4348196f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:11.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.815+0000 7f434d51e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4348102250 0x7f434819dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:11.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.815+0000 7f434d51e640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f434810c350 con 0x7f4348102250 2024-04-21T23:18:11.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.815+0000 7f434d51e640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f434810c1d0 con 0x7f43481006a0 2024-04-21T23:18:11.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.815+0000 7f434d51e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f434810c4d0 con 0x7f4348101890 2024-04-21T23:18:11.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.815+0000 7f43477fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4348102250 0x7f434819dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:11.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.815+0000 7f43477fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4348102250 0x7f434819dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45890/0 (socket says 172.21.15.40:45890) 2024-04-21T23:18:11.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.815+0000 7f43477fe640 1 -- 172.21.15.40:0/1523823700 learned_addr learned my addr 172.21.15.40:0/1523823700 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:11.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.816+0000 7f4346ffd640 1 --2- 172.21.15.40:0/1523823700 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43481006a0 0x7f43481969f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:11.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.816+0000 7f433ffff640 1 --2- 172.21.15.40:0/1523823700 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4348101890 0x7f4348196f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:11.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.816+0000 7f43477fe640 1 -- 172.21.15.40:0/1523823700 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4348101890 msgr2=0x7f4348196f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:11.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.816+0000 7f43477fe640 1 --2- 172.21.15.40:0/1523823700 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4348101890 0x7f4348196f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:11.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.816+0000 7f43477fe640 1 -- 172.21.15.40:0/1523823700 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43481006a0 msgr2=0x7f43481969f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:11.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.816+0000 7f43477fe640 1 --2- 172.21.15.40:0/1523823700 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43481006a0 0x7f43481969f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:11.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.816+0000 7f43477fe640 1 -- 172.21.15.40:0/1523823700 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f434819e610 con 0x7f4348102250 2024-04-21T23:18:11.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.816+0000 7f43477fe640 1 --2- 172.21.15.40:0/1523823700 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4348102250 0x7f434819dfa0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f4340009fd0 tx=0x7f434000b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:11.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.816+0000 7f4344ff9640 1 -- 172.21.15.40:0/1523823700 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4340017070 con 0x7f4348102250 2024-04-21T23:18:11.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.816+0000 7f434d51e640 1 -- 172.21.15.40:0/1523823700 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f434819e900 con 0x7f4348102250 2024-04-21T23:18:11.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.816+0000 7f4344ff9640 1 -- 172.21.15.40:0/1523823700 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4340003e50 con 0x7f4348102250 2024-04-21T23:18:11.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.816+0000 7f4344ff9640 1 -- 172.21.15.40:0/1523823700 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43400106c0 con 0x7f4348102250 2024-04-21T23:18:11.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.816+0000 7f434d51e640 1 -- 172.21.15.40:0/1523823700 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f434819ee20 con 0x7f4348102250 2024-04-21T23:18:11.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.817+0000 7f434d51e640 1 -- 172.21.15.40:0/1523823700 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4348107a90 con 0x7f4348102250 2024-04-21T23:18:11.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.818+0000 7f4344ff9640 1 -- 172.21.15.40:0/1523823700 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f4340003970 con 0x7f4348102250 2024-04-21T23:18:11.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.819+0000 7f4344ff9640 1 --2- 172.21.15.40:0/1523823700 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f43340a2b90 0x7f43340a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:11.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.819+0000 7f4346ffd640 1 --2- 172.21.15.40:0/1523823700 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f43340a2b90 0x7f43340a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:11.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.819+0000 7f4344ff9640 1 -- 172.21.15.40:0/1523823700 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f43400cf2e0 con 0x7f4348102250 2024-04-21T23:18:11.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.819+0000 7f4346ffd640 1 --2- 172.21.15.40:0/1523823700 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f43340a2b90 0x7f43340a5050 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f43481016f0 tx=0x7f4338005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:11.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:11.823+0000 7f4344ff9640 1 -- 172.21.15.40:0/1523823700 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f434009be10 con 0x7f4348102250 2024-04-21T23:18:12.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.007+0000 7f434d51e640 1 -- 172.21.15.40:0/1523823700 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f4348062340 con 0x7f4348102250 2024-04-21T23:18:12.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.007+0000 7f4344ff9640 1 -- 172.21.15.40:0/1523823700 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7f434009b710 con 0x7f4348102250 2024-04-21T23:18:12.006 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-21T23:18:12.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.009+0000 7f434d51e640 1 -- 172.21.15.40:0/1523823700 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f43340a2b90 msgr2=0x7f43340a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:12.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.009+0000 7f434d51e640 1 --2- 172.21.15.40:0/1523823700 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f43340a2b90 0x7f43340a5050 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f43481016f0 tx=0x7f4338005f90 comp rx=0 tx=0).stop 2024-04-21T23:18:12.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.009+0000 7f434d51e640 1 -- 172.21.15.40:0/1523823700 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4348102250 msgr2=0x7f434819dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:12.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.009+0000 7f434d51e640 1 --2- 172.21.15.40:0/1523823700 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4348102250 0x7f434819dfa0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f4340009fd0 tx=0x7f434000b260 comp rx=0 tx=0).stop 2024-04-21T23:18:12.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.009+0000 7f434d51e640 1 -- 172.21.15.40:0/1523823700 shutdown_connections 2024-04-21T23:18:12.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.009+0000 7f434d51e640 1 --2- 172.21.15.40:0/1523823700 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f43340a2b90 0x7f43340a5050 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:12.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.009+0000 7f434d51e640 1 --2- 172.21.15.40:0/1523823700 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4348102250 0x7f434819dfa0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:12.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.009+0000 7f434d51e640 1 --2- 172.21.15.40:0/1523823700 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4348101890 0x7f4348196f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:12.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.009+0000 7f434d51e640 1 --2- 172.21.15.40:0/1523823700 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43481006a0 0x7f43481969f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:12.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.009+0000 7f434d51e640 1 -- 172.21.15.40:0/1523823700 >> 172.21.15.40:0/1523823700 conn(0x7f43480fbc30 msgr2=0x7f4348104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:12.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.009+0000 7f434d51e640 1 -- 172.21.15.40:0/1523823700 shutdown_connections 2024-04-21T23:18:12.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.009+0000 7f434d51e640 1 -- 172.21.15.40:0/1523823700 wait complete. 2024-04-21T23:18:12.020 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:18:12.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.122+0000 7f37722fa640 1 -- 172.21.15.40:0/1107019943 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f376c1017b0 msgr2=0x7f376c101bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:12.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.122+0000 7f37722fa640 1 --2- 172.21.15.40:0/1107019943 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f376c1017b0 0x7f376c101bd0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f375c009900 tx=0x7f375c02bc30 comp rx=0 tx=0).stop 2024-04-21T23:18:12.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.122+0000 7f37722fa640 1 -- 172.21.15.40:0/1107019943 shutdown_connections 2024-04-21T23:18:12.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.122+0000 7f37722fa640 1 --2- 172.21.15.40:0/1107019943 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f376c103360 0x7f376c109bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:12.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.122+0000 7f37722fa640 1 --2- 172.21.15.40:0/1107019943 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f376c1029a0 0x7f376c102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:12.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.122+0000 7f37722fa640 1 --2- 172.21.15.40:0/1107019943 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f376c1017b0 0x7f376c101bd0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:12.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.122+0000 7f37722fa640 1 -- 172.21.15.40:0/1107019943 >> 172.21.15.40:0/1107019943 conn(0x7f376c0fcd40 msgr2=0x7f376c0ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:12.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.122+0000 7f37722fa640 1 -- 172.21.15.40:0/1107019943 shutdown_connections 2024-04-21T23:18:12.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.122+0000 7f37722fa640 1 -- 172.21.15.40:0/1107019943 wait complete. 2024-04-21T23:18:12.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.123+0000 7f37722fa640 1 Processor -- start 2024-04-21T23:18:12.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.123+0000 7f37722fa640 1 -- start start 2024-04-21T23:18:12.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.123+0000 7f37722fa640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f376c1017b0 0x7f376c197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:12.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.123+0000 7f37722fa640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f376c1029a0 0x7f376c197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:12.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.123+0000 7f37722fa640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f376c103360 0x7f376c19f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:12.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.123+0000 7f37722fa640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f376c10d420 con 0x7f376c1029a0 2024-04-21T23:18:12.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.123+0000 7f37722fa640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f376c10d2a0 con 0x7f376c1017b0 2024-04-21T23:18:12.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.123+0000 7f37722fa640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f376c10d5a0 con 0x7f376c103360 2024-04-21T23:18:12.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.123+0000 7f376b7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f376c1029a0 0x7f376c197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:12.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.123+0000 7f376b7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f376c1029a0 0x7f376c197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45906/0 (socket says 172.21.15.40:45906) 2024-04-21T23:18:12.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.123+0000 7f376b7fe640 1 -- 172.21.15.40:0/323817127 learned_addr learned my addr 172.21.15.40:0/323817127 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:12.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.123+0000 7f3770897640 1 --2- 172.21.15.40:0/323817127 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f376c103360 0x7f376c19f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:12.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.123+0000 7f376bfff640 1 --2- 172.21.15.40:0/323817127 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f376c1017b0 0x7f376c197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:12.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.123+0000 7f376b7fe640 1 -- 172.21.15.40:0/323817127 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f376c103360 msgr2=0x7f376c19f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:12.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.123+0000 7f376b7fe640 1 --2- 172.21.15.40:0/323817127 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f376c103360 0x7f376c19f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:12.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.123+0000 7f376b7fe640 1 -- 172.21.15.40:0/323817127 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f376c1017b0 msgr2=0x7f376c197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:12.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.123+0000 7f376b7fe640 1 --2- 172.21.15.40:0/323817127 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f376c1017b0 0x7f376c197ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:12.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.123+0000 7f376b7fe640 1 -- 172.21.15.40:0/323817127 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f376c19f6d0 con 0x7f376c1029a0 2024-04-21T23:18:12.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.124+0000 7f376b7fe640 1 --2- 172.21.15.40:0/323817127 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f376c1029a0 0x7f376c197ff0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f376000d770 tx=0x7f376000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:12.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.124+0000 7f37697fa640 1 -- 172.21.15.40:0/323817127 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3760017070 con 0x7f376c1029a0 2024-04-21T23:18:12.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.124+0000 7f37697fa640 1 -- 172.21.15.40:0/323817127 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3760005500 con 0x7f376c1029a0 2024-04-21T23:18:12.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.124+0000 7f37722fa640 1 -- 172.21.15.40:0/323817127 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f376c19f9c0 con 0x7f376c1029a0 2024-04-21T23:18:12.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.124+0000 7f37697fa640 1 -- 172.21.15.40:0/323817127 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3760004e00 con 0x7f376c1029a0 2024-04-21T23:18:12.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.124+0000 7f37722fa640 1 -- 172.21.15.40:0/323817127 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f376c19fec0 con 0x7f376c1029a0 2024-04-21T23:18:12.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.125+0000 7f37722fa640 1 -- 172.21.15.40:0/323817127 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f376c108b70 con 0x7f376c1029a0 2024-04-21T23:18:12.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.126+0000 7f37697fa640 1 -- 172.21.15.40:0/323817127 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f3760004fa0 con 0x7f376c1029a0 2024-04-21T23:18:12.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.126+0000 7f37697fa640 1 --2- 172.21.15.40:0/323817127 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f37540a2b90 0x7f37540a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:12.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.127+0000 7f376bfff640 1 --2- 172.21.15.40:0/323817127 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f37540a2b90 0x7f37540a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:12.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.127+0000 7f37697fa640 1 -- 172.21.15.40:0/323817127 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f37600ceef0 con 0x7f376c1029a0 2024-04-21T23:18:12.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.127+0000 7f376bfff640 1 --2- 172.21.15.40:0/323817127 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f37540a2b90 0x7f37540a5050 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f375c036040 tx=0x7f375c005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:12.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.131+0000 7f37697fa640 1 -- 172.21.15.40:0/323817127 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f376009ba80 con 0x7f376c1029a0 2024-04-21T23:18:12.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.361+0000 7f37722fa640 1 -- 172.21.15.40:0/323817127 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f376c02d170 con 0x7f376c1029a0 2024-04-21T23:18:12.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.361+0000 7f37697fa640 1 -- 172.21.15.40:0/323817127 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11882 (secure 0 0 0) 0x7f376009b380 con 0x7f376c1029a0 2024-04-21T23:18:12.361 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:18:12.361 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/1571948830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1571948830},{"type":"v1","addr":"172.21.15.40:6835","nonce":1571948830}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/969356642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":969356642},{"type":"v1","addr":"172.21.15.40:6837","nonce":969356642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6839/3025057845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":3025057845},{"type":"v1","addr":"172.21.15.40:6839","nonce":3025057845}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6841/3350488804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":3350488804},{"type":"v1","addr":"172.21.15.40:6841","nonce":3350488804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/3438493203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":3438493203},{"type":"v1","addr":"172.21.15.174:6833","nonce":3438493203}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/3105752678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":3105752678},{"type":"v1","addr":"172.21.15.111:6833","nonce":3105752678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T23:15:55.742233+0000","modified":"2024-04-21T23:16:22.498043+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24521,"mds_1":24502,"mds_2":24494},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24461":{"gid":24461,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6835/2704462092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2704462092},{"type":"v1","addr":"172.21.15.111:6835","nonce":2704462092}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/3905733045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3905733045},{"type":"v1","addr":"172.21.15.174:6835","nonce":3905733045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6837/465579746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":465579746},{"type":"v1","addr":"172.21.15.174:6837","nonce":465579746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/2703764182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":2703764182},{"type":"v1","addr":"172.21.15.111:6837","nonce":2703764182}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24502":{"gid":24502,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.174:6839/2467509274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":2467509274},{"type":"v1","addr":"172.21.15.174:6839","nonce":2467509274}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3712235548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3712235548},{"type":"v1","addr":"172.21.15.111:6839","nonce":3712235548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24494,24502,24521]},"id":1}]} 2024-04-21T23:18:12.361 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 24 2024-04-21T23:18:12.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.364+0000 7f37722fa640 1 -- 172.21.15.40:0/323817127 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f37540a2b90 msgr2=0x7f37540a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:12.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.364+0000 7f37722fa640 1 --2- 172.21.15.40:0/323817127 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f37540a2b90 0x7f37540a5050 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f375c036040 tx=0x7f375c005fd0 comp rx=0 tx=0).stop 2024-04-21T23:18:12.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.364+0000 7f37722fa640 1 -- 172.21.15.40:0/323817127 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f376c1029a0 msgr2=0x7f376c197ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:12.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.364+0000 7f37722fa640 1 --2- 172.21.15.40:0/323817127 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f376c1029a0 0x7f376c197ff0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f376000d770 tx=0x7f376000dc30 comp rx=0 tx=0).stop 2024-04-21T23:18:12.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.364+0000 7f37722fa640 1 -- 172.21.15.40:0/323817127 shutdown_connections 2024-04-21T23:18:12.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.364+0000 7f37722fa640 1 --2- 172.21.15.40:0/323817127 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f37540a2b90 0x7f37540a5050 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:12.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.364+0000 7f37722fa640 1 --2- 172.21.15.40:0/323817127 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f376c103360 0x7f376c19f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:12.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.364+0000 7f37722fa640 1 --2- 172.21.15.40:0/323817127 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f376c1029a0 0x7f376c197ff0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:12.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.364+0000 7f37722fa640 1 --2- 172.21.15.40:0/323817127 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f376c1017b0 0x7f376c197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:12.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.364+0000 7f37722fa640 1 -- 172.21.15.40:0/323817127 >> 172.21.15.40:0/323817127 conn(0x7f376c0fcd40 msgr2=0x7f376c105d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:12.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.364+0000 7f37722fa640 1 -- 172.21.15.40:0/323817127 shutdown_connections 2024-04-21T23:18:12.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.364+0000 7f37722fa640 1 -- 172.21.15.40:0/323817127 wait complete. 2024-04-21T23:18:12.373 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:12 smithi040 ceph-mon[27984]: pgmap v364: 129 pgs: 129 active+clean; 641 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:18:12.373 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:12 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/2433129222' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:18:12.373 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:12 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/374431911' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T23:18:12.373 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:12 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/320094489' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T23:18:12.373 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:12 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/1523823700' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T23:18:12.377 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T23:18:12.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:12 smithi174 ceph-mon[31409]: pgmap v364: 129 pgs: 129 active+clean; 641 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:18:12.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:12 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/2433129222' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:18:12.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:12 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/374431911' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T23:18:12.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:12 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/320094489' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T23:18:12.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:12 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/1523823700' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T23:18:12.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:12 smithi111 ceph-mon[31085]: pgmap v364: 129 pgs: 129 active+clean; 641 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:18:12.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:12 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/2433129222' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:18:12.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:12 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/374431911' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T23:18:12.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:12 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/320094489' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T23:18:12.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:12 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/1523823700' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T23:18:12.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.480+0000 7f522f5d7640 1 -- 172.21.15.40:0/3813466045 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5228101890 msgr2=0x7f5228101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:12.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.480+0000 7f522f5d7640 1 --2- 172.21.15.40:0/3813466045 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5228101890 0x7f5228101d10 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f5218009900 tx=0x7f521802bc30 comp rx=0 tx=0).stop 2024-04-21T23:18:12.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.480+0000 7f522f5d7640 1 -- 172.21.15.40:0/3813466045 shutdown_connections 2024-04-21T23:18:12.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.480+0000 7f522f5d7640 1 --2- 172.21.15.40:0/3813466045 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5228102250 0x7f5228108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:12.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.480+0000 7f522f5d7640 1 --2- 172.21.15.40:0/3813466045 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5228101890 0x7f5228101d10 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:12.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.480+0000 7f522f5d7640 1 --2- 172.21.15.40:0/3813466045 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52281006a0 0x7f5228100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:12.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.480+0000 7f522f5d7640 1 -- 172.21.15.40:0/3813466045 >> 172.21.15.40:0/3813466045 conn(0x7f52280fbc30 msgr2=0x7f52280fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:12.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.480+0000 7f522f5d7640 1 -- 172.21.15.40:0/3813466045 shutdown_connections 2024-04-21T23:18:12.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.480+0000 7f522f5d7640 1 -- 172.21.15.40:0/3813466045 wait complete. 2024-04-21T23:18:12.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.480+0000 7f522f5d7640 1 Processor -- start 2024-04-21T23:18:12.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.480+0000 7f522f5d7640 1 -- start start 2024-04-21T23:18:12.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.480+0000 7f522f5d7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52281006a0 0x7f5228196960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:12.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.481+0000 7f522f5d7640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5228101890 0x7f5228196ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:12.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.481+0000 7f522f5d7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5228102250 0x7f522819df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:12.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.481+0000 7f522f5d7640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f522810c320 con 0x7f5228102250 2024-04-21T23:18:12.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.481+0000 7f522f5d7640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f522810c1a0 con 0x7f5228101890 2024-04-21T23:18:12.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.481+0000 7f522f5d7640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f522810c4a0 con 0x7f52281006a0 2024-04-21T23:18:12.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.481+0000 7f522db74640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5228102250 0x7f522819df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:12.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.481+0000 7f522db74640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5228102250 0x7f522819df10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45922/0 (socket says 172.21.15.40:45922) 2024-04-21T23:18:12.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.481+0000 7f522db74640 1 -- 172.21.15.40:0/596923679 learned_addr learned my addr 172.21.15.40:0/596923679 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:12.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.481+0000 7f522cb72640 1 --2- 172.21.15.40:0/596923679 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5228101890 0x7f5228196ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:12.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.481+0000 7f522d373640 1 --2- 172.21.15.40:0/596923679 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52281006a0 0x7f5228196960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:12.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.481+0000 7f522db74640 1 -- 172.21.15.40:0/596923679 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52281006a0 msgr2=0x7f5228196960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:12.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.481+0000 7f522db74640 1 --2- 172.21.15.40:0/596923679 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52281006a0 0x7f5228196960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:12.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.481+0000 7f522db74640 1 -- 172.21.15.40:0/596923679 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5228101890 msgr2=0x7f5228196ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:12.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.481+0000 7f522db74640 1 --2- 172.21.15.40:0/596923679 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5228101890 0x7f5228196ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:12.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.481+0000 7f522db74640 1 -- 172.21.15.40:0/596923679 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f522819e580 con 0x7f5228102250 2024-04-21T23:18:12.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.481+0000 7f522d373640 1 --2- 172.21.15.40:0/596923679 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52281006a0 0x7f5228196960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:18:12.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.481+0000 7f522db74640 1 --2- 172.21.15.40:0/596923679 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5228102250 0x7f522819df10 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f522000cce0 tx=0x7f522000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:12.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.481+0000 7f521e7fc640 1 -- 172.21.15.40:0/596923679 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f522000f450 con 0x7f5228102250 2024-04-21T23:18:12.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.481+0000 7f521e7fc640 1 -- 172.21.15.40:0/596923679 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f522000fad0 con 0x7f5228102250 2024-04-21T23:18:12.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.481+0000 7f522f5d7640 1 -- 172.21.15.40:0/596923679 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f522819e870 con 0x7f5228102250 2024-04-21T23:18:12.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.481+0000 7f521e7fc640 1 -- 172.21.15.40:0/596923679 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5220018c10 con 0x7f5228102250 2024-04-21T23:18:12.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.481+0000 7f522f5d7640 1 -- 172.21.15.40:0/596923679 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f522819ed40 con 0x7f5228102250 2024-04-21T23:18:12.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.482+0000 7f522f5d7640 1 -- 172.21.15.40:0/596923679 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52281079c0 con 0x7f5228102250 2024-04-21T23:18:12.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.484+0000 7f521e7fc640 1 -- 172.21.15.40:0/596923679 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f522000f5f0 con 0x7f5228102250 2024-04-21T23:18:12.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.484+0000 7f521e7fc640 1 --2- 172.21.15.40:0/596923679 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f52140a2b90 0x7f52140a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:12.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.484+0000 7f522d373640 1 --2- 172.21.15.40:0/596923679 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f52140a2b90 0x7f52140a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:12.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.485+0000 7f521e7fc640 1 -- 172.21.15.40:0/596923679 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f52200cfb60 con 0x7f5228102250 2024-04-21T23:18:12.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.485+0000 7f522d373640 1 --2- 172.21.15.40:0/596923679 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f52140a2b90 0x7f52140a5050 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f52240058c0 tx=0x7f5224005850 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:12.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.488+0000 7f521e7fc640 1 -- 172.21.15.40:0/596923679 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f5220014030 con 0x7f5228102250 2024-04-21T23:18:12.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.674+0000 7f522f5d7640 1 -- 172.21.15.40:0/596923679 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f522802d170 con 0x7f5228102250 2024-04-21T23:18:12.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.676+0000 7f521e7fc640 1 -- 172.21.15.40:0/596923679 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+24853 (secure 0 0 0) 0x7f522009c780 con 0x7f5228102250 2024-04-21T23:18:12.675 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:18:12.675 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":94,"fsid":"d814e4dc-0033-11ef-bc93-c7b262605968","created":"2024-04-21T23:07:52.659207+0000","modified":"2024-04-21T23:17:52.907466+0000","last_up_change":"2024-04-21T23:15:05.308821+0000","last_in_change":"2024-04-21T23:14:45.447491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:11:14.824775+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T23:15:54.322571+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T23:15:55.164493+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T23:15:58.838266+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"0e8d692a-22fe-4a87-8d99-9151f96cbb02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6803","nonce":2009758784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6805","nonce":2009758784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6809","nonce":2009758784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6807","nonce":2009758784}]},"public_addr":"172.21.15.40:6803/2009758784","cluster_addr":"172.21.15.40:6805/2009758784","heartbeat_back_addr":"172.21.15.40:6809/2009758784","heartbeat_front_addr":"172.21.15.40:6807/2009758784","state":["exists","up"]},{"osd":1,"uuid":"fe5fe11b-0e84-4359-83a7-1ba45105755a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6801","nonce":1763903836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6803","nonce":1763903836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6807","nonce":1763903836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6805","nonce":1763903836}]},"public_addr":"172.21.15.111:6801/1763903836","cluster_addr":"172.21.15.111:6803/1763903836","heartbeat_back_addr":"172.21.15.111:6807/1763903836","heartbeat_front_addr":"172.21.15.111:6805/1763903836","state":["exists","up"]},{"osd":2,"uuid":"813ad735-5cc2-4da0-ae16-6db238395fbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6801","nonce":439695825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6803","nonce":439695825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6807","nonce":439695825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6805","nonce":439695825}]},"public_addr":"172.21.15.174:6801/439695825","cluster_addr":"172.21.15.174:6803/439695825","heartbeat_back_addr":"172.21.15.174:6807/439695825","heartbeat_front_addr":"172.21.15.174:6805/439695825","state":["exists","up"]},{"osd":3,"uuid":"09becb03-b7ce-4ffc-917d-b7900ec37e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6811","nonce":3454312436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6813","nonce":3454312436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6817","nonce":3454312436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6815","nonce":3454312436}]},"public_addr":"172.21.15.40:6811/3454312436","cluster_addr":"172.21.15.40:6813/3454312436","heartbeat_back_addr":"172.21.15.40:6817/3454312436","heartbeat_front_addr":"172.21.15.40:6815/3454312436","state":["exists","up"]},{"osd":4,"uuid":"46eabfd7-d88a-4efd-94eb-04354d3a9d7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6809","nonce":1810866070}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6811","nonce":1810866070}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6815","nonce":1810866070}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6813","nonce":1810866070}]},"public_addr":"172.21.15.111:6809/1810866070","cluster_addr":"172.21.15.111:6811/1810866070","heartbeat_back_addr":"172.21.15.111:6815/1810866070","heartbeat_front_addr":"172.21.15.111:6813/1810866070","state":["exists","up"]},{"osd":5,"uuid":"6f73b582-6c65-41db-aab0-9b360f83d5c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6809","nonce":3752986711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6811","nonce":3752986711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6815","nonce":3752986711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6813","nonce":3752986711}]},"public_addr":"172.21.15.174:6809/3752986711","cluster_addr":"172.21.15.174:6811/3752986711","heartbeat_back_addr":"172.21.15.174:6815/3752986711","heartbeat_front_addr":"172.21.15.174:6813/3752986711","state":["exists","up"]},{"osd":6,"uuid":"39f305f0-12d2-4732-aaa0-cd66700af538","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6819","nonce":1618855427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6821","nonce":1618855427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6825","nonce":1618855427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6823","nonce":1618855427}]},"public_addr":"172.21.15.40:6819/1618855427","cluster_addr":"172.21.15.40:6821/1618855427","heartbeat_back_addr":"172.21.15.40:6825/1618855427","heartbeat_front_addr":"172.21.15.40:6823/1618855427","state":["exists","up"]},{"osd":7,"uuid":"769f2468-5102-490b-89f8-d0444d3169a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6817","nonce":927054162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6819","nonce":927054162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6823","nonce":927054162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6821","nonce":927054162}]},"public_addr":"172.21.15.111:6817/927054162","cluster_addr":"172.21.15.111:6819/927054162","heartbeat_back_addr":"172.21.15.111:6823/927054162","heartbeat_front_addr":"172.21.15.111:6821/927054162","state":["exists","up"]},{"osd":8,"uuid":"1a444e80-007d-473b-b972-4c814a813c22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6817","nonce":3817848746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6819","nonce":3817848746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6823","nonce":3817848746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6821","nonce":3817848746}]},"public_addr":"172.21.15.174:6817/3817848746","cluster_addr":"172.21.15.174:6819/3817848746","heartbeat_back_addr":"172.21.15.174:6823/3817848746","heartbeat_front_addr":"172.21.15.174:6821/3817848746","state":["exists","up"]},{"osd":9,"uuid":"9a6b6a56-6ae5-49fe-9e5b-cb17d686e0dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6827","nonce":1649446531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6829","nonce":1649446531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6833","nonce":1649446531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6831","nonce":1649446531}]},"public_addr":"172.21.15.40:6827/1649446531","cluster_addr":"172.21.15.40:6829/1649446531","heartbeat_back_addr":"172.21.15.40:6833/1649446531","heartbeat_front_addr":"172.21.15.40:6831/1649446531","state":["exists","up"]},{"osd":10,"uuid":"a70038ba-e2ad-4170-85cc-ea0470a2edef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6825","nonce":1227538935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6827","nonce":1227538935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6831","nonce":1227538935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6829","nonce":1227538935}]},"public_addr":"172.21.15.111:6825/1227538935","cluster_addr":"172.21.15.111:6827/1227538935","heartbeat_back_addr":"172.21.15.111:6831/1227538935","heartbeat_front_addr":"172.21.15.111:6829/1227538935","state":["exists","up"]},{"osd":11,"uuid":"781532bb-7a5e-44e3-8eb7-b584b0054666","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6825","nonce":2045865553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6827","nonce":2045865553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6831","nonce":2045865553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6829","nonce":2045865553}]},"public_addr":"172.21.15.174:6825/2045865553","cluster_addr":"172.21.15.174:6827/2045865553","heartbeat_back_addr":"172.21.15.174:6831/2045865553","heartbeat_front_addr":"172.21.15.174:6829/2045865553","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:10:24.761290+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:10:48.565428+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:11:12.307818+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:11:37.951529+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:02.832523+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:27.568879+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:52.997707+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:13:18.937274+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:13:44.708657+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:14:10.833855+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:14:37.078272+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:15:03.878076+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":1}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.2a","mappings":[{"from":3,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"4.9","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":10}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:6801/526554319":"2024-04-22T23:08:50.305770+0000","172.21.15.40:6800/526554319":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/3178763471":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/4045911364":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/276300080":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/3689649814":"2024-04-22T23:08:23.743030+0000","172.21.15.40:0/2241116715":"2024-04-22T23:08:23.743030+0000","172.21.15.40:6801/1479040326":"2024-04-22T23:08:23.743030+0000","172.21.15.40:6800/1479040326":"2024-04-22T23:08:23.743030+0000","172.21.15.40:0/681298099":"2024-04-22T23:08:23.743030+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T23:18:12.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.678+0000 7f522f5d7640 1 -- 172.21.15.40:0/596923679 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f52140a2b90 msgr2=0x7f52140a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:12.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.678+0000 7f522f5d7640 1 --2- 172.21.15.40:0/596923679 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f52140a2b90 0x7f52140a5050 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f52240058c0 tx=0x7f5224005850 comp rx=0 tx=0).stop 2024-04-21T23:18:12.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.678+0000 7f522f5d7640 1 -- 172.21.15.40:0/596923679 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5228102250 msgr2=0x7f522819df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:12.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.678+0000 7f522f5d7640 1 --2- 172.21.15.40:0/596923679 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5228102250 0x7f522819df10 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f522000cce0 tx=0x7f522000a620 comp rx=0 tx=0).stop 2024-04-21T23:18:12.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.678+0000 7f522f5d7640 1 -- 172.21.15.40:0/596923679 shutdown_connections 2024-04-21T23:18:12.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.678+0000 7f522f5d7640 1 --2- 172.21.15.40:0/596923679 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f52140a2b90 0x7f52140a5050 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:12.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.678+0000 7f522f5d7640 1 --2- 172.21.15.40:0/596923679 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5228101890 0x7f5228196ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:12.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.678+0000 7f522f5d7640 1 --2- 172.21.15.40:0/596923679 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5228102250 0x7f522819df10 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:12.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.678+0000 7f522f5d7640 1 --2- 172.21.15.40:0/596923679 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52281006a0 0x7f5228196960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:12.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.678+0000 7f522f5d7640 1 -- 172.21.15.40:0/596923679 >> 172.21.15.40:0/596923679 conn(0x7f52280fbc30 msgr2=0x7f5228104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:12.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.678+0000 7f522f5d7640 1 -- 172.21.15.40:0/596923679 shutdown_connections 2024-04-21T23:18:12.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.678+0000 7f522f5d7640 1 -- 172.21.15.40:0/596923679 wait complete. 2024-04-21T23:18:12.689 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T23:11:14.824775+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T23:15:54.322571+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T23:15:55.164493+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T23:15:58.838266+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T23:18:12.689 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T23:18:12.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.795+0000 7ff92ac88640 1 -- 172.21.15.40:0/2086755414 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff924103350 msgr2=0x7ff924109be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:12.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.795+0000 7ff92ac88640 1 --2- 172.21.15.40:0/2086755414 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff924103350 0x7ff924109be0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7ff91c009ca0 tx=0x7ff91c02c0f0 comp rx=0 tx=0).stop 2024-04-21T23:18:12.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.795+0000 7ff92ac88640 1 -- 172.21.15.40:0/2086755414 shutdown_connections 2024-04-21T23:18:12.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.795+0000 7ff92ac88640 1 --2- 172.21.15.40:0/2086755414 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff924102990 0x7ff924102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:12.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.795+0000 7ff92ac88640 1 --2- 172.21.15.40:0/2086755414 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff924103350 0x7ff924109be0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:12.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.795+0000 7ff92ac88640 1 --2- 172.21.15.40:0/2086755414 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff9241017a0 0x7ff924101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:12.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.795+0000 7ff92ac88640 1 -- 172.21.15.40:0/2086755414 >> 172.21.15.40:0/2086755414 conn(0x7ff9240fcd10 msgr2=0x7ff9240ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:12.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.795+0000 7ff92ac88640 1 -- 172.21.15.40:0/2086755414 shutdown_connections 2024-04-21T23:18:12.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.795+0000 7ff92ac88640 1 -- 172.21.15.40:0/2086755414 wait complete. 2024-04-21T23:18:12.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.796+0000 7ff92ac88640 1 Processor -- start 2024-04-21T23:18:12.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.796+0000 7ff92ac88640 1 -- start start 2024-04-21T23:18:12.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.796+0000 7ff92ac88640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9241017a0 0x7ff9240754f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:12.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.796+0000 7ff92ac88640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff924102990 0x7ff924073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:12.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.796+0000 7ff92ac88640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff924103350 0x7ff924074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:12.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.796+0000 7ff92ac88640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff92410d400 con 0x7ff9241017a0 2024-04-21T23:18:12.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.796+0000 7ff92ac88640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff92410d280 con 0x7ff924103350 2024-04-21T23:18:12.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.796+0000 7ff92ac88640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff92410d580 con 0x7ff924102990 2024-04-21T23:18:12.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.796+0000 7ff928a24640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9241017a0 0x7ff9240754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:12.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.796+0000 7ff928a24640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9241017a0 0x7ff9240754f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45932/0 (socket says 172.21.15.40:45932) 2024-04-21T23:18:12.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.796+0000 7ff928a24640 1 -- 172.21.15.40:0/3015213118 learned_addr learned my addr 172.21.15.40:0/3015213118 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:12.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.796+0000 7ff929225640 1 --2- 172.21.15.40:0/3015213118 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff924103350 0x7ff924074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:12.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.796+0000 7ff91bfff640 1 --2- 172.21.15.40:0/3015213118 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff924102990 0x7ff924073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:12.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.796+0000 7ff928a24640 1 -- 172.21.15.40:0/3015213118 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff924102990 msgr2=0x7ff924073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:12.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.796+0000 7ff928a24640 1 --2- 172.21.15.40:0/3015213118 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff924102990 0x7ff924073b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:12.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.796+0000 7ff928a24640 1 -- 172.21.15.40:0/3015213118 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff924103350 msgr2=0x7ff924074070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:12.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.796+0000 7ff928a24640 1 --2- 172.21.15.40:0/3015213118 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff924103350 0x7ff924074070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:12.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.796+0000 7ff928a24640 1 -- 172.21.15.40:0/3015213118 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff924074900 con 0x7ff9241017a0 2024-04-21T23:18:12.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.797+0000 7ff928a24640 1 --2- 172.21.15.40:0/3015213118 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9241017a0 0x7ff9240754f0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7ff92000c970 tx=0x7ff92000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:12.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.797+0000 7ff919ffb640 1 -- 172.21.15.40:0/3015213118 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff920007c10 con 0x7ff9241017a0 2024-04-21T23:18:12.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.797+0000 7ff919ffb640 1 -- 172.21.15.40:0/3015213118 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff920007db0 con 0x7ff9241017a0 2024-04-21T23:18:12.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.797+0000 7ff92ac88640 1 -- 172.21.15.40:0/3015213118 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9241a1b40 con 0x7ff9241017a0 2024-04-21T23:18:12.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.797+0000 7ff919ffb640 1 -- 172.21.15.40:0/3015213118 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff92000f940 con 0x7ff9241017a0 2024-04-21T23:18:12.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.797+0000 7ff92ac88640 1 -- 172.21.15.40:0/3015213118 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9241a2090 con 0x7ff9241017a0 2024-04-21T23:18:12.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.798+0000 7ff92ac88640 1 -- 172.21.15.40:0/3015213118 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff924108a60 con 0x7ff9241017a0 2024-04-21T23:18:12.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.799+0000 7ff919ffb640 1 -- 172.21.15.40:0/3015213118 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7ff92000fae0 con 0x7ff9241017a0 2024-04-21T23:18:12.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.800+0000 7ff919ffb640 1 --2- 172.21.15.40:0/3015213118 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7ff90c0a2b90 0x7ff90c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:12.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.800+0000 7ff91bfff640 1 --2- 172.21.15.40:0/3015213118 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7ff90c0a2b90 0x7ff90c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:12.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.800+0000 7ff919ffb640 1 -- 172.21.15.40:0/3015213118 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7ff9200cf890 con 0x7ff9241017a0 2024-04-21T23:18:12.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.800+0000 7ff91bfff640 1 --2- 172.21.15.40:0/3015213118 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7ff90c0a2b90 0x7ff90c0a5050 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7ff924075260 tx=0x7ff91400a4c0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:12.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.804+0000 7ff919ffb640 1 -- 172.21.15.40:0/3015213118 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7ff92009c4b0 con 0x7ff9241017a0 2024-04-21T23:18:12.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.988+0000 7ff92ac88640 1 -- 172.21.15.40:0/3015213118 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff924062340 con 0x7ff9241017a0 2024-04-21T23:18:12.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.989+0000 7ff919ffb640 1 -- 172.21.15.40:0/3015213118 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7ff92009bdb0 con 0x7ff9241017a0 2024-04-21T23:18:12.988 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-04-21T23:18:12.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.991+0000 7ff92ac88640 1 -- 172.21.15.40:0/3015213118 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7ff90c0a2b90 msgr2=0x7ff90c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:12.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.991+0000 7ff92ac88640 1 --2- 172.21.15.40:0/3015213118 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7ff90c0a2b90 0x7ff90c0a5050 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7ff924075260 tx=0x7ff91400a4c0 comp rx=0 tx=0).stop 2024-04-21T23:18:12.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.991+0000 7ff92ac88640 1 -- 172.21.15.40:0/3015213118 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9241017a0 msgr2=0x7ff9240754f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:12.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.991+0000 7ff92ac88640 1 --2- 172.21.15.40:0/3015213118 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9241017a0 0x7ff9240754f0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7ff92000c970 tx=0x7ff92000ce30 comp rx=0 tx=0).stop 2024-04-21T23:18:12.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.991+0000 7ff92ac88640 1 -- 172.21.15.40:0/3015213118 shutdown_connections 2024-04-21T23:18:12.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.991+0000 7ff92ac88640 1 --2- 172.21.15.40:0/3015213118 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7ff90c0a2b90 0x7ff90c0a5050 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:12.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.991+0000 7ff92ac88640 1 --2- 172.21.15.40:0/3015213118 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff924103350 0x7ff924074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:12.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.991+0000 7ff92ac88640 1 --2- 172.21.15.40:0/3015213118 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff924102990 0x7ff924073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:12.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.991+0000 7ff92ac88640 1 --2- 172.21.15.40:0/3015213118 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9241017a0 0x7ff9240754f0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:12.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.991+0000 7ff92ac88640 1 -- 172.21.15.40:0/3015213118 >> 172.21.15.40:0/3015213118 conn(0x7ff9240fcd10 msgr2=0x7ff924105c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:12.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.991+0000 7ff92ac88640 1 -- 172.21.15.40:0/3015213118 shutdown_connections 2024-04-21T23:18:12.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:12.991+0000 7ff92ac88640 1 -- 172.21.15.40:0/3015213118 wait complete. 2024-04-21T23:18:13.001 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T23:18:13.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.107+0000 7f0e331b8640 1 -- 172.21.15.40:0/524037726 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e2c109650 msgr2=0x7f0e2c10bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:13.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.107+0000 7f0e331b8640 1 --2- 172.21.15.40:0/524037726 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e2c109650 0x7f0e2c10bae0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f0e24009ca0 tx=0x7f0e2402c180 comp rx=0 tx=0).stop 2024-04-21T23:18:13.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.107+0000 7f0e331b8640 1 -- 172.21.15.40:0/524037726 shutdown_connections 2024-04-21T23:18:13.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.107+0000 7f0e331b8640 1 --2- 172.21.15.40:0/524037726 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e2c074120 0x7f0e2c074580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:13.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.107+0000 7f0e331b8640 1 --2- 172.21.15.40:0/524037726 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e2c109650 0x7f0e2c10bae0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:13.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.107+0000 7f0e331b8640 1 --2- 172.21.15.40:0/524037726 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0e2c075700 0x7f0e2c073b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:13.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.107+0000 7f0e331b8640 1 -- 172.21.15.40:0/524037726 >> 172.21.15.40:0/524037726 conn(0x7f0e2c0fcd10 msgr2=0x7f0e2c0ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:13.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.107+0000 7f0e331b8640 1 -- 172.21.15.40:0/524037726 shutdown_connections 2024-04-21T23:18:13.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.107+0000 7f0e331b8640 1 -- 172.21.15.40:0/524037726 wait complete. 2024-04-21T23:18:13.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.108+0000 7f0e331b8640 1 Processor -- start 2024-04-21T23:18:13.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.108+0000 7f0e331b8640 1 -- start start 2024-04-21T23:18:13.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.108+0000 7f0e331b8640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e2c074120 0x7f0e2c199c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:13.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.108+0000 7f0e331b8640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e2c075700 0x7f0e2c19a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:13.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.108+0000 7f0e331b8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0e2c109650 0x7f0e2c1a11d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:13.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.108+0000 7f0e331b8640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0e2c10f630 con 0x7f0e2c074120 2024-04-21T23:18:13.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.108+0000 7f0e331b8640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f0e2c10f4b0 con 0x7f0e2c075700 2024-04-21T23:18:13.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.108+0000 7f0e331b8640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0e2c10f7b0 con 0x7f0e2c109650 2024-04-21T23:18:13.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.108+0000 7f0e30f54640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e2c074120 0x7f0e2c199c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:13.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.108+0000 7f0e30f54640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e2c074120 0x7f0e2c199c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45948/0 (socket says 172.21.15.40:45948) 2024-04-21T23:18:13.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.108+0000 7f0e30f54640 1 -- 172.21.15.40:0/3835126950 learned_addr learned my addr 172.21.15.40:0/3835126950 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:13.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.108+0000 7f0e2bfff640 1 --2- 172.21.15.40:0/3835126950 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e2c075700 0x7f0e2c19a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:13.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.108+0000 7f0e31755640 1 --2- 172.21.15.40:0/3835126950 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0e2c109650 0x7f0e2c1a11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:13.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.108+0000 7f0e30f54640 1 -- 172.21.15.40:0/3835126950 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0e2c109650 msgr2=0x7f0e2c1a11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:13.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.108+0000 7f0e30f54640 1 --2- 172.21.15.40:0/3835126950 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0e2c109650 0x7f0e2c1a11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:13.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.108+0000 7f0e30f54640 1 -- 172.21.15.40:0/3835126950 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e2c075700 msgr2=0x7f0e2c19a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:13.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.108+0000 7f0e30f54640 1 --2- 172.21.15.40:0/3835126950 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e2c075700 0x7f0e2c19a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:13.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.109+0000 7f0e30f54640 1 -- 172.21.15.40:0/3835126950 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e2c1a1870 con 0x7f0e2c074120 2024-04-21T23:18:13.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.109+0000 7f0e31755640 1 --2- 172.21.15.40:0/3835126950 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0e2c109650 0x7f0e2c1a11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:18:13.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.109+0000 7f0e30f54640 1 --2- 172.21.15.40:0/3835126950 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e2c074120 0x7f0e2c199c60 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c00d770 tx=0x7f0e1c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:13.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.109+0000 7f0e29ffb640 1 -- 172.21.15.40:0/3835126950 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e1c017070 con 0x7f0e2c074120 2024-04-21T23:18:13.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.109+0000 7f0e29ffb640 1 -- 172.21.15.40:0/3835126950 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e1c005500 con 0x7f0e2c074120 2024-04-21T23:18:13.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.109+0000 7f0e331b8640 1 -- 172.21.15.40:0/3835126950 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e2c1a1b60 con 0x7f0e2c074120 2024-04-21T23:18:13.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.109+0000 7f0e29ffb640 1 -- 172.21.15.40:0/3835126950 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e1c004e00 con 0x7f0e2c074120 2024-04-21T23:18:13.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.109+0000 7f0e331b8640 1 -- 172.21.15.40:0/3835126950 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e2c1a2060 con 0x7f0e2c074120 2024-04-21T23:18:13.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.110+0000 7f0e331b8640 1 -- 172.21.15.40:0/3835126950 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e2c108410 con 0x7f0e2c074120 2024-04-21T23:18:13.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.111+0000 7f0e29ffb640 1 -- 172.21.15.40:0/3835126950 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f0e1c004fa0 con 0x7f0e2c074120 2024-04-21T23:18:13.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.112+0000 7f0e29ffb640 1 --2- 172.21.15.40:0/3835126950 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f0e140a2b90 0x7f0e140a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:13.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.112+0000 7f0e2bfff640 1 --2- 172.21.15.40:0/3835126950 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f0e140a2b90 0x7f0e140a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:13.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.112+0000 7f0e29ffb640 1 -- 172.21.15.40:0/3835126950 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f0e1c0cef80 con 0x7f0e2c074120 2024-04-21T23:18:13.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.112+0000 7f0e2bfff640 1 --2- 172.21.15.40:0/3835126950 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f0e140a2b90 0x7f0e140a5050 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f0e2000bbe0 tx=0x7f0e2000b390 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:13.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.116+0000 7f0e29ffb640 1 -- 172.21.15.40:0/3835126950 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f0e1c09bab0 con 0x7f0e2c074120 2024-04-21T23:18:13.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.300+0000 7f0e331b8640 1 -- 172.21.15.40:0/3835126950 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f0e2c0623e0 con 0x7f0e2c074120 2024-04-21T23:18:13.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.300+0000 7f0e29ffb640 1 -- 172.21.15.40:0/3835126950 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7f0e1c09b3b0 con 0x7f0e2c074120 2024-04-21T23:18:13.299 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-21T23:18:13.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.302+0000 7f0e331b8640 1 -- 172.21.15.40:0/3835126950 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f0e140a2b90 msgr2=0x7f0e140a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:13.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.302+0000 7f0e331b8640 1 --2- 172.21.15.40:0/3835126950 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f0e140a2b90 0x7f0e140a5050 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f0e2000bbe0 tx=0x7f0e2000b390 comp rx=0 tx=0).stop 2024-04-21T23:18:13.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.302+0000 7f0e331b8640 1 -- 172.21.15.40:0/3835126950 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e2c074120 msgr2=0x7f0e2c199c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:13.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.302+0000 7f0e331b8640 1 --2- 172.21.15.40:0/3835126950 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e2c074120 0x7f0e2c199c60 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c00d770 tx=0x7f0e1c00dc30 comp rx=0 tx=0).stop 2024-04-21T23:18:13.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.302+0000 7f0e331b8640 1 -- 172.21.15.40:0/3835126950 shutdown_connections 2024-04-21T23:18:13.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.302+0000 7f0e331b8640 1 --2- 172.21.15.40:0/3835126950 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f0e140a2b90 0x7f0e140a5050 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:13.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.302+0000 7f0e331b8640 1 --2- 172.21.15.40:0/3835126950 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e2c075700 0x7f0e2c19a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:13.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.302+0000 7f0e331b8640 1 --2- 172.21.15.40:0/3835126950 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0e2c109650 0x7f0e2c1a11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:13.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.302+0000 7f0e331b8640 1 --2- 172.21.15.40:0/3835126950 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e2c074120 0x7f0e2c199c60 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:13.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.302+0000 7f0e331b8640 1 -- 172.21.15.40:0/3835126950 >> 172.21.15.40:0/3835126950 conn(0x7f0e2c0fcd10 msgr2=0x7f0e2c1055e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:13.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.302+0000 7f0e331b8640 1 -- 172.21.15.40:0/3835126950 shutdown_connections 2024-04-21T23:18:13.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.302+0000 7f0e331b8640 1 -- 172.21.15.40:0/3835126950 wait complete. 2024-04-21T23:18:13.311 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:13 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/323817127' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:18:13.311 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:13 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/596923679' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:18:13.312 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:13 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/3015213118' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:18:13.313 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T23:18:13.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.418+0000 7f7bc21af640 1 -- 172.21.15.40:0/2846133740 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7bbc101890 msgr2=0x7f7bbc101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:13.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.418+0000 7f7bc21af640 1 --2- 172.21.15.40:0/2846133740 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7bbc101890 0x7f7bbc101d10 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f7bb0009900 tx=0x7f7bb002bc30 comp rx=0 tx=0).stop 2024-04-21T23:18:13.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.418+0000 7f7bc21af640 1 -- 172.21.15.40:0/2846133740 shutdown_connections 2024-04-21T23:18:13.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.418+0000 7f7bc21af640 1 --2- 172.21.15.40:0/2846133740 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7bbc102250 0x7f7bbc108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:13.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.418+0000 7f7bc21af640 1 --2- 172.21.15.40:0/2846133740 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7bbc101890 0x7f7bbc101d10 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:13.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.418+0000 7f7bc21af640 1 --2- 172.21.15.40:0/2846133740 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7bbc1006a0 0x7f7bbc100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:13.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.418+0000 7f7bc21af640 1 -- 172.21.15.40:0/2846133740 >> 172.21.15.40:0/2846133740 conn(0x7f7bbc0fbc30 msgr2=0x7f7bbc0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:13.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.418+0000 7f7bc21af640 1 -- 172.21.15.40:0/2846133740 shutdown_connections 2024-04-21T23:18:13.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.418+0000 7f7bc21af640 1 -- 172.21.15.40:0/2846133740 wait complete. 2024-04-21T23:18:13.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.419+0000 7f7bc21af640 1 Processor -- start 2024-04-21T23:18:13.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.419+0000 7f7bc21af640 1 -- start start 2024-04-21T23:18:13.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.419+0000 7f7bc21af640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7bbc1006a0 0x7f7bbc06a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:13.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.419+0000 7f7bc21af640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7bbc101890 0x7f7bbc19c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:13.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.419+0000 7f7bc21af640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7bbc102250 0x7f7bbc19eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:13.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.419+0000 7f7bc21af640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7bbc10c320 con 0x7f7bbc1006a0 2024-04-21T23:18:13.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.419+0000 7f7bc21af640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f7bbc10c1a0 con 0x7f7bbc101890 2024-04-21T23:18:13.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.419+0000 7f7bc21af640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7bbc10c4a0 con 0x7f7bbc102250 2024-04-21T23:18:13.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.419+0000 7f7bbb7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7bbc1006a0 0x7f7bbc06a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:13.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.419+0000 7f7bbb7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7bbc1006a0 0x7f7bbc06a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45968/0 (socket says 172.21.15.40:45968) 2024-04-21T23:18:13.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.419+0000 7f7bbb7fe640 1 -- 172.21.15.40:0/3292457023 learned_addr learned my addr 172.21.15.40:0/3292457023 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:13.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.420+0000 7f7bbbfff640 1 --2- 172.21.15.40:0/3292457023 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7bbc102250 0x7f7bbc19eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:13.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.420+0000 7f7bbaffd640 1 --2- 172.21.15.40:0/3292457023 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7bbc101890 0x7f7bbc19c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:13.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.420+0000 7f7bbb7fe640 1 -- 172.21.15.40:0/3292457023 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7bbc102250 msgr2=0x7f7bbc19eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:13.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.420+0000 7f7bbb7fe640 1 --2- 172.21.15.40:0/3292457023 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7bbc102250 0x7f7bbc19eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:13.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.420+0000 7f7bbb7fe640 1 -- 172.21.15.40:0/3292457023 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7bbc101890 msgr2=0x7f7bbc19c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:13.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.420+0000 7f7bbb7fe640 1 --2- 172.21.15.40:0/3292457023 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7bbc101890 0x7f7bbc19c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:13.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.420+0000 7f7bbb7fe640 1 -- 172.21.15.40:0/3292457023 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bbc19f770 con 0x7f7bbc1006a0 2024-04-21T23:18:13.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.420+0000 7f7bbb7fe640 1 --2- 172.21.15.40:0/3292457023 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7bbc1006a0 0x7f7bbc06a8d0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f7bac00d950 tx=0x7f7bac00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:13.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.420+0000 7f7bb8ff9640 1 -- 172.21.15.40:0/3292457023 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bac017070 con 0x7f7bbc1006a0 2024-04-21T23:18:13.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.420+0000 7f7bb8ff9640 1 -- 172.21.15.40:0/3292457023 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7bac009c30 con 0x7f7bbc1006a0 2024-04-21T23:18:13.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.420+0000 7f7bc21af640 1 -- 172.21.15.40:0/3292457023 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bbc19fa60 con 0x7f7bbc1006a0 2024-04-21T23:18:13.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.420+0000 7f7bb8ff9640 1 -- 172.21.15.40:0/3292457023 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bac00f600 con 0x7f7bbc1006a0 2024-04-21T23:18:13.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.420+0000 7f7bc21af640 1 -- 172.21.15.40:0/3292457023 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bbc19ff30 con 0x7f7bbc1006a0 2024-04-21T23:18:13.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.421+0000 7f7bc21af640 1 -- 172.21.15.40:0/3292457023 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7bbc107a80 con 0x7f7bbc1006a0 2024-04-21T23:18:13.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.422+0000 7f7bb8ff9640 1 -- 172.21.15.40:0/3292457023 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f7bac01a070 con 0x7f7bbc1006a0 2024-04-21T23:18:13.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.423+0000 7f7bb8ff9640 1 --2- 172.21.15.40:0/3292457023 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f7ba40a2b90 0x7f7ba40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:13.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.423+0000 7f7bbaffd640 1 --2- 172.21.15.40:0/3292457023 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f7ba40a2b90 0x7f7ba40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:13.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.423+0000 7f7bb8ff9640 1 -- 172.21.15.40:0/3292457023 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f7bac0cef00 con 0x7f7bbc1006a0 2024-04-21T23:18:13.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.424+0000 7f7bbaffd640 1 --2- 172.21.15.40:0/3292457023 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f7ba40a2b90 0x7f7ba40a5050 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f7bb0006010 tx=0x7f7bb0000bc0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:13.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.427+0000 7f7bb8ff9640 1 -- 172.21.15.40:0/3292457023 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f7bac009de0 con 0x7f7bbc1006a0 2024-04-21T23:18:13.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:13 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/323817127' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:18:13.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:13 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/596923679' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:18:13.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:13 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/3015213118' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:18:13.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:13 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/323817127' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:18:13.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:13 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/596923679' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:18:13.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:13 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/3015213118' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:18:13.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.614+0000 7f7bc21af640 1 -- 172.21.15.40:0/3292457023 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f7bbc02d3b0 con 0x7f7bbc1006a0 2024-04-21T23:18:13.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.615+0000 7f7bb8ff9640 1 -- 172.21.15.40:0/3292457023 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7f7bac09ba30 con 0x7f7bbc1006a0 2024-04-21T23:18:13.614 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-04-21T23:18:13.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.617+0000 7f7bc21af640 1 -- 172.21.15.40:0/3292457023 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f7ba40a2b90 msgr2=0x7f7ba40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:13.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.617+0000 7f7bc21af640 1 --2- 172.21.15.40:0/3292457023 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f7ba40a2b90 0x7f7ba40a5050 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f7bb0006010 tx=0x7f7bb0000bc0 comp rx=0 tx=0).stop 2024-04-21T23:18:13.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.617+0000 7f7bc21af640 1 -- 172.21.15.40:0/3292457023 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7bbc1006a0 msgr2=0x7f7bbc06a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:13.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.617+0000 7f7bc21af640 1 --2- 172.21.15.40:0/3292457023 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7bbc1006a0 0x7f7bbc06a8d0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f7bac00d950 tx=0x7f7bac00de10 comp rx=0 tx=0).stop 2024-04-21T23:18:13.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.617+0000 7f7bc21af640 1 -- 172.21.15.40:0/3292457023 shutdown_connections 2024-04-21T23:18:13.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.617+0000 7f7bc21af640 1 --2- 172.21.15.40:0/3292457023 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f7ba40a2b90 0x7f7ba40a5050 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:13.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.617+0000 7f7bc21af640 1 --2- 172.21.15.40:0/3292457023 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7bbc101890 0x7f7bbc19c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:13.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.617+0000 7f7bc21af640 1 --2- 172.21.15.40:0/3292457023 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7bbc102250 0x7f7bbc19eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:13.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.617+0000 7f7bc21af640 1 --2- 172.21.15.40:0/3292457023 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7bbc1006a0 0x7f7bbc06a8d0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:13.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.617+0000 7f7bc21af640 1 -- 172.21.15.40:0/3292457023 >> 172.21.15.40:0/3292457023 conn(0x7f7bbc0fbc30 msgr2=0x7f7bbc104c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:13.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.617+0000 7f7bc21af640 1 -- 172.21.15.40:0/3292457023 shutdown_connections 2024-04-21T23:18:13.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.617+0000 7f7bc21af640 1 -- 172.21.15.40:0/3292457023 wait complete. 2024-04-21T23:18:13.627 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T23:18:13.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.733+0000 7f6ff27a2640 1 -- 172.21.15.40:0/383972884 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6fec102870 msgr2=0x7f6fec102cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:13.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.733+0000 7f6ff27a2640 1 --2- 172.21.15.40:0/383972884 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6fec102870 0x7f6fec102cf0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc009900 tx=0x7f6fdc02bc30 comp rx=0 tx=0).stop 2024-04-21T23:18:13.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.733+0000 7f6ff27a2640 1 -- 172.21.15.40:0/383972884 shutdown_connections 2024-04-21T23:18:13.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.733+0000 7f6ff27a2640 1 --2- 172.21.15.40:0/383972884 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6fec103230 0x7f6fec109ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:13.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.733+0000 7f6ff27a2640 1 --2- 172.21.15.40:0/383972884 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6fec102870 0x7f6fec102cf0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:13.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.733+0000 7f6ff27a2640 1 --2- 172.21.15.40:0/383972884 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6fec101680 0x7f6fec101aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:13.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.733+0000 7f6ff27a2640 1 -- 172.21.15.40:0/383972884 >> 172.21.15.40:0/383972884 conn(0x7f6fec0fcc10 msgr2=0x7f6fec0ff030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:13.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.733+0000 7f6ff27a2640 1 -- 172.21.15.40:0/383972884 shutdown_connections 2024-04-21T23:18:13.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.733+0000 7f6ff27a2640 1 -- 172.21.15.40:0/383972884 wait complete. 2024-04-21T23:18:13.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.734+0000 7f6ff27a2640 1 Processor -- start 2024-04-21T23:18:13.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.734+0000 7f6ff27a2640 1 -- start start 2024-04-21T23:18:13.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.734+0000 7f6ff27a2640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6fec101680 0x7f6fec197980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:13.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.734+0000 7f6ff27a2640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6fec102870 0x7f6fec197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:13.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.734+0000 7f6ff27a2640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6fec103230 0x7f6fec19ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:13.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.734+0000 7f6ff27a2640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6fec10d360 con 0x7f6fec101680 2024-04-21T23:18:13.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.734+0000 7f6ff27a2640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f6fec10d1e0 con 0x7f6fec102870 2024-04-21T23:18:13.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.734+0000 7f6ff27a2640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6fec10d4e0 con 0x7f6fec103230 2024-04-21T23:18:13.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.734+0000 7f6ff17a0640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6fec101680 0x7f6fec197980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:13.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.734+0000 7f6ff17a0640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6fec101680 0x7f6fec197980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45992/0 (socket says 172.21.15.40:45992) 2024-04-21T23:18:13.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.734+0000 7f6ff17a0640 1 -- 172.21.15.40:0/1993999169 learned_addr learned my addr 172.21.15.40:0/1993999169 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:13.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.734+0000 7f6ff0f9f640 1 --2- 172.21.15.40:0/1993999169 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6fec102870 0x7f6fec197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:13.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.734+0000 7f6ff17a0640 1 -- 172.21.15.40:0/1993999169 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6fec103230 msgr2=0x7f6fec19ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:13.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.734+0000 7f6ff1fa1640 1 --2- 172.21.15.40:0/1993999169 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6fec103230 0x7f6fec19ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:13.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.734+0000 7f6ff17a0640 1 --2- 172.21.15.40:0/1993999169 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6fec103230 0x7f6fec19ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:13.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.734+0000 7f6ff17a0640 1 -- 172.21.15.40:0/1993999169 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6fec102870 msgr2=0x7f6fec197ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:13.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.734+0000 7f6ff17a0640 1 --2- 172.21.15.40:0/1993999169 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6fec102870 0x7f6fec197ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:13.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.734+0000 7f6ff17a0640 1 -- 172.21.15.40:0/1993999169 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fec19f5a0 con 0x7f6fec101680 2024-04-21T23:18:13.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.734+0000 7f6ff0f9f640 1 --2- 172.21.15.40:0/1993999169 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6fec102870 0x7f6fec197ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:18:13.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.734+0000 7f6ff1fa1640 1 --2- 172.21.15.40:0/1993999169 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6fec103230 0x7f6fec19ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:18:13.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.735+0000 7f6ff17a0640 1 --2- 172.21.15.40:0/1993999169 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6fec101680 0x7f6fec197980 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f6fe800c970 tx=0x7f6fe800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:13.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.735+0000 7f6fe27fc640 1 -- 172.21.15.40:0/1993999169 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fe8007c10 con 0x7f6fec101680 2024-04-21T23:18:13.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.735+0000 7f6fe27fc640 1 -- 172.21.15.40:0/1993999169 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6fe8007db0 con 0x7f6fec101680 2024-04-21T23:18:13.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.735+0000 7f6ff27a2640 1 -- 172.21.15.40:0/1993999169 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fec19f890 con 0x7f6fec101680 2024-04-21T23:18:13.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.735+0000 7f6fe27fc640 1 -- 172.21.15.40:0/1993999169 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fe800f940 con 0x7f6fec101680 2024-04-21T23:18:13.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.735+0000 7f6ff27a2640 1 -- 172.21.15.40:0/1993999169 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fec19fde0 con 0x7f6fec101680 2024-04-21T23:18:13.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.736+0000 7f6ff27a2640 1 -- 172.21.15.40:0/1993999169 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6fec108940 con 0x7f6fec101680 2024-04-21T23:18:13.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.737+0000 7f6fe27fc640 1 -- 172.21.15.40:0/1993999169 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f6fe800fae0 con 0x7f6fec101680 2024-04-21T23:18:13.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.738+0000 7f6fe27fc640 1 --2- 172.21.15.40:0/1993999169 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f6fd80a2b90 0x7f6fd80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:13.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.738+0000 7f6ff0f9f640 1 --2- 172.21.15.40:0/1993999169 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f6fd80a2b90 0x7f6fd80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:13.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.738+0000 7f6fe27fc640 1 -- 172.21.15.40:0/1993999169 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f6fe80cf840 con 0x7f6fec101680 2024-04-21T23:18:13.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.738+0000 7f6ff0f9f640 1 --2- 172.21.15.40:0/1993999169 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f6fd80a2b90 0x7f6fd80a5050 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f6fec198e90 tx=0x7f6fdc000bc0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:13.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.743+0000 7f6fe27fc640 1 -- 172.21.15.40:0/1993999169 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f6fe809c460 con 0x7f6fec101680 2024-04-21T23:18:13.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.925+0000 7f6ff27a2640 1 -- 172.21.15.40:0/1993999169 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f6fec02d420 con 0x7f6fec101680 2024-04-21T23:18:13.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.925+0000 7f6fe27fc640 1 -- 172.21.15.40:0/1993999169 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7f6fe809bd60 con 0x7f6fec101680 2024-04-21T23:18:13.925 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-21T23:18:13.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.927+0000 7f6ff27a2640 1 -- 172.21.15.40:0/1993999169 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f6fd80a2b90 msgr2=0x7f6fd80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:13.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.927+0000 7f6ff27a2640 1 --2- 172.21.15.40:0/1993999169 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f6fd80a2b90 0x7f6fd80a5050 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f6fec198e90 tx=0x7f6fdc000bc0 comp rx=0 tx=0).stop 2024-04-21T23:18:13.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.927+0000 7f6ff27a2640 1 -- 172.21.15.40:0/1993999169 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6fec101680 msgr2=0x7f6fec197980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:13.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.927+0000 7f6ff27a2640 1 --2- 172.21.15.40:0/1993999169 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6fec101680 0x7f6fec197980 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f6fe800c970 tx=0x7f6fe800ce30 comp rx=0 tx=0).stop 2024-04-21T23:18:13.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.927+0000 7f6ff27a2640 1 -- 172.21.15.40:0/1993999169 shutdown_connections 2024-04-21T23:18:13.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.927+0000 7f6ff27a2640 1 --2- 172.21.15.40:0/1993999169 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f6fd80a2b90 0x7f6fd80a5050 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:13.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.927+0000 7f6ff27a2640 1 --2- 172.21.15.40:0/1993999169 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6fec102870 0x7f6fec197ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:13.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.927+0000 7f6ff27a2640 1 --2- 172.21.15.40:0/1993999169 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6fec103230 0x7f6fec19ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:13.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.927+0000 7f6ff27a2640 1 --2- 172.21.15.40:0/1993999169 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6fec101680 0x7f6fec197980 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:13.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.927+0000 7f6ff27a2640 1 -- 172.21.15.40:0/1993999169 >> 172.21.15.40:0/1993999169 conn(0x7f6fec0fcc10 msgr2=0x7f6fec105b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:13.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.928+0000 7f6ff27a2640 1 -- 172.21.15.40:0/1993999169 shutdown_connections 2024-04-21T23:18:13.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:13.928+0000 7f6ff27a2640 1 -- 172.21.15.40:0/1993999169 wait complete. 2024-04-21T23:18:13.938 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:18:14.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.042+0000 7fbd5906b640 1 -- 172.21.15.40:0/1784749859 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd54102250 msgr2=0x7fbd54108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:14.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.042+0000 7fbd5906b640 1 --2- 172.21.15.40:0/1784749859 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd54102250 0x7fbd54108ae0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c009ca0 tx=0x7fbd4c02c0f0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.042+0000 7fbd5906b640 1 -- 172.21.15.40:0/1784749859 shutdown_connections 2024-04-21T23:18:14.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.042+0000 7fbd5906b640 1 --2- 172.21.15.40:0/1784749859 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd54102250 0x7fbd54108ae0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.042+0000 7fbd5906b640 1 --2- 172.21.15.40:0/1784749859 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbd54101890 0x7fbd54101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.042+0000 7fbd5906b640 1 --2- 172.21.15.40:0/1784749859 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbd541006a0 0x7fbd54100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.042+0000 7fbd5906b640 1 -- 172.21.15.40:0/1784749859 >> 172.21.15.40:0/1784749859 conn(0x7fbd540fbc30 msgr2=0x7fbd540fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:14.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.042+0000 7fbd5906b640 1 -- 172.21.15.40:0/1784749859 shutdown_connections 2024-04-21T23:18:14.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.042+0000 7fbd5906b640 1 -- 172.21.15.40:0/1784749859 wait complete. 2024-04-21T23:18:14.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.043+0000 7fbd5906b640 1 Processor -- start 2024-04-21T23:18:14.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.043+0000 7fbd5906b640 1 -- start start 2024-04-21T23:18:14.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.043+0000 7fbd5906b640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbd541006a0 0x7fbd54197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:14.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.043+0000 7fbd5906b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbd54101890 0x7fbd54197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:14.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.043+0000 7fbd5906b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd54102250 0x7fbd5419efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:14.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.043+0000 7fbd5906b640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fbd5410d440 con 0x7fbd54102250 2024-04-21T23:18:14.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.043+0000 7fbd5906b640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fbd5410d2c0 con 0x7fbd541006a0 2024-04-21T23:18:14.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.043+0000 7fbd5906b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fbd5410d5c0 con 0x7fbd54101890 2024-04-21T23:18:14.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.043+0000 7fbd5359e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd54102250 0x7fbd5419efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:14.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.043+0000 7fbd5359e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd54102250 0x7fbd5419efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46014/0 (socket says 172.21.15.40:46014) 2024-04-21T23:18:14.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.043+0000 7fbd5359e640 1 -- 172.21.15.40:0/2057284756 learned_addr learned my addr 172.21.15.40:0/2057284756 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:14.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.043+0000 7fbd52d9d640 1 --2- 172.21.15.40:0/2057284756 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbd541006a0 0x7fbd54197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:14.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.043+0000 7fbd5259c640 1 --2- 172.21.15.40:0/2057284756 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbd54101890 0x7fbd54197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:14.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.043+0000 7fbd5359e640 1 -- 172.21.15.40:0/2057284756 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbd54101890 msgr2=0x7fbd54197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:14.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.043+0000 7fbd5359e640 1 --2- 172.21.15.40:0/2057284756 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbd54101890 0x7fbd54197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.043+0000 7fbd5359e640 1 -- 172.21.15.40:0/2057284756 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbd541006a0 msgr2=0x7fbd54197a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:14.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.043+0000 7fbd5359e640 1 --2- 172.21.15.40:0/2057284756 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbd541006a0 0x7fbd54197a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.043+0000 7fbd5359e640 1 -- 172.21.15.40:0/2057284756 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd5419f650 con 0x7fbd54102250 2024-04-21T23:18:14.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.043+0000 7fbd5259c640 1 --2- 172.21.15.40:0/2057284756 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbd54101890 0x7fbd54197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:18:14.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.044+0000 7fbd5359e640 1 --2- 172.21.15.40:0/2057284756 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd54102250 0x7fbd5419efe0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c02c700 tx=0x7fbd4c032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:14.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.044+0000 7fbd3bfff640 1 -- 172.21.15.40:0/2057284756 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd4c039040 con 0x7fbd54102250 2024-04-21T23:18:14.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.044+0000 7fbd5906b640 1 -- 172.21.15.40:0/2057284756 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd5419f8e0 con 0x7fbd54102250 2024-04-21T23:18:14.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.044+0000 7fbd3bfff640 1 -- 172.21.15.40:0/2057284756 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd4c044070 con 0x7fbd54102250 2024-04-21T23:18:14.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.044+0000 7fbd3bfff640 1 -- 172.21.15.40:0/2057284756 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd4c0042a0 con 0x7fbd54102250 2024-04-21T23:18:14.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.044+0000 7fbd5906b640 1 -- 172.21.15.40:0/2057284756 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd5419fe20 con 0x7fbd54102250 2024-04-21T23:18:14.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.045+0000 7fbd5906b640 1 -- 172.21.15.40:0/2057284756 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd54107960 con 0x7fbd54102250 2024-04-21T23:18:14.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.046+0000 7fbd3bfff640 1 -- 172.21.15.40:0/2057284756 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fbd4c003970 con 0x7fbd54102250 2024-04-21T23:18:14.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.046+0000 7fbd3bfff640 1 --2- 172.21.15.40:0/2057284756 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fbd3c0a2b90 0x7fbd3c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:14.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.047+0000 7fbd52d9d640 1 --2- 172.21.15.40:0/2057284756 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fbd3c0a2b90 0x7fbd3c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:14.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.047+0000 7fbd3bfff640 1 -- 172.21.15.40:0/2057284756 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fbd4c0f1990 con 0x7fbd54102250 2024-04-21T23:18:14.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.047+0000 7fbd52d9d640 1 --2- 172.21.15.40:0/2057284756 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fbd3c0a2b90 0x7fbd3c0a5050 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fbd541016f0 tx=0x7fbd4400b390 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:14.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.051+0000 7fbd3bfff640 1 -- 172.21.15.40:0/2057284756 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fbd4c0be550 con 0x7fbd54102250 2024-04-21T23:18:14.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.276+0000 7fbd5906b640 1 -- 172.21.15.40:0/2057284756 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbd5402d170 con 0x7fbd54102250 2024-04-21T23:18:14.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.276+0000 7fbd3bfff640 1 -- 172.21.15.40:0/2057284756 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11882 (secure 0 0 0) 0x7fbd4c0bde50 con 0x7fbd54102250 2024-04-21T23:18:14.276 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:18:14.276 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/1571948830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1571948830},{"type":"v1","addr":"172.21.15.40:6835","nonce":1571948830}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/969356642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":969356642},{"type":"v1","addr":"172.21.15.40:6837","nonce":969356642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6839/3025057845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":3025057845},{"type":"v1","addr":"172.21.15.40:6839","nonce":3025057845}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6841/3350488804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":3350488804},{"type":"v1","addr":"172.21.15.40:6841","nonce":3350488804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/3438493203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":3438493203},{"type":"v1","addr":"172.21.15.174:6833","nonce":3438493203}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/3105752678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":3105752678},{"type":"v1","addr":"172.21.15.111:6833","nonce":3105752678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T23:15:55.742233+0000","modified":"2024-04-21T23:16:22.498043+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24521,"mds_1":24502,"mds_2":24494},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24461":{"gid":24461,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6835/2704462092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2704462092},{"type":"v1","addr":"172.21.15.111:6835","nonce":2704462092}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/3905733045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3905733045},{"type":"v1","addr":"172.21.15.174:6835","nonce":3905733045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6837/465579746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":465579746},{"type":"v1","addr":"172.21.15.174:6837","nonce":465579746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/2703764182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":2703764182},{"type":"v1","addr":"172.21.15.111:6837","nonce":2703764182}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24502":{"gid":24502,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.174:6839/2467509274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":2467509274},{"type":"v1","addr":"172.21.15.174:6839","nonce":2467509274}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3712235548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3712235548},{"type":"v1","addr":"172.21.15.111:6839","nonce":3712235548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24494,24502,24521]},"id":1}]} 2024-04-21T23:18:14.276 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 24 2024-04-21T23:18:14.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.278+0000 7fbd5906b640 1 -- 172.21.15.40:0/2057284756 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fbd3c0a2b90 msgr2=0x7fbd3c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:14.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.278+0000 7fbd5906b640 1 --2- 172.21.15.40:0/2057284756 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fbd3c0a2b90 0x7fbd3c0a5050 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fbd541016f0 tx=0x7fbd4400b390 comp rx=0 tx=0).stop 2024-04-21T23:18:14.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.278+0000 7fbd5906b640 1 -- 172.21.15.40:0/2057284756 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd54102250 msgr2=0x7fbd5419efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:14.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.278+0000 7fbd5906b640 1 --2- 172.21.15.40:0/2057284756 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd54102250 0x7fbd5419efe0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c02c700 tx=0x7fbd4c032f20 comp rx=0 tx=0).stop 2024-04-21T23:18:14.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.278+0000 7fbd5906b640 1 -- 172.21.15.40:0/2057284756 shutdown_connections 2024-04-21T23:18:14.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.278+0000 7fbd5906b640 1 --2- 172.21.15.40:0/2057284756 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fbd3c0a2b90 0x7fbd3c0a5050 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.278+0000 7fbd5906b640 1 --2- 172.21.15.40:0/2057284756 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd54102250 0x7fbd5419efe0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.278+0000 7fbd5906b640 1 --2- 172.21.15.40:0/2057284756 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbd54101890 0x7fbd54197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.278+0000 7fbd5906b640 1 --2- 172.21.15.40:0/2057284756 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbd541006a0 0x7fbd54197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.278+0000 7fbd5906b640 1 -- 172.21.15.40:0/2057284756 >> 172.21.15.40:0/2057284756 conn(0x7fbd540fbc30 msgr2=0x7fbd54104b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:14.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.279+0000 7fbd5906b640 1 -- 172.21.15.40:0/2057284756 shutdown_connections 2024-04-21T23:18:14.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.279+0000 7fbd5906b640 1 -- 172.21.15.40:0/2057284756 wait complete. 2024-04-21T23:18:14.288 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:14 smithi040 ceph-mon[27984]: pgmap v365: 129 pgs: 129 active+clean; 641 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-21T23:18:14.288 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:14 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/3835126950' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T23:18:14.288 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:14 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/3292457023' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T23:18:14.288 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:14 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/1993999169' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T23:18:14.289 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T23:18:14.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.392+0000 7fb9227ba640 1 -- 172.21.15.40:0/424116033 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb91c1029a0 msgr2=0x7fb91c102e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:14.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.392+0000 7fb9227ba640 1 --2- 172.21.15.40:0/424116033 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb91c1029a0 0x7fb91c102e20 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fb910009900 tx=0x7fb91002bc30 comp rx=0 tx=0).stop 2024-04-21T23:18:14.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.392+0000 7fb9227ba640 1 -- 172.21.15.40:0/424116033 shutdown_connections 2024-04-21T23:18:14.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.392+0000 7fb9227ba640 1 --2- 172.21.15.40:0/424116033 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb91c103360 0x7fb91c109bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.392+0000 7fb9227ba640 1 --2- 172.21.15.40:0/424116033 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb91c1029a0 0x7fb91c102e20 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.392+0000 7fb9227ba640 1 --2- 172.21.15.40:0/424116033 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb91c1017b0 0x7fb91c101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.392+0000 7fb9227ba640 1 -- 172.21.15.40:0/424116033 >> 172.21.15.40:0/424116033 conn(0x7fb91c0fcd40 msgr2=0x7fb91c0ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:14.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.392+0000 7fb9227ba640 1 -- 172.21.15.40:0/424116033 shutdown_connections 2024-04-21T23:18:14.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.392+0000 7fb9227ba640 1 -- 172.21.15.40:0/424116033 wait complete. 2024-04-21T23:18:14.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.393+0000 7fb9227ba640 1 Processor -- start 2024-04-21T23:18:14.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.393+0000 7fb9227ba640 1 -- start start 2024-04-21T23:18:14.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.393+0000 7fb9227ba640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb91c1017b0 0x7fb91c197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:14.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.393+0000 7fb9227ba640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb91c1029a0 0x7fb91c197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:14.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.393+0000 7fb9227ba640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb91c103360 0x7fb91c19eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:14.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.393+0000 7fb9227ba640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb91c10d430 con 0x7fb91c103360 2024-04-21T23:18:14.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.393+0000 7fb9227ba640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb91c10d2b0 con 0x7fb91c1029a0 2024-04-21T23:18:14.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.393+0000 7fb9227ba640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb91c10d5b0 con 0x7fb91c1017b0 2024-04-21T23:18:14.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.393+0000 7fb920d57640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb91c103360 0x7fb91c19eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:14.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.393+0000 7fb920d57640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb91c103360 0x7fb91c19eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46026/0 (socket says 172.21.15.40:46026) 2024-04-21T23:18:14.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.394+0000 7fb920d57640 1 -- 172.21.15.40:0/2115309490 learned_addr learned my addr 172.21.15.40:0/2115309490 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:14.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.394+0000 7fb91b7fe640 1 --2- 172.21.15.40:0/2115309490 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb91c1029a0 0x7fb91c197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:14.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.394+0000 7fb91bfff640 1 --2- 172.21.15.40:0/2115309490 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb91c1017b0 0x7fb91c197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:14.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.394+0000 7fb920d57640 1 -- 172.21.15.40:0/2115309490 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb91c1017b0 msgr2=0x7fb91c197a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:14.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.394+0000 7fb920d57640 1 --2- 172.21.15.40:0/2115309490 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb91c1017b0 0x7fb91c197a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.394+0000 7fb920d57640 1 -- 172.21.15.40:0/2115309490 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb91c1029a0 msgr2=0x7fb91c197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:14.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.394+0000 7fb920d57640 1 --2- 172.21.15.40:0/2115309490 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb91c1029a0 0x7fb91c197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.394+0000 7fb920d57640 1 -- 172.21.15.40:0/2115309490 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb91c19f660 con 0x7fb91c103360 2024-04-21T23:18:14.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.394+0000 7fb920d57640 1 --2- 172.21.15.40:0/2115309490 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb91c103360 0x7fb91c19eff0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fb91400cce0 tx=0x7fb91400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:14.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.394+0000 7fb9197fa640 1 -- 172.21.15.40:0/2115309490 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb91400f450 con 0x7fb91c103360 2024-04-21T23:18:14.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.394+0000 7fb9197fa640 1 -- 172.21.15.40:0/2115309490 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb91400fad0 con 0x7fb91c103360 2024-04-21T23:18:14.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.394+0000 7fb9227ba640 1 -- 172.21.15.40:0/2115309490 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb91c19f950 con 0x7fb91c103360 2024-04-21T23:18:14.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.394+0000 7fb9197fa640 1 -- 172.21.15.40:0/2115309490 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb914018c10 con 0x7fb91c103360 2024-04-21T23:18:14.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.394+0000 7fb9227ba640 1 -- 172.21.15.40:0/2115309490 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb91c19fe20 con 0x7fb91c103360 2024-04-21T23:18:14.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.395+0000 7fb9227ba640 1 -- 172.21.15.40:0/2115309490 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb91c108a70 con 0x7fb91c103360 2024-04-21T23:18:14.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.396+0000 7fb9197fa640 1 -- 172.21.15.40:0/2115309490 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fb91400f5f0 con 0x7fb91c103360 2024-04-21T23:18:14.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.397+0000 7fb9197fa640 1 --2- 172.21.15.40:0/2115309490 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fb9040a2b90 0x7fb9040a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:14.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.397+0000 7fb91bfff640 1 --2- 172.21.15.40:0/2115309490 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fb9040a2b90 0x7fb9040a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:14.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.397+0000 7fb9197fa640 1 -- 172.21.15.40:0/2115309490 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fb9140cfb60 con 0x7fb91c103360 2024-04-21T23:18:14.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.397+0000 7fb91bfff640 1 --2- 172.21.15.40:0/2115309490 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fb9040a2b90 0x7fb9040a5050 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fb90c005fd0 tx=0x7fb90c005850 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:14.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.401+0000 7fb9197fa640 1 -- 172.21.15.40:0/2115309490 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fb914014030 con 0x7fb91c103360 2024-04-21T23:18:14.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:14 smithi174 ceph-mon[31409]: pgmap v365: 129 pgs: 129 active+clean; 641 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-21T23:18:14.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:14 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/3835126950' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T23:18:14.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:14 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/3292457023' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T23:18:14.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:14 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/1993999169' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T23:18:14.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:14 smithi111 ceph-mon[31085]: pgmap v365: 129 pgs: 129 active+clean; 641 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-21T23:18:14.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:14 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/3835126950' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T23:18:14.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:14 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/3292457023' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T23:18:14.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:14 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/1993999169' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T23:18:14.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.590+0000 7fb9227ba640 1 -- 172.21.15.40:0/2115309490 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb91c062340 con 0x7fb91c103360 2024-04-21T23:18:14.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.592+0000 7fb9197fa640 1 -- 172.21.15.40:0/2115309490 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+24853 (secure 0 0 0) 0x7fb91409c690 con 0x7fb91c103360 2024-04-21T23:18:14.591 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:18:14.591 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":94,"fsid":"d814e4dc-0033-11ef-bc93-c7b262605968","created":"2024-04-21T23:07:52.659207+0000","modified":"2024-04-21T23:17:52.907466+0000","last_up_change":"2024-04-21T23:15:05.308821+0000","last_in_change":"2024-04-21T23:14:45.447491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:11:14.824775+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T23:15:54.322571+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T23:15:55.164493+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T23:15:58.838266+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"0e8d692a-22fe-4a87-8d99-9151f96cbb02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6803","nonce":2009758784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6805","nonce":2009758784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6809","nonce":2009758784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6807","nonce":2009758784}]},"public_addr":"172.21.15.40:6803/2009758784","cluster_addr":"172.21.15.40:6805/2009758784","heartbeat_back_addr":"172.21.15.40:6809/2009758784","heartbeat_front_addr":"172.21.15.40:6807/2009758784","state":["exists","up"]},{"osd":1,"uuid":"fe5fe11b-0e84-4359-83a7-1ba45105755a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6801","nonce":1763903836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6803","nonce":1763903836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6807","nonce":1763903836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6805","nonce":1763903836}]},"public_addr":"172.21.15.111:6801/1763903836","cluster_addr":"172.21.15.111:6803/1763903836","heartbeat_back_addr":"172.21.15.111:6807/1763903836","heartbeat_front_addr":"172.21.15.111:6805/1763903836","state":["exists","up"]},{"osd":2,"uuid":"813ad735-5cc2-4da0-ae16-6db238395fbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6801","nonce":439695825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6803","nonce":439695825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6807","nonce":439695825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6805","nonce":439695825}]},"public_addr":"172.21.15.174:6801/439695825","cluster_addr":"172.21.15.174:6803/439695825","heartbeat_back_addr":"172.21.15.174:6807/439695825","heartbeat_front_addr":"172.21.15.174:6805/439695825","state":["exists","up"]},{"osd":3,"uuid":"09becb03-b7ce-4ffc-917d-b7900ec37e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6811","nonce":3454312436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6813","nonce":3454312436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6817","nonce":3454312436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6815","nonce":3454312436}]},"public_addr":"172.21.15.40:6811/3454312436","cluster_addr":"172.21.15.40:6813/3454312436","heartbeat_back_addr":"172.21.15.40:6817/3454312436","heartbeat_front_addr":"172.21.15.40:6815/3454312436","state":["exists","up"]},{"osd":4,"uuid":"46eabfd7-d88a-4efd-94eb-04354d3a9d7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6809","nonce":1810866070}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6811","nonce":1810866070}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6815","nonce":1810866070}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6813","nonce":1810866070}]},"public_addr":"172.21.15.111:6809/1810866070","cluster_addr":"172.21.15.111:6811/1810866070","heartbeat_back_addr":"172.21.15.111:6815/1810866070","heartbeat_front_addr":"172.21.15.111:6813/1810866070","state":["exists","up"]},{"osd":5,"uuid":"6f73b582-6c65-41db-aab0-9b360f83d5c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6809","nonce":3752986711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6811","nonce":3752986711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6815","nonce":3752986711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6813","nonce":3752986711}]},"public_addr":"172.21.15.174:6809/3752986711","cluster_addr":"172.21.15.174:6811/3752986711","heartbeat_back_addr":"172.21.15.174:6815/3752986711","heartbeat_front_addr":"172.21.15.174:6813/3752986711","state":["exists","up"]},{"osd":6,"uuid":"39f305f0-12d2-4732-aaa0-cd66700af538","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6819","nonce":1618855427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6821","nonce":1618855427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6825","nonce":1618855427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6823","nonce":1618855427}]},"public_addr":"172.21.15.40:6819/1618855427","cluster_addr":"172.21.15.40:6821/1618855427","heartbeat_back_addr":"172.21.15.40:6825/1618855427","heartbeat_front_addr":"172.21.15.40:6823/1618855427","state":["exists","up"]},{"osd":7,"uuid":"769f2468-5102-490b-89f8-d0444d3169a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6817","nonce":927054162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6819","nonce":927054162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6823","nonce":927054162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6821","nonce":927054162}]},"public_addr":"172.21.15.111:6817/927054162","cluster_addr":"172.21.15.111:6819/927054162","heartbeat_back_addr":"172.21.15.111:6823/927054162","heartbeat_front_addr":"172.21.15.111:6821/927054162","state":["exists","up"]},{"osd":8,"uuid":"1a444e80-007d-473b-b972-4c814a813c22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6817","nonce":3817848746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6819","nonce":3817848746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6823","nonce":3817848746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6821","nonce":3817848746}]},"public_addr":"172.21.15.174:6817/3817848746","cluster_addr":"172.21.15.174:6819/3817848746","heartbeat_back_addr":"172.21.15.174:6823/3817848746","heartbeat_front_addr":"172.21.15.174:6821/3817848746","state":["exists","up"]},{"osd":9,"uuid":"9a6b6a56-6ae5-49fe-9e5b-cb17d686e0dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6827","nonce":1649446531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6829","nonce":1649446531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6833","nonce":1649446531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6831","nonce":1649446531}]},"public_addr":"172.21.15.40:6827/1649446531","cluster_addr":"172.21.15.40:6829/1649446531","heartbeat_back_addr":"172.21.15.40:6833/1649446531","heartbeat_front_addr":"172.21.15.40:6831/1649446531","state":["exists","up"]},{"osd":10,"uuid":"a70038ba-e2ad-4170-85cc-ea0470a2edef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6825","nonce":1227538935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6827","nonce":1227538935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6831","nonce":1227538935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6829","nonce":1227538935}]},"public_addr":"172.21.15.111:6825/1227538935","cluster_addr":"172.21.15.111:6827/1227538935","heartbeat_back_addr":"172.21.15.111:6831/1227538935","heartbeat_front_addr":"172.21.15.111:6829/1227538935","state":["exists","up"]},{"osd":11,"uuid":"781532bb-7a5e-44e3-8eb7-b584b0054666","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6825","nonce":2045865553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6827","nonce":2045865553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6831","nonce":2045865553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6829","nonce":2045865553}]},"public_addr":"172.21.15.174:6825/2045865553","cluster_addr":"172.21.15.174:6827/2045865553","heartbeat_back_addr":"172.21.15.174:6831/2045865553","heartbeat_front_addr":"172.21.15.174:6829/2045865553","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:10:24.761290+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:10:48.565428+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:11:12.307818+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:11:37.951529+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:02.832523+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:27.568879+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:52.997707+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:13:18.937274+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:13:44.708657+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:14:10.833855+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:14:37.078272+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:15:03.878076+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":1}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.2a","mappings":[{"from":3,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"4.9","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":10}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:6801/526554319":"2024-04-22T23:08:50.305770+0000","172.21.15.40:6800/526554319":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/3178763471":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/4045911364":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/276300080":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/3689649814":"2024-04-22T23:08:23.743030+0000","172.21.15.40:0/2241116715":"2024-04-22T23:08:23.743030+0000","172.21.15.40:6801/1479040326":"2024-04-22T23:08:23.743030+0000","172.21.15.40:6800/1479040326":"2024-04-22T23:08:23.743030+0000","172.21.15.40:0/681298099":"2024-04-22T23:08:23.743030+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T23:18:14.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.594+0000 7fb9227ba640 1 -- 172.21.15.40:0/2115309490 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fb9040a2b90 msgr2=0x7fb9040a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:14.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.594+0000 7fb9227ba640 1 --2- 172.21.15.40:0/2115309490 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fb9040a2b90 0x7fb9040a5050 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fb90c005fd0 tx=0x7fb90c005850 comp rx=0 tx=0).stop 2024-04-21T23:18:14.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.594+0000 7fb9227ba640 1 -- 172.21.15.40:0/2115309490 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb91c103360 msgr2=0x7fb91c19eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:14.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.594+0000 7fb9227ba640 1 --2- 172.21.15.40:0/2115309490 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb91c103360 0x7fb91c19eff0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fb91400cce0 tx=0x7fb91400a620 comp rx=0 tx=0).stop 2024-04-21T23:18:14.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.594+0000 7fb9227ba640 1 -- 172.21.15.40:0/2115309490 shutdown_connections 2024-04-21T23:18:14.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.594+0000 7fb9227ba640 1 --2- 172.21.15.40:0/2115309490 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7fb9040a2b90 0x7fb9040a5050 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.594+0000 7fb9227ba640 1 --2- 172.21.15.40:0/2115309490 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb91c1029a0 0x7fb91c197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.594+0000 7fb9227ba640 1 --2- 172.21.15.40:0/2115309490 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb91c103360 0x7fb91c19eff0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.594+0000 7fb9227ba640 1 --2- 172.21.15.40:0/2115309490 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb91c1017b0 0x7fb91c197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.594+0000 7fb9227ba640 1 -- 172.21.15.40:0/2115309490 >> 172.21.15.40:0/2115309490 conn(0x7fb91c0fcd40 msgr2=0x7fb91c105c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:14.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.594+0000 7fb9227ba640 1 -- 172.21.15.40:0/2115309490 shutdown_connections 2024-04-21T23:18:14.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.594+0000 7fb9227ba640 1 -- 172.21.15.40:0/2115309490 wait complete. 2024-04-21T23:18:14.605 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-21T23:15:55.742233+0000', 'modified': '2024-04-21T23:16:22.498043+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24521, 'mds_1': 24502, 'mds_2': 24494}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24461': {'gid': 24461, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.111:6835/2704462092', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6834', 'nonce': 2704462092}, {'type': 'v1', 'addr': '172.21.15.111:6835', 'nonce': 2704462092}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24466': {'gid': 24466, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.174:6835/3905733045', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6834', 'nonce': 3905733045}, {'type': 'v1', 'addr': '172.21.15.174:6835', 'nonce': 3905733045}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24484': {'gid': 24484, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.174:6837/465579746', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6836', 'nonce': 465579746}, {'type': 'v1', 'addr': '172.21.15.174:6837', 'nonce': 465579746}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24494': {'gid': 24494, 'name': 'h', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.111:6837/2703764182', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6836', 'nonce': 2703764182}, {'type': 'v1', 'addr': '172.21.15.111:6837', 'nonce': 2703764182}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24502': {'gid': 24502, 'name': 'i', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.174:6839/2467509274', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6838', 'nonce': 2467509274}, {'type': 'v1', 'addr': '172.21.15.174:6839', 'nonce': 2467509274}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24521': {'gid': 24521, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.111:6839/3712235548', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6838', 'nonce': 3712235548}, {'type': 'v1', 'addr': '172.21.15.111:6839', 'nonce': 3712235548}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24521, 'qdb_cluster': [24494, 24502, 24521]} 2024-04-21T23:18:14.605 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-21T23:18:14.605 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-04-21T23:18:14.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.709+0000 7f6b5f520640 1 -- 172.21.15.40:0/2990555114 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b58100070 msgr2=0x7f6b581024a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:14.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.709+0000 7f6b5f520640 1 --2- 172.21.15.40:0/2990555114 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b58100070 0x7f6b581024a0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f6b54009900 tx=0x7f6b5402bc30 comp rx=0 tx=0).stop 2024-04-21T23:18:14.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.709+0000 7f6b5f520640 1 -- 172.21.15.40:0/2990555114 shutdown_connections 2024-04-21T23:18:14.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.709+0000 7f6b5f520640 1 --2- 172.21.15.40:0/2990555114 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6b58105390 0x7f6b58105810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.709+0000 7f6b5f520640 1 --2- 172.21.15.40:0/2990555114 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b581029e0 0x7f6b58104e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.709+0000 7f6b5f520640 1 --2- 172.21.15.40:0/2990555114 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b58100070 0x7f6b581024a0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.709+0000 7f6b5f520640 1 -- 172.21.15.40:0/2990555114 >> 172.21.15.40:0/2990555114 conn(0x7f6b580fbc90 msgr2=0x7f6b580fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:14.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.710+0000 7f6b5f520640 1 -- 172.21.15.40:0/2990555114 shutdown_connections 2024-04-21T23:18:14.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.710+0000 7f6b5f520640 1 -- 172.21.15.40:0/2990555114 wait complete. 2024-04-21T23:18:14.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.710+0000 7f6b5f520640 1 Processor -- start 2024-04-21T23:18:14.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.710+0000 7f6b5f520640 1 -- start start 2024-04-21T23:18:14.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.710+0000 7f6b5f520640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b58100070 0x7f6b5819e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:14.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.710+0000 7f6b5f520640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b581029e0 0x7f6b5819e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:14.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.710+0000 7f6b5f520640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6b58105390 0x7f6b581a56b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:14.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.710+0000 7f6b5f520640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6b58113a00 con 0x7f6b58100070 2024-04-21T23:18:14.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.710+0000 7f6b5f520640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f6b58113880 con 0x7f6b58105390 2024-04-21T23:18:14.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.710+0000 7f6b5f520640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6b58113b80 con 0x7f6b581029e0 2024-04-21T23:18:14.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.710+0000 7f6b5d2bc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b58100070 0x7f6b5819e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:14.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.710+0000 7f6b5d2bc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b58100070 0x7f6b5819e100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46048/0 (socket says 172.21.15.40:46048) 2024-04-21T23:18:14.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.710+0000 7f6b5d2bc640 1 -- 172.21.15.40:0/999148888 learned_addr learned my addr 172.21.15.40:0/999148888 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:14.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.711+0000 7f6b5dabd640 1 --2- 172.21.15.40:0/999148888 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6b58105390 0x7f6b581a56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:14.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.711+0000 7f6b5cabb640 1 --2- 172.21.15.40:0/999148888 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b581029e0 0x7f6b5819e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:14.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.711+0000 7f6b5d2bc640 1 -- 172.21.15.40:0/999148888 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b581029e0 msgr2=0x7f6b5819e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:14.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.711+0000 7f6b5d2bc640 1 --2- 172.21.15.40:0/999148888 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b581029e0 0x7f6b5819e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.711+0000 7f6b5d2bc640 1 -- 172.21.15.40:0/999148888 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6b58105390 msgr2=0x7f6b581a56b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:14.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.711+0000 7f6b5d2bc640 1 --2- 172.21.15.40:0/999148888 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6b58105390 0x7f6b581a56b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.711+0000 7f6b5d2bc640 1 -- 172.21.15.40:0/999148888 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b581a5d20 con 0x7f6b58100070 2024-04-21T23:18:14.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.711+0000 7f6b5cabb640 1 --2- 172.21.15.40:0/999148888 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b581029e0 0x7f6b5819e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:18:14.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.711+0000 7f6b5d2bc640 1 --2- 172.21.15.40:0/999148888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b58100070 0x7f6b5819e100 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f6b54009230 tx=0x7f6b54005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:14.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.711+0000 7f6b4e7fc640 1 -- 172.21.15.40:0/999148888 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b54041070 con 0x7f6b58100070 2024-04-21T23:18:14.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.711+0000 7f6b4e7fc640 1 -- 172.21.15.40:0/999148888 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b540044d0 con 0x7f6b58100070 2024-04-21T23:18:14.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.711+0000 7f6b5f520640 1 -- 172.21.15.40:0/999148888 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b581a5fb0 con 0x7f6b58100070 2024-04-21T23:18:14.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.711+0000 7f6b4e7fc640 1 -- 172.21.15.40:0/999148888 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b54034440 con 0x7f6b58100070 2024-04-21T23:18:14.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.711+0000 7f6b5f520640 1 -- 172.21.15.40:0/999148888 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b581a63b0 con 0x7f6b58100070 2024-04-21T23:18:14.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.713+0000 7f6b4e7fc640 1 -- 172.21.15.40:0/999148888 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f6b54044070 con 0x7f6b58100070 2024-04-21T23:18:14.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.714+0000 7f6b621e0740 1 Processor -- start 2024-04-21T23:18:14.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.714+0000 7f6b621e0740 1 -- start start 2024-04-21T23:18:14.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.714+0000 7f6b621e0740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e1d94e5070 0x55e1d94e5490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:14.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.714+0000 7f6b621e0740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e1d94e5a60 0x55e1d94ea560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:14.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.714+0000 7f6b621e0740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e1d94eabd0 0x55e1d94ed060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:14.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.714+0000 7f6b621e0740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55e1d9449df0 con 0x55e1d94e5a60 2024-04-21T23:18:14.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.714+0000 7f6b621e0740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55e1d93f2640 con 0x55e1d94e5070 2024-04-21T23:18:14.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.714+0000 7f6b621e0740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55e1d9445db0 con 0x55e1d94eabd0 2024-04-21T23:18:14.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.714+0000 7f6b4e7fc640 1 --2- 172.21.15.40:0/999148888 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f6b440a2b90 0x7f6b440a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:14.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.714+0000 7f6b5d2bc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e1d94e5a60 0x55e1d94ea560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:14.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.714+0000 7f6b5d2bc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e1d94e5a60 0x55e1d94ea560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46056/0 (socket says 172.21.15.40:46056) 2024-04-21T23:18:14.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.714+0000 7f6b5d2bc640 1 -- 172.21.15.40:0/2348555370 learned_addr learned my addr 172.21.15.40:0/2348555370 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:14.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.714+0000 7f6b5dabd640 1 --2- 172.21.15.40:0/999148888 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f6b440a2b90 0x7f6b440a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:14.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.714+0000 7f6b4e7fc640 1 -- 172.21.15.40:0/999148888 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f6b540f10f0 con 0x7f6b58100070 2024-04-21T23:18:14.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.714+0000 7f6b5cabb640 1 --2- 172.21.15.40:0/2348555370 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e1d94eabd0 0x55e1d94ed060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:14.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.714+0000 7f6b5dabd640 1 --2- 172.21.15.40:0/2348555370 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e1d94e5070 0x55e1d94e5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:14.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.714+0000 7f6b5d2bc640 1 -- 172.21.15.40:0/2348555370 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e1d94eabd0 msgr2=0x55e1d94ed060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:14.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.714+0000 7f6b5d2bc640 1 --2- 172.21.15.40:0/2348555370 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e1d94eabd0 0x55e1d94ed060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.714+0000 7f6b5d2bc640 1 -- 172.21.15.40:0/2348555370 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e1d94e5070 msgr2=0x55e1d94e5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:14.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.714+0000 7f6b5d2bc640 1 --2- 172.21.15.40:0/2348555370 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e1d94e5070 0x55e1d94e5490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.714+0000 7f6b5d2bc640 1 -- 172.21.15.40:0/2348555370 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e1d915e2d0 con 0x55e1d94e5a60 2024-04-21T23:18:14.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.715+0000 7f6b5cabb640 1 --2- 172.21.15.40:0/2348555370 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e1d94eabd0 0x55e1d94ed060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:18:14.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.715+0000 7f6b5d2bc640 1 --2- 172.21.15.40:0/2348555370 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e1d94e5a60 0x55e1d94ea560 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f6b54009430 tx=0x7f6b540905e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c882d92deaf8af76 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:14.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.715+0000 7f6b5dabd640 1 --2- 172.21.15.40:0/999148888 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f6b440a2b90 0x7f6b440a5050 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x55e1d94e62c0 tx=0x7f6b50009b50 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:14.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.715+0000 7f6b437fe640 1 -- 172.21.15.40:0/2348555370 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b5404b070 con 0x55e1d94e5a60 2024-04-21T23:18:14.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.715+0000 7f6b437fe640 1 -- 172.21.15.40:0/2348555370 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b540bdc20 con 0x55e1d94e5a60 2024-04-21T23:18:14.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.715+0000 7f6b437fe640 1 -- 172.21.15.40:0/2348555370 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b540bd520 con 0x55e1d94e5a60 2024-04-21T23:18:14.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.716+0000 7f6b621e0740 1 -- 172.21.15.40:0/2348555370 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e1d94e5a60 msgr2=0x55e1d94ea560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:14.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.716+0000 7f6b621e0740 1 --2- 172.21.15.40:0/2348555370 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e1d94e5a60 0x55e1d94ea560 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f6b54009430 tx=0x7f6b540905e0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.716+0000 7f6b621e0740 1 -- 172.21.15.40:0/2348555370 shutdown_connections 2024-04-21T23:18:14.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.716+0000 7f6b621e0740 1 --2- 172.21.15.40:0/2348555370 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e1d94eabd0 0x55e1d94ed060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.716+0000 7f6b621e0740 1 --2- 172.21.15.40:0/2348555370 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e1d94e5a60 0x55e1d94ea560 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.716+0000 7f6b621e0740 1 --2- 172.21.15.40:0/2348555370 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e1d94e5070 0x55e1d94e5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.716+0000 7f6b621e0740 1 -- 172.21.15.40:0/2348555370 >> 172.21.15.40:0/2348555370 conn(0x55e1d94e05e0 msgr2=0x55e1d94e09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:14.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.716+0000 7f6b621e0740 1 -- 172.21.15.40:0/2348555370 shutdown_connections 2024-04-21T23:18:14.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.716+0000 7f6b621e0740 1 -- 172.21.15.40:0/2348555370 wait complete. 2024-04-21T23:18:14.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.716+0000 7f6b621e0740 1 -- start start 2024-04-21T23:18:14.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.716+0000 7f6b621e0740 1 Processor -- start 2024-04-21T23:18:14.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.716+0000 7f6b621e0740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e1d94e5070 0x55e1d94ef1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:14.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.716+0000 7f6b621e0740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e1d94e5a60 0x55e1d94ef720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:14.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.716+0000 7f6b621e0740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e1d94eabd0 0x55e1d94f6760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:14.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.716+0000 7f6b621e0740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55e1d94ed940 con 0x55e1d94e5a60 2024-04-21T23:18:14.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.716+0000 7f6b621e0740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55e1d9405be0 con 0x55e1d94e5070 2024-04-21T23:18:14.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.716+0000 7f6b621e0740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55e1d94edac0 con 0x55e1d94eabd0 2024-04-21T23:18:14.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.717+0000 7f6b621e0740 10 client.0 _ll_register_callbacks cb 0x55e1d94055d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:18:14.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.717+0000 7f6b5cabb640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e1d94e5a60 0x55e1d94ef720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:14.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.717+0000 7f6b5cabb640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e1d94e5a60 0x55e1d94ef720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46062/0 (socket says 172.21.15.40:46062) 2024-04-21T23:18:14.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.717+0000 7f6b5cabb640 1 -- 172.21.15.40:0/3715290446 learned_addr learned my addr 172.21.15.40:0/3715290446 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:14.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.717+0000 7f6b5d2bc640 1 --2- 172.21.15.40:0/3715290446 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e1d94e5070 0x55e1d94ef1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:14.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.717+0000 7f6b5dabd640 1 --2- 172.21.15.40:0/3715290446 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e1d94eabd0 0x55e1d94f6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:14.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.717+0000 7f6b5cabb640 1 -- 172.21.15.40:0/3715290446 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e1d94eabd0 msgr2=0x55e1d94f6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:14.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.717+0000 7f6b5cabb640 1 --2- 172.21.15.40:0/3715290446 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e1d94eabd0 0x55e1d94f6760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.717+0000 7f6b5cabb640 1 -- 172.21.15.40:0/3715290446 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e1d94e5070 msgr2=0x55e1d94ef1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:14.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.717+0000 7f6b5cabb640 1 --2- 172.21.15.40:0/3715290446 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e1d94e5070 0x55e1d94ef1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.717+0000 7f6b5cabb640 1 -- 172.21.15.40:0/3715290446 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e1d92286b0 con 0x55e1d94e5a60 2024-04-21T23:18:14.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.717+0000 7f6b5cabb640 1 --2- 172.21.15.40:0/3715290446 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e1d94e5a60 0x55e1d94ef720 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f6b48013710 tx=0x7f6b48013bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c1d2d110e812597 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:14.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.717+0000 7f6b40ff9640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-21T23:18:14.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.717+0000 7f6b40ff9640 1 -- 172.21.15.40:0/3715290446 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b48020060 con 0x55e1d94e5a60 2024-04-21T23:18:14.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.717+0000 7f6b40ff9640 1 -- 172.21.15.40:0/3715290446 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f6b48011d70 con 0x55e1d94e5a60 2024-04-21T23:18:14.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.717+0000 7f6b621e0740 1 -- 172.21.15.40:0/3715290446 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6b540bdc20 con 0x55e1d94e5a60 2024-04-21T23:18:14.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.717+0000 7f6b40ff9640 1 -- 172.21.15.40:0/3715290446 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f6b48027050 con 0x55e1d94e5a60 2024-04-21T23:18:14.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.717+0000 7f6b621e0740 10 client.15393 fetch_fsmap learned FSMap version 24 2024-04-21T23:18:14.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.717+0000 7f6b621e0740 1 -- 172.21.15.40:0/3715290446 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55e1d94f9810 con 0x55e1d94e5a60 2024-04-21T23:18:14.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.718+0000 7f6b40ff9640 1 -- 172.21.15.40:0/3715290446 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b540bdc20 con 0x55e1d94e5a60 2024-04-21T23:18:14.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.718+0000 7f6b40ff9640 1 -- 172.21.15.40:0/3715290446 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f6b4801a030 con 0x55e1d94e5a60 2024-04-21T23:18:14.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.718+0000 7f6b621e0740 10 client.15393 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T23:18:14.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.718+0000 7f6b621e0740 10 client.15393 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-21T23:18:14.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.718+0000 7f6b621e0740 20 client.15393 populate_metadata read hostname 'smithi040' 2024-04-21T23:18:14.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.718+0000 7f6b621e0740 1 --2- 172.21.15.40:0/3715290446 >> [v2:172.21.15.111:6836/2703764182,v1:172.21.15.111:6837/2703764182] conn(0x55e1d94fbf10 0x55e1d94fd380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:14.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.718+0000 7f6b621e0740 4 client.15393 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:18:14.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.718+0000 7f6b621e0740 1 -- 172.21.15.40:0/3715290446 --> [v2:172.21.15.111:6836/2703764182,v1:172.21.15.111:6837/2703764182] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e1d9457330 con 0x55e1d94fbf10 2024-04-21T23:18:14.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.718+0000 7f6b5d2bc640 1 --2- 172.21.15.40:0/3715290446 >> [v2:172.21.15.111:6836/2703764182,v1:172.21.15.111:6837/2703764182] conn(0x55e1d94fbf10 0x55e1d94fd380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:14.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.719+0000 7f6b5d2bc640 1 --2- 172.21.15.40:0/3715290446 >> [v2:172.21.15.111:6836/2703764182,v1:172.21.15.111:6837/2703764182] conn(0x55e1d94fbf10 0x55e1d94fd380 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=aa487a30cf7cdf60 server_cookie=f5e55124ecd3320f in_seq=0 out_seq=0 2024-04-21T23:18:14.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.719+0000 7f6b40ff9640 10 client.15393 ms_handle_connect on v2:172.21.15.111:6836/2703764182 2024-04-21T23:18:14.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.719+0000 7f6b40ff9640 1 -- 172.21.15.40:0/3715290446 <== mds.2 v2:172.21.15.111:6836/2703764182 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e1d9457330 con 0x55e1d94fbf10 2024-04-21T23:18:14.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.719+0000 7f6b40ff9640 10 client.15393 handle_command_reply: tid=0 2024-04-21T23:18:14.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.720+0000 7f6b621e0740 1 client.15393 shutdown 2024-04-21T23:18:14.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.720+0000 7f6b621e0740 1 -- 172.21.15.40:0/3715290446 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e1d94e5a60 msgr2=0x55e1d94ef720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:14.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.720+0000 7f6b621e0740 1 --2- 172.21.15.40:0/3715290446 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e1d94e5a60 0x55e1d94ef720 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f6b48013710 tx=0x7f6b48013bd0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.720+0000 7f6b621e0740 1 -- 172.21.15.40:0/3715290446 shutdown_connections 2024-04-21T23:18:14.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.720+0000 7f6b621e0740 1 --2- 172.21.15.40:0/3715290446 >> [v2:172.21.15.111:6836/2703764182,v1:172.21.15.111:6837/2703764182] conn(0x55e1d94fbf10 0x55e1d94fd380 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.720+0000 7f6b621e0740 1 --2- 172.21.15.40:0/3715290446 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e1d94eabd0 0x55e1d94f6760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.720+0000 7f6b621e0740 1 --2- 172.21.15.40:0/3715290446 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e1d94e5a60 0x55e1d94ef720 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.720+0000 7f6b621e0740 1 --2- 172.21.15.40:0/3715290446 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e1d94e5070 0x55e1d94ef1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.720+0000 7f6b40ff9640 0 client.15393 ms_handle_reset on v2:172.21.15.111:6836/2703764182 2024-04-21T23:18:14.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.720+0000 7f6b621e0740 1 -- 172.21.15.40:0/3715290446 >> 172.21.15.40:0/3715290446 conn(0x55e1d94e05e0 msgr2=0x55e1d94df720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:14.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.720+0000 7f6b621e0740 1 -- 172.21.15.40:0/3715290446 shutdown_connections 2024-04-21T23:18:14.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.720+0000 7f6b621e0740 1 -- 172.21.15.40:0/3715290446 wait complete. 2024-04-21T23:18:14.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.720+0000 7f6b621e0740 20 client.15393 trim_cache size 0 max 16384 2024-04-21T23:18:14.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.738+0000 7f6b621e0740 1 Processor -- start 2024-04-21T23:18:14.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.738+0000 7f6b621e0740 1 -- start start 2024-04-21T23:18:14.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.738+0000 7f6b621e0740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e1d94e5070 0x55e1d94db340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:14.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.738+0000 7f6b621e0740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e1d94e5a60 0x55e1d94f6720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:14.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.738+0000 7f6b621e0740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e1d94eabd0 0x55e1d94e6ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:14.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.738+0000 7f6b621e0740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55e1d94fb6c0 con 0x55e1d94e5070 2024-04-21T23:18:14.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.738+0000 7f6b621e0740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55e1d94fbd90 con 0x55e1d94eabd0 2024-04-21T23:18:14.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.738+0000 7f6b621e0740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55e1d94fc460 con 0x55e1d94e5a60 2024-04-21T23:18:14.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.738+0000 7f6b5d2bc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e1d94e5070 0x55e1d94db340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:14.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.738+0000 7f6b5d2bc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e1d94e5070 0x55e1d94db340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46076/0 (socket says 172.21.15.40:46076) 2024-04-21T23:18:14.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.738+0000 7f6b5d2bc640 1 -- 172.21.15.40:0/973059460 learned_addr learned my addr 172.21.15.40:0/973059460 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:14.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.739+0000 7f6b5dabd640 1 --2- 172.21.15.40:0/973059460 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e1d94eabd0 0x55e1d94e6ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:14.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.739+0000 7f6b5d2bc640 1 -- 172.21.15.40:0/973059460 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e1d94e5a60 msgr2=0x55e1d94f6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:14.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.739+0000 7f6b5d2bc640 1 --2- 172.21.15.40:0/973059460 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e1d94e5a60 0x55e1d94f6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.739+0000 7f6b5d2bc640 1 -- 172.21.15.40:0/973059460 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e1d94eabd0 msgr2=0x55e1d94e6ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:14.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.739+0000 7f6b5d2bc640 1 --2- 172.21.15.40:0/973059460 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e1d94eabd0 0x55e1d94e6ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.739+0000 7f6b5d2bc640 1 -- 172.21.15.40:0/973059460 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e1d94f0080 con 0x55e1d94e5070 2024-04-21T23:18:14.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.739+0000 7f6b5dabd640 1 --2- 172.21.15.40:0/973059460 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e1d94eabd0 0x55e1d94e6ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:18:14.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.739+0000 7f6b5d2bc640 1 --2- 172.21.15.40:0/973059460 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e1d94e5070 0x55e1d94db340 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f6b540bd4b0 tx=0x7f6b54000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27c4fda7ccbeb502 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:14.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.739+0000 7f6b42ffd640 1 -- 172.21.15.40:0/973059460 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b5404b040 con 0x55e1d94e5070 2024-04-21T23:18:14.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.739+0000 7f6b42ffd640 1 -- 172.21.15.40:0/973059460 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b540bdc20 con 0x55e1d94e5070 2024-04-21T23:18:14.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.739+0000 7f6b42ffd640 1 -- 172.21.15.40:0/973059460 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b540b9b90 con 0x55e1d94e5070 2024-04-21T23:18:14.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.739+0000 7f6b621e0740 1 -- 172.21.15.40:0/973059460 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e1d94e5070 msgr2=0x55e1d94db340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:14.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.740+0000 7f6b621e0740 1 --2- 172.21.15.40:0/973059460 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e1d94e5070 0x55e1d94db340 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f6b540bd4b0 tx=0x7f6b54000bc0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.740+0000 7f6b621e0740 1 -- 172.21.15.40:0/973059460 shutdown_connections 2024-04-21T23:18:14.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.740+0000 7f6b621e0740 1 --2- 172.21.15.40:0/973059460 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e1d94eabd0 0x55e1d94e6ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.740+0000 7f6b621e0740 1 --2- 172.21.15.40:0/973059460 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e1d94e5a60 0x55e1d94f6720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.740+0000 7f6b621e0740 1 --2- 172.21.15.40:0/973059460 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e1d94e5070 0x55e1d94db340 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.740+0000 7f6b621e0740 1 -- 172.21.15.40:0/973059460 >> 172.21.15.40:0/973059460 conn(0x55e1d94e05e0 msgr2=0x55e1d94e09c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:14.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.740+0000 7f6b621e0740 1 -- 172.21.15.40:0/973059460 shutdown_connections 2024-04-21T23:18:14.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.740+0000 7f6b621e0740 1 -- 172.21.15.40:0/973059460 wait complete. 2024-04-21T23:18:14.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.740+0000 7f6b621e0740 1 -- start start 2024-04-21T23:18:14.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.740+0000 7f6b621e0740 1 Processor -- start 2024-04-21T23:18:14.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.740+0000 7f6b621e0740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e1d94e5070 0x55e1d94f1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:14.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.740+0000 7f6b621e0740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e1d94e5a60 0x55e1d94f22e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:14.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.740+0000 7f6b621e0740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e1d94eabd0 0x55e1d94f51a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:14.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.740+0000 7f6b621e0740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55e1d94fd200 con 0x55e1d94e5070 2024-04-21T23:18:14.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.740+0000 7f6b621e0740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55e1d94fcb30 con 0x55e1d94eabd0 2024-04-21T23:18:14.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.740+0000 7f6b621e0740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55e1d94fd8d0 con 0x55e1d94e5a60 2024-04-21T23:18:14.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.740+0000 7f6b621e0740 10 client.0 _ll_register_callbacks cb 0x7f6b48009d00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:18:14.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.740+0000 7f6b5d2bc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e1d94e5070 0x55e1d94f1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:14.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.740+0000 7f6b5d2bc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e1d94e5070 0x55e1d94f1da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46084/0 (socket says 172.21.15.40:46084) 2024-04-21T23:18:14.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.740+0000 7f6b5d2bc640 1 -- 172.21.15.40:0/3769877233 learned_addr learned my addr 172.21.15.40:0/3769877233 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:14.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.741+0000 7f6b5dabd640 1 --2- 172.21.15.40:0/3769877233 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e1d94eabd0 0x55e1d94f51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:14.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.741+0000 7f6b5d2bc640 1 -- 172.21.15.40:0/3769877233 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e1d94e5a60 msgr2=0x55e1d94f22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:14.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.741+0000 7f6b5d2bc640 1 --2- 172.21.15.40:0/3769877233 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e1d94e5a60 0x55e1d94f22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.741+0000 7f6b5d2bc640 1 -- 172.21.15.40:0/3769877233 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e1d94eabd0 msgr2=0x55e1d94f51a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:14.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.741+0000 7f6b5d2bc640 1 --2- 172.21.15.40:0/3769877233 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e1d94eabd0 0x55e1d94f51a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.741+0000 7f6b5d2bc640 1 -- 172.21.15.40:0/3769877233 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e1d94f57e0 con 0x55e1d94e5070 2024-04-21T23:18:14.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.741+0000 7f6b5dabd640 1 --2- 172.21.15.40:0/3769877233 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e1d94eabd0 0x55e1d94f51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:18:14.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.741+0000 7f6b5d2bc640 1 --2- 172.21.15.40:0/3769877233 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e1d94e5070 0x55e1d94f1da0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f6b540c5c60 tx=0x7f6b5402ce80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa8fe14f0de660fc server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:14.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.741+0000 7f6b40ff9640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-21T23:18:14.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.741+0000 7f6b40ff9640 1 -- 172.21.15.40:0/3769877233 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b5404b070 con 0x55e1d94e5070 2024-04-21T23:18:14.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.741+0000 7f6b621e0740 1 -- 172.21.15.40:0/3769877233 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6b540bdc20 con 0x55e1d94e5070 2024-04-21T23:18:14.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.741+0000 7f6b40ff9640 1 -- 172.21.15.40:0/3769877233 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f6b540b9b90 con 0x55e1d94e5070 2024-04-21T23:18:14.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.741+0000 7f6b40ff9640 1 -- 172.21.15.40:0/3769877233 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f6b540bce80 con 0x55e1d94e5070 2024-04-21T23:18:14.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.741+0000 7f6b621e0740 10 client.15399 fetch_fsmap learned FSMap version 24 2024-04-21T23:18:14.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.741+0000 7f6b621e0740 1 -- 172.21.15.40:0/3769877233 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55e1d94f6a10 con 0x55e1d94e5070 2024-04-21T23:18:14.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.741+0000 7f6b40ff9640 1 -- 172.21.15.40:0/3769877233 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b540bdc20 con 0x55e1d94e5070 2024-04-21T23:18:14.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.741+0000 7f6b40ff9640 1 -- 172.21.15.40:0/3769877233 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f6b540c5d00 con 0x55e1d94e5070 2024-04-21T23:18:14.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.741+0000 7f6b621e0740 10 client.15399 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T23:18:14.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.741+0000 7f6b621e0740 10 client.15399 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-21T23:18:14.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.742+0000 7f6b621e0740 20 client.15399 populate_metadata read hostname 'smithi040' 2024-04-21T23:18:14.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.742+0000 7f6b621e0740 1 --2- 172.21.15.40:0/3769877233 >> [v2:172.21.15.111:6836/2703764182,v1:172.21.15.111:6837/2703764182] conn(0x55e1d9566390 0x55e1d95650a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:14.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.742+0000 7f6b621e0740 4 client.15399 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-21T23:18:14.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.742+0000 7f6b621e0740 1 -- 172.21.15.40:0/3769877233 --> [v2:172.21.15.111:6836/2703764182,v1:172.21.15.111:6837/2703764182] -- command(tid 0: {"prefix": "status"}) -- 0x55e1d9451680 con 0x55e1d9566390 2024-04-21T23:18:14.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.742+0000 7f6b5cabb640 1 --2- 172.21.15.40:0/3769877233 >> [v2:172.21.15.111:6836/2703764182,v1:172.21.15.111:6837/2703764182] conn(0x55e1d9566390 0x55e1d95650a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:14.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.742+0000 7f6b5cabb640 1 --2- 172.21.15.40:0/3769877233 >> [v2:172.21.15.111:6836/2703764182,v1:172.21.15.111:6837/2703764182] conn(0x55e1d9566390 0x55e1d95650a0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8ea262b253743cf3 server_cookie=7118b1cb54204394 in_seq=0 out_seq=0 2024-04-21T23:18:14.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.742+0000 7f6b40ff9640 10 client.15399 ms_handle_connect on v2:172.21.15.111:6836/2703764182 2024-04-21T23:18:14.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.742+0000 7f6b40ff9640 1 -- 172.21.15.40:0/3769877233 <== mds.2 v2:172.21.15.111:6836/2703764182 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x55e1d9451680 con 0x55e1d9566390 2024-04-21T23:18:14.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.742+0000 7f6b40ff9640 10 client.15399 handle_command_reply: tid=0 2024-04-21T23:18:14.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.742+0000 7f6b621e0740 1 client.15399 shutdown 2024-04-21T23:18:14.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.742+0000 7f6b621e0740 1 -- 172.21.15.40:0/3769877233 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e1d94e5070 msgr2=0x55e1d94f1da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:14.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.742+0000 7f6b621e0740 1 --2- 172.21.15.40:0/3769877233 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e1d94e5070 0x55e1d94f1da0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f6b540c5c60 tx=0x7f6b5402ce80 comp rx=0 tx=0).stop 2024-04-21T23:18:14.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.743+0000 7f6b621e0740 1 -- 172.21.15.40:0/3769877233 shutdown_connections 2024-04-21T23:18:14.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.743+0000 7f6b621e0740 1 --2- 172.21.15.40:0/3769877233 >> [v2:172.21.15.111:6836/2703764182,v1:172.21.15.111:6837/2703764182] conn(0x55e1d9566390 0x55e1d95650a0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.743+0000 7f6b621e0740 1 --2- 172.21.15.40:0/3769877233 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e1d94eabd0 0x55e1d94f51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.743+0000 7f6b40ff9640 0 client.15399 ms_handle_reset on v2:172.21.15.111:6836/2703764182 2024-04-21T23:18:14.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.743+0000 7f6b621e0740 1 --2- 172.21.15.40:0/3769877233 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e1d94e5a60 0x55e1d94f22e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.743+0000 7f6b621e0740 1 --2- 172.21.15.40:0/3769877233 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e1d94e5070 0x55e1d94f1da0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.743+0000 7f6b621e0740 1 -- 172.21.15.40:0/3769877233 >> 172.21.15.40:0/3769877233 conn(0x55e1d94e05e0 msgr2=0x55e1d94db340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:14.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.743+0000 7f6b621e0740 1 -- 172.21.15.40:0/3769877233 shutdown_connections 2024-04-21T23:18:14.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.743+0000 7f6b621e0740 1 -- 172.21.15.40:0/3769877233 wait complete. 2024-04-21T23:18:14.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.743+0000 7f6b621e0740 20 client.15399 trim_cache size 0 max 16384 2024-04-21T23:18:14.749 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-21T23:18:14.749 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "d814e4dc-0033-11ef-bc93-c7b262605968", 2024-04-21T23:18:14.749 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 2, 2024-04-21T23:18:14.749 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 24494, 2024-04-21T23:18:14.750 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-04-21T23:18:14.750 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-04-21T23:18:14.750 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-04-21T23:18:14.750 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 113.258543633, 2024-04-21T23:18:14.750 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 24, 2024-04-21T23:18:14.750 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 94, 2024-04-21T23:18:14.750 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 90, 2024-04-21T23:18:14.750 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 155.981935064 2024-04-21T23:18:14.750 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-21T23:18:14.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.743+0000 7f6b5f520640 1 -- 172.21.15.40:0/999148888 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f6b440a2b90 msgr2=0x7f6b440a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:14.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.743+0000 7f6b5f520640 1 --2- 172.21.15.40:0/999148888 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f6b440a2b90 0x7f6b440a5050 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x55e1d94e62c0 tx=0x7f6b50009b50 comp rx=0 tx=0).stop 2024-04-21T23:18:14.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.743+0000 7f6b5f520640 1 -- 172.21.15.40:0/999148888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b58100070 msgr2=0x7f6b5819e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:14.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.743+0000 7f6b5f520640 1 --2- 172.21.15.40:0/999148888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b58100070 0x7f6b5819e100 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f6b54009230 tx=0x7f6b54005d50 comp rx=0 tx=0).stop 2024-04-21T23:18:14.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.743+0000 7f6b5f520640 1 -- 172.21.15.40:0/999148888 shutdown_connections 2024-04-21T23:18:14.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.743+0000 7f6b5f520640 1 --2- 172.21.15.40:0/999148888 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f6b440a2b90 0x7f6b440a5050 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.743+0000 7f6b5f520640 1 --2- 172.21.15.40:0/999148888 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6b58105390 0x7f6b581a56b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.743+0000 7f6b5f520640 1 --2- 172.21.15.40:0/999148888 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b581029e0 0x7f6b5819e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.743+0000 7f6b5f520640 1 --2- 172.21.15.40:0/999148888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b58100070 0x7f6b5819e100 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.743+0000 7f6b5f520640 1 -- 172.21.15.40:0/999148888 >> 172.21.15.40:0/999148888 conn(0x7f6b580fbc90 msgr2=0x7f6b581034f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:14.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.743+0000 7f6b5f520640 1 -- 172.21.15.40:0/999148888 shutdown_connections 2024-04-21T23:18:14.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.743+0000 7f6b5f520640 1 -- 172.21.15.40:0/999148888 wait complete. 2024-04-21T23:18:14.753 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-21T23:18:14.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.857+0000 7f4d1e4c6640 1 -- 172.21.15.40:0/3350286384 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d181074e0 msgr2=0x7f4d181098d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:14.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.857+0000 7f4d1e4c6640 1 --2- 172.21.15.40:0/3350286384 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d181074e0 0x7f4d181098d0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f4d10009ca0 tx=0x7f4d1002c0f0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.857+0000 7f4d1e4c6640 1 -- 172.21.15.40:0/3350286384 shutdown_connections 2024-04-21T23:18:14.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.857+0000 7f4d1e4c6640 1 --2- 172.21.15.40:0/3350286384 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d18104bb0 0x7f4d18106fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.857+0000 7f4d1e4c6640 1 --2- 172.21.15.40:0/3350286384 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d181074e0 0x7f4d181098d0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.857+0000 7f4d1e4c6640 1 --2- 172.21.15.40:0/3350286384 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4d18102280 0x7f4d18104670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.857+0000 7f4d1e4c6640 1 -- 172.21.15.40:0/3350286384 >> 172.21.15.40:0/3350286384 conn(0x7f4d180fbc30 msgr2=0x7f4d180fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:14.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.857+0000 7f4d1e4c6640 1 -- 172.21.15.40:0/3350286384 shutdown_connections 2024-04-21T23:18:14.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.857+0000 7f4d1e4c6640 1 -- 172.21.15.40:0/3350286384 wait complete. 2024-04-21T23:18:14.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.858+0000 7f4d1e4c6640 1 Processor -- start 2024-04-21T23:18:14.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.858+0000 7f4d1e4c6640 1 -- start start 2024-04-21T23:18:14.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.858+0000 7f4d1e4c6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4d18102280 0x7f4d18197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:14.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.858+0000 7f4d1e4c6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d18104bb0 0x7f4d18197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:14.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.858+0000 7f4d1e4c6640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d181074e0 0x7f4d1819efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:14.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.858+0000 7f4d1e4c6640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4d1810d440 con 0x7f4d18104bb0 2024-04-21T23:18:14.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.858+0000 7f4d1e4c6640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f4d1810d2c0 con 0x7f4d181074e0 2024-04-21T23:18:14.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.858+0000 7f4d1e4c6640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4d1810d5c0 con 0x7f4d18102280 2024-04-21T23:18:14.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.858+0000 7f4d0ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d18104bb0 0x7f4d18197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:14.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.858+0000 7f4d0ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d18104bb0 0x7f4d18197f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46098/0 (socket says 172.21.15.40:46098) 2024-04-21T23:18:14.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.858+0000 7f4d0ffff640 1 -- 172.21.15.40:0/2923019130 learned_addr learned my addr 172.21.15.40:0/2923019130 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:14.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.858+0000 7f4d17fff640 1 --2- 172.21.15.40:0/2923019130 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4d18102280 0x7f4d18197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:14.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.858+0000 7f4d1ca63640 1 --2- 172.21.15.40:0/2923019130 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d181074e0 0x7f4d1819efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:14.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.859+0000 7f4d0ffff640 1 -- 172.21.15.40:0/2923019130 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4d18102280 msgr2=0x7f4d18197a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:14.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.859+0000 7f4d0ffff640 1 --2- 172.21.15.40:0/2923019130 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4d18102280 0x7f4d18197a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.859+0000 7f4d0ffff640 1 -- 172.21.15.40:0/2923019130 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d181074e0 msgr2=0x7f4d1819efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:14.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.859+0000 7f4d0ffff640 1 --2- 172.21.15.40:0/2923019130 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d181074e0 0x7f4d1819efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.859+0000 7f4d0ffff640 1 -- 172.21.15.40:0/2923019130 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d1819f620 con 0x7f4d18104bb0 2024-04-21T23:18:14.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.859+0000 7f4d1ca63640 1 --2- 172.21.15.40:0/2923019130 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d181074e0 0x7f4d1819efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:18:14.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.859+0000 7f4d0ffff640 1 --2- 172.21.15.40:0/2923019130 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d18104bb0 0x7f4d18197f40 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f4d0000c970 tx=0x7f4d0000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:14.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.859+0000 7f4d15ffb640 1 -- 172.21.15.40:0/2923019130 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d00014070 con 0x7f4d18104bb0 2024-04-21T23:18:14.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.859+0000 7f4d15ffb640 1 -- 172.21.15.40:0/2923019130 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d00004440 con 0x7f4d18104bb0 2024-04-21T23:18:14.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.859+0000 7f4d1e4c6640 1 -- 172.21.15.40:0/2923019130 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d1819f910 con 0x7f4d18104bb0 2024-04-21T23:18:14.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.859+0000 7f4d15ffb640 1 -- 172.21.15.40:0/2923019130 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d0000f450 con 0x7f4d18104bb0 2024-04-21T23:18:14.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.859+0000 7f4d1e4c6640 1 -- 172.21.15.40:0/2923019130 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d1819feb0 con 0x7f4d18104bb0 2024-04-21T23:18:14.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.861+0000 7f4d15ffb640 1 -- 172.21.15.40:0/2923019130 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f4d00017070 con 0x7f4d18104bb0 2024-04-21T23:18:14.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.862+0000 7f4d21193740 1 Processor -- start 2024-04-21T23:18:14.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.862+0000 7f4d21193740 1 -- start start 2024-04-21T23:18:14.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.862+0000 7f4d15ffb640 1 --2- 172.21.15.40:0/2923019130 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f4d040a2c10 0x7f4d040a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:14.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.862+0000 7f4d21193740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556d23a2c070 0x556d23a2c490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:14.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.862+0000 7f4d21193740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d23a2ca60 0x556d23a31560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:14.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.862+0000 7f4d21193740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d23a31bd0 0x556d23a34060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:14.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.862+0000 7f4d21193740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x556d23990df0 con 0x556d23a2ca60 2024-04-21T23:18:14.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.862+0000 7f4d21193740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x556d23939640 con 0x556d23a2c070 2024-04-21T23:18:14.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.862+0000 7f4d21193740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x556d2398cdb0 con 0x556d23a31bd0 2024-04-21T23:18:14.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.862+0000 7f4d1ca63640 1 --2- 172.21.15.40:0/2923019130 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f4d040a2c10 0x7f4d040a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:14.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.862+0000 7f4d0ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d23a2ca60 0x556d23a31560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:14.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.862+0000 7f4d0ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d23a2ca60 0x556d23a31560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46106/0 (socket says 172.21.15.40:46106) 2024-04-21T23:18:14.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.862+0000 7f4d0ffff640 1 -- 172.21.15.40:0/3368460856 learned_addr learned my addr 172.21.15.40:0/3368460856 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:14.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.862+0000 7f4d15ffb640 1 -- 172.21.15.40:0/2923019130 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f4d000cf250 con 0x7f4d18104bb0 2024-04-21T23:18:14.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.862+0000 7f4d1ca63640 1 --2- 172.21.15.40:0/3368460856 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d23a31bd0 0x556d23a34060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:14.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.862+0000 7f4d17fff640 1 --2- 172.21.15.40:0/3368460856 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556d23a2c070 0x556d23a2c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:14.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.862+0000 7f4d0ffff640 1 -- 172.21.15.40:0/3368460856 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d23a31bd0 msgr2=0x556d23a34060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:14.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.862+0000 7f4d0ffff640 1 --2- 172.21.15.40:0/3368460856 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d23a31bd0 0x556d23a34060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.862+0000 7f4d0ffff640 1 -- 172.21.15.40:0/3368460856 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556d23a2c070 msgr2=0x556d23a2c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:14.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.862+0000 7f4d0ffff640 1 --2- 172.21.15.40:0/3368460856 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556d23a2c070 0x556d23a2c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.862+0000 7f4d0ffff640 1 -- 172.21.15.40:0/3368460856 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d236a52d0 con 0x556d23a2ca60 2024-04-21T23:18:14.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.862+0000 7f4d17fff640 1 --2- 172.21.15.40:0/3368460856 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556d23a2c070 0x556d23a2c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:18:14.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.863+0000 7f4d1ca63640 1 --2- 172.21.15.40:0/2923019130 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f4d040a2c10 0x7f4d040a50d0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f4d10032420 tx=0x7f4d10009910 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:14.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.863+0000 7f4d0ffff640 1 --2- 172.21.15.40:0/3368460856 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d23a2ca60 0x556d23a31560 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f4d00000c00 tx=0x7f4d00008d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c298b874668c05a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:14.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.863+0000 7f4d0e7fc640 1 -- 172.21.15.40:0/3368460856 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d000bba40 con 0x556d23a2ca60 2024-04-21T23:18:14.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.863+0000 7f4d0e7fc640 1 -- 172.21.15.40:0/3368460856 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d0009bd80 con 0x556d23a2ca60 2024-04-21T23:18:14.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.863+0000 7f4d0e7fc640 1 -- 172.21.15.40:0/3368460856 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d000860d0 con 0x556d23a2ca60 2024-04-21T23:18:14.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.863+0000 7f4d21193740 1 -- 172.21.15.40:0/3368460856 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d23a2ca60 msgr2=0x556d23a31560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:14.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.863+0000 7f4d21193740 1 --2- 172.21.15.40:0/3368460856 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d23a2ca60 0x556d23a31560 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f4d00000c00 tx=0x7f4d00008d10 comp rx=0 tx=0).stop 2024-04-21T23:18:14.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.863+0000 7f4d21193740 1 -- 172.21.15.40:0/3368460856 shutdown_connections 2024-04-21T23:18:14.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.863+0000 7f4d21193740 1 --2- 172.21.15.40:0/3368460856 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d23a31bd0 0x556d23a34060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.863+0000 7f4d21193740 1 --2- 172.21.15.40:0/3368460856 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d23a2ca60 0x556d23a31560 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.863+0000 7f4d21193740 1 --2- 172.21.15.40:0/3368460856 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556d23a2c070 0x556d23a2c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.863+0000 7f4d21193740 1 -- 172.21.15.40:0/3368460856 >> 172.21.15.40:0/3368460856 conn(0x556d23a275e0 msgr2=0x556d23a279e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:14.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.863+0000 7f4d21193740 1 -- 172.21.15.40:0/3368460856 shutdown_connections 2024-04-21T23:18:14.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.863+0000 7f4d21193740 1 -- 172.21.15.40:0/3368460856 wait complete. 2024-04-21T23:18:14.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.864+0000 7f4d21193740 1 -- start start 2024-04-21T23:18:14.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.864+0000 7f4d21193740 1 Processor -- start 2024-04-21T23:18:14.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.864+0000 7f4d21193740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d23a2c070 0x556d23a365b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:14.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.864+0000 7f4d21193740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556d23a2ca60 0x556d23a36af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:14.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.864+0000 7f4d21193740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d23a31bd0 0x556d23a3db30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:14.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.864+0000 7f4d21193740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x556d23a34b80 con 0x556d23a2c070 2024-04-21T23:18:14.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.864+0000 7f4d21193740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x556d2394cbe0 con 0x556d23a2ca60 2024-04-21T23:18:14.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.864+0000 7f4d21193740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x556d23a34d00 con 0x556d23a31bd0 2024-04-21T23:18:14.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.864+0000 7f4d21193740 10 client.0 _ll_register_callbacks cb 0x556d2394c5d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:18:14.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.864+0000 7f4d17fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d23a2c070 0x556d23a365b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:14.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.864+0000 7f4d17fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d23a2c070 0x556d23a365b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46120/0 (socket says 172.21.15.40:46120) 2024-04-21T23:18:14.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.864+0000 7f4d17fff640 1 -- 172.21.15.40:0/712907796 learned_addr learned my addr 172.21.15.40:0/712907796 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:14.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.865+0000 7f4d0ffff640 1 --2- 172.21.15.40:0/712907796 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556d23a2ca60 0x556d23a36af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:14.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.865+0000 7f4d1ca63640 1 --2- 172.21.15.40:0/712907796 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d23a31bd0 0x556d23a3db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:14.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.865+0000 7f4d17fff640 1 -- 172.21.15.40:0/712907796 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d23a31bd0 msgr2=0x556d23a3db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:14.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.865+0000 7f4d17fff640 1 --2- 172.21.15.40:0/712907796 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d23a31bd0 0x556d23a3db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.865+0000 7f4d17fff640 1 -- 172.21.15.40:0/712907796 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556d23a2ca60 msgr2=0x556d23a36af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:14.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.865+0000 7f4d17fff640 1 --2- 172.21.15.40:0/712907796 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556d23a2ca60 0x556d23a36af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.865+0000 7f4d17fff640 1 -- 172.21.15.40:0/712907796 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556d2376f6b0 con 0x556d23a2c070 2024-04-21T23:18:14.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.865+0000 7f4d17fff640 1 --2- 172.21.15.40:0/712907796 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d23a2c070 0x556d23a365b0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f4d0800b9a0 tx=0x7f4d0800be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43d693436963268c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:14.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.865+0000 7f4cf77fe640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-21T23:18:14.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.865+0000 7f4cf77fe640 1 -- 172.21.15.40:0/712907796 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d08005020 con 0x556d23a2c070 2024-04-21T23:18:14.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.865+0000 7f4cf77fe640 1 -- 172.21.15.40:0/712907796 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f4d080051c0 con 0x556d23a2c070 2024-04-21T23:18:14.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.865+0000 7f4d21193740 1 -- 172.21.15.40:0/712907796 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4d0009bd80 con 0x556d23a2c070 2024-04-21T23:18:14.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.865+0000 7f4cf77fe640 1 -- 172.21.15.40:0/712907796 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f4d08023050 con 0x556d23a2c070 2024-04-21T23:18:14.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.865+0000 7f4d21193740 10 client.15411 fetch_fsmap learned FSMap version 24 2024-04-21T23:18:14.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.865+0000 7f4d21193740 1 -- 172.21.15.40:0/712907796 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x556d23a40be0 con 0x556d23a2c070 2024-04-21T23:18:14.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.865+0000 7f4cf77fe640 1 -- 172.21.15.40:0/712907796 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d0009bd80 con 0x556d23a2c070 2024-04-21T23:18:14.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.866+0000 7f4cf77fe640 1 -- 172.21.15.40:0/712907796 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f4d08019030 con 0x556d23a2c070 2024-04-21T23:18:14.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.866+0000 7f4d21193740 10 client.15411 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T23:18:14.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.866+0000 7f4d21193740 10 client.15411 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-21T23:18:14.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.866+0000 7f4d21193740 20 client.15411 populate_metadata read hostname 'smithi040' 2024-04-21T23:18:14.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.866+0000 7f4d21193740 1 --2- 172.21.15.40:0/712907796 >> [v2:172.21.15.174:6838/2467509274,v1:172.21.15.174:6839/2467509274] conn(0x556d23a432d0 0x556d23a44740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:14.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.866+0000 7f4d21193740 4 client.15411 mds_command: new command op to 24502 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:18:14.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.866+0000 7f4d21193740 1 -- 172.21.15.40:0/712907796 --> [v2:172.21.15.174:6838/2467509274,v1:172.21.15.174:6839/2467509274] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556d2399e330 con 0x556d23a432d0 2024-04-21T23:18:14.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.866+0000 7f4d0ffff640 1 --2- 172.21.15.40:0/712907796 >> [v2:172.21.15.174:6838/2467509274,v1:172.21.15.174:6839/2467509274] conn(0x556d23a432d0 0x556d23a44740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:14.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.866+0000 7f4d0ffff640 1 --2- 172.21.15.40:0/712907796 >> [v2:172.21.15.174:6838/2467509274,v1:172.21.15.174:6839/2467509274] conn(0x556d23a432d0 0x556d23a44740 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4dd8fca50a02cd25 server_cookie=e3c13a59b6ba6620 in_seq=0 out_seq=0 2024-04-21T23:18:14.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.866+0000 7f4cf77fe640 10 client.15411 ms_handle_connect on v2:172.21.15.174:6838/2467509274 2024-04-21T23:18:14.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.867+0000 7f4cf77fe640 1 -- 172.21.15.40:0/712907796 <== mds.1 v2:172.21.15.174:6838/2467509274 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556d2399e330 con 0x556d23a432d0 2024-04-21T23:18:14.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.867+0000 7f4cf77fe640 10 client.15411 handle_command_reply: tid=0 2024-04-21T23:18:14.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.867+0000 7f4d21193740 1 client.15411 shutdown 2024-04-21T23:18:14.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.867+0000 7f4d21193740 1 -- 172.21.15.40:0/712907796 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d23a2c070 msgr2=0x556d23a365b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:14.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.867+0000 7f4d21193740 1 --2- 172.21.15.40:0/712907796 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d23a2c070 0x556d23a365b0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f4d0800b9a0 tx=0x7f4d0800be60 comp rx=0 tx=0).stop 2024-04-21T23:18:14.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.867+0000 7f4d21193740 1 -- 172.21.15.40:0/712907796 shutdown_connections 2024-04-21T23:18:14.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.867+0000 7f4d21193740 1 --2- 172.21.15.40:0/712907796 >> [v2:172.21.15.174:6838/2467509274,v1:172.21.15.174:6839/2467509274] conn(0x556d23a432d0 0x556d23a44740 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.867+0000 7f4d21193740 1 --2- 172.21.15.40:0/712907796 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556d23a2ca60 0x556d23a36af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.867+0000 7f4d21193740 1 --2- 172.21.15.40:0/712907796 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d23a31bd0 0x556d23a3db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.867+0000 7f4cf77fe640 0 client.15411 ms_handle_reset on v2:172.21.15.174:6838/2467509274 2024-04-21T23:18:14.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.867+0000 7f4d21193740 1 --2- 172.21.15.40:0/712907796 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d23a2c070 0x556d23a365b0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.867+0000 7f4d21193740 1 -- 172.21.15.40:0/712907796 >> 172.21.15.40:0/712907796 conn(0x556d23a275e0 msgr2=0x556d23a26720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:14.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.868+0000 7f4d21193740 1 -- 172.21.15.40:0/712907796 shutdown_connections 2024-04-21T23:18:14.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.868+0000 7f4d21193740 1 -- 172.21.15.40:0/712907796 wait complete. 2024-04-21T23:18:14.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.868+0000 7f4d21193740 20 client.15411 trim_cache size 0 max 16384 2024-04-21T23:18:14.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.886+0000 7f4d21193740 1 Processor -- start 2024-04-21T23:18:14.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.886+0000 7f4d21193740 1 -- start start 2024-04-21T23:18:14.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.886+0000 7f4d21193740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d23a2c070 0x556d23a25e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:14.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.886+0000 7f4d21193740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d23a2ca60 0x556d23a3daf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:14.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.886+0000 7f4d21193740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556d23a31bd0 0x556d23a263d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:14.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.886+0000 7f4d21193740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x556d23a42a80 con 0x556d23a2ca60 2024-04-21T23:18:14.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.886+0000 7f4d21193740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x556d23a43150 con 0x556d23a31bd0 2024-04-21T23:18:14.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.886+0000 7f4d21193740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x556d23a43820 con 0x556d23a2c070 2024-04-21T23:18:14.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.886+0000 7f4d0ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d23a2ca60 0x556d23a3daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:14.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.886+0000 7f4d0ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d23a2ca60 0x556d23a3daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46122/0 (socket says 172.21.15.40:46122) 2024-04-21T23:18:14.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.886+0000 7f4d0ffff640 1 -- 172.21.15.40:0/3393609311 learned_addr learned my addr 172.21.15.40:0/3393609311 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:14.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.886+0000 7f4d17fff640 1 --2- 172.21.15.40:0/3393609311 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d23a2c070 0x556d23a25e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:14.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.886+0000 7f4d1ca63640 1 --2- 172.21.15.40:0/3393609311 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556d23a31bd0 0x556d23a263d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:14.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.886+0000 7f4d0ffff640 1 -- 172.21.15.40:0/3393609311 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d23a2c070 msgr2=0x556d23a25e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:14.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.886+0000 7f4d0ffff640 1 --2- 172.21.15.40:0/3393609311 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d23a2c070 0x556d23a25e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.886+0000 7f4d0ffff640 1 -- 172.21.15.40:0/3393609311 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556d23a31bd0 msgr2=0x556d23a263d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:14.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.886+0000 7f4d0ffff640 1 --2- 172.21.15.40:0/3393609311 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556d23a31bd0 0x556d23a263d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.886+0000 7f4d0ffff640 1 -- 172.21.15.40:0/3393609311 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d23a25c20 con 0x556d23a2ca60 2024-04-21T23:18:14.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.886+0000 7f4d17fff640 1 --2- 172.21.15.40:0/3393609311 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d23a2c070 0x556d23a25e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:18:14.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.886+0000 7f4d1ca63640 1 --2- 172.21.15.40:0/3393609311 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556d23a31bd0 0x556d23a263d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:18:14.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.887+0000 7f4d0ffff640 1 --2- 172.21.15.40:0/3393609311 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d23a2ca60 0x556d23a3daf0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f4d0009b380 tx=0x7f4d000045f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19a13af6e9c7cc57 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:14.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.887+0000 7f4d0dffb640 1 -- 172.21.15.40:0/3393609311 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d0009b680 con 0x556d23a2ca60 2024-04-21T23:18:14.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.887+0000 7f4d0dffb640 1 -- 172.21.15.40:0/3393609311 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d000a42f0 con 0x556d23a2ca60 2024-04-21T23:18:14.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.887+0000 7f4d0dffb640 1 -- 172.21.15.40:0/3393609311 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d00099a10 con 0x556d23a2ca60 2024-04-21T23:18:14.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.887+0000 7f4d21193740 1 -- 172.21.15.40:0/3393609311 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d23a2ca60 msgr2=0x556d23a3daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:14.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.887+0000 7f4d21193740 1 --2- 172.21.15.40:0/3393609311 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d23a2ca60 0x556d23a3daf0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f4d0009b380 tx=0x7f4d000045f0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.887+0000 7f4d21193740 1 -- 172.21.15.40:0/3393609311 shutdown_connections 2024-04-21T23:18:14.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.887+0000 7f4d21193740 1 --2- 172.21.15.40:0/3393609311 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556d23a31bd0 0x556d23a263d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.887+0000 7f4d21193740 1 --2- 172.21.15.40:0/3393609311 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d23a2ca60 0x556d23a3daf0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.887+0000 7f4d21193740 1 --2- 172.21.15.40:0/3393609311 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d23a2c070 0x556d23a25e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.887+0000 7f4d21193740 1 -- 172.21.15.40:0/3393609311 >> 172.21.15.40:0/3393609311 conn(0x556d23a275e0 msgr2=0x556d23a279c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:14.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.887+0000 7f4d21193740 1 -- 172.21.15.40:0/3393609311 shutdown_connections 2024-04-21T23:18:14.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.887+0000 7f4d21193740 1 -- 172.21.15.40:0/3393609311 wait complete. 2024-04-21T23:18:14.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.888+0000 7f4d21193740 1 -- start start 2024-04-21T23:18:14.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.888+0000 7f4d21193740 1 Processor -- start 2024-04-21T23:18:14.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.888+0000 7f4d21193740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556d23a2c070 0x556d23a39400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:14.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.888+0000 7f4d21193740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d23a2ca60 0x556d23a39940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:14.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.888+0000 7f4d21193740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d23a31bd0 0x556d23a3c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:14.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.888+0000 7f4d21193740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x556d23a445c0 con 0x556d23a31bd0 2024-04-21T23:18:14.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.888+0000 7f4d21193740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x556d23a43ef0 con 0x556d23a2c070 2024-04-21T23:18:14.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.888+0000 7f4d21193740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x556d23a44c90 con 0x556d23a2ca60 2024-04-21T23:18:14.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.888+0000 7f4d21193740 10 client.0 _ll_register_callbacks cb 0x7f4d08004440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:18:14.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.888+0000 7f4d1ca63640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d23a31bd0 0x556d23a3c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:14.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.888+0000 7f4d1ca63640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d23a31bd0 0x556d23a3c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46134/0 (socket says 172.21.15.40:46134) 2024-04-21T23:18:14.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.888+0000 7f4d1ca63640 1 -- 172.21.15.40:0/3286855094 learned_addr learned my addr 172.21.15.40:0/3286855094 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:14.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.888+0000 7f4d1ca63640 1 -- 172.21.15.40:0/3286855094 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d23a2ca60 msgr2=0x556d23a39940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:14.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.888+0000 7f4d1ca63640 1 --2- 172.21.15.40:0/3286855094 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d23a2ca60 0x556d23a39940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.888+0000 7f4d1ca63640 1 -- 172.21.15.40:0/3286855094 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556d23a2c070 msgr2=0x556d23a39400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:14.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.888+0000 7f4d1ca63640 1 --2- 172.21.15.40:0/3286855094 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556d23a2c070 0x556d23a39400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.888+0000 7f4d1ca63640 1 -- 172.21.15.40:0/3286855094 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556d23a3cf50 con 0x556d23a31bd0 2024-04-21T23:18:14.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.889+0000 7f4d1ca63640 1 --2- 172.21.15.40:0/3286855094 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d23a31bd0 0x556d23a3c960 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f4d10009ca0 tx=0x7f4d1002c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f36f149c35abd694 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:14.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.889+0000 7f4cf77fe640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-21T23:18:14.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.889+0000 7f4cf77fe640 1 -- 172.21.15.40:0/3286855094 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d10052070 con 0x556d23a31bd0 2024-04-21T23:18:14.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.889+0000 7f4cf77fe640 1 -- 172.21.15.40:0/3286855094 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f4d10004030 con 0x556d23a31bd0 2024-04-21T23:18:14.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.889+0000 7f4d21193740 1 -- 172.21.15.40:0/3286855094 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4d000a42f0 con 0x556d23a31bd0 2024-04-21T23:18:14.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.889+0000 7f4cf77fe640 1 -- 172.21.15.40:0/3286855094 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f4d1005e050 con 0x556d23a31bd0 2024-04-21T23:18:14.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.889+0000 7f4d21193740 10 client.15417 fetch_fsmap learned FSMap version 24 2024-04-21T23:18:14.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.889+0000 7f4d21193740 1 -- 172.21.15.40:0/3286855094 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x556d23aa94c0 con 0x556d23a31bd0 2024-04-21T23:18:14.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.889+0000 7f4cf77fe640 1 -- 172.21.15.40:0/3286855094 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d000a42f0 con 0x556d23a31bd0 2024-04-21T23:18:14.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.889+0000 7f4cf77fe640 1 -- 172.21.15.40:0/3286855094 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f4d1004d070 con 0x556d23a31bd0 2024-04-21T23:18:14.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.889+0000 7f4d21193740 10 client.15417 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T23:18:14.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.889+0000 7f4d21193740 10 client.15417 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-21T23:18:14.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.889+0000 7f4d21193740 20 client.15417 populate_metadata read hostname 'smithi040' 2024-04-21T23:18:14.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.889+0000 7f4d21193740 1 --2- 172.21.15.40:0/3286855094 >> [v2:172.21.15.174:6838/2467509274,v1:172.21.15.174:6839/2467509274] conn(0x556d23aae230 0x556d23aacf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:14.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.889+0000 7f4d21193740 4 client.15417 mds_command: new command op to 24502 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-21T23:18:14.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.889+0000 7f4d21193740 1 -- 172.21.15.40:0/3286855094 --> [v2:172.21.15.174:6838/2467509274,v1:172.21.15.174:6839/2467509274] -- command(tid 0: {"prefix": "status"}) -- 0x556d23998680 con 0x556d23aae230 2024-04-21T23:18:14.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.890+0000 7f4d17fff640 1 --2- 172.21.15.40:0/3286855094 >> [v2:172.21.15.174:6838/2467509274,v1:172.21.15.174:6839/2467509274] conn(0x556d23aae230 0x556d23aacf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:14.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.890+0000 7f4cf77fe640 10 client.15417 ms_handle_connect on v2:172.21.15.174:6838/2467509274 2024-04-21T23:18:14.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.890+0000 7f4d17fff640 1 --2- 172.21.15.40:0/3286855094 >> [v2:172.21.15.174:6838/2467509274,v1:172.21.15.174:6839/2467509274] conn(0x556d23aae230 0x556d23aacf40 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c56a784c8c59469a server_cookie=28030c1da015455e in_seq=0 out_seq=0 2024-04-21T23:18:14.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.890+0000 7f4cf77fe640 1 -- 172.21.15.40:0/3286855094 <== mds.1 v2:172.21.15.174:6838/2467509274 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x556d23998680 con 0x556d23aae230 2024-04-21T23:18:14.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.890+0000 7f4cf77fe640 10 client.15417 handle_command_reply: tid=0 2024-04-21T23:18:14.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.890+0000 7f4d21193740 1 client.15417 shutdown 2024-04-21T23:18:14.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.890+0000 7f4d21193740 1 -- 172.21.15.40:0/3286855094 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d23a31bd0 msgr2=0x556d23a3c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:14.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.890+0000 7f4d21193740 1 --2- 172.21.15.40:0/3286855094 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d23a31bd0 0x556d23a3c960 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f4d10009ca0 tx=0x7f4d1002c0f0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.891+0000 7f4d21193740 1 -- 172.21.15.40:0/3286855094 shutdown_connections 2024-04-21T23:18:14.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.891+0000 7f4d21193740 1 --2- 172.21.15.40:0/3286855094 >> [v2:172.21.15.174:6838/2467509274,v1:172.21.15.174:6839/2467509274] conn(0x556d23aae230 0x556d23aacf40 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.891+0000 7f4d21193740 1 --2- 172.21.15.40:0/3286855094 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d23a31bd0 0x556d23a3c960 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.891+0000 7f4d21193740 1 --2- 172.21.15.40:0/3286855094 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d23a2ca60 0x556d23a39940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.891+0000 7f4cf77fe640 0 client.15417 ms_handle_reset on v2:172.21.15.174:6838/2467509274 2024-04-21T23:18:14.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.891+0000 7f4d21193740 1 --2- 172.21.15.40:0/3286855094 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556d23a2c070 0x556d23a39400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.891+0000 7f4d21193740 1 -- 172.21.15.40:0/3286855094 >> 172.21.15.40:0/3286855094 conn(0x556d23a275e0 msgr2=0x556d23a25e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:14.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.891+0000 7f4d21193740 1 -- 172.21.15.40:0/3286855094 shutdown_connections 2024-04-21T23:18:14.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.891+0000 7f4d21193740 1 -- 172.21.15.40:0/3286855094 wait complete. 2024-04-21T23:18:14.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.891+0000 7f4d21193740 20 client.15417 trim_cache size 0 max 16384 2024-04-21T23:18:14.897 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-21T23:18:14.897 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "d814e4dc-0033-11ef-bc93-c7b262605968", 2024-04-21T23:18:14.897 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 1, 2024-04-21T23:18:14.897 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 24502, 2024-04-21T23:18:14.897 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-04-21T23:18:14.898 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-04-21T23:18:14.898 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-04-21T23:18:14.898 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 114.411918603, 2024-04-21T23:18:14.898 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 24, 2024-04-21T23:18:14.898 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 94, 2024-04-21T23:18:14.898 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 90, 2024-04-21T23:18:14.898 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 153.36355549000001 2024-04-21T23:18:14.898 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-21T23:18:14.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.891+0000 7f4d1e4c6640 1 -- 172.21.15.40:0/2923019130 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f4d040a2c10 msgr2=0x7f4d040a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:14.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.891+0000 7f4d1e4c6640 1 --2- 172.21.15.40:0/2923019130 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f4d040a2c10 0x7f4d040a50d0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f4d10032420 tx=0x7f4d10009910 comp rx=0 tx=0).stop 2024-04-21T23:18:14.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.891+0000 7f4d1e4c6640 1 -- 172.21.15.40:0/2923019130 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d18104bb0 msgr2=0x7f4d18197f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:14.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.891+0000 7f4d1e4c6640 1 --2- 172.21.15.40:0/2923019130 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d18104bb0 0x7f4d18197f40 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f4d0000c970 tx=0x7f4d0000ce30 comp rx=0 tx=0).stop 2024-04-21T23:18:14.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.891+0000 7f4d1e4c6640 1 -- 172.21.15.40:0/2923019130 shutdown_connections 2024-04-21T23:18:14.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.891+0000 7f4d1e4c6640 1 --2- 172.21.15.40:0/2923019130 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f4d040a2c10 0x7f4d040a50d0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.891+0000 7f4d1e4c6640 1 --2- 172.21.15.40:0/2923019130 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d181074e0 0x7f4d1819efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.891+0000 7f4d1e4c6640 1 --2- 172.21.15.40:0/2923019130 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d18104bb0 0x7f4d18197f40 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.891+0000 7f4d1e4c6640 1 --2- 172.21.15.40:0/2923019130 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4d18102280 0x7f4d18197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:14.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.891+0000 7f4d1e4c6640 1 -- 172.21.15.40:0/2923019130 >> 172.21.15.40:0/2923019130 conn(0x7f4d180fbc30 msgr2=0x7f4d18102bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:14.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.891+0000 7f4d1e4c6640 1 -- 172.21.15.40:0/2923019130 shutdown_connections 2024-04-21T23:18:14.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:14.891+0000 7f4d1e4c6640 1 -- 172.21.15.40:0/2923019130 wait complete. 2024-04-21T23:18:14.902 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-21T23:18:15.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.005+0000 7f6be0c39640 1 -- 172.21.15.40:0/1382571872 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6bdc102250 msgr2=0x7f6bdc108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:15.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.005+0000 7f6be0c39640 1 --2- 172.21.15.40:0/1382571872 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6bdc102250 0x7f6bdc108ae0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f6bd4009ca0 tx=0x7f6bd402c0f0 comp rx=0 tx=0).stop 2024-04-21T23:18:15.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.005+0000 7f6be0c39640 1 -- 172.21.15.40:0/1382571872 shutdown_connections 2024-04-21T23:18:15.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.005+0000 7f6be0c39640 1 --2- 172.21.15.40:0/1382571872 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6bdc102250 0x7f6bdc108ae0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:15.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.005+0000 7f6be0c39640 1 --2- 172.21.15.40:0/1382571872 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6bdc101890 0x7f6bdc101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:15.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.005+0000 7f6be0c39640 1 --2- 172.21.15.40:0/1382571872 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6bdc1006a0 0x7f6bdc100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:15.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.005+0000 7f6be0c39640 1 -- 172.21.15.40:0/1382571872 >> 172.21.15.40:0/1382571872 conn(0x7f6bdc0fbc30 msgr2=0x7f6bdc0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:15.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.005+0000 7f6be0c39640 1 -- 172.21.15.40:0/1382571872 shutdown_connections 2024-04-21T23:18:15.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.005+0000 7f6be0c39640 1 -- 172.21.15.40:0/1382571872 wait complete. 2024-04-21T23:18:15.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.006+0000 7f6be0c39640 1 Processor -- start 2024-04-21T23:18:15.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.006+0000 7f6be0c39640 1 -- start start 2024-04-21T23:18:15.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.006+0000 7f6be0c39640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6bdc1006a0 0x7f6bdc197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:15.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.006+0000 7f6be0c39640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6bdc101890 0x7f6bdc197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:15.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.006+0000 7f6be0c39640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6bdc102250 0x7f6bdc19efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:15.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.006+0000 7f6be0c39640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6bdc10c300 con 0x7f6bdc102250 2024-04-21T23:18:15.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.006+0000 7f6be0c39640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f6bdc10c180 con 0x7f6bdc101890 2024-04-21T23:18:15.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.006+0000 7f6be0c39640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6bdc10c480 con 0x7f6bdc1006a0 2024-04-21T23:18:15.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.006+0000 7f6bdad9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6bdc102250 0x7f6bdc19efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:15.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.006+0000 7f6bdad9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6bdc102250 0x7f6bdc19efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46146/0 (socket says 172.21.15.40:46146) 2024-04-21T23:18:15.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.006+0000 7f6bdad9d640 1 -- 172.21.15.40:0/4099055858 learned_addr learned my addr 172.21.15.40:0/4099055858 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:15.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.006+0000 7f6bd9d9b640 1 --2- 172.21.15.40:0/4099055858 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6bdc101890 0x7f6bdc197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:15.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.006+0000 7f6bda59c640 1 --2- 172.21.15.40:0/4099055858 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6bdc1006a0 0x7f6bdc197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:15.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.007+0000 7f6bdad9d640 1 -- 172.21.15.40:0/4099055858 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6bdc1006a0 msgr2=0x7f6bdc197a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:15.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.007+0000 7f6bdad9d640 1 --2- 172.21.15.40:0/4099055858 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6bdc1006a0 0x7f6bdc197a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:15.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.007+0000 7f6bdad9d640 1 -- 172.21.15.40:0/4099055858 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6bdc101890 msgr2=0x7f6bdc197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:15.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.007+0000 7f6bdad9d640 1 --2- 172.21.15.40:0/4099055858 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6bdc101890 0x7f6bdc197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:15.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.007+0000 7f6bdad9d640 1 -- 172.21.15.40:0/4099055858 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bdc19f630 con 0x7f6bdc102250 2024-04-21T23:18:15.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.007+0000 7f6bdad9d640 1 --2- 172.21.15.40:0/4099055858 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6bdc102250 0x7f6bdc19efc0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f6bd402c700 tx=0x7f6bd4032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:15.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.007+0000 7f6bcb7fe640 1 -- 172.21.15.40:0/4099055858 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bd4039040 con 0x7f6bdc102250 2024-04-21T23:18:15.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.007+0000 7f6bcb7fe640 1 -- 172.21.15.40:0/4099055858 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6bd4044070 con 0x7f6bdc102250 2024-04-21T23:18:15.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.007+0000 7f6be0c39640 1 -- 172.21.15.40:0/4099055858 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bdc19f8c0 con 0x7f6bdc102250 2024-04-21T23:18:15.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.007+0000 7f6bcb7fe640 1 -- 172.21.15.40:0/4099055858 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bd40042a0 con 0x7f6bdc102250 2024-04-21T23:18:15.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.007+0000 7f6be0c39640 1 -- 172.21.15.40:0/4099055858 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bdc19fdb0 con 0x7f6bdc102250 2024-04-21T23:18:15.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.009+0000 7f6bcb7fe640 1 -- 172.21.15.40:0/4099055858 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f6bd4003970 con 0x7f6bdc102250 2024-04-21T23:18:15.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.010+0000 7f6be3828740 1 Processor -- start 2024-04-21T23:18:15.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.010+0000 7f6be3828740 1 -- start start 2024-04-21T23:18:15.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.010+0000 7f6be3828740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ea01ea8070 0x55ea01ea8490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:15.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.010+0000 7f6be3828740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ea01ea8a60 0x55ea01ead560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:15.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.010+0000 7f6be3828740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea01eadbd0 0x55ea01eb0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:15.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.010+0000 7f6be3828740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55ea01e0cdf0 con 0x55ea01eadbd0 2024-04-21T23:18:15.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.010+0000 7f6be3828740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55ea01db5640 con 0x55ea01ea8070 2024-04-21T23:18:15.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.010+0000 7f6be3828740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55ea01e08db0 con 0x55ea01ea8a60 2024-04-21T23:18:15.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.010+0000 7f6bcb7fe640 1 --2- 172.21.15.40:0/4099055858 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f6bc40a2b90 0x7f6bc40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:15.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.010+0000 7f6bd9d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea01eadbd0 0x55ea01eb0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:15.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.010+0000 7f6bdad9d640 1 --2- 172.21.15.40:0/4099055858 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f6bc40a2b90 0x7f6bc40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:15.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.010+0000 7f6bd9d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea01eadbd0 0x55ea01eb0060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46160/0 (socket says 172.21.15.40:46160) 2024-04-21T23:18:15.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.010+0000 7f6bd9d9b640 1 -- 172.21.15.40:0/559508972 learned_addr learned my addr 172.21.15.40:0/559508972 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:15.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.010+0000 7f6bda59c640 1 --2- 172.21.15.40:0/559508972 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ea01ea8a60 0x55ea01ead560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:15.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.010+0000 7f6bcb7fe640 1 -- 172.21.15.40:0/4099055858 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f6bd40f1990 con 0x7f6bdc102250 2024-04-21T23:18:15.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.010+0000 7f6bd9d9b640 1 --2- 172.21.15.40:0/559508972 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ea01ea8070 0x55ea01ea8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:15.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.011+0000 7f6bd9d9b640 1 -- 172.21.15.40:0/559508972 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ea01ea8a60 msgr2=0x55ea01ead560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:15.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.011+0000 7f6bd9d9b640 1 --2- 172.21.15.40:0/559508972 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ea01ea8a60 0x55ea01ead560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:15.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.011+0000 7f6bd9d9b640 1 -- 172.21.15.40:0/559508972 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ea01ea8070 msgr2=0x55ea01ea8490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:15.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.011+0000 7f6bd9d9b640 1 --2- 172.21.15.40:0/559508972 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ea01ea8070 0x55ea01ea8490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:15.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.011+0000 7f6bd9d9b640 1 -- 172.21.15.40:0/559508972 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ea01b212d0 con 0x55ea01eadbd0 2024-04-21T23:18:15.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.011+0000 7f6bda59c640 1 --2- 172.21.15.40:0/559508972 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ea01ea8a60 0x55ea01ead560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:18:15.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.011+0000 7f6bdad9d640 1 --2- 172.21.15.40:0/4099055858 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f6bc40a2b90 0x7f6bc40a5050 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f6bd40ba2b0 tx=0x7f6bd40095c0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:15.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.011+0000 7f6bd9d9b640 1 --2- 172.21.15.40:0/559508972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea01eadbd0 0x55ea01eb0060 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f6bd0012590 tx=0x7f6bd000b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c3cc2b5213542b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:15.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.011+0000 7f6bc8ff9640 1 -- 172.21.15.40:0/559508972 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bd00106b0 con 0x55ea01eadbd0 2024-04-21T23:18:15.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.011+0000 7f6bc8ff9640 1 -- 172.21.15.40:0/559508972 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6bd0010d30 con 0x55ea01eadbd0 2024-04-21T23:18:15.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.011+0000 7f6bc8ff9640 1 -- 172.21.15.40:0/559508972 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bd001ed80 con 0x55ea01eadbd0 2024-04-21T23:18:15.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.012+0000 7f6be3828740 1 -- 172.21.15.40:0/559508972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea01eadbd0 msgr2=0x55ea01eb0060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:15.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.012+0000 7f6be3828740 1 --2- 172.21.15.40:0/559508972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea01eadbd0 0x55ea01eb0060 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f6bd0012590 tx=0x7f6bd000b870 comp rx=0 tx=0).stop 2024-04-21T23:18:15.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.012+0000 7f6be3828740 1 -- 172.21.15.40:0/559508972 shutdown_connections 2024-04-21T23:18:15.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.012+0000 7f6be3828740 1 --2- 172.21.15.40:0/559508972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea01eadbd0 0x55ea01eb0060 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:15.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.012+0000 7f6be3828740 1 --2- 172.21.15.40:0/559508972 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ea01ea8a60 0x55ea01ead560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:15.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.012+0000 7f6be3828740 1 --2- 172.21.15.40:0/559508972 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ea01ea8070 0x55ea01ea8490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:15.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.012+0000 7f6be3828740 1 -- 172.21.15.40:0/559508972 >> 172.21.15.40:0/559508972 conn(0x55ea01ea35e0 msgr2=0x55ea01ea39e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:15.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.012+0000 7f6be3828740 1 -- 172.21.15.40:0/559508972 shutdown_connections 2024-04-21T23:18:15.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.012+0000 7f6be3828740 1 -- 172.21.15.40:0/559508972 wait complete. 2024-04-21T23:18:15.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.012+0000 7f6be3828740 1 -- start start 2024-04-21T23:18:15.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.012+0000 7f6be3828740 1 Processor -- start 2024-04-21T23:18:15.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.012+0000 7f6be3828740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea01ea8070 0x55ea01eb25e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:15.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.012+0000 7f6be3828740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ea01ea8a60 0x55ea01eb2b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:15.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.012+0000 7f6be3828740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ea01eadbd0 0x55ea01eb9b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:15.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.012+0000 7f6be3828740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55ea01eb0b80 con 0x55ea01ea8070 2024-04-21T23:18:15.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.012+0000 7f6be3828740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55ea01dc8be0 con 0x55ea01ea8a60 2024-04-21T23:18:15.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.012+0000 7f6be3828740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55ea01eb0d00 con 0x55ea01eadbd0 2024-04-21T23:18:15.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.013+0000 7f6be3828740 10 client.0 _ll_register_callbacks cb 0x55ea01dc85d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:18:15.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.013+0000 7f6bd9d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea01ea8070 0x55ea01eb25e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:15.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.013+0000 7f6bd9d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea01ea8070 0x55ea01eb25e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46176/0 (socket says 172.21.15.40:46176) 2024-04-21T23:18:15.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.013+0000 7f6bd9d9b640 1 -- 172.21.15.40:0/2654724957 learned_addr learned my addr 172.21.15.40:0/2654724957 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:15.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.013+0000 7f6bda59c640 1 --2- 172.21.15.40:0/2654724957 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ea01ea8a60 0x55ea01eb2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:15.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.013+0000 7f6bd9d9b640 1 -- 172.21.15.40:0/2654724957 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ea01eadbd0 msgr2=0x55ea01eb9b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:15.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.013+0000 7f6bd9d9b640 1 --2- 172.21.15.40:0/2654724957 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ea01eadbd0 0x55ea01eb9b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:15.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.013+0000 7f6bd9d9b640 1 -- 172.21.15.40:0/2654724957 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ea01ea8a60 msgr2=0x55ea01eb2b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:15.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.013+0000 7f6bd9d9b640 1 --2- 172.21.15.40:0/2654724957 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ea01ea8a60 0x55ea01eb2b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:15.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.013+0000 7f6bd9d9b640 1 -- 172.21.15.40:0/2654724957 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ea01beb6b0 con 0x55ea01ea8070 2024-04-21T23:18:15.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.013+0000 7f6bd9d9b640 1 --2- 172.21.15.40:0/2654724957 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea01ea8070 0x55ea01eb25e0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f6bd0006010 tx=0x7f6bd000bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d05ff203d4976763 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:15.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.013+0000 7f6bc1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-21T23:18:15.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.013+0000 7f6bc1ffb640 1 -- 172.21.15.40:0/2654724957 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bd002e060 con 0x55ea01ea8070 2024-04-21T23:18:15.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.013+0000 7f6bc1ffb640 1 -- 172.21.15.40:0/2654724957 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f6bd00104b0 con 0x55ea01ea8070 2024-04-21T23:18:15.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.013+0000 7f6be3828740 1 -- 172.21.15.40:0/2654724957 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6bd0010d30 con 0x55ea01ea8070 2024-04-21T23:18:15.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.013+0000 7f6bc1ffb640 1 -- 172.21.15.40:0/2654724957 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f6bd0035050 con 0x55ea01ea8070 2024-04-21T23:18:15.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.013+0000 7f6be3828740 10 client.15429 fetch_fsmap learned FSMap version 24 2024-04-21T23:18:15.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.013+0000 7f6be3828740 1 -- 172.21.15.40:0/2654724957 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55ea01ebcb40 con 0x55ea01ea8070 2024-04-21T23:18:15.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.014+0000 7f6bc1ffb640 1 -- 172.21.15.40:0/2654724957 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6bd0010d30 con 0x55ea01ea8070 2024-04-21T23:18:15.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.014+0000 7f6bc1ffb640 1 -- 172.21.15.40:0/2654724957 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f6bd001c020 con 0x55ea01ea8070 2024-04-21T23:18:15.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.014+0000 7f6be3828740 10 client.15429 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T23:18:15.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.014+0000 7f6be3828740 10 client.15429 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-21T23:18:15.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.014+0000 7f6be3828740 20 client.15429 populate_metadata read hostname 'smithi040' 2024-04-21T23:18:15.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.014+0000 7f6be3828740 1 --2- 172.21.15.40:0/2654724957 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55ea01ebeea0 0x55ea01ec0310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:15.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.014+0000 7f6be3828740 4 client.15429 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:18:15.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.014+0000 7f6be3828740 1 -- 172.21.15.40:0/2654724957 --> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ea01e1a330 con 0x55ea01ebeea0 2024-04-21T23:18:15.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.014+0000 7f6bda59c640 1 --2- 172.21.15.40:0/2654724957 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55ea01ebeea0 0x55ea01ec0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:15.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.014+0000 7f6bda59c640 1 --2- 172.21.15.40:0/2654724957 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55ea01ebeea0 0x55ea01ec0310 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=33f037f8f015518d server_cookie=586fd6644b0ac7dd in_seq=0 out_seq=0 2024-04-21T23:18:15.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.014+0000 7f6bc1ffb640 10 client.15429 ms_handle_connect on v2:172.21.15.111:6838/3712235548 2024-04-21T23:18:15.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.015+0000 7f6bc1ffb640 1 -- 172.21.15.40:0/2654724957 <== mds.0 v2:172.21.15.111:6838/3712235548 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ea01e1a330 con 0x55ea01ebeea0 2024-04-21T23:18:15.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.015+0000 7f6bc1ffb640 10 client.15429 handle_command_reply: tid=0 2024-04-21T23:18:15.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.015+0000 7f6be3828740 1 client.15429 shutdown 2024-04-21T23:18:15.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.015+0000 7f6be3828740 1 -- 172.21.15.40:0/2654724957 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea01ea8070 msgr2=0x55ea01eb25e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:15.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.015+0000 7f6be3828740 1 --2- 172.21.15.40:0/2654724957 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea01ea8070 0x55ea01eb25e0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f6bd0006010 tx=0x7f6bd000bdc0 comp rx=0 tx=0).stop 2024-04-21T23:18:15.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.015+0000 7f6be3828740 1 -- 172.21.15.40:0/2654724957 shutdown_connections 2024-04-21T23:18:15.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.015+0000 7f6be3828740 1 --2- 172.21.15.40:0/2654724957 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55ea01ebeea0 0x55ea01ec0310 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:15.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.015+0000 7f6be3828740 1 --2- 172.21.15.40:0/2654724957 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ea01ea8a60 0x55ea01eb2b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:15.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.015+0000 7f6be3828740 1 --2- 172.21.15.40:0/2654724957 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ea01eadbd0 0x55ea01eb9b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:15.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.015+0000 7f6bc1ffb640 0 client.15429 ms_handle_reset on v2:172.21.15.111:6838/3712235548 2024-04-21T23:18:15.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.015+0000 7f6be3828740 1 --2- 172.21.15.40:0/2654724957 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea01ea8070 0x55ea01eb25e0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:15.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.015+0000 7f6be3828740 1 -- 172.21.15.40:0/2654724957 >> 172.21.15.40:0/2654724957 conn(0x55ea01ea35e0 msgr2=0x55ea01ea2720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:15.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.015+0000 7f6be3828740 1 -- 172.21.15.40:0/2654724957 shutdown_connections 2024-04-21T23:18:15.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.015+0000 7f6be3828740 1 -- 172.21.15.40:0/2654724957 wait complete. 2024-04-21T23:18:15.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.015+0000 7f6be3828740 20 client.15429 trim_cache size 0 max 16384 2024-04-21T23:18:15.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.034+0000 7f6be3828740 1 Processor -- start 2024-04-21T23:18:15.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.034+0000 7f6be3828740 1 -- start start 2024-04-21T23:18:15.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.034+0000 7f6be3828740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ea01ea8070 0x55ea01eb21f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:15.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.034+0000 7f6be3828740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea01ea8a60 0x55ea01eb2730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:15.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.034+0000 7f6be3828740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ea01eadbd0 0x55ea01ea1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:15.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.034+0000 7f6be3828740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55ea01ebe650 con 0x55ea01ea8a60 2024-04-21T23:18:15.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.034+0000 7f6be3828740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55ea01ebed20 con 0x55ea01eadbd0 2024-04-21T23:18:15.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.034+0000 7f6be3828740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55ea01ebf3f0 con 0x55ea01ea8070 2024-04-21T23:18:15.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.034+0000 7f6bda59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea01ea8a60 0x55ea01eb2730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:15.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.034+0000 7f6bda59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea01ea8a60 0x55ea01eb2730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46186/0 (socket says 172.21.15.40:46186) 2024-04-21T23:18:15.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.034+0000 7f6bda59c640 1 -- 172.21.15.40:0/2226560187 learned_addr learned my addr 172.21.15.40:0/2226560187 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:15.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.035+0000 7f6bd9d9b640 1 --2- 172.21.15.40:0/2226560187 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ea01ea8070 0x55ea01eb21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:15.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.035+0000 7f6bda59c640 1 -- 172.21.15.40:0/2226560187 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ea01ea8070 msgr2=0x55ea01eb21f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:15.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.035+0000 7f6bda59c640 1 --2- 172.21.15.40:0/2226560187 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ea01ea8070 0x55ea01eb21f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:15.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.035+0000 7f6bda59c640 1 -- 172.21.15.40:0/2226560187 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ea01eadbd0 msgr2=0x55ea01ea1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:15.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.035+0000 7f6bda59c640 1 --2- 172.21.15.40:0/2226560187 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ea01eadbd0 0x55ea01ea1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:15.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.035+0000 7f6bda59c640 1 -- 172.21.15.40:0/2226560187 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ea01eb0ec0 con 0x55ea01ea8a60 2024-04-21T23:18:15.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.035+0000 7f6bda59c640 1 --2- 172.21.15.40:0/2226560187 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea01ea8a60 0x55ea01eb2730 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x55ea01ea94d0 tx=0x7f6bcc0138f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5691f394b5d70778 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:15.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.035+0000 7f6bc3fff640 1 -- 172.21.15.40:0/2226560187 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bcc004030 con 0x55ea01ea8a60 2024-04-21T23:18:15.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.035+0000 7f6bc3fff640 1 -- 172.21.15.40:0/2226560187 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6bcc01b890 con 0x55ea01ea8a60 2024-04-21T23:18:15.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.035+0000 7f6bc3fff640 1 -- 172.21.15.40:0/2226560187 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bcc003b80 con 0x55ea01ea8a60 2024-04-21T23:18:15.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.036+0000 7f6be3828740 1 -- 172.21.15.40:0/2226560187 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea01ea8a60 msgr2=0x55ea01eb2730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:15.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.036+0000 7f6be3828740 1 --2- 172.21.15.40:0/2226560187 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea01ea8a60 0x55ea01eb2730 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x55ea01ea94d0 tx=0x7f6bcc0138f0 comp rx=0 tx=0).stop 2024-04-21T23:18:15.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.036+0000 7f6be3828740 1 -- 172.21.15.40:0/2226560187 shutdown_connections 2024-04-21T23:18:15.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.036+0000 7f6be3828740 1 --2- 172.21.15.40:0/2226560187 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ea01eadbd0 0x55ea01ea1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:15.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.036+0000 7f6be3828740 1 --2- 172.21.15.40:0/2226560187 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea01ea8a60 0x55ea01eb2730 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:15.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.036+0000 7f6be3828740 1 --2- 172.21.15.40:0/2226560187 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ea01ea8070 0x55ea01eb21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:15.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.036+0000 7f6be3828740 1 -- 172.21.15.40:0/2226560187 >> 172.21.15.40:0/2226560187 conn(0x55ea01ea35e0 msgr2=0x55ea01ea39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:15.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.036+0000 7f6be3828740 1 -- 172.21.15.40:0/2226560187 shutdown_connections 2024-04-21T23:18:15.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.036+0000 7f6be3828740 1 -- 172.21.15.40:0/2226560187 wait complete. 2024-04-21T23:18:15.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.036+0000 7f6be3828740 1 -- start start 2024-04-21T23:18:15.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.036+0000 7f6be3828740 1 Processor -- start 2024-04-21T23:18:15.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.036+0000 7f6be3828740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ea01ea8070 0x55ea01f21080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:15.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.036+0000 7f6be3828740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea01ea8a60 0x55ea01f215c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:15.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.036+0000 7f6be3828740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ea01eadbd0 0x55ea01f28180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:15.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.036+0000 7f6be3828740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55ea01ec0190 con 0x55ea01ea8a60 2024-04-21T23:18:15.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.036+0000 7f6be3828740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55ea01ebfac0 con 0x55ea01eadbd0 2024-04-21T23:18:15.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.036+0000 7f6be3828740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55ea01ec0860 con 0x55ea01ea8070 2024-04-21T23:18:15.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.036+0000 7f6be3828740 10 client.0 _ll_register_callbacks cb 0x7f6bd00056d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:18:15.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.036+0000 7f6bda59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea01ea8a60 0x55ea01f215c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:15.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.036+0000 7f6bda59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea01ea8a60 0x55ea01f215c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46196/0 (socket says 172.21.15.40:46196) 2024-04-21T23:18:15.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.036+0000 7f6bda59c640 1 -- 172.21.15.40:0/2471084966 learned_addr learned my addr 172.21.15.40:0/2471084966 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:15.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.037+0000 7f6bd9d9b640 1 --2- 172.21.15.40:0/2471084966 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ea01eadbd0 0x55ea01f28180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:15.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.037+0000 7f6bda59c640 1 -- 172.21.15.40:0/2471084966 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ea01ea8070 msgr2=0x55ea01f21080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:15.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.037+0000 7f6bd9d9b640 1 --2- 172.21.15.40:0/2471084966 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ea01ea8070 0x55ea01f21080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:15.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.037+0000 7f6bda59c640 1 --2- 172.21.15.40:0/2471084966 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ea01ea8070 0x55ea01f21080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:15.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.037+0000 7f6bda59c640 1 -- 172.21.15.40:0/2471084966 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ea01eadbd0 msgr2=0x55ea01f28180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:15.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.037+0000 7f6bda59c640 1 --2- 172.21.15.40:0/2471084966 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ea01eadbd0 0x55ea01f28180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:15.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.037+0000 7f6bda59c640 1 -- 172.21.15.40:0/2471084966 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ea01f28790 con 0x55ea01ea8a60 2024-04-21T23:18:15.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.037+0000 7f6bd9d9b640 1 --2- 172.21.15.40:0/2471084966 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ea01eadbd0 0x55ea01f28180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:18:15.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.037+0000 7f6bda59c640 1 --2- 172.21.15.40:0/2471084966 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea01ea8a60 0x55ea01f215c0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc01e040 tx=0x7f6bcc0036e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6137f658bd3e2b0c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:15.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.037+0000 7f6bc1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-21T23:18:15.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.037+0000 7f6bc1ffb640 1 -- 172.21.15.40:0/2471084966 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bcc02d440 con 0x55ea01ea8a60 2024-04-21T23:18:15.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.037+0000 7f6bc1ffb640 1 -- 172.21.15.40:0/2471084966 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f6bcc011440 con 0x55ea01ea8a60 2024-04-21T23:18:15.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.037+0000 7f6be3828740 1 -- 172.21.15.40:0/2471084966 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6bcc01b890 con 0x55ea01ea8a60 2024-04-21T23:18:15.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.037+0000 7f6bc1ffb640 1 -- 172.21.15.40:0/2471084966 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f6bcc011640 con 0x55ea01ea8a60 2024-04-21T23:18:15.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.037+0000 7f6be3828740 10 client.15435 fetch_fsmap learned FSMap version 24 2024-04-21T23:18:15.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.037+0000 7f6be3828740 1 -- 172.21.15.40:0/2471084966 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55ea01f29ac0 con 0x55ea01ea8a60 2024-04-21T23:18:15.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.037+0000 7f6bc1ffb640 1 -- 172.21.15.40:0/2471084966 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6bcc01b890 con 0x55ea01ea8a60 2024-04-21T23:18:15.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.038+0000 7f6bc1ffb640 1 -- 172.21.15.40:0/2471084966 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f6bcc02b070 con 0x55ea01ea8a60 2024-04-21T23:18:15.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.038+0000 7f6be3828740 10 client.15435 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T23:18:15.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.038+0000 7f6be3828740 10 client.15435 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-21T23:18:15.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.038+0000 7f6be3828740 20 client.15435 populate_metadata read hostname 'smithi040' 2024-04-21T23:18:15.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.038+0000 7f6be3828740 1 --2- 172.21.15.40:0/2471084966 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55ea01f2e4f0 0x55ea01f2d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:15.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.038+0000 7f6be3828740 4 client.15435 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-21T23:18:15.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.038+0000 7f6be3828740 1 -- 172.21.15.40:0/2471084966 --> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] -- command(tid 0: {"prefix": "status"}) -- 0x55ea01e14680 con 0x55ea01f2e4f0 2024-04-21T23:18:15.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.038+0000 7f6bd9d9b640 1 --2- 172.21.15.40:0/2471084966 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55ea01f2e4f0 0x55ea01f2d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:15.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.038+0000 7f6bc1ffb640 10 client.15435 ms_handle_connect on v2:172.21.15.111:6838/3712235548 2024-04-21T23:18:15.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.038+0000 7f6bd9d9b640 1 --2- 172.21.15.40:0/2471084966 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55ea01f2e4f0 0x55ea01f2d200 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=99d42db28ab9f66e server_cookie=10f1e3002d14f108 in_seq=0 out_seq=0 2024-04-21T23:18:15.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.039+0000 7f6bc1ffb640 1 -- 172.21.15.40:0/2471084966 <== mds.0 v2:172.21.15.111:6838/3712235548 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55ea01e14680 con 0x55ea01f2e4f0 2024-04-21T23:18:15.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.039+0000 7f6bc1ffb640 10 client.15435 handle_command_reply: tid=0 2024-04-21T23:18:15.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.039+0000 7f6be3828740 1 client.15435 shutdown 2024-04-21T23:18:15.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.039+0000 7f6be3828740 1 -- 172.21.15.40:0/2471084966 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea01ea8a60 msgr2=0x55ea01f215c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:15.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.039+0000 7f6be3828740 1 --2- 172.21.15.40:0/2471084966 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea01ea8a60 0x55ea01f215c0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc01e040 tx=0x7f6bcc0036e0 comp rx=0 tx=0).stop 2024-04-21T23:18:15.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.039+0000 7f6be3828740 1 -- 172.21.15.40:0/2471084966 shutdown_connections 2024-04-21T23:18:15.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.039+0000 7f6be3828740 1 --2- 172.21.15.40:0/2471084966 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55ea01f2e4f0 0x55ea01f2d200 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:15.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.039+0000 7f6be3828740 1 --2- 172.21.15.40:0/2471084966 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ea01eadbd0 0x55ea01f28180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:15.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.039+0000 7f6be3828740 1 --2- 172.21.15.40:0/2471084966 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea01ea8a60 0x55ea01f215c0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:15.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.039+0000 7f6be3828740 1 --2- 172.21.15.40:0/2471084966 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ea01ea8070 0x55ea01f21080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:15.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.039+0000 7f6bc1ffb640 0 client.15435 ms_handle_reset on v2:172.21.15.111:6838/3712235548 2024-04-21T23:18:15.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.039+0000 7f6be3828740 1 -- 172.21.15.40:0/2471084966 >> 172.21.15.40:0/2471084966 conn(0x55ea01ea35e0 msgr2=0x55ea01eb21f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:15.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.039+0000 7f6be3828740 1 -- 172.21.15.40:0/2471084966 shutdown_connections 2024-04-21T23:18:15.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.039+0000 7f6be3828740 1 -- 172.21.15.40:0/2471084966 wait complete. 2024-04-21T23:18:15.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.039+0000 7f6be3828740 20 client.15435 trim_cache size 0 max 16384 2024-04-21T23:18:15.045 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-21T23:18:15.045 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "d814e4dc-0033-11ef-bc93-c7b262605968", 2024-04-21T23:18:15.045 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 0, 2024-04-21T23:18:15.045 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 24521, 2024-04-21T23:18:15.045 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-04-21T23:18:15.045 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-04-21T23:18:15.046 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-04-21T23:18:15.046 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 138.65566787500001, 2024-04-21T23:18:15.046 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 24, 2024-04-21T23:18:15.046 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 94, 2024-04-21T23:18:15.046 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 90, 2024-04-21T23:18:15.046 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 145.04702328299999 2024-04-21T23:18:15.046 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-21T23:18:15.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.039+0000 7f6be0c39640 1 -- 172.21.15.40:0/4099055858 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f6bc40a2b90 msgr2=0x7f6bc40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:15.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.039+0000 7f6be0c39640 1 --2- 172.21.15.40:0/4099055858 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f6bc40a2b90 0x7f6bc40a5050 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f6bd40ba2b0 tx=0x7f6bd40095c0 comp rx=0 tx=0).stop 2024-04-21T23:18:15.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.039+0000 7f6be0c39640 1 -- 172.21.15.40:0/4099055858 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6bdc102250 msgr2=0x7f6bdc19efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:15.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.039+0000 7f6be0c39640 1 --2- 172.21.15.40:0/4099055858 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6bdc102250 0x7f6bdc19efc0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f6bd402c700 tx=0x7f6bd4032f20 comp rx=0 tx=0).stop 2024-04-21T23:18:15.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.039+0000 7f6be0c39640 1 -- 172.21.15.40:0/4099055858 shutdown_connections 2024-04-21T23:18:15.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.039+0000 7f6be0c39640 1 --2- 172.21.15.40:0/4099055858 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f6bc40a2b90 0x7f6bc40a5050 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:15.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.039+0000 7f6be0c39640 1 --2- 172.21.15.40:0/4099055858 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6bdc101890 0x7f6bdc197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:15.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.039+0000 7f6be0c39640 1 --2- 172.21.15.40:0/4099055858 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6bdc102250 0x7f6bdc19efc0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:15.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.039+0000 7f6be0c39640 1 --2- 172.21.15.40:0/4099055858 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6bdc1006a0 0x7f6bdc197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:15.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.039+0000 7f6be0c39640 1 -- 172.21.15.40:0/4099055858 >> 172.21.15.40:0/4099055858 conn(0x7f6bdc0fbc30 msgr2=0x7f6bdc0fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:15.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.039+0000 7f6be0c39640 1 -- 172.21.15.40:0/4099055858 shutdown_connections 2024-04-21T23:18:15.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.040+0000 7f6be0c39640 1 -- 172.21.15.40:0/4099055858 wait complete. 2024-04-21T23:18:15.050 INFO:teuthology.run_tasks:Running task sequential... 2024-04-21T23:18:15.062 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-21T23:18:15.062 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-21T23:18:15.062 INFO:teuthology.task.print:Enabling mgr modules 2024-04-21T23:18:15.062 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-04-21T23:18:15.063 INFO:teuthology.task.exec:Executing custom commands... 2024-04-21T23:18:15.063 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi040.front.sepia.ceph.com 2024-04-21T23:18:15.063 DEBUG:teuthology.orchestra.run.smithi040:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-04-21T23:18:15.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.165+0000 7feec6927640 1 -- 172.21.15.40:0/3331486401 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feec0103380 msgr2=0x7feec0109c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:15.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.165+0000 7feec6927640 1 --2- 172.21.15.40:0/3331486401 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feec0103380 0x7feec0109c10 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7feeb8009ca0 tx=0x7feeb802c0f0 comp rx=0 tx=0).stop 2024-04-21T23:18:15.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.166+0000 7feec6927640 1 -- 172.21.15.40:0/3331486401 shutdown_connections 2024-04-21T23:18:15.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.166+0000 7feec6927640 1 --2- 172.21.15.40:0/3331486401 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feec01029c0 0x7feec0102e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:15.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.166+0000 7feec6927640 1 --2- 172.21.15.40:0/3331486401 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feec0103380 0x7feec0109c10 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:15.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.166+0000 7feec6927640 1 --2- 172.21.15.40:0/3331486401 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feec01017d0 0x7feec0101bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:15.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.166+0000 7feec6927640 1 -- 172.21.15.40:0/3331486401 >> 172.21.15.40:0/3331486401 conn(0x7feec00fcd60 msgr2=0x7feec00ff180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:15.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.166+0000 7feec6927640 1 -- 172.21.15.40:0/3331486401 shutdown_connections 2024-04-21T23:18:15.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.166+0000 7feec6927640 1 -- 172.21.15.40:0/3331486401 wait complete. 2024-04-21T23:18:15.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.166+0000 7feec6927640 1 Processor -- start 2024-04-21T23:18:15.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.166+0000 7feec6927640 1 -- start start 2024-04-21T23:18:15.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.166+0000 7feec6927640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feec01017d0 0x7feec01979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:15.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.166+0000 7feec6927640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feec01029c0 0x7feec0197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:15.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.166+0000 7feec6927640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feec0103380 0x7feec019ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:15.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.166+0000 7feec6927640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7feec010d450 con 0x7feec01029c0 2024-04-21T23:18:15.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.166+0000 7feec6927640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7feec010d2d0 con 0x7feec0103380 2024-04-21T23:18:15.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.166+0000 7feec6927640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7feec010d5d0 con 0x7feec01017d0 2024-04-21T23:18:15.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.167+0000 7feebf7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feec01029c0 0x7feec0197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:15.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.167+0000 7feebf7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feec01029c0 0x7feec0197f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46210/0 (socket says 172.21.15.40:46210) 2024-04-21T23:18:15.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.167+0000 7feebf7fe640 1 -- 172.21.15.40:0/4042304536 learned_addr learned my addr 172.21.15.40:0/4042304536 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:15.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.167+0000 7feec4ec4640 1 --2- 172.21.15.40:0/4042304536 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feec0103380 0x7feec019ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:15.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.167+0000 7feebffff640 1 --2- 172.21.15.40:0/4042304536 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feec01017d0 0x7feec01979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:15.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.167+0000 7feebf7fe640 1 -- 172.21.15.40:0/4042304536 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feec01017d0 msgr2=0x7feec01979d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:15.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.167+0000 7feebf7fe640 1 --2- 172.21.15.40:0/4042304536 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feec01017d0 0x7feec01979d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:15.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.167+0000 7feebf7fe640 1 -- 172.21.15.40:0/4042304536 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feec0103380 msgr2=0x7feec019ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:15.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.167+0000 7feebf7fe640 1 --2- 172.21.15.40:0/4042304536 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feec0103380 0x7feec019ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:15.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.167+0000 7feebf7fe640 1 -- 172.21.15.40:0/4042304536 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feec019f5f0 con 0x7feec01029c0 2024-04-21T23:18:15.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.167+0000 7feebffff640 1 --2- 172.21.15.40:0/4042304536 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feec01017d0 0x7feec01979d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:18:15.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.167+0000 7feec4ec4640 1 --2- 172.21.15.40:0/4042304536 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feec0103380 0x7feec019ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:18:15.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.167+0000 7feebf7fe640 1 --2- 172.21.15.40:0/4042304536 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feec01029c0 0x7feec0197f10 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7feeb400d950 tx=0x7feeb400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:15.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.167+0000 7feebd7fa640 1 -- 172.21.15.40:0/4042304536 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeb4017030 con 0x7feec01029c0 2024-04-21T23:18:15.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.167+0000 7feebd7fa640 1 -- 172.21.15.40:0/4042304536 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feeb4004020 con 0x7feec01029c0 2024-04-21T23:18:15.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.167+0000 7feebd7fa640 1 -- 172.21.15.40:0/4042304536 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeb40047a0 con 0x7feec01029c0 2024-04-21T23:18:15.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.167+0000 7feec6927640 1 -- 172.21.15.40:0/4042304536 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feec019f8e0 con 0x7feec01029c0 2024-04-21T23:18:15.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.167+0000 7feec6927640 1 -- 172.21.15.40:0/4042304536 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feec019fe30 con 0x7feec01029c0 2024-04-21T23:18:15.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.168+0000 7feec6927640 1 -- 172.21.15.40:0/4042304536 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feec0108a90 con 0x7feec01029c0 2024-04-21T23:18:15.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.169+0000 7feebd7fa640 1 -- 172.21.15.40:0/4042304536 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7feeb40041d0 con 0x7feec01029c0 2024-04-21T23:18:15.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.170+0000 7feebd7fa640 1 --2- 172.21.15.40:0/4042304536 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7feea80a2c30 0x7feea80a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:15.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.170+0000 7feebffff640 1 --2- 172.21.15.40:0/4042304536 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7feea80a2c30 0x7feea80a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:15.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.170+0000 7feebd7fa640 1 -- 172.21.15.40:0/4042304536 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7feeb40ceeb0 con 0x7feec01029c0 2024-04-21T23:18:15.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.171+0000 7feebffff640 1 --2- 172.21.15.40:0/4042304536 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7feea80a2c30 0x7feea80a50f0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7feec0102820 tx=0x7feeb0005e70 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:15.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.175+0000 7feebd7fa640 1 -- 172.21.15.40:0/4042304536 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7feeb409ba40 con 0x7feec01029c0 2024-04-21T23:18:15.181 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:15 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/2057284756' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:18:15.181 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:15 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/2115309490' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:18:15.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:15.413+0000 7feec6927640 1 -- 172.21.15.40:0/4042304536 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7feec0062340 con 0x7feec01029c0 2024-04-21T23:18:15.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:15 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/2057284756' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:18:15.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:15 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/2115309490' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:18:15.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:15 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/2057284756' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:18:15.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:15 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/2115309490' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:18:16.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.032+0000 7feebd7fa640 1 -- 172.21.15.40:0/4042304536 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) ==== 92+0+0 (secure 0 0 0) 0x7feeb409b340 con 0x7feec01029c0 2024-04-21T23:18:16.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.035+0000 7feec6927640 1 -- 172.21.15.40:0/4042304536 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7feea80a2c30 msgr2=0x7feea80a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:16.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.035+0000 7feec6927640 1 --2- 172.21.15.40:0/4042304536 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7feea80a2c30 0x7feea80a50f0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7feec0102820 tx=0x7feeb0005e70 comp rx=0 tx=0).stop 2024-04-21T23:18:16.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.035+0000 7feec6927640 1 -- 172.21.15.40:0/4042304536 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feec01029c0 msgr2=0x7feec0197f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:16.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.035+0000 7feec6927640 1 --2- 172.21.15.40:0/4042304536 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feec01029c0 0x7feec0197f10 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7feeb400d950 tx=0x7feeb400de10 comp rx=0 tx=0).stop 2024-04-21T23:18:16.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.036+0000 7feec6927640 1 -- 172.21.15.40:0/4042304536 shutdown_connections 2024-04-21T23:18:16.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.036+0000 7feec6927640 1 --2- 172.21.15.40:0/4042304536 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7feea80a2c30 0x7feea80a50f0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:16.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.036+0000 7feec6927640 1 --2- 172.21.15.40:0/4042304536 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feec0103380 0x7feec019ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:16.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.036+0000 7feec6927640 1 --2- 172.21.15.40:0/4042304536 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feec01029c0 0x7feec0197f10 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:16.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.036+0000 7feec6927640 1 --2- 172.21.15.40:0/4042304536 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feec01017d0 0x7feec01979d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:16.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.036+0000 7feec6927640 1 -- 172.21.15.40:0/4042304536 >> 172.21.15.40:0/4042304536 conn(0x7feec00fcd60 msgr2=0x7feec0105c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:16.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.036+0000 7feec6927640 1 -- 172.21.15.40:0/4042304536 shutdown_connections 2024-04-21T23:18:16.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.036+0000 7feec6927640 1 -- 172.21.15.40:0/4042304536 wait complete. 2024-04-21T23:18:16.054 DEBUG:teuthology.orchestra.run.smithi040:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-04-21T23:18:16.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.165+0000 7f65661e5640 1 -- 172.21.15.40:0/4122588056 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6560101890 msgr2=0x7f6560101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:16.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.165+0000 7f65661e5640 1 --2- 172.21.15.40:0/4122588056 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6560101890 0x7f6560101d10 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f6550009900 tx=0x7f655002bc30 comp rx=0 tx=0).stop 2024-04-21T23:18:16.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.165+0000 7f65661e5640 1 -- 172.21.15.40:0/4122588056 shutdown_connections 2024-04-21T23:18:16.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.165+0000 7f65661e5640 1 --2- 172.21.15.40:0/4122588056 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6560102250 0x7f6560108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:16.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.165+0000 7f65661e5640 1 --2- 172.21.15.40:0/4122588056 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6560101890 0x7f6560101d10 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:16.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.165+0000 7f65661e5640 1 --2- 172.21.15.40:0/4122588056 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65601006a0 0x7f6560100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:16.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.165+0000 7f65661e5640 1 -- 172.21.15.40:0/4122588056 >> 172.21.15.40:0/4122588056 conn(0x7f65600fbc10 msgr2=0x7f65600fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:16.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.166+0000 7f65661e5640 1 -- 172.21.15.40:0/4122588056 shutdown_connections 2024-04-21T23:18:16.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.166+0000 7f65661e5640 1 -- 172.21.15.40:0/4122588056 wait complete. 2024-04-21T23:18:16.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.166+0000 7f65661e5640 1 Processor -- start 2024-04-21T23:18:16.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.166+0000 7f65661e5640 1 -- start start 2024-04-21T23:18:16.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.166+0000 7f65661e5640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65601006a0 0x7f6560199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:16.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.166+0000 7f65661e5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6560101890 0x7f656019a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:16.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.167+0000 7f65661e5640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6560102250 0x7f65601a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:16.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.167+0000 7f65661e5640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f656010c320 con 0x7f6560101890 2024-04-21T23:18:16.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.167+0000 7f65661e5640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f656010c1a0 con 0x7f65601006a0 2024-04-21T23:18:16.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.167+0000 7f65661e5640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f656010c4a0 con 0x7f6560102250 2024-04-21T23:18:16.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.167+0000 7f655f7fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65601006a0 0x7f6560199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:16.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.167+0000 7f655effd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6560101890 0x7f656019a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:16.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.167+0000 7f655f7fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65601006a0 0x7f6560199c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.40:46194/0 (socket says 172.21.15.40:46194) 2024-04-21T23:18:16.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.167+0000 7f655f7fe640 1 -- 172.21.15.40:0/42127499 learned_addr learned my addr 172.21.15.40:0/42127499 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:16.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.167+0000 7f655ffff640 1 --2- 172.21.15.40:0/42127499 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6560102250 0x7f65601a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:16.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.167+0000 7f655f7fe640 1 -- 172.21.15.40:0/42127499 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6560102250 msgr2=0x7f65601a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:16.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.167+0000 7f655f7fe640 1 --2- 172.21.15.40:0/42127499 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6560102250 0x7f65601a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:16.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.167+0000 7f655f7fe640 1 -- 172.21.15.40:0/42127499 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6560101890 msgr2=0x7f656019a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:16.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.167+0000 7f655f7fe640 1 --2- 172.21.15.40:0/42127499 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6560101890 0x7f656019a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:16.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.167+0000 7f655f7fe640 1 -- 172.21.15.40:0/42127499 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65601a1830 con 0x7f65601006a0 2024-04-21T23:18:16.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.167+0000 7f655f7fe640 1 --2- 172.21.15.40:0/42127499 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65601006a0 0x7f6560199c20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f654800e9c0 tx=0x7f654800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:16.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.167+0000 7f655cff9640 1 -- 172.21.15.40:0/42127499 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f654800cd80 con 0x7f65601006a0 2024-04-21T23:18:16.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.167+0000 7f65661e5640 1 -- 172.21.15.40:0/42127499 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65601a1b20 con 0x7f65601006a0 2024-04-21T23:18:16.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.167+0000 7f655cff9640 1 -- 172.21.15.40:0/42127499 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65480035f0 con 0x7f65601006a0 2024-04-21T23:18:16.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.167+0000 7f655cff9640 1 -- 172.21.15.40:0/42127499 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6548010690 con 0x7f65601006a0 2024-04-21T23:18:16.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.167+0000 7f65661e5640 1 -- 172.21.15.40:0/42127499 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65601a20c0 con 0x7f65601006a0 2024-04-21T23:18:16.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.168+0000 7f65661e5640 1 -- 172.21.15.40:0/42127499 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65601a2290 con 0x7f65601006a0 2024-04-21T23:18:16.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.170+0000 7f655cff9640 1 -- 172.21.15.40:0/42127499 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 19) ==== 137850+0+0 (secure 0 0 0) 0x7f65480037a0 con 0x7f65601006a0 2024-04-21T23:18:16.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.170+0000 7f655cff9640 1 --2- 172.21.15.40:0/42127499 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f654c0a2c80 0x7f654c0a5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:16.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.170+0000 7f655effd640 1 -- 172.21.15.40:0/42127499 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f654c0a2c80 msgr2=0x7f654c0a5140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:6800/2500760788 2024-04-21T23:18:16.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.170+0000 7f655cff9640 1 -- 172.21.15.40:0/42127499 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f65480cf870 con 0x7f65601006a0 2024-04-21T23:18:16.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.170+0000 7f655effd640 1 --2- 172.21.15.40:0/42127499 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f654c0a2c80 0x7f654c0a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T23:18:16.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.174+0000 7f655cff9640 1 -- 172.21.15.40:0/42127499 <== mon.2 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f654809c400 con 0x7f65601006a0 2024-04-21T23:18:16.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.364+0000 7f65661e5640 1 -- 172.21.15.40:0/42127499 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7f65601a2460 con 0x7f65601006a0 2024-04-21T23:18:16.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.370+0000 7f655cff9640 1 -- 172.21.15.40:0/42127499 <== mon.2 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v27)=0 v27) ==== 138+0+0 (secure 0 0 0) 0x7f6548086020 con 0x7f65601006a0 2024-04-21T23:18:16.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.371+0000 7f655effd640 1 -- 172.21.15.40:0/42127499 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f654c0a2c80 msgr2=0x7f654c0a5140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:6800/2500760788 2024-04-21T23:18:16.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.371+0000 7f655effd640 1 --2- 172.21.15.40:0/42127499 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f654c0a2c80 0x7f654c0a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-21T23:18:16.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.372+0000 7f65661e5640 1 -- 172.21.15.40:0/42127499 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f654c0a2c80 msgr2=0x7f654c0a5140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T23:18:16.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.372+0000 7f65661e5640 1 --2- 172.21.15.40:0/42127499 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f654c0a2c80 0x7f654c0a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:16.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.372+0000 7f65661e5640 1 -- 172.21.15.40:0/42127499 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65601006a0 msgr2=0x7f6560199c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:16.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.372+0000 7f65661e5640 1 --2- 172.21.15.40:0/42127499 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65601006a0 0x7f6560199c20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f654800e9c0 tx=0x7f654800ee80 comp rx=0 tx=0).stop 2024-04-21T23:18:16.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.372+0000 7f65661e5640 1 -- 172.21.15.40:0/42127499 shutdown_connections 2024-04-21T23:18:16.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.372+0000 7f65661e5640 1 --2- 172.21.15.40:0/42127499 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f654c0a2c80 0x7f654c0a5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:16.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.372+0000 7f65661e5640 1 --2- 172.21.15.40:0/42127499 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6560102250 0x7f65601a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:16.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.372+0000 7f65661e5640 1 --2- 172.21.15.40:0/42127499 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6560101890 0x7f656019a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:16.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.372+0000 7f65661e5640 1 --2- 172.21.15.40:0/42127499 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65601006a0 0x7f6560199c20 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:16.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.372+0000 7f65661e5640 1 -- 172.21.15.40:0/42127499 >> 172.21.15.40:0/42127499 conn(0x7f65600fbc10 msgr2=0x7f65600fd630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:16.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.372+0000 7f65661e5640 1 -- 172.21.15.40:0/42127499 shutdown_connections 2024-04-21T23:18:16.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.372+0000 7f65661e5640 1 -- 172.21.15.40:0/42127499 wait complete. 2024-04-21T23:18:16.382 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:16 smithi040 ceph-mon[27984]: pgmap v366: 129 pgs: 129 active+clean; 641 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T23:18:16.382 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:16 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/4042304536' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-21T23:18:16.382 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:18:16 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: ignoring --setuser ceph since I am not root 2024-04-21T23:18:16.382 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:18:16 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: ignoring --setgroup ceph since I am not root 2024-04-21T23:18:16.382 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:18:16 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:18:16.225+0000 7f2f22aac200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-21T23:18:16.383 DEBUG:teuthology.orchestra.run.smithi040:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-04-21T23:18:16.391 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:16 smithi111 ceph-mon[31085]: pgmap v366: 129 pgs: 129 active+clean; 641 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T23:18:16.391 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:16 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/4042304536' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-21T23:18:16.392 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:18:16 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-y[32479]: ignoring --setuser ceph since I am not root 2024-04-21T23:18:16.392 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:18:16 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-y[32479]: ignoring --setgroup ceph since I am not root 2024-04-21T23:18:16.392 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:18:16 smithi111 ceph-mgr[32483]: -- 172.21.15.111:0/2218879040 <== mon.2 v2:172.21.15.111:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559f366765a0 con 0x559f3664b000 2024-04-21T23:18:16.392 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:18:16 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-y[32479]: 2024-04-21T23:18:16.210+0000 7f9dc849f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-21T23:18:16.392 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:18:16 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-y[32479]: 2024-04-21T23:18:16.392+0000 7f9dc849f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-21T23:18:16.399 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:16 smithi174 ceph-mon[31409]: pgmap v366: 129 pgs: 129 active+clean; 641 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T23:18:16.399 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:16 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/4042304536' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-21T23:18:16.399 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Apr 21 23:18:16 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-z[33257]: ignoring --setuser ceph since I am not root 2024-04-21T23:18:16.400 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Apr 21 23:18:16 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-z[33257]: ignoring --setgroup ceph since I am not root 2024-04-21T23:18:16.400 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Apr 21 23:18:16 smithi174 ceph-mgr[33261]: -- 172.21.15.174:0/3314697049 <== mon.0 v2:172.21.15.40:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559cdc18c5a0 con 0x559cdc160800 2024-04-21T23:18:16.400 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Apr 21 23:18:16 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-z[33257]: 2024-04-21T23:18:16.214+0000 7f31e2b84200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-21T23:18:16.400 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Apr 21 23:18:16 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-z[33257]: 2024-04-21T23:18:16.399+0000 7f31e2b84200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-21T23:18:16.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.490+0000 7f12596b5640 1 -- 172.21.15.40:0/3519409448 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1254101940 msgr2=0x7f1254101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:16.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.490+0000 7f12596b5640 1 --2- 172.21.15.40:0/3519409448 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1254101940 0x7f1254101dc0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f1248009900 tx=0x7f124802bc30 comp rx=0 tx=0).stop 2024-04-21T23:18:16.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.490+0000 7f12596b5640 1 -- 172.21.15.40:0/3519409448 shutdown_connections 2024-04-21T23:18:16.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.490+0000 7f12596b5640 1 --2- 172.21.15.40:0/3519409448 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1254102300 0x7f1254108b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:16.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.490+0000 7f12596b5640 1 --2- 172.21.15.40:0/3519409448 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1254101940 0x7f1254101dc0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:16.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.490+0000 7f12596b5640 1 --2- 172.21.15.40:0/3519409448 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1254100750 0x7f1254100b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:16.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.490+0000 7f12596b5640 1 -- 172.21.15.40:0/3519409448 >> 172.21.15.40:0/3519409448 conn(0x7f12540fbcc0 msgr2=0x7f12540fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:16.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.490+0000 7f12596b5640 1 -- 172.21.15.40:0/3519409448 shutdown_connections 2024-04-21T23:18:16.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.490+0000 7f12596b5640 1 -- 172.21.15.40:0/3519409448 wait complete. 2024-04-21T23:18:16.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.491+0000 7f12596b5640 1 Processor -- start 2024-04-21T23:18:16.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.491+0000 7f12596b5640 1 -- start start 2024-04-21T23:18:16.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.491+0000 7f12596b5640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1254100750 0x7f1254197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:16.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.491+0000 7f12596b5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1254101940 0x7f1254197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:16.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.491+0000 7f12596b5640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1254102300 0x7f125419efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:16.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.491+0000 7f12596b5640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f125410c340 con 0x7f1254101940 2024-04-21T23:18:16.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.491+0000 7f12596b5640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f125410c1c0 con 0x7f1254100750 2024-04-21T23:18:16.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.491+0000 7f12596b5640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f125410c4c0 con 0x7f1254102300 2024-04-21T23:18:16.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.491+0000 7f12527fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1254101940 0x7f1254197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:16.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.491+0000 7f12527fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1254101940 0x7f1254197f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46272/0 (socket says 172.21.15.40:46272) 2024-04-21T23:18:16.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.491+0000 7f12527fc640 1 -- 172.21.15.40:0/770199545 learned_addr learned my addr 172.21.15.40:0/770199545 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:16.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.491+0000 7f1252ffd640 1 --2- 172.21.15.40:0/770199545 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1254100750 0x7f1254197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:16.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.491+0000 7f12537fe640 1 --2- 172.21.15.40:0/770199545 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1254102300 0x7f125419efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:16.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.492+0000 7f1252ffd640 1 -- 172.21.15.40:0/770199545 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1254102300 msgr2=0x7f125419efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:16.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.492+0000 7f1252ffd640 1 --2- 172.21.15.40:0/770199545 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1254102300 0x7f125419efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:16.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.492+0000 7f1252ffd640 1 -- 172.21.15.40:0/770199545 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1254101940 msgr2=0x7f1254197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:16.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.492+0000 7f1252ffd640 1 --2- 172.21.15.40:0/770199545 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1254101940 0x7f1254197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:16.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.492+0000 7f1252ffd640 1 -- 172.21.15.40:0/770199545 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f125419f610 con 0x7f1254100750 2024-04-21T23:18:16.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.492+0000 7f12527fc640 1 --2- 172.21.15.40:0/770199545 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1254101940 0x7f1254197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:18:16.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.492+0000 7f1252ffd640 1 --2- 172.21.15.40:0/770199545 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1254100750 0x7f1254197a40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f124400e9c0 tx=0x7f124400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:16.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.492+0000 7f123bfff640 1 -- 172.21.15.40:0/770199545 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f124400cd80 con 0x7f1254100750 2024-04-21T23:18:16.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.492+0000 7f123bfff640 1 -- 172.21.15.40:0/770199545 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1244010910 con 0x7f1254100750 2024-04-21T23:18:16.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.492+0000 7f12596b5640 1 -- 172.21.15.40:0/770199545 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f125419f900 con 0x7f1254100750 2024-04-21T23:18:16.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.492+0000 7f123bfff640 1 -- 172.21.15.40:0/770199545 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1244018980 con 0x7f1254100750 2024-04-21T23:18:16.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.492+0000 7f12596b5640 1 -- 172.21.15.40:0/770199545 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f125419fe20 con 0x7f1254100750 2024-04-21T23:18:16.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.493+0000 7f12596b5640 1 -- 172.21.15.40:0/770199545 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12541a0050 con 0x7f1254100750 2024-04-21T23:18:16.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.494+0000 7f123bfff640 1 -- 172.21.15.40:0/770199545 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 19) ==== 137850+0+0 (secure 0 0 0) 0x7f124401f020 con 0x7f1254100750 2024-04-21T23:18:16.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.495+0000 7f123bfff640 1 --2- 172.21.15.40:0/770199545 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f123c0a2c80 0x7f123c0a5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:16.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.495+0000 7f12527fc640 1 -- 172.21.15.40:0/770199545 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f123c0a2c80 msgr2=0x7f123c0a5140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:6800/2500760788 2024-04-21T23:18:16.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.495+0000 7f123bfff640 1 -- 172.21.15.40:0/770199545 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f12440cf630 con 0x7f1254100750 2024-04-21T23:18:16.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.495+0000 7f12527fc640 1 --2- 172.21.15.40:0/770199545 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f123c0a2c80 0x7f123c0a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T23:18:16.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.498+0000 7f123bfff640 1 -- 172.21.15.40:0/770199545 <== mon.2 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f124409c160 con 0x7f1254100750 2024-04-21T23:18:16.682 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:18:16 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:18:16.420+0000 7f2f22aac200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-21T23:18:16.682 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:18:16 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:18:16.533+0000 7f2f22aac200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-21T23:18:16.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.687+0000 7f12596b5640 1 -- 172.21.15.40:0/770199545 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f12541a0220 con 0x7f1254100750 2024-04-21T23:18:16.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.693+0000 7f123bfff640 1 -- 172.21.15.40:0/770199545 <== mon.2 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v28)=0 v28) ==== 133+0+0 (secure 0 0 0) 0x7f124409ba60 con 0x7f1254100750 2024-04-21T23:18:16.693 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Apr 21 23:18:16 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-z[33257]: 2024-04-21T23:18:16.505+0000 7f31e2b84200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-21T23:18:16.694 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:18:16 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-y[32479]: 2024-04-21T23:18:16.498+0000 7f9dc849f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-21T23:18:16.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.695+0000 7f12596b5640 1 -- 172.21.15.40:0/770199545 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f123c0a2c80 msgr2=0x7f123c0a5140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T23:18:16.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.695+0000 7f12596b5640 1 --2- 172.21.15.40:0/770199545 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f123c0a2c80 0x7f123c0a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:16.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.695+0000 7f12596b5640 1 -- 172.21.15.40:0/770199545 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1254100750 msgr2=0x7f1254197a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:16.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.695+0000 7f12596b5640 1 --2- 172.21.15.40:0/770199545 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1254100750 0x7f1254197a40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f124400e9c0 tx=0x7f124400ee80 comp rx=0 tx=0).stop 2024-04-21T23:18:16.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.695+0000 7f12596b5640 1 -- 172.21.15.40:0/770199545 shutdown_connections 2024-04-21T23:18:16.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.695+0000 7f12596b5640 1 --2- 172.21.15.40:0/770199545 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f123c0a2c80 0x7f123c0a5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:16.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.695+0000 7f12596b5640 1 --2- 172.21.15.40:0/770199545 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1254102300 0x7f125419efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:16.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.695+0000 7f12596b5640 1 --2- 172.21.15.40:0/770199545 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1254101940 0x7f1254197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:16.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.695+0000 7f12596b5640 1 --2- 172.21.15.40:0/770199545 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1254100750 0x7f1254197a40 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:16.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.695+0000 7f12596b5640 1 -- 172.21.15.40:0/770199545 >> 172.21.15.40:0/770199545 conn(0x7f12540fbcc0 msgr2=0x7f12540fd690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:16.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.695+0000 7f12596b5640 1 -- 172.21.15.40:0/770199545 shutdown_connections 2024-04-21T23:18:16.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:16.695+0000 7f12596b5640 1 -- 172.21.15.40:0/770199545 wait complete. 2024-04-21T23:18:16.706 INFO:teuthology.run_tasks:Running task kclient... 2024-04-21T23:18:16.719 INFO:tasks.kclient:Mounting kernel clients... 2024-04-21T23:18:16.720 INFO:tasks.kclient:config is {'client.0': {}} 2024-04-21T23:18:16.720 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:18:16.720 DEBUG:teuthology.orchestra.run.smithi040:> ip netns list 2024-04-21T23:18:16.736 INFO:teuthology.orchestra.run.smithi040.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-04-21T23:18:16.736 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:18:16.736 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-21T23:18:16.801 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:18:16.801 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link delete ceph-brx 2024-04-21T23:18:16.922 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=crc', 'nowsync'], 'syntax': 'v1', 'mount_subvol_num': 1} 2024-04-21T23:18:16.922 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-21T23:18:16.922 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-21T23:18:16.922 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph fs ls 2024-04-21T23:18:17.033 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:18:16 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:18:16.862+0000 7f2f22aac200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-21T23:18:17.033 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:18:16 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:18:16.954+0000 7f2f22aac200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-21T23:18:17.186 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:18:17.193 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Apr 21 23:18:16 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-z[33257]: 2024-04-21T23:18:16.816+0000 7f31e2b84200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-21T23:18:17.193 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Apr 21 23:18:16 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-z[33257]: 2024-04-21T23:18:16.902+0000 7f31e2b84200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-21T23:18:17.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:17 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/4042304536' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-21T23:18:17.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:17 smithi174 ceph-mon[31409]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-04-21T23:18:17.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:17 smithi174 ceph-mon[31409]: from='client.? ' entity='client.admin' 2024-04-21T23:18:17.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:17 smithi174 ceph-mon[31409]: from='client.? ' entity='client.admin' 2024-04-21T23:18:17.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:17 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/4042304536' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-21T23:18:17.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:17 smithi111 ceph-mon[31085]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-04-21T23:18:17.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:17 smithi111 ceph-mon[31085]: from='client.? ' entity='client.admin' 2024-04-21T23:18:17.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:17 smithi111 ceph-mon[31085]: from='client.? ' entity='client.admin' 2024-04-21T23:18:17.195 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:18:16 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-y[32479]: 2024-04-21T23:18:16.806+0000 7f9dc849f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-21T23:18:17.195 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:18:16 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-y[32479]: 2024-04-21T23:18:16.890+0000 7f9dc849f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-21T23:18:17.308 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:17 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/4042304536' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-21T23:18:17.309 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:17 smithi040 ceph-mon[27984]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-04-21T23:18:17.309 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:17 smithi040 ceph-mon[27984]: from='client.? ' entity='client.admin' 2024-04-21T23:18:17.309 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:17 smithi040 ceph-mon[27984]: from='client.? ' entity='client.admin' 2024-04-21T23:18:18.393 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:18:18 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-y[32479]: 2024-04-21T23:18:18.057+0000 7f9dc849f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-21T23:18:18.393 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:18:18 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-y[32479]: 2024-04-21T23:18:18.225+0000 7f9dc849f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-21T23:18:18.419 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Apr 21 23:18:18 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-z[33257]: 2024-04-21T23:18:18.079+0000 7f31e2b84200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-21T23:18:18.419 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Apr 21 23:18:18 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-z[33257]: 2024-04-21T23:18:18.249+0000 7f31e2b84200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-21T23:18:18.419 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Apr 21 23:18:18 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-z[33257]: 2024-04-21T23:18:18.420+0000 7f31e2b84200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-21T23:18:18.432 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:18:18 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:18:18.229+0000 7f2f22aac200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-21T23:18:18.432 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:18:18 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:18:18.406+0000 7f2f22aac200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-21T23:18:18.677 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:18:18 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-y[32479]: 2024-04-21T23:18:18.393+0000 7f9dc849f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-21T23:18:18.792 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:18:18 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:18:18.587+0000 7f2f22aac200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-21T23:18:18.944 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:18:18 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-y[32479]: 2024-04-21T23:18:18.678+0000 7f9dc849f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-21T23:18:18.944 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:18:18 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-y[32479]: 2024-04-21T23:18:18.770+0000 7f9dc849f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-21T23:18:19.004 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Apr 21 23:18:18 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-z[33257]: 2024-04-21T23:18:18.708+0000 7f31e2b84200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-21T23:18:19.005 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Apr 21 23:18:18 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-z[33257]: 2024-04-21T23:18:18.800+0000 7f31e2b84200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-21T23:18:19.005 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Apr 21 23:18:19 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-z[33257]: 2024-04-21T23:18:19.005+0000 7f31e2b84200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-21T23:18:19.073 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:18:18 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:18:18.893+0000 7f2f22aac200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-21T23:18:19.073 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:18:18 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:18:18.991+0000 7f2f22aac200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-21T23:18:19.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:19.074+0000 7f7e10abd700 1 -- 172.21.15.40:0/1323058337 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e0c06a7d0 msgr2=0x7f7e0c06ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:19.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:19.074+0000 7f7e10abd700 1 --2- 172.21.15.40:0/1323058337 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e0c06a7d0 0x7f7e0c06ac30 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f7df4009990 tx=0x7f7df4009ca0 comp rx=0 tx=0).stop 2024-04-21T23:18:19.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:19.074+0000 7f7e10abd700 1 -- 172.21.15.40:0/1323058337 shutdown_connections 2024-04-21T23:18:19.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:19.074+0000 7f7e10abd700 1 --2- 172.21.15.40:0/1323058337 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7e0c06b170 0x7f7e0c10f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:19.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:19.074+0000 7f7e10abd700 1 --2- 172.21.15.40:0/1323058337 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7e0c1082a0 0x7f7e0c108680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:19.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:19.074+0000 7f7e10abd700 1 --2- 172.21.15.40:0/1323058337 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e0c06a7d0 0x7f7e0c06ac30 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:19.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:19.074+0000 7f7e10abd700 1 -- 172.21.15.40:0/1323058337 >> 172.21.15.40:0/1323058337 conn(0x7f7e0c0780a0 msgr2=0x7f7e0c0784b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:19.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:19.074+0000 7f7e10abd700 1 -- 172.21.15.40:0/1323058337 shutdown_connections 2024-04-21T23:18:19.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:19.074+0000 7f7e10abd700 1 -- 172.21.15.40:0/1323058337 wait complete. 2024-04-21T23:18:19.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:19.075+0000 7f7e10abd700 1 Processor -- start 2024-04-21T23:18:19.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:19.075+0000 7f7e10abd700 1 -- start start 2024-04-21T23:18:19.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:19.075+0000 7f7e10abd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e0c06a7d0 0x7f7e0c1a0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:19.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:19.075+0000 7f7e10abd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7e0c06b170 0x7f7e0c1a0d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:19.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:19.075+0000 7f7e10abd700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7e0c1082a0 0x7f7e0c19a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:19.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:19.075+0000 7f7e10abd700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7e0c112500 con 0x7f7e0c06a7d0 2024-04-21T23:18:19.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:19.075+0000 7f7e10abd700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f7e0c112380 con 0x7f7e0c06b170 2024-04-21T23:18:19.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:19.075+0000 7f7e10abd700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7e0c112680 con 0x7f7e0c1082a0 2024-04-21T23:18:19.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:19.075+0000 7f7e0a59c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e0c06a7d0 0x7f7e0c1a0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:19.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:19.075+0000 7f7e0a59c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e0c06a7d0 0x7f7e0c1a0810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46294/0 (socket says 172.21.15.40:46294) 2024-04-21T23:18:19.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:19.075+0000 7f7e0a59c700 1 -- 172.21.15.40:0/3222297511 learned_addr learned my addr 172.21.15.40:0/3222297511 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:19.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:19.075+0000 7f7e09d9b700 1 --2- 172.21.15.40:0/3222297511 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7e0c06b170 0x7f7e0c1a0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:19.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:19.075+0000 7f7e0ad9d700 1 --2- 172.21.15.40:0/3222297511 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7e0c1082a0 0x7f7e0c19a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:19.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:19.076+0000 7f7e0a59c700 1 -- 172.21.15.40:0/3222297511 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7e0c1082a0 msgr2=0x7f7e0c19a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:19.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:19.076+0000 7f7e0a59c700 1 --2- 172.21.15.40:0/3222297511 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7e0c1082a0 0x7f7e0c19a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:19.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:19.076+0000 7f7e0a59c700 1 -- 172.21.15.40:0/3222297511 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7e0c06b170 msgr2=0x7f7e0c1a0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:19.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:19.076+0000 7f7e0a59c700 1 --2- 172.21.15.40:0/3222297511 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7e0c06b170 0x7f7e0c1a0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:19.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:19.076+0000 7f7e0a59c700 1 -- 172.21.15.40:0/3222297511 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e0c19b130 con 0x7f7e0c06a7d0 2024-04-21T23:18:19.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:19.076+0000 7f7e0a59c700 1 --2- 172.21.15.40:0/3222297511 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e0c06a7d0 0x7f7e0c1a0810 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f7dfc00e9a0 tx=0x7f7dfc00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:19.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:19.076+0000 7f7e037fe700 1 -- 172.21.15.40:0/3222297511 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dfc00ca10 con 0x7f7e0c06a7d0 2024-04-21T23:18:19.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:19.076+0000 7f7e10abd700 1 -- 172.21.15.40:0/3222297511 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e0c19b420 con 0x7f7e0c06a7d0 2024-04-21T23:18:19.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:19.076+0000 7f7e037fe700 1 -- 172.21.15.40:0/3222297511 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7dfc00cbb0 con 0x7f7e0c06a7d0 2024-04-21T23:18:19.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:19.076+0000 7f7e10abd700 1 -- 172.21.15.40:0/3222297511 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e0c1a76e0 con 0x7f7e0c06a7d0 2024-04-21T23:18:19.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:19.076+0000 7f7e037fe700 1 -- 172.21.15.40:0/3222297511 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dfc018680 con 0x7f7e0c06a7d0 2024-04-21T23:18:19.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:19.077+0000 7f7e10abd700 1 -- 172.21.15.40:0/3222297511 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e0c103950 con 0x7f7e0c06a7d0 2024-04-21T23:18:19.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:19.078+0000 7f7e037fe700 1 -- 172.21.15.40:0/3222297511 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 19) ==== 137850+0+0 (secure 0 0 0) 0x7f7dfc010430 con 0x7f7e0c06a7d0 2024-04-21T23:18:19.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:19.079+0000 7f7e037fe700 1 --2- 172.21.15.40:0/3222297511 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f7df80a2c70 0x7f7df80a5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:19.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:19.079+0000 7f7e09d9b700 1 -- 172.21.15.40:0/3222297511 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f7df80a2c70 msgr2=0x7f7df80a5130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:6800/2500760788 2024-04-21T23:18:19.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:19.079+0000 7f7e037fe700 1 -- 172.21.15.40:0/3222297511 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f7dfc0cf660 con 0x7f7e0c06a7d0 2024-04-21T23:18:19.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:19.079+0000 7f7e09d9b700 1 --2- 172.21.15.40:0/3222297511 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f7df80a2c70 0x7f7df80a5130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T23:18:19.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:19.084+0000 7f7e037fe700 1 -- 172.21.15.40:0/3222297511 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f7dfc09c1f0 con 0x7f7e0c06a7d0 2024-04-21T23:18:19.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:19.280+0000 7f7e09d9b700 1 -- 172.21.15.40:0/3222297511 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f7df80a2c70 msgr2=0x7f7df80a5130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:6800/2500760788 2024-04-21T23:18:19.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:19.280+0000 7f7e09d9b700 1 --2- 172.21.15.40:0/3222297511 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f7df80a2c70 0x7f7df80a5130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-21T23:18:19.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:19.354+0000 7f7e10abd700 1 -- 172.21.15.40:0/3222297511 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f7e0c19bf00 con 0x7f7e0c06a7d0 2024-04-21T23:18:19.353 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:18:19 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:18:19.220+0000 7f2f22aac200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-21T23:18:19.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:19.354+0000 7f7e037fe700 1 -- 172.21.15.40:0/3222297511 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v24) ==== 53+0+88 (secure 0 0 0) 0x7f7dfc09baf0 con 0x7f7e0c06a7d0 2024-04-21T23:18:19.354 INFO:teuthology.orchestra.run.smithi040.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-21T23:18:19.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:19.356+0000 7f7e10abd700 1 -- 172.21.15.40:0/3222297511 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f7df80a2c70 msgr2=0x7f7df80a5130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T23:18:19.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:19.356+0000 7f7e10abd700 1 --2- 172.21.15.40:0/3222297511 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f7df80a2c70 0x7f7df80a5130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:19.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:19.356+0000 7f7e10abd700 1 -- 172.21.15.40:0/3222297511 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e0c06a7d0 msgr2=0x7f7e0c1a0810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:19.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:19.356+0000 7f7e10abd700 1 --2- 172.21.15.40:0/3222297511 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e0c06a7d0 0x7f7e0c1a0810 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f7dfc00e9a0 tx=0x7f7dfc00ecb0 comp rx=0 tx=0).stop 2024-04-21T23:18:19.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:19.356+0000 7f7e10abd700 1 -- 172.21.15.40:0/3222297511 shutdown_connections 2024-04-21T23:18:19.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:19.356+0000 7f7e10abd700 1 --2- 172.21.15.40:0/3222297511 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7e0c1082a0 0x7f7e0c19a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:19.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:19.356+0000 7f7e10abd700 1 --2- 172.21.15.40:0/3222297511 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f7df80a2c70 0x7f7df80a5130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:19.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:19.356+0000 7f7e10abd700 1 --2- 172.21.15.40:0/3222297511 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e0c06a7d0 0x7f7e0c1a0810 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:19.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:19.356+0000 7f7e10abd700 1 --2- 172.21.15.40:0/3222297511 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7e0c06b170 0x7f7e0c1a0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:19.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:19.356+0000 7f7e10abd700 1 -- 172.21.15.40:0/3222297511 >> 172.21.15.40:0/3222297511 conn(0x7f7e0c0780a0 msgr2=0x7f7e0c10d4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:19.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:19.357+0000 7f7e10abd700 1 -- 172.21.15.40:0/3222297511 shutdown_connections 2024-04-21T23:18:19.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:19.357+0000 7f7e10abd700 1 -- 172.21.15.40:0/3222297511 wait complete. 2024-04-21T23:18:19.405 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Apr 21 23:18:19 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-z[33257]: 2024-04-21T23:18:19.198+0000 7f31e2b84200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-21T23:18:19.405 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:18:18 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-y[32479]: 2024-04-21T23:18:18.972+0000 7f9dc849f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-21T23:18:19.406 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:18:19 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-y[32479]: 2024-04-21T23:18:19.164+0000 7f9dc849f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-21T23:18:19.681 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:19 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/3222297511' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-21T23:18:19.682 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:18:19 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:18:19.428+0000 7f2f22aac200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-21T23:18:19.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:19 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/3222297511' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-21T23:18:19.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:19 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/3222297511' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-21T23:18:20.063 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-21T23:18:20.063 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-21T23:18:20.063 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi040.front.sepia.ceph.com 2024-04-21T23:18:20.063 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-21T23:18:20.063 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-21T23:18:20.063 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-21T23:18:20.063 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-21T23:18:20.063 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-21T23:18:20.064 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-21T23:18:20.064 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:18:20.064 DEBUG:teuthology.orchestra.run.smithi040:> ip addr 2024-04-21T23:18:20.080 INFO:teuthology.orchestra.run.smithi040.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-21T23:18:20.080 INFO:teuthology.orchestra.run.smithi040.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-21T23:18:20.080 INFO:teuthology.orchestra.run.smithi040.stdout: inet 127.0.0.1/8 scope host lo 2024-04-21T23:18:20.081 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-04-21T23:18:20.081 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 ::1/128 scope host 2024-04-21T23:18:20.081 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-04-21T23:18:20.081 INFO:teuthology.orchestra.run.smithi040.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-21T23:18:20.081 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:6b:fd:16 brd ff:ff:ff:ff:ff:ff 2024-04-21T23:18:20.081 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp7s0f0 2024-04-21T23:18:20.081 INFO:teuthology.orchestra.run.smithi040.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-21T23:18:20.081 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:6b:fd:17 brd ff:ff:ff:ff:ff:ff 2024-04-21T23:18:20.081 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp7s0f1 2024-04-21T23:18:20.081 INFO:teuthology.orchestra.run.smithi040.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-21T23:18:20.081 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:bd:12:cc brd ff:ff:ff:ff:ff:ff 2024-04-21T23:18:20.081 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp1s0f0 2024-04-21T23:18:20.081 INFO:teuthology.orchestra.run.smithi040.stdout: inet 172.21.15.40/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-21T23:18:20.081 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft 42071sec preferred_lft 42071sec 2024-04-21T23:18:20.081 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 fe80::ec4:7aff:febd:12cc/64 scope link noprefixroute 2024-04-21T23:18:20.082 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-04-21T23:18:20.082 INFO:teuthology.orchestra.run.smithi040.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-21T23:18:20.082 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:bd:12:cd brd ff:ff:ff:ff:ff:ff 2024-04-21T23:18:20.082 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp1s0f1 2024-04-21T23:18:20.082 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-21T23:18:20.082 DEBUG:teuthology.orchestra.run.smithi040:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-21T23:18:20.082 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2024-04-21T23:18:20.082 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link add name ceph-brx type bridge 2024-04-21T23:18:20.082 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip addr flush dev ceph-brx 2024-04-21T23:18:20.082 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link set ceph-brx up 2024-04-21T23:18:20.083 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-21T23:18:20.083 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-04-21T23:18:20.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:20 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-21T23:18:20.172 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:18:19 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-y[32479]: 2024-04-21T23:18:19.758+0000 7f9dc849f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-21T23:18:20.193 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Apr 21 23:18:19 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-z[33257]: 2024-04-21T23:18:19.799+0000 7f31e2b84200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-21T23:18:20.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T23:18:20.224 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:18:20.225 DEBUG:teuthology.orchestra.run.smithi040:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-21T23:18:20.249 INFO:teuthology.orchestra.run.smithi040.stdout:1 2024-04-21T23:18:20.251 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:18:20.251 DEBUG:teuthology.orchestra.run.smithi040:> route 2024-04-21T23:18:20.307 INFO:teuthology.orchestra.run.smithi040.stdout:Kernel IP routing table 2024-04-21T23:18:20.307 INFO:teuthology.orchestra.run.smithi040.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-21T23:18:20.308 INFO:teuthology.orchestra.run.smithi040.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-21T23:18:20.308 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-21T23:18:20.308 INFO:teuthology.orchestra.run.smithi040.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-21T23:18:20.308 DEBUG:teuthology.orchestra.run.smithi040:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-21T23:18:20.308 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2024-04-21T23:18:20.308 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-21T23:18:20.308 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-21T23:18:20.308 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-21T23:18:20.308 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-04-21T23:18:20.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:20 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-21T23:18:20.406 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:18:20 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:18:20.067+0000 7f2f22aac200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-21T23:18:20.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T23:18:20.439 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:18:20.439 DEBUG:teuthology.orchestra.run.smithi040:> ip netns list 2024-04-21T23:18:20.444 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:18:20 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-y[32479]: 2024-04-21T23:18:20.173+0000 7f9dc849f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-21T23:18:20.444 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:18:20 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-y[32479]: 2024-04-21T23:18:20.270+0000 7f9dc849f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-21T23:18:20.454 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:18:20.455 DEBUG:teuthology.orchestra.run.smithi040:> ip netns list-id 2024-04-21T23:18:20.511 DEBUG:teuthology.orchestra.run.smithi040:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-21T23:18:20.512 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2024-04-21T23:18:20.512 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-21T23:18:20.512 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-21T23:18:20.512 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-04-21T23:18:20.525 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Apr 21 23:18:20 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-z[33257]: 2024-04-21T23:18:20.223+0000 7f31e2b84200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-21T23:18:20.525 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Apr 21 23:18:20 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-z[33257]: 2024-04-21T23:18:20.319+0000 7f31e2b84200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-21T23:18:20.525 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Apr 21 23:18:20 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-z[33257]: 2024-04-21T23:18:20.525+0000 7f31e2b84200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-21T23:18:20.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:20 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-21T23:18:20.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T23:18:20.632 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-21T23:18:20.632 DEBUG:teuthology.orchestra.run.smithi040:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-21T23:18:20.632 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2024-04-21T23:18:20.632 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-21T23:18:20.632 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-21T23:18:20.633 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-21T23:18:20.633 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-21T23:18:20.633 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-21T23:18:20.633 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-04-21T23:18:20.681 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:18:20 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:18:20.535+0000 7f2f22aac200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-21T23:18:20.682 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:18:20 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:18:20.645+0000 7f2f22aac200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-21T23:18:20.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:20 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-21T23:18:20.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T23:18:20.748 DEBUG:teuthology.orchestra.run.smithi040:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-21T23:18:20.748 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2024-04-21T23:18:20.748 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link set brx.0 up 2024-04-21T23:18:20.748 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link set dev brx.0 master ceph-brx 2024-04-21T23:18:20.748 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-04-21T23:18:20.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:20 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-21T23:18:20.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T23:18:20.829 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-21T23:18:20.829 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-21T23:18:20.845 INFO:teuthology.orchestra.run.smithi040.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-21T23:18:20.845 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-21T23:18:20.845 DEBUG:teuthology.orchestra.run.smithi040:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-21T23:18:20.904 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-04-21T23:18:20.905 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-04-21T23:18:20.905 INFO:tasks.cephfs.kernel_mount:mounting using device: :/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66 2024-04-21T23:18:20.905 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:18:20.905 DEBUG:teuthology.orchestra.run.smithi040:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,nowsync 2024-04-21T23:18:20.944 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:18:20 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-y[32479]: 2024-04-21T23:18:20.472+0000 7f9dc849f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-21T23:18:20.967 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:18:20 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:18:20.870+0000 7f2f22aac200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-21T23:18:20.989 INFO:teuthology.orchestra.run.smithi040.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,nowsync 2024-04-21T23:18:20.989 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,nowsync". 2024-04-21T23:18:20.989 INFO:teuthology.orchestra.run.smithi040.stdout:invalid new device string format 2024-04-21T23:18:21.114 INFO:teuthology.orchestra.run.smithi040.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,nowsync 2024-04-21T23:18:21.114 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,nowsync". 2024-04-21T23:18:21.114 INFO:teuthology.orchestra.run.smithi040.stdout:invalid new device string format 2024-04-21T23:18:21.114 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: resolved to: "172.21.15.40:3300,172.21.15.111:3300,172.21.15.174:3300" 2024-04-21T23:18:21.114 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.40:3300,172.21.15.111:3300,172.21.15.174:3300:/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66 2024-04-21T23:18:21.114 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,nowsync,key=0,fsid=d814e4dc-0033-11ef-bc93-c7b262605968" will pass to kernel 2024-04-21T23:18:21.116 INFO:teuthology.orchestra.run.smithi040.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-21T23:18:21.116 INFO:teuthology.orchestra.run.smithi040.stdout: You just mounted a file system that supports labels which does not 2024-04-21T23:18:21.116 INFO:teuthology.orchestra.run.smithi040.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-21T23:18:21.116 INFO:teuthology.orchestra.run.smithi040.stdout: applications will generate AVC messages and not be allowed access to 2024-04-21T23:18:21.116 INFO:teuthology.orchestra.run.smithi040.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-21T23:18:21.117 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-21T23:18:21.117 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:18:21.118 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-21T23:18:21.187 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-21T23:18:21.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:21 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-21T23:18:21.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T23:18:21.285 INFO:teuthology.orchestra.run.smithi040.stdout:client25132 2024-04-21T23:18:21.285 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-21T23:18:21.285 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-21T23:18:21.302 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-21T23:18:21.321 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:18:20 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-y[32479]: 2024-04-21T23:18:20.989+0000 7f9dc849f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-21T23:18:21.321 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:18:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-y[32479]: 2024-04-21T23:18:21.080+0000 7f9dc849f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-21T23:18:21.386 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Apr 21 23:18:21 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-z[33257]: 2024-04-21T23:18:21.051+0000 7f31e2b84200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-21T23:18:21.386 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Apr 21 23:18:21 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-z[33257]: 2024-04-21T23:18:21.144+0000 7f31e2b84200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-21T23:18:21.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:21 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-21T23:18:21.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T23:18:21.400 INFO:teuthology.orchestra.run.smithi040.stdout:client25132 2024-04-21T23:18:21.401 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-21T23:18:21.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:21 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-21T23:18:21.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T23:18:21.459 INFO:teuthology.orchestra.run.smithi040.stdout:client25132 2024-04-21T23:18:21.459 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/d814e4dc-0033-11ef-bc93-c7b262605968.client25132/status') 2024-04-21T23:18:21.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:21 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-21T23:18:21.519 INFO:teuthology.orchestra.run.smithi040.stdout:instance: client.25132 (3)172.21.15.40:0/46935991 2024-04-21T23:18:21.519 INFO:teuthology.orchestra.run.smithi040.stdout:blocklisted: false 2024-04-21T23:18:21.519 INFO:teuthology.orchestra.run.smithi040.stderr:0+1 records in 2024-04-21T23:18:21.519 INFO:teuthology.orchestra.run.smithi040.stderr:0+1 records out 2024-04-21T23:18:21.519 INFO:teuthology.orchestra.run.smithi040.stderr:69 bytes copied, 2.8158e-05 s, 2.5 MB/s 2024-04-21T23:18:21.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T23:18:21.527 INFO:teuthology.run_tasks:Running task exec... 2024-04-21T23:18:21.537 INFO:teuthology.task.exec:Executing custom commands... 2024-04-21T23:18:21.537 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi040.front.sepia.ceph.com 2024-04-21T23:18:21.538 DEBUG:teuthology.orchestra.run.smithi040:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate false' 2024-04-21T23:18:21.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:21.642+0000 7ff92f1be640 1 -- 172.21.15.40:0/1503866879 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff928102250 msgr2=0x7ff928108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:21.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:21.642+0000 7ff92f1be640 1 --2- 172.21.15.40:0/1503866879 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff928102250 0x7ff928108ae0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7ff920009ca0 tx=0x7ff92002c0f0 comp rx=0 tx=0).stop 2024-04-21T23:18:21.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:21.642+0000 7ff92f1be640 1 -- 172.21.15.40:0/1503866879 shutdown_connections 2024-04-21T23:18:21.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:21.642+0000 7ff92f1be640 1 --2- 172.21.15.40:0/1503866879 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff928101890 0x7ff928101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:21.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:21.642+0000 7ff92f1be640 1 --2- 172.21.15.40:0/1503866879 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff928102250 0x7ff928108ae0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:21.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:21.642+0000 7ff92f1be640 1 --2- 172.21.15.40:0/1503866879 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff9281006a0 0x7ff928100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:21.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:21.642+0000 7ff92f1be640 1 -- 172.21.15.40:0/1503866879 >> 172.21.15.40:0/1503866879 conn(0x7ff9280fbc10 msgr2=0x7ff9280fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:21.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:21.642+0000 7ff92f1be640 1 -- 172.21.15.40:0/1503866879 shutdown_connections 2024-04-21T23:18:21.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:21.642+0000 7ff92f1be640 1 -- 172.21.15.40:0/1503866879 wait complete. 2024-04-21T23:18:21.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:21.643+0000 7ff92f1be640 1 Processor -- start 2024-04-21T23:18:21.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:21.643+0000 7ff92f1be640 1 -- start start 2024-04-21T23:18:21.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:21.643+0000 7ff92f1be640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9281006a0 0x7ff928104630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:21.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:21.643+0000 7ff92f1be640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff928101890 0x7ff928102c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:21.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:21.643+0000 7ff92f1be640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff928102250 0x7ff9281031b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:21.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:21.643+0000 7ff92f1be640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff92810c320 con 0x7ff9281006a0 2024-04-21T23:18:21.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:21.643+0000 7ff92f1be640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff92810c1a0 con 0x7ff928101890 2024-04-21T23:18:21.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:21.643+0000 7ff92f1be640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff92810c4a0 con 0x7ff928102250 2024-04-21T23:18:21.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:21.643+0000 7ff92cf5a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9281006a0 0x7ff928104630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:21.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:21.643+0000 7ff927fff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff928101890 0x7ff928102c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:21.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:21.643+0000 7ff92cf5a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9281006a0 0x7ff928104630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46302/0 (socket says 172.21.15.40:46302) 2024-04-21T23:18:21.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:21.643+0000 7ff92cf5a640 1 -- 172.21.15.40:0/191514372 learned_addr learned my addr 172.21.15.40:0/191514372 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:21.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:21.643+0000 7ff92d75b640 1 --2- 172.21.15.40:0/191514372 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff928102250 0x7ff9281031b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:21.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:21.644+0000 7ff927fff640 1 -- 172.21.15.40:0/191514372 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff928102250 msgr2=0x7ff9281031b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:21.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:21.644+0000 7ff927fff640 1 --2- 172.21.15.40:0/191514372 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff928102250 0x7ff9281031b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:21.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:21.644+0000 7ff927fff640 1 -- 172.21.15.40:0/191514372 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9281006a0 msgr2=0x7ff928104630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:21.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:21.644+0000 7ff927fff640 1 --2- 172.21.15.40:0/191514372 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9281006a0 0x7ff928104630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:21.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:21.644+0000 7ff927fff640 1 -- 172.21.15.40:0/191514372 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff928103a40 con 0x7ff928101890 2024-04-21T23:18:21.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:21.644+0000 7ff92cf5a640 1 --2- 172.21.15.40:0/191514372 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9281006a0 0x7ff928104630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:18:21.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:21.644+0000 7ff927fff640 1 --2- 172.21.15.40:0/191514372 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff928101890 0x7ff928102c70 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff91400e9f0 tx=0x7ff91400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:21.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:21.644+0000 7ff925ffb640 1 -- 172.21.15.40:0/191514372 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff91400ccc0 con 0x7ff928101890 2024-04-21T23:18:21.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:21.644+0000 7ff92f1be640 1 -- 172.21.15.40:0/191514372 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9281a1ae0 con 0x7ff928101890 2024-04-21T23:18:21.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:21.644+0000 7ff925ffb640 1 -- 172.21.15.40:0/191514372 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff914010910 con 0x7ff928101890 2024-04-21T23:18:21.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:21.644+0000 7ff925ffb640 1 -- 172.21.15.40:0/191514372 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff914018960 con 0x7ff928101890 2024-04-21T23:18:21.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:21.644+0000 7ff92f1be640 1 -- 172.21.15.40:0/191514372 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9281a2080 con 0x7ff928101890 2024-04-21T23:18:21.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:21.645+0000 7ff92f1be640 1 -- 172.21.15.40:0/191514372 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9281a2250 con 0x7ff928101890 2024-04-21T23:18:21.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:21.646+0000 7ff925ffb640 1 -- 172.21.15.40:0/191514372 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 19) ==== 137850+0+0 (secure 0 0 0) 0x7ff914010ac0 con 0x7ff928101890 2024-04-21T23:18:21.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:21.647+0000 7ff925ffb640 1 --2- 172.21.15.40:0/191514372 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7ff9180a2c80 0x7ff9180a5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:21.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:21.647+0000 7ff92cf5a640 1 -- 172.21.15.40:0/191514372 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7ff9180a2c80 msgr2=0x7ff9180a5140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:6800/2500760788 2024-04-21T23:18:21.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:21.647+0000 7ff925ffb640 1 -- 172.21.15.40:0/191514372 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7ff9140cfbd0 con 0x7ff928101890 2024-04-21T23:18:21.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:21.647+0000 7ff92cf5a640 1 --2- 172.21.15.40:0/191514372 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7ff9180a2c80 0x7ff9180a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T23:18:21.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:21.650+0000 7ff925ffb640 1 -- 172.21.15.40:0/191514372 <== mon.2 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7ff91409c0b0 con 0x7ff928101890 2024-04-21T23:18:21.663 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:18:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-y[32479]: 2024-04-21T23:18:21.322+0000 7f9dc849f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-21T23:18:21.663 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:18:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-y[32479]: 2024-04-21T23:18:21.453+0000 7f9dc849f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-21T23:18:21.663 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:18:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-y[32479]: 2024-04-21T23:18:21.549+0000 7f9dc849f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-21T23:18:21.663 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:18:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-y[32479]: 2024-04-21T23:18:21.664+0000 7f9dc849f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-21T23:18:21.693 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Apr 21 23:18:21 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-z[33257]: 2024-04-21T23:18:21.386+0000 7f31e2b84200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-21T23:18:21.694 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Apr 21 23:18:21 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-z[33257]: 2024-04-21T23:18:21.523+0000 7f31e2b84200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-21T23:18:21.694 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Apr 21 23:18:21 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-z[33257]: 2024-04-21T23:18:21.619+0000 7f31e2b84200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-21T23:18:21.819 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:18:21 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:18:21.456+0000 7f2f22aac200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-21T23:18:21.820 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:18:21 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:18:21.557+0000 7f2f22aac200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-21T23:18:21.820 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:18:21 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:18:21.820+0000 7f2f22aac200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-21T23:18:21.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:21.848+0000 7ff92cf5a640 1 -- 172.21.15.40:0/191514372 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7ff9180a2c80 msgr2=0x7ff9180a5140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:6800/2500760788 2024-04-21T23:18:21.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:21.848+0000 7ff92cf5a640 1 --2- 172.21.15.40:0/191514372 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7ff9180a2c80 0x7ff9180a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-21T23:18:21.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:21.877+0000 7ff92f1be640 1 -- 172.21.15.40:0/191514372 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} v 0) -- 0x7ff9281a2520 con 0x7ff928101890 2024-04-21T23:18:21.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:21.932+0000 7ff925ffb640 1 -- 172.21.15.40:0/191514372 <== mon.2 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]=0 v25) ==== 118+0+0 (secure 0 0 0) 0x7ff9140a03b0 con 0x7ff928101890 2024-04-21T23:18:21.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:21.935+0000 7ff92f1be640 1 -- 172.21.15.40:0/191514372 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7ff9180a2c80 msgr2=0x7ff9180a5140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T23:18:21.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:21.935+0000 7ff92f1be640 1 --2- 172.21.15.40:0/191514372 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7ff9180a2c80 0x7ff9180a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:21.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:21.935+0000 7ff92f1be640 1 -- 172.21.15.40:0/191514372 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff928101890 msgr2=0x7ff928102c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:21.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:21.935+0000 7ff92f1be640 1 --2- 172.21.15.40:0/191514372 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff928101890 0x7ff928102c70 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff91400e9f0 tx=0x7ff91400eeb0 comp rx=0 tx=0).stop 2024-04-21T23:18:21.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:21.935+0000 7ff92f1be640 1 -- 172.21.15.40:0/191514372 shutdown_connections 2024-04-21T23:18:21.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:21.935+0000 7ff92f1be640 1 --2- 172.21.15.40:0/191514372 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7ff9180a2c80 0x7ff9180a5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:21.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:21.935+0000 7ff92f1be640 1 --2- 172.21.15.40:0/191514372 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff928101890 0x7ff928102c70 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:21.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:21.935+0000 7ff92f1be640 1 --2- 172.21.15.40:0/191514372 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff928102250 0x7ff9281031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:21.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:21.935+0000 7ff92f1be640 1 --2- 172.21.15.40:0/191514372 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9281006a0 0x7ff928104630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:21.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:21.935+0000 7ff92f1be640 1 -- 172.21.15.40:0/191514372 >> 172.21.15.40:0/191514372 conn(0x7ff9280fbc10 msgr2=0x7ff9280fd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:21.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:21.935+0000 7ff92f1be640 1 -- 172.21.15.40:0/191514372 shutdown_connections 2024-04-21T23:18:21.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:21.935+0000 7ff92f1be640 1 -- 172.21.15.40:0/191514372 wait complete. 2024-04-21T23:18:21.935 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:18:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-y[32479]: 2024-04-21T23:18:21.757+0000 7f9dc849f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-21T23:18:21.947 DEBUG:teuthology.orchestra.run.smithi040:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10' 2024-04-21T23:18:22.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:22.054+0000 7f75af627640 1 -- 172.21.15.40:0/1038931495 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75a81006a0 msgr2=0x7f75a8100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:22.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:22.054+0000 7f75af627640 1 --2- 172.21.15.40:0/1038931495 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75a81006a0 0x7f75a8100ac0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f7598009900 tx=0x7f759802bc30 comp rx=0 tx=0).stop 2024-04-21T23:18:22.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:22.054+0000 7f75af627640 1 -- 172.21.15.40:0/1038931495 shutdown_connections 2024-04-21T23:18:22.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:22.054+0000 7f75af627640 1 --2- 172.21.15.40:0/1038931495 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75a8102250 0x7f75a8108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:22.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:22.054+0000 7f75af627640 1 --2- 172.21.15.40:0/1038931495 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f75a8101890 0x7f75a8101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:22.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:22.054+0000 7f75af627640 1 --2- 172.21.15.40:0/1038931495 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75a81006a0 0x7f75a8100ac0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:22.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:22.054+0000 7f75af627640 1 -- 172.21.15.40:0/1038931495 >> 172.21.15.40:0/1038931495 conn(0x7f75a80fbc10 msgr2=0x7f75a80fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:22.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:22.054+0000 7f75af627640 1 -- 172.21.15.40:0/1038931495 shutdown_connections 2024-04-21T23:18:22.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:22.054+0000 7f75af627640 1 -- 172.21.15.40:0/1038931495 wait complete. 2024-04-21T23:18:22.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:22.055+0000 7f75af627640 1 Processor -- start 2024-04-21T23:18:22.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:22.055+0000 7f75af627640 1 -- start start 2024-04-21T23:18:22.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:22.055+0000 7f75af627640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f75a81006a0 0x7f75a80699c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:22.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:22.055+0000 7f75af627640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75a8101890 0x7f75a819c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:22.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:22.055+0000 7f75af627640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75a8102250 0x7f75a819eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:22.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:22.055+0000 7f75af627640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f75a810c320 con 0x7f75a8102250 2024-04-21T23:18:22.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:22.055+0000 7f75af627640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f75a810c1a0 con 0x7f75a8101890 2024-04-21T23:18:22.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:22.055+0000 7f75af627640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f75a810c4a0 con 0x7f75a81006a0 2024-04-21T23:18:22.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:22.055+0000 7f75adbc4640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75a8102250 0x7f75a819eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:22.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:22.055+0000 7f75adbc4640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75a8102250 0x7f75a819eda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60646/0 (socket says 172.21.15.40:60646) 2024-04-21T23:18:22.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:22.055+0000 7f75adbc4640 1 -- 172.21.15.40:0/3187747537 learned_addr learned my addr 172.21.15.40:0/3187747537 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:22.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:22.055+0000 7f75ad3c3640 1 --2- 172.21.15.40:0/3187747537 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f75a81006a0 0x7f75a80699c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:22.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:22.055+0000 7f75acbc2640 1 --2- 172.21.15.40:0/3187747537 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75a8101890 0x7f75a819c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:22.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:22.056+0000 7f75adbc4640 1 -- 172.21.15.40:0/3187747537 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f75a81006a0 msgr2=0x7f75a80699c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:22.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:22.056+0000 7f75adbc4640 1 --2- 172.21.15.40:0/3187747537 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f75a81006a0 0x7f75a80699c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:22.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:22.056+0000 7f75adbc4640 1 -- 172.21.15.40:0/3187747537 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75a8101890 msgr2=0x7f75a819c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:22.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:22.056+0000 7f75adbc4640 1 --2- 172.21.15.40:0/3187747537 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75a8101890 0x7f75a819c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:22.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:22.056+0000 7f75adbc4640 1 -- 172.21.15.40:0/3187747537 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75a819f660 con 0x7f75a8102250 2024-04-21T23:18:22.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:22.056+0000 7f75adbc4640 1 --2- 172.21.15.40:0/3187747537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75a8102250 0x7f75a819eda0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f75a000eb10 tx=0x7f75a000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:22.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:22.056+0000 7f759e7fc640 1 -- 172.21.15.40:0/3187747537 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a0004b20 con 0x7f75a8102250 2024-04-21T23:18:22.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:22.056+0000 7f759e7fc640 1 -- 172.21.15.40:0/3187747537 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75a0004cc0 con 0x7f75a8102250 2024-04-21T23:18:22.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:22.056+0000 7f759e7fc640 1 -- 172.21.15.40:0/3187747537 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a0010760 con 0x7f75a8102250 2024-04-21T23:18:22.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:22.056+0000 7f75af627640 1 -- 172.21.15.40:0/3187747537 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75a819f950 con 0x7f75a8102250 2024-04-21T23:18:22.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:22.056+0000 7f75af627640 1 -- 172.21.15.40:0/3187747537 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75a819fea0 con 0x7f75a8102250 2024-04-21T23:18:22.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:22.057+0000 7f75af627640 1 -- 172.21.15.40:0/3187747537 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75a81079c0 con 0x7f75a8102250 2024-04-21T23:18:22.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:22.058+0000 7f759e7fc640 1 -- 172.21.15.40:0/3187747537 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 19) ==== 137850+0+0 (secure 0 0 0) 0x7f75a0005020 con 0x7f75a8102250 2024-04-21T23:18:22.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:22.059+0000 7f759e7fc640 1 --2- 172.21.15.40:0/3187747537 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f75940a2d00 0x7f75940a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:22.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:22.059+0000 7f75ad3c3640 1 -- 172.21.15.40:0/3187747537 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f75940a2d00 msgr2=0x7f75940a51c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:6800/2500760788 2024-04-21T23:18:22.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:22.059+0000 7f759e7fc640 1 -- 172.21.15.40:0/3187747537 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f75a00cf9a0 con 0x7f75a8102250 2024-04-21T23:18:22.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:22.059+0000 7f75ad3c3640 1 --2- 172.21.15.40:0/3187747537 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f75940a2d00 0x7f75940a51c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T23:18:22.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:22.063+0000 7f759e7fc640 1 -- 172.21.15.40:0/3187747537 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f75a009c530 con 0x7f75a8102250 2024-04-21T23:18:22.072 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:18:21 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:18:21.965+0000 7f2f22aac200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-21T23:18:22.072 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:21 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/191514372' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-21T23:18:22.072 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:21 smithi040 ceph-mon[27984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-21T23:18:22.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:21 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/191514372' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-21T23:18:22.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:21 smithi111 ceph-mon[31085]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-21T23:18:22.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:21 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/191514372' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-21T23:18:22.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:21 smithi174 ceph-mon[31409]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-21T23:18:22.195 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Apr 21 23:18:21 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-z[33257]: 2024-04-21T23:18:21.735+0000 7f31e2b84200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-21T23:18:22.195 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Apr 21 23:18:21 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-z[33257]: 2024-04-21T23:18:21.827+0000 7f31e2b84200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-21T23:18:22.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:22.260+0000 7f75ad3c3640 1 -- 172.21.15.40:0/3187747537 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f75940a2d00 msgr2=0x7f75940a51c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:6800/2500760788 2024-04-21T23:18:22.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:22.260+0000 7f75ad3c3640 1 --2- 172.21.15.40:0/3187747537 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f75940a2d00 0x7f75940a51c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-21T23:18:22.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:22.280+0000 7f75af627640 1 -- 172.21.15.40:0/3187747537 --> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7f75a8061c20 con 0x7f75940a2d00 2024-04-21T23:18:22.432 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:18:22 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:18:22.073+0000 7f2f22aac200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-21T23:18:22.432 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:18:22 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:18:22.199+0000 7f2f22aac200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-21T23:18:22.432 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:18:22 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:18:22.298+0000 7f2f22aac200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-21T23:18:22.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:22.661+0000 7f75ad3c3640 1 -- 172.21.15.40:0/3187747537 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f75940a2d00 msgr2=0x7f75940a51c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:6800/2500760788 2024-04-21T23:18:22.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:22.661+0000 7f75ad3c3640 1 --2- 172.21.15.40:0/3187747537 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f75940a2d00 0x7f75940a51c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-21T23:18:22.944 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Apr 21 23:18:22 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-z[33257]: 2024-04-21T23:18:22.648+0000 7f31e2b84200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-21T23:18:22.944 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Apr 21 23:18:22 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-z[33257]: 2024-04-21T23:18:22.864+0000 7f31e2b84200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-21T23:18:22.944 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:18:22 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-y[32479]: 2024-04-21T23:18:22.560+0000 7f9dc849f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-21T23:18:22.944 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:18:22 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-y[32479]: 2024-04-21T23:18:22.773+0000 7f9dc849f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-21T23:18:22.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:22.945+0000 7f759e7fc640 1 -- 172.21.15.40:0/3187747537 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mgrmap(e 20) ==== 137850+0+0 (secure 0 0 0) 0x7f75a00984a0 con 0x7f75a8102250 2024-04-21T23:18:23.385 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:18:23 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:18:23.156+0000 7f2f22aac200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-21T23:18:23.386 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:22 smithi040 ceph-mon[27984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-21T23:18:23.386 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:22 smithi040 ceph-mon[27984]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-21T23:18:23.386 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:22 smithi040 ceph-mon[27984]: Standby manager daemon y restarted 2024-04-21T23:18:23.386 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:22 smithi040 ceph-mon[27984]: Standby manager daemon y started 2024-04-21T23:18:23.386 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:22 smithi040 ceph-mon[27984]: from='mgr.? 172.21.15.111:0/3220365648' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-21T23:18:23.386 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:22 smithi040 ceph-mon[27984]: from='mgr.? 172.21.15.111:0/3220365648' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T23:18:23.386 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:22 smithi040 ceph-mon[27984]: from='mgr.? 172.21.15.111:0/3220365648' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-21T23:18:23.386 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:22 smithi040 ceph-mon[27984]: from='mgr.? 172.21.15.111:0/3220365648' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T23:18:23.386 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:22 smithi040 ceph-mon[27984]: Standby manager daemon z restarted 2024-04-21T23:18:23.386 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:22 smithi040 ceph-mon[27984]: Standby manager daemon z started 2024-04-21T23:18:23.386 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:22 smithi040 ceph-mon[27984]: from='mgr.? 172.21.15.174:0/346040755' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-21T23:18:23.386 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:22 smithi040 ceph-mon[27984]: from='mgr.? 172.21.15.174:0/346040755' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T23:18:23.387 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:22 smithi040 ceph-mon[27984]: from='mgr.? 172.21.15.174:0/346040755' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-21T23:18:23.387 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:22 smithi040 ceph-mon[27984]: from='mgr.? 172.21.15.174:0/346040755' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T23:18:23.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:23.398+0000 7f759e7fc640 1 -- 172.21.15.40:0/3187747537 <== mon.0 v2:172.21.15.40:3300/0 8 ==== mgrmap(e 21) ==== 137518+0+0 (secure 0 0 0) 0x7f75a009c090 con 0x7f75a8102250 2024-04-21T23:18:23.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:23.399+0000 7f759e7fc640 1 -- 172.21.15.40:0/3187747537 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f75940a2d00 msgr2=0x7f75940a51c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T23:18:23.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:23.399+0000 7f759e7fc640 1 --2- 172.21.15.40:0/3187747537 >> [v2:172.21.15.40:6800/2500760788,v1:172.21.15.40:6801/2500760788] conn(0x7f75940a2d00 0x7f75940a51c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:23.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:22 smithi174 ceph-mon[31409]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-21T23:18:23.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:22 smithi174 ceph-mon[31409]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-21T23:18:23.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:22 smithi174 ceph-mon[31409]: Standby manager daemon y restarted 2024-04-21T23:18:23.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:22 smithi174 ceph-mon[31409]: Standby manager daemon y started 2024-04-21T23:18:23.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:22 smithi174 ceph-mon[31409]: from='mgr.? 172.21.15.111:0/3220365648' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-21T23:18:23.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:22 smithi174 ceph-mon[31409]: from='mgr.? 172.21.15.111:0/3220365648' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T23:18:23.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:22 smithi174 ceph-mon[31409]: from='mgr.? 172.21.15.111:0/3220365648' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-21T23:18:23.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:22 smithi174 ceph-mon[31409]: from='mgr.? 172.21.15.111:0/3220365648' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T23:18:23.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:22 smithi174 ceph-mon[31409]: Standby manager daemon z restarted 2024-04-21T23:18:23.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:22 smithi174 ceph-mon[31409]: Standby manager daemon z started 2024-04-21T23:18:23.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:22 smithi174 ceph-mon[31409]: from='mgr.? 172.21.15.174:0/346040755' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-21T23:18:23.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:22 smithi174 ceph-mon[31409]: from='mgr.? 172.21.15.174:0/346040755' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T23:18:23.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:22 smithi174 ceph-mon[31409]: from='mgr.? 172.21.15.174:0/346040755' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-21T23:18:23.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:22 smithi174 ceph-mon[31409]: from='mgr.? 172.21.15.174:0/346040755' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T23:18:23.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:22 smithi111 ceph-mon[31085]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-21T23:18:23.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:22 smithi111 ceph-mon[31085]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-21T23:18:23.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:22 smithi111 ceph-mon[31085]: Standby manager daemon y restarted 2024-04-21T23:18:23.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:22 smithi111 ceph-mon[31085]: Standby manager daemon y started 2024-04-21T23:18:23.446 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:22 smithi111 ceph-mon[31085]: from='mgr.? 172.21.15.111:0/3220365648' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-21T23:18:23.446 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:22 smithi111 ceph-mon[31085]: from='mgr.? 172.21.15.111:0/3220365648' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T23:18:23.446 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:22 smithi111 ceph-mon[31085]: from='mgr.? 172.21.15.111:0/3220365648' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-21T23:18:23.446 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:22 smithi111 ceph-mon[31085]: from='mgr.? 172.21.15.111:0/3220365648' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T23:18:23.446 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:22 smithi111 ceph-mon[31085]: Standby manager daemon z restarted 2024-04-21T23:18:23.446 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:22 smithi111 ceph-mon[31085]: Standby manager daemon z started 2024-04-21T23:18:23.446 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:22 smithi111 ceph-mon[31085]: from='mgr.? 172.21.15.174:0/346040755' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-21T23:18:23.446 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:22 smithi111 ceph-mon[31085]: from='mgr.? 172.21.15.174:0/346040755' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T23:18:23.446 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:22 smithi111 ceph-mon[31085]: from='mgr.? 172.21.15.174:0/346040755' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-21T23:18:23.446 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:22 smithi111 ceph-mon[31085]: from='mgr.? 172.21.15.174:0/346040755' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T23:18:23.644 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:18:23 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:18:23.386+0000 7f2f22aac200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-21T23:18:23.932 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:18:23 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:18:23.652+0000 7f2e83bb4700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T23:18:23.932 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:18:23 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:18:23.652+0000 7f2e83bb4700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T23:18:23.932 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:18:23 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:18:23.652+0000 7f2e83bb4700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T23:18:23.932 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:18:23 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:18:23.652+0000 7f2e83bb4700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T23:18:23.932 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:18:23 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:18:23.652+0000 7f2e83bb4700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T23:18:23.932 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:18:23 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:18:23.654+0000 7f2e81bb0700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T23:18:23.932 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:18:23 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:18:23.654+0000 7f2e81bb0700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T23:18:23.933 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:18:23 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:18:23.654+0000 7f2e81bb0700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T23:18:23.933 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:18:23 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:18:23.654+0000 7f2e81bb0700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T23:18:23.933 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:18:23 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:18:23.654+0000 7f2e81bb0700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T23:18:24.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.405+0000 7f759e7fc640 1 -- 172.21.15.40:0/3187747537 <== mon.0 v2:172.21.15.40:3300/0 9 ==== mgrmap(e 22) ==== 137645+0+0 (secure 0 0 0) 0x7f75a004d800 con 0x7f75a8102250 2024-04-21T23:18:24.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.406+0000 7f759e7fc640 1 --2- 172.21.15.40:0/3187747537 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f75940356a0 0x7f7594035a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:24.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.406+0000 7f759e7fc640 1 -- 172.21.15.40:0/3187747537 --> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7f75a8061c20 con 0x7f75940356a0 2024-04-21T23:18:24.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.408+0000 7f75ad3c3640 1 --2- 172.21.15.40:0/3187747537 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f75940356a0 0x7f7594035a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:24.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.411+0000 7f75ad3c3640 1 --2- 172.21.15.40:0/3187747537 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f75940356a0 0x7f7594035a80 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f7598000c00 tx=0x7f7598005fd0 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:24.420 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:23 smithi040 ceph-mon[27984]: mgrmap e20: x(active, since 9m), standbys: y, z 2024-04-21T23:18:24.420 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:23 smithi040 ceph-mon[27984]: Active manager daemon x restarted 2024-04-21T23:18:24.420 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:23 smithi040 ceph-mon[27984]: Activating manager daemon x 2024-04-21T23:18:24.420 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:23 smithi040 ceph-mon[27984]: osdmap e95: 12 total, 12 up, 12 in 2024-04-21T23:18:24.420 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:23 smithi040 ceph-mon[27984]: mgrmap e21: x(active, starting, since 0.010744s), standbys: y, z 2024-04-21T23:18:24.420 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:23 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T23:18:24.420 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:23 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:18:24.420 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:23 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:18:24.420 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:23 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-21T23:18:24.420 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:23 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-21T23:18:24.421 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:23 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-21T23:18:24.421 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:23 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-21T23:18:24.421 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:23 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-21T23:18:24.421 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:23 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-21T23:18:24.421 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:23 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-21T23:18:24.421 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:23 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-21T23:18:24.421 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:23 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-21T23:18:24.421 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:23 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-21T23:18:24.421 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:23 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-21T23:18:24.421 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:23 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-21T23:18:24.421 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:23 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-21T23:18:24.421 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:23 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-21T23:18:24.421 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:23 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-21T23:18:24.421 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:23 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T23:18:24.422 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:23 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T23:18:24.422 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:23 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T23:18:24.422 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:23 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T23:18:24.422 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:23 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T23:18:24.422 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:23 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T23:18:24.422 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:23 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T23:18:24.422 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:23 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T23:18:24.422 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:23 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T23:18:24.422 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:23 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T23:18:24.422 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:23 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T23:18:24.422 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:23 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T23:18:24.422 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:23 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-21T23:18:24.422 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:23 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-21T23:18:24.423 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:23 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-21T23:18:24.423 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:23 smithi040 ceph-mon[27984]: Manager daemon x is now available 2024-04-21T23:18:24.423 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:23 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:18:24.423 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:23 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-21T23:18:24.423 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:23 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-21T23:18:24.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.433+0000 7f759e7fc640 1 -- 172.21.15.40:0/3187747537 <== mgr.25111 v2:172.21.15.40:6800/478611259 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+3 (secure 0 0 0) 0x7f75a8061c20 con 0x7f75940356a0 2024-04-21T23:18:24.432 INFO:teuthology.orchestra.run.smithi040.stdout:{} 2024-04-21T23:18:24.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.436+0000 7f75af627640 1 -- 172.21.15.40:0/3187747537 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f75940356a0 msgr2=0x7f7594035a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:24.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.436+0000 7f75af627640 1 --2- 172.21.15.40:0/3187747537 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f75940356a0 0x7f7594035a80 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f7598000c00 tx=0x7f7598005fd0 comp rx=0 tx=0).stop 2024-04-21T23:18:24.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.436+0000 7f75af627640 1 -- 172.21.15.40:0/3187747537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75a8102250 msgr2=0x7f75a819eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:24.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.436+0000 7f75af627640 1 --2- 172.21.15.40:0/3187747537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75a8102250 0x7f75a819eda0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f75a000eb10 tx=0x7f75a000c2d0 comp rx=0 tx=0).stop 2024-04-21T23:18:24.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.436+0000 7f75adbc4640 1 -- 172.21.15.40:0/3187747537 reap_dead start 2024-04-21T23:18:24.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.436+0000 7f75af627640 1 -- 172.21.15.40:0/3187747537 shutdown_connections 2024-04-21T23:18:24.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.436+0000 7f75af627640 1 -- 172.21.15.40:0/3187747537 >> 172.21.15.40:0/3187747537 conn(0x7f75a80fbc10 msgr2=0x7f75a8104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:24.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.436+0000 7f75af627640 1 -- 172.21.15.40:0/3187747537 shutdown_connections 2024-04-21T23:18:24.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.436+0000 7f75af627640 1 -- 172.21.15.40:0/3187747537 wait complete. 2024-04-21T23:18:24.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:23 smithi174 ceph-mon[31409]: mgrmap e20: x(active, since 9m), standbys: y, z 2024-04-21T23:18:24.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:23 smithi174 ceph-mon[31409]: Active manager daemon x restarted 2024-04-21T23:18:24.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:23 smithi174 ceph-mon[31409]: Activating manager daemon x 2024-04-21T23:18:24.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:23 smithi174 ceph-mon[31409]: osdmap e95: 12 total, 12 up, 12 in 2024-04-21T23:18:24.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:23 smithi174 ceph-mon[31409]: mgrmap e21: x(active, starting, since 0.010744s), standbys: y, z 2024-04-21T23:18:24.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:23 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T23:18:24.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:23 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:18:24.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:23 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:18:24.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:23 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-21T23:18:24.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:23 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-21T23:18:24.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:23 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-21T23:18:24.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:23 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-21T23:18:24.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:23 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-21T23:18:24.446 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:23 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-21T23:18:24.446 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:23 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-21T23:18:24.446 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:23 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-21T23:18:24.446 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:23 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-21T23:18:24.446 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:23 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-21T23:18:24.446 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:23 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-21T23:18:24.446 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:23 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-21T23:18:24.446 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:23 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-21T23:18:24.447 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:23 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-21T23:18:24.447 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:23 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-21T23:18:24.447 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:23 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T23:18:24.447 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:23 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T23:18:24.447 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:23 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T23:18:24.447 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:23 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T23:18:24.447 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:23 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T23:18:24.447 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:23 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T23:18:24.447 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:23 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T23:18:24.448 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:23 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T23:18:24.448 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:23 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T23:18:24.448 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:23 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T23:18:24.448 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:23 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T23:18:24.448 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:23 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T23:18:24.448 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:23 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-21T23:18:24.448 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:23 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-21T23:18:24.448 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:23 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-21T23:18:24.449 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:23 smithi174 ceph-mon[31409]: Manager daemon x is now available 2024-04-21T23:18:24.449 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:23 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:18:24.449 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:23 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-21T23:18:24.449 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:23 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-21T23:18:24.450 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:23 smithi111 ceph-mon[31085]: mgrmap e20: x(active, since 9m), standbys: y, z 2024-04-21T23:18:24.450 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:23 smithi111 ceph-mon[31085]: Active manager daemon x restarted 2024-04-21T23:18:24.450 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:23 smithi111 ceph-mon[31085]: Activating manager daemon x 2024-04-21T23:18:24.450 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:23 smithi111 ceph-mon[31085]: osdmap e95: 12 total, 12 up, 12 in 2024-04-21T23:18:24.450 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:23 smithi111 ceph-mon[31085]: mgrmap e21: x(active, starting, since 0.010744s), standbys: y, z 2024-04-21T23:18:24.450 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:23 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T23:18:24.450 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:23 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:18:24.450 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:23 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:18:24.450 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:23 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-21T23:18:24.451 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:23 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-21T23:18:24.451 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:23 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-21T23:18:24.451 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:23 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-21T23:18:24.451 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:23 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-21T23:18:24.451 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:23 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-21T23:18:24.451 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:23 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-21T23:18:24.451 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:23 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-21T23:18:24.451 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:23 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-21T23:18:24.451 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:23 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-21T23:18:24.451 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:23 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-21T23:18:24.451 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:23 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-21T23:18:24.451 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:23 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-21T23:18:24.451 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:23 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-21T23:18:24.451 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:23 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-21T23:18:24.451 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:23 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T23:18:24.452 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:23 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T23:18:24.452 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:23 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T23:18:24.452 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:23 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T23:18:24.452 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:23 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T23:18:24.452 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:23 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T23:18:24.452 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:23 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T23:18:24.452 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:23 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T23:18:24.452 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:23 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T23:18:24.452 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:23 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T23:18:24.452 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:23 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T23:18:24.452 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:23 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T23:18:24.452 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:23 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-21T23:18:24.452 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:23 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-21T23:18:24.452 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:23 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-21T23:18:24.452 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:23 smithi111 ceph-mon[31085]: Manager daemon x is now available 2024-04-21T23:18:24.453 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:23 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:18:24.453 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:23 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-21T23:18:24.453 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:23 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-21T23:18:24.453 INFO:teuthology.run_tasks:Running task check-counter... 2024-04-21T23:18:24.468 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-04-21T23:18:24.507 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T23:18:24.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.612+0000 7fd4c5597640 1 -- 172.21.15.40:0/2322136438 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd4c0102250 msgr2=0x7fd4c0108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:24.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.612+0000 7fd4c5597640 1 --2- 172.21.15.40:0/2322136438 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd4c0102250 0x7fd4c0108ae0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd4b8009ca0 tx=0x7fd4b802c0f0 comp rx=0 tx=0).stop 2024-04-21T23:18:24.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.612+0000 7fd4c5597640 1 -- 172.21.15.40:0/2322136438 shutdown_connections 2024-04-21T23:18:24.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.612+0000 7fd4c5597640 1 --2- 172.21.15.40:0/2322136438 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4c0101890 0x7fd4c0101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:24.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.612+0000 7fd4c5597640 1 --2- 172.21.15.40:0/2322136438 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd4c0102250 0x7fd4c0108ae0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:24.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.612+0000 7fd4c5597640 1 --2- 172.21.15.40:0/2322136438 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4c01006a0 0x7fd4c0100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:24.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.612+0000 7fd4c5597640 1 -- 172.21.15.40:0/2322136438 >> 172.21.15.40:0/2322136438 conn(0x7fd4c00fbc30 msgr2=0x7fd4c00fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:24.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.612+0000 7fd4c5597640 1 -- 172.21.15.40:0/2322136438 shutdown_connections 2024-04-21T23:18:24.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.612+0000 7fd4c5597640 1 -- 172.21.15.40:0/2322136438 wait complete. 2024-04-21T23:18:24.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.612+0000 7fd4c5597640 1 Processor -- start 2024-04-21T23:18:24.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.613+0000 7fd4c5597640 1 -- start start 2024-04-21T23:18:24.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.613+0000 7fd4c5597640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4c01006a0 0x7fd4c0104630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:24.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.613+0000 7fd4c5597640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4c0101890 0x7fd4c0102c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:24.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.613+0000 7fd4c5597640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd4c0102250 0x7fd4c01031b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:24.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.613+0000 7fd4c5597640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd4c010c320 con 0x7fd4c01006a0 2024-04-21T23:18:24.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.613+0000 7fd4c5597640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd4c010c1a0 con 0x7fd4c0101890 2024-04-21T23:18:24.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.613+0000 7fd4c5597640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd4c010c4a0 con 0x7fd4c0102250 2024-04-21T23:18:24.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.613+0000 7fd4beffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4c01006a0 0x7fd4c0104630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:24.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.613+0000 7fd4beffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4c01006a0 0x7fd4c0104630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60816/0 (socket says 172.21.15.40:60816) 2024-04-21T23:18:24.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.613+0000 7fd4beffd640 1 -- 172.21.15.40:0/3993811885 learned_addr learned my addr 172.21.15.40:0/3993811885 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:24.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.613+0000 7fd4be7fc640 1 --2- 172.21.15.40:0/3993811885 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4c0101890 0x7fd4c0102c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:24.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.613+0000 7fd4bf7fe640 1 --2- 172.21.15.40:0/3993811885 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd4c0102250 0x7fd4c01031b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:24.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.613+0000 7fd4beffd640 1 -- 172.21.15.40:0/3993811885 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd4c0102250 msgr2=0x7fd4c01031b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:24.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.613+0000 7fd4beffd640 1 --2- 172.21.15.40:0/3993811885 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd4c0102250 0x7fd4c01031b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:24.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.613+0000 7fd4beffd640 1 -- 172.21.15.40:0/3993811885 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4c0101890 msgr2=0x7fd4c0102c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:24.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.613+0000 7fd4beffd640 1 --2- 172.21.15.40:0/3993811885 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4c0101890 0x7fd4c0102c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:24.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.613+0000 7fd4beffd640 1 -- 172.21.15.40:0/3993811885 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4c0103a40 con 0x7fd4c01006a0 2024-04-21T23:18:24.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.613+0000 7fd4beffd640 1 --2- 172.21.15.40:0/3993811885 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4c01006a0 0x7fd4c0104630 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fd4b000e9c0 tx=0x7fd4b000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:24.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.614+0000 7fd4a7fff640 1 -- 172.21.15.40:0/3993811885 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4b000cd80 con 0x7fd4c01006a0 2024-04-21T23:18:24.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.614+0000 7fd4a7fff640 1 -- 172.21.15.40:0/3993811885 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd4b00035f0 con 0x7fd4c01006a0 2024-04-21T23:18:24.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.614+0000 7fd4c5597640 1 -- 172.21.15.40:0/3993811885 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4c01a1b40 con 0x7fd4c01006a0 2024-04-21T23:18:24.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.614+0000 7fd4a7fff640 1 -- 172.21.15.40:0/3993811885 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4b0010690 con 0x7fd4c01006a0 2024-04-21T23:18:24.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.614+0000 7fd4c5597640 1 -- 172.21.15.40:0/3993811885 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4c01a2090 con 0x7fd4c01006a0 2024-04-21T23:18:24.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.615+0000 7fd4c5597640 1 -- 172.21.15.40:0/3993811885 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4c01a22c0 con 0x7fd4c01006a0 2024-04-21T23:18:24.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.616+0000 7fd4a7fff640 1 -- 172.21.15.40:0/3993811885 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 22) ==== 137645+0+0 (secure 0 0 0) 0x7fd4b00037a0 con 0x7fd4c01006a0 2024-04-21T23:18:24.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.616+0000 7fd4a7fff640 1 --2- 172.21.15.40:0/3993811885 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fd4a80a2990 0x7fd4a80a4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:24.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.617+0000 7fd4be7fc640 1 --2- 172.21.15.40:0/3993811885 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fd4a80a2990 0x7fd4a80a4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:24.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.617+0000 7fd4a7fff640 1 -- 172.21.15.40:0/3993811885 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7fd4b00cf480 con 0x7fd4c01006a0 2024-04-21T23:18:24.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.617+0000 7fd4be7fc640 1 --2- 172.21.15.40:0/3993811885 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fd4a80a2990 0x7fd4a80a4e50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd4c01043a0 tx=0x7fd4b4005f90 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:24.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.621+0000 7fd4a7fff640 1 -- 172.21.15.40:0/3993811885 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fd4b009c2d0 con 0x7fd4c01006a0 2024-04-21T23:18:24.682 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:18:24 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:18:24.419+0000 7f2e7f3ab700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T23:18:24.682 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:18:24 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:18:24.419+0000 7f2e7f3ab700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T23:18:24.682 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:18:24 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:18:24.419+0000 7f2e7f3ab700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T23:18:24.682 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:18:24 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:18:24.419+0000 7f2e7f3ab700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T23:18:24.682 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:18:24 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:18:24.419+0000 7f2e7f3ab700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T23:18:24.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.811+0000 7fd4c5597640 1 -- 172.21.15.40:0/3993811885 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd4c01a2990 con 0x7fd4c01006a0 2024-04-21T23:18:24.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.814+0000 7fd4a7fff640 1 -- 172.21.15.40:0/3993811885 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) ==== 74+0+25231 (secure 0 0 0) 0x7fd4b0086020 con 0x7fd4c01006a0 2024-04-21T23:18:24.813 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:18:24.813 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":95,"fsid":"d814e4dc-0033-11ef-bc93-c7b262605968","created":"2024-04-21T23:07:52.659207+0000","modified":"2024-04-21T23:18:23.389706+0000","last_up_change":"2024-04-21T23:15:05.308821+0000","last_in_change":"2024-04-21T23:14:45.447491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:11:14.824775+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T23:15:54.322571+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T23:15:55.164493+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T23:15:58.838266+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"0e8d692a-22fe-4a87-8d99-9151f96cbb02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6803","nonce":2009758784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6805","nonce":2009758784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6809","nonce":2009758784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6807","nonce":2009758784}]},"public_addr":"172.21.15.40:6803/2009758784","cluster_addr":"172.21.15.40:6805/2009758784","heartbeat_back_addr":"172.21.15.40:6809/2009758784","heartbeat_front_addr":"172.21.15.40:6807/2009758784","state":["exists","up"]},{"osd":1,"uuid":"fe5fe11b-0e84-4359-83a7-1ba45105755a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6801","nonce":1763903836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6803","nonce":1763903836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6807","nonce":1763903836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6805","nonce":1763903836}]},"public_addr":"172.21.15.111:6801/1763903836","cluster_addr":"172.21.15.111:6803/1763903836","heartbeat_back_addr":"172.21.15.111:6807/1763903836","heartbeat_front_addr":"172.21.15.111:6805/1763903836","state":["exists","up"]},{"osd":2,"uuid":"813ad735-5cc2-4da0-ae16-6db238395fbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6801","nonce":439695825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6803","nonce":439695825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6807","nonce":439695825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6805","nonce":439695825}]},"public_addr":"172.21.15.174:6801/439695825","cluster_addr":"172.21.15.174:6803/439695825","heartbeat_back_addr":"172.21.15.174:6807/439695825","heartbeat_front_addr":"172.21.15.174:6805/439695825","state":["exists","up"]},{"osd":3,"uuid":"09becb03-b7ce-4ffc-917d-b7900ec37e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6811","nonce":3454312436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6813","nonce":3454312436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6817","nonce":3454312436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6815","nonce":3454312436}]},"public_addr":"172.21.15.40:6811/3454312436","cluster_addr":"172.21.15.40:6813/3454312436","heartbeat_back_addr":"172.21.15.40:6817/3454312436","heartbeat_front_addr":"172.21.15.40:6815/3454312436","state":["exists","up"]},{"osd":4,"uuid":"46eabfd7-d88a-4efd-94eb-04354d3a9d7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6809","nonce":1810866070}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6811","nonce":1810866070}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6815","nonce":1810866070}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6813","nonce":1810866070}]},"public_addr":"172.21.15.111:6809/1810866070","cluster_addr":"172.21.15.111:6811/1810866070","heartbeat_back_addr":"172.21.15.111:6815/1810866070","heartbeat_front_addr":"172.21.15.111:6813/1810866070","state":["exists","up"]},{"osd":5,"uuid":"6f73b582-6c65-41db-aab0-9b360f83d5c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6809","nonce":3752986711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6811","nonce":3752986711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6815","nonce":3752986711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6813","nonce":3752986711}]},"public_addr":"172.21.15.174:6809/3752986711","cluster_addr":"172.21.15.174:6811/3752986711","heartbeat_back_addr":"172.21.15.174:6815/3752986711","heartbeat_front_addr":"172.21.15.174:6813/3752986711","state":["exists","up"]},{"osd":6,"uuid":"39f305f0-12d2-4732-aaa0-cd66700af538","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6819","nonce":1618855427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6821","nonce":1618855427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6825","nonce":1618855427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6823","nonce":1618855427}]},"public_addr":"172.21.15.40:6819/1618855427","cluster_addr":"172.21.15.40:6821/1618855427","heartbeat_back_addr":"172.21.15.40:6825/1618855427","heartbeat_front_addr":"172.21.15.40:6823/1618855427","state":["exists","up"]},{"osd":7,"uuid":"769f2468-5102-490b-89f8-d0444d3169a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6817","nonce":927054162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6819","nonce":927054162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6823","nonce":927054162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6821","nonce":927054162}]},"public_addr":"172.21.15.111:6817/927054162","cluster_addr":"172.21.15.111:6819/927054162","heartbeat_back_addr":"172.21.15.111:6823/927054162","heartbeat_front_addr":"172.21.15.111:6821/927054162","state":["exists","up"]},{"osd":8,"uuid":"1a444e80-007d-473b-b972-4c814a813c22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6817","nonce":3817848746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6819","nonce":3817848746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6823","nonce":3817848746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6821","nonce":3817848746}]},"public_addr":"172.21.15.174:6817/3817848746","cluster_addr":"172.21.15.174:6819/3817848746","heartbeat_back_addr":"172.21.15.174:6823/3817848746","heartbeat_front_addr":"172.21.15.174:6821/3817848746","state":["exists","up"]},{"osd":9,"uuid":"9a6b6a56-6ae5-49fe-9e5b-cb17d686e0dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6827","nonce":1649446531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6829","nonce":1649446531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6833","nonce":1649446531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6831","nonce":1649446531}]},"public_addr":"172.21.15.40:6827/1649446531","cluster_addr":"172.21.15.40:6829/1649446531","heartbeat_back_addr":"172.21.15.40:6833/1649446531","heartbeat_front_addr":"172.21.15.40:6831/1649446531","state":["exists","up"]},{"osd":10,"uuid":"a70038ba-e2ad-4170-85cc-ea0470a2edef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6825","nonce":1227538935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6827","nonce":1227538935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6831","nonce":1227538935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6829","nonce":1227538935}]},"public_addr":"172.21.15.111:6825/1227538935","cluster_addr":"172.21.15.111:6827/1227538935","heartbeat_back_addr":"172.21.15.111:6831/1227538935","heartbeat_front_addr":"172.21.15.111:6829/1227538935","state":["exists","up"]},{"osd":11,"uuid":"781532bb-7a5e-44e3-8eb7-b584b0054666","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6825","nonce":2045865553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6827","nonce":2045865553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6831","nonce":2045865553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6829","nonce":2045865553}]},"public_addr":"172.21.15.174:6825/2045865553","cluster_addr":"172.21.15.174:6827/2045865553","heartbeat_back_addr":"172.21.15.174:6831/2045865553","heartbeat_front_addr":"172.21.15.174:6829/2045865553","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:10:24.761290+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:10:48.565428+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:11:12.307818+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:11:37.951529+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:02.832523+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:27.568879+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:52.997707+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:13:18.937274+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:13:44.708657+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:14:10.833855+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:14:37.078272+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:15:03.878076+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":1}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.2a","mappings":[{"from":3,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"4.9","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":10}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:6801/2500760788":"2024-04-22T23:18:23.389674+0000","172.21.15.40:6800/2500760788":"2024-04-22T23:18:23.389674+0000","172.21.15.40:0/2742002992":"2024-04-22T23:18:23.389674+0000","172.21.15.40:0/2494672676":"2024-04-22T23:18:23.389674+0000","172.21.15.40:6801/526554319":"2024-04-22T23:08:50.305770+0000","172.21.15.40:6800/526554319":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/3178763471":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/4045911364":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/276300080":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/3689649814":"2024-04-22T23:08:23.743030+0000","172.21.15.40:0/2241116715":"2024-04-22T23:08:23.743030+0000","172.21.15.40:0/1406242364":"2024-04-22T23:18:23.389674+0000","172.21.15.40:0/1737227818":"2024-04-22T23:18:23.389674+0000","172.21.15.40:6801/1479040326":"2024-04-22T23:08:23.743030+0000","172.21.15.40:6800/1479040326":"2024-04-22T23:08:23.743030+0000","172.21.15.40:0/681298099":"2024-04-22T23:08:23.743030+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T23:18:24.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.816+0000 7fd4c5597640 1 -- 172.21.15.40:0/3993811885 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fd4a80a2990 msgr2=0x7fd4a80a4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:24.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.816+0000 7fd4c5597640 1 --2- 172.21.15.40:0/3993811885 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fd4a80a2990 0x7fd4a80a4e50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd4c01043a0 tx=0x7fd4b4005f90 comp rx=0 tx=0).stop 2024-04-21T23:18:24.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.816+0000 7fd4c5597640 1 -- 172.21.15.40:0/3993811885 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4c01006a0 msgr2=0x7fd4c0104630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:24.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.816+0000 7fd4c5597640 1 --2- 172.21.15.40:0/3993811885 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4c01006a0 0x7fd4c0104630 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fd4b000e9c0 tx=0x7fd4b000ee80 comp rx=0 tx=0).stop 2024-04-21T23:18:24.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.816+0000 7fd4c5597640 1 -- 172.21.15.40:0/3993811885 shutdown_connections 2024-04-21T23:18:24.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.816+0000 7fd4c5597640 1 --2- 172.21.15.40:0/3993811885 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fd4a80a2990 0x7fd4a80a4e50 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:24.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.816+0000 7fd4c5597640 1 --2- 172.21.15.40:0/3993811885 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4c0101890 0x7fd4c0102c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:24.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.816+0000 7fd4c5597640 1 --2- 172.21.15.40:0/3993811885 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd4c0102250 0x7fd4c01031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:24.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.816+0000 7fd4c5597640 1 --2- 172.21.15.40:0/3993811885 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4c01006a0 0x7fd4c0104630 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:24.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.816+0000 7fd4c5597640 1 -- 172.21.15.40:0/3993811885 >> 172.21.15.40:0/3993811885 conn(0x7fd4c00fbc30 msgr2=0x7fd4c00fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:24.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.816+0000 7fd4c5597640 1 -- 172.21.15.40:0/3993811885 shutdown_connections 2024-04-21T23:18:24.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.816+0000 7fd4c5597640 1 -- 172.21.15.40:0/3993811885 wait complete. 2024-04-21T23:18:24.827 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T23:11:14.824775+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T23:15:54.322571+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T23:15:55.164493+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T23:15:58.838266+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T23:18:24.827 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T23:18:24.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.934+0000 7f293f59e640 1 -- 172.21.15.40:0/1200130375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2940101680 msgr2=0x7f2940101aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:24.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.934+0000 7f293f59e640 1 --2- 172.21.15.40:0/1200130375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2940101680 0x7f2940101aa0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f2930009900 tx=0x7f293002bc30 comp rx=0 tx=0).stop 2024-04-21T23:18:24.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.934+0000 7f293f59e640 1 -- 172.21.15.40:0/1200130375 shutdown_connections 2024-04-21T23:18:24.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.934+0000 7f293f59e640 1 --2- 172.21.15.40:0/1200130375 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2940103230 0x7f2940109ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:24.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.934+0000 7f293f59e640 1 --2- 172.21.15.40:0/1200130375 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2940102870 0x7f2940102cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:24.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.934+0000 7f293f59e640 1 --2- 172.21.15.40:0/1200130375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2940101680 0x7f2940101aa0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:24.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.934+0000 7f293f59e640 1 -- 172.21.15.40:0/1200130375 >> 172.21.15.40:0/1200130375 conn(0x7f29400fcc10 msgr2=0x7f29400ff030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:24.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.934+0000 7f293f59e640 1 -- 172.21.15.40:0/1200130375 shutdown_connections 2024-04-21T23:18:24.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.934+0000 7f293f59e640 1 -- 172.21.15.40:0/1200130375 wait complete. 2024-04-21T23:18:24.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.935+0000 7f293f59e640 1 Processor -- start 2024-04-21T23:18:24.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.935+0000 7f293f59e640 1 -- start start 2024-04-21T23:18:24.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.935+0000 7f293f59e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2940101680 0x7f2940197910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:24.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.935+0000 7f293f59e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2940102870 0x7f2940197e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:24.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.935+0000 7f293f59e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2940103230 0x7f294019eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:24.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.935+0000 7f293f59e640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f294010d2d0 con 0x7f2940101680 2024-04-21T23:18:24.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.935+0000 7f293f59e640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f294010d150 con 0x7f2940102870 2024-04-21T23:18:24.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.935+0000 7f293f59e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f294010d450 con 0x7f2940103230 2024-04-21T23:18:24.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.935+0000 7f293e59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2940101680 0x7f2940197910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:24.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.935+0000 7f293e59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2940101680 0x7f2940197910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60836/0 (socket says 172.21.15.40:60836) 2024-04-21T23:18:24.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.935+0000 7f293e59c640 1 -- 172.21.15.40:0/1884359250 learned_addr learned my addr 172.21.15.40:0/1884359250 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:24.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.935+0000 7f2937fff640 1 --2- 172.21.15.40:0/1884359250 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2940102870 0x7f2940197e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:24.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.935+0000 7f293ed9d640 1 --2- 172.21.15.40:0/1884359250 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2940103230 0x7f294019eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:24.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.936+0000 7f293e59c640 1 -- 172.21.15.40:0/1884359250 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2940103230 msgr2=0x7f294019eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:24.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.936+0000 7f293e59c640 1 --2- 172.21.15.40:0/1884359250 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2940103230 0x7f294019eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:24.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.936+0000 7f293e59c640 1 -- 172.21.15.40:0/1884359250 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2940102870 msgr2=0x7f2940197e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:24.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.936+0000 7f293e59c640 1 --2- 172.21.15.40:0/1884359250 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2940102870 0x7f2940197e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:24.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.936+0000 7f293e59c640 1 -- 172.21.15.40:0/1884359250 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f294019f530 con 0x7f2940101680 2024-04-21T23:18:24.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.936+0000 7f293ed9d640 1 --2- 172.21.15.40:0/1884359250 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2940103230 0x7f294019eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:18:24.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.936+0000 7f293e59c640 1 --2- 172.21.15.40:0/1884359250 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2940101680 0x7f2940197910 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f2930000c00 tx=0x7f2930004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:24.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.936+0000 7f29377fe640 1 -- 172.21.15.40:0/1884359250 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2930041070 con 0x7f2940101680 2024-04-21T23:18:24.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.936+0000 7f29377fe640 1 -- 172.21.15.40:0/1884359250 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f29300054e0 con 0x7f2940101680 2024-04-21T23:18:24.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.936+0000 7f293f59e640 1 -- 172.21.15.40:0/1884359250 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f294019f7c0 con 0x7f2940101680 2024-04-21T23:18:24.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.936+0000 7f29377fe640 1 -- 172.21.15.40:0/1884359250 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29300346a0 con 0x7f2940101680 2024-04-21T23:18:24.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.936+0000 7f293f59e640 1 -- 172.21.15.40:0/1884359250 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f294019fd00 con 0x7f2940101680 2024-04-21T23:18:24.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.937+0000 7f293f59e640 1 -- 172.21.15.40:0/1884359250 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29401089a0 con 0x7f2940101680 2024-04-21T23:18:24.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.938+0000 7f29377fe640 1 -- 172.21.15.40:0/1884359250 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 22) ==== 137645+0+0 (secure 0 0 0) 0x7f2930044070 con 0x7f2940101680 2024-04-21T23:18:24.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.939+0000 7f29377fe640 1 --2- 172.21.15.40:0/1884359250 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f292c0a2a10 0x7f292c0a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:24.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.939+0000 7f2937fff640 1 --2- 172.21.15.40:0/1884359250 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f292c0a2a10 0x7f292c0a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:24.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.939+0000 7f29377fe640 1 -- 172.21.15.40:0/1884359250 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f29300f0de0 con 0x7f2940101680 2024-04-21T23:18:24.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.939+0000 7f2937fff640 1 --2- 172.21.15.40:0/1884359250 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f292c0a2a10 0x7f292c0a4ed0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2940198e20 tx=0x7f292800a2d0 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:24.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:24.943+0000 7f29377fe640 1 -- 172.21.15.40:0/1884359250 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f29300bdb60 con 0x7f2940101680 2024-04-21T23:18:25.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.128+0000 7f293f59e640 1 -- 172.21.15.40:0/1884359250 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f2940062340 con 0x7f2940101680 2024-04-21T23:18:25.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.129+0000 7f29377fe640 1 -- 172.21.15.40:0/1884359250 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v95) ==== 93+0+10 (secure 0 0 0) 0x7f29300bd460 con 0x7f2940101680 2024-04-21T23:18:25.128 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-04-21T23:18:25.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.131+0000 7f293f59e640 1 -- 172.21.15.40:0/1884359250 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f292c0a2a10 msgr2=0x7f292c0a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:25.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.131+0000 7f293f59e640 1 --2- 172.21.15.40:0/1884359250 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f292c0a2a10 0x7f292c0a4ed0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2940198e20 tx=0x7f292800a2d0 comp rx=0 tx=0).stop 2024-04-21T23:18:25.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.131+0000 7f293f59e640 1 -- 172.21.15.40:0/1884359250 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2940101680 msgr2=0x7f2940197910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:25.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.131+0000 7f293f59e640 1 --2- 172.21.15.40:0/1884359250 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2940101680 0x7f2940197910 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f2930000c00 tx=0x7f2930004040 comp rx=0 tx=0).stop 2024-04-21T23:18:25.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.131+0000 7f293f59e640 1 -- 172.21.15.40:0/1884359250 shutdown_connections 2024-04-21T23:18:25.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.131+0000 7f293f59e640 1 --2- 172.21.15.40:0/1884359250 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f292c0a2a10 0x7f292c0a4ed0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:25.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.131+0000 7f293f59e640 1 --2- 172.21.15.40:0/1884359250 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2940102870 0x7f2940197e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:25.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.131+0000 7f293f59e640 1 --2- 172.21.15.40:0/1884359250 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2940103230 0x7f294019eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:25.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.131+0000 7f293f59e640 1 --2- 172.21.15.40:0/1884359250 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2940101680 0x7f2940197910 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:25.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.131+0000 7f293f59e640 1 -- 172.21.15.40:0/1884359250 >> 172.21.15.40:0/1884359250 conn(0x7f29400fcc10 msgr2=0x7f2940105b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:25.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.131+0000 7f293f59e640 1 -- 172.21.15.40:0/1884359250 shutdown_connections 2024-04-21T23:18:25.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.131+0000 7f293f59e640 1 -- 172.21.15.40:0/1884359250 wait complete. 2024-04-21T23:18:25.141 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T23:18:25.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.253+0000 7fc3155be640 1 -- 172.21.15.40:0/4094440845 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc310102230 msgr2=0x7fc310108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:25.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.253+0000 7fc3155be640 1 --2- 172.21.15.40:0/4094440845 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc310102230 0x7fc310108ac0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fc308009ca0 tx=0x7fc30802c0f0 comp rx=0 tx=0).stop 2024-04-21T23:18:25.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.253+0000 7fc3155be640 1 -- 172.21.15.40:0/4094440845 shutdown_connections 2024-04-21T23:18:25.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.253+0000 7fc3155be640 1 --2- 172.21.15.40:0/4094440845 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc310102230 0x7fc310108ac0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:25.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.253+0000 7fc3155be640 1 --2- 172.21.15.40:0/4094440845 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc310101870 0x7fc310101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:25.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.253+0000 7fc3155be640 1 --2- 172.21.15.40:0/4094440845 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc310100680 0x7fc310100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:25.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.253+0000 7fc3155be640 1 -- 172.21.15.40:0/4094440845 >> 172.21.15.40:0/4094440845 conn(0x7fc3100fbbf0 msgr2=0x7fc3100fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:25.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.253+0000 7fc3155be640 1 -- 172.21.15.40:0/4094440845 shutdown_connections 2024-04-21T23:18:25.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.253+0000 7fc3155be640 1 -- 172.21.15.40:0/4094440845 wait complete. 2024-04-21T23:18:25.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.254+0000 7fc3155be640 1 Processor -- start 2024-04-21T23:18:25.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.254+0000 7fc3155be640 1 -- start start 2024-04-21T23:18:25.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.254+0000 7fc3155be640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc310100680 0x7fc3100694a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:25.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.254+0000 7fc3155be640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc310101870 0x7fc31019c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:25.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.254+0000 7fc3155be640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc310102230 0x7fc31019edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:25.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.254+0000 7fc3155be640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc31010c2e0 con 0x7fc310101870 2024-04-21T23:18:25.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.254+0000 7fc3155be640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc31010c160 con 0x7fc310100680 2024-04-21T23:18:25.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.254+0000 7fc3155be640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc31010c460 con 0x7fc310102230 2024-04-21T23:18:25.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.254+0000 7fc30f7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc310102230 0x7fc31019edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:25.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.254+0000 7fc30e7fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc310101870 0x7fc31019c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:25.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.254+0000 7fc30effd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc310100680 0x7fc3100694a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:25.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.254+0000 7fc30f7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc310102230 0x7fc31019edc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.40:34532/0 (socket says 172.21.15.40:34532) 2024-04-21T23:18:25.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.254+0000 7fc30f7fe640 1 -- 172.21.15.40:0/4032454917 learned_addr learned my addr 172.21.15.40:0/4032454917 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:25.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.254+0000 7fc30effd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc310100680 0x7fc3100694a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.40:59954/0 (socket says 172.21.15.40:59954) 2024-04-21T23:18:25.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.255+0000 7fc30e7fc640 1 -- 172.21.15.40:0/4032454917 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc310102230 msgr2=0x7fc31019edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:25.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.255+0000 7fc30e7fc640 1 --2- 172.21.15.40:0/4032454917 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc310102230 0x7fc31019edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:25.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.255+0000 7fc30e7fc640 1 -- 172.21.15.40:0/4032454917 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc310100680 msgr2=0x7fc3100694a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:25.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.255+0000 7fc30e7fc640 1 --2- 172.21.15.40:0/4032454917 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc310100680 0x7fc3100694a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:25.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.255+0000 7fc30e7fc640 1 -- 172.21.15.40:0/4032454917 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc31019f680 con 0x7fc310101870 2024-04-21T23:18:25.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.255+0000 7fc30f7fe640 1 --2- 172.21.15.40:0/4032454917 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc310102230 0x7fc31019edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:18:25.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.255+0000 7fc30e7fc640 1 --2- 172.21.15.40:0/4032454917 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc310101870 0x7fc31019c750 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fc30400e9f0 tx=0x7fc30400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:25.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.255+0000 7fc2f7fff640 1 -- 172.21.15.40:0/4032454917 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc30400ccc0 con 0x7fc310101870 2024-04-21T23:18:25.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.255+0000 7fc2f7fff640 1 -- 172.21.15.40:0/4032454917 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc304010910 con 0x7fc310101870 2024-04-21T23:18:25.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.255+0000 7fc3155be640 1 -- 172.21.15.40:0/4032454917 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc31019f970 con 0x7fc310101870 2024-04-21T23:18:25.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.255+0000 7fc2f7fff640 1 -- 172.21.15.40:0/4032454917 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc304018960 con 0x7fc310101870 2024-04-21T23:18:25.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.255+0000 7fc3155be640 1 -- 172.21.15.40:0/4032454917 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc31019fec0 con 0x7fc310101870 2024-04-21T23:18:25.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.256+0000 7fc3155be640 1 -- 172.21.15.40:0/4032454917 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc310107a20 con 0x7fc310101870 2024-04-21T23:18:25.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.257+0000 7fc2f7fff640 1 -- 172.21.15.40:0/4032454917 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 22) ==== 137645+0+0 (secure 0 0 0) 0x7fc304010ac0 con 0x7fc310101870 2024-04-21T23:18:25.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.258+0000 7fc2f7fff640 1 --2- 172.21.15.40:0/4032454917 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fc2f80a2990 0x7fc2f80a4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:25.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.258+0000 7fc30effd640 1 --2- 172.21.15.40:0/4032454917 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fc2f80a2990 0x7fc2f80a4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:25.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.258+0000 7fc2f7fff640 1 -- 172.21.15.40:0/4032454917 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7fc3040cf4a0 con 0x7fc310101870 2024-04-21T23:18:25.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.259+0000 7fc30effd640 1 --2- 172.21.15.40:0/4032454917 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fc2f80a2990 0x7fc2f80a4e50 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fc3101016d0 tx=0x7fc300005f90 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:25.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.263+0000 7fc2f7fff640 1 -- 172.21.15.40:0/4032454917 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fc30409bbd0 con 0x7fc310101870 2024-04-21T23:18:25.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:24 smithi040 ceph-mon[27984]: [21/Apr/2024:23:18:23] ENGINE Bus STARTING 2024-04-21T23:18:25.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:24 smithi040 ceph-mon[27984]: [21/Apr/2024:23:18:24] ENGINE Serving on https://172.21.15.40:7150 2024-04-21T23:18:25.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:24 smithi040 ceph-mon[27984]: mgrmap e22: x(active, since 1.01751s), standbys: y, z 2024-04-21T23:18:25.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:24 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/3993811885' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:18:25.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:24 smithi174 ceph-mon[31409]: [21/Apr/2024:23:18:23] ENGINE Bus STARTING 2024-04-21T23:18:25.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:24 smithi174 ceph-mon[31409]: [21/Apr/2024:23:18:24] ENGINE Serving on https://172.21.15.40:7150 2024-04-21T23:18:25.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:24 smithi174 ceph-mon[31409]: mgrmap e22: x(active, since 1.01751s), standbys: y, z 2024-04-21T23:18:25.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:24 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/3993811885' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:18:25.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:24 smithi111 ceph-mon[31085]: [21/Apr/2024:23:18:23] ENGINE Bus STARTING 2024-04-21T23:18:25.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:24 smithi111 ceph-mon[31085]: [21/Apr/2024:23:18:24] ENGINE Serving on https://172.21.15.40:7150 2024-04-21T23:18:25.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:24 smithi111 ceph-mon[31085]: mgrmap e22: x(active, since 1.01751s), standbys: y, z 2024-04-21T23:18:25.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:24 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/3993811885' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:18:25.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.466+0000 7fc3155be640 1 -- 172.21.15.40:0/4032454917 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fc31002d170 con 0x7fc310101870 2024-04-21T23:18:25.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.467+0000 7fc2f7fff640 1 -- 172.21.15.40:0/4032454917 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v95) ==== 104+0+11 (secure 0 0 0) 0x7fc304086020 con 0x7fc310101870 2024-04-21T23:18:25.466 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-21T23:18:25.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.469+0000 7fc3155be640 1 -- 172.21.15.40:0/4032454917 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fc2f80a2990 msgr2=0x7fc2f80a4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:25.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.469+0000 7fc3155be640 1 --2- 172.21.15.40:0/4032454917 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fc2f80a2990 0x7fc2f80a4e50 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fc3101016d0 tx=0x7fc300005f90 comp rx=0 tx=0).stop 2024-04-21T23:18:25.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.469+0000 7fc3155be640 1 -- 172.21.15.40:0/4032454917 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc310101870 msgr2=0x7fc31019c750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:25.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.469+0000 7fc3155be640 1 --2- 172.21.15.40:0/4032454917 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc310101870 0x7fc31019c750 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fc30400e9f0 tx=0x7fc30400eeb0 comp rx=0 tx=0).stop 2024-04-21T23:18:25.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.469+0000 7fc3155be640 1 -- 172.21.15.40:0/4032454917 shutdown_connections 2024-04-21T23:18:25.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.469+0000 7fc3155be640 1 --2- 172.21.15.40:0/4032454917 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fc2f80a2990 0x7fc2f80a4e50 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:25.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.469+0000 7fc3155be640 1 --2- 172.21.15.40:0/4032454917 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc310102230 0x7fc31019edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:25.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.469+0000 7fc3155be640 1 --2- 172.21.15.40:0/4032454917 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc310101870 0x7fc31019c750 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:25.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.469+0000 7fc3155be640 1 --2- 172.21.15.40:0/4032454917 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc310100680 0x7fc3100694a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:25.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.469+0000 7fc3155be640 1 -- 172.21.15.40:0/4032454917 >> 172.21.15.40:0/4032454917 conn(0x7fc3100fbbf0 msgr2=0x7fc310104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:25.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.469+0000 7fc3155be640 1 -- 172.21.15.40:0/4032454917 shutdown_connections 2024-04-21T23:18:25.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.469+0000 7fc3155be640 1 -- 172.21.15.40:0/4032454917 wait complete. 2024-04-21T23:18:25.480 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T23:18:25.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.613+0000 7f9566c4e640 1 -- 172.21.15.40:0/1277788189 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9560107500 msgr2=0x7f95601098f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:25.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.613+0000 7f9566c4e640 1 --2- 172.21.15.40:0/1277788189 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9560107500 0x7f95601098f0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f9558009ca0 tx=0x7f955802c0f0 comp rx=0 tx=0).stop 2024-04-21T23:18:25.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.613+0000 7f9566c4e640 1 -- 172.21.15.40:0/1277788189 shutdown_connections 2024-04-21T23:18:25.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.613+0000 7f9566c4e640 1 --2- 172.21.15.40:0/1277788189 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9560104bd0 0x7f9560106fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:25.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.613+0000 7f9566c4e640 1 --2- 172.21.15.40:0/1277788189 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9560107500 0x7f95601098f0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:25.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.613+0000 7f9566c4e640 1 --2- 172.21.15.40:0/1277788189 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f95601022a0 0x7f9560104690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:25.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.613+0000 7f9566c4e640 1 -- 172.21.15.40:0/1277788189 >> 172.21.15.40:0/1277788189 conn(0x7f95600fbbf0 msgr2=0x7f95600fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:25.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.613+0000 7f9566c4e640 1 -- 172.21.15.40:0/1277788189 shutdown_connections 2024-04-21T23:18:25.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.613+0000 7f9566c4e640 1 -- 172.21.15.40:0/1277788189 wait complete. 2024-04-21T23:18:25.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.613+0000 7f9566c4e640 1 Processor -- start 2024-04-21T23:18:25.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.613+0000 7f9566c4e640 1 -- start start 2024-04-21T23:18:25.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.614+0000 7f9566c4e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95601022a0 0x7f9560197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:25.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.614+0000 7f9566c4e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9560104bd0 0x7f9560197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:25.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.614+0000 7f9566c4e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9560107500 0x7f956019f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:25.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.614+0000 7f9566c4e640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f956010d420 con 0x7f9560104bd0 2024-04-21T23:18:25.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.614+0000 7f9566c4e640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f956010d2a0 con 0x7f95601022a0 2024-04-21T23:18:25.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.614+0000 7f9566c4e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f956010d5a0 con 0x7f9560107500 2024-04-21T23:18:25.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.614+0000 7f955ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9560104bd0 0x7f9560197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:25.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.614+0000 7f955ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9560104bd0 0x7f9560197fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60874/0 (socket says 172.21.15.40:60874) 2024-04-21T23:18:25.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.614+0000 7f955ffff640 1 -- 172.21.15.40:0/2007241096 learned_addr learned my addr 172.21.15.40:0/2007241096 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:25.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.614+0000 7f95649ea640 1 --2- 172.21.15.40:0/2007241096 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95601022a0 0x7f9560197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:25.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.614+0000 7f95651eb640 1 --2- 172.21.15.40:0/2007241096 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9560107500 0x7f956019f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:25.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.614+0000 7f955ffff640 1 -- 172.21.15.40:0/2007241096 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9560107500 msgr2=0x7f956019f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:25.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.614+0000 7f955ffff640 1 --2- 172.21.15.40:0/2007241096 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9560107500 0x7f956019f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:25.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.614+0000 7f955ffff640 1 -- 172.21.15.40:0/2007241096 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95601022a0 msgr2=0x7f9560197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:25.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.614+0000 7f955ffff640 1 --2- 172.21.15.40:0/2007241096 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95601022a0 0x7f9560197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:25.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.614+0000 7f955ffff640 1 -- 172.21.15.40:0/2007241096 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f956019f6b0 con 0x7f9560104bd0 2024-04-21T23:18:25.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.614+0000 7f955ffff640 1 --2- 172.21.15.40:0/2007241096 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9560104bd0 0x7f9560197fd0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f954c00e9c0 tx=0x7f954c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:25.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.614+0000 7f955dffb640 1 -- 172.21.15.40:0/2007241096 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f954c00cd80 con 0x7f9560104bd0 2024-04-21T23:18:25.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.614+0000 7f9566c4e640 1 -- 172.21.15.40:0/2007241096 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f956019f9a0 con 0x7f9560104bd0 2024-04-21T23:18:25.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.614+0000 7f955dffb640 1 -- 172.21.15.40:0/2007241096 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f954c005500 con 0x7f9560104bd0 2024-04-21T23:18:25.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.614+0000 7f955dffb640 1 -- 172.21.15.40:0/2007241096 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f954c010690 con 0x7f9560104bd0 2024-04-21T23:18:25.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.615+0000 7f9566c4e640 1 -- 172.21.15.40:0/2007241096 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f956019ff40 con 0x7f9560104bd0 2024-04-21T23:18:25.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.616+0000 7f9566c4e640 1 -- 172.21.15.40:0/2007241096 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95600fd6e0 con 0x7f9560104bd0 2024-04-21T23:18:25.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.617+0000 7f955dffb640 1 -- 172.21.15.40:0/2007241096 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 22) ==== 137645+0+0 (secure 0 0 0) 0x7f954c0046c0 con 0x7f9560104bd0 2024-04-21T23:18:25.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.617+0000 7f955dffb640 1 --2- 172.21.15.40:0/2007241096 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f95500a2990 0x7f95500a4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:25.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.617+0000 7f955dffb640 1 -- 172.21.15.40:0/2007241096 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f954c0cf4b0 con 0x7f9560104bd0 2024-04-21T23:18:25.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.617+0000 7f95649ea640 1 --2- 172.21.15.40:0/2007241096 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f95500a2990 0x7f95500a4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:25.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.618+0000 7f95649ea640 1 --2- 172.21.15.40:0/2007241096 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f95500a2990 0x7f95500a4e50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9554009bd0 tx=0x7f9554005f90 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:25.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.622+0000 7f955dffb640 1 -- 172.21.15.40:0/2007241096 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f954c09c230 con 0x7f9560104bd0 2024-04-21T23:18:25.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.817+0000 7f9566c4e640 1 -- 172.21.15.40:0/2007241096 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f9560062340 con 0x7f9560104bd0 2024-04-21T23:18:25.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.818+0000 7f955dffb640 1 -- 172.21.15.40:0/2007241096 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v95) ==== 100+0+11 (secure 0 0 0) 0x7f954c09bb30 con 0x7f9560104bd0 2024-04-21T23:18:25.817 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-04-21T23:18:25.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.820+0000 7f9566c4e640 1 -- 172.21.15.40:0/2007241096 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f95500a2990 msgr2=0x7f95500a4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:25.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.820+0000 7f9566c4e640 1 --2- 172.21.15.40:0/2007241096 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f95500a2990 0x7f95500a4e50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9554009bd0 tx=0x7f9554005f90 comp rx=0 tx=0).stop 2024-04-21T23:18:25.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.820+0000 7f9566c4e640 1 -- 172.21.15.40:0/2007241096 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9560104bd0 msgr2=0x7f9560197fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:25.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.820+0000 7f9566c4e640 1 --2- 172.21.15.40:0/2007241096 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9560104bd0 0x7f9560197fd0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f954c00e9c0 tx=0x7f954c00ee80 comp rx=0 tx=0).stop 2024-04-21T23:18:25.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.820+0000 7f9566c4e640 1 -- 172.21.15.40:0/2007241096 shutdown_connections 2024-04-21T23:18:25.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.820+0000 7f9566c4e640 1 --2- 172.21.15.40:0/2007241096 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f95500a2990 0x7f95500a4e50 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:25.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.820+0000 7f9566c4e640 1 --2- 172.21.15.40:0/2007241096 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9560107500 0x7f956019f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:25.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.820+0000 7f9566c4e640 1 --2- 172.21.15.40:0/2007241096 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9560104bd0 0x7f9560197fd0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:25.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.820+0000 7f9566c4e640 1 --2- 172.21.15.40:0/2007241096 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95601022a0 0x7f9560197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:25.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.820+0000 7f9566c4e640 1 -- 172.21.15.40:0/2007241096 >> 172.21.15.40:0/2007241096 conn(0x7f95600fbbf0 msgr2=0x7f9560102d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:25.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.820+0000 7f9566c4e640 1 -- 172.21.15.40:0/2007241096 shutdown_connections 2024-04-21T23:18:25.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.820+0000 7f9566c4e640 1 -- 172.21.15.40:0/2007241096 wait complete. 2024-04-21T23:18:25.831 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T23:18:25.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.939+0000 7fd1a587d640 1 -- 172.21.15.40:0/2276764125 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1a0104b30 msgr2=0x7fd1a0106f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:25.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.939+0000 7fd1a587d640 1 --2- 172.21.15.40:0/2276764125 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1a0104b30 0x7fd1a0106f20 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fd188009900 tx=0x7fd18802bc30 comp rx=0 tx=0).stop 2024-04-21T23:18:25.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.939+0000 7fd1a587d640 1 -- 172.21.15.40:0/2276764125 shutdown_connections 2024-04-21T23:18:25.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.939+0000 7fd1a587d640 1 --2- 172.21.15.40:0/2276764125 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1a0107460 0x7fd1a01098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:25.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.939+0000 7fd1a587d640 1 --2- 172.21.15.40:0/2276764125 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1a0104b30 0x7fd1a0106f20 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:25.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.939+0000 7fd1a587d640 1 --2- 172.21.15.40:0/2276764125 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd1a0102200 0x7fd1a01045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:25.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.939+0000 7fd1a587d640 1 -- 172.21.15.40:0/2276764125 >> 172.21.15.40:0/2276764125 conn(0x7fd1a00fbbf0 msgr2=0x7fd1a00fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:25.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.939+0000 7fd1a587d640 1 -- 172.21.15.40:0/2276764125 shutdown_connections 2024-04-21T23:18:25.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.939+0000 7fd1a587d640 1 -- 172.21.15.40:0/2276764125 wait complete. 2024-04-21T23:18:25.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.939+0000 7fd1a587d640 1 Processor -- start 2024-04-21T23:18:25.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.939+0000 7fd1a587d640 1 -- start start 2024-04-21T23:18:25.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.940+0000 7fd1a587d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd1a0102200 0x7fd1a0197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:25.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.940+0000 7fd1a587d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1a0104b30 0x7fd1a0198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:25.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.940+0000 7fd1a587d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1a0107460 0x7fd1a019f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:25.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.940+0000 7fd1a587d640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd1a010d410 con 0x7fd1a0107460 2024-04-21T23:18:25.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.940+0000 7fd1a587d640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd1a010d290 con 0x7fd1a0104b30 2024-04-21T23:18:25.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.940+0000 7fd1a587d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd1a010d590 con 0x7fd1a0102200 2024-04-21T23:18:25.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.940+0000 7fd19f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1a0107460 0x7fd1a019f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:25.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.940+0000 7fd19f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1a0107460 0x7fd1a019f0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60898/0 (socket says 172.21.15.40:60898) 2024-04-21T23:18:25.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.940+0000 7fd19f7fe640 1 -- 172.21.15.40:0/266022039 learned_addr learned my addr 172.21.15.40:0/266022039 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:25.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.940+0000 7fd19effd640 1 --2- 172.21.15.40:0/266022039 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd1a0102200 0x7fd1a0197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:25.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.940+0000 7fd197fff640 1 --2- 172.21.15.40:0/266022039 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1a0104b30 0x7fd1a0198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:25.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.940+0000 7fd197fff640 1 -- 172.21.15.40:0/266022039 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd1a0102200 msgr2=0x7fd1a0197b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:25.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.940+0000 7fd197fff640 1 --2- 172.21.15.40:0/266022039 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd1a0102200 0x7fd1a0197b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:25.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.940+0000 7fd197fff640 1 -- 172.21.15.40:0/266022039 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1a0107460 msgr2=0x7fd1a019f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:25.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.940+0000 7fd197fff640 1 --2- 172.21.15.40:0/266022039 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1a0107460 0x7fd1a019f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:25.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.940+0000 7fd197fff640 1 -- 172.21.15.40:0/266022039 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1a019f730 con 0x7fd1a0104b30 2024-04-21T23:18:25.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.940+0000 7fd19f7fe640 1 --2- 172.21.15.40:0/266022039 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1a0107460 0x7fd1a019f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:18:25.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.940+0000 7fd197fff640 1 --2- 172.21.15.40:0/266022039 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1a0104b30 0x7fd1a0198050 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd18802c170 tx=0x7fd18802c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:25.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.940+0000 7fd1a487b640 1 -- 172.21.15.40:0/266022039 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd188041070 con 0x7fd1a0104b30 2024-04-21T23:18:25.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.941+0000 7fd1a587d640 1 -- 172.21.15.40:0/266022039 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1a019f9c0 con 0x7fd1a0104b30 2024-04-21T23:18:25.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.941+0000 7fd1a487b640 1 -- 172.21.15.40:0/266022039 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd188004450 con 0x7fd1a0104b30 2024-04-21T23:18:25.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.941+0000 7fd1a487b640 1 -- 172.21.15.40:0/266022039 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd188003e50 con 0x7fd1a0104b30 2024-04-21T23:18:25.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.941+0000 7fd1a587d640 1 -- 172.21.15.40:0/266022039 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1a019fe80 con 0x7fd1a0104b30 2024-04-21T23:18:25.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.942+0000 7fd1a587d640 1 -- 172.21.15.40:0/266022039 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1a00fd6e0 con 0x7fd1a0104b30 2024-04-21T23:18:25.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.943+0000 7fd1a487b640 1 -- 172.21.15.40:0/266022039 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 22) ==== 137645+0+0 (secure 0 0 0) 0x7fd188044070 con 0x7fd1a0104b30 2024-04-21T23:18:25.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.943+0000 7fd1a487b640 1 --2- 172.21.15.40:0/266022039 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fd18c0a2b30 0x7fd18c0a4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:25.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.943+0000 7fd1a487b640 1 -- 172.21.15.40:0/266022039 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7fd1880f0d70 con 0x7fd1a0104b30 2024-04-21T23:18:25.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.943+0000 7fd19effd640 1 --2- 172.21.15.40:0/266022039 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fd18c0a2b30 0x7fd18c0a4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:25.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.944+0000 7fd19effd640 1 --2- 172.21.15.40:0/266022039 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fd18c0a2b30 0x7fd18c0a4ff0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd190009bd0 tx=0x7fd190005f90 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:25.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.947+0000 7fd1a487b640 1 -- 172.21.15.40:0/266022039 <== mon.2 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fd1880bd4a0 con 0x7fd1a0104b30 2024-04-21T23:18:25.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:25.970+0000 7fd1a487b640 1 -- 172.21.15.40:0/266022039 <== mon.2 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 23) ==== 138004+0+0 (secure 0 0 0) 0x7fd1880035f0 con 0x7fd1a0104b30 2024-04-21T23:18:26.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.143+0000 7fd1a587d640 1 -- 172.21.15.40:0/266022039 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fd1a002d3c0 con 0x7fd1a0104b30 2024-04-21T23:18:26.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.144+0000 7fd1a487b640 1 -- 172.21.15.40:0/266022039 <== mon.2 v2:172.21.15.111:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v95) ==== 103+0+11 (secure 0 0 0) 0x7fd1880a9690 con 0x7fd1a0104b30 2024-04-21T23:18:26.143 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-21T23:18:26.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.146+0000 7fd1a587d640 1 -- 172.21.15.40:0/266022039 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fd18c0a2b30 msgr2=0x7fd18c0a4ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:26.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.146+0000 7fd1a587d640 1 --2- 172.21.15.40:0/266022039 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fd18c0a2b30 0x7fd18c0a4ff0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd190009bd0 tx=0x7fd190005f90 comp rx=0 tx=0).stop 2024-04-21T23:18:26.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.146+0000 7fd1a587d640 1 -- 172.21.15.40:0/266022039 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1a0104b30 msgr2=0x7fd1a0198050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:26.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.146+0000 7fd1a587d640 1 --2- 172.21.15.40:0/266022039 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1a0104b30 0x7fd1a0198050 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd18802c170 tx=0x7fd18802c650 comp rx=0 tx=0).stop 2024-04-21T23:18:26.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.146+0000 7fd1a587d640 1 -- 172.21.15.40:0/266022039 shutdown_connections 2024-04-21T23:18:26.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.146+0000 7fd1a587d640 1 --2- 172.21.15.40:0/266022039 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fd18c0a2b30 0x7fd18c0a4ff0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:26.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.146+0000 7fd1a587d640 1 --2- 172.21.15.40:0/266022039 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1a0104b30 0x7fd1a0198050 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:26.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.146+0000 7fd1a587d640 1 --2- 172.21.15.40:0/266022039 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1a0107460 0x7fd1a019f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:26.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.146+0000 7fd1a587d640 1 --2- 172.21.15.40:0/266022039 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd1a0102200 0x7fd1a0197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:26.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.146+0000 7fd1a587d640 1 -- 172.21.15.40:0/266022039 >> 172.21.15.40:0/266022039 conn(0x7fd1a00fbbf0 msgr2=0x7fd1a0102d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:26.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.146+0000 7fd1a587d640 1 -- 172.21.15.40:0/266022039 shutdown_connections 2024-04-21T23:18:26.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.146+0000 7fd1a587d640 1 -- 172.21.15.40:0/266022039 wait complete. 2024-04-21T23:18:26.155 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:25 smithi040 ceph-mon[27984]: [21/Apr/2024:23:18:24] ENGINE Serving on http://172.21.15.40:8765 2024-04-21T23:18:26.155 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:25 smithi040 ceph-mon[27984]: [21/Apr/2024:23:18:24] ENGINE Bus STARTED 2024-04-21T23:18:26.155 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:25 smithi040 ceph-mon[27984]: from='client.15489 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:18:26.156 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:25 smithi040 ceph-mon[27984]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:18:26.156 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:25 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/1884359250' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:18:26.156 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:25 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/4032454917' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T23:18:26.156 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:25 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:18:26.156 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:25 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:18:26.156 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:25 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:18:26.156 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:25 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:18:26.156 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:25 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:18:26.156 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:25 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:18:26.156 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:25 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/2007241096' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T23:18:26.158 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T23:18:26.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.276+0000 7f967a639640 1 -- 172.21.15.40:0/1787845419 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9674101870 msgr2=0x7f9674101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:26.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.276+0000 7f967a639640 1 --2- 172.21.15.40:0/1787845419 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9674101870 0x7f9674101cf0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f9668009900 tx=0x7f966802bc30 comp rx=0 tx=0).stop 2024-04-21T23:18:26.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.276+0000 7f967a639640 1 -- 172.21.15.40:0/1787845419 shutdown_connections 2024-04-21T23:18:26.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.276+0000 7f967a639640 1 --2- 172.21.15.40:0/1787845419 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9674102230 0x7f9674108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:26.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.276+0000 7f967a639640 1 --2- 172.21.15.40:0/1787845419 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9674101870 0x7f9674101cf0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:26.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.276+0000 7f967a639640 1 --2- 172.21.15.40:0/1787845419 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9674100680 0x7f9674100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:26.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.276+0000 7f967a639640 1 -- 172.21.15.40:0/1787845419 >> 172.21.15.40:0/1787845419 conn(0x7f96740fbbf0 msgr2=0x7f96740fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:26.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.276+0000 7f967a639640 1 -- 172.21.15.40:0/1787845419 shutdown_connections 2024-04-21T23:18:26.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.276+0000 7f967a639640 1 -- 172.21.15.40:0/1787845419 wait complete. 2024-04-21T23:18:26.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.276+0000 7f967a639640 1 Processor -- start 2024-04-21T23:18:26.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.276+0000 7f967a639640 1 -- start start 2024-04-21T23:18:26.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.277+0000 7f967a639640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9674100680 0x7f9674069460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:26.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.277+0000 7f967a639640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9674101870 0x7f967419c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:26.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.277+0000 7f967a639640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9674102230 0x7f967419ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:26.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.277+0000 7f967a639640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f967410c2e0 con 0x7f9674102230 2024-04-21T23:18:26.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.277+0000 7f967a639640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f967410c160 con 0x7f9674101870 2024-04-21T23:18:26.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.277+0000 7f967a639640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f967410c460 con 0x7f9674100680 2024-04-21T23:18:26.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.277+0000 7f9678bd6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9674102230 0x7f967419ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:26.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.277+0000 7f9678bd6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9674102230 0x7f967419ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60918/0 (socket says 172.21.15.40:60918) 2024-04-21T23:18:26.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.277+0000 7f9678bd6640 1 -- 172.21.15.40:0/2893086835 learned_addr learned my addr 172.21.15.40:0/2893086835 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:26.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.277+0000 7f96737fe640 1 --2- 172.21.15.40:0/2893086835 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9674101870 0x7f967419c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:26.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.277+0000 7f9673fff640 1 --2- 172.21.15.40:0/2893086835 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9674100680 0x7f9674069460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:26.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.277+0000 7f9678bd6640 1 -- 172.21.15.40:0/2893086835 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9674100680 msgr2=0x7f9674069460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:26.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.277+0000 7f9678bd6640 1 --2- 172.21.15.40:0/2893086835 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9674100680 0x7f9674069460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:26.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.277+0000 7f9678bd6640 1 -- 172.21.15.40:0/2893086835 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9674101870 msgr2=0x7f967419c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:26.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.277+0000 7f9678bd6640 1 --2- 172.21.15.40:0/2893086835 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9674101870 0x7f967419c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:26.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.277+0000 7f9678bd6640 1 -- 172.21.15.40:0/2893086835 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f967419f6d0 con 0x7f9674102230 2024-04-21T23:18:26.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.277+0000 7f9678bd6640 1 --2- 172.21.15.40:0/2893086835 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9674102230 0x7f967419ee10 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f966c00eb10 tx=0x7f966c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:26.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.277+0000 7f96717fa640 1 -- 172.21.15.40:0/2893086835 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f966c004b20 con 0x7f9674102230 2024-04-21T23:18:26.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.277+0000 7f96717fa640 1 -- 172.21.15.40:0/2893086835 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f966c004cc0 con 0x7f9674102230 2024-04-21T23:18:26.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.277+0000 7f967a639640 1 -- 172.21.15.40:0/2893086835 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f967419f9c0 con 0x7f9674102230 2024-04-21T23:18:26.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.278+0000 7f96717fa640 1 -- 172.21.15.40:0/2893086835 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f966c010760 con 0x7f9674102230 2024-04-21T23:18:26.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.278+0000 7f967a639640 1 -- 172.21.15.40:0/2893086835 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f967419fe90 con 0x7f9674102230 2024-04-21T23:18:26.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.279+0000 7f967a639640 1 -- 172.21.15.40:0/2893086835 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96741079a0 con 0x7f9674102230 2024-04-21T23:18:26.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.280+0000 7f96717fa640 1 -- 172.21.15.40:0/2893086835 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 23) ==== 138004+0+0 (secure 0 0 0) 0x7f966c005020 con 0x7f9674102230 2024-04-21T23:18:26.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.280+0000 7f96717fa640 1 --2- 172.21.15.40:0/2893086835 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f965c0a2d90 0x7f965c0a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:26.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.281+0000 7f9673fff640 1 --2- 172.21.15.40:0/2893086835 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f965c0a2d90 0x7f965c0a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:26.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.281+0000 7f96717fa640 1 -- 172.21.15.40:0/2893086835 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f966c0cfe90 con 0x7f9674102230 2024-04-21T23:18:26.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.281+0000 7f9673fff640 1 --2- 172.21.15.40:0/2893086835 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f965c0a2d90 0x7f965c0a5250 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9664005fd0 tx=0x7f966400b410 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:26.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.285+0000 7f96717fa640 1 -- 172.21.15.40:0/2893086835 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f966c09c590 con 0x7f9674102230 2024-04-21T23:18:26.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:25 smithi174 ceph-mon[31409]: [21/Apr/2024:23:18:24] ENGINE Serving on http://172.21.15.40:8765 2024-04-21T23:18:26.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:25 smithi174 ceph-mon[31409]: [21/Apr/2024:23:18:24] ENGINE Bus STARTED 2024-04-21T23:18:26.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:25 smithi174 ceph-mon[31409]: from='client.15489 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:18:26.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:25 smithi174 ceph-mon[31409]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:18:26.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:25 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/1884359250' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:18:26.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:25 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/4032454917' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T23:18:26.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:25 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:18:26.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:25 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:18:26.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:25 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:18:26.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:25 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:18:26.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:25 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:18:26.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:25 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:18:26.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:25 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/2007241096' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T23:18:26.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:25 smithi111 ceph-mon[31085]: [21/Apr/2024:23:18:24] ENGINE Serving on http://172.21.15.40:8765 2024-04-21T23:18:26.446 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:25 smithi111 ceph-mon[31085]: [21/Apr/2024:23:18:24] ENGINE Bus STARTED 2024-04-21T23:18:26.446 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:25 smithi111 ceph-mon[31085]: from='client.15489 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:18:26.446 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:25 smithi111 ceph-mon[31085]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:18:26.446 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:25 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/1884359250' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:18:26.446 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:25 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/4032454917' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T23:18:26.446 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:25 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:18:26.446 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:25 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:18:26.447 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:25 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:18:26.447 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:25 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:18:26.447 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:25 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:18:26.447 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:25 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:18:26.447 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:25 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/2007241096' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T23:18:26.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.471+0000 7f967a639640 1 -- 172.21.15.40:0/2893086835 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f967402d1c0 con 0x7f9674102230 2024-04-21T23:18:26.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.473+0000 7f96717fa640 1 -- 172.21.15.40:0/2893086835 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) ==== 74+0+25231 (secure 0 0 0) 0x7f966c09be90 con 0x7f9674102230 2024-04-21T23:18:26.473 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:18:26.473 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":95,"fsid":"d814e4dc-0033-11ef-bc93-c7b262605968","created":"2024-04-21T23:07:52.659207+0000","modified":"2024-04-21T23:18:23.389706+0000","last_up_change":"2024-04-21T23:15:05.308821+0000","last_in_change":"2024-04-21T23:14:45.447491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:11:14.824775+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T23:15:54.322571+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T23:15:55.164493+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T23:15:58.838266+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"0e8d692a-22fe-4a87-8d99-9151f96cbb02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6803","nonce":2009758784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6805","nonce":2009758784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6809","nonce":2009758784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6807","nonce":2009758784}]},"public_addr":"172.21.15.40:6803/2009758784","cluster_addr":"172.21.15.40:6805/2009758784","heartbeat_back_addr":"172.21.15.40:6809/2009758784","heartbeat_front_addr":"172.21.15.40:6807/2009758784","state":["exists","up"]},{"osd":1,"uuid":"fe5fe11b-0e84-4359-83a7-1ba45105755a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6801","nonce":1763903836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6803","nonce":1763903836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6807","nonce":1763903836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6805","nonce":1763903836}]},"public_addr":"172.21.15.111:6801/1763903836","cluster_addr":"172.21.15.111:6803/1763903836","heartbeat_back_addr":"172.21.15.111:6807/1763903836","heartbeat_front_addr":"172.21.15.111:6805/1763903836","state":["exists","up"]},{"osd":2,"uuid":"813ad735-5cc2-4da0-ae16-6db238395fbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6801","nonce":439695825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6803","nonce":439695825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6807","nonce":439695825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6805","nonce":439695825}]},"public_addr":"172.21.15.174:6801/439695825","cluster_addr":"172.21.15.174:6803/439695825","heartbeat_back_addr":"172.21.15.174:6807/439695825","heartbeat_front_addr":"172.21.15.174:6805/439695825","state":["exists","up"]},{"osd":3,"uuid":"09becb03-b7ce-4ffc-917d-b7900ec37e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6811","nonce":3454312436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6813","nonce":3454312436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6817","nonce":3454312436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6815","nonce":3454312436}]},"public_addr":"172.21.15.40:6811/3454312436","cluster_addr":"172.21.15.40:6813/3454312436","heartbeat_back_addr":"172.21.15.40:6817/3454312436","heartbeat_front_addr":"172.21.15.40:6815/3454312436","state":["exists","up"]},{"osd":4,"uuid":"46eabfd7-d88a-4efd-94eb-04354d3a9d7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6809","nonce":1810866070}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6811","nonce":1810866070}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6815","nonce":1810866070}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6813","nonce":1810866070}]},"public_addr":"172.21.15.111:6809/1810866070","cluster_addr":"172.21.15.111:6811/1810866070","heartbeat_back_addr":"172.21.15.111:6815/1810866070","heartbeat_front_addr":"172.21.15.111:6813/1810866070","state":["exists","up"]},{"osd":5,"uuid":"6f73b582-6c65-41db-aab0-9b360f83d5c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6809","nonce":3752986711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6811","nonce":3752986711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6815","nonce":3752986711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6813","nonce":3752986711}]},"public_addr":"172.21.15.174:6809/3752986711","cluster_addr":"172.21.15.174:6811/3752986711","heartbeat_back_addr":"172.21.15.174:6815/3752986711","heartbeat_front_addr":"172.21.15.174:6813/3752986711","state":["exists","up"]},{"osd":6,"uuid":"39f305f0-12d2-4732-aaa0-cd66700af538","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6819","nonce":1618855427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6821","nonce":1618855427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6825","nonce":1618855427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6823","nonce":1618855427}]},"public_addr":"172.21.15.40:6819/1618855427","cluster_addr":"172.21.15.40:6821/1618855427","heartbeat_back_addr":"172.21.15.40:6825/1618855427","heartbeat_front_addr":"172.21.15.40:6823/1618855427","state":["exists","up"]},{"osd":7,"uuid":"769f2468-5102-490b-89f8-d0444d3169a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6817","nonce":927054162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6819","nonce":927054162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6823","nonce":927054162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6821","nonce":927054162}]},"public_addr":"172.21.15.111:6817/927054162","cluster_addr":"172.21.15.111:6819/927054162","heartbeat_back_addr":"172.21.15.111:6823/927054162","heartbeat_front_addr":"172.21.15.111:6821/927054162","state":["exists","up"]},{"osd":8,"uuid":"1a444e80-007d-473b-b972-4c814a813c22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6817","nonce":3817848746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6819","nonce":3817848746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6823","nonce":3817848746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6821","nonce":3817848746}]},"public_addr":"172.21.15.174:6817/3817848746","cluster_addr":"172.21.15.174:6819/3817848746","heartbeat_back_addr":"172.21.15.174:6823/3817848746","heartbeat_front_addr":"172.21.15.174:6821/3817848746","state":["exists","up"]},{"osd":9,"uuid":"9a6b6a56-6ae5-49fe-9e5b-cb17d686e0dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6827","nonce":1649446531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6829","nonce":1649446531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6833","nonce":1649446531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6831","nonce":1649446531}]},"public_addr":"172.21.15.40:6827/1649446531","cluster_addr":"172.21.15.40:6829/1649446531","heartbeat_back_addr":"172.21.15.40:6833/1649446531","heartbeat_front_addr":"172.21.15.40:6831/1649446531","state":["exists","up"]},{"osd":10,"uuid":"a70038ba-e2ad-4170-85cc-ea0470a2edef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6825","nonce":1227538935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6827","nonce":1227538935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6831","nonce":1227538935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6829","nonce":1227538935}]},"public_addr":"172.21.15.111:6825/1227538935","cluster_addr":"172.21.15.111:6827/1227538935","heartbeat_back_addr":"172.21.15.111:6831/1227538935","heartbeat_front_addr":"172.21.15.111:6829/1227538935","state":["exists","up"]},{"osd":11,"uuid":"781532bb-7a5e-44e3-8eb7-b584b0054666","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6825","nonce":2045865553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6827","nonce":2045865553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6831","nonce":2045865553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6829","nonce":2045865553}]},"public_addr":"172.21.15.174:6825/2045865553","cluster_addr":"172.21.15.174:6827/2045865553","heartbeat_back_addr":"172.21.15.174:6831/2045865553","heartbeat_front_addr":"172.21.15.174:6829/2045865553","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:10:24.761290+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:10:48.565428+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:11:12.307818+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:11:37.951529+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:02.832523+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:27.568879+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:52.997707+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:13:18.937274+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:13:44.708657+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:14:10.833855+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:14:37.078272+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:15:03.878076+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":1}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.2a","mappings":[{"from":3,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"4.9","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":10}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:6801/2500760788":"2024-04-22T23:18:23.389674+0000","172.21.15.40:6800/2500760788":"2024-04-22T23:18:23.389674+0000","172.21.15.40:0/2742002992":"2024-04-22T23:18:23.389674+0000","172.21.15.40:0/2494672676":"2024-04-22T23:18:23.389674+0000","172.21.15.40:6801/526554319":"2024-04-22T23:08:50.305770+0000","172.21.15.40:6800/526554319":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/3178763471":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/4045911364":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/276300080":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/3689649814":"2024-04-22T23:08:23.743030+0000","172.21.15.40:0/2241116715":"2024-04-22T23:08:23.743030+0000","172.21.15.40:0/1406242364":"2024-04-22T23:18:23.389674+0000","172.21.15.40:0/1737227818":"2024-04-22T23:18:23.389674+0000","172.21.15.40:6801/1479040326":"2024-04-22T23:08:23.743030+0000","172.21.15.40:6800/1479040326":"2024-04-22T23:08:23.743030+0000","172.21.15.40:0/681298099":"2024-04-22T23:08:23.743030+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T23:18:26.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.475+0000 7f967a639640 1 -- 172.21.15.40:0/2893086835 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f965c0a2d90 msgr2=0x7f965c0a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:26.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.475+0000 7f967a639640 1 --2- 172.21.15.40:0/2893086835 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f965c0a2d90 0x7f965c0a5250 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9664005fd0 tx=0x7f966400b410 comp rx=0 tx=0).stop 2024-04-21T23:18:26.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.475+0000 7f967a639640 1 -- 172.21.15.40:0/2893086835 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9674102230 msgr2=0x7f967419ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:26.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.475+0000 7f967a639640 1 --2- 172.21.15.40:0/2893086835 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9674102230 0x7f967419ee10 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f966c00eb10 tx=0x7f966c00c2d0 comp rx=0 tx=0).stop 2024-04-21T23:18:26.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.476+0000 7f967a639640 1 -- 172.21.15.40:0/2893086835 shutdown_connections 2024-04-21T23:18:26.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.476+0000 7f967a639640 1 --2- 172.21.15.40:0/2893086835 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f965c0a2d90 0x7f965c0a5250 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:26.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.476+0000 7f967a639640 1 --2- 172.21.15.40:0/2893086835 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9674101870 0x7f967419c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:26.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.476+0000 7f967a639640 1 --2- 172.21.15.40:0/2893086835 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9674102230 0x7f967419ee10 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:26.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.476+0000 7f967a639640 1 --2- 172.21.15.40:0/2893086835 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9674100680 0x7f9674069460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:26.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.476+0000 7f967a639640 1 -- 172.21.15.40:0/2893086835 >> 172.21.15.40:0/2893086835 conn(0x7f96740fbbf0 msgr2=0x7f9674104b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:26.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.476+0000 7f967a639640 1 -- 172.21.15.40:0/2893086835 shutdown_connections 2024-04-21T23:18:26.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.476+0000 7f967a639640 1 -- 172.21.15.40:0/2893086835 wait complete. 2024-04-21T23:18:26.487 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T23:11:14.824775+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T23:15:54.322571+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T23:15:55.164493+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T23:15:58.838266+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T23:18:26.488 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T23:18:26.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.596+0000 7fec2047d640 1 -- 172.21.15.40:0/2296414362 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec18104b30 msgr2=0x7fec18106f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:26.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.596+0000 7fec2047d640 1 --2- 172.21.15.40:0/2296414362 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec18104b30 0x7fec18106f20 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fec14009900 tx=0x7fec1402bc30 comp rx=0 tx=0).stop 2024-04-21T23:18:26.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.596+0000 7fec2047d640 1 -- 172.21.15.40:0/2296414362 shutdown_connections 2024-04-21T23:18:26.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.596+0000 7fec2047d640 1 --2- 172.21.15.40:0/2296414362 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fec18107460 0x7fec181098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:26.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.596+0000 7fec2047d640 1 --2- 172.21.15.40:0/2296414362 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec18104b30 0x7fec18106f20 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:26.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.596+0000 7fec2047d640 1 --2- 172.21.15.40:0/2296414362 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec18102200 0x7fec181045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:26.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.596+0000 7fec2047d640 1 -- 172.21.15.40:0/2296414362 >> 172.21.15.40:0/2296414362 conn(0x7fec180fbbf0 msgr2=0x7fec180fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:26.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.596+0000 7fec2047d640 1 -- 172.21.15.40:0/2296414362 shutdown_connections 2024-04-21T23:18:26.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.596+0000 7fec2047d640 1 -- 172.21.15.40:0/2296414362 wait complete. 2024-04-21T23:18:26.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.596+0000 7fec2047d640 1 Processor -- start 2024-04-21T23:18:26.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.596+0000 7fec2047d640 1 -- start start 2024-04-21T23:18:26.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.597+0000 7fec2047d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec18102200 0x7fec18101b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:26.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.597+0000 7fec2047d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec18104b30 0x7fec181001b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:26.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.597+0000 7fec2047d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fec18107460 0x7fec181006f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:26.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.597+0000 7fec2047d640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fec1810d450 con 0x7fec18102200 2024-04-21T23:18:26.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.597+0000 7fec2047d640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fec1810d2d0 con 0x7fec18104b30 2024-04-21T23:18:26.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.597+0000 7fec2047d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fec1810d5d0 con 0x7fec18107460 2024-04-21T23:18:26.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.597+0000 7fec1e219640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec18102200 0x7fec18101b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:26.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.597+0000 7fec1e219640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec18102200 0x7fec18101b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60924/0 (socket says 172.21.15.40:60924) 2024-04-21T23:18:26.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.597+0000 7fec1e219640 1 -- 172.21.15.40:0/3005973933 learned_addr learned my addr 172.21.15.40:0/3005973933 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:26.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.597+0000 7fec1da18640 1 --2- 172.21.15.40:0/3005973933 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec18104b30 0x7fec181001b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:26.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.597+0000 7fec1ea1a640 1 --2- 172.21.15.40:0/3005973933 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fec18107460 0x7fec181006f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:26.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.597+0000 7fec1e219640 1 -- 172.21.15.40:0/3005973933 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fec18107460 msgr2=0x7fec181006f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:26.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.597+0000 7fec1e219640 1 --2- 172.21.15.40:0/3005973933 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fec18107460 0x7fec181006f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:26.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.597+0000 7fec1e219640 1 -- 172.21.15.40:0/3005973933 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec18104b30 msgr2=0x7fec181001b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:26.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.597+0000 7fec1e219640 1 --2- 172.21.15.40:0/3005973933 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec18104b30 0x7fec181001b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:26.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.597+0000 7fec1e219640 1 -- 172.21.15.40:0/3005973933 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec18100f80 con 0x7fec18102200 2024-04-21T23:18:26.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.597+0000 7fec1da18640 1 --2- 172.21.15.40:0/3005973933 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec18104b30 0x7fec181001b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:18:26.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.597+0000 7fec1e219640 1 --2- 172.21.15.40:0/3005973933 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec18102200 0x7fec18101b70 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fec0800ea60 tx=0x7fec0800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:26.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.598+0000 7fec0f7fe640 1 -- 172.21.15.40:0/3005973933 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec0800cde0 con 0x7fec18102200 2024-04-21T23:18:26.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.598+0000 7fec0f7fe640 1 -- 172.21.15.40:0/3005973933 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec080035f0 con 0x7fec18102200 2024-04-21T23:18:26.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.598+0000 7fec2047d640 1 -- 172.21.15.40:0/3005973933 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec181a1c40 con 0x7fec18102200 2024-04-21T23:18:26.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.598+0000 7fec0f7fe640 1 -- 172.21.15.40:0/3005973933 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec08010670 con 0x7fec18102200 2024-04-21T23:18:26.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.598+0000 7fec2047d640 1 -- 172.21.15.40:0/3005973933 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec181a2160 con 0x7fec18102200 2024-04-21T23:18:26.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.599+0000 7fec2047d640 1 -- 172.21.15.40:0/3005973933 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec18068560 con 0x7fec18102200 2024-04-21T23:18:26.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.600+0000 7fec0f7fe640 1 -- 172.21.15.40:0/3005973933 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 23) ==== 138004+0+0 (secure 0 0 0) 0x7fec080037a0 con 0x7fec18102200 2024-04-21T23:18:26.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.600+0000 7fec0f7fe640 1 --2- 172.21.15.40:0/3005973933 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fec040a2d90 0x7fec040a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:26.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.601+0000 7fec1da18640 1 --2- 172.21.15.40:0/3005973933 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fec040a2d90 0x7fec040a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:26.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.601+0000 7fec0f7fe640 1 -- 172.21.15.40:0/3005973933 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7fec080cfd70 con 0x7fec18102200 2024-04-21T23:18:26.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.601+0000 7fec1da18640 1 --2- 172.21.15.40:0/3005973933 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fec040a2d90 0x7fec040a5250 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fec1402c170 tx=0x7fec1402df90 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:26.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.608+0000 7fec0f7fe640 1 -- 172.21.15.40:0/3005973933 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fec0809c500 con 0x7fec18102200 2024-04-21T23:18:26.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.833+0000 7fec2047d640 1 -- 172.21.15.40:0/3005973933 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fec1802d170 con 0x7fec18102200 2024-04-21T23:18:26.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.833+0000 7fec0f7fe640 1 -- 172.21.15.40:0/3005973933 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v95) ==== 93+0+10 (secure 0 0 0) 0x7fec1802d170 con 0x7fec18102200 2024-04-21T23:18:26.832 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-04-21T23:18:26.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.835+0000 7fec2047d640 1 -- 172.21.15.40:0/3005973933 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fec040a2d90 msgr2=0x7fec040a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:26.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.835+0000 7fec2047d640 1 --2- 172.21.15.40:0/3005973933 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fec040a2d90 0x7fec040a5250 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fec1402c170 tx=0x7fec1402df90 comp rx=0 tx=0).stop 2024-04-21T23:18:26.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.835+0000 7fec2047d640 1 -- 172.21.15.40:0/3005973933 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec18102200 msgr2=0x7fec18101b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:26.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.835+0000 7fec2047d640 1 --2- 172.21.15.40:0/3005973933 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec18102200 0x7fec18101b70 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fec0800ea60 tx=0x7fec0800ef20 comp rx=0 tx=0).stop 2024-04-21T23:18:26.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.835+0000 7fec2047d640 1 -- 172.21.15.40:0/3005973933 shutdown_connections 2024-04-21T23:18:26.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.835+0000 7fec2047d640 1 --2- 172.21.15.40:0/3005973933 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fec040a2d90 0x7fec040a5250 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:26.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.835+0000 7fec2047d640 1 --2- 172.21.15.40:0/3005973933 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec18104b30 0x7fec181001b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:26.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.835+0000 7fec2047d640 1 --2- 172.21.15.40:0/3005973933 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fec18107460 0x7fec181006f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:26.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.835+0000 7fec2047d640 1 --2- 172.21.15.40:0/3005973933 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec18102200 0x7fec18101b70 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:26.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.835+0000 7fec2047d640 1 -- 172.21.15.40:0/3005973933 >> 172.21.15.40:0/3005973933 conn(0x7fec180fbbf0 msgr2=0x7fec18102c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:26.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.835+0000 7fec2047d640 1 -- 172.21.15.40:0/3005973933 shutdown_connections 2024-04-21T23:18:26.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.835+0000 7fec2047d640 1 -- 172.21.15.40:0/3005973933 wait complete. 2024-04-21T23:18:26.849 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T23:18:26.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.957+0000 7f8edcf0c640 1 -- 172.21.15.40:0/1172414681 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ed81006a0 msgr2=0x7f8ed8100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:26.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.957+0000 7f8edcf0c640 1 --2- 172.21.15.40:0/1172414681 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ed81006a0 0x7f8ed8100ac0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f8ec8009900 tx=0x7f8ec802bc30 comp rx=0 tx=0).stop 2024-04-21T23:18:26.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.957+0000 7f8edcf0c640 1 -- 172.21.15.40:0/1172414681 shutdown_connections 2024-04-21T23:18:26.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.957+0000 7f8edcf0c640 1 --2- 172.21.15.40:0/1172414681 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ed8101890 0x7f8ed8101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:26.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.957+0000 7f8edcf0c640 1 --2- 172.21.15.40:0/1172414681 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ed8102250 0x7f8ed8108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:26.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.957+0000 7f8edcf0c640 1 --2- 172.21.15.40:0/1172414681 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ed81006a0 0x7f8ed8100ac0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:26.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.957+0000 7f8edcf0c640 1 -- 172.21.15.40:0/1172414681 >> 172.21.15.40:0/1172414681 conn(0x7f8ed80fbc10 msgr2=0x7f8ed80fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:26.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.957+0000 7f8edcf0c640 1 -- 172.21.15.40:0/1172414681 shutdown_connections 2024-04-21T23:18:26.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.957+0000 7f8edcf0c640 1 -- 172.21.15.40:0/1172414681 wait complete. 2024-04-21T23:18:26.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.958+0000 7f8edcf0c640 1 Processor -- start 2024-04-21T23:18:26.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.958+0000 7f8edcf0c640 1 -- start start 2024-04-21T23:18:26.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.958+0000 7f8edcf0c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ed81006a0 0x7f8ed8197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:26.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.958+0000 7f8edcf0c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ed8101890 0x7f8ed8198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:26.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.958+0000 7f8edcf0c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ed8102250 0x7f8ed819f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:26.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.958+0000 7f8edcf0c640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8ed810c340 con 0x7f8ed8102250 2024-04-21T23:18:26.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.958+0000 7f8edcf0c640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f8ed810c1c0 con 0x7f8ed81006a0 2024-04-21T23:18:26.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.958+0000 7f8edcf0c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8ed810c4c0 con 0x7f8ed8101890 2024-04-21T23:18:26.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.958+0000 7f8ed6d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ed8102250 0x7f8ed819f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:26.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.959+0000 7f8ecdd9b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ed8101890 0x7f8ed8198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:26.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.959+0000 7f8ed659c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ed81006a0 0x7f8ed8197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:26.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.959+0000 7f8ed6d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ed8102250 0x7f8ed819f0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60934/0 (socket says 172.21.15.40:60934) 2024-04-21T23:18:26.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.959+0000 7f8ed6d9d640 1 -- 172.21.15.40:0/584075735 learned_addr learned my addr 172.21.15.40:0/584075735 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:26.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.959+0000 7f8ecdd9b640 1 -- 172.21.15.40:0/584075735 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ed81006a0 msgr2=0x7f8ed8197b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:26.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.959+0000 7f8ecdd9b640 1 --2- 172.21.15.40:0/584075735 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ed81006a0 0x7f8ed8197b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:26.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.959+0000 7f8ecdd9b640 1 -- 172.21.15.40:0/584075735 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ed8102250 msgr2=0x7f8ed819f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:26.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.959+0000 7f8ecdd9b640 1 --2- 172.21.15.40:0/584075735 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ed8102250 0x7f8ed819f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:26.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.959+0000 7f8ecdd9b640 1 -- 172.21.15.40:0/584075735 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ed819f740 con 0x7f8ed8101890 2024-04-21T23:18:26.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.959+0000 7f8ed659c640 1 --2- 172.21.15.40:0/584075735 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ed81006a0 0x7f8ed8197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:18:26.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.959+0000 7f8ecdd9b640 1 --2- 172.21.15.40:0/584075735 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ed8101890 0x7f8ed8198060 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8ec000e9f0 tx=0x7f8ec000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:26.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.959+0000 7f8ecffff640 1 -- 172.21.15.40:0/584075735 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ec000ccc0 con 0x7f8ed8101890 2024-04-21T23:18:26.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.959+0000 7f8ecffff640 1 -- 172.21.15.40:0/584075735 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ec0010910 con 0x7f8ed8101890 2024-04-21T23:18:26.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.959+0000 7f8ecffff640 1 -- 172.21.15.40:0/584075735 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ec0018960 con 0x7f8ed8101890 2024-04-21T23:18:26.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.959+0000 7f8edcf0c640 1 -- 172.21.15.40:0/584075735 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ed819fa30 con 0x7f8ed8101890 2024-04-21T23:18:26.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.959+0000 7f8edcf0c640 1 -- 172.21.15.40:0/584075735 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ed819ffd0 con 0x7f8ed8101890 2024-04-21T23:18:26.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.961+0000 7f8edcf0c640 1 -- 172.21.15.40:0/584075735 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ed81a0260 con 0x7f8ed8101890 2024-04-21T23:18:26.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.963+0000 7f8ecffff640 1 -- 172.21.15.40:0/584075735 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 23) ==== 138004+0+0 (secure 0 0 0) 0x7f8ec0010ac0 con 0x7f8ed8101890 2024-04-21T23:18:26.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.963+0000 7f8ecffff640 1 --2- 172.21.15.40:0/584075735 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f8ec40a2e10 0x7f8ec40a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:26.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.963+0000 7f8ecffff640 1 -- 172.21.15.40:0/584075735 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f8ec00cff80 con 0x7f8ed8101890 2024-04-21T23:18:26.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.963+0000 7f8ed659c640 1 --2- 172.21.15.40:0/584075735 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f8ec40a2e10 0x7f8ec40a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:26.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.964+0000 7f8ed659c640 1 --2- 172.21.15.40:0/584075735 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f8ec40a2e10 0x7f8ec40a52d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f8ec802c170 tx=0x7f8ec802df90 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:26.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:26.966+0000 7f8ecffff640 1 -- 172.21.15.40:0/584075735 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f8ec009c030 con 0x7f8ed8101890 2024-04-21T23:18:27.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.157+0000 7f8edcf0c640 1 -- 172.21.15.40:0/584075735 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f8ed81a0430 con 0x7f8ed8101890 2024-04-21T23:18:27.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.158+0000 7f8ecffff640 1 -- 172.21.15.40:0/584075735 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v95) ==== 104+0+11 (secure 0 0 0) 0x7f8ec00a0330 con 0x7f8ed8101890 2024-04-21T23:18:27.157 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-21T23:18:27.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.160+0000 7f8edcf0c640 1 -- 172.21.15.40:0/584075735 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f8ec40a2e10 msgr2=0x7f8ec40a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:27.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.160+0000 7f8edcf0c640 1 --2- 172.21.15.40:0/584075735 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f8ec40a2e10 0x7f8ec40a52d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f8ec802c170 tx=0x7f8ec802df90 comp rx=0 tx=0).stop 2024-04-21T23:18:27.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.160+0000 7f8edcf0c640 1 -- 172.21.15.40:0/584075735 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ed8101890 msgr2=0x7f8ed8198060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:27.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.160+0000 7f8edcf0c640 1 --2- 172.21.15.40:0/584075735 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ed8101890 0x7f8ed8198060 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8ec000e9f0 tx=0x7f8ec000eeb0 comp rx=0 tx=0).stop 2024-04-21T23:18:27.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.160+0000 7f8edcf0c640 1 -- 172.21.15.40:0/584075735 shutdown_connections 2024-04-21T23:18:27.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.160+0000 7f8edcf0c640 1 --2- 172.21.15.40:0/584075735 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f8ec40a2e10 0x7f8ec40a52d0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:27.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.160+0000 7f8edcf0c640 1 --2- 172.21.15.40:0/584075735 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ed8102250 0x7f8ed819f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:27.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.160+0000 7f8edcf0c640 1 --2- 172.21.15.40:0/584075735 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ed8101890 0x7f8ed8198060 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:27.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.160+0000 7f8edcf0c640 1 --2- 172.21.15.40:0/584075735 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ed81006a0 0x7f8ed8197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:27.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.160+0000 7f8edcf0c640 1 -- 172.21.15.40:0/584075735 >> 172.21.15.40:0/584075735 conn(0x7f8ed80fbc10 msgr2=0x7f8ed80fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:27.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.160+0000 7f8edcf0c640 1 -- 172.21.15.40:0/584075735 shutdown_connections 2024-04-21T23:18:27.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.160+0000 7f8edcf0c640 1 -- 172.21.15.40:0/584075735 wait complete. 2024-04-21T23:18:27.170 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:26 smithi040 ceph-mon[27984]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:18:27.171 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:26 smithi040 ceph-mon[27984]: mgrmap e23: x(active, since 2s), standbys: y, z 2024-04-21T23:18:27.171 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:26 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/266022039' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T23:18:27.171 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:26 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/2893086835' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:18:27.171 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:26 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:18:27.171 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:26 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:18:27.171 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:26 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:18:27.171 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:26 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:18:27.171 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:26 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:18:27.171 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:26 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:18:27.171 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:26 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:18:27.171 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:26 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:18:27.172 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:26 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:18:27.172 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:26 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:18:27.172 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:26 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:18:27.172 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:26 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:18:27.172 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:26 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:18:27.172 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:26 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:18:27.172 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:26 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:18:27.172 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:26 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:18:27.172 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:26 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:18:27.172 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:26 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:18:27.172 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:26 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/3005973933' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:18:27.172 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:26 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:18:27.173 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:26 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:18:27.173 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T23:18:27.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.281+0000 7f4fe3f5b640 1 -- 172.21.15.40:0/2203798886 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fdc107460 msgr2=0x7f4fdc1098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:27.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.281+0000 7f4fe3f5b640 1 --2- 172.21.15.40:0/2203798886 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fdc107460 0x7f4fdc1098b0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f4fd4009ca0 tx=0x7f4fd402c0f0 comp rx=0 tx=0).stop 2024-04-21T23:18:27.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.281+0000 7f4fe3f5b640 1 -- 172.21.15.40:0/2203798886 shutdown_connections 2024-04-21T23:18:27.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.281+0000 7f4fe3f5b640 1 --2- 172.21.15.40:0/2203798886 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4fdc104b30 0x7f4fdc106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:27.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.281+0000 7f4fe3f5b640 1 --2- 172.21.15.40:0/2203798886 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fdc107460 0x7f4fdc1098b0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:27.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.281+0000 7f4fe3f5b640 1 --2- 172.21.15.40:0/2203798886 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4fdc102200 0x7f4fdc1045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:27.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.281+0000 7f4fe3f5b640 1 -- 172.21.15.40:0/2203798886 >> 172.21.15.40:0/2203798886 conn(0x7f4fdc0fbbf0 msgr2=0x7f4fdc0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:27.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.281+0000 7f4fe3f5b640 1 -- 172.21.15.40:0/2203798886 shutdown_connections 2024-04-21T23:18:27.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.281+0000 7f4fe3f5b640 1 -- 172.21.15.40:0/2203798886 wait complete. 2024-04-21T23:18:27.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.281+0000 7f4fe3f5b640 1 Processor -- start 2024-04-21T23:18:27.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.282+0000 7f4fe3f5b640 1 -- start start 2024-04-21T23:18:27.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.282+0000 7f4fe3f5b640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4fdc102200 0x7f4fdc101b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:27.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.282+0000 7f4fe3f5b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fdc104b30 0x7f4fdc1001b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:27.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.282+0000 7f4fe3f5b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4fdc107460 0x7f4fdc1006f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:27.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.282+0000 7f4fe3f5b640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4fdc10d420 con 0x7f4fdc104b30 2024-04-21T23:18:27.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.282+0000 7f4fe3f5b640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f4fdc10d2a0 con 0x7f4fdc102200 2024-04-21T23:18:27.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.282+0000 7f4fe3f5b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4fdc10d5a0 con 0x7f4fdc107460 2024-04-21T23:18:27.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.282+0000 7f4fe14f6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fdc104b30 0x7f4fdc1001b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:27.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.282+0000 7f4fe14f6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fdc104b30 0x7f4fdc1001b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60940/0 (socket says 172.21.15.40:60940) 2024-04-21T23:18:27.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.282+0000 7f4fe14f6640 1 -- 172.21.15.40:0/42623367 learned_addr learned my addr 172.21.15.40:0/42623367 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:27.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.282+0000 7f4fe1cf7640 1 --2- 172.21.15.40:0/42623367 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4fdc102200 0x7f4fdc101b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:27.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.282+0000 7f4fe24f8640 1 --2- 172.21.15.40:0/42623367 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4fdc107460 0x7f4fdc1006f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:27.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.282+0000 7f4fe14f6640 1 -- 172.21.15.40:0/42623367 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4fdc107460 msgr2=0x7f4fdc1006f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:27.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.282+0000 7f4fe14f6640 1 --2- 172.21.15.40:0/42623367 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4fdc107460 0x7f4fdc1006f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:27.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.282+0000 7f4fe14f6640 1 -- 172.21.15.40:0/42623367 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4fdc102200 msgr2=0x7f4fdc101b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:27.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.282+0000 7f4fe14f6640 1 --2- 172.21.15.40:0/42623367 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4fdc102200 0x7f4fdc101b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:27.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.282+0000 7f4fe14f6640 1 -- 172.21.15.40:0/42623367 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fdc100f80 con 0x7f4fdc104b30 2024-04-21T23:18:27.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.282+0000 7f4fe1cf7640 1 --2- 172.21.15.40:0/42623367 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4fdc102200 0x7f4fdc101b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:18:27.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.282+0000 7f4fe14f6640 1 --2- 172.21.15.40:0/42623367 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fdc104b30 0x7f4fdc1001b0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f4fcc00e9c0 tx=0x7f4fcc00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:27.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.283+0000 7f4fd2ffd640 1 -- 172.21.15.40:0/42623367 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fcc00cd80 con 0x7f4fdc104b30 2024-04-21T23:18:27.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.283+0000 7f4fe3f5b640 1 -- 172.21.15.40:0/42623367 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fdc1a1bb0 con 0x7f4fdc104b30 2024-04-21T23:18:27.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.283+0000 7f4fd2ffd640 1 -- 172.21.15.40:0/42623367 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4fcc005500 con 0x7f4fdc104b30 2024-04-21T23:18:27.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.283+0000 7f4fd2ffd640 1 -- 172.21.15.40:0/42623367 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fcc010690 con 0x7f4fdc104b30 2024-04-21T23:18:27.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.283+0000 7f4fe3f5b640 1 -- 172.21.15.40:0/42623367 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fdc1a2100 con 0x7f4fdc104b30 2024-04-21T23:18:27.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.284+0000 7f4fe3f5b640 1 -- 172.21.15.40:0/42623367 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4fdc0fd6e0 con 0x7f4fdc104b30 2024-04-21T23:18:27.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.285+0000 7f4fd2ffd640 1 -- 172.21.15.40:0/42623367 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 23) ==== 138004+0+0 (secure 0 0 0) 0x7f4fcc0046c0 con 0x7f4fdc104b30 2024-04-21T23:18:27.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.285+0000 7f4fd2ffd640 1 --2- 172.21.15.40:0/42623367 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f4fc80a2d90 0x7f4fc80a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:27.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.286+0000 7f4fe1cf7640 1 --2- 172.21.15.40:0/42623367 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f4fc80a2d90 0x7f4fc80a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:27.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.286+0000 7f4fd2ffd640 1 -- 172.21.15.40:0/42623367 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f4fcc0cfd80 con 0x7f4fdc104b30 2024-04-21T23:18:27.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.286+0000 7f4fe1cf7640 1 --2- 172.21.15.40:0/42623367 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f4fc80a2d90 0x7f4fc80a5250 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4fd8009fc0 tx=0x7f4fd8005f90 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:27.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.290+0000 7f4fd2ffd640 1 -- 172.21.15.40:0/42623367 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f4fcc09c480 con 0x7f4fdc104b30 2024-04-21T23:18:27.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:26 smithi111 ceph-mon[31085]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:18:27.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:26 smithi111 ceph-mon[31085]: mgrmap e23: x(active, since 2s), standbys: y, z 2024-04-21T23:18:27.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:26 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/266022039' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T23:18:27.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:26 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/2893086835' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:18:27.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:26 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:18:27.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:26 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:18:27.446 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:26 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:18:27.446 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:26 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:18:27.446 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:26 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:18:27.446 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:26 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:18:27.446 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:26 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:18:27.447 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:26 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:18:27.447 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:26 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:18:27.447 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:26 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:18:27.447 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:26 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:18:27.447 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:26 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:18:27.447 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:26 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:18:27.448 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:26 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:18:27.448 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:26 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:18:27.448 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:26 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:18:27.450 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:26 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:18:27.450 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:26 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:18:27.450 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:26 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/3005973933' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:18:27.450 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:26 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:18:27.450 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:26 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:18:27.450 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:26 smithi174 ceph-mon[31409]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:18:27.450 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:26 smithi174 ceph-mon[31409]: mgrmap e23: x(active, since 2s), standbys: y, z 2024-04-21T23:18:27.451 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:26 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/266022039' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T23:18:27.451 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:26 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/2893086835' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:18:27.451 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:26 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:18:27.451 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:26 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:18:27.451 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:26 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:18:27.451 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:26 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:18:27.451 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:26 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:18:27.451 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:26 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:18:27.452 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:26 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:18:27.452 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:26 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:18:27.452 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:26 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:18:27.452 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:26 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:18:27.452 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:26 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:18:27.452 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:26 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:18:27.453 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:26 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:18:27.453 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:26 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:18:27.453 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:26 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:18:27.453 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:26 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:18:27.453 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:26 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:18:27.454 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:26 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:18:27.454 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:26 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/3005973933' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:18:27.454 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:26 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:18:27.454 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:26 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:18:27.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.479+0000 7f4fe3f5b640 1 -- 172.21.15.40:0/42623367 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f4fdc02d340 con 0x7f4fdc104b30 2024-04-21T23:18:27.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.480+0000 7f4fd2ffd640 1 -- 172.21.15.40:0/42623367 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v95) ==== 100+0+11 (secure 0 0 0) 0x7f4fcc09bd80 con 0x7f4fdc104b30 2024-04-21T23:18:27.479 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-04-21T23:18:27.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.482+0000 7f4fe3f5b640 1 -- 172.21.15.40:0/42623367 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f4fc80a2d90 msgr2=0x7f4fc80a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:27.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.482+0000 7f4fe3f5b640 1 --2- 172.21.15.40:0/42623367 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f4fc80a2d90 0x7f4fc80a5250 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4fd8009fc0 tx=0x7f4fd8005f90 comp rx=0 tx=0).stop 2024-04-21T23:18:27.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.482+0000 7f4fe3f5b640 1 -- 172.21.15.40:0/42623367 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fdc104b30 msgr2=0x7f4fdc1001b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:27.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.482+0000 7f4fe3f5b640 1 --2- 172.21.15.40:0/42623367 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fdc104b30 0x7f4fdc1001b0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f4fcc00e9c0 tx=0x7f4fcc00ee80 comp rx=0 tx=0).stop 2024-04-21T23:18:27.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.482+0000 7f4fe3f5b640 1 -- 172.21.15.40:0/42623367 shutdown_connections 2024-04-21T23:18:27.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.482+0000 7f4fe3f5b640 1 --2- 172.21.15.40:0/42623367 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f4fc80a2d90 0x7f4fc80a5250 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:27.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.482+0000 7f4fe3f5b640 1 --2- 172.21.15.40:0/42623367 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4fdc107460 0x7f4fdc1006f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:27.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.482+0000 7f4fe3f5b640 1 --2- 172.21.15.40:0/42623367 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fdc104b30 0x7f4fdc1001b0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:27.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.482+0000 7f4fe3f5b640 1 --2- 172.21.15.40:0/42623367 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4fdc102200 0x7f4fdc101b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:27.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.482+0000 7f4fe3f5b640 1 -- 172.21.15.40:0/42623367 >> 172.21.15.40:0/42623367 conn(0x7f4fdc0fbbf0 msgr2=0x7f4fdc102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:27.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.483+0000 7f4fe3f5b640 1 -- 172.21.15.40:0/42623367 shutdown_connections 2024-04-21T23:18:27.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.483+0000 7f4fe3f5b640 1 -- 172.21.15.40:0/42623367 wait complete. 2024-04-21T23:18:27.494 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T23:18:27.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.602+0000 7f851cbf7640 1 -- 172.21.15.40:0/2146275737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8518102230 msgr2=0x7f8518108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:27.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.602+0000 7f851cbf7640 1 --2- 172.21.15.40:0/2146275737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8518102230 0x7f8518108ac0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f8510009ca0 tx=0x7f851002c0f0 comp rx=0 tx=0).stop 2024-04-21T23:18:27.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.602+0000 7f851cbf7640 1 -- 172.21.15.40:0/2146275737 shutdown_connections 2024-04-21T23:18:27.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.602+0000 7f851cbf7640 1 --2- 172.21.15.40:0/2146275737 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8518101870 0x7f8518101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:27.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.602+0000 7f851cbf7640 1 --2- 172.21.15.40:0/2146275737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8518102230 0x7f8518108ac0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:27.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.602+0000 7f851cbf7640 1 --2- 172.21.15.40:0/2146275737 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8518100680 0x7f8518100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:27.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.602+0000 7f851cbf7640 1 -- 172.21.15.40:0/2146275737 >> 172.21.15.40:0/2146275737 conn(0x7f85180fbbf0 msgr2=0x7f85180fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:27.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.602+0000 7f851cbf7640 1 -- 172.21.15.40:0/2146275737 shutdown_connections 2024-04-21T23:18:27.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.602+0000 7f851cbf7640 1 -- 172.21.15.40:0/2146275737 wait complete. 2024-04-21T23:18:27.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.603+0000 7f851cbf7640 1 Processor -- start 2024-04-21T23:18:27.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.603+0000 7f851cbf7640 1 -- start start 2024-04-21T23:18:27.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.603+0000 7f851cbf7640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8518100680 0x7f8518197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:27.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.603+0000 7f851cbf7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8518101870 0x7f8518198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:27.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.603+0000 7f851cbf7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8518102230 0x7f851819f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:27.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.603+0000 7f851cbf7640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f851810c300 con 0x7f8518101870 2024-04-21T23:18:27.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.603+0000 7f851cbf7640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f851810c180 con 0x7f8518100680 2024-04-21T23:18:27.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.603+0000 7f851cbf7640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f851810c480 con 0x7f8518102230 2024-04-21T23:18:27.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.603+0000 7f8515d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8518101870 0x7f8518198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:27.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.603+0000 7f8515d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8518101870 0x7f8518198010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60966/0 (socket says 172.21.15.40:60966) 2024-04-21T23:18:27.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.603+0000 7f8515d9b640 1 -- 172.21.15.40:0/3094926743 learned_addr learned my addr 172.21.15.40:0/3094926743 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:27.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.603+0000 7f8516d9d640 1 --2- 172.21.15.40:0/3094926743 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8518102230 0x7f851819f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:27.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.603+0000 7f851659c640 1 --2- 172.21.15.40:0/3094926743 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8518100680 0x7f8518197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:27.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.603+0000 7f8515d9b640 1 -- 172.21.15.40:0/3094926743 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8518102230 msgr2=0x7f851819f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:27.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.603+0000 7f8515d9b640 1 --2- 172.21.15.40:0/3094926743 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8518102230 0x7f851819f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:27.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.603+0000 7f8515d9b640 1 -- 172.21.15.40:0/3094926743 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8518100680 msgr2=0x7f8518197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:27.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.603+0000 7f8515d9b640 1 --2- 172.21.15.40:0/3094926743 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8518100680 0x7f8518197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:27.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.603+0000 7f8515d9b640 1 -- 172.21.15.40:0/3094926743 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f851819f680 con 0x7f8518101870 2024-04-21T23:18:27.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.604+0000 7f8515d9b640 1 --2- 172.21.15.40:0/3094926743 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8518101870 0x7f8518198010 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f850c00ea60 tx=0x7f850c00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:27.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.604+0000 7f85077fe640 1 -- 172.21.15.40:0/3094926743 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f850c00cde0 con 0x7f8518101870 2024-04-21T23:18:27.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.604+0000 7f851cbf7640 1 -- 172.21.15.40:0/3094926743 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f851819f970 con 0x7f8518101870 2024-04-21T23:18:27.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.604+0000 7f851cbf7640 1 -- 172.21.15.40:0/3094926743 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f851819ff10 con 0x7f8518101870 2024-04-21T23:18:27.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.604+0000 7f85077fe640 1 -- 172.21.15.40:0/3094926743 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f850c0035f0 con 0x7f8518101870 2024-04-21T23:18:27.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.604+0000 7f85077fe640 1 -- 172.21.15.40:0/3094926743 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f850c010670 con 0x7f8518101870 2024-04-21T23:18:27.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.605+0000 7f851cbf7640 1 -- 172.21.15.40:0/3094926743 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84f80052c0 con 0x7f8518101870 2024-04-21T23:18:27.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.606+0000 7f85077fe640 1 -- 172.21.15.40:0/3094926743 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 23) ==== 138004+0+0 (secure 0 0 0) 0x7f850c004000 con 0x7f8518101870 2024-04-21T23:18:27.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.607+0000 7f85077fe640 1 --2- 172.21.15.40:0/3094926743 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f85000a2e10 0x7f85000a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:27.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.607+0000 7f851659c640 1 --2- 172.21.15.40:0/3094926743 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f85000a2e10 0x7f85000a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:27.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.607+0000 7f85077fe640 1 -- 172.21.15.40:0/3094926743 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f850c0cfb40 con 0x7f8518101870 2024-04-21T23:18:27.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.607+0000 7f851659c640 1 --2- 172.21.15.40:0/3094926743 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f85000a2e10 0x7f85000a52d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8508005fd0 tx=0x7f850800b390 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:27.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.612+0000 7f85077fe640 1 -- 172.21.15.40:0/3094926743 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f850c09c1e0 con 0x7f8518101870 2024-04-21T23:18:27.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.804+0000 7f851cbf7640 1 -- 172.21.15.40:0/3094926743 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f84f8005880 con 0x7f8518101870 2024-04-21T23:18:27.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.805+0000 7f85077fe640 1 -- 172.21.15.40:0/3094926743 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v95) ==== 103+0+11 (secure 0 0 0) 0x7f850c09bae0 con 0x7f8518101870 2024-04-21T23:18:27.804 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-21T23:18:27.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.808+0000 7f851cbf7640 1 -- 172.21.15.40:0/3094926743 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f85000a2e10 msgr2=0x7f85000a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:27.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.808+0000 7f851cbf7640 1 --2- 172.21.15.40:0/3094926743 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f85000a2e10 0x7f85000a52d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8508005fd0 tx=0x7f850800b390 comp rx=0 tx=0).stop 2024-04-21T23:18:27.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.808+0000 7f851cbf7640 1 -- 172.21.15.40:0/3094926743 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8518101870 msgr2=0x7f8518198010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:27.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.808+0000 7f851cbf7640 1 --2- 172.21.15.40:0/3094926743 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8518101870 0x7f8518198010 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f850c00ea60 tx=0x7f850c00ef20 comp rx=0 tx=0).stop 2024-04-21T23:18:27.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.808+0000 7f851cbf7640 1 -- 172.21.15.40:0/3094926743 shutdown_connections 2024-04-21T23:18:27.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.808+0000 7f851cbf7640 1 --2- 172.21.15.40:0/3094926743 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f85000a2e10 0x7f85000a52d0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:27.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.808+0000 7f851cbf7640 1 --2- 172.21.15.40:0/3094926743 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8518102230 0x7f851819f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:27.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.808+0000 7f851cbf7640 1 --2- 172.21.15.40:0/3094926743 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8518101870 0x7f8518198010 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:27.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.808+0000 7f851cbf7640 1 --2- 172.21.15.40:0/3094926743 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8518100680 0x7f8518197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:27.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.808+0000 7f851cbf7640 1 -- 172.21.15.40:0/3094926743 >> 172.21.15.40:0/3094926743 conn(0x7f85180fbbf0 msgr2=0x7f85180fd6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:27.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.808+0000 7f851cbf7640 1 -- 172.21.15.40:0/3094926743 shutdown_connections 2024-04-21T23:18:27.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.808+0000 7f851cbf7640 1 -- 172.21.15.40:0/3094926743 wait complete. 2024-04-21T23:18:27.822 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-04-21T23:18:27.822 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:18:27.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.932+0000 7f65f2f47640 1 -- 172.21.15.40:0/2563615459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f65ec100620 msgr2=0x7f65ec100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:27.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.932+0000 7f65f2f47640 1 --2- 172.21.15.40:0/2563615459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f65ec100620 0x7f65ec100aa0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f65d8009930 tx=0x7f65d802bc30 comp rx=0 tx=0).stop 2024-04-21T23:18:27.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.932+0000 7f65f2f47640 1 -- 172.21.15.40:0/2563615459 shutdown_connections 2024-04-21T23:18:27.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.932+0000 7f65f2f47640 1 --2- 172.21.15.40:0/2563615459 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65ec100fe0 0x7f65ec10d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:27.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.932+0000 7f65f2f47640 1 --2- 172.21.15.40:0/2563615459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f65ec100620 0x7f65ec100aa0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:27.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.932+0000 7f65f2f47640 1 --2- 172.21.15.40:0/2563615459 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f65ec106630 0x7f65ec106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:27.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.932+0000 7f65f2f47640 1 -- 172.21.15.40:0/2563615459 >> 172.21.15.40:0/2563615459 conn(0x7f65ec077740 msgr2=0x7f65ec077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:27.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.932+0000 7f65f2f47640 1 -- 172.21.15.40:0/2563615459 shutdown_connections 2024-04-21T23:18:27.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.932+0000 7f65f2f47640 1 -- 172.21.15.40:0/2563615459 wait complete. 2024-04-21T23:18:27.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.933+0000 7f65f2f47640 1 Processor -- start 2024-04-21T23:18:27.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.933+0000 7f65f2f47640 1 -- start start 2024-04-21T23:18:27.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.933+0000 7f65f2f47640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f65ec100620 0x7f65ec102c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:27.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.933+0000 7f65f2f47640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65ec100fe0 0x7f65ec1031d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:27.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.933+0000 7f65f2f47640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f65ec106630 0x7f65ec103710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:27.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.933+0000 7f65f2f47640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f65ec110a40 con 0x7f65ec100620 2024-04-21T23:18:27.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.933+0000 7f65f2f47640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f65ec1108c0 con 0x7f65ec100fe0 2024-04-21T23:18:27.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.933+0000 7f65f2f47640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f65ec110bc0 con 0x7f65ec106630 2024-04-21T23:18:27.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.933+0000 7f65f0ce3640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f65ec100620 0x7f65ec102c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:27.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.933+0000 7f65f0ce3640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f65ec100620 0x7f65ec102c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60992/0 (socket says 172.21.15.40:60992) 2024-04-21T23:18:27.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.933+0000 7f65f0ce3640 1 -- 172.21.15.40:0/1320803449 learned_addr learned my addr 172.21.15.40:0/1320803449 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:27.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.933+0000 7f65f14e4640 1 --2- 172.21.15.40:0/1320803449 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f65ec106630 0x7f65ec103710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:27.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.933+0000 7f65ebfff640 1 --2- 172.21.15.40:0/1320803449 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65ec100fe0 0x7f65ec1031d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:27.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.933+0000 7f65f0ce3640 1 -- 172.21.15.40:0/1320803449 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f65ec106630 msgr2=0x7f65ec103710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:27.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.933+0000 7f65f0ce3640 1 --2- 172.21.15.40:0/1320803449 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f65ec106630 0x7f65ec103710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:27.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.933+0000 7f65f0ce3640 1 -- 172.21.15.40:0/1320803449 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65ec100fe0 msgr2=0x7f65ec1031d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:27.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.933+0000 7f65f0ce3640 1 --2- 172.21.15.40:0/1320803449 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65ec100fe0 0x7f65ec1031d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:27.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.933+0000 7f65f0ce3640 1 -- 172.21.15.40:0/1320803449 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65ec1a6510 con 0x7f65ec100620 2024-04-21T23:18:27.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.933+0000 7f65f14e4640 1 --2- 172.21.15.40:0/1320803449 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f65ec106630 0x7f65ec103710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:18:27.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.934+0000 7f65f0ce3640 1 --2- 172.21.15.40:0/1320803449 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f65ec100620 0x7f65ec102c90 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f65e000e9f0 tx=0x7f65e000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:27.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.934+0000 7f65e9ffb640 1 -- 172.21.15.40:0/1320803449 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65e000cd80 con 0x7f65ec100620 2024-04-21T23:18:27.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.934+0000 7f65e9ffb640 1 -- 172.21.15.40:0/1320803449 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65e0010910 con 0x7f65ec100620 2024-04-21T23:18:27.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.934+0000 7f65f2f47640 1 -- 172.21.15.40:0/1320803449 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65ec1a6800 con 0x7f65ec100620 2024-04-21T23:18:27.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.934+0000 7f65e9ffb640 1 -- 172.21.15.40:0/1320803449 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65e0018990 con 0x7f65ec100620 2024-04-21T23:18:27.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.934+0000 7f65f2f47640 1 -- 172.21.15.40:0/1320803449 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65ec1a6d20 con 0x7f65ec100620 2024-04-21T23:18:27.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.935+0000 7f65f2f47640 1 -- 172.21.15.40:0/1320803449 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65ec068560 con 0x7f65ec100620 2024-04-21T23:18:27.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.936+0000 7f65e9ffb640 1 -- 172.21.15.40:0/1320803449 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 23) ==== 138004+0+0 (secure 0 0 0) 0x7f65e0010ac0 con 0x7f65ec100620 2024-04-21T23:18:27.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.937+0000 7f65e9ffb640 1 --2- 172.21.15.40:0/1320803449 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f65dc0a2d90 0x7f65dc0a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:27.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.937+0000 7f65ebfff640 1 --2- 172.21.15.40:0/1320803449 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f65dc0a2d90 0x7f65dc0a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:27.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.937+0000 7f65e9ffb640 1 -- 172.21.15.40:0/1320803449 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f65e00cfdf0 con 0x7f65ec100620 2024-04-21T23:18:27.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.938+0000 7f65ebfff640 1 --2- 172.21.15.40:0/1320803449 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f65dc0a2d90 0x7f65dc0a5250 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f65d8006010 tx=0x7f65d8000bc0 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:27.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.942+0000 7f65e9ffb640 1 -- 172.21.15.40:0/1320803449 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f65e009c490 con 0x7f65ec100620 2024-04-21T23:18:27.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:27.985+0000 7f65e9ffb640 1 -- 172.21.15.40:0/1320803449 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7f65e0098400 con 0x7f65ec100620 2024-04-21T23:18:28.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.176+0000 7f65f2f47640 1 -- 172.21.15.40:0/1320803449 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f65ec109ba0 con 0x7f65ec100620 2024-04-21T23:18:28.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.176+0000 7f65e9ffb640 1 -- 172.21.15.40:0/1320803449 <== mon.0 v2:172.21.15.40:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11882 (secure 0 0 0) 0x7f65e0091050 con 0x7f65ec100620 2024-04-21T23:18:28.176 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:18:28.176 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/1571948830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1571948830},{"type":"v1","addr":"172.21.15.40:6835","nonce":1571948830}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/969356642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":969356642},{"type":"v1","addr":"172.21.15.40:6837","nonce":969356642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6839/3025057845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":3025057845},{"type":"v1","addr":"172.21.15.40:6839","nonce":3025057845}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6841/3350488804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":3350488804},{"type":"v1","addr":"172.21.15.40:6841","nonce":3350488804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/3438493203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":3438493203},{"type":"v1","addr":"172.21.15.174:6833","nonce":3438493203}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/3105752678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":3105752678},{"type":"v1","addr":"172.21.15.111:6833","nonce":3105752678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T23:15:55.742233+0000","modified":"2024-04-21T23:18:21.879184+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24521,"mds_1":24502,"mds_2":24494},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24461":{"gid":24461,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6835/2704462092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2704462092},{"type":"v1","addr":"172.21.15.111:6835","nonce":2704462092}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/3905733045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3905733045},{"type":"v1","addr":"172.21.15.174:6835","nonce":3905733045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6837/465579746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":465579746},{"type":"v1","addr":"172.21.15.174:6837","nonce":465579746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/2703764182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":2703764182},{"type":"v1","addr":"172.21.15.111:6837","nonce":2703764182}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24502":{"gid":24502,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.174:6839/2467509274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":2467509274},{"type":"v1","addr":"172.21.15.174:6839","nonce":2467509274}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3712235548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3712235548},{"type":"v1","addr":"172.21.15.111:6839","nonce":3712235548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24521,24494,24502]},"id":1}]} 2024-04-21T23:18:28.176 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 25 2024-04-21T23:18:28.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.178+0000 7f65f2f47640 1 -- 172.21.15.40:0/1320803449 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f65dc0a2d90 msgr2=0x7f65dc0a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:28.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.178+0000 7f65f2f47640 1 --2- 172.21.15.40:0/1320803449 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f65dc0a2d90 0x7f65dc0a5250 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f65d8006010 tx=0x7f65d8000bc0 comp rx=0 tx=0).stop 2024-04-21T23:18:28.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.178+0000 7f65f2f47640 1 -- 172.21.15.40:0/1320803449 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f65ec100620 msgr2=0x7f65ec102c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:28.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.178+0000 7f65f2f47640 1 --2- 172.21.15.40:0/1320803449 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f65ec100620 0x7f65ec102c90 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f65e000e9f0 tx=0x7f65e000eeb0 comp rx=0 tx=0).stop 2024-04-21T23:18:28.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.178+0000 7f65f2f47640 1 -- 172.21.15.40:0/1320803449 shutdown_connections 2024-04-21T23:18:28.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.178+0000 7f65f2f47640 1 --2- 172.21.15.40:0/1320803449 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f65dc0a2d90 0x7f65dc0a5250 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:28.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.178+0000 7f65f2f47640 1 --2- 172.21.15.40:0/1320803449 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65ec100fe0 0x7f65ec1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:28.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.178+0000 7f65f2f47640 1 --2- 172.21.15.40:0/1320803449 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f65ec106630 0x7f65ec103710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:28.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.178+0000 7f65f2f47640 1 --2- 172.21.15.40:0/1320803449 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f65ec100620 0x7f65ec102c90 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:28.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.178+0000 7f65f2f47640 1 -- 172.21.15.40:0/1320803449 >> 172.21.15.40:0/1320803449 conn(0x7f65ec077740 msgr2=0x7f65ec068c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:28.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.178+0000 7f65f2f47640 1 -- 172.21.15.40:0/1320803449 shutdown_connections 2024-04-21T23:18:28.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.178+0000 7f65f2f47640 1 -- 172.21.15.40:0/1320803449 wait complete. 2024-04-21T23:18:28.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:27 smithi040 ceph-mon[27984]: Adjusting osd_memory_target on smithi174 to 176.2M 2024-04-21T23:18:28.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:27 smithi040 ceph-mon[27984]: Unable to set osd_memory_target on smithi174 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-04-21T23:18:28.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:27 smithi040 ceph-mon[27984]: Adjusting osd_memory_target on smithi111 to 176.2M 2024-04-21T23:18:28.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:27 smithi040 ceph-mon[27984]: Unable to set osd_memory_target on smithi111 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-04-21T23:18:28.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:27 smithi040 ceph-mon[27984]: Adjusting osd_memory_target on smithi040 to 177.2M 2024-04-21T23:18:28.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:27 smithi040 ceph-mon[27984]: Unable to set osd_memory_target on smithi040 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-04-21T23:18:28.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:27 smithi040 ceph-mon[27984]: Updating smithi040:/etc/ceph/ceph.conf 2024-04-21T23:18:28.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:27 smithi040 ceph-mon[27984]: Updating smithi111:/etc/ceph/ceph.conf 2024-04-21T23:18:28.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:27 smithi040 ceph-mon[27984]: Updating smithi174:/etc/ceph/ceph.conf 2024-04-21T23:18:28.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:27 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/584075735' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T23:18:28.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:27 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/42623367' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T23:18:28.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:27 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/3094926743' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T23:18:28.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:27 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:18:28.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:27 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:18:28.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:27 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:18:28.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:27 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:18:28.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:27 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:18:28.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:27 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:18:28.184 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:27 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:18:28.190 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-04-21T23:18:28.190 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-04-21T23:18:28.190 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-21T23:18:28.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.293+0000 7ff5a83c9640 1 -- 172.21.15.40:0/3615429613 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5a0100690 msgr2=0x7ff5a0100b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:28.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.293+0000 7ff5a83c9640 1 --2- 172.21.15.40:0/3615429613 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5a0100690 0x7ff5a0100b10 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7ff590009930 tx=0x7ff59002bc30 comp rx=0 tx=0).stop 2024-04-21T23:18:28.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.293+0000 7ff5a83c9640 1 -- 172.21.15.40:0/3615429613 shutdown_connections 2024-04-21T23:18:28.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.293+0000 7ff5a83c9640 1 --2- 172.21.15.40:0/3615429613 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff5a0101050 0x7ff5a010d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:28.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.293+0000 7ff5a83c9640 1 --2- 172.21.15.40:0/3615429613 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5a0100690 0x7ff5a0100b10 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:28.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.293+0000 7ff5a83c9640 1 --2- 172.21.15.40:0/3615429613 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff5a0106650 0x7ff5a0106a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:28.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.293+0000 7ff5a83c9640 1 -- 172.21.15.40:0/3615429613 >> 172.21.15.40:0/3615429613 conn(0x7ff5a00777a0 msgr2=0x7ff5a0077bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:28.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.293+0000 7ff5a83c9640 1 -- 172.21.15.40:0/3615429613 shutdown_connections 2024-04-21T23:18:28.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.293+0000 7ff5a83c9640 1 -- 172.21.15.40:0/3615429613 wait complete. 2024-04-21T23:18:28.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.294+0000 7ff5a83c9640 1 Processor -- start 2024-04-21T23:18:28.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.294+0000 7ff5a83c9640 1 -- start start 2024-04-21T23:18:28.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.294+0000 7ff5a83c9640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff5a0100690 0x7ff5a01980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:28.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.294+0000 7ff5a83c9640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff5a0101050 0x7ff5a0198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:28.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.294+0000 7ff5a83c9640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5a0106650 0x7ff5a019c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:28.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.294+0000 7ff5a83c9640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff5a0111b60 con 0x7ff5a0106650 2024-04-21T23:18:28.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.294+0000 7ff5a83c9640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff5a01119e0 con 0x7ff5a0100690 2024-04-21T23:18:28.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.294+0000 7ff5a83c9640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff5a0111ce0 con 0x7ff5a0101050 2024-04-21T23:18:28.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.294+0000 7ff5a6165640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff5a0100690 0x7ff5a01980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:28.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.294+0000 7ff5a6966640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5a0106650 0x7ff5a019c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:28.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.294+0000 7ff5a6165640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff5a0100690 0x7ff5a01980c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.40:60154/0 (socket says 172.21.15.40:60154) 2024-04-21T23:18:28.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.294+0000 7ff5a6165640 1 -- 172.21.15.40:0/3397121576 learned_addr learned my addr 172.21.15.40:0/3397121576 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:28.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.295+0000 7ff5a5964640 1 --2- 172.21.15.40:0/3397121576 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff5a0101050 0x7ff5a0198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:28.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.295+0000 7ff5a6165640 1 -- 172.21.15.40:0/3397121576 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff5a0101050 msgr2=0x7ff5a0198600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:28.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.295+0000 7ff5a6165640 1 --2- 172.21.15.40:0/3397121576 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff5a0101050 0x7ff5a0198600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:28.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.295+0000 7ff5a6165640 1 -- 172.21.15.40:0/3397121576 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5a0106650 msgr2=0x7ff5a019c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:28.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.295+0000 7ff5a6165640 1 --2- 172.21.15.40:0/3397121576 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5a0106650 0x7ff5a019c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:28.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.295+0000 7ff5a6165640 1 -- 172.21.15.40:0/3397121576 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5a019d0b0 con 0x7ff5a0100690 2024-04-21T23:18:28.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.295+0000 7ff5a6966640 1 --2- 172.21.15.40:0/3397121576 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5a0106650 0x7ff5a019c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:18:28.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.295+0000 7ff5a6165640 1 --2- 172.21.15.40:0/3397121576 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff5a0100690 0x7ff5a01980c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff59c00d770 tx=0x7ff59c00dc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:28.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.295+0000 7ff5977fe640 1 -- 172.21.15.40:0/3397121576 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff59c017070 con 0x7ff5a0100690 2024-04-21T23:18:28.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.295+0000 7ff5a83c9640 1 -- 172.21.15.40:0/3397121576 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5a019d3a0 con 0x7ff5a0100690 2024-04-21T23:18:28.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.295+0000 7ff5977fe640 1 -- 172.21.15.40:0/3397121576 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff59c005500 con 0x7ff5a0100690 2024-04-21T23:18:28.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.295+0000 7ff5977fe640 1 -- 172.21.15.40:0/3397121576 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff59c004e00 con 0x7ff5a0100690 2024-04-21T23:18:28.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.295+0000 7ff5a83c9640 1 -- 172.21.15.40:0/3397121576 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5a01a4c30 con 0x7ff5a0100690 2024-04-21T23:18:28.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.296+0000 7ff5a83c9640 1 -- 172.21.15.40:0/3397121576 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5a00689b0 con 0x7ff5a0100690 2024-04-21T23:18:28.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.297+0000 7ff5977fe640 1 -- 172.21.15.40:0/3397121576 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7ff59c004fa0 con 0x7ff5a0100690 2024-04-21T23:18:28.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.298+0000 7ff5977fe640 1 --2- 172.21.15.40:0/3397121576 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7ff58c0a2e20 0x7ff58c0a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:28.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.298+0000 7ff5a5964640 1 --2- 172.21.15.40:0/3397121576 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7ff58c0a2e20 0x7ff58c0a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:28.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.298+0000 7ff5977fe640 1 -- 172.21.15.40:0/3397121576 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7ff59c0cf400 con 0x7ff5a0100690 2024-04-21T23:18:28.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.298+0000 7ff5a5964640 1 --2- 172.21.15.40:0/3397121576 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7ff58c0a2e20 0x7ff58c0a52e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff590005ab0 tx=0x7ff590005fd0 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:28.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.301+0000 7ff5977fe640 1 -- 172.21.15.40:0/3397121576 <== mon.2 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7ff59c09b370 con 0x7ff5a0100690 2024-04-21T23:18:28.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:27 smithi174 ceph-mon[31409]: Adjusting osd_memory_target on smithi174 to 176.2M 2024-04-21T23:18:28.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:27 smithi174 ceph-mon[31409]: Unable to set osd_memory_target on smithi174 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-04-21T23:18:28.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:27 smithi174 ceph-mon[31409]: Adjusting osd_memory_target on smithi111 to 176.2M 2024-04-21T23:18:28.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:27 smithi174 ceph-mon[31409]: Unable to set osd_memory_target on smithi111 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-04-21T23:18:28.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:27 smithi174 ceph-mon[31409]: Adjusting osd_memory_target on smithi040 to 177.2M 2024-04-21T23:18:28.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:27 smithi174 ceph-mon[31409]: Unable to set osd_memory_target on smithi040 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-04-21T23:18:28.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:27 smithi174 ceph-mon[31409]: Updating smithi040:/etc/ceph/ceph.conf 2024-04-21T23:18:28.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:27 smithi174 ceph-mon[31409]: Updating smithi111:/etc/ceph/ceph.conf 2024-04-21T23:18:28.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:27 smithi174 ceph-mon[31409]: Updating smithi174:/etc/ceph/ceph.conf 2024-04-21T23:18:28.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:27 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/584075735' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T23:18:28.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:27 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/42623367' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T23:18:28.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:27 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/3094926743' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T23:18:28.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:27 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:18:28.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:27 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:18:28.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:27 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:18:28.446 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:27 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:18:28.446 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:27 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:18:28.446 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:27 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:18:28.446 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:27 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:18:28.446 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:27 smithi111 ceph-mon[31085]: Adjusting osd_memory_target on smithi174 to 176.2M 2024-04-21T23:18:28.446 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:27 smithi111 ceph-mon[31085]: Unable to set osd_memory_target on smithi174 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-04-21T23:18:28.447 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:27 smithi111 ceph-mon[31085]: Adjusting osd_memory_target on smithi111 to 176.2M 2024-04-21T23:18:28.447 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:27 smithi111 ceph-mon[31085]: Unable to set osd_memory_target on smithi111 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-04-21T23:18:28.447 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:27 smithi111 ceph-mon[31085]: Adjusting osd_memory_target on smithi040 to 177.2M 2024-04-21T23:18:28.447 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:27 smithi111 ceph-mon[31085]: Unable to set osd_memory_target on smithi040 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-04-21T23:18:28.447 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:27 smithi111 ceph-mon[31085]: Updating smithi040:/etc/ceph/ceph.conf 2024-04-21T23:18:28.447 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:27 smithi111 ceph-mon[31085]: Updating smithi111:/etc/ceph/ceph.conf 2024-04-21T23:18:28.447 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:27 smithi111 ceph-mon[31085]: Updating smithi174:/etc/ceph/ceph.conf 2024-04-21T23:18:28.447 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:27 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/584075735' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T23:18:28.447 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:27 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/42623367' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T23:18:28.447 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:27 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/3094926743' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T23:18:28.447 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:27 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:18:28.447 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:27 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:18:28.447 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:27 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:18:28.447 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:27 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:18:28.447 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:27 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:18:28.448 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:27 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:18:28.448 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:27 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:18:28.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.485+0000 7ff5a83c9640 1 -- 172.21.15.40:0/3397121576 --> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff5a0061b80 con 0x7ff58c0a2e20 2024-04-21T23:18:28.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.495+0000 7ff5977fe640 1 -- 172.21.15.40:0/3397121576 <== mgr.25111 v2:172.21.15.40:6800/478611259 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339387 (secure 0 0 0) 0x7ff5a0061b80 con 0x7ff58c0a2e20 2024-04-21T23:18:28.499 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:18:28.501 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-21T23:18:28.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.498+0000 7ff5a83c9640 1 -- 172.21.15.40:0/3397121576 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7ff58c0a2e20 msgr2=0x7ff58c0a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:28.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.498+0000 7ff5a83c9640 1 --2- 172.21.15.40:0/3397121576 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7ff58c0a2e20 0x7ff58c0a52e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff590005ab0 tx=0x7ff590005fd0 comp rx=0 tx=0).stop 2024-04-21T23:18:28.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.498+0000 7ff5a83c9640 1 -- 172.21.15.40:0/3397121576 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff5a0100690 msgr2=0x7ff5a01980c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:28.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.498+0000 7ff5a83c9640 1 --2- 172.21.15.40:0/3397121576 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff5a0100690 0x7ff5a01980c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff59c00d770 tx=0x7ff59c00dc30 comp rx=0 tx=0).stop 2024-04-21T23:18:28.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.498+0000 7ff5a83c9640 1 -- 172.21.15.40:0/3397121576 shutdown_connections 2024-04-21T23:18:28.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.498+0000 7ff5a83c9640 1 --2- 172.21.15.40:0/3397121576 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7ff58c0a2e20 0x7ff58c0a52e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:28.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.498+0000 7ff5a83c9640 1 --2- 172.21.15.40:0/3397121576 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5a0106650 0x7ff5a019c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:28.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.498+0000 7ff5a83c9640 1 --2- 172.21.15.40:0/3397121576 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff5a0101050 0x7ff5a0198600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:28.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.498+0000 7ff5a83c9640 1 --2- 172.21.15.40:0/3397121576 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff5a0100690 0x7ff5a01980c0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:28.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.498+0000 7ff5a83c9640 1 -- 172.21.15.40:0/3397121576 >> 172.21.15.40:0/3397121576 conn(0x7ff5a00777a0 msgr2=0x7ff5a0103310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:28.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.498+0000 7ff5a83c9640 1 -- 172.21.15.40:0/3397121576 shutdown_connections 2024-04-21T23:18:28.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.498+0000 7ff5a83c9640 1 -- 172.21.15.40:0/3397121576 wait complete. 2024-04-21T23:18:28.509 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-04-21T23:18:27.410882+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1188469,"num_objects":66,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":990,"num_read_kb":655,"num_write":471,"num_write_kb":4380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1180848,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":867,"ondisk_log_size":867,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":373920,"kb_used_data":16356,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124567904,"statfs":{"total":1151940427776,"available":1151557533696,"internally_reserved":0,"allocated":16748544,"data_stored":9961440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":365933882},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.993386"},"pg_stats":[{"pgid":"3.3f","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.399227+0000","last_change":"2024-04-21T23:15:56.349368+0000","last_active":"2024-04-21T23:18:23.399227+0000","last_peered":"2024-04-21T23:18:23.399227+0000","last_clean":"2024-04-21T23:18:23.399227+0000","last_became_active":"2024-04-21T23:15:56.345743+0000","last_became_peered":"2024-04-21T23:15:56.345743+0000","last_unstale":"2024-04-21T23:18:23.399227+0000","last_undegraded":"2024-04-21T23:18:23.399227+0000","last_fullsized":"2024-04-21T23:18:23.399227+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:57:21.574012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"92'4","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.832418+0000","last_change":"2024-04-21T23:15:56.347231+0000","last_active":"2024-04-21T23:18:23.832418+0000","last_peered":"2024-04-21T23:18:23.832418+0000","last_clean":"2024-04-21T23:18:23.832418+0000","last_became_active":"2024-04-21T23:15:56.346847+0000","last_became_peered":"2024-04-21T23:15:56.346847+0000","last_unstale":"2024-04-21T23:18:23.832418+0000","last_undegraded":"2024-04-21T23:18:23.832418+0000","last_fullsized":"2024-04-21T23:18:23.832418+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:20:15.884789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.831956+0000","last_change":"2024-04-21T23:15:56.354897+0000","last_active":"2024-04-21T23:18:23.831956+0000","last_peered":"2024-04-21T23:18:23.831956+0000","last_clean":"2024-04-21T23:18:23.831956+0000","last_became_active":"2024-04-21T23:15:56.354494+0000","last_became_peered":"2024-04-21T23:15:56.354494+0000","last_unstale":"2024-04-21T23:18:23.831956+0000","last_undegraded":"2024-04-21T23:18:23.831956+0000","last_fullsized":"2024-04-21T23:18:23.831956+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:51:11.865310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.660564+0000","last_change":"2024-04-21T23:15:56.355184+0000","last_active":"2024-04-21T23:18:23.660564+0000","last_peered":"2024-04-21T23:18:23.660564+0000","last_clean":"2024-04-21T23:18:23.660564+0000","last_became_active":"2024-04-21T23:15:56.354748+0000","last_became_peered":"2024-04-21T23:15:56.354748+0000","last_unstale":"2024-04-21T23:18:23.660564+0000","last_undegraded":"2024-04-21T23:18:23.660564+0000","last_fullsized":"2024-04-21T23:18:23.660564+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:46:24.867488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"92'4","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.500211+0000","last_change":"2024-04-21T23:15:56.349446+0000","last_active":"2024-04-21T23:18:23.500211+0000","last_peered":"2024-04-21T23:18:23.500211+0000","last_clean":"2024-04-21T23:18:23.500211+0000","last_became_active":"2024-04-21T23:15:56.349050+0000","last_became_peered":"2024-04-21T23:15:56.349050+0000","last_unstale":"2024-04-21T23:18:23.500211+0000","last_undegraded":"2024-04-21T23:18:23.500211+0000","last_fullsized":"2024-04-21T23:18:23.500211+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:15:25.489044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"92'6","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.399061+0000","last_change":"2024-04-21T23:15:56.356095+0000","last_active":"2024-04-21T23:18:23.399061+0000","last_peered":"2024-04-21T23:18:23.399061+0000","last_clean":"2024-04-21T23:18:23.399061+0000","last_became_active":"2024-04-21T23:15:56.355754+0000","last_became_peered":"2024-04-21T23:15:56.355754+0000","last_unstale":"2024-04-21T23:18:23.399061+0000","last_undegraded":"2024-04-21T23:18:23.399061+0000","last_fullsized":"2024-04-21T23:18:23.399061+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:00:27.296457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"92'5","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.662728+0000","last_change":"2024-04-21T23:15:56.349733+0000","last_active":"2024-04-21T23:18:23.662728+0000","last_peered":"2024-04-21T23:18:23.662728+0000","last_clean":"2024-04-21T23:18:23.662728+0000","last_became_active":"2024-04-21T23:15:56.347820+0000","last_became_peered":"2024-04-21T23:15:56.347820+0000","last_unstale":"2024-04-21T23:18:23.662728+0000","last_undegraded":"2024-04-21T23:18:23.662728+0000","last_fullsized":"2024-04-21T23:18:23.662728+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:31:34.453626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"92'9","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.500348+0000","last_change":"2024-04-21T23:15:56.359385+0000","last_active":"2024-04-21T23:18:23.500348+0000","last_peered":"2024-04-21T23:18:23.500348+0000","last_clean":"2024-04-21T23:18:23.500348+0000","last_became_active":"2024-04-21T23:15:56.359074+0000","last_became_peered":"2024-04-21T23:15:56.359074+0000","last_unstale":"2024-04-21T23:18:23.500348+0000","last_undegraded":"2024-04-21T23:18:23.500348+0000","last_fullsized":"2024-04-21T23:18:23.500348+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:40:23.904592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"92'11","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.661940+0000","last_change":"2024-04-21T23:15:56.350886+0000","last_active":"2024-04-21T23:18:23.661940+0000","last_peered":"2024-04-21T23:18:23.661940+0000","last_clean":"2024-04-21T23:18:23.661940+0000","last_became_active":"2024-04-21T23:15:56.350531+0000","last_became_peered":"2024-04-21T23:15:56.350531+0000","last_unstale":"2024-04-21T23:18:23.661940+0000","last_undegraded":"2024-04-21T23:18:23.661940+0000","last_fullsized":"2024-04-21T23:18:23.661940+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:13:12.354473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"92'16","reported_seq":48,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.399459+0000","last_change":"2024-04-21T23:15:56.347376+0000","last_active":"2024-04-21T23:18:23.399459+0000","last_peered":"2024-04-21T23:18:23.399459+0000","last_clean":"2024-04-21T23:18:23.399459+0000","last_became_active":"2024-04-21T23:15:56.347016+0000","last_became_peered":"2024-04-21T23:15:56.347016+0000","last_unstale":"2024-04-21T23:18:23.399459+0000","last_undegraded":"2024-04-21T23:18:23.399459+0000","last_fullsized":"2024-04-21T23:18:23.399459+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:49:46.722931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.663443+0000","last_change":"2024-04-21T23:15:56.351754+0000","last_active":"2024-04-21T23:18:23.663443+0000","last_peered":"2024-04-21T23:18:23.663443+0000","last_clean":"2024-04-21T23:18:23.663443+0000","last_became_active":"2024-04-21T23:15:56.349860+0000","last_became_peered":"2024-04-21T23:15:56.349860+0000","last_unstale":"2024-04-21T23:18:23.663443+0000","last_undegraded":"2024-04-21T23:18:23.663443+0000","last_fullsized":"2024-04-21T23:18:23.663443+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:48:41.469708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.660467+0000","last_change":"2024-04-21T23:15:56.345177+0000","last_active":"2024-04-21T23:18:23.660467+0000","last_peered":"2024-04-21T23:18:23.660467+0000","last_clean":"2024-04-21T23:18:23.660467+0000","last_became_active":"2024-04-21T23:15:56.344791+0000","last_became_peered":"2024-04-21T23:15:56.344791+0000","last_unstale":"2024-04-21T23:18:23.660467+0000","last_undegraded":"2024-04-21T23:18:23.660467+0000","last_fullsized":"2024-04-21T23:18:23.660467+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:33:33.418383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"92'10","reported_seq":48,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.688996+0000","last_change":"2024-04-21T23:16:52.556505+0000","last_active":"2024-04-21T23:18:23.688996+0000","last_peered":"2024-04-21T23:18:23.688996+0000","last_clean":"2024-04-21T23:18:23.688996+0000","last_became_active":"2024-04-21T23:16:52.556177+0000","last_became_peered":"2024-04-21T23:16:52.556177+0000","last_unstale":"2024-04-21T23:18:23.688996+0000","last_undegraded":"2024-04-21T23:18:23.688996+0000","last_fullsized":"2024-04-21T23:18:23.688996+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:17:13.546746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.832040+0000","last_change":"2024-04-21T23:15:56.354544+0000","last_active":"2024-04-21T23:18:23.832040+0000","last_peered":"2024-04-21T23:18:23.832040+0000","last_clean":"2024-04-21T23:18:23.832040+0000","last_became_active":"2024-04-21T23:15:56.354206+0000","last_became_peered":"2024-04-21T23:15:56.354206+0000","last_unstale":"2024-04-21T23:18:23.832040+0000","last_undegraded":"2024-04-21T23:18:23.832040+0000","last_fullsized":"2024-04-21T23:18:23.832040+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:37:19.985422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.831952+0000","last_change":"2024-04-21T23:15:56.357447+0000","last_active":"2024-04-21T23:18:23.831952+0000","last_peered":"2024-04-21T23:18:23.831952+0000","last_clean":"2024-04-21T23:18:23.831952+0000","last_became_active":"2024-04-21T23:15:56.355850+0000","last_became_peered":"2024-04-21T23:15:56.355850+0000","last_unstale":"2024-04-21T23:18:23.831952+0000","last_undegraded":"2024-04-21T23:18:23.831952+0000","last_fullsized":"2024-04-21T23:18:23.831952+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:48:56.648994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.660691+0000","last_change":"2024-04-21T23:15:56.353712+0000","last_active":"2024-04-21T23:18:23.660691+0000","last_peered":"2024-04-21T23:18:23.660691+0000","last_clean":"2024-04-21T23:18:23.660691+0000","last_became_active":"2024-04-21T23:15:56.353388+0000","last_became_peered":"2024-04-21T23:15:56.353388+0000","last_unstale":"2024-04-21T23:18:23.660691+0000","last_undegraded":"2024-04-21T23:18:23.660691+0000","last_fullsized":"2024-04-21T23:18:23.660691+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:43:33.581222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.399252+0000","last_change":"2024-04-21T23:15:56.347041+0000","last_active":"2024-04-21T23:18:23.399252+0000","last_peered":"2024-04-21T23:18:23.399252+0000","last_clean":"2024-04-21T23:18:23.399252+0000","last_became_active":"2024-04-21T23:15:56.346630+0000","last_became_peered":"2024-04-21T23:15:56.346630+0000","last_unstale":"2024-04-21T23:18:23.399252+0000","last_undegraded":"2024-04-21T23:18:23.399252+0000","last_fullsized":"2024-04-21T23:18:23.399252+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:57:42.553666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"92'10","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.498505+0000","last_change":"2024-04-21T23:15:56.353809+0000","last_active":"2024-04-21T23:18:23.498505+0000","last_peered":"2024-04-21T23:18:23.498505+0000","last_clean":"2024-04-21T23:18:23.498505+0000","last_became_active":"2024-04-21T23:15:56.353469+0000","last_became_peered":"2024-04-21T23:15:56.353469+0000","last_unstale":"2024-04-21T23:18:23.498505+0000","last_undegraded":"2024-04-21T23:18:23.498505+0000","last_fullsized":"2024-04-21T23:18:23.498505+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:11:14.136845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.395486+0000","last_change":"2024-04-21T23:15:56.354500+0000","last_active":"2024-04-21T23:18:23.395486+0000","last_peered":"2024-04-21T23:18:23.395486+0000","last_clean":"2024-04-21T23:18:23.395486+0000","last_became_active":"2024-04-21T23:15:56.353893+0000","last_became_peered":"2024-04-21T23:15:56.353893+0000","last_unstale":"2024-04-21T23:18:23.395486+0000","last_undegraded":"2024-04-21T23:18:23.395486+0000","last_fullsized":"2024-04-21T23:18:23.395486+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:14:38.121448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.395523+0000","last_change":"2024-04-21T23:17:53.274634+0000","last_active":"2024-04-21T23:18:23.395523+0000","last_peered":"2024-04-21T23:18:23.395523+0000","last_clean":"2024-04-21T23:18:23.395523+0000","last_became_active":"2024-04-21T23:17:53.274018+0000","last_became_peered":"2024-04-21T23:17:53.274018+0000","last_unstale":"2024-04-21T23:18:23.395523+0000","last_undegraded":"2024-04-21T23:18:23.395523+0000","last_fullsized":"2024-04-21T23:18:23.395523+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:04:55.017447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8,0],"acting":[2,3,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"90'9","reported_seq":44,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.498460+0000","last_change":"2024-04-21T23:15:55.402553+0000","last_active":"2024-04-21T23:18:23.498460+0000","last_peered":"2024-04-21T23:18:23.498460+0000","last_clean":"2024-04-21T23:18:23.498460+0000","last_became_active":"2024-04-21T23:15:55.402192+0000","last_became_peered":"2024-04-21T23:15:55.402192+0000","last_unstale":"2024-04-21T23:18:23.498460+0000","last_undegraded":"2024-04-21T23:18:23.498460+0000","last_fullsized":"2024-04-21T23:18:23.498460+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:24:57.467906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"92'5","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.501309+0000","last_change":"2024-04-21T23:15:56.358797+0000","last_active":"2024-04-21T23:18:23.501309+0000","last_peered":"2024-04-21T23:18:23.501309+0000","last_clean":"2024-04-21T23:18:23.501309+0000","last_became_active":"2024-04-21T23:15:56.357997+0000","last_became_peered":"2024-04-21T23:15:56.357997+0000","last_unstale":"2024-04-21T23:18:23.501309+0000","last_undegraded":"2024-04-21T23:18:23.501309+0000","last_fullsized":"2024-04-21T23:18:23.501309+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:20:50.891506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.659831+0000","last_change":"2024-04-21T23:16:00.443368+0000","last_active":"2024-04-21T23:18:23.659831+0000","last_peered":"2024-04-21T23:18:23.659831+0000","last_clean":"2024-04-21T23:18:23.659831+0000","last_became_active":"2024-04-21T23:16:00.442940+0000","last_became_peered":"2024-04-21T23:16:00.442940+0000","last_unstale":"2024-04-21T23:18:23.659831+0000","last_undegraded":"2024-04-21T23:18:23.659831+0000","last_fullsized":"2024-04-21T23:18:23.659831+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:17:15.706460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"90'2","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.501369+0000","last_change":"2024-04-21T23:15:55.405216+0000","last_active":"2024-04-21T23:18:23.501369+0000","last_peered":"2024-04-21T23:18:23.501369+0000","last_clean":"2024-04-21T23:18:23.501369+0000","last_became_active":"2024-04-21T23:15:55.404844+0000","last_became_peered":"2024-04-21T23:15:55.404844+0000","last_unstale":"2024-04-21T23:18:23.501369+0000","last_undegraded":"2024-04-21T23:18:23.501369+0000","last_fullsized":"2024-04-21T23:18:23.501369+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:56:28.691844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.832115+0000","last_change":"2024-04-21T23:15:56.355789+0000","last_active":"2024-04-21T23:18:23.832115+0000","last_peered":"2024-04-21T23:18:23.832115+0000","last_clean":"2024-04-21T23:18:23.832115+0000","last_became_active":"2024-04-21T23:15:56.354972+0000","last_became_peered":"2024-04-21T23:15:56.354972+0000","last_unstale":"2024-04-21T23:18:23.832115+0000","last_undegraded":"2024-04-21T23:18:23.832115+0000","last_fullsized":"2024-04-21T23:18:23.832115+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:11:36.831453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.660103+0000","last_change":"2024-04-21T23:16:00.443241+0000","last_active":"2024-04-21T23:18:23.660103+0000","last_peered":"2024-04-21T23:18:23.660103+0000","last_clean":"2024-04-21T23:18:23.660103+0000","last_became_active":"2024-04-21T23:16:00.442820+0000","last_became_peered":"2024-04-21T23:16:00.442820+0000","last_unstale":"2024-04-21T23:18:23.660103+0000","last_undegraded":"2024-04-21T23:18:23.660103+0000","last_fullsized":"2024-04-21T23:18:23.660103+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:28:04.032621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"90'4","reported_seq":49,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.500708+0000","last_change":"2024-04-21T23:16:52.212906+0000","last_active":"2024-04-21T23:18:23.500708+0000","last_peered":"2024-04-21T23:18:23.500708+0000","last_clean":"2024-04-21T23:18:23.500708+0000","last_became_active":"2024-04-21T23:16:52.205828+0000","last_became_peered":"2024-04-21T23:16:52.205828+0000","last_unstale":"2024-04-21T23:18:23.500708+0000","last_undegraded":"2024-04-21T23:18:23.500708+0000","last_fullsized":"2024-04-21T23:18:23.500708+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:20:01.936578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.662333+0000","last_change":"2024-04-21T23:15:56.354462+0000","last_active":"2024-04-21T23:18:23.662333+0000","last_peered":"2024-04-21T23:18:23.662333+0000","last_clean":"2024-04-21T23:18:23.662333+0000","last_became_active":"2024-04-21T23:15:56.354134+0000","last_became_peered":"2024-04-21T23:15:56.354134+0000","last_unstale":"2024-04-21T23:18:23.662333+0000","last_undegraded":"2024-04-21T23:18:23.662333+0000","last_fullsized":"2024-04-21T23:18:23.662333+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:27:40.809483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.660710+0000","last_change":"2024-04-21T23:16:00.427717+0000","last_active":"2024-04-21T23:18:23.660710+0000","last_peered":"2024-04-21T23:18:23.660710+0000","last_clean":"2024-04-21T23:18:23.660710+0000","last_became_active":"2024-04-21T23:16:00.427296+0000","last_became_peered":"2024-04-21T23:16:00.427296+0000","last_unstale":"2024-04-21T23:18:23.660710+0000","last_undegraded":"2024-04-21T23:18:23.660710+0000","last_fullsized":"2024-04-21T23:18:23.660710+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:03:41.650045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.500411+0000","last_change":"2024-04-21T23:15:55.398641+0000","last_active":"2024-04-21T23:18:23.500411+0000","last_peered":"2024-04-21T23:18:23.500411+0000","last_clean":"2024-04-21T23:18:23.500411+0000","last_became_active":"2024-04-21T23:15:55.397712+0000","last_became_peered":"2024-04-21T23:15:55.397712+0000","last_unstale":"2024-04-21T23:18:23.500411+0000","last_undegraded":"2024-04-21T23:18:23.500411+0000","last_fullsized":"2024-04-21T23:18:23.500411+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:28:09.266740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"92'15","reported_seq":46,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.507482+0000","last_change":"2024-04-21T23:15:56.345359+0000","last_active":"2024-04-21T23:18:23.507482+0000","last_peered":"2024-04-21T23:18:23.507482+0000","last_clean":"2024-04-21T23:18:23.507482+0000","last_became_active":"2024-04-21T23:15:56.343652+0000","last_became_peered":"2024-04-21T23:15:56.343652+0000","last_unstale":"2024-04-21T23:18:23.507482+0000","last_undegraded":"2024-04-21T23:18:23.507482+0000","last_fullsized":"2024-04-21T23:18:23.507482+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:18:10.057999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.395459+0000","last_change":"2024-04-21T23:16:00.430640+0000","last_active":"2024-04-21T23:18:23.395459+0000","last_peered":"2024-04-21T23:18:23.395459+0000","last_clean":"2024-04-21T23:18:23.395459+0000","last_became_active":"2024-04-21T23:16:00.430141+0000","last_became_peered":"2024-04-21T23:16:00.430141+0000","last_unstale":"2024-04-21T23:18:23.395459+0000","last_undegraded":"2024-04-21T23:18:23.395459+0000","last_fullsized":"2024-04-21T23:18:23.395459+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:41:13.749780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"95'6","reported_seq":165,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.688107+0000","last_change":"2024-04-21T23:15:55.399823+0000","last_active":"2024-04-21T23:18:23.688107+0000","last_peered":"2024-04-21T23:18:23.688107+0000","last_clean":"2024-04-21T23:18:23.688107+0000","last_became_active":"2024-04-21T23:15:55.399490+0000","last_became_peered":"2024-04-21T23:15:55.399490+0000","last_unstale":"2024-04-21T23:18:23.688107+0000","last_undegraded":"2024-04-21T23:18:23.688107+0000","last_fullsized":"2024-04-21T23:18:23.688107+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:54:38.124841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":125,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.500231+0000","last_change":"2024-04-21T23:15:56.346444+0000","last_active":"2024-04-21T23:18:23.500231+0000","last_peered":"2024-04-21T23:18:23.500231+0000","last_clean":"2024-04-21T23:18:23.500231+0000","last_became_active":"2024-04-21T23:15:56.346091+0000","last_became_peered":"2024-04-21T23:15:56.346091+0000","last_unstale":"2024-04-21T23:18:23.500231+0000","last_undegraded":"2024-04-21T23:18:23.500231+0000","last_fullsized":"2024-04-21T23:18:23.500231+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:41:11.144570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.395544+0000","last_change":"2024-04-21T23:16:00.438198+0000","last_active":"2024-04-21T23:18:23.395544+0000","last_peered":"2024-04-21T23:18:23.395544+0000","last_clean":"2024-04-21T23:18:23.395544+0000","last_became_active":"2024-04-21T23:16:00.437767+0000","last_became_peered":"2024-04-21T23:16:00.437767+0000","last_unstale":"2024-04-21T23:18:23.395544+0000","last_undegraded":"2024-04-21T23:18:23.395544+0000","last_fullsized":"2024-04-21T23:18:23.395544+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:35:11.356414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"90'3","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.399081+0000","last_change":"2024-04-21T23:15:55.403896+0000","last_active":"2024-04-21T23:18:23.399081+0000","last_peered":"2024-04-21T23:18:23.399081+0000","last_clean":"2024-04-21T23:18:23.399081+0000","last_became_active":"2024-04-21T23:15:55.403423+0000","last_became_peered":"2024-04-21T23:15:55.403423+0000","last_unstale":"2024-04-21T23:18:23.399081+0000","last_undegraded":"2024-04-21T23:18:23.399081+0000","last_fullsized":"2024-04-21T23:18:23.399081+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:34:44.252259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"92'8","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.498057+0000","last_change":"2024-04-21T23:16:52.349354+0000","last_active":"2024-04-21T23:18:23.498057+0000","last_peered":"2024-04-21T23:18:23.498057+0000","last_clean":"2024-04-21T23:18:23.498057+0000","last_became_active":"2024-04-21T23:16:52.348991+0000","last_became_peered":"2024-04-21T23:16:52.348991+0000","last_unstale":"2024-04-21T23:18:23.498057+0000","last_undegraded":"2024-04-21T23:18:23.498057+0000","last_fullsized":"2024-04-21T23:18:23.498057+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:42:57.975963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.399121+0000","last_change":"2024-04-21T23:16:00.434104+0000","last_active":"2024-04-21T23:18:23.399121+0000","last_peered":"2024-04-21T23:18:23.399121+0000","last_clean":"2024-04-21T23:18:23.399121+0000","last_became_active":"2024-04-21T23:16:00.433681+0000","last_became_peered":"2024-04-21T23:16:00.433681+0000","last_unstale":"2024-04-21T23:18:23.399121+0000","last_undegraded":"2024-04-21T23:18:23.399121+0000","last_fullsized":"2024-04-21T23:18:23.399121+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:10:17.986652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"90'2","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.688659+0000","last_change":"2024-04-21T23:15:55.397536+0000","last_active":"2024-04-21T23:18:23.688659+0000","last_peered":"2024-04-21T23:18:23.688659+0000","last_clean":"2024-04-21T23:18:23.688659+0000","last_became_active":"2024-04-21T23:15:55.397163+0000","last_became_peered":"2024-04-21T23:15:55.397163+0000","last_unstale":"2024-04-21T23:18:23.688659+0000","last_undegraded":"2024-04-21T23:18:23.688659+0000","last_fullsized":"2024-04-21T23:18:23.688659+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:05:27.777804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.500246+0000","last_change":"2024-04-21T23:15:56.351840+0000","last_active":"2024-04-21T23:18:23.500246+0000","last_peered":"2024-04-21T23:18:23.500246+0000","last_clean":"2024-04-21T23:18:23.500246+0000","last_became_active":"2024-04-21T23:15:56.351523+0000","last_became_peered":"2024-04-21T23:15:56.351523+0000","last_unstale":"2024-04-21T23:18:23.500246+0000","last_undegraded":"2024-04-21T23:18:23.500246+0000","last_fullsized":"2024-04-21T23:18:23.500246+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:57:11.714284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.832073+0000","last_change":"2024-04-21T23:16:00.437473+0000","last_active":"2024-04-21T23:18:23.832073+0000","last_peered":"2024-04-21T23:18:23.832073+0000","last_clean":"2024-04-21T23:18:23.832073+0000","last_became_active":"2024-04-21T23:16:00.437014+0000","last_became_peered":"2024-04-21T23:16:00.437014+0000","last_unstale":"2024-04-21T23:18:23.832073+0000","last_undegraded":"2024-04-21T23:18:23.832073+0000","last_fullsized":"2024-04-21T23:18:23.832073+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:37:14.241896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"90'1","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.507228+0000","last_change":"2024-04-21T23:15:55.399213+0000","last_active":"2024-04-21T23:18:23.507228+0000","last_peered":"2024-04-21T23:18:23.507228+0000","last_clean":"2024-04-21T23:18:23.507228+0000","last_became_active":"2024-04-21T23:15:55.398880+0000","last_became_peered":"2024-04-21T23:15:55.398880+0000","last_unstale":"2024-04-21T23:18:23.507228+0000","last_undegraded":"2024-04-21T23:18:23.507228+0000","last_fullsized":"2024-04-21T23:18:23.507228+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:56:32.236862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"92'11","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.497698+0000","last_change":"2024-04-21T23:15:56.349330+0000","last_active":"2024-04-21T23:18:23.497698+0000","last_peered":"2024-04-21T23:18:23.497698+0000","last_clean":"2024-04-21T23:18:23.497698+0000","last_became_active":"2024-04-21T23:15:56.348988+0000","last_became_peered":"2024-04-21T23:15:56.348988+0000","last_unstale":"2024-04-21T23:18:23.497698+0000","last_undegraded":"2024-04-21T23:18:23.497698+0000","last_fullsized":"2024-04-21T23:18:23.497698+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:11:19.650119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.500463+0000","last_change":"2024-04-21T23:16:00.431959+0000","last_active":"2024-04-21T23:18:23.500463+0000","last_peered":"2024-04-21T23:18:23.500463+0000","last_clean":"2024-04-21T23:18:23.500463+0000","last_became_active":"2024-04-21T23:16:00.431570+0000","last_became_peered":"2024-04-21T23:16:00.431570+0000","last_unstale":"2024-04-21T23:18:23.500463+0000","last_undegraded":"2024-04-21T23:18:23.500463+0000","last_fullsized":"2024-04-21T23:18:23.500463+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:02:54.057326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"95'4","reported_seq":160,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.686988+0000","last_change":"2024-04-21T23:15:55.404505+0000","last_active":"2024-04-21T23:18:23.686988+0000","last_peered":"2024-04-21T23:18:23.686988+0000","last_clean":"2024-04-21T23:18:23.686988+0000","last_became_active":"2024-04-21T23:15:55.403945+0000","last_became_peered":"2024-04-21T23:15:55.403945+0000","last_unstale":"2024-04-21T23:18:23.686988+0000","last_undegraded":"2024-04-21T23:18:23.686988+0000","last_fullsized":"2024-04-21T23:18:23.686988+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:54:37.647597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":122,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"92'10","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.661999+0000","last_change":"2024-04-21T23:15:56.356238+0000","last_active":"2024-04-21T23:18:23.661999+0000","last_peered":"2024-04-21T23:18:23.661999+0000","last_clean":"2024-04-21T23:18:23.661999+0000","last_became_active":"2024-04-21T23:15:56.355116+0000","last_became_peered":"2024-04-21T23:15:56.355116+0000","last_unstale":"2024-04-21T23:18:23.661999+0000","last_undegraded":"2024-04-21T23:18:23.661999+0000","last_fullsized":"2024-04-21T23:18:23.661999+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:43:20.468549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.497631+0000","last_change":"2024-04-21T23:16:00.439069+0000","last_active":"2024-04-21T23:18:23.497631+0000","last_peered":"2024-04-21T23:18:23.497631+0000","last_clean":"2024-04-21T23:18:23.497631+0000","last_became_active":"2024-04-21T23:16:00.438510+0000","last_became_peered":"2024-04-21T23:16:00.438510+0000","last_unstale":"2024-04-21T23:18:23.497631+0000","last_undegraded":"2024-04-21T23:18:23.497631+0000","last_fullsized":"2024-04-21T23:18:23.497631+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:41:50.814112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"90'2","reported_seq":46,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.663309+0000","last_change":"2024-04-21T23:16:52.206602+0000","last_active":"2024-04-21T23:18:23.663309+0000","last_peered":"2024-04-21T23:18:23.663309+0000","last_clean":"2024-04-21T23:18:23.663309+0000","last_became_active":"2024-04-21T23:16:52.202755+0000","last_became_peered":"2024-04-21T23:16:52.202755+0000","last_unstale":"2024-04-21T23:18:23.663309+0000","last_undegraded":"2024-04-21T23:18:23.663309+0000","last_fullsized":"2024-04-21T23:18:23.663309+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:01:02.437392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.832264+0000","last_change":"2024-04-21T23:15:56.357228+0000","last_active":"2024-04-21T23:18:23.832264+0000","last_peered":"2024-04-21T23:18:23.832264+0000","last_clean":"2024-04-21T23:18:23.832264+0000","last_became_active":"2024-04-21T23:15:56.356808+0000","last_became_peered":"2024-04-21T23:15:56.356808+0000","last_unstale":"2024-04-21T23:18:23.832264+0000","last_undegraded":"2024-04-21T23:18:23.832264+0000","last_fullsized":"2024-04-21T23:18:23.832264+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:15:10.728530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.660434+0000","last_change":"2024-04-21T23:16:00.443569+0000","last_active":"2024-04-21T23:18:23.660434+0000","last_peered":"2024-04-21T23:18:23.660434+0000","last_clean":"2024-04-21T23:18:23.660434+0000","last_became_active":"2024-04-21T23:16:00.442972+0000","last_became_peered":"2024-04-21T23:16:00.442972+0000","last_unstale":"2024-04-21T23:18:23.660434+0000","last_undegraded":"2024-04-21T23:18:23.660434+0000","last_fullsized":"2024-04-21T23:18:23.660434+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:35:21.471407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"90'4","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.507401+0000","last_change":"2024-04-21T23:15:55.400134+0000","last_active":"2024-04-21T23:18:23.507401+0000","last_peered":"2024-04-21T23:18:23.507401+0000","last_clean":"2024-04-21T23:18:23.507401+0000","last_became_active":"2024-04-21T23:15:55.399768+0000","last_became_peered":"2024-04-21T23:15:55.399768+0000","last_unstale":"2024-04-21T23:18:23.507401+0000","last_undegraded":"2024-04-21T23:18:23.507401+0000","last_fullsized":"2024-04-21T23:18:23.507401+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:15:09.323035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"92'11","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.662422+0000","last_change":"2024-04-21T23:15:56.357278+0000","last_active":"2024-04-21T23:18:23.662422+0000","last_peered":"2024-04-21T23:18:23.662422+0000","last_clean":"2024-04-21T23:18:23.662422+0000","last_became_active":"2024-04-21T23:15:56.356902+0000","last_became_peered":"2024-04-21T23:15:56.356902+0000","last_unstale":"2024-04-21T23:18:23.662422+0000","last_undegraded":"2024-04-21T23:18:23.662422+0000","last_fullsized":"2024-04-21T23:18:23.662422+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:19:05.269109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.497842+0000","last_change":"2024-04-21T23:16:00.439704+0000","last_active":"2024-04-21T23:18:23.497842+0000","last_peered":"2024-04-21T23:18:23.497842+0000","last_clean":"2024-04-21T23:18:23.497842+0000","last_became_active":"2024-04-21T23:16:00.439286+0000","last_became_peered":"2024-04-21T23:16:00.439286+0000","last_unstale":"2024-04-21T23:18:23.497842+0000","last_undegraded":"2024-04-21T23:18:23.497842+0000","last_fullsized":"2024-04-21T23:18:23.497842+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:08:59.194481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"95'38","reported_seq":326,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.687640+0000","last_change":"2024-04-21T23:15:55.403344+0000","last_active":"2024-04-21T23:18:23.687640+0000","last_peered":"2024-04-21T23:18:23.687640+0000","last_clean":"2024-04-21T23:18:23.687640+0000","last_became_active":"2024-04-21T23:15:55.402876+0000","last_became_peered":"2024-04-21T23:15:55.402876+0000","last_unstale":"2024-04-21T23:18:23.687640+0000","last_undegraded":"2024-04-21T23:18:23.687640+0000","last_fullsized":"2024-04-21T23:18:23.687640+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:55:59.082964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67428,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":254,"num_read_kb":193,"num_write":39,"num_write_kb":85,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.500611+0000","last_change":"2024-04-21T23:15:56.343897+0000","last_active":"2024-04-21T23:18:23.500611+0000","last_peered":"2024-04-21T23:18:23.500611+0000","last_clean":"2024-04-21T23:18:23.500611+0000","last_became_active":"2024-04-21T23:15:56.343567+0000","last_became_peered":"2024-04-21T23:15:56.343567+0000","last_unstale":"2024-04-21T23:18:23.500611+0000","last_undegraded":"2024-04-21T23:18:23.500611+0000","last_fullsized":"2024-04-21T23:18:23.500611+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:57:07.955722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.507737+0000","last_change":"2024-04-21T23:16:00.432722+0000","last_active":"2024-04-21T23:18:23.507737+0000","last_peered":"2024-04-21T23:18:23.507737+0000","last_clean":"2024-04-21T23:18:23.507737+0000","last_became_active":"2024-04-21T23:16:00.432257+0000","last_became_peered":"2024-04-21T23:16:00.432257+0000","last_unstale":"2024-04-21T23:18:23.507737+0000","last_undegraded":"2024-04-21T23:18:23.507737+0000","last_fullsized":"2024-04-21T23:18:23.507737+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:10:14.804490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"90'6","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.395461+0000","last_change":"2024-04-21T23:15:55.401300+0000","last_active":"2024-04-21T23:18:23.395461+0000","last_peered":"2024-04-21T23:18:23.395461+0000","last_clean":"2024-04-21T23:18:23.395461+0000","last_became_active":"2024-04-21T23:15:55.400904+0000","last_became_peered":"2024-04-21T23:15:55.400904+0000","last_unstale":"2024-04-21T23:18:23.395461+0000","last_undegraded":"2024-04-21T23:18:23.395461+0000","last_fullsized":"2024-04-21T23:18:23.395461+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:29:20.439936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.688806+0000","last_change":"2024-04-21T23:15:56.351046+0000","last_active":"2024-04-21T23:18:23.688806+0000","last_peered":"2024-04-21T23:18:23.688806+0000","last_clean":"2024-04-21T23:18:23.688806+0000","last_became_active":"2024-04-21T23:15:56.350662+0000","last_became_peered":"2024-04-21T23:15:56.350662+0000","last_unstale":"2024-04-21T23:18:23.688806+0000","last_undegraded":"2024-04-21T23:18:23.688806+0000","last_fullsized":"2024-04-21T23:18:23.688806+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:40:16.482061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.398794+0000","last_change":"2024-04-21T23:16:00.439816+0000","last_active":"2024-04-21T23:18:23.398794+0000","last_peered":"2024-04-21T23:18:23.398794+0000","last_clean":"2024-04-21T23:18:23.398794+0000","last_became_active":"2024-04-21T23:16:00.439330+0000","last_became_peered":"2024-04-21T23:16:00.439330+0000","last_unstale":"2024-04-21T23:18:23.398794+0000","last_undegraded":"2024-04-21T23:18:23.398794+0000","last_fullsized":"2024-04-21T23:18:23.398794+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:36:24.472968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"90'3","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.507439+0000","last_change":"2024-04-21T23:15:55.398588+0000","last_active":"2024-04-21T23:18:23.507439+0000","last_peered":"2024-04-21T23:18:23.507439+0000","last_clean":"2024-04-21T23:18:23.507439+0000","last_became_active":"2024-04-21T23:15:55.398264+0000","last_became_peered":"2024-04-21T23:15:55.398264+0000","last_unstale":"2024-04-21T23:18:23.507439+0000","last_undegraded":"2024-04-21T23:18:23.507439+0000","last_fullsized":"2024-04-21T23:18:23.507439+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:48:09.293481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"92'15","reported_seq":53,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.663205+0000","last_change":"2024-04-21T23:16:52.202037+0000","last_active":"2024-04-21T23:18:23.663205+0000","last_peered":"2024-04-21T23:18:23.663205+0000","last_clean":"2024-04-21T23:18:23.663205+0000","last_became_active":"2024-04-21T23:16:52.201244+0000","last_became_peered":"2024-04-21T23:16:52.201244+0000","last_unstale":"2024-04-21T23:18:23.663205+0000","last_undegraded":"2024-04-21T23:18:23.663205+0000","last_fullsized":"2024-04-21T23:18:23.663205+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:37:34.530110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.399332+0000","last_change":"2024-04-21T23:17:53.363902+0000","last_active":"2024-04-21T23:18:23.399332+0000","last_peered":"2024-04-21T23:18:23.399332+0000","last_clean":"2024-04-21T23:18:23.399332+0000","last_became_active":"2024-04-21T23:17:53.363433+0000","last_became_peered":"2024-04-21T23:17:53.363433+0000","last_unstale":"2024-04-21T23:18:23.399332+0000","last_undegraded":"2024-04-21T23:18:23.399332+0000","last_fullsized":"2024-04-21T23:18:23.399332+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:52:07.725946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,0,3],"acting":[10,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.11","version":"95'8","reported_seq":170,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.689594+0000","last_change":"2024-04-21T23:15:55.397503+0000","last_active":"2024-04-21T23:18:23.689594+0000","last_peered":"2024-04-21T23:18:23.689594+0000","last_clean":"2024-04-21T23:18:23.689594+0000","last_became_active":"2024-04-21T23:15:55.397181+0000","last_became_peered":"2024-04-21T23:15:55.397181+0000","last_unstale":"2024-04-21T23:18:23.689594+0000","last_undegraded":"2024-04-21T23:18:23.689594+0000","last_fullsized":"2024-04-21T23:18:23.689594+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:47:08.775783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2210,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":129,"num_read_kb":7,"num_write":7,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.663078+0000","last_change":"2024-04-21T23:15:56.350872+0000","last_active":"2024-04-21T23:18:23.663078+0000","last_peered":"2024-04-21T23:18:23.663078+0000","last_clean":"2024-04-21T23:18:23.663078+0000","last_became_active":"2024-04-21T23:15:56.350519+0000","last_became_peered":"2024-04-21T23:15:56.350519+0000","last_unstale":"2024-04-21T23:18:23.663078+0000","last_undegraded":"2024-04-21T23:18:23.663078+0000","last_fullsized":"2024-04-21T23:18:23.663078+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:23:37.753893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.689394+0000","last_change":"2024-04-21T23:16:00.436069+0000","last_active":"2024-04-21T23:18:23.689394+0000","last_peered":"2024-04-21T23:18:23.689394+0000","last_clean":"2024-04-21T23:18:23.689394+0000","last_became_active":"2024-04-21T23:16:00.435617+0000","last_became_peered":"2024-04-21T23:16:00.435617+0000","last_unstale":"2024-04-21T23:18:23.689394+0000","last_undegraded":"2024-04-21T23:18:23.689394+0000","last_fullsized":"2024-04-21T23:18:23.689394+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:21:44.256928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"90'15","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.398588+0000","last_change":"2024-04-21T23:16:52.220742+0000","last_active":"2024-04-21T23:18:23.398588+0000","last_peered":"2024-04-21T23:18:23.398588+0000","last_clean":"2024-04-21T23:18:23.398588+0000","last_became_active":"2024-04-21T23:16:52.204787+0000","last_became_peered":"2024-04-21T23:16:52.204787+0000","last_unstale":"2024-04-21T23:18:23.398588+0000","last_undegraded":"2024-04-21T23:18:23.398588+0000","last_fullsized":"2024-04-21T23:18:23.398588+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:53:12.057014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"92'6","reported_seq":44,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.507781+0000","last_change":"2024-04-21T23:16:52.202341+0000","last_active":"2024-04-21T23:18:23.507781+0000","last_peered":"2024-04-21T23:18:23.507781+0000","last_clean":"2024-04-21T23:18:23.507781+0000","last_became_active":"2024-04-21T23:16:52.201528+0000","last_became_peered":"2024-04-21T23:16:52.201528+0000","last_unstale":"2024-04-21T23:18:23.507781+0000","last_undegraded":"2024-04-21T23:18:23.507781+0000","last_fullsized":"2024-04-21T23:18:23.507781+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:41:24.767379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.500204+0000","last_change":"2024-04-21T23:17:53.274665+0000","last_active":"2024-04-21T23:18:23.500204+0000","last_peered":"2024-04-21T23:18:23.500204+0000","last_clean":"2024-04-21T23:18:23.500204+0000","last_became_active":"2024-04-21T23:17:53.274146+0000","last_became_peered":"2024-04-21T23:17:53.274146+0000","last_unstale":"2024-04-21T23:18:23.500204+0000","last_undegraded":"2024-04-21T23:18:23.500204+0000","last_fullsized":"2024-04-21T23:18:23.500204+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:15:08.842226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,3,1],"acting":[8,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.f","version":"94'3","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.500747+0000","last_change":"2024-04-21T23:15:55.399032+0000","last_active":"2024-04-21T23:18:23.500747+0000","last_peered":"2024-04-21T23:18:23.500747+0000","last_clean":"2024-04-21T23:18:23.500747+0000","last_became_active":"2024-04-21T23:15:55.398208+0000","last_became_peered":"2024-04-21T23:15:55.398208+0000","last_unstale":"2024-04-21T23:18:23.500747+0000","last_undegraded":"2024-04-21T23:18:23.500747+0000","last_fullsized":"2024-04-21T23:18:23.500747+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:14:59.520571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"92'5","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.399041+0000","last_change":"2024-04-21T23:15:56.355357+0000","last_active":"2024-04-21T23:18:23.399041+0000","last_peered":"2024-04-21T23:18:23.399041+0000","last_clean":"2024-04-21T23:18:23.399041+0000","last_became_active":"2024-04-21T23:15:56.354704+0000","last_became_peered":"2024-04-21T23:15:56.354704+0000","last_unstale":"2024-04-21T23:18:23.399041+0000","last_undegraded":"2024-04-21T23:18:23.399041+0000","last_fullsized":"2024-04-21T23:18:23.399041+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:14:02.368659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.500362+0000","last_change":"2024-04-21T23:16:00.427781+0000","last_active":"2024-04-21T23:18:23.500362+0000","last_peered":"2024-04-21T23:18:23.500362+0000","last_clean":"2024-04-21T23:18:23.500362+0000","last_became_active":"2024-04-21T23:16:00.427244+0000","last_became_peered":"2024-04-21T23:16:00.427244+0000","last_unstale":"2024-04-21T23:18:23.500362+0000","last_undegraded":"2024-04-21T23:18:23.500362+0000","last_fullsized":"2024-04-21T23:18:23.500362+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:21:30.548865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.498142+0000","last_change":"2024-04-21T23:15:55.405687+0000","last_active":"2024-04-21T23:18:23.498142+0000","last_peered":"2024-04-21T23:18:23.498142+0000","last_clean":"2024-04-21T23:18:23.498142+0000","last_became_active":"2024-04-21T23:15:55.405379+0000","last_became_peered":"2024-04-21T23:15:55.405379+0000","last_unstale":"2024-04-21T23:18:23.498142+0000","last_undegraded":"2024-04-21T23:18:23.498142+0000","last_fullsized":"2024-04-21T23:18:23.498142+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:12:44.192092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"92'4","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.399301+0000","last_change":"2024-04-21T23:15:56.348989+0000","last_active":"2024-04-21T23:18:23.399301+0000","last_peered":"2024-04-21T23:18:23.399301+0000","last_clean":"2024-04-21T23:18:23.399301+0000","last_became_active":"2024-04-21T23:15:56.348039+0000","last_became_peered":"2024-04-21T23:15:56.348039+0000","last_unstale":"2024-04-21T23:18:23.399301+0000","last_undegraded":"2024-04-21T23:18:23.399301+0000","last_fullsized":"2024-04-21T23:18:23.399301+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:39:15.577567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.688857+0000","last_change":"2024-04-21T23:16:00.434557+0000","last_active":"2024-04-21T23:18:23.688857+0000","last_peered":"2024-04-21T23:18:23.688857+0000","last_clean":"2024-04-21T23:18:23.688857+0000","last_became_active":"2024-04-21T23:16:00.433737+0000","last_became_peered":"2024-04-21T23:16:00.433737+0000","last_unstale":"2024-04-21T23:18:23.688857+0000","last_undegraded":"2024-04-21T23:18:23.688857+0000","last_fullsized":"2024-04-21T23:18:23.688857+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:38:00.295054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"90'1","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.662784+0000","last_change":"2024-04-21T23:15:55.396792+0000","last_active":"2024-04-21T23:18:23.662784+0000","last_peered":"2024-04-21T23:18:23.662784+0000","last_clean":"2024-04-21T23:18:23.662784+0000","last_became_active":"2024-04-21T23:15:55.395740+0000","last_became_peered":"2024-04-21T23:15:55.395740+0000","last_unstale":"2024-04-21T23:18:23.662784+0000","last_undegraded":"2024-04-21T23:18:23.662784+0000","last_fullsized":"2024-04-21T23:18:23.662784+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:39:22.366310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.399163+0000","last_change":"2024-04-21T23:15:56.346372+0000","last_active":"2024-04-21T23:18:23.399163+0000","last_peered":"2024-04-21T23:18:23.399163+0000","last_clean":"2024-04-21T23:18:23.399163+0000","last_became_active":"2024-04-21T23:15:56.346047+0000","last_became_peered":"2024-04-21T23:15:56.346047+0000","last_unstale":"2024-04-21T23:18:23.399163+0000","last_undegraded":"2024-04-21T23:18:23.399163+0000","last_fullsized":"2024-04-21T23:18:23.399163+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:32:29.684227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.689210+0000","last_change":"2024-04-21T23:16:00.434747+0000","last_active":"2024-04-21T23:18:23.689210+0000","last_peered":"2024-04-21T23:18:23.689210+0000","last_clean":"2024-04-21T23:18:23.689210+0000","last_became_active":"2024-04-21T23:16:00.433950+0000","last_became_peered":"2024-04-21T23:16:00.433950+0000","last_unstale":"2024-04-21T23:18:23.689210+0000","last_undegraded":"2024-04-21T23:18:23.689210+0000","last_fullsized":"2024-04-21T23:18:23.689210+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:02:10.562135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"92'8","reported_seq":46,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.507442+0000","last_change":"2024-04-21T23:16:52.206022+0000","last_active":"2024-04-21T23:18:23.507442+0000","last_peered":"2024-04-21T23:18:23.507442+0000","last_clean":"2024-04-21T23:18:23.507442+0000","last_became_active":"2024-04-21T23:16:52.205469+0000","last_became_peered":"2024-04-21T23:16:52.205469+0000","last_unstale":"2024-04-21T23:18:23.507442+0000","last_undegraded":"2024-04-21T23:18:23.507442+0000","last_fullsized":"2024-04-21T23:18:23.507442+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:34:46.264003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,8],"acting":[5,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"94'6","reported_seq":162,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.688568+0000","last_change":"2024-04-21T23:15:55.394900+0000","last_active":"2024-04-21T23:18:23.688568+0000","last_peered":"2024-04-21T23:18:23.688568+0000","last_clean":"2024-04-21T23:18:23.688568+0000","last_became_active":"2024-04-21T23:15:55.393556+0000","last_became_peered":"2024-04-21T23:15:55.393556+0000","last_unstale":"2024-04-21T23:18:23.688568+0000","last_undegraded":"2024-04-21T23:18:23.688568+0000","last_fullsized":"2024-04-21T23:18:23.688568+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:16:24.691930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":122,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":26,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.662759+0000","last_change":"2024-04-21T23:16:00.443338+0000","last_active":"2024-04-21T23:18:23.662759+0000","last_peered":"2024-04-21T23:18:23.662759+0000","last_clean":"2024-04-21T23:18:23.662759+0000","last_became_active":"2024-04-21T23:16:00.442855+0000","last_became_peered":"2024-04-21T23:16:00.442855+0000","last_unstale":"2024-04-21T23:18:23.662759+0000","last_undegraded":"2024-04-21T23:18:23.662759+0000","last_fullsized":"2024-04-21T23:18:23.662759+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:04:36.763027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.500267+0000","last_change":"2024-04-21T23:15:56.356788+0000","last_active":"2024-04-21T23:18:23.500267+0000","last_peered":"2024-04-21T23:18:23.500267+0000","last_clean":"2024-04-21T23:18:23.500267+0000","last_became_active":"2024-04-21T23:15:56.356474+0000","last_became_peered":"2024-04-21T23:15:56.356474+0000","last_unstale":"2024-04-21T23:18:23.500267+0000","last_undegraded":"2024-04-21T23:18:23.500267+0000","last_fullsized":"2024-04-21T23:18:23.500267+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:34:33.700897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"90'3","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.660276+0000","last_change":"2024-04-21T23:15:55.395398+0000","last_active":"2024-04-21T23:18:23.660276+0000","last_peered":"2024-04-21T23:18:23.660276+0000","last_clean":"2024-04-21T23:18:23.660276+0000","last_became_active":"2024-04-21T23:15:55.394978+0000","last_became_peered":"2024-04-21T23:15:55.394978+0000","last_unstale":"2024-04-21T23:18:23.660276+0000","last_undegraded":"2024-04-21T23:18:23.660276+0000","last_fullsized":"2024-04-21T23:18:23.660276+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:07:01.266885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":368,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.662639+0000","last_change":"2024-04-21T23:14:42.877959+0000","last_active":"2024-04-21T23:18:23.662639+0000","last_peered":"2024-04-21T23:18:23.662639+0000","last_clean":"2024-04-21T23:18:23.662639+0000","last_became_active":"2024-04-21T23:14:42.877384+0000","last_became_peered":"2024-04-21T23:14:42.877384+0000","last_unstale":"2024-04-21T23:18:23.662639+0000","last_undegraded":"2024-04-21T23:18:23.662639+0000","last_fullsized":"2024-04-21T23:18:23.662639+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:11:15.009349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:11:15.009349+0000","last_clean_scrub_stamp":"2024-04-21T23:11:15.009349+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:18:55.621052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.507306+0000","last_change":"2024-04-21T23:16:00.444576+0000","last_active":"2024-04-21T23:18:23.507306+0000","last_peered":"2024-04-21T23:18:23.507306+0000","last_clean":"2024-04-21T23:18:23.507306+0000","last_became_active":"2024-04-21T23:16:00.444153+0000","last_became_peered":"2024-04-21T23:16:00.444153+0000","last_unstale":"2024-04-21T23:18:23.507306+0000","last_undegraded":"2024-04-21T23:18:23.507306+0000","last_fullsized":"2024-04-21T23:18:23.507306+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:11:48.040383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"92'5","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.688448+0000","last_change":"2024-04-21T23:15:56.349870+0000","last_active":"2024-04-21T23:18:23.688448+0000","last_peered":"2024-04-21T23:18:23.688448+0000","last_clean":"2024-04-21T23:18:23.688448+0000","last_became_active":"2024-04-21T23:15:56.349535+0000","last_became_peered":"2024-04-21T23:15:56.349535+0000","last_unstale":"2024-04-21T23:18:23.688448+0000","last_undegraded":"2024-04-21T23:18:23.688448+0000","last_fullsized":"2024-04-21T23:18:23.688448+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:12:50.702556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"90'3","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.662748+0000","last_change":"2024-04-21T23:15:55.402406+0000","last_active":"2024-04-21T23:18:23.662748+0000","last_peered":"2024-04-21T23:18:23.662748+0000","last_clean":"2024-04-21T23:18:23.662748+0000","last_became_active":"2024-04-21T23:15:55.402075+0000","last_became_peered":"2024-04-21T23:15:55.402075+0000","last_unstale":"2024-04-21T23:18:23.662748+0000","last_undegraded":"2024-04-21T23:18:23.662748+0000","last_fullsized":"2024-04-21T23:18:23.662748+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:44:18.013031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.398753+0000","last_change":"2024-04-21T23:16:00.439206+0000","last_active":"2024-04-21T23:18:23.398753+0000","last_peered":"2024-04-21T23:18:23.398753+0000","last_clean":"2024-04-21T23:18:23.398753+0000","last_became_active":"2024-04-21T23:16:00.438601+0000","last_became_peered":"2024-04-21T23:16:00.438601+0000","last_unstale":"2024-04-21T23:18:23.398753+0000","last_undegraded":"2024-04-21T23:18:23.398753+0000","last_fullsized":"2024-04-21T23:18:23.398753+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:58:46.771456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"90'1","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.500582+0000","last_change":"2024-04-21T23:15:55.401154+0000","last_active":"2024-04-21T23:18:23.500582+0000","last_peered":"2024-04-21T23:18:23.500582+0000","last_clean":"2024-04-21T23:18:23.500582+0000","last_became_active":"2024-04-21T23:15:55.400579+0000","last_became_peered":"2024-04-21T23:15:55.400579+0000","last_unstale":"2024-04-21T23:18:23.500582+0000","last_undegraded":"2024-04-21T23:18:23.500582+0000","last_fullsized":"2024-04-21T23:18:23.500582+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:20:03.879660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.689222+0000","last_change":"2024-04-21T23:15:56.349837+0000","last_active":"2024-04-21T23:18:23.689222+0000","last_peered":"2024-04-21T23:18:23.689222+0000","last_clean":"2024-04-21T23:18:23.689222+0000","last_became_active":"2024-04-21T23:15:56.349466+0000","last_became_peered":"2024-04-21T23:15:56.349466+0000","last_unstale":"2024-04-21T23:18:23.689222+0000","last_undegraded":"2024-04-21T23:18:23.689222+0000","last_fullsized":"2024-04-21T23:18:23.689222+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:15:32.675770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.500545+0000","last_change":"2024-04-21T23:16:00.432599+0000","last_active":"2024-04-21T23:18:23.500545+0000","last_peered":"2024-04-21T23:18:23.500545+0000","last_clean":"2024-04-21T23:18:23.500545+0000","last_became_active":"2024-04-21T23:16:00.432194+0000","last_became_peered":"2024-04-21T23:16:00.432194+0000","last_unstale":"2024-04-21T23:18:23.500545+0000","last_undegraded":"2024-04-21T23:18:23.500545+0000","last_fullsized":"2024-04-21T23:18:23.500545+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:10:21.354338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"95'44","reported_seq":101,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.615014+0000","last_change":"2024-04-21T23:15:55.400328+0000","last_active":"2024-04-21T23:18:23.615014+0000","last_peered":"2024-04-21T23:18:23.615014+0000","last_clean":"2024-04-21T23:18:23.615014+0000","last_became_active":"2024-04-21T23:15:55.399991+0000","last_became_peered":"2024-04-21T23:15:55.399991+0000","last_unstale":"2024-04-21T23:18:23.615014+0000","last_undegraded":"2024-04-21T23:18:23.615014+0000","last_fullsized":"2024-04-21T23:18:23.615014+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:58:54.480720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":78,"num_write_kb":836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"92'5","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.501203+0000","last_change":"2024-04-21T23:15:56.358698+0000","last_active":"2024-04-21T23:18:23.501203+0000","last_peered":"2024-04-21T23:18:23.501203+0000","last_clean":"2024-04-21T23:18:23.501203+0000","last_became_active":"2024-04-21T23:15:56.358364+0000","last_became_peered":"2024-04-21T23:15:56.358364+0000","last_unstale":"2024-04-21T23:18:23.501203+0000","last_undegraded":"2024-04-21T23:18:23.501203+0000","last_fullsized":"2024-04-21T23:18:23.501203+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:34:15.281142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.395058+0000","last_change":"2024-04-21T23:16:00.432237+0000","last_active":"2024-04-21T23:18:23.395058+0000","last_peered":"2024-04-21T23:18:23.395058+0000","last_clean":"2024-04-21T23:18:23.395058+0000","last_became_active":"2024-04-21T23:16:00.431745+0000","last_became_peered":"2024-04-21T23:16:00.431745+0000","last_unstale":"2024-04-21T23:18:23.395058+0000","last_undegraded":"2024-04-21T23:18:23.395058+0000","last_fullsized":"2024-04-21T23:18:23.395058+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:25:00.477154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"90'4","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.661315+0000","last_change":"2024-04-21T23:15:55.398667+0000","last_active":"2024-04-21T23:18:23.661315+0000","last_peered":"2024-04-21T23:18:23.661315+0000","last_clean":"2024-04-21T23:18:23.661315+0000","last_became_active":"2024-04-21T23:15:55.398290+0000","last_became_peered":"2024-04-21T23:15:55.398290+0000","last_unstale":"2024-04-21T23:18:23.661315+0000","last_undegraded":"2024-04-21T23:18:23.661315+0000","last_fullsized":"2024-04-21T23:18:23.661315+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:57:58.992615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.832119+0000","last_change":"2024-04-21T23:15:56.359518+0000","last_active":"2024-04-21T23:18:23.832119+0000","last_peered":"2024-04-21T23:18:23.832119+0000","last_clean":"2024-04-21T23:18:23.832119+0000","last_became_active":"2024-04-21T23:15:56.358970+0000","last_became_peered":"2024-04-21T23:15:56.358970+0000","last_unstale":"2024-04-21T23:18:23.832119+0000","last_undegraded":"2024-04-21T23:18:23.832119+0000","last_fullsized":"2024-04-21T23:18:23.832119+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:41:22.113877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.660034+0000","last_change":"2024-04-21T23:16:00.443380+0000","last_active":"2024-04-21T23:18:23.660034+0000","last_peered":"2024-04-21T23:18:23.660034+0000","last_clean":"2024-04-21T23:18:23.660034+0000","last_became_active":"2024-04-21T23:16:00.442896+0000","last_became_peered":"2024-04-21T23:16:00.442896+0000","last_unstale":"2024-04-21T23:18:23.660034+0000","last_undegraded":"2024-04-21T23:18:23.660034+0000","last_fullsized":"2024-04-21T23:18:23.660034+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:51:00.089766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"90'2","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.660516+0000","last_change":"2024-04-21T23:15:55.395287+0000","last_active":"2024-04-21T23:18:23.660516+0000","last_peered":"2024-04-21T23:18:23.660516+0000","last_clean":"2024-04-21T23:18:23.660516+0000","last_became_active":"2024-04-21T23:15:55.394908+0000","last_became_peered":"2024-04-21T23:15:55.394908+0000","last_unstale":"2024-04-21T23:18:23.660516+0000","last_undegraded":"2024-04-21T23:18:23.660516+0000","last_fullsized":"2024-04-21T23:18:23.660516+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:57:29.915581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"92'2","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.500115+0000","last_change":"2024-04-21T23:15:56.348016+0000","last_active":"2024-04-21T23:18:23.500115+0000","last_peered":"2024-04-21T23:18:23.500115+0000","last_clean":"2024-04-21T23:18:23.500115+0000","last_became_active":"2024-04-21T23:15:56.347630+0000","last_became_peered":"2024-04-21T23:15:56.347630+0000","last_unstale":"2024-04-21T23:18:23.500115+0000","last_undegraded":"2024-04-21T23:18:23.500115+0000","last_fullsized":"2024-04-21T23:18:23.500115+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:17:27.372880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.500678+0000","last_change":"2024-04-21T23:17:52.925060+0000","last_active":"2024-04-21T23:18:23.500678+0000","last_peered":"2024-04-21T23:18:23.500678+0000","last_clean":"2024-04-21T23:18:23.500678+0000","last_became_active":"2024-04-21T23:17:52.924560+0000","last_became_peered":"2024-04-21T23:17:52.924560+0000","last_unstale":"2024-04-21T23:18:23.500678+0000","last_undegraded":"2024-04-21T23:18:23.500678+0000","last_fullsized":"2024-04-21T23:18:23.500678+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:24:19.265334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,5],"acting":[8,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"90'3","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.500520+0000","last_change":"2024-04-21T23:15:55.400859+0000","last_active":"2024-04-21T23:18:23.500520+0000","last_peered":"2024-04-21T23:18:23.500520+0000","last_clean":"2024-04-21T23:18:23.500520+0000","last_became_active":"2024-04-21T23:15:55.400460+0000","last_became_peered":"2024-04-21T23:15:55.400460+0000","last_unstale":"2024-04-21T23:18:23.500520+0000","last_undegraded":"2024-04-21T23:18:23.500520+0000","last_fullsized":"2024-04-21T23:18:23.500520+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:31:14.113476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.398661+0000","last_change":"2024-04-21T23:15:56.350354+0000","last_active":"2024-04-21T23:18:23.398661+0000","last_peered":"2024-04-21T23:18:23.398661+0000","last_clean":"2024-04-21T23:18:23.398661+0000","last_became_active":"2024-04-21T23:15:56.349954+0000","last_became_peered":"2024-04-21T23:15:56.349954+0000","last_unstale":"2024-04-21T23:18:23.398661+0000","last_undegraded":"2024-04-21T23:18:23.398661+0000","last_fullsized":"2024-04-21T23:18:23.398661+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:42:59.165610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":26,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.660123+0000","last_change":"2024-04-21T23:16:00.444829+0000","last_active":"2024-04-21T23:18:23.660123+0000","last_peered":"2024-04-21T23:18:23.660123+0000","last_clean":"2024-04-21T23:18:23.660123+0000","last_became_active":"2024-04-21T23:16:00.444365+0000","last_became_peered":"2024-04-21T23:16:00.444365+0000","last_unstale":"2024-04-21T23:18:23.660123+0000","last_undegraded":"2024-04-21T23:18:23.660123+0000","last_fullsized":"2024-04-21T23:18:23.660123+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:23:16.747684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"90'1","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.395250+0000","last_change":"2024-04-21T23:15:55.397855+0000","last_active":"2024-04-21T23:18:23.395250+0000","last_peered":"2024-04-21T23:18:23.395250+0000","last_clean":"2024-04-21T23:18:23.395250+0000","last_became_active":"2024-04-21T23:15:55.397498+0000","last_became_peered":"2024-04-21T23:15:55.397498+0000","last_unstale":"2024-04-21T23:18:23.395250+0000","last_undegraded":"2024-04-21T23:18:23.395250+0000","last_fullsized":"2024-04-21T23:18:23.395250+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:45:44.680350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.500943+0000","last_change":"2024-04-21T23:15:56.353023+0000","last_active":"2024-04-21T23:18:23.500943+0000","last_peered":"2024-04-21T23:18:23.500943+0000","last_clean":"2024-04-21T23:18:23.500943+0000","last_became_active":"2024-04-21T23:15:56.352209+0000","last_became_peered":"2024-04-21T23:15:56.352209+0000","last_unstale":"2024-04-21T23:18:23.500943+0000","last_undegraded":"2024-04-21T23:18:23.500943+0000","last_fullsized":"2024-04-21T23:18:23.500943+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:20:58.075627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.660994+0000","last_change":"2024-04-21T23:16:00.429258+0000","last_active":"2024-04-21T23:18:23.660994+0000","last_peered":"2024-04-21T23:18:23.660994+0000","last_clean":"2024-04-21T23:18:23.660994+0000","last_became_active":"2024-04-21T23:16:00.428856+0000","last_became_peered":"2024-04-21T23:16:00.428856+0000","last_unstale":"2024-04-21T23:18:23.660994+0000","last_undegraded":"2024-04-21T23:18:23.660994+0000","last_fullsized":"2024-04-21T23:18:23.660994+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:07:44.608370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"90'4","reported_seq":23,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.398659+0000","last_change":"2024-04-21T23:16:52.209766+0000","last_active":"2024-04-21T23:18:23.398659+0000","last_peered":"2024-04-21T23:18:23.398659+0000","last_clean":"2024-04-21T23:18:23.398659+0000","last_became_active":"2024-04-21T23:16:52.204654+0000","last_became_peered":"2024-04-21T23:16:52.204654+0000","last_unstale":"2024-04-21T23:18:23.398659+0000","last_undegraded":"2024-04-21T23:18:23.398659+0000","last_fullsized":"2024-04-21T23:18:23.398659+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:40:41.230460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"92'5","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.662982+0000","last_change":"2024-04-21T23:15:56.352024+0000","last_active":"2024-04-21T23:18:23.662982+0000","last_peered":"2024-04-21T23:18:23.662982+0000","last_clean":"2024-04-21T23:18:23.662982+0000","last_became_active":"2024-04-21T23:15:56.351052+0000","last_became_peered":"2024-04-21T23:15:56.351052+0000","last_unstale":"2024-04-21T23:18:23.662982+0000","last_undegraded":"2024-04-21T23:18:23.662982+0000","last_fullsized":"2024-04-21T23:18:23.662982+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:36:42.777601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.660850+0000","last_change":"2024-04-21T23:16:00.432371+0000","last_active":"2024-04-21T23:18:23.660850+0000","last_peered":"2024-04-21T23:18:23.660850+0000","last_clean":"2024-04-21T23:18:23.660850+0000","last_became_active":"2024-04-21T23:16:00.431828+0000","last_became_peered":"2024-04-21T23:16:00.431828+0000","last_unstale":"2024-04-21T23:18:23.660850+0000","last_undegraded":"2024-04-21T23:18:23.660850+0000","last_fullsized":"2024-04-21T23:18:23.660850+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:02:06.500075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"90'3","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.395651+0000","last_change":"2024-04-21T23:15:55.407018+0000","last_active":"2024-04-21T23:18:23.395651+0000","last_peered":"2024-04-21T23:18:23.395651+0000","last_clean":"2024-04-21T23:18:23.395651+0000","last_became_active":"2024-04-21T23:15:55.406709+0000","last_became_peered":"2024-04-21T23:15:55.406709+0000","last_unstale":"2024-04-21T23:18:23.395651+0000","last_undegraded":"2024-04-21T23:18:23.395651+0000","last_fullsized":"2024-04-21T23:18:23.395651+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:10:40.138470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.663535+0000","last_change":"2024-04-21T23:15:56.358569+0000","last_active":"2024-04-21T23:18:23.663535+0000","last_peered":"2024-04-21T23:18:23.663535+0000","last_clean":"2024-04-21T23:18:23.663535+0000","last_became_active":"2024-04-21T23:15:56.357533+0000","last_became_peered":"2024-04-21T23:15:56.357533+0000","last_unstale":"2024-04-21T23:18:23.663535+0000","last_undegraded":"2024-04-21T23:18:23.663535+0000","last_fullsized":"2024-04-21T23:18:23.663535+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:59:58.525340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.832026+0000","last_change":"2024-04-21T23:16:00.443981+0000","last_active":"2024-04-21T23:18:23.832026+0000","last_peered":"2024-04-21T23:18:23.832026+0000","last_clean":"2024-04-21T23:18:23.832026+0000","last_became_active":"2024-04-21T23:16:00.443368+0000","last_became_peered":"2024-04-21T23:16:00.443368+0000","last_unstale":"2024-04-21T23:18:23.832026+0000","last_undegraded":"2024-04-21T23:18:23.832026+0000","last_fullsized":"2024-04-21T23:18:23.832026+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:56:48.776540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"95'39","reported_seq":81,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.612653+0000","last_change":"2024-04-21T23:15:55.397334+0000","last_active":"2024-04-21T23:18:23.612653+0000","last_peered":"2024-04-21T23:18:23.612653+0000","last_clean":"2024-04-21T23:18:23.612653+0000","last_became_active":"2024-04-21T23:15:55.396745+0000","last_became_peered":"2024-04-21T23:15:55.396745+0000","last_unstale":"2024-04-21T23:18:23.612653+0000","last_undegraded":"2024-04-21T23:18:23.612653+0000","last_fullsized":"2024-04-21T23:18:23.612653+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:56:21.042187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.663027+0000","last_change":"2024-04-21T23:15:56.351012+0000","last_active":"2024-04-21T23:18:23.663027+0000","last_peered":"2024-04-21T23:18:23.663027+0000","last_clean":"2024-04-21T23:18:23.663027+0000","last_became_active":"2024-04-21T23:15:56.350042+0000","last_became_peered":"2024-04-21T23:15:56.350042+0000","last_unstale":"2024-04-21T23:18:23.663027+0000","last_undegraded":"2024-04-21T23:18:23.663027+0000","last_fullsized":"2024-04-21T23:18:23.663027+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:47:02.629870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.507013+0000","last_change":"2024-04-21T23:16:00.428775+0000","last_active":"2024-04-21T23:18:23.507013+0000","last_peered":"2024-04-21T23:18:23.507013+0000","last_clean":"2024-04-21T23:18:23.507013+0000","last_became_active":"2024-04-21T23:16:00.428278+0000","last_became_peered":"2024-04-21T23:16:00.428278+0000","last_unstale":"2024-04-21T23:18:23.507013+0000","last_undegraded":"2024-04-21T23:18:23.507013+0000","last_fullsized":"2024-04-21T23:18:23.507013+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:24:04.397387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.689442+0000","last_change":"2024-04-21T23:15:55.395524+0000","last_active":"2024-04-21T23:18:23.689442+0000","last_peered":"2024-04-21T23:18:23.689442+0000","last_clean":"2024-04-21T23:18:23.689442+0000","last_became_active":"2024-04-21T23:15:55.395220+0000","last_became_peered":"2024-04-21T23:15:55.395220+0000","last_unstale":"2024-04-21T23:18:23.689442+0000","last_undegraded":"2024-04-21T23:18:23.689442+0000","last_fullsized":"2024-04-21T23:18:23.689442+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:01:37.517620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.663388+0000","last_change":"2024-04-21T23:15:56.352128+0000","last_active":"2024-04-21T23:18:23.663388+0000","last_peered":"2024-04-21T23:18:23.663388+0000","last_clean":"2024-04-21T23:18:23.663388+0000","last_became_active":"2024-04-21T23:15:56.349349+0000","last_became_peered":"2024-04-21T23:15:56.349349+0000","last_unstale":"2024-04-21T23:18:23.663388+0000","last_undegraded":"2024-04-21T23:18:23.663388+0000","last_fullsized":"2024-04-21T23:18:23.663388+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:12:23.247084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.398793+0000","last_change":"2024-04-21T23:15:56.350830+0000","last_active":"2024-04-21T23:18:23.398793+0000","last_peered":"2024-04-21T23:18:23.398793+0000","last_clean":"2024-04-21T23:18:23.398793+0000","last_became_active":"2024-04-21T23:15:56.350385+0000","last_became_peered":"2024-04-21T23:15:56.350385+0000","last_unstale":"2024-04-21T23:18:23.398793+0000","last_undegraded":"2024-04-21T23:18:23.398793+0000","last_fullsized":"2024-04-21T23:18:23.398793+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:18:48.242375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"92'12","reported_seq":44,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.395348+0000","last_change":"2024-04-21T23:15:56.350947+0000","last_active":"2024-04-21T23:18:23.395348+0000","last_peered":"2024-04-21T23:18:23.395348+0000","last_clean":"2024-04-21T23:18:23.395348+0000","last_became_active":"2024-04-21T23:15:56.350383+0000","last_became_peered":"2024-04-21T23:15:56.350383+0000","last_unstale":"2024-04-21T23:18:23.395348+0000","last_undegraded":"2024-04-21T23:18:23.395348+0000","last_fullsized":"2024-04-21T23:18:23.395348+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:07:28.879822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.395154+0000","last_change":"2024-04-21T23:15:56.354559+0000","last_active":"2024-04-21T23:18:23.395154+0000","last_peered":"2024-04-21T23:18:23.395154+0000","last_clean":"2024-04-21T23:18:23.395154+0000","last_became_active":"2024-04-21T23:15:56.354205+0000","last_became_peered":"2024-04-21T23:15:56.354205+0000","last_unstale":"2024-04-21T23:18:23.395154+0000","last_undegraded":"2024-04-21T23:18:23.395154+0000","last_fullsized":"2024-04-21T23:18:23.395154+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:33:50.157971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.832253+0000","last_change":"2024-04-21T23:15:56.358395+0000","last_active":"2024-04-21T23:18:23.832253+0000","last_peered":"2024-04-21T23:18:23.832253+0000","last_clean":"2024-04-21T23:18:23.832253+0000","last_became_active":"2024-04-21T23:15:56.358077+0000","last_became_peered":"2024-04-21T23:15:56.358077+0000","last_unstale":"2024-04-21T23:18:23.832253+0000","last_undegraded":"2024-04-21T23:18:23.832253+0000","last_fullsized":"2024-04-21T23:18:23.832253+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:12:43.124588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"92'4","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.689166+0000","last_change":"2024-04-21T23:15:56.347476+0000","last_active":"2024-04-21T23:18:23.689166+0000","last_peered":"2024-04-21T23:18:23.689166+0000","last_clean":"2024-04-21T23:18:23.689166+0000","last_became_active":"2024-04-21T23:15:56.347097+0000","last_became_peered":"2024-04-21T23:15:56.347097+0000","last_unstale":"2024-04-21T23:18:23.689166+0000","last_undegraded":"2024-04-21T23:18:23.689166+0000","last_fullsized":"2024-04-21T23:18:23.689166+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:11:58.715526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.399175+0000","last_change":"2024-04-21T23:15:56.346413+0000","last_active":"2024-04-21T23:18:23.399175+0000","last_peered":"2024-04-21T23:18:23.399175+0000","last_clean":"2024-04-21T23:18:23.399175+0000","last_became_active":"2024-04-21T23:15:56.346056+0000","last_became_peered":"2024-04-21T23:15:56.346056+0000","last_unstale":"2024-04-21T23:18:23.399175+0000","last_undegraded":"2024-04-21T23:18:23.399175+0000","last_fullsized":"2024-04-21T23:18:23.399175+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:47:16.340323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"92'15","reported_seq":47,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.395818+0000","last_change":"2024-04-21T23:15:56.354876+0000","last_active":"2024-04-21T23:18:23.395818+0000","last_peered":"2024-04-21T23:18:23.395818+0000","last_clean":"2024-04-21T23:18:23.395818+0000","last_became_active":"2024-04-21T23:15:56.354525+0000","last_became_peered":"2024-04-21T23:15:56.354525+0000","last_unstale":"2024-04-21T23:18:23.395818+0000","last_undegraded":"2024-04-21T23:18:23.395818+0000","last_fullsized":"2024-04-21T23:18:23.395818+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:09:23.662664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"92'2","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.501341+0000","last_change":"2024-04-21T23:15:56.358456+0000","last_active":"2024-04-21T23:18:23.501341+0000","last_peered":"2024-04-21T23:18:23.501341+0000","last_clean":"2024-04-21T23:18:23.501341+0000","last_became_active":"2024-04-21T23:15:56.358060+0000","last_became_peered":"2024-04-21T23:15:56.358060+0000","last_unstale":"2024-04-21T23:18:23.501341+0000","last_undegraded":"2024-04-21T23:18:23.501341+0000","last_fullsized":"2024-04-21T23:18:23.501341+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:47:58.928438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.507107+0000","last_change":"2024-04-21T23:15:56.352870+0000","last_active":"2024-04-21T23:18:23.507107+0000","last_peered":"2024-04-21T23:18:23.507107+0000","last_clean":"2024-04-21T23:18:23.507107+0000","last_became_active":"2024-04-21T23:15:56.352267+0000","last_became_peered":"2024-04-21T23:15:56.352267+0000","last_unstale":"2024-04-21T23:18:23.507107+0000","last_undegraded":"2024-04-21T23:18:23.507107+0000","last_fullsized":"2024-04-21T23:18:23.507107+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:32:22.244773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.500521+0000","last_change":"2024-04-21T23:15:56.357468+0000","last_active":"2024-04-21T23:18:23.500521+0000","last_peered":"2024-04-21T23:18:23.500521+0000","last_clean":"2024-04-21T23:18:23.500521+0000","last_became_active":"2024-04-21T23:15:56.357104+0000","last_became_peered":"2024-04-21T23:15:56.357104+0000","last_unstale":"2024-04-21T23:18:23.500521+0000","last_undegraded":"2024-04-21T23:18:23.500521+0000","last_fullsized":"2024-04-21T23:18:23.500521+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:37:36.884036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"92'10","reported_seq":48,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.661120+0000","last_change":"2024-04-21T23:16:52.203142+0000","last_active":"2024-04-21T23:18:23.661120+0000","last_peered":"2024-04-21T23:18:23.661120+0000","last_clean":"2024-04-21T23:18:23.661120+0000","last_became_active":"2024-04-21T23:16:52.202558+0000","last_became_peered":"2024-04-21T23:16:52.202558+0000","last_unstale":"2024-04-21T23:18:23.661120+0000","last_undegraded":"2024-04-21T23:18:23.661120+0000","last_fullsized":"2024-04-21T23:18:23.661120+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:29:05.428123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,8],"acting":[1,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"92'11","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.688864+0000","last_change":"2024-04-21T23:15:56.351901+0000","last_active":"2024-04-21T23:18:23.688864+0000","last_peered":"2024-04-21T23:18:23.688864+0000","last_clean":"2024-04-21T23:18:23.688864+0000","last_became_active":"2024-04-21T23:15:56.351545+0000","last_became_peered":"2024-04-21T23:15:56.351545+0000","last_unstale":"2024-04-21T23:18:23.688864+0000","last_undegraded":"2024-04-21T23:18:23.688864+0000","last_fullsized":"2024-04-21T23:18:23.688864+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:37:20.855631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.398840+0000","last_change":"2024-04-21T23:15:56.355773+0000","last_active":"2024-04-21T23:18:23.398840+0000","last_peered":"2024-04-21T23:18:23.398840+0000","last_clean":"2024-04-21T23:18:23.398840+0000","last_became_active":"2024-04-21T23:15:56.355269+0000","last_became_peered":"2024-04-21T23:15:56.355269+0000","last_unstale":"2024-04-21T23:18:23.398840+0000","last_undegraded":"2024-04-21T23:18:23.398840+0000","last_fullsized":"2024-04-21T23:18:23.398840+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:04:54.540817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":597867,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":854,"num_read_kb":536,"num_write":288,"num_write_kb":1440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":112,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2027520,"data_stored":1793601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":228,"ondisk_log_size":228,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351018,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1093632,"data_stored":536470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612656,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37268,"kb_used_data":1872,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93707884,"statfs":{"total":95995035648,"available":95956873216,"internally_reserved":0,"allocated":1916928,"data_stored":1324276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808885,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36804,"kb_used_data":1464,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708348,"statfs":{"total":95995035648,"available":95957348352,"internally_reserved":0,"allocated":1499136,"data_stored":933906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070522,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28472,"kb_used_data":1452,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716680,"statfs":{"total":95995035648,"available":95965880320,"internally_reserved":0,"allocated":1486848,"data_stored":929618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332159,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28676,"kb_used_data":1656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716476,"statfs":{"total":95995035648,"available":95965671424,"internally_reserved":0,"allocated":1695744,"data_stored":1134045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495684,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1081344,"data_stored":531770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574942,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28148,"kb_used_data":1128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717004,"statfs":{"total":95995035648,"available":95966212096,"internally_reserved":0,"allocated":1155072,"data_stored":599108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738466,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28100,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717052,"statfs":{"total":95995035648,"available":95966261248,"internally_reserved":0,"allocated":1105920,"data_stored":533158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411417,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28168,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716984,"statfs":{"total":95995035648,"available":95966191616,"internally_reserved":0,"allocated":1175552,"data_stored":595362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182483,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36920,"kb_used_data":1584,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708232,"statfs":{"total":95995035648,"available":95957229568,"internally_reserved":0,"allocated":1622016,"data_stored":1056780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920847,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1077248,"data_stored":527336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757321,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37132,"kb_used_data":1796,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708020,"statfs":{"total":95995035648,"available":95957012480,"internally_reserved":0,"allocated":1839104,"data_stored":1259611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":67428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":69169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":544768,"data_stored":525100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":133467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":393842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":394034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":202228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-21T23:18:28.515 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-21T23:18:28.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.653+0000 7f8a442b5640 1 -- 172.21.15.40:0/2494492429 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a3c101030 msgr2=0x7f8a3c10d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:28.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.653+0000 7f8a442b5640 1 --2- 172.21.15.40:0/2494492429 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a3c101030 0x7f8a3c10d4c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f8a34009c10 tx=0x7f8a3402bef0 comp rx=0 tx=0).stop 2024-04-21T23:18:28.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.653+0000 7f8a442b5640 1 -- 172.21.15.40:0/2494492429 shutdown_connections 2024-04-21T23:18:28.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.653+0000 7f8a442b5640 1 --2- 172.21.15.40:0/2494492429 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a3c101030 0x7f8a3c10d4c0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:28.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.653+0000 7f8a442b5640 1 --2- 172.21.15.40:0/2494492429 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a3c100670 0x7f8a3c100af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:28.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.653+0000 7f8a442b5640 1 --2- 172.21.15.40:0/2494492429 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a3c106690 0x7f8a3c106a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:28.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.653+0000 7f8a442b5640 1 -- 172.21.15.40:0/2494492429 >> 172.21.15.40:0/2494492429 conn(0x7f8a3c076750 msgr2=0x7f8a3c076b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:28.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.653+0000 7f8a442b5640 1 -- 172.21.15.40:0/2494492429 shutdown_connections 2024-04-21T23:18:28.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.653+0000 7f8a442b5640 1 -- 172.21.15.40:0/2494492429 wait complete. 2024-04-21T23:18:28.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.653+0000 7f8a442b5640 1 Processor -- start 2024-04-21T23:18:28.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.653+0000 7f8a442b5640 1 -- start start 2024-04-21T23:18:28.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.653+0000 7f8a442b5640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a3c100670 0x7f8a3c197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:28.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.653+0000 7f8a442b5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a3c101030 0x7f8a3c198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:28.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.653+0000 7f8a442b5640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a3c106690 0x7f8a3c19c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:28.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.653+0000 7f8a442b5640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8a3c111af0 con 0x7f8a3c101030 2024-04-21T23:18:28.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.653+0000 7f8a442b5640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f8a3c111970 con 0x7f8a3c106690 2024-04-21T23:18:28.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.653+0000 7f8a442b5640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8a3c111c70 con 0x7f8a3c100670 2024-04-21T23:18:28.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.654+0000 7f8a41850640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a3c101030 0x7f8a3c198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:28.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.654+0000 7f8a41850640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a3c101030 0x7f8a3c198490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32816/0 (socket says 172.21.15.40:32816) 2024-04-21T23:18:28.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.654+0000 7f8a41850640 1 -- 172.21.15.40:0/463765212 learned_addr learned my addr 172.21.15.40:0/463765212 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:28.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.654+0000 7f8a42852640 1 --2- 172.21.15.40:0/463765212 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a3c106690 0x7f8a3c19c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:28.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.654+0000 7f8a42051640 1 --2- 172.21.15.40:0/463765212 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a3c100670 0x7f8a3c197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:28.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.654+0000 7f8a41850640 1 -- 172.21.15.40:0/463765212 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a3c100670 msgr2=0x7f8a3c197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:28.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.654+0000 7f8a41850640 1 --2- 172.21.15.40:0/463765212 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a3c100670 0x7f8a3c197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:28.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.654+0000 7f8a41850640 1 -- 172.21.15.40:0/463765212 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a3c106690 msgr2=0x7f8a3c19c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:28.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.654+0000 7f8a41850640 1 --2- 172.21.15.40:0/463765212 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a3c106690 0x7f8a3c19c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:28.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.654+0000 7f8a41850640 1 -- 172.21.15.40:0/463765212 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a3c19cf40 con 0x7f8a3c101030 2024-04-21T23:18:28.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.654+0000 7f8a42852640 1 --2- 172.21.15.40:0/463765212 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a3c106690 0x7f8a3c19c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:18:28.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.654+0000 7f8a41850640 1 --2- 172.21.15.40:0/463765212 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a3c101030 0x7f8a3c198490 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f8a2c00e9c0 tx=0x7f8a2c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:28.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.654+0000 7f8a337fe640 1 -- 172.21.15.40:0/463765212 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a2c00cde0 con 0x7f8a3c101030 2024-04-21T23:18:28.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.654+0000 7f8a442b5640 1 -- 172.21.15.40:0/463765212 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a3c19d230 con 0x7f8a3c101030 2024-04-21T23:18:28.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.654+0000 7f8a337fe640 1 -- 172.21.15.40:0/463765212 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a2c0035f0 con 0x7f8a3c101030 2024-04-21T23:18:28.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.654+0000 7f8a337fe640 1 -- 172.21.15.40:0/463765212 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a2c010650 con 0x7f8a3c101030 2024-04-21T23:18:28.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.654+0000 7f8a442b5640 1 -- 172.21.15.40:0/463765212 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a3c1a4ac0 con 0x7f8a3c101030 2024-04-21T23:18:28.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.655+0000 7f8a442b5640 1 -- 172.21.15.40:0/463765212 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a3c101d30 con 0x7f8a3c101030 2024-04-21T23:18:28.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.657+0000 7f8a337fe640 1 -- 172.21.15.40:0/463765212 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7f8a2c0037a0 con 0x7f8a3c101030 2024-04-21T23:18:28.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.657+0000 7f8a337fe640 1 --2- 172.21.15.40:0/463765212 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f8a280a2e20 0x7f8a280a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:28.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.657+0000 7f8a42051640 1 --2- 172.21.15.40:0/463765212 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f8a280a2e20 0x7f8a280a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:28.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.657+0000 7f8a337fe640 1 -- 172.21.15.40:0/463765212 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f8a2c0cfdd0 con 0x7f8a3c101030 2024-04-21T23:18:28.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.658+0000 7f8a42051640 1 --2- 172.21.15.40:0/463765212 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f8a280a2e20 0x7f8a280a52e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c101730 tx=0x7f8a38005f90 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:28.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.662+0000 7f8a337fe640 1 -- 172.21.15.40:0/463765212 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f8a2c09c3f0 con 0x7f8a3c101030 2024-04-21T23:18:28.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.842+0000 7f8a442b5640 1 -- 172.21.15.40:0/463765212 --> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8a3c0ff810 con 0x7f8a280a2e20 2024-04-21T23:18:28.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.851+0000 7f8a337fe640 1 -- 172.21.15.40:0/463765212 <== mgr.25111 v2:172.21.15.40:6800/478611259 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339387 (secure 0 0 0) 0x7f8a3c0ff810 con 0x7f8a280a2e20 2024-04-21T23:18:28.854 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:18:28.856 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-21T23:18:28.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.854+0000 7f8a442b5640 1 -- 172.21.15.40:0/463765212 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f8a280a2e20 msgr2=0x7f8a280a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:28.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.854+0000 7f8a442b5640 1 --2- 172.21.15.40:0/463765212 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f8a280a2e20 0x7f8a280a52e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c101730 tx=0x7f8a38005f90 comp rx=0 tx=0).stop 2024-04-21T23:18:28.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.854+0000 7f8a442b5640 1 -- 172.21.15.40:0/463765212 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a3c101030 msgr2=0x7f8a3c198490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:28.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.854+0000 7f8a442b5640 1 --2- 172.21.15.40:0/463765212 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a3c101030 0x7f8a3c198490 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f8a2c00e9c0 tx=0x7f8a2c00ee80 comp rx=0 tx=0).stop 2024-04-21T23:18:28.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.854+0000 7f8a442b5640 1 -- 172.21.15.40:0/463765212 shutdown_connections 2024-04-21T23:18:28.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.854+0000 7f8a442b5640 1 --2- 172.21.15.40:0/463765212 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f8a280a2e20 0x7f8a280a52e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:28.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.854+0000 7f8a442b5640 1 --2- 172.21.15.40:0/463765212 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a3c106690 0x7f8a3c19c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:28.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.854+0000 7f8a442b5640 1 --2- 172.21.15.40:0/463765212 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a3c101030 0x7f8a3c198490 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:28.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.854+0000 7f8a442b5640 1 --2- 172.21.15.40:0/463765212 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a3c100670 0x7f8a3c197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:28.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.854+0000 7f8a442b5640 1 -- 172.21.15.40:0/463765212 >> 172.21.15.40:0/463765212 conn(0x7f8a3c076750 msgr2=0x7f8a3c0ff100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:28.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.855+0000 7f8a442b5640 1 -- 172.21.15.40:0/463765212 shutdown_connections 2024-04-21T23:18:28.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:28.855+0000 7f8a442b5640 1 -- 172.21.15.40:0/463765212 wait complete. 2024-04-21T23:18:28.864 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-04-21T23:18:27.410882+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1188469,"num_objects":66,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":990,"num_read_kb":655,"num_write":471,"num_write_kb":4380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1180848,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":867,"ondisk_log_size":867,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":373920,"kb_used_data":16356,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124567904,"statfs":{"total":1151940427776,"available":1151557533696,"internally_reserved":0,"allocated":16748544,"data_stored":9961440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":365933882},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.993386"},"pg_stats":[{"pgid":"3.3f","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.399227+0000","last_change":"2024-04-21T23:15:56.349368+0000","last_active":"2024-04-21T23:18:23.399227+0000","last_peered":"2024-04-21T23:18:23.399227+0000","last_clean":"2024-04-21T23:18:23.399227+0000","last_became_active":"2024-04-21T23:15:56.345743+0000","last_became_peered":"2024-04-21T23:15:56.345743+0000","last_unstale":"2024-04-21T23:18:23.399227+0000","last_undegraded":"2024-04-21T23:18:23.399227+0000","last_fullsized":"2024-04-21T23:18:23.399227+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:57:21.574012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"92'4","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.832418+0000","last_change":"2024-04-21T23:15:56.347231+0000","last_active":"2024-04-21T23:18:23.832418+0000","last_peered":"2024-04-21T23:18:23.832418+0000","last_clean":"2024-04-21T23:18:23.832418+0000","last_became_active":"2024-04-21T23:15:56.346847+0000","last_became_peered":"2024-04-21T23:15:56.346847+0000","last_unstale":"2024-04-21T23:18:23.832418+0000","last_undegraded":"2024-04-21T23:18:23.832418+0000","last_fullsized":"2024-04-21T23:18:23.832418+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:20:15.884789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.831956+0000","last_change":"2024-04-21T23:15:56.354897+0000","last_active":"2024-04-21T23:18:23.831956+0000","last_peered":"2024-04-21T23:18:23.831956+0000","last_clean":"2024-04-21T23:18:23.831956+0000","last_became_active":"2024-04-21T23:15:56.354494+0000","last_became_peered":"2024-04-21T23:15:56.354494+0000","last_unstale":"2024-04-21T23:18:23.831956+0000","last_undegraded":"2024-04-21T23:18:23.831956+0000","last_fullsized":"2024-04-21T23:18:23.831956+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:51:11.865310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.660564+0000","last_change":"2024-04-21T23:15:56.355184+0000","last_active":"2024-04-21T23:18:23.660564+0000","last_peered":"2024-04-21T23:18:23.660564+0000","last_clean":"2024-04-21T23:18:23.660564+0000","last_became_active":"2024-04-21T23:15:56.354748+0000","last_became_peered":"2024-04-21T23:15:56.354748+0000","last_unstale":"2024-04-21T23:18:23.660564+0000","last_undegraded":"2024-04-21T23:18:23.660564+0000","last_fullsized":"2024-04-21T23:18:23.660564+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:46:24.867488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"92'4","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.500211+0000","last_change":"2024-04-21T23:15:56.349446+0000","last_active":"2024-04-21T23:18:23.500211+0000","last_peered":"2024-04-21T23:18:23.500211+0000","last_clean":"2024-04-21T23:18:23.500211+0000","last_became_active":"2024-04-21T23:15:56.349050+0000","last_became_peered":"2024-04-21T23:15:56.349050+0000","last_unstale":"2024-04-21T23:18:23.500211+0000","last_undegraded":"2024-04-21T23:18:23.500211+0000","last_fullsized":"2024-04-21T23:18:23.500211+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:15:25.489044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"92'6","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.399061+0000","last_change":"2024-04-21T23:15:56.356095+0000","last_active":"2024-04-21T23:18:23.399061+0000","last_peered":"2024-04-21T23:18:23.399061+0000","last_clean":"2024-04-21T23:18:23.399061+0000","last_became_active":"2024-04-21T23:15:56.355754+0000","last_became_peered":"2024-04-21T23:15:56.355754+0000","last_unstale":"2024-04-21T23:18:23.399061+0000","last_undegraded":"2024-04-21T23:18:23.399061+0000","last_fullsized":"2024-04-21T23:18:23.399061+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:00:27.296457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"92'5","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.662728+0000","last_change":"2024-04-21T23:15:56.349733+0000","last_active":"2024-04-21T23:18:23.662728+0000","last_peered":"2024-04-21T23:18:23.662728+0000","last_clean":"2024-04-21T23:18:23.662728+0000","last_became_active":"2024-04-21T23:15:56.347820+0000","last_became_peered":"2024-04-21T23:15:56.347820+0000","last_unstale":"2024-04-21T23:18:23.662728+0000","last_undegraded":"2024-04-21T23:18:23.662728+0000","last_fullsized":"2024-04-21T23:18:23.662728+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:31:34.453626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"92'9","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.500348+0000","last_change":"2024-04-21T23:15:56.359385+0000","last_active":"2024-04-21T23:18:23.500348+0000","last_peered":"2024-04-21T23:18:23.500348+0000","last_clean":"2024-04-21T23:18:23.500348+0000","last_became_active":"2024-04-21T23:15:56.359074+0000","last_became_peered":"2024-04-21T23:15:56.359074+0000","last_unstale":"2024-04-21T23:18:23.500348+0000","last_undegraded":"2024-04-21T23:18:23.500348+0000","last_fullsized":"2024-04-21T23:18:23.500348+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:40:23.904592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"92'11","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.661940+0000","last_change":"2024-04-21T23:15:56.350886+0000","last_active":"2024-04-21T23:18:23.661940+0000","last_peered":"2024-04-21T23:18:23.661940+0000","last_clean":"2024-04-21T23:18:23.661940+0000","last_became_active":"2024-04-21T23:15:56.350531+0000","last_became_peered":"2024-04-21T23:15:56.350531+0000","last_unstale":"2024-04-21T23:18:23.661940+0000","last_undegraded":"2024-04-21T23:18:23.661940+0000","last_fullsized":"2024-04-21T23:18:23.661940+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:13:12.354473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"92'16","reported_seq":48,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.399459+0000","last_change":"2024-04-21T23:15:56.347376+0000","last_active":"2024-04-21T23:18:23.399459+0000","last_peered":"2024-04-21T23:18:23.399459+0000","last_clean":"2024-04-21T23:18:23.399459+0000","last_became_active":"2024-04-21T23:15:56.347016+0000","last_became_peered":"2024-04-21T23:15:56.347016+0000","last_unstale":"2024-04-21T23:18:23.399459+0000","last_undegraded":"2024-04-21T23:18:23.399459+0000","last_fullsized":"2024-04-21T23:18:23.399459+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:49:46.722931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.663443+0000","last_change":"2024-04-21T23:15:56.351754+0000","last_active":"2024-04-21T23:18:23.663443+0000","last_peered":"2024-04-21T23:18:23.663443+0000","last_clean":"2024-04-21T23:18:23.663443+0000","last_became_active":"2024-04-21T23:15:56.349860+0000","last_became_peered":"2024-04-21T23:15:56.349860+0000","last_unstale":"2024-04-21T23:18:23.663443+0000","last_undegraded":"2024-04-21T23:18:23.663443+0000","last_fullsized":"2024-04-21T23:18:23.663443+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:48:41.469708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.660467+0000","last_change":"2024-04-21T23:15:56.345177+0000","last_active":"2024-04-21T23:18:23.660467+0000","last_peered":"2024-04-21T23:18:23.660467+0000","last_clean":"2024-04-21T23:18:23.660467+0000","last_became_active":"2024-04-21T23:15:56.344791+0000","last_became_peered":"2024-04-21T23:15:56.344791+0000","last_unstale":"2024-04-21T23:18:23.660467+0000","last_undegraded":"2024-04-21T23:18:23.660467+0000","last_fullsized":"2024-04-21T23:18:23.660467+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:33:33.418383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"92'10","reported_seq":48,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.688996+0000","last_change":"2024-04-21T23:16:52.556505+0000","last_active":"2024-04-21T23:18:23.688996+0000","last_peered":"2024-04-21T23:18:23.688996+0000","last_clean":"2024-04-21T23:18:23.688996+0000","last_became_active":"2024-04-21T23:16:52.556177+0000","last_became_peered":"2024-04-21T23:16:52.556177+0000","last_unstale":"2024-04-21T23:18:23.688996+0000","last_undegraded":"2024-04-21T23:18:23.688996+0000","last_fullsized":"2024-04-21T23:18:23.688996+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:17:13.546746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.832040+0000","last_change":"2024-04-21T23:15:56.354544+0000","last_active":"2024-04-21T23:18:23.832040+0000","last_peered":"2024-04-21T23:18:23.832040+0000","last_clean":"2024-04-21T23:18:23.832040+0000","last_became_active":"2024-04-21T23:15:56.354206+0000","last_became_peered":"2024-04-21T23:15:56.354206+0000","last_unstale":"2024-04-21T23:18:23.832040+0000","last_undegraded":"2024-04-21T23:18:23.832040+0000","last_fullsized":"2024-04-21T23:18:23.832040+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:37:19.985422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.831952+0000","last_change":"2024-04-21T23:15:56.357447+0000","last_active":"2024-04-21T23:18:23.831952+0000","last_peered":"2024-04-21T23:18:23.831952+0000","last_clean":"2024-04-21T23:18:23.831952+0000","last_became_active":"2024-04-21T23:15:56.355850+0000","last_became_peered":"2024-04-21T23:15:56.355850+0000","last_unstale":"2024-04-21T23:18:23.831952+0000","last_undegraded":"2024-04-21T23:18:23.831952+0000","last_fullsized":"2024-04-21T23:18:23.831952+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:48:56.648994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.660691+0000","last_change":"2024-04-21T23:15:56.353712+0000","last_active":"2024-04-21T23:18:23.660691+0000","last_peered":"2024-04-21T23:18:23.660691+0000","last_clean":"2024-04-21T23:18:23.660691+0000","last_became_active":"2024-04-21T23:15:56.353388+0000","last_became_peered":"2024-04-21T23:15:56.353388+0000","last_unstale":"2024-04-21T23:18:23.660691+0000","last_undegraded":"2024-04-21T23:18:23.660691+0000","last_fullsized":"2024-04-21T23:18:23.660691+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:43:33.581222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.399252+0000","last_change":"2024-04-21T23:15:56.347041+0000","last_active":"2024-04-21T23:18:23.399252+0000","last_peered":"2024-04-21T23:18:23.399252+0000","last_clean":"2024-04-21T23:18:23.399252+0000","last_became_active":"2024-04-21T23:15:56.346630+0000","last_became_peered":"2024-04-21T23:15:56.346630+0000","last_unstale":"2024-04-21T23:18:23.399252+0000","last_undegraded":"2024-04-21T23:18:23.399252+0000","last_fullsized":"2024-04-21T23:18:23.399252+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:57:42.553666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"92'10","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.498505+0000","last_change":"2024-04-21T23:15:56.353809+0000","last_active":"2024-04-21T23:18:23.498505+0000","last_peered":"2024-04-21T23:18:23.498505+0000","last_clean":"2024-04-21T23:18:23.498505+0000","last_became_active":"2024-04-21T23:15:56.353469+0000","last_became_peered":"2024-04-21T23:15:56.353469+0000","last_unstale":"2024-04-21T23:18:23.498505+0000","last_undegraded":"2024-04-21T23:18:23.498505+0000","last_fullsized":"2024-04-21T23:18:23.498505+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:11:14.136845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.395486+0000","last_change":"2024-04-21T23:15:56.354500+0000","last_active":"2024-04-21T23:18:23.395486+0000","last_peered":"2024-04-21T23:18:23.395486+0000","last_clean":"2024-04-21T23:18:23.395486+0000","last_became_active":"2024-04-21T23:15:56.353893+0000","last_became_peered":"2024-04-21T23:15:56.353893+0000","last_unstale":"2024-04-21T23:18:23.395486+0000","last_undegraded":"2024-04-21T23:18:23.395486+0000","last_fullsized":"2024-04-21T23:18:23.395486+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:14:38.121448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.395523+0000","last_change":"2024-04-21T23:17:53.274634+0000","last_active":"2024-04-21T23:18:23.395523+0000","last_peered":"2024-04-21T23:18:23.395523+0000","last_clean":"2024-04-21T23:18:23.395523+0000","last_became_active":"2024-04-21T23:17:53.274018+0000","last_became_peered":"2024-04-21T23:17:53.274018+0000","last_unstale":"2024-04-21T23:18:23.395523+0000","last_undegraded":"2024-04-21T23:18:23.395523+0000","last_fullsized":"2024-04-21T23:18:23.395523+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:04:55.017447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8,0],"acting":[2,3,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"90'9","reported_seq":44,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.498460+0000","last_change":"2024-04-21T23:15:55.402553+0000","last_active":"2024-04-21T23:18:23.498460+0000","last_peered":"2024-04-21T23:18:23.498460+0000","last_clean":"2024-04-21T23:18:23.498460+0000","last_became_active":"2024-04-21T23:15:55.402192+0000","last_became_peered":"2024-04-21T23:15:55.402192+0000","last_unstale":"2024-04-21T23:18:23.498460+0000","last_undegraded":"2024-04-21T23:18:23.498460+0000","last_fullsized":"2024-04-21T23:18:23.498460+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:24:57.467906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"92'5","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.501309+0000","last_change":"2024-04-21T23:15:56.358797+0000","last_active":"2024-04-21T23:18:23.501309+0000","last_peered":"2024-04-21T23:18:23.501309+0000","last_clean":"2024-04-21T23:18:23.501309+0000","last_became_active":"2024-04-21T23:15:56.357997+0000","last_became_peered":"2024-04-21T23:15:56.357997+0000","last_unstale":"2024-04-21T23:18:23.501309+0000","last_undegraded":"2024-04-21T23:18:23.501309+0000","last_fullsized":"2024-04-21T23:18:23.501309+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:20:50.891506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.659831+0000","last_change":"2024-04-21T23:16:00.443368+0000","last_active":"2024-04-21T23:18:23.659831+0000","last_peered":"2024-04-21T23:18:23.659831+0000","last_clean":"2024-04-21T23:18:23.659831+0000","last_became_active":"2024-04-21T23:16:00.442940+0000","last_became_peered":"2024-04-21T23:16:00.442940+0000","last_unstale":"2024-04-21T23:18:23.659831+0000","last_undegraded":"2024-04-21T23:18:23.659831+0000","last_fullsized":"2024-04-21T23:18:23.659831+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:17:15.706460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"90'2","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.501369+0000","last_change":"2024-04-21T23:15:55.405216+0000","last_active":"2024-04-21T23:18:23.501369+0000","last_peered":"2024-04-21T23:18:23.501369+0000","last_clean":"2024-04-21T23:18:23.501369+0000","last_became_active":"2024-04-21T23:15:55.404844+0000","last_became_peered":"2024-04-21T23:15:55.404844+0000","last_unstale":"2024-04-21T23:18:23.501369+0000","last_undegraded":"2024-04-21T23:18:23.501369+0000","last_fullsized":"2024-04-21T23:18:23.501369+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:56:28.691844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.832115+0000","last_change":"2024-04-21T23:15:56.355789+0000","last_active":"2024-04-21T23:18:23.832115+0000","last_peered":"2024-04-21T23:18:23.832115+0000","last_clean":"2024-04-21T23:18:23.832115+0000","last_became_active":"2024-04-21T23:15:56.354972+0000","last_became_peered":"2024-04-21T23:15:56.354972+0000","last_unstale":"2024-04-21T23:18:23.832115+0000","last_undegraded":"2024-04-21T23:18:23.832115+0000","last_fullsized":"2024-04-21T23:18:23.832115+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:11:36.831453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.660103+0000","last_change":"2024-04-21T23:16:00.443241+0000","last_active":"2024-04-21T23:18:23.660103+0000","last_peered":"2024-04-21T23:18:23.660103+0000","last_clean":"2024-04-21T23:18:23.660103+0000","last_became_active":"2024-04-21T23:16:00.442820+0000","last_became_peered":"2024-04-21T23:16:00.442820+0000","last_unstale":"2024-04-21T23:18:23.660103+0000","last_undegraded":"2024-04-21T23:18:23.660103+0000","last_fullsized":"2024-04-21T23:18:23.660103+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:28:04.032621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"90'4","reported_seq":49,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.500708+0000","last_change":"2024-04-21T23:16:52.212906+0000","last_active":"2024-04-21T23:18:23.500708+0000","last_peered":"2024-04-21T23:18:23.500708+0000","last_clean":"2024-04-21T23:18:23.500708+0000","last_became_active":"2024-04-21T23:16:52.205828+0000","last_became_peered":"2024-04-21T23:16:52.205828+0000","last_unstale":"2024-04-21T23:18:23.500708+0000","last_undegraded":"2024-04-21T23:18:23.500708+0000","last_fullsized":"2024-04-21T23:18:23.500708+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:20:01.936578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.662333+0000","last_change":"2024-04-21T23:15:56.354462+0000","last_active":"2024-04-21T23:18:23.662333+0000","last_peered":"2024-04-21T23:18:23.662333+0000","last_clean":"2024-04-21T23:18:23.662333+0000","last_became_active":"2024-04-21T23:15:56.354134+0000","last_became_peered":"2024-04-21T23:15:56.354134+0000","last_unstale":"2024-04-21T23:18:23.662333+0000","last_undegraded":"2024-04-21T23:18:23.662333+0000","last_fullsized":"2024-04-21T23:18:23.662333+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:27:40.809483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.660710+0000","last_change":"2024-04-21T23:16:00.427717+0000","last_active":"2024-04-21T23:18:23.660710+0000","last_peered":"2024-04-21T23:18:23.660710+0000","last_clean":"2024-04-21T23:18:23.660710+0000","last_became_active":"2024-04-21T23:16:00.427296+0000","last_became_peered":"2024-04-21T23:16:00.427296+0000","last_unstale":"2024-04-21T23:18:23.660710+0000","last_undegraded":"2024-04-21T23:18:23.660710+0000","last_fullsized":"2024-04-21T23:18:23.660710+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:03:41.650045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.500411+0000","last_change":"2024-04-21T23:15:55.398641+0000","last_active":"2024-04-21T23:18:23.500411+0000","last_peered":"2024-04-21T23:18:23.500411+0000","last_clean":"2024-04-21T23:18:23.500411+0000","last_became_active":"2024-04-21T23:15:55.397712+0000","last_became_peered":"2024-04-21T23:15:55.397712+0000","last_unstale":"2024-04-21T23:18:23.500411+0000","last_undegraded":"2024-04-21T23:18:23.500411+0000","last_fullsized":"2024-04-21T23:18:23.500411+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:28:09.266740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"92'15","reported_seq":46,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.507482+0000","last_change":"2024-04-21T23:15:56.345359+0000","last_active":"2024-04-21T23:18:23.507482+0000","last_peered":"2024-04-21T23:18:23.507482+0000","last_clean":"2024-04-21T23:18:23.507482+0000","last_became_active":"2024-04-21T23:15:56.343652+0000","last_became_peered":"2024-04-21T23:15:56.343652+0000","last_unstale":"2024-04-21T23:18:23.507482+0000","last_undegraded":"2024-04-21T23:18:23.507482+0000","last_fullsized":"2024-04-21T23:18:23.507482+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:18:10.057999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.395459+0000","last_change":"2024-04-21T23:16:00.430640+0000","last_active":"2024-04-21T23:18:23.395459+0000","last_peered":"2024-04-21T23:18:23.395459+0000","last_clean":"2024-04-21T23:18:23.395459+0000","last_became_active":"2024-04-21T23:16:00.430141+0000","last_became_peered":"2024-04-21T23:16:00.430141+0000","last_unstale":"2024-04-21T23:18:23.395459+0000","last_undegraded":"2024-04-21T23:18:23.395459+0000","last_fullsized":"2024-04-21T23:18:23.395459+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:41:13.749780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"95'6","reported_seq":165,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.688107+0000","last_change":"2024-04-21T23:15:55.399823+0000","last_active":"2024-04-21T23:18:23.688107+0000","last_peered":"2024-04-21T23:18:23.688107+0000","last_clean":"2024-04-21T23:18:23.688107+0000","last_became_active":"2024-04-21T23:15:55.399490+0000","last_became_peered":"2024-04-21T23:15:55.399490+0000","last_unstale":"2024-04-21T23:18:23.688107+0000","last_undegraded":"2024-04-21T23:18:23.688107+0000","last_fullsized":"2024-04-21T23:18:23.688107+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:54:38.124841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":125,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.500231+0000","last_change":"2024-04-21T23:15:56.346444+0000","last_active":"2024-04-21T23:18:23.500231+0000","last_peered":"2024-04-21T23:18:23.500231+0000","last_clean":"2024-04-21T23:18:23.500231+0000","last_became_active":"2024-04-21T23:15:56.346091+0000","last_became_peered":"2024-04-21T23:15:56.346091+0000","last_unstale":"2024-04-21T23:18:23.500231+0000","last_undegraded":"2024-04-21T23:18:23.500231+0000","last_fullsized":"2024-04-21T23:18:23.500231+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:41:11.144570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.395544+0000","last_change":"2024-04-21T23:16:00.438198+0000","last_active":"2024-04-21T23:18:23.395544+0000","last_peered":"2024-04-21T23:18:23.395544+0000","last_clean":"2024-04-21T23:18:23.395544+0000","last_became_active":"2024-04-21T23:16:00.437767+0000","last_became_peered":"2024-04-21T23:16:00.437767+0000","last_unstale":"2024-04-21T23:18:23.395544+0000","last_undegraded":"2024-04-21T23:18:23.395544+0000","last_fullsized":"2024-04-21T23:18:23.395544+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:35:11.356414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"90'3","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.399081+0000","last_change":"2024-04-21T23:15:55.403896+0000","last_active":"2024-04-21T23:18:23.399081+0000","last_peered":"2024-04-21T23:18:23.399081+0000","last_clean":"2024-04-21T23:18:23.399081+0000","last_became_active":"2024-04-21T23:15:55.403423+0000","last_became_peered":"2024-04-21T23:15:55.403423+0000","last_unstale":"2024-04-21T23:18:23.399081+0000","last_undegraded":"2024-04-21T23:18:23.399081+0000","last_fullsized":"2024-04-21T23:18:23.399081+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:34:44.252259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"92'8","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.498057+0000","last_change":"2024-04-21T23:16:52.349354+0000","last_active":"2024-04-21T23:18:23.498057+0000","last_peered":"2024-04-21T23:18:23.498057+0000","last_clean":"2024-04-21T23:18:23.498057+0000","last_became_active":"2024-04-21T23:16:52.348991+0000","last_became_peered":"2024-04-21T23:16:52.348991+0000","last_unstale":"2024-04-21T23:18:23.498057+0000","last_undegraded":"2024-04-21T23:18:23.498057+0000","last_fullsized":"2024-04-21T23:18:23.498057+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:42:57.975963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.399121+0000","last_change":"2024-04-21T23:16:00.434104+0000","last_active":"2024-04-21T23:18:23.399121+0000","last_peered":"2024-04-21T23:18:23.399121+0000","last_clean":"2024-04-21T23:18:23.399121+0000","last_became_active":"2024-04-21T23:16:00.433681+0000","last_became_peered":"2024-04-21T23:16:00.433681+0000","last_unstale":"2024-04-21T23:18:23.399121+0000","last_undegraded":"2024-04-21T23:18:23.399121+0000","last_fullsized":"2024-04-21T23:18:23.399121+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:10:17.986652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"90'2","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.688659+0000","last_change":"2024-04-21T23:15:55.397536+0000","last_active":"2024-04-21T23:18:23.688659+0000","last_peered":"2024-04-21T23:18:23.688659+0000","last_clean":"2024-04-21T23:18:23.688659+0000","last_became_active":"2024-04-21T23:15:55.397163+0000","last_became_peered":"2024-04-21T23:15:55.397163+0000","last_unstale":"2024-04-21T23:18:23.688659+0000","last_undegraded":"2024-04-21T23:18:23.688659+0000","last_fullsized":"2024-04-21T23:18:23.688659+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:05:27.777804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.500246+0000","last_change":"2024-04-21T23:15:56.351840+0000","last_active":"2024-04-21T23:18:23.500246+0000","last_peered":"2024-04-21T23:18:23.500246+0000","last_clean":"2024-04-21T23:18:23.500246+0000","last_became_active":"2024-04-21T23:15:56.351523+0000","last_became_peered":"2024-04-21T23:15:56.351523+0000","last_unstale":"2024-04-21T23:18:23.500246+0000","last_undegraded":"2024-04-21T23:18:23.500246+0000","last_fullsized":"2024-04-21T23:18:23.500246+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:57:11.714284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.832073+0000","last_change":"2024-04-21T23:16:00.437473+0000","last_active":"2024-04-21T23:18:23.832073+0000","last_peered":"2024-04-21T23:18:23.832073+0000","last_clean":"2024-04-21T23:18:23.832073+0000","last_became_active":"2024-04-21T23:16:00.437014+0000","last_became_peered":"2024-04-21T23:16:00.437014+0000","last_unstale":"2024-04-21T23:18:23.832073+0000","last_undegraded":"2024-04-21T23:18:23.832073+0000","last_fullsized":"2024-04-21T23:18:23.832073+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:37:14.241896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"90'1","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.507228+0000","last_change":"2024-04-21T23:15:55.399213+0000","last_active":"2024-04-21T23:18:23.507228+0000","last_peered":"2024-04-21T23:18:23.507228+0000","last_clean":"2024-04-21T23:18:23.507228+0000","last_became_active":"2024-04-21T23:15:55.398880+0000","last_became_peered":"2024-04-21T23:15:55.398880+0000","last_unstale":"2024-04-21T23:18:23.507228+0000","last_undegraded":"2024-04-21T23:18:23.507228+0000","last_fullsized":"2024-04-21T23:18:23.507228+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:56:32.236862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"92'11","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.497698+0000","last_change":"2024-04-21T23:15:56.349330+0000","last_active":"2024-04-21T23:18:23.497698+0000","last_peered":"2024-04-21T23:18:23.497698+0000","last_clean":"2024-04-21T23:18:23.497698+0000","last_became_active":"2024-04-21T23:15:56.348988+0000","last_became_peered":"2024-04-21T23:15:56.348988+0000","last_unstale":"2024-04-21T23:18:23.497698+0000","last_undegraded":"2024-04-21T23:18:23.497698+0000","last_fullsized":"2024-04-21T23:18:23.497698+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:11:19.650119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.500463+0000","last_change":"2024-04-21T23:16:00.431959+0000","last_active":"2024-04-21T23:18:23.500463+0000","last_peered":"2024-04-21T23:18:23.500463+0000","last_clean":"2024-04-21T23:18:23.500463+0000","last_became_active":"2024-04-21T23:16:00.431570+0000","last_became_peered":"2024-04-21T23:16:00.431570+0000","last_unstale":"2024-04-21T23:18:23.500463+0000","last_undegraded":"2024-04-21T23:18:23.500463+0000","last_fullsized":"2024-04-21T23:18:23.500463+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:02:54.057326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"95'4","reported_seq":160,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.686988+0000","last_change":"2024-04-21T23:15:55.404505+0000","last_active":"2024-04-21T23:18:23.686988+0000","last_peered":"2024-04-21T23:18:23.686988+0000","last_clean":"2024-04-21T23:18:23.686988+0000","last_became_active":"2024-04-21T23:15:55.403945+0000","last_became_peered":"2024-04-21T23:15:55.403945+0000","last_unstale":"2024-04-21T23:18:23.686988+0000","last_undegraded":"2024-04-21T23:18:23.686988+0000","last_fullsized":"2024-04-21T23:18:23.686988+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:54:37.647597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":122,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"92'10","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.661999+0000","last_change":"2024-04-21T23:15:56.356238+0000","last_active":"2024-04-21T23:18:23.661999+0000","last_peered":"2024-04-21T23:18:23.661999+0000","last_clean":"2024-04-21T23:18:23.661999+0000","last_became_active":"2024-04-21T23:15:56.355116+0000","last_became_peered":"2024-04-21T23:15:56.355116+0000","last_unstale":"2024-04-21T23:18:23.661999+0000","last_undegraded":"2024-04-21T23:18:23.661999+0000","last_fullsized":"2024-04-21T23:18:23.661999+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:43:20.468549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.497631+0000","last_change":"2024-04-21T23:16:00.439069+0000","last_active":"2024-04-21T23:18:23.497631+0000","last_peered":"2024-04-21T23:18:23.497631+0000","last_clean":"2024-04-21T23:18:23.497631+0000","last_became_active":"2024-04-21T23:16:00.438510+0000","last_became_peered":"2024-04-21T23:16:00.438510+0000","last_unstale":"2024-04-21T23:18:23.497631+0000","last_undegraded":"2024-04-21T23:18:23.497631+0000","last_fullsized":"2024-04-21T23:18:23.497631+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:41:50.814112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"90'2","reported_seq":46,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.663309+0000","last_change":"2024-04-21T23:16:52.206602+0000","last_active":"2024-04-21T23:18:23.663309+0000","last_peered":"2024-04-21T23:18:23.663309+0000","last_clean":"2024-04-21T23:18:23.663309+0000","last_became_active":"2024-04-21T23:16:52.202755+0000","last_became_peered":"2024-04-21T23:16:52.202755+0000","last_unstale":"2024-04-21T23:18:23.663309+0000","last_undegraded":"2024-04-21T23:18:23.663309+0000","last_fullsized":"2024-04-21T23:18:23.663309+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:01:02.437392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.832264+0000","last_change":"2024-04-21T23:15:56.357228+0000","last_active":"2024-04-21T23:18:23.832264+0000","last_peered":"2024-04-21T23:18:23.832264+0000","last_clean":"2024-04-21T23:18:23.832264+0000","last_became_active":"2024-04-21T23:15:56.356808+0000","last_became_peered":"2024-04-21T23:15:56.356808+0000","last_unstale":"2024-04-21T23:18:23.832264+0000","last_undegraded":"2024-04-21T23:18:23.832264+0000","last_fullsized":"2024-04-21T23:18:23.832264+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:15:10.728530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.660434+0000","last_change":"2024-04-21T23:16:00.443569+0000","last_active":"2024-04-21T23:18:23.660434+0000","last_peered":"2024-04-21T23:18:23.660434+0000","last_clean":"2024-04-21T23:18:23.660434+0000","last_became_active":"2024-04-21T23:16:00.442972+0000","last_became_peered":"2024-04-21T23:16:00.442972+0000","last_unstale":"2024-04-21T23:18:23.660434+0000","last_undegraded":"2024-04-21T23:18:23.660434+0000","last_fullsized":"2024-04-21T23:18:23.660434+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:35:21.471407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"90'4","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.507401+0000","last_change":"2024-04-21T23:15:55.400134+0000","last_active":"2024-04-21T23:18:23.507401+0000","last_peered":"2024-04-21T23:18:23.507401+0000","last_clean":"2024-04-21T23:18:23.507401+0000","last_became_active":"2024-04-21T23:15:55.399768+0000","last_became_peered":"2024-04-21T23:15:55.399768+0000","last_unstale":"2024-04-21T23:18:23.507401+0000","last_undegraded":"2024-04-21T23:18:23.507401+0000","last_fullsized":"2024-04-21T23:18:23.507401+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:15:09.323035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"92'11","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.662422+0000","last_change":"2024-04-21T23:15:56.357278+0000","last_active":"2024-04-21T23:18:23.662422+0000","last_peered":"2024-04-21T23:18:23.662422+0000","last_clean":"2024-04-21T23:18:23.662422+0000","last_became_active":"2024-04-21T23:15:56.356902+0000","last_became_peered":"2024-04-21T23:15:56.356902+0000","last_unstale":"2024-04-21T23:18:23.662422+0000","last_undegraded":"2024-04-21T23:18:23.662422+0000","last_fullsized":"2024-04-21T23:18:23.662422+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:19:05.269109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.497842+0000","last_change":"2024-04-21T23:16:00.439704+0000","last_active":"2024-04-21T23:18:23.497842+0000","last_peered":"2024-04-21T23:18:23.497842+0000","last_clean":"2024-04-21T23:18:23.497842+0000","last_became_active":"2024-04-21T23:16:00.439286+0000","last_became_peered":"2024-04-21T23:16:00.439286+0000","last_unstale":"2024-04-21T23:18:23.497842+0000","last_undegraded":"2024-04-21T23:18:23.497842+0000","last_fullsized":"2024-04-21T23:18:23.497842+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:08:59.194481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"95'38","reported_seq":326,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.687640+0000","last_change":"2024-04-21T23:15:55.403344+0000","last_active":"2024-04-21T23:18:23.687640+0000","last_peered":"2024-04-21T23:18:23.687640+0000","last_clean":"2024-04-21T23:18:23.687640+0000","last_became_active":"2024-04-21T23:15:55.402876+0000","last_became_peered":"2024-04-21T23:15:55.402876+0000","last_unstale":"2024-04-21T23:18:23.687640+0000","last_undegraded":"2024-04-21T23:18:23.687640+0000","last_fullsized":"2024-04-21T23:18:23.687640+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:55:59.082964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67428,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":254,"num_read_kb":193,"num_write":39,"num_write_kb":85,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.500611+0000","last_change":"2024-04-21T23:15:56.343897+0000","last_active":"2024-04-21T23:18:23.500611+0000","last_peered":"2024-04-21T23:18:23.500611+0000","last_clean":"2024-04-21T23:18:23.500611+0000","last_became_active":"2024-04-21T23:15:56.343567+0000","last_became_peered":"2024-04-21T23:15:56.343567+0000","last_unstale":"2024-04-21T23:18:23.500611+0000","last_undegraded":"2024-04-21T23:18:23.500611+0000","last_fullsized":"2024-04-21T23:18:23.500611+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:57:07.955722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.507737+0000","last_change":"2024-04-21T23:16:00.432722+0000","last_active":"2024-04-21T23:18:23.507737+0000","last_peered":"2024-04-21T23:18:23.507737+0000","last_clean":"2024-04-21T23:18:23.507737+0000","last_became_active":"2024-04-21T23:16:00.432257+0000","last_became_peered":"2024-04-21T23:16:00.432257+0000","last_unstale":"2024-04-21T23:18:23.507737+0000","last_undegraded":"2024-04-21T23:18:23.507737+0000","last_fullsized":"2024-04-21T23:18:23.507737+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:10:14.804490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"90'6","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.395461+0000","last_change":"2024-04-21T23:15:55.401300+0000","last_active":"2024-04-21T23:18:23.395461+0000","last_peered":"2024-04-21T23:18:23.395461+0000","last_clean":"2024-04-21T23:18:23.395461+0000","last_became_active":"2024-04-21T23:15:55.400904+0000","last_became_peered":"2024-04-21T23:15:55.400904+0000","last_unstale":"2024-04-21T23:18:23.395461+0000","last_undegraded":"2024-04-21T23:18:23.395461+0000","last_fullsized":"2024-04-21T23:18:23.395461+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:29:20.439936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.688806+0000","last_change":"2024-04-21T23:15:56.351046+0000","last_active":"2024-04-21T23:18:23.688806+0000","last_peered":"2024-04-21T23:18:23.688806+0000","last_clean":"2024-04-21T23:18:23.688806+0000","last_became_active":"2024-04-21T23:15:56.350662+0000","last_became_peered":"2024-04-21T23:15:56.350662+0000","last_unstale":"2024-04-21T23:18:23.688806+0000","last_undegraded":"2024-04-21T23:18:23.688806+0000","last_fullsized":"2024-04-21T23:18:23.688806+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:40:16.482061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.398794+0000","last_change":"2024-04-21T23:16:00.439816+0000","last_active":"2024-04-21T23:18:23.398794+0000","last_peered":"2024-04-21T23:18:23.398794+0000","last_clean":"2024-04-21T23:18:23.398794+0000","last_became_active":"2024-04-21T23:16:00.439330+0000","last_became_peered":"2024-04-21T23:16:00.439330+0000","last_unstale":"2024-04-21T23:18:23.398794+0000","last_undegraded":"2024-04-21T23:18:23.398794+0000","last_fullsized":"2024-04-21T23:18:23.398794+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:36:24.472968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"90'3","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.507439+0000","last_change":"2024-04-21T23:15:55.398588+0000","last_active":"2024-04-21T23:18:23.507439+0000","last_peered":"2024-04-21T23:18:23.507439+0000","last_clean":"2024-04-21T23:18:23.507439+0000","last_became_active":"2024-04-21T23:15:55.398264+0000","last_became_peered":"2024-04-21T23:15:55.398264+0000","last_unstale":"2024-04-21T23:18:23.507439+0000","last_undegraded":"2024-04-21T23:18:23.507439+0000","last_fullsized":"2024-04-21T23:18:23.507439+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:48:09.293481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"92'15","reported_seq":53,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.663205+0000","last_change":"2024-04-21T23:16:52.202037+0000","last_active":"2024-04-21T23:18:23.663205+0000","last_peered":"2024-04-21T23:18:23.663205+0000","last_clean":"2024-04-21T23:18:23.663205+0000","last_became_active":"2024-04-21T23:16:52.201244+0000","last_became_peered":"2024-04-21T23:16:52.201244+0000","last_unstale":"2024-04-21T23:18:23.663205+0000","last_undegraded":"2024-04-21T23:18:23.663205+0000","last_fullsized":"2024-04-21T23:18:23.663205+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:37:34.530110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.399332+0000","last_change":"2024-04-21T23:17:53.363902+0000","last_active":"2024-04-21T23:18:23.399332+0000","last_peered":"2024-04-21T23:18:23.399332+0000","last_clean":"2024-04-21T23:18:23.399332+0000","last_became_active":"2024-04-21T23:17:53.363433+0000","last_became_peered":"2024-04-21T23:17:53.363433+0000","last_unstale":"2024-04-21T23:18:23.399332+0000","last_undegraded":"2024-04-21T23:18:23.399332+0000","last_fullsized":"2024-04-21T23:18:23.399332+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:52:07.725946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,0,3],"acting":[10,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.11","version":"95'8","reported_seq":170,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.689594+0000","last_change":"2024-04-21T23:15:55.397503+0000","last_active":"2024-04-21T23:18:23.689594+0000","last_peered":"2024-04-21T23:18:23.689594+0000","last_clean":"2024-04-21T23:18:23.689594+0000","last_became_active":"2024-04-21T23:15:55.397181+0000","last_became_peered":"2024-04-21T23:15:55.397181+0000","last_unstale":"2024-04-21T23:18:23.689594+0000","last_undegraded":"2024-04-21T23:18:23.689594+0000","last_fullsized":"2024-04-21T23:18:23.689594+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:47:08.775783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2210,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":129,"num_read_kb":7,"num_write":7,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.663078+0000","last_change":"2024-04-21T23:15:56.350872+0000","last_active":"2024-04-21T23:18:23.663078+0000","last_peered":"2024-04-21T23:18:23.663078+0000","last_clean":"2024-04-21T23:18:23.663078+0000","last_became_active":"2024-04-21T23:15:56.350519+0000","last_became_peered":"2024-04-21T23:15:56.350519+0000","last_unstale":"2024-04-21T23:18:23.663078+0000","last_undegraded":"2024-04-21T23:18:23.663078+0000","last_fullsized":"2024-04-21T23:18:23.663078+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:23:37.753893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.689394+0000","last_change":"2024-04-21T23:16:00.436069+0000","last_active":"2024-04-21T23:18:23.689394+0000","last_peered":"2024-04-21T23:18:23.689394+0000","last_clean":"2024-04-21T23:18:23.689394+0000","last_became_active":"2024-04-21T23:16:00.435617+0000","last_became_peered":"2024-04-21T23:16:00.435617+0000","last_unstale":"2024-04-21T23:18:23.689394+0000","last_undegraded":"2024-04-21T23:18:23.689394+0000","last_fullsized":"2024-04-21T23:18:23.689394+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:21:44.256928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"90'15","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.398588+0000","last_change":"2024-04-21T23:16:52.220742+0000","last_active":"2024-04-21T23:18:23.398588+0000","last_peered":"2024-04-21T23:18:23.398588+0000","last_clean":"2024-04-21T23:18:23.398588+0000","last_became_active":"2024-04-21T23:16:52.204787+0000","last_became_peered":"2024-04-21T23:16:52.204787+0000","last_unstale":"2024-04-21T23:18:23.398588+0000","last_undegraded":"2024-04-21T23:18:23.398588+0000","last_fullsized":"2024-04-21T23:18:23.398588+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:53:12.057014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"92'6","reported_seq":44,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.507781+0000","last_change":"2024-04-21T23:16:52.202341+0000","last_active":"2024-04-21T23:18:23.507781+0000","last_peered":"2024-04-21T23:18:23.507781+0000","last_clean":"2024-04-21T23:18:23.507781+0000","last_became_active":"2024-04-21T23:16:52.201528+0000","last_became_peered":"2024-04-21T23:16:52.201528+0000","last_unstale":"2024-04-21T23:18:23.507781+0000","last_undegraded":"2024-04-21T23:18:23.507781+0000","last_fullsized":"2024-04-21T23:18:23.507781+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:41:24.767379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.500204+0000","last_change":"2024-04-21T23:17:53.274665+0000","last_active":"2024-04-21T23:18:23.500204+0000","last_peered":"2024-04-21T23:18:23.500204+0000","last_clean":"2024-04-21T23:18:23.500204+0000","last_became_active":"2024-04-21T23:17:53.274146+0000","last_became_peered":"2024-04-21T23:17:53.274146+0000","last_unstale":"2024-04-21T23:18:23.500204+0000","last_undegraded":"2024-04-21T23:18:23.500204+0000","last_fullsized":"2024-04-21T23:18:23.500204+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:15:08.842226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,3,1],"acting":[8,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.f","version":"94'3","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.500747+0000","last_change":"2024-04-21T23:15:55.399032+0000","last_active":"2024-04-21T23:18:23.500747+0000","last_peered":"2024-04-21T23:18:23.500747+0000","last_clean":"2024-04-21T23:18:23.500747+0000","last_became_active":"2024-04-21T23:15:55.398208+0000","last_became_peered":"2024-04-21T23:15:55.398208+0000","last_unstale":"2024-04-21T23:18:23.500747+0000","last_undegraded":"2024-04-21T23:18:23.500747+0000","last_fullsized":"2024-04-21T23:18:23.500747+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:14:59.520571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"92'5","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.399041+0000","last_change":"2024-04-21T23:15:56.355357+0000","last_active":"2024-04-21T23:18:23.399041+0000","last_peered":"2024-04-21T23:18:23.399041+0000","last_clean":"2024-04-21T23:18:23.399041+0000","last_became_active":"2024-04-21T23:15:56.354704+0000","last_became_peered":"2024-04-21T23:15:56.354704+0000","last_unstale":"2024-04-21T23:18:23.399041+0000","last_undegraded":"2024-04-21T23:18:23.399041+0000","last_fullsized":"2024-04-21T23:18:23.399041+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:14:02.368659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.500362+0000","last_change":"2024-04-21T23:16:00.427781+0000","last_active":"2024-04-21T23:18:23.500362+0000","last_peered":"2024-04-21T23:18:23.500362+0000","last_clean":"2024-04-21T23:18:23.500362+0000","last_became_active":"2024-04-21T23:16:00.427244+0000","last_became_peered":"2024-04-21T23:16:00.427244+0000","last_unstale":"2024-04-21T23:18:23.500362+0000","last_undegraded":"2024-04-21T23:18:23.500362+0000","last_fullsized":"2024-04-21T23:18:23.500362+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:21:30.548865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.498142+0000","last_change":"2024-04-21T23:15:55.405687+0000","last_active":"2024-04-21T23:18:23.498142+0000","last_peered":"2024-04-21T23:18:23.498142+0000","last_clean":"2024-04-21T23:18:23.498142+0000","last_became_active":"2024-04-21T23:15:55.405379+0000","last_became_peered":"2024-04-21T23:15:55.405379+0000","last_unstale":"2024-04-21T23:18:23.498142+0000","last_undegraded":"2024-04-21T23:18:23.498142+0000","last_fullsized":"2024-04-21T23:18:23.498142+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:12:44.192092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"92'4","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.399301+0000","last_change":"2024-04-21T23:15:56.348989+0000","last_active":"2024-04-21T23:18:23.399301+0000","last_peered":"2024-04-21T23:18:23.399301+0000","last_clean":"2024-04-21T23:18:23.399301+0000","last_became_active":"2024-04-21T23:15:56.348039+0000","last_became_peered":"2024-04-21T23:15:56.348039+0000","last_unstale":"2024-04-21T23:18:23.399301+0000","last_undegraded":"2024-04-21T23:18:23.399301+0000","last_fullsized":"2024-04-21T23:18:23.399301+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:39:15.577567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.688857+0000","last_change":"2024-04-21T23:16:00.434557+0000","last_active":"2024-04-21T23:18:23.688857+0000","last_peered":"2024-04-21T23:18:23.688857+0000","last_clean":"2024-04-21T23:18:23.688857+0000","last_became_active":"2024-04-21T23:16:00.433737+0000","last_became_peered":"2024-04-21T23:16:00.433737+0000","last_unstale":"2024-04-21T23:18:23.688857+0000","last_undegraded":"2024-04-21T23:18:23.688857+0000","last_fullsized":"2024-04-21T23:18:23.688857+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:38:00.295054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"90'1","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.662784+0000","last_change":"2024-04-21T23:15:55.396792+0000","last_active":"2024-04-21T23:18:23.662784+0000","last_peered":"2024-04-21T23:18:23.662784+0000","last_clean":"2024-04-21T23:18:23.662784+0000","last_became_active":"2024-04-21T23:15:55.395740+0000","last_became_peered":"2024-04-21T23:15:55.395740+0000","last_unstale":"2024-04-21T23:18:23.662784+0000","last_undegraded":"2024-04-21T23:18:23.662784+0000","last_fullsized":"2024-04-21T23:18:23.662784+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:39:22.366310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.399163+0000","last_change":"2024-04-21T23:15:56.346372+0000","last_active":"2024-04-21T23:18:23.399163+0000","last_peered":"2024-04-21T23:18:23.399163+0000","last_clean":"2024-04-21T23:18:23.399163+0000","last_became_active":"2024-04-21T23:15:56.346047+0000","last_became_peered":"2024-04-21T23:15:56.346047+0000","last_unstale":"2024-04-21T23:18:23.399163+0000","last_undegraded":"2024-04-21T23:18:23.399163+0000","last_fullsized":"2024-04-21T23:18:23.399163+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:32:29.684227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.689210+0000","last_change":"2024-04-21T23:16:00.434747+0000","last_active":"2024-04-21T23:18:23.689210+0000","last_peered":"2024-04-21T23:18:23.689210+0000","last_clean":"2024-04-21T23:18:23.689210+0000","last_became_active":"2024-04-21T23:16:00.433950+0000","last_became_peered":"2024-04-21T23:16:00.433950+0000","last_unstale":"2024-04-21T23:18:23.689210+0000","last_undegraded":"2024-04-21T23:18:23.689210+0000","last_fullsized":"2024-04-21T23:18:23.689210+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:02:10.562135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"92'8","reported_seq":46,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.507442+0000","last_change":"2024-04-21T23:16:52.206022+0000","last_active":"2024-04-21T23:18:23.507442+0000","last_peered":"2024-04-21T23:18:23.507442+0000","last_clean":"2024-04-21T23:18:23.507442+0000","last_became_active":"2024-04-21T23:16:52.205469+0000","last_became_peered":"2024-04-21T23:16:52.205469+0000","last_unstale":"2024-04-21T23:18:23.507442+0000","last_undegraded":"2024-04-21T23:18:23.507442+0000","last_fullsized":"2024-04-21T23:18:23.507442+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:34:46.264003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,8],"acting":[5,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"94'6","reported_seq":162,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.688568+0000","last_change":"2024-04-21T23:15:55.394900+0000","last_active":"2024-04-21T23:18:23.688568+0000","last_peered":"2024-04-21T23:18:23.688568+0000","last_clean":"2024-04-21T23:18:23.688568+0000","last_became_active":"2024-04-21T23:15:55.393556+0000","last_became_peered":"2024-04-21T23:15:55.393556+0000","last_unstale":"2024-04-21T23:18:23.688568+0000","last_undegraded":"2024-04-21T23:18:23.688568+0000","last_fullsized":"2024-04-21T23:18:23.688568+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:16:24.691930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":122,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":26,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.662759+0000","last_change":"2024-04-21T23:16:00.443338+0000","last_active":"2024-04-21T23:18:23.662759+0000","last_peered":"2024-04-21T23:18:23.662759+0000","last_clean":"2024-04-21T23:18:23.662759+0000","last_became_active":"2024-04-21T23:16:00.442855+0000","last_became_peered":"2024-04-21T23:16:00.442855+0000","last_unstale":"2024-04-21T23:18:23.662759+0000","last_undegraded":"2024-04-21T23:18:23.662759+0000","last_fullsized":"2024-04-21T23:18:23.662759+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:04:36.763027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.500267+0000","last_change":"2024-04-21T23:15:56.356788+0000","last_active":"2024-04-21T23:18:23.500267+0000","last_peered":"2024-04-21T23:18:23.500267+0000","last_clean":"2024-04-21T23:18:23.500267+0000","last_became_active":"2024-04-21T23:15:56.356474+0000","last_became_peered":"2024-04-21T23:15:56.356474+0000","last_unstale":"2024-04-21T23:18:23.500267+0000","last_undegraded":"2024-04-21T23:18:23.500267+0000","last_fullsized":"2024-04-21T23:18:23.500267+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:34:33.700897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"90'3","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.660276+0000","last_change":"2024-04-21T23:15:55.395398+0000","last_active":"2024-04-21T23:18:23.660276+0000","last_peered":"2024-04-21T23:18:23.660276+0000","last_clean":"2024-04-21T23:18:23.660276+0000","last_became_active":"2024-04-21T23:15:55.394978+0000","last_became_peered":"2024-04-21T23:15:55.394978+0000","last_unstale":"2024-04-21T23:18:23.660276+0000","last_undegraded":"2024-04-21T23:18:23.660276+0000","last_fullsized":"2024-04-21T23:18:23.660276+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:07:01.266885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":368,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.662639+0000","last_change":"2024-04-21T23:14:42.877959+0000","last_active":"2024-04-21T23:18:23.662639+0000","last_peered":"2024-04-21T23:18:23.662639+0000","last_clean":"2024-04-21T23:18:23.662639+0000","last_became_active":"2024-04-21T23:14:42.877384+0000","last_became_peered":"2024-04-21T23:14:42.877384+0000","last_unstale":"2024-04-21T23:18:23.662639+0000","last_undegraded":"2024-04-21T23:18:23.662639+0000","last_fullsized":"2024-04-21T23:18:23.662639+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:11:15.009349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:11:15.009349+0000","last_clean_scrub_stamp":"2024-04-21T23:11:15.009349+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:18:55.621052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.507306+0000","last_change":"2024-04-21T23:16:00.444576+0000","last_active":"2024-04-21T23:18:23.507306+0000","last_peered":"2024-04-21T23:18:23.507306+0000","last_clean":"2024-04-21T23:18:23.507306+0000","last_became_active":"2024-04-21T23:16:00.444153+0000","last_became_peered":"2024-04-21T23:16:00.444153+0000","last_unstale":"2024-04-21T23:18:23.507306+0000","last_undegraded":"2024-04-21T23:18:23.507306+0000","last_fullsized":"2024-04-21T23:18:23.507306+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:11:48.040383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"92'5","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.688448+0000","last_change":"2024-04-21T23:15:56.349870+0000","last_active":"2024-04-21T23:18:23.688448+0000","last_peered":"2024-04-21T23:18:23.688448+0000","last_clean":"2024-04-21T23:18:23.688448+0000","last_became_active":"2024-04-21T23:15:56.349535+0000","last_became_peered":"2024-04-21T23:15:56.349535+0000","last_unstale":"2024-04-21T23:18:23.688448+0000","last_undegraded":"2024-04-21T23:18:23.688448+0000","last_fullsized":"2024-04-21T23:18:23.688448+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:12:50.702556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"90'3","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.662748+0000","last_change":"2024-04-21T23:15:55.402406+0000","last_active":"2024-04-21T23:18:23.662748+0000","last_peered":"2024-04-21T23:18:23.662748+0000","last_clean":"2024-04-21T23:18:23.662748+0000","last_became_active":"2024-04-21T23:15:55.402075+0000","last_became_peered":"2024-04-21T23:15:55.402075+0000","last_unstale":"2024-04-21T23:18:23.662748+0000","last_undegraded":"2024-04-21T23:18:23.662748+0000","last_fullsized":"2024-04-21T23:18:23.662748+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:44:18.013031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.398753+0000","last_change":"2024-04-21T23:16:00.439206+0000","last_active":"2024-04-21T23:18:23.398753+0000","last_peered":"2024-04-21T23:18:23.398753+0000","last_clean":"2024-04-21T23:18:23.398753+0000","last_became_active":"2024-04-21T23:16:00.438601+0000","last_became_peered":"2024-04-21T23:16:00.438601+0000","last_unstale":"2024-04-21T23:18:23.398753+0000","last_undegraded":"2024-04-21T23:18:23.398753+0000","last_fullsized":"2024-04-21T23:18:23.398753+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:58:46.771456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"90'1","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.500582+0000","last_change":"2024-04-21T23:15:55.401154+0000","last_active":"2024-04-21T23:18:23.500582+0000","last_peered":"2024-04-21T23:18:23.500582+0000","last_clean":"2024-04-21T23:18:23.500582+0000","last_became_active":"2024-04-21T23:15:55.400579+0000","last_became_peered":"2024-04-21T23:15:55.400579+0000","last_unstale":"2024-04-21T23:18:23.500582+0000","last_undegraded":"2024-04-21T23:18:23.500582+0000","last_fullsized":"2024-04-21T23:18:23.500582+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:20:03.879660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.689222+0000","last_change":"2024-04-21T23:15:56.349837+0000","last_active":"2024-04-21T23:18:23.689222+0000","last_peered":"2024-04-21T23:18:23.689222+0000","last_clean":"2024-04-21T23:18:23.689222+0000","last_became_active":"2024-04-21T23:15:56.349466+0000","last_became_peered":"2024-04-21T23:15:56.349466+0000","last_unstale":"2024-04-21T23:18:23.689222+0000","last_undegraded":"2024-04-21T23:18:23.689222+0000","last_fullsized":"2024-04-21T23:18:23.689222+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:15:32.675770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.500545+0000","last_change":"2024-04-21T23:16:00.432599+0000","last_active":"2024-04-21T23:18:23.500545+0000","last_peered":"2024-04-21T23:18:23.500545+0000","last_clean":"2024-04-21T23:18:23.500545+0000","last_became_active":"2024-04-21T23:16:00.432194+0000","last_became_peered":"2024-04-21T23:16:00.432194+0000","last_unstale":"2024-04-21T23:18:23.500545+0000","last_undegraded":"2024-04-21T23:18:23.500545+0000","last_fullsized":"2024-04-21T23:18:23.500545+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:10:21.354338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"95'44","reported_seq":101,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.615014+0000","last_change":"2024-04-21T23:15:55.400328+0000","last_active":"2024-04-21T23:18:23.615014+0000","last_peered":"2024-04-21T23:18:23.615014+0000","last_clean":"2024-04-21T23:18:23.615014+0000","last_became_active":"2024-04-21T23:15:55.399991+0000","last_became_peered":"2024-04-21T23:15:55.399991+0000","last_unstale":"2024-04-21T23:18:23.615014+0000","last_undegraded":"2024-04-21T23:18:23.615014+0000","last_fullsized":"2024-04-21T23:18:23.615014+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:58:54.480720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":78,"num_write_kb":836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"92'5","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.501203+0000","last_change":"2024-04-21T23:15:56.358698+0000","last_active":"2024-04-21T23:18:23.501203+0000","last_peered":"2024-04-21T23:18:23.501203+0000","last_clean":"2024-04-21T23:18:23.501203+0000","last_became_active":"2024-04-21T23:15:56.358364+0000","last_became_peered":"2024-04-21T23:15:56.358364+0000","last_unstale":"2024-04-21T23:18:23.501203+0000","last_undegraded":"2024-04-21T23:18:23.501203+0000","last_fullsized":"2024-04-21T23:18:23.501203+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:34:15.281142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.395058+0000","last_change":"2024-04-21T23:16:00.432237+0000","last_active":"2024-04-21T23:18:23.395058+0000","last_peered":"2024-04-21T23:18:23.395058+0000","last_clean":"2024-04-21T23:18:23.395058+0000","last_became_active":"2024-04-21T23:16:00.431745+0000","last_became_peered":"2024-04-21T23:16:00.431745+0000","last_unstale":"2024-04-21T23:18:23.395058+0000","last_undegraded":"2024-04-21T23:18:23.395058+0000","last_fullsized":"2024-04-21T23:18:23.395058+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:25:00.477154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"90'4","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.661315+0000","last_change":"2024-04-21T23:15:55.398667+0000","last_active":"2024-04-21T23:18:23.661315+0000","last_peered":"2024-04-21T23:18:23.661315+0000","last_clean":"2024-04-21T23:18:23.661315+0000","last_became_active":"2024-04-21T23:15:55.398290+0000","last_became_peered":"2024-04-21T23:15:55.398290+0000","last_unstale":"2024-04-21T23:18:23.661315+0000","last_undegraded":"2024-04-21T23:18:23.661315+0000","last_fullsized":"2024-04-21T23:18:23.661315+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:57:58.992615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.832119+0000","last_change":"2024-04-21T23:15:56.359518+0000","last_active":"2024-04-21T23:18:23.832119+0000","last_peered":"2024-04-21T23:18:23.832119+0000","last_clean":"2024-04-21T23:18:23.832119+0000","last_became_active":"2024-04-21T23:15:56.358970+0000","last_became_peered":"2024-04-21T23:15:56.358970+0000","last_unstale":"2024-04-21T23:18:23.832119+0000","last_undegraded":"2024-04-21T23:18:23.832119+0000","last_fullsized":"2024-04-21T23:18:23.832119+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:41:22.113877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.660034+0000","last_change":"2024-04-21T23:16:00.443380+0000","last_active":"2024-04-21T23:18:23.660034+0000","last_peered":"2024-04-21T23:18:23.660034+0000","last_clean":"2024-04-21T23:18:23.660034+0000","last_became_active":"2024-04-21T23:16:00.442896+0000","last_became_peered":"2024-04-21T23:16:00.442896+0000","last_unstale":"2024-04-21T23:18:23.660034+0000","last_undegraded":"2024-04-21T23:18:23.660034+0000","last_fullsized":"2024-04-21T23:18:23.660034+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:51:00.089766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"90'2","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.660516+0000","last_change":"2024-04-21T23:15:55.395287+0000","last_active":"2024-04-21T23:18:23.660516+0000","last_peered":"2024-04-21T23:18:23.660516+0000","last_clean":"2024-04-21T23:18:23.660516+0000","last_became_active":"2024-04-21T23:15:55.394908+0000","last_became_peered":"2024-04-21T23:15:55.394908+0000","last_unstale":"2024-04-21T23:18:23.660516+0000","last_undegraded":"2024-04-21T23:18:23.660516+0000","last_fullsized":"2024-04-21T23:18:23.660516+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:57:29.915581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"92'2","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.500115+0000","last_change":"2024-04-21T23:15:56.348016+0000","last_active":"2024-04-21T23:18:23.500115+0000","last_peered":"2024-04-21T23:18:23.500115+0000","last_clean":"2024-04-21T23:18:23.500115+0000","last_became_active":"2024-04-21T23:15:56.347630+0000","last_became_peered":"2024-04-21T23:15:56.347630+0000","last_unstale":"2024-04-21T23:18:23.500115+0000","last_undegraded":"2024-04-21T23:18:23.500115+0000","last_fullsized":"2024-04-21T23:18:23.500115+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:17:27.372880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.500678+0000","last_change":"2024-04-21T23:17:52.925060+0000","last_active":"2024-04-21T23:18:23.500678+0000","last_peered":"2024-04-21T23:18:23.500678+0000","last_clean":"2024-04-21T23:18:23.500678+0000","last_became_active":"2024-04-21T23:17:52.924560+0000","last_became_peered":"2024-04-21T23:17:52.924560+0000","last_unstale":"2024-04-21T23:18:23.500678+0000","last_undegraded":"2024-04-21T23:18:23.500678+0000","last_fullsized":"2024-04-21T23:18:23.500678+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:24:19.265334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,5],"acting":[8,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"90'3","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.500520+0000","last_change":"2024-04-21T23:15:55.400859+0000","last_active":"2024-04-21T23:18:23.500520+0000","last_peered":"2024-04-21T23:18:23.500520+0000","last_clean":"2024-04-21T23:18:23.500520+0000","last_became_active":"2024-04-21T23:15:55.400460+0000","last_became_peered":"2024-04-21T23:15:55.400460+0000","last_unstale":"2024-04-21T23:18:23.500520+0000","last_undegraded":"2024-04-21T23:18:23.500520+0000","last_fullsized":"2024-04-21T23:18:23.500520+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:31:14.113476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.398661+0000","last_change":"2024-04-21T23:15:56.350354+0000","last_active":"2024-04-21T23:18:23.398661+0000","last_peered":"2024-04-21T23:18:23.398661+0000","last_clean":"2024-04-21T23:18:23.398661+0000","last_became_active":"2024-04-21T23:15:56.349954+0000","last_became_peered":"2024-04-21T23:15:56.349954+0000","last_unstale":"2024-04-21T23:18:23.398661+0000","last_undegraded":"2024-04-21T23:18:23.398661+0000","last_fullsized":"2024-04-21T23:18:23.398661+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:42:59.165610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":26,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.660123+0000","last_change":"2024-04-21T23:16:00.444829+0000","last_active":"2024-04-21T23:18:23.660123+0000","last_peered":"2024-04-21T23:18:23.660123+0000","last_clean":"2024-04-21T23:18:23.660123+0000","last_became_active":"2024-04-21T23:16:00.444365+0000","last_became_peered":"2024-04-21T23:16:00.444365+0000","last_unstale":"2024-04-21T23:18:23.660123+0000","last_undegraded":"2024-04-21T23:18:23.660123+0000","last_fullsized":"2024-04-21T23:18:23.660123+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:23:16.747684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"90'1","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.395250+0000","last_change":"2024-04-21T23:15:55.397855+0000","last_active":"2024-04-21T23:18:23.395250+0000","last_peered":"2024-04-21T23:18:23.395250+0000","last_clean":"2024-04-21T23:18:23.395250+0000","last_became_active":"2024-04-21T23:15:55.397498+0000","last_became_peered":"2024-04-21T23:15:55.397498+0000","last_unstale":"2024-04-21T23:18:23.395250+0000","last_undegraded":"2024-04-21T23:18:23.395250+0000","last_fullsized":"2024-04-21T23:18:23.395250+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:45:44.680350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.500943+0000","last_change":"2024-04-21T23:15:56.353023+0000","last_active":"2024-04-21T23:18:23.500943+0000","last_peered":"2024-04-21T23:18:23.500943+0000","last_clean":"2024-04-21T23:18:23.500943+0000","last_became_active":"2024-04-21T23:15:56.352209+0000","last_became_peered":"2024-04-21T23:15:56.352209+0000","last_unstale":"2024-04-21T23:18:23.500943+0000","last_undegraded":"2024-04-21T23:18:23.500943+0000","last_fullsized":"2024-04-21T23:18:23.500943+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:20:58.075627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.660994+0000","last_change":"2024-04-21T23:16:00.429258+0000","last_active":"2024-04-21T23:18:23.660994+0000","last_peered":"2024-04-21T23:18:23.660994+0000","last_clean":"2024-04-21T23:18:23.660994+0000","last_became_active":"2024-04-21T23:16:00.428856+0000","last_became_peered":"2024-04-21T23:16:00.428856+0000","last_unstale":"2024-04-21T23:18:23.660994+0000","last_undegraded":"2024-04-21T23:18:23.660994+0000","last_fullsized":"2024-04-21T23:18:23.660994+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:07:44.608370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"90'4","reported_seq":23,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.398659+0000","last_change":"2024-04-21T23:16:52.209766+0000","last_active":"2024-04-21T23:18:23.398659+0000","last_peered":"2024-04-21T23:18:23.398659+0000","last_clean":"2024-04-21T23:18:23.398659+0000","last_became_active":"2024-04-21T23:16:52.204654+0000","last_became_peered":"2024-04-21T23:16:52.204654+0000","last_unstale":"2024-04-21T23:18:23.398659+0000","last_undegraded":"2024-04-21T23:18:23.398659+0000","last_fullsized":"2024-04-21T23:18:23.398659+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:40:41.230460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"92'5","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.662982+0000","last_change":"2024-04-21T23:15:56.352024+0000","last_active":"2024-04-21T23:18:23.662982+0000","last_peered":"2024-04-21T23:18:23.662982+0000","last_clean":"2024-04-21T23:18:23.662982+0000","last_became_active":"2024-04-21T23:15:56.351052+0000","last_became_peered":"2024-04-21T23:15:56.351052+0000","last_unstale":"2024-04-21T23:18:23.662982+0000","last_undegraded":"2024-04-21T23:18:23.662982+0000","last_fullsized":"2024-04-21T23:18:23.662982+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:36:42.777601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.660850+0000","last_change":"2024-04-21T23:16:00.432371+0000","last_active":"2024-04-21T23:18:23.660850+0000","last_peered":"2024-04-21T23:18:23.660850+0000","last_clean":"2024-04-21T23:18:23.660850+0000","last_became_active":"2024-04-21T23:16:00.431828+0000","last_became_peered":"2024-04-21T23:16:00.431828+0000","last_unstale":"2024-04-21T23:18:23.660850+0000","last_undegraded":"2024-04-21T23:18:23.660850+0000","last_fullsized":"2024-04-21T23:18:23.660850+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:02:06.500075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"90'3","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.395651+0000","last_change":"2024-04-21T23:15:55.407018+0000","last_active":"2024-04-21T23:18:23.395651+0000","last_peered":"2024-04-21T23:18:23.395651+0000","last_clean":"2024-04-21T23:18:23.395651+0000","last_became_active":"2024-04-21T23:15:55.406709+0000","last_became_peered":"2024-04-21T23:15:55.406709+0000","last_unstale":"2024-04-21T23:18:23.395651+0000","last_undegraded":"2024-04-21T23:18:23.395651+0000","last_fullsized":"2024-04-21T23:18:23.395651+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:10:40.138470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.663535+0000","last_change":"2024-04-21T23:15:56.358569+0000","last_active":"2024-04-21T23:18:23.663535+0000","last_peered":"2024-04-21T23:18:23.663535+0000","last_clean":"2024-04-21T23:18:23.663535+0000","last_became_active":"2024-04-21T23:15:56.357533+0000","last_became_peered":"2024-04-21T23:15:56.357533+0000","last_unstale":"2024-04-21T23:18:23.663535+0000","last_undegraded":"2024-04-21T23:18:23.663535+0000","last_fullsized":"2024-04-21T23:18:23.663535+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:59:58.525340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.832026+0000","last_change":"2024-04-21T23:16:00.443981+0000","last_active":"2024-04-21T23:18:23.832026+0000","last_peered":"2024-04-21T23:18:23.832026+0000","last_clean":"2024-04-21T23:18:23.832026+0000","last_became_active":"2024-04-21T23:16:00.443368+0000","last_became_peered":"2024-04-21T23:16:00.443368+0000","last_unstale":"2024-04-21T23:18:23.832026+0000","last_undegraded":"2024-04-21T23:18:23.832026+0000","last_fullsized":"2024-04-21T23:18:23.832026+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:56:48.776540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"95'39","reported_seq":81,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.612653+0000","last_change":"2024-04-21T23:15:55.397334+0000","last_active":"2024-04-21T23:18:23.612653+0000","last_peered":"2024-04-21T23:18:23.612653+0000","last_clean":"2024-04-21T23:18:23.612653+0000","last_became_active":"2024-04-21T23:15:55.396745+0000","last_became_peered":"2024-04-21T23:15:55.396745+0000","last_unstale":"2024-04-21T23:18:23.612653+0000","last_undegraded":"2024-04-21T23:18:23.612653+0000","last_fullsized":"2024-04-21T23:18:23.612653+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:56:21.042187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.663027+0000","last_change":"2024-04-21T23:15:56.351012+0000","last_active":"2024-04-21T23:18:23.663027+0000","last_peered":"2024-04-21T23:18:23.663027+0000","last_clean":"2024-04-21T23:18:23.663027+0000","last_became_active":"2024-04-21T23:15:56.350042+0000","last_became_peered":"2024-04-21T23:15:56.350042+0000","last_unstale":"2024-04-21T23:18:23.663027+0000","last_undegraded":"2024-04-21T23:18:23.663027+0000","last_fullsized":"2024-04-21T23:18:23.663027+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:47:02.629870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.507013+0000","last_change":"2024-04-21T23:16:00.428775+0000","last_active":"2024-04-21T23:18:23.507013+0000","last_peered":"2024-04-21T23:18:23.507013+0000","last_clean":"2024-04-21T23:18:23.507013+0000","last_became_active":"2024-04-21T23:16:00.428278+0000","last_became_peered":"2024-04-21T23:16:00.428278+0000","last_unstale":"2024-04-21T23:18:23.507013+0000","last_undegraded":"2024-04-21T23:18:23.507013+0000","last_fullsized":"2024-04-21T23:18:23.507013+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:24:04.397387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.689442+0000","last_change":"2024-04-21T23:15:55.395524+0000","last_active":"2024-04-21T23:18:23.689442+0000","last_peered":"2024-04-21T23:18:23.689442+0000","last_clean":"2024-04-21T23:18:23.689442+0000","last_became_active":"2024-04-21T23:15:55.395220+0000","last_became_peered":"2024-04-21T23:15:55.395220+0000","last_unstale":"2024-04-21T23:18:23.689442+0000","last_undegraded":"2024-04-21T23:18:23.689442+0000","last_fullsized":"2024-04-21T23:18:23.689442+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:01:37.517620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.663388+0000","last_change":"2024-04-21T23:15:56.352128+0000","last_active":"2024-04-21T23:18:23.663388+0000","last_peered":"2024-04-21T23:18:23.663388+0000","last_clean":"2024-04-21T23:18:23.663388+0000","last_became_active":"2024-04-21T23:15:56.349349+0000","last_became_peered":"2024-04-21T23:15:56.349349+0000","last_unstale":"2024-04-21T23:18:23.663388+0000","last_undegraded":"2024-04-21T23:18:23.663388+0000","last_fullsized":"2024-04-21T23:18:23.663388+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:12:23.247084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.398793+0000","last_change":"2024-04-21T23:15:56.350830+0000","last_active":"2024-04-21T23:18:23.398793+0000","last_peered":"2024-04-21T23:18:23.398793+0000","last_clean":"2024-04-21T23:18:23.398793+0000","last_became_active":"2024-04-21T23:15:56.350385+0000","last_became_peered":"2024-04-21T23:15:56.350385+0000","last_unstale":"2024-04-21T23:18:23.398793+0000","last_undegraded":"2024-04-21T23:18:23.398793+0000","last_fullsized":"2024-04-21T23:18:23.398793+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:18:48.242375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"92'12","reported_seq":44,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.395348+0000","last_change":"2024-04-21T23:15:56.350947+0000","last_active":"2024-04-21T23:18:23.395348+0000","last_peered":"2024-04-21T23:18:23.395348+0000","last_clean":"2024-04-21T23:18:23.395348+0000","last_became_active":"2024-04-21T23:15:56.350383+0000","last_became_peered":"2024-04-21T23:15:56.350383+0000","last_unstale":"2024-04-21T23:18:23.395348+0000","last_undegraded":"2024-04-21T23:18:23.395348+0000","last_fullsized":"2024-04-21T23:18:23.395348+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:07:28.879822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.395154+0000","last_change":"2024-04-21T23:15:56.354559+0000","last_active":"2024-04-21T23:18:23.395154+0000","last_peered":"2024-04-21T23:18:23.395154+0000","last_clean":"2024-04-21T23:18:23.395154+0000","last_became_active":"2024-04-21T23:15:56.354205+0000","last_became_peered":"2024-04-21T23:15:56.354205+0000","last_unstale":"2024-04-21T23:18:23.395154+0000","last_undegraded":"2024-04-21T23:18:23.395154+0000","last_fullsized":"2024-04-21T23:18:23.395154+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:33:50.157971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.832253+0000","last_change":"2024-04-21T23:15:56.358395+0000","last_active":"2024-04-21T23:18:23.832253+0000","last_peered":"2024-04-21T23:18:23.832253+0000","last_clean":"2024-04-21T23:18:23.832253+0000","last_became_active":"2024-04-21T23:15:56.358077+0000","last_became_peered":"2024-04-21T23:15:56.358077+0000","last_unstale":"2024-04-21T23:18:23.832253+0000","last_undegraded":"2024-04-21T23:18:23.832253+0000","last_fullsized":"2024-04-21T23:18:23.832253+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:12:43.124588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"92'4","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.689166+0000","last_change":"2024-04-21T23:15:56.347476+0000","last_active":"2024-04-21T23:18:23.689166+0000","last_peered":"2024-04-21T23:18:23.689166+0000","last_clean":"2024-04-21T23:18:23.689166+0000","last_became_active":"2024-04-21T23:15:56.347097+0000","last_became_peered":"2024-04-21T23:15:56.347097+0000","last_unstale":"2024-04-21T23:18:23.689166+0000","last_undegraded":"2024-04-21T23:18:23.689166+0000","last_fullsized":"2024-04-21T23:18:23.689166+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:11:58.715526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.399175+0000","last_change":"2024-04-21T23:15:56.346413+0000","last_active":"2024-04-21T23:18:23.399175+0000","last_peered":"2024-04-21T23:18:23.399175+0000","last_clean":"2024-04-21T23:18:23.399175+0000","last_became_active":"2024-04-21T23:15:56.346056+0000","last_became_peered":"2024-04-21T23:15:56.346056+0000","last_unstale":"2024-04-21T23:18:23.399175+0000","last_undegraded":"2024-04-21T23:18:23.399175+0000","last_fullsized":"2024-04-21T23:18:23.399175+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:47:16.340323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"92'15","reported_seq":47,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.395818+0000","last_change":"2024-04-21T23:15:56.354876+0000","last_active":"2024-04-21T23:18:23.395818+0000","last_peered":"2024-04-21T23:18:23.395818+0000","last_clean":"2024-04-21T23:18:23.395818+0000","last_became_active":"2024-04-21T23:15:56.354525+0000","last_became_peered":"2024-04-21T23:15:56.354525+0000","last_unstale":"2024-04-21T23:18:23.395818+0000","last_undegraded":"2024-04-21T23:18:23.395818+0000","last_fullsized":"2024-04-21T23:18:23.395818+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:09:23.662664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"92'2","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.501341+0000","last_change":"2024-04-21T23:15:56.358456+0000","last_active":"2024-04-21T23:18:23.501341+0000","last_peered":"2024-04-21T23:18:23.501341+0000","last_clean":"2024-04-21T23:18:23.501341+0000","last_became_active":"2024-04-21T23:15:56.358060+0000","last_became_peered":"2024-04-21T23:15:56.358060+0000","last_unstale":"2024-04-21T23:18:23.501341+0000","last_undegraded":"2024-04-21T23:18:23.501341+0000","last_fullsized":"2024-04-21T23:18:23.501341+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:47:58.928438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.507107+0000","last_change":"2024-04-21T23:15:56.352870+0000","last_active":"2024-04-21T23:18:23.507107+0000","last_peered":"2024-04-21T23:18:23.507107+0000","last_clean":"2024-04-21T23:18:23.507107+0000","last_became_active":"2024-04-21T23:15:56.352267+0000","last_became_peered":"2024-04-21T23:15:56.352267+0000","last_unstale":"2024-04-21T23:18:23.507107+0000","last_undegraded":"2024-04-21T23:18:23.507107+0000","last_fullsized":"2024-04-21T23:18:23.507107+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:32:22.244773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.500521+0000","last_change":"2024-04-21T23:15:56.357468+0000","last_active":"2024-04-21T23:18:23.500521+0000","last_peered":"2024-04-21T23:18:23.500521+0000","last_clean":"2024-04-21T23:18:23.500521+0000","last_became_active":"2024-04-21T23:15:56.357104+0000","last_became_peered":"2024-04-21T23:15:56.357104+0000","last_unstale":"2024-04-21T23:18:23.500521+0000","last_undegraded":"2024-04-21T23:18:23.500521+0000","last_fullsized":"2024-04-21T23:18:23.500521+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:37:36.884036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"92'10","reported_seq":48,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.661120+0000","last_change":"2024-04-21T23:16:52.203142+0000","last_active":"2024-04-21T23:18:23.661120+0000","last_peered":"2024-04-21T23:18:23.661120+0000","last_clean":"2024-04-21T23:18:23.661120+0000","last_became_active":"2024-04-21T23:16:52.202558+0000","last_became_peered":"2024-04-21T23:16:52.202558+0000","last_unstale":"2024-04-21T23:18:23.661120+0000","last_undegraded":"2024-04-21T23:18:23.661120+0000","last_fullsized":"2024-04-21T23:18:23.661120+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:29:05.428123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,8],"acting":[1,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"92'11","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.688864+0000","last_change":"2024-04-21T23:15:56.351901+0000","last_active":"2024-04-21T23:18:23.688864+0000","last_peered":"2024-04-21T23:18:23.688864+0000","last_clean":"2024-04-21T23:18:23.688864+0000","last_became_active":"2024-04-21T23:15:56.351545+0000","last_became_peered":"2024-04-21T23:15:56.351545+0000","last_unstale":"2024-04-21T23:18:23.688864+0000","last_undegraded":"2024-04-21T23:18:23.688864+0000","last_fullsized":"2024-04-21T23:18:23.688864+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:37:20.855631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.398840+0000","last_change":"2024-04-21T23:15:56.355773+0000","last_active":"2024-04-21T23:18:23.398840+0000","last_peered":"2024-04-21T23:18:23.398840+0000","last_clean":"2024-04-21T23:18:23.398840+0000","last_became_active":"2024-04-21T23:15:56.355269+0000","last_became_peered":"2024-04-21T23:15:56.355269+0000","last_unstale":"2024-04-21T23:18:23.398840+0000","last_undegraded":"2024-04-21T23:18:23.398840+0000","last_fullsized":"2024-04-21T23:18:23.398840+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:04:54.540817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":597867,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":854,"num_read_kb":536,"num_write":288,"num_write_kb":1440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":112,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2027520,"data_stored":1793601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":228,"ondisk_log_size":228,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351018,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1093632,"data_stored":536470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612656,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37268,"kb_used_data":1872,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93707884,"statfs":{"total":95995035648,"available":95956873216,"internally_reserved":0,"allocated":1916928,"data_stored":1324276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808885,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36804,"kb_used_data":1464,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708348,"statfs":{"total":95995035648,"available":95957348352,"internally_reserved":0,"allocated":1499136,"data_stored":933906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070522,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28472,"kb_used_data":1452,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716680,"statfs":{"total":95995035648,"available":95965880320,"internally_reserved":0,"allocated":1486848,"data_stored":929618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332159,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28676,"kb_used_data":1656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716476,"statfs":{"total":95995035648,"available":95965671424,"internally_reserved":0,"allocated":1695744,"data_stored":1134045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495684,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1081344,"data_stored":531770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574942,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28148,"kb_used_data":1128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717004,"statfs":{"total":95995035648,"available":95966212096,"internally_reserved":0,"allocated":1155072,"data_stored":599108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738466,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28100,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717052,"statfs":{"total":95995035648,"available":95966261248,"internally_reserved":0,"allocated":1105920,"data_stored":533158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411417,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28168,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716984,"statfs":{"total":95995035648,"available":95966191616,"internally_reserved":0,"allocated":1175552,"data_stored":595362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182483,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36920,"kb_used_data":1584,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708232,"statfs":{"total":95995035648,"available":95957229568,"internally_reserved":0,"allocated":1622016,"data_stored":1056780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920847,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1077248,"data_stored":527336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757321,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37132,"kb_used_data":1796,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708020,"statfs":{"total":95995035648,"available":95957012480,"internally_reserved":0,"allocated":1839104,"data_stored":1259611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":67428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":69169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":544768,"data_stored":525100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":133467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":393842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":394034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":202228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-21T23:18:28.869 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-04-21T23:18:28.869 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-21T23:18:29.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.008+0000 7fca629de640 1 -- 172.21.15.40:0/1241660351 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca5c100fe0 msgr2=0x7fca5c10d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:29.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.008+0000 7fca629de640 1 --2- 172.21.15.40:0/1241660351 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca5c100fe0 0x7fca5c10d4f0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fca54009cd0 tx=0x7fca5402c0f0 comp rx=0 tx=0).stop 2024-04-21T23:18:29.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.008+0000 7fca629de640 1 -- 172.21.15.40:0/1241660351 shutdown_connections 2024-04-21T23:18:29.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.008+0000 7fca629de640 1 --2- 172.21.15.40:0/1241660351 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca5c100fe0 0x7fca5c10d4f0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:29.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.008+0000 7fca629de640 1 --2- 172.21.15.40:0/1241660351 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fca5c100620 0x7fca5c100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:29.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.008+0000 7fca629de640 1 --2- 172.21.15.40:0/1241660351 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca5c106630 0x7fca5c106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:29.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.008+0000 7fca629de640 1 -- 172.21.15.40:0/1241660351 >> 172.21.15.40:0/1241660351 conn(0x7fca5c077740 msgr2=0x7fca5c077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:29.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.008+0000 7fca629de640 1 -- 172.21.15.40:0/1241660351 shutdown_connections 2024-04-21T23:18:29.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.008+0000 7fca629de640 1 -- 172.21.15.40:0/1241660351 wait complete. 2024-04-21T23:18:29.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.009+0000 7fca629de640 1 Processor -- start 2024-04-21T23:18:29.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.009+0000 7fca629de640 1 -- start start 2024-04-21T23:18:29.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.009+0000 7fca629de640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca5c100620 0x7fca5c198e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:29.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.009+0000 7fca629de640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca5c100fe0 0x7fca5c199360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:29.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.009+0000 7fca629de640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fca5c106630 0x7fca5c1a3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:29.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.009+0000 7fca629de640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fca5c110a40 con 0x7fca5c100fe0 2024-04-21T23:18:29.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.009+0000 7fca629de640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fca5c1108c0 con 0x7fca5c100620 2024-04-21T23:18:29.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.009+0000 7fca629de640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fca5c110bc0 con 0x7fca5c106630 2024-04-21T23:18:29.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.010+0000 7fca5b7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca5c100fe0 0x7fca5c199360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:29.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.010+0000 7fca5bfff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca5c100620 0x7fca5c198e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:29.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.010+0000 7fca5bfff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca5c100620 0x7fca5c198e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.40:60176/0 (socket says 172.21.15.40:60176) 2024-04-21T23:18:29.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.010+0000 7fca5bfff640 1 -- 172.21.15.40:0/2234502079 learned_addr learned my addr 172.21.15.40:0/2234502079 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:29.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.010+0000 7fca60f7b640 1 --2- 172.21.15.40:0/2234502079 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fca5c106630 0x7fca5c1a3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:29.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.010+0000 7fca5bfff640 1 -- 172.21.15.40:0/2234502079 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fca5c106630 msgr2=0x7fca5c1a3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:29.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.010+0000 7fca5bfff640 1 --2- 172.21.15.40:0/2234502079 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fca5c106630 0x7fca5c1a3ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:29.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.010+0000 7fca5bfff640 1 -- 172.21.15.40:0/2234502079 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca5c100fe0 msgr2=0x7fca5c199360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:29.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.010+0000 7fca5bfff640 1 --2- 172.21.15.40:0/2234502079 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca5c100fe0 0x7fca5c199360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:29.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.010+0000 7fca5bfff640 1 -- 172.21.15.40:0/2234502079 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca5c1a43e0 con 0x7fca5c100620 2024-04-21T23:18:29.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.010+0000 7fca5b7fe640 1 --2- 172.21.15.40:0/2234502079 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca5c100fe0 0x7fca5c199360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:18:29.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.010+0000 7fca5bfff640 1 --2- 172.21.15.40:0/2234502079 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca5c100620 0x7fca5c198e20 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fca4c00e9f0 tx=0x7fca4c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:29.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.010+0000 7fca597fa640 1 -- 172.21.15.40:0/2234502079 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca4c00cd80 con 0x7fca5c100620 2024-04-21T23:18:29.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.010+0000 7fca597fa640 1 -- 172.21.15.40:0/2234502079 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fca4c010910 con 0x7fca5c100620 2024-04-21T23:18:29.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.010+0000 7fca597fa640 1 -- 172.21.15.40:0/2234502079 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca4c018990 con 0x7fca5c100620 2024-04-21T23:18:29.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.010+0000 7fca629de640 1 -- 172.21.15.40:0/2234502079 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca5c1a45b0 con 0x7fca5c100620 2024-04-21T23:18:29.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.010+0000 7fca629de640 1 -- 172.21.15.40:0/2234502079 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca5c1a4ad0 con 0x7fca5c100620 2024-04-21T23:18:29.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.011+0000 7fca629de640 1 -- 172.21.15.40:0/2234502079 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca5c068560 con 0x7fca5c100620 2024-04-21T23:18:29.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.013+0000 7fca597fa640 1 -- 172.21.15.40:0/2234502079 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7fca4c010ac0 con 0x7fca5c100620 2024-04-21T23:18:29.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.013+0000 7fca597fa640 1 --2- 172.21.15.40:0/2234502079 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fca440a2e20 0x7fca440a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:29.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.013+0000 7fca5b7fe640 1 --2- 172.21.15.40:0/2234502079 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fca440a2e20 0x7fca440a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:29.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.013+0000 7fca597fa640 1 -- 172.21.15.40:0/2234502079 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7fca4c0d01e0 con 0x7fca5c100620 2024-04-21T23:18:29.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.014+0000 7fca5b7fe640 1 --2- 172.21.15.40:0/2234502079 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fca440a2e20 0x7fca440a52e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fca5c069430 tx=0x7fca5000b390 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:29.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.017+0000 7fca597fa640 1 -- 172.21.15.40:0/2234502079 <== mon.2 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fca4c09c1b0 con 0x7fca5c100620 2024-04-21T23:18:29.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:28 smithi040 ceph-mon[27984]: Updating smithi040:/var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-21T23:18:29.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:28 smithi040 ceph-mon[27984]: Updating smithi111:/var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-21T23:18:29.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:28 smithi040 ceph-mon[27984]: Updating smithi174:/var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-21T23:18:29.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:28 smithi040 ceph-mon[27984]: Updating smithi040:/etc/ceph/ceph.client.admin.keyring 2024-04-21T23:18:29.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:28 smithi040 ceph-mon[27984]: Updating smithi111:/etc/ceph/ceph.client.admin.keyring 2024-04-21T23:18:29.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:28 smithi040 ceph-mon[27984]: Updating smithi174:/etc/ceph/ceph.client.admin.keyring 2024-04-21T23:18:29.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:28 smithi040 ceph-mon[27984]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:18:29.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:28 smithi040 ceph-mon[27984]: Updating smithi040:/var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T23:18:29.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:28 smithi040 ceph-mon[27984]: Updating smithi111:/var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T23:18:29.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:28 smithi040 ceph-mon[27984]: Updating smithi174:/var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T23:18:29.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:28 smithi040 ceph-mon[27984]: mgrmap e24: x(active, since 4s), standbys: y, z 2024-04-21T23:18:29.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:28 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:18:29.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:28 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:18:29.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:28 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:18:29.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:28 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:18:29.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:28 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/1320803449' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:18:29.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.202+0000 7fca629de640 1 -- 172.21.15.40:0/2234502079 --> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fca5c108c20 con 0x7fca440a2e20 2024-04-21T23:18:29.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.211+0000 7fca597fa640 1 -- 172.21.15.40:0/2234502079 <== mgr.25111 v2:172.21.15.40:6800/478611259 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339387 (secure 0 0 0) 0x7fca5000bc60 con 0x7fca440a2e20 2024-04-21T23:18:29.214 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:18:29.216 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-21T23:18:29.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.215+0000 7fca629de640 1 -- 172.21.15.40:0/2234502079 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fca440a2e20 msgr2=0x7fca440a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:29.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.215+0000 7fca629de640 1 --2- 172.21.15.40:0/2234502079 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fca440a2e20 0x7fca440a52e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fca5c069430 tx=0x7fca5000b390 comp rx=0 tx=0).stop 2024-04-21T23:18:29.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.215+0000 7fca629de640 1 -- 172.21.15.40:0/2234502079 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca5c100620 msgr2=0x7fca5c198e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:29.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.215+0000 7fca629de640 1 --2- 172.21.15.40:0/2234502079 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca5c100620 0x7fca5c198e20 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fca4c00e9f0 tx=0x7fca4c00eeb0 comp rx=0 tx=0).stop 2024-04-21T23:18:29.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.215+0000 7fca629de640 1 -- 172.21.15.40:0/2234502079 shutdown_connections 2024-04-21T23:18:29.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.215+0000 7fca629de640 1 --2- 172.21.15.40:0/2234502079 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fca440a2e20 0x7fca440a52e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:29.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.215+0000 7fca629de640 1 --2- 172.21.15.40:0/2234502079 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fca5c106630 0x7fca5c1a3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:29.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.215+0000 7fca629de640 1 --2- 172.21.15.40:0/2234502079 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca5c100fe0 0x7fca5c199360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:29.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.215+0000 7fca629de640 1 --2- 172.21.15.40:0/2234502079 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca5c100620 0x7fca5c198e20 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:29.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.215+0000 7fca629de640 1 -- 172.21.15.40:0/2234502079 >> 172.21.15.40:0/2234502079 conn(0x7fca5c077740 msgr2=0x7fca5c102d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:29.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.215+0000 7fca629de640 1 -- 172.21.15.40:0/2234502079 shutdown_connections 2024-04-21T23:18:29.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.215+0000 7fca629de640 1 -- 172.21.15.40:0/2234502079 wait complete. 2024-04-21T23:18:29.225 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-04-21T23:18:27.410882+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1188469,"num_objects":66,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":990,"num_read_kb":655,"num_write":471,"num_write_kb":4380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1180848,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":867,"ondisk_log_size":867,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":373920,"kb_used_data":16356,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124567904,"statfs":{"total":1151940427776,"available":1151557533696,"internally_reserved":0,"allocated":16748544,"data_stored":9961440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":365933882},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.993386"},"pg_stats":[{"pgid":"3.3f","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.399227+0000","last_change":"2024-04-21T23:15:56.349368+0000","last_active":"2024-04-21T23:18:23.399227+0000","last_peered":"2024-04-21T23:18:23.399227+0000","last_clean":"2024-04-21T23:18:23.399227+0000","last_became_active":"2024-04-21T23:15:56.345743+0000","last_became_peered":"2024-04-21T23:15:56.345743+0000","last_unstale":"2024-04-21T23:18:23.399227+0000","last_undegraded":"2024-04-21T23:18:23.399227+0000","last_fullsized":"2024-04-21T23:18:23.399227+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:57:21.574012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"92'4","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.832418+0000","last_change":"2024-04-21T23:15:56.347231+0000","last_active":"2024-04-21T23:18:23.832418+0000","last_peered":"2024-04-21T23:18:23.832418+0000","last_clean":"2024-04-21T23:18:23.832418+0000","last_became_active":"2024-04-21T23:15:56.346847+0000","last_became_peered":"2024-04-21T23:15:56.346847+0000","last_unstale":"2024-04-21T23:18:23.832418+0000","last_undegraded":"2024-04-21T23:18:23.832418+0000","last_fullsized":"2024-04-21T23:18:23.832418+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:20:15.884789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.831956+0000","last_change":"2024-04-21T23:15:56.354897+0000","last_active":"2024-04-21T23:18:23.831956+0000","last_peered":"2024-04-21T23:18:23.831956+0000","last_clean":"2024-04-21T23:18:23.831956+0000","last_became_active":"2024-04-21T23:15:56.354494+0000","last_became_peered":"2024-04-21T23:15:56.354494+0000","last_unstale":"2024-04-21T23:18:23.831956+0000","last_undegraded":"2024-04-21T23:18:23.831956+0000","last_fullsized":"2024-04-21T23:18:23.831956+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:51:11.865310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.660564+0000","last_change":"2024-04-21T23:15:56.355184+0000","last_active":"2024-04-21T23:18:23.660564+0000","last_peered":"2024-04-21T23:18:23.660564+0000","last_clean":"2024-04-21T23:18:23.660564+0000","last_became_active":"2024-04-21T23:15:56.354748+0000","last_became_peered":"2024-04-21T23:15:56.354748+0000","last_unstale":"2024-04-21T23:18:23.660564+0000","last_undegraded":"2024-04-21T23:18:23.660564+0000","last_fullsized":"2024-04-21T23:18:23.660564+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:46:24.867488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"92'4","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.500211+0000","last_change":"2024-04-21T23:15:56.349446+0000","last_active":"2024-04-21T23:18:23.500211+0000","last_peered":"2024-04-21T23:18:23.500211+0000","last_clean":"2024-04-21T23:18:23.500211+0000","last_became_active":"2024-04-21T23:15:56.349050+0000","last_became_peered":"2024-04-21T23:15:56.349050+0000","last_unstale":"2024-04-21T23:18:23.500211+0000","last_undegraded":"2024-04-21T23:18:23.500211+0000","last_fullsized":"2024-04-21T23:18:23.500211+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:15:25.489044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"92'6","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.399061+0000","last_change":"2024-04-21T23:15:56.356095+0000","last_active":"2024-04-21T23:18:23.399061+0000","last_peered":"2024-04-21T23:18:23.399061+0000","last_clean":"2024-04-21T23:18:23.399061+0000","last_became_active":"2024-04-21T23:15:56.355754+0000","last_became_peered":"2024-04-21T23:15:56.355754+0000","last_unstale":"2024-04-21T23:18:23.399061+0000","last_undegraded":"2024-04-21T23:18:23.399061+0000","last_fullsized":"2024-04-21T23:18:23.399061+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:00:27.296457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"92'5","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.662728+0000","last_change":"2024-04-21T23:15:56.349733+0000","last_active":"2024-04-21T23:18:23.662728+0000","last_peered":"2024-04-21T23:18:23.662728+0000","last_clean":"2024-04-21T23:18:23.662728+0000","last_became_active":"2024-04-21T23:15:56.347820+0000","last_became_peered":"2024-04-21T23:15:56.347820+0000","last_unstale":"2024-04-21T23:18:23.662728+0000","last_undegraded":"2024-04-21T23:18:23.662728+0000","last_fullsized":"2024-04-21T23:18:23.662728+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:31:34.453626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"92'9","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.500348+0000","last_change":"2024-04-21T23:15:56.359385+0000","last_active":"2024-04-21T23:18:23.500348+0000","last_peered":"2024-04-21T23:18:23.500348+0000","last_clean":"2024-04-21T23:18:23.500348+0000","last_became_active":"2024-04-21T23:15:56.359074+0000","last_became_peered":"2024-04-21T23:15:56.359074+0000","last_unstale":"2024-04-21T23:18:23.500348+0000","last_undegraded":"2024-04-21T23:18:23.500348+0000","last_fullsized":"2024-04-21T23:18:23.500348+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:40:23.904592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"92'11","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.661940+0000","last_change":"2024-04-21T23:15:56.350886+0000","last_active":"2024-04-21T23:18:23.661940+0000","last_peered":"2024-04-21T23:18:23.661940+0000","last_clean":"2024-04-21T23:18:23.661940+0000","last_became_active":"2024-04-21T23:15:56.350531+0000","last_became_peered":"2024-04-21T23:15:56.350531+0000","last_unstale":"2024-04-21T23:18:23.661940+0000","last_undegraded":"2024-04-21T23:18:23.661940+0000","last_fullsized":"2024-04-21T23:18:23.661940+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:13:12.354473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"92'16","reported_seq":48,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.399459+0000","last_change":"2024-04-21T23:15:56.347376+0000","last_active":"2024-04-21T23:18:23.399459+0000","last_peered":"2024-04-21T23:18:23.399459+0000","last_clean":"2024-04-21T23:18:23.399459+0000","last_became_active":"2024-04-21T23:15:56.347016+0000","last_became_peered":"2024-04-21T23:15:56.347016+0000","last_unstale":"2024-04-21T23:18:23.399459+0000","last_undegraded":"2024-04-21T23:18:23.399459+0000","last_fullsized":"2024-04-21T23:18:23.399459+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:49:46.722931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.663443+0000","last_change":"2024-04-21T23:15:56.351754+0000","last_active":"2024-04-21T23:18:23.663443+0000","last_peered":"2024-04-21T23:18:23.663443+0000","last_clean":"2024-04-21T23:18:23.663443+0000","last_became_active":"2024-04-21T23:15:56.349860+0000","last_became_peered":"2024-04-21T23:15:56.349860+0000","last_unstale":"2024-04-21T23:18:23.663443+0000","last_undegraded":"2024-04-21T23:18:23.663443+0000","last_fullsized":"2024-04-21T23:18:23.663443+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:48:41.469708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.660467+0000","last_change":"2024-04-21T23:15:56.345177+0000","last_active":"2024-04-21T23:18:23.660467+0000","last_peered":"2024-04-21T23:18:23.660467+0000","last_clean":"2024-04-21T23:18:23.660467+0000","last_became_active":"2024-04-21T23:15:56.344791+0000","last_became_peered":"2024-04-21T23:15:56.344791+0000","last_unstale":"2024-04-21T23:18:23.660467+0000","last_undegraded":"2024-04-21T23:18:23.660467+0000","last_fullsized":"2024-04-21T23:18:23.660467+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:33:33.418383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"92'10","reported_seq":48,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.688996+0000","last_change":"2024-04-21T23:16:52.556505+0000","last_active":"2024-04-21T23:18:23.688996+0000","last_peered":"2024-04-21T23:18:23.688996+0000","last_clean":"2024-04-21T23:18:23.688996+0000","last_became_active":"2024-04-21T23:16:52.556177+0000","last_became_peered":"2024-04-21T23:16:52.556177+0000","last_unstale":"2024-04-21T23:18:23.688996+0000","last_undegraded":"2024-04-21T23:18:23.688996+0000","last_fullsized":"2024-04-21T23:18:23.688996+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:17:13.546746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.832040+0000","last_change":"2024-04-21T23:15:56.354544+0000","last_active":"2024-04-21T23:18:23.832040+0000","last_peered":"2024-04-21T23:18:23.832040+0000","last_clean":"2024-04-21T23:18:23.832040+0000","last_became_active":"2024-04-21T23:15:56.354206+0000","last_became_peered":"2024-04-21T23:15:56.354206+0000","last_unstale":"2024-04-21T23:18:23.832040+0000","last_undegraded":"2024-04-21T23:18:23.832040+0000","last_fullsized":"2024-04-21T23:18:23.832040+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:37:19.985422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.831952+0000","last_change":"2024-04-21T23:15:56.357447+0000","last_active":"2024-04-21T23:18:23.831952+0000","last_peered":"2024-04-21T23:18:23.831952+0000","last_clean":"2024-04-21T23:18:23.831952+0000","last_became_active":"2024-04-21T23:15:56.355850+0000","last_became_peered":"2024-04-21T23:15:56.355850+0000","last_unstale":"2024-04-21T23:18:23.831952+0000","last_undegraded":"2024-04-21T23:18:23.831952+0000","last_fullsized":"2024-04-21T23:18:23.831952+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:48:56.648994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.660691+0000","last_change":"2024-04-21T23:15:56.353712+0000","last_active":"2024-04-21T23:18:23.660691+0000","last_peered":"2024-04-21T23:18:23.660691+0000","last_clean":"2024-04-21T23:18:23.660691+0000","last_became_active":"2024-04-21T23:15:56.353388+0000","last_became_peered":"2024-04-21T23:15:56.353388+0000","last_unstale":"2024-04-21T23:18:23.660691+0000","last_undegraded":"2024-04-21T23:18:23.660691+0000","last_fullsized":"2024-04-21T23:18:23.660691+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:43:33.581222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.399252+0000","last_change":"2024-04-21T23:15:56.347041+0000","last_active":"2024-04-21T23:18:23.399252+0000","last_peered":"2024-04-21T23:18:23.399252+0000","last_clean":"2024-04-21T23:18:23.399252+0000","last_became_active":"2024-04-21T23:15:56.346630+0000","last_became_peered":"2024-04-21T23:15:56.346630+0000","last_unstale":"2024-04-21T23:18:23.399252+0000","last_undegraded":"2024-04-21T23:18:23.399252+0000","last_fullsized":"2024-04-21T23:18:23.399252+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:57:42.553666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"92'10","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.498505+0000","last_change":"2024-04-21T23:15:56.353809+0000","last_active":"2024-04-21T23:18:23.498505+0000","last_peered":"2024-04-21T23:18:23.498505+0000","last_clean":"2024-04-21T23:18:23.498505+0000","last_became_active":"2024-04-21T23:15:56.353469+0000","last_became_peered":"2024-04-21T23:15:56.353469+0000","last_unstale":"2024-04-21T23:18:23.498505+0000","last_undegraded":"2024-04-21T23:18:23.498505+0000","last_fullsized":"2024-04-21T23:18:23.498505+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:11:14.136845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.395486+0000","last_change":"2024-04-21T23:15:56.354500+0000","last_active":"2024-04-21T23:18:23.395486+0000","last_peered":"2024-04-21T23:18:23.395486+0000","last_clean":"2024-04-21T23:18:23.395486+0000","last_became_active":"2024-04-21T23:15:56.353893+0000","last_became_peered":"2024-04-21T23:15:56.353893+0000","last_unstale":"2024-04-21T23:18:23.395486+0000","last_undegraded":"2024-04-21T23:18:23.395486+0000","last_fullsized":"2024-04-21T23:18:23.395486+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:14:38.121448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.395523+0000","last_change":"2024-04-21T23:17:53.274634+0000","last_active":"2024-04-21T23:18:23.395523+0000","last_peered":"2024-04-21T23:18:23.395523+0000","last_clean":"2024-04-21T23:18:23.395523+0000","last_became_active":"2024-04-21T23:17:53.274018+0000","last_became_peered":"2024-04-21T23:17:53.274018+0000","last_unstale":"2024-04-21T23:18:23.395523+0000","last_undegraded":"2024-04-21T23:18:23.395523+0000","last_fullsized":"2024-04-21T23:18:23.395523+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:04:55.017447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8,0],"acting":[2,3,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"90'9","reported_seq":44,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.498460+0000","last_change":"2024-04-21T23:15:55.402553+0000","last_active":"2024-04-21T23:18:23.498460+0000","last_peered":"2024-04-21T23:18:23.498460+0000","last_clean":"2024-04-21T23:18:23.498460+0000","last_became_active":"2024-04-21T23:15:55.402192+0000","last_became_peered":"2024-04-21T23:15:55.402192+0000","last_unstale":"2024-04-21T23:18:23.498460+0000","last_undegraded":"2024-04-21T23:18:23.498460+0000","last_fullsized":"2024-04-21T23:18:23.498460+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:24:57.467906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"92'5","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.501309+0000","last_change":"2024-04-21T23:15:56.358797+0000","last_active":"2024-04-21T23:18:23.501309+0000","last_peered":"2024-04-21T23:18:23.501309+0000","last_clean":"2024-04-21T23:18:23.501309+0000","last_became_active":"2024-04-21T23:15:56.357997+0000","last_became_peered":"2024-04-21T23:15:56.357997+0000","last_unstale":"2024-04-21T23:18:23.501309+0000","last_undegraded":"2024-04-21T23:18:23.501309+0000","last_fullsized":"2024-04-21T23:18:23.501309+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:20:50.891506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.659831+0000","last_change":"2024-04-21T23:16:00.443368+0000","last_active":"2024-04-21T23:18:23.659831+0000","last_peered":"2024-04-21T23:18:23.659831+0000","last_clean":"2024-04-21T23:18:23.659831+0000","last_became_active":"2024-04-21T23:16:00.442940+0000","last_became_peered":"2024-04-21T23:16:00.442940+0000","last_unstale":"2024-04-21T23:18:23.659831+0000","last_undegraded":"2024-04-21T23:18:23.659831+0000","last_fullsized":"2024-04-21T23:18:23.659831+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:17:15.706460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"90'2","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.501369+0000","last_change":"2024-04-21T23:15:55.405216+0000","last_active":"2024-04-21T23:18:23.501369+0000","last_peered":"2024-04-21T23:18:23.501369+0000","last_clean":"2024-04-21T23:18:23.501369+0000","last_became_active":"2024-04-21T23:15:55.404844+0000","last_became_peered":"2024-04-21T23:15:55.404844+0000","last_unstale":"2024-04-21T23:18:23.501369+0000","last_undegraded":"2024-04-21T23:18:23.501369+0000","last_fullsized":"2024-04-21T23:18:23.501369+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:56:28.691844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.832115+0000","last_change":"2024-04-21T23:15:56.355789+0000","last_active":"2024-04-21T23:18:23.832115+0000","last_peered":"2024-04-21T23:18:23.832115+0000","last_clean":"2024-04-21T23:18:23.832115+0000","last_became_active":"2024-04-21T23:15:56.354972+0000","last_became_peered":"2024-04-21T23:15:56.354972+0000","last_unstale":"2024-04-21T23:18:23.832115+0000","last_undegraded":"2024-04-21T23:18:23.832115+0000","last_fullsized":"2024-04-21T23:18:23.832115+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:11:36.831453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.660103+0000","last_change":"2024-04-21T23:16:00.443241+0000","last_active":"2024-04-21T23:18:23.660103+0000","last_peered":"2024-04-21T23:18:23.660103+0000","last_clean":"2024-04-21T23:18:23.660103+0000","last_became_active":"2024-04-21T23:16:00.442820+0000","last_became_peered":"2024-04-21T23:16:00.442820+0000","last_unstale":"2024-04-21T23:18:23.660103+0000","last_undegraded":"2024-04-21T23:18:23.660103+0000","last_fullsized":"2024-04-21T23:18:23.660103+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:28:04.032621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"90'4","reported_seq":49,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.500708+0000","last_change":"2024-04-21T23:16:52.212906+0000","last_active":"2024-04-21T23:18:23.500708+0000","last_peered":"2024-04-21T23:18:23.500708+0000","last_clean":"2024-04-21T23:18:23.500708+0000","last_became_active":"2024-04-21T23:16:52.205828+0000","last_became_peered":"2024-04-21T23:16:52.205828+0000","last_unstale":"2024-04-21T23:18:23.500708+0000","last_undegraded":"2024-04-21T23:18:23.500708+0000","last_fullsized":"2024-04-21T23:18:23.500708+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:20:01.936578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.662333+0000","last_change":"2024-04-21T23:15:56.354462+0000","last_active":"2024-04-21T23:18:23.662333+0000","last_peered":"2024-04-21T23:18:23.662333+0000","last_clean":"2024-04-21T23:18:23.662333+0000","last_became_active":"2024-04-21T23:15:56.354134+0000","last_became_peered":"2024-04-21T23:15:56.354134+0000","last_unstale":"2024-04-21T23:18:23.662333+0000","last_undegraded":"2024-04-21T23:18:23.662333+0000","last_fullsized":"2024-04-21T23:18:23.662333+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:27:40.809483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.660710+0000","last_change":"2024-04-21T23:16:00.427717+0000","last_active":"2024-04-21T23:18:23.660710+0000","last_peered":"2024-04-21T23:18:23.660710+0000","last_clean":"2024-04-21T23:18:23.660710+0000","last_became_active":"2024-04-21T23:16:00.427296+0000","last_became_peered":"2024-04-21T23:16:00.427296+0000","last_unstale":"2024-04-21T23:18:23.660710+0000","last_undegraded":"2024-04-21T23:18:23.660710+0000","last_fullsized":"2024-04-21T23:18:23.660710+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:03:41.650045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.500411+0000","last_change":"2024-04-21T23:15:55.398641+0000","last_active":"2024-04-21T23:18:23.500411+0000","last_peered":"2024-04-21T23:18:23.500411+0000","last_clean":"2024-04-21T23:18:23.500411+0000","last_became_active":"2024-04-21T23:15:55.397712+0000","last_became_peered":"2024-04-21T23:15:55.397712+0000","last_unstale":"2024-04-21T23:18:23.500411+0000","last_undegraded":"2024-04-21T23:18:23.500411+0000","last_fullsized":"2024-04-21T23:18:23.500411+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:28:09.266740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"92'15","reported_seq":46,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.507482+0000","last_change":"2024-04-21T23:15:56.345359+0000","last_active":"2024-04-21T23:18:23.507482+0000","last_peered":"2024-04-21T23:18:23.507482+0000","last_clean":"2024-04-21T23:18:23.507482+0000","last_became_active":"2024-04-21T23:15:56.343652+0000","last_became_peered":"2024-04-21T23:15:56.343652+0000","last_unstale":"2024-04-21T23:18:23.507482+0000","last_undegraded":"2024-04-21T23:18:23.507482+0000","last_fullsized":"2024-04-21T23:18:23.507482+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:18:10.057999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.395459+0000","last_change":"2024-04-21T23:16:00.430640+0000","last_active":"2024-04-21T23:18:23.395459+0000","last_peered":"2024-04-21T23:18:23.395459+0000","last_clean":"2024-04-21T23:18:23.395459+0000","last_became_active":"2024-04-21T23:16:00.430141+0000","last_became_peered":"2024-04-21T23:16:00.430141+0000","last_unstale":"2024-04-21T23:18:23.395459+0000","last_undegraded":"2024-04-21T23:18:23.395459+0000","last_fullsized":"2024-04-21T23:18:23.395459+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:41:13.749780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"95'6","reported_seq":165,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.688107+0000","last_change":"2024-04-21T23:15:55.399823+0000","last_active":"2024-04-21T23:18:23.688107+0000","last_peered":"2024-04-21T23:18:23.688107+0000","last_clean":"2024-04-21T23:18:23.688107+0000","last_became_active":"2024-04-21T23:15:55.399490+0000","last_became_peered":"2024-04-21T23:15:55.399490+0000","last_unstale":"2024-04-21T23:18:23.688107+0000","last_undegraded":"2024-04-21T23:18:23.688107+0000","last_fullsized":"2024-04-21T23:18:23.688107+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:54:38.124841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":125,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.500231+0000","last_change":"2024-04-21T23:15:56.346444+0000","last_active":"2024-04-21T23:18:23.500231+0000","last_peered":"2024-04-21T23:18:23.500231+0000","last_clean":"2024-04-21T23:18:23.500231+0000","last_became_active":"2024-04-21T23:15:56.346091+0000","last_became_peered":"2024-04-21T23:15:56.346091+0000","last_unstale":"2024-04-21T23:18:23.500231+0000","last_undegraded":"2024-04-21T23:18:23.500231+0000","last_fullsized":"2024-04-21T23:18:23.500231+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:41:11.144570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.395544+0000","last_change":"2024-04-21T23:16:00.438198+0000","last_active":"2024-04-21T23:18:23.395544+0000","last_peered":"2024-04-21T23:18:23.395544+0000","last_clean":"2024-04-21T23:18:23.395544+0000","last_became_active":"2024-04-21T23:16:00.437767+0000","last_became_peered":"2024-04-21T23:16:00.437767+0000","last_unstale":"2024-04-21T23:18:23.395544+0000","last_undegraded":"2024-04-21T23:18:23.395544+0000","last_fullsized":"2024-04-21T23:18:23.395544+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:35:11.356414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"90'3","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.399081+0000","last_change":"2024-04-21T23:15:55.403896+0000","last_active":"2024-04-21T23:18:23.399081+0000","last_peered":"2024-04-21T23:18:23.399081+0000","last_clean":"2024-04-21T23:18:23.399081+0000","last_became_active":"2024-04-21T23:15:55.403423+0000","last_became_peered":"2024-04-21T23:15:55.403423+0000","last_unstale":"2024-04-21T23:18:23.399081+0000","last_undegraded":"2024-04-21T23:18:23.399081+0000","last_fullsized":"2024-04-21T23:18:23.399081+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:34:44.252259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"92'8","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.498057+0000","last_change":"2024-04-21T23:16:52.349354+0000","last_active":"2024-04-21T23:18:23.498057+0000","last_peered":"2024-04-21T23:18:23.498057+0000","last_clean":"2024-04-21T23:18:23.498057+0000","last_became_active":"2024-04-21T23:16:52.348991+0000","last_became_peered":"2024-04-21T23:16:52.348991+0000","last_unstale":"2024-04-21T23:18:23.498057+0000","last_undegraded":"2024-04-21T23:18:23.498057+0000","last_fullsized":"2024-04-21T23:18:23.498057+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:42:57.975963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.399121+0000","last_change":"2024-04-21T23:16:00.434104+0000","last_active":"2024-04-21T23:18:23.399121+0000","last_peered":"2024-04-21T23:18:23.399121+0000","last_clean":"2024-04-21T23:18:23.399121+0000","last_became_active":"2024-04-21T23:16:00.433681+0000","last_became_peered":"2024-04-21T23:16:00.433681+0000","last_unstale":"2024-04-21T23:18:23.399121+0000","last_undegraded":"2024-04-21T23:18:23.399121+0000","last_fullsized":"2024-04-21T23:18:23.399121+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:10:17.986652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"90'2","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.688659+0000","last_change":"2024-04-21T23:15:55.397536+0000","last_active":"2024-04-21T23:18:23.688659+0000","last_peered":"2024-04-21T23:18:23.688659+0000","last_clean":"2024-04-21T23:18:23.688659+0000","last_became_active":"2024-04-21T23:15:55.397163+0000","last_became_peered":"2024-04-21T23:15:55.397163+0000","last_unstale":"2024-04-21T23:18:23.688659+0000","last_undegraded":"2024-04-21T23:18:23.688659+0000","last_fullsized":"2024-04-21T23:18:23.688659+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:05:27.777804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.500246+0000","last_change":"2024-04-21T23:15:56.351840+0000","last_active":"2024-04-21T23:18:23.500246+0000","last_peered":"2024-04-21T23:18:23.500246+0000","last_clean":"2024-04-21T23:18:23.500246+0000","last_became_active":"2024-04-21T23:15:56.351523+0000","last_became_peered":"2024-04-21T23:15:56.351523+0000","last_unstale":"2024-04-21T23:18:23.500246+0000","last_undegraded":"2024-04-21T23:18:23.500246+0000","last_fullsized":"2024-04-21T23:18:23.500246+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:57:11.714284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.832073+0000","last_change":"2024-04-21T23:16:00.437473+0000","last_active":"2024-04-21T23:18:23.832073+0000","last_peered":"2024-04-21T23:18:23.832073+0000","last_clean":"2024-04-21T23:18:23.832073+0000","last_became_active":"2024-04-21T23:16:00.437014+0000","last_became_peered":"2024-04-21T23:16:00.437014+0000","last_unstale":"2024-04-21T23:18:23.832073+0000","last_undegraded":"2024-04-21T23:18:23.832073+0000","last_fullsized":"2024-04-21T23:18:23.832073+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:37:14.241896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"90'1","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.507228+0000","last_change":"2024-04-21T23:15:55.399213+0000","last_active":"2024-04-21T23:18:23.507228+0000","last_peered":"2024-04-21T23:18:23.507228+0000","last_clean":"2024-04-21T23:18:23.507228+0000","last_became_active":"2024-04-21T23:15:55.398880+0000","last_became_peered":"2024-04-21T23:15:55.398880+0000","last_unstale":"2024-04-21T23:18:23.507228+0000","last_undegraded":"2024-04-21T23:18:23.507228+0000","last_fullsized":"2024-04-21T23:18:23.507228+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:56:32.236862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"92'11","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.497698+0000","last_change":"2024-04-21T23:15:56.349330+0000","last_active":"2024-04-21T23:18:23.497698+0000","last_peered":"2024-04-21T23:18:23.497698+0000","last_clean":"2024-04-21T23:18:23.497698+0000","last_became_active":"2024-04-21T23:15:56.348988+0000","last_became_peered":"2024-04-21T23:15:56.348988+0000","last_unstale":"2024-04-21T23:18:23.497698+0000","last_undegraded":"2024-04-21T23:18:23.497698+0000","last_fullsized":"2024-04-21T23:18:23.497698+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:11:19.650119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.500463+0000","last_change":"2024-04-21T23:16:00.431959+0000","last_active":"2024-04-21T23:18:23.500463+0000","last_peered":"2024-04-21T23:18:23.500463+0000","last_clean":"2024-04-21T23:18:23.500463+0000","last_became_active":"2024-04-21T23:16:00.431570+0000","last_became_peered":"2024-04-21T23:16:00.431570+0000","last_unstale":"2024-04-21T23:18:23.500463+0000","last_undegraded":"2024-04-21T23:18:23.500463+0000","last_fullsized":"2024-04-21T23:18:23.500463+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:02:54.057326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"95'4","reported_seq":160,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.686988+0000","last_change":"2024-04-21T23:15:55.404505+0000","last_active":"2024-04-21T23:18:23.686988+0000","last_peered":"2024-04-21T23:18:23.686988+0000","last_clean":"2024-04-21T23:18:23.686988+0000","last_became_active":"2024-04-21T23:15:55.403945+0000","last_became_peered":"2024-04-21T23:15:55.403945+0000","last_unstale":"2024-04-21T23:18:23.686988+0000","last_undegraded":"2024-04-21T23:18:23.686988+0000","last_fullsized":"2024-04-21T23:18:23.686988+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:54:37.647597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":122,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"92'10","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.661999+0000","last_change":"2024-04-21T23:15:56.356238+0000","last_active":"2024-04-21T23:18:23.661999+0000","last_peered":"2024-04-21T23:18:23.661999+0000","last_clean":"2024-04-21T23:18:23.661999+0000","last_became_active":"2024-04-21T23:15:56.355116+0000","last_became_peered":"2024-04-21T23:15:56.355116+0000","last_unstale":"2024-04-21T23:18:23.661999+0000","last_undegraded":"2024-04-21T23:18:23.661999+0000","last_fullsized":"2024-04-21T23:18:23.661999+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:43:20.468549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.497631+0000","last_change":"2024-04-21T23:16:00.439069+0000","last_active":"2024-04-21T23:18:23.497631+0000","last_peered":"2024-04-21T23:18:23.497631+0000","last_clean":"2024-04-21T23:18:23.497631+0000","last_became_active":"2024-04-21T23:16:00.438510+0000","last_became_peered":"2024-04-21T23:16:00.438510+0000","last_unstale":"2024-04-21T23:18:23.497631+0000","last_undegraded":"2024-04-21T23:18:23.497631+0000","last_fullsized":"2024-04-21T23:18:23.497631+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:41:50.814112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"90'2","reported_seq":46,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.663309+0000","last_change":"2024-04-21T23:16:52.206602+0000","last_active":"2024-04-21T23:18:23.663309+0000","last_peered":"2024-04-21T23:18:23.663309+0000","last_clean":"2024-04-21T23:18:23.663309+0000","last_became_active":"2024-04-21T23:16:52.202755+0000","last_became_peered":"2024-04-21T23:16:52.202755+0000","last_unstale":"2024-04-21T23:18:23.663309+0000","last_undegraded":"2024-04-21T23:18:23.663309+0000","last_fullsized":"2024-04-21T23:18:23.663309+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:01:02.437392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.832264+0000","last_change":"2024-04-21T23:15:56.357228+0000","last_active":"2024-04-21T23:18:23.832264+0000","last_peered":"2024-04-21T23:18:23.832264+0000","last_clean":"2024-04-21T23:18:23.832264+0000","last_became_active":"2024-04-21T23:15:56.356808+0000","last_became_peered":"2024-04-21T23:15:56.356808+0000","last_unstale":"2024-04-21T23:18:23.832264+0000","last_undegraded":"2024-04-21T23:18:23.832264+0000","last_fullsized":"2024-04-21T23:18:23.832264+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:15:10.728530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.660434+0000","last_change":"2024-04-21T23:16:00.443569+0000","last_active":"2024-04-21T23:18:23.660434+0000","last_peered":"2024-04-21T23:18:23.660434+0000","last_clean":"2024-04-21T23:18:23.660434+0000","last_became_active":"2024-04-21T23:16:00.442972+0000","last_became_peered":"2024-04-21T23:16:00.442972+0000","last_unstale":"2024-04-21T23:18:23.660434+0000","last_undegraded":"2024-04-21T23:18:23.660434+0000","last_fullsized":"2024-04-21T23:18:23.660434+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:35:21.471407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"90'4","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.507401+0000","last_change":"2024-04-21T23:15:55.400134+0000","last_active":"2024-04-21T23:18:23.507401+0000","last_peered":"2024-04-21T23:18:23.507401+0000","last_clean":"2024-04-21T23:18:23.507401+0000","last_became_active":"2024-04-21T23:15:55.399768+0000","last_became_peered":"2024-04-21T23:15:55.399768+0000","last_unstale":"2024-04-21T23:18:23.507401+0000","last_undegraded":"2024-04-21T23:18:23.507401+0000","last_fullsized":"2024-04-21T23:18:23.507401+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:15:09.323035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"92'11","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.662422+0000","last_change":"2024-04-21T23:15:56.357278+0000","last_active":"2024-04-21T23:18:23.662422+0000","last_peered":"2024-04-21T23:18:23.662422+0000","last_clean":"2024-04-21T23:18:23.662422+0000","last_became_active":"2024-04-21T23:15:56.356902+0000","last_became_peered":"2024-04-21T23:15:56.356902+0000","last_unstale":"2024-04-21T23:18:23.662422+0000","last_undegraded":"2024-04-21T23:18:23.662422+0000","last_fullsized":"2024-04-21T23:18:23.662422+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:19:05.269109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.497842+0000","last_change":"2024-04-21T23:16:00.439704+0000","last_active":"2024-04-21T23:18:23.497842+0000","last_peered":"2024-04-21T23:18:23.497842+0000","last_clean":"2024-04-21T23:18:23.497842+0000","last_became_active":"2024-04-21T23:16:00.439286+0000","last_became_peered":"2024-04-21T23:16:00.439286+0000","last_unstale":"2024-04-21T23:18:23.497842+0000","last_undegraded":"2024-04-21T23:18:23.497842+0000","last_fullsized":"2024-04-21T23:18:23.497842+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:08:59.194481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"95'38","reported_seq":326,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.687640+0000","last_change":"2024-04-21T23:15:55.403344+0000","last_active":"2024-04-21T23:18:23.687640+0000","last_peered":"2024-04-21T23:18:23.687640+0000","last_clean":"2024-04-21T23:18:23.687640+0000","last_became_active":"2024-04-21T23:15:55.402876+0000","last_became_peered":"2024-04-21T23:15:55.402876+0000","last_unstale":"2024-04-21T23:18:23.687640+0000","last_undegraded":"2024-04-21T23:18:23.687640+0000","last_fullsized":"2024-04-21T23:18:23.687640+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:55:59.082964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67428,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":254,"num_read_kb":193,"num_write":39,"num_write_kb":85,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.500611+0000","last_change":"2024-04-21T23:15:56.343897+0000","last_active":"2024-04-21T23:18:23.500611+0000","last_peered":"2024-04-21T23:18:23.500611+0000","last_clean":"2024-04-21T23:18:23.500611+0000","last_became_active":"2024-04-21T23:15:56.343567+0000","last_became_peered":"2024-04-21T23:15:56.343567+0000","last_unstale":"2024-04-21T23:18:23.500611+0000","last_undegraded":"2024-04-21T23:18:23.500611+0000","last_fullsized":"2024-04-21T23:18:23.500611+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:57:07.955722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.507737+0000","last_change":"2024-04-21T23:16:00.432722+0000","last_active":"2024-04-21T23:18:23.507737+0000","last_peered":"2024-04-21T23:18:23.507737+0000","last_clean":"2024-04-21T23:18:23.507737+0000","last_became_active":"2024-04-21T23:16:00.432257+0000","last_became_peered":"2024-04-21T23:16:00.432257+0000","last_unstale":"2024-04-21T23:18:23.507737+0000","last_undegraded":"2024-04-21T23:18:23.507737+0000","last_fullsized":"2024-04-21T23:18:23.507737+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:10:14.804490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"90'6","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.395461+0000","last_change":"2024-04-21T23:15:55.401300+0000","last_active":"2024-04-21T23:18:23.395461+0000","last_peered":"2024-04-21T23:18:23.395461+0000","last_clean":"2024-04-21T23:18:23.395461+0000","last_became_active":"2024-04-21T23:15:55.400904+0000","last_became_peered":"2024-04-21T23:15:55.400904+0000","last_unstale":"2024-04-21T23:18:23.395461+0000","last_undegraded":"2024-04-21T23:18:23.395461+0000","last_fullsized":"2024-04-21T23:18:23.395461+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:29:20.439936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.688806+0000","last_change":"2024-04-21T23:15:56.351046+0000","last_active":"2024-04-21T23:18:23.688806+0000","last_peered":"2024-04-21T23:18:23.688806+0000","last_clean":"2024-04-21T23:18:23.688806+0000","last_became_active":"2024-04-21T23:15:56.350662+0000","last_became_peered":"2024-04-21T23:15:56.350662+0000","last_unstale":"2024-04-21T23:18:23.688806+0000","last_undegraded":"2024-04-21T23:18:23.688806+0000","last_fullsized":"2024-04-21T23:18:23.688806+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:40:16.482061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.398794+0000","last_change":"2024-04-21T23:16:00.439816+0000","last_active":"2024-04-21T23:18:23.398794+0000","last_peered":"2024-04-21T23:18:23.398794+0000","last_clean":"2024-04-21T23:18:23.398794+0000","last_became_active":"2024-04-21T23:16:00.439330+0000","last_became_peered":"2024-04-21T23:16:00.439330+0000","last_unstale":"2024-04-21T23:18:23.398794+0000","last_undegraded":"2024-04-21T23:18:23.398794+0000","last_fullsized":"2024-04-21T23:18:23.398794+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:36:24.472968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"90'3","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.507439+0000","last_change":"2024-04-21T23:15:55.398588+0000","last_active":"2024-04-21T23:18:23.507439+0000","last_peered":"2024-04-21T23:18:23.507439+0000","last_clean":"2024-04-21T23:18:23.507439+0000","last_became_active":"2024-04-21T23:15:55.398264+0000","last_became_peered":"2024-04-21T23:15:55.398264+0000","last_unstale":"2024-04-21T23:18:23.507439+0000","last_undegraded":"2024-04-21T23:18:23.507439+0000","last_fullsized":"2024-04-21T23:18:23.507439+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:48:09.293481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"92'15","reported_seq":53,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.663205+0000","last_change":"2024-04-21T23:16:52.202037+0000","last_active":"2024-04-21T23:18:23.663205+0000","last_peered":"2024-04-21T23:18:23.663205+0000","last_clean":"2024-04-21T23:18:23.663205+0000","last_became_active":"2024-04-21T23:16:52.201244+0000","last_became_peered":"2024-04-21T23:16:52.201244+0000","last_unstale":"2024-04-21T23:18:23.663205+0000","last_undegraded":"2024-04-21T23:18:23.663205+0000","last_fullsized":"2024-04-21T23:18:23.663205+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:37:34.530110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.399332+0000","last_change":"2024-04-21T23:17:53.363902+0000","last_active":"2024-04-21T23:18:23.399332+0000","last_peered":"2024-04-21T23:18:23.399332+0000","last_clean":"2024-04-21T23:18:23.399332+0000","last_became_active":"2024-04-21T23:17:53.363433+0000","last_became_peered":"2024-04-21T23:17:53.363433+0000","last_unstale":"2024-04-21T23:18:23.399332+0000","last_undegraded":"2024-04-21T23:18:23.399332+0000","last_fullsized":"2024-04-21T23:18:23.399332+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:52:07.725946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,0,3],"acting":[10,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.11","version":"95'8","reported_seq":170,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.689594+0000","last_change":"2024-04-21T23:15:55.397503+0000","last_active":"2024-04-21T23:18:23.689594+0000","last_peered":"2024-04-21T23:18:23.689594+0000","last_clean":"2024-04-21T23:18:23.689594+0000","last_became_active":"2024-04-21T23:15:55.397181+0000","last_became_peered":"2024-04-21T23:15:55.397181+0000","last_unstale":"2024-04-21T23:18:23.689594+0000","last_undegraded":"2024-04-21T23:18:23.689594+0000","last_fullsized":"2024-04-21T23:18:23.689594+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:47:08.775783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2210,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":129,"num_read_kb":7,"num_write":7,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.663078+0000","last_change":"2024-04-21T23:15:56.350872+0000","last_active":"2024-04-21T23:18:23.663078+0000","last_peered":"2024-04-21T23:18:23.663078+0000","last_clean":"2024-04-21T23:18:23.663078+0000","last_became_active":"2024-04-21T23:15:56.350519+0000","last_became_peered":"2024-04-21T23:15:56.350519+0000","last_unstale":"2024-04-21T23:18:23.663078+0000","last_undegraded":"2024-04-21T23:18:23.663078+0000","last_fullsized":"2024-04-21T23:18:23.663078+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:23:37.753893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.689394+0000","last_change":"2024-04-21T23:16:00.436069+0000","last_active":"2024-04-21T23:18:23.689394+0000","last_peered":"2024-04-21T23:18:23.689394+0000","last_clean":"2024-04-21T23:18:23.689394+0000","last_became_active":"2024-04-21T23:16:00.435617+0000","last_became_peered":"2024-04-21T23:16:00.435617+0000","last_unstale":"2024-04-21T23:18:23.689394+0000","last_undegraded":"2024-04-21T23:18:23.689394+0000","last_fullsized":"2024-04-21T23:18:23.689394+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:21:44.256928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"90'15","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.398588+0000","last_change":"2024-04-21T23:16:52.220742+0000","last_active":"2024-04-21T23:18:23.398588+0000","last_peered":"2024-04-21T23:18:23.398588+0000","last_clean":"2024-04-21T23:18:23.398588+0000","last_became_active":"2024-04-21T23:16:52.204787+0000","last_became_peered":"2024-04-21T23:16:52.204787+0000","last_unstale":"2024-04-21T23:18:23.398588+0000","last_undegraded":"2024-04-21T23:18:23.398588+0000","last_fullsized":"2024-04-21T23:18:23.398588+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:53:12.057014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"92'6","reported_seq":44,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.507781+0000","last_change":"2024-04-21T23:16:52.202341+0000","last_active":"2024-04-21T23:18:23.507781+0000","last_peered":"2024-04-21T23:18:23.507781+0000","last_clean":"2024-04-21T23:18:23.507781+0000","last_became_active":"2024-04-21T23:16:52.201528+0000","last_became_peered":"2024-04-21T23:16:52.201528+0000","last_unstale":"2024-04-21T23:18:23.507781+0000","last_undegraded":"2024-04-21T23:18:23.507781+0000","last_fullsized":"2024-04-21T23:18:23.507781+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:41:24.767379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.500204+0000","last_change":"2024-04-21T23:17:53.274665+0000","last_active":"2024-04-21T23:18:23.500204+0000","last_peered":"2024-04-21T23:18:23.500204+0000","last_clean":"2024-04-21T23:18:23.500204+0000","last_became_active":"2024-04-21T23:17:53.274146+0000","last_became_peered":"2024-04-21T23:17:53.274146+0000","last_unstale":"2024-04-21T23:18:23.500204+0000","last_undegraded":"2024-04-21T23:18:23.500204+0000","last_fullsized":"2024-04-21T23:18:23.500204+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:15:08.842226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,3,1],"acting":[8,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.f","version":"94'3","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.500747+0000","last_change":"2024-04-21T23:15:55.399032+0000","last_active":"2024-04-21T23:18:23.500747+0000","last_peered":"2024-04-21T23:18:23.500747+0000","last_clean":"2024-04-21T23:18:23.500747+0000","last_became_active":"2024-04-21T23:15:55.398208+0000","last_became_peered":"2024-04-21T23:15:55.398208+0000","last_unstale":"2024-04-21T23:18:23.500747+0000","last_undegraded":"2024-04-21T23:18:23.500747+0000","last_fullsized":"2024-04-21T23:18:23.500747+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:14:59.520571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"92'5","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.399041+0000","last_change":"2024-04-21T23:15:56.355357+0000","last_active":"2024-04-21T23:18:23.399041+0000","last_peered":"2024-04-21T23:18:23.399041+0000","last_clean":"2024-04-21T23:18:23.399041+0000","last_became_active":"2024-04-21T23:15:56.354704+0000","last_became_peered":"2024-04-21T23:15:56.354704+0000","last_unstale":"2024-04-21T23:18:23.399041+0000","last_undegraded":"2024-04-21T23:18:23.399041+0000","last_fullsized":"2024-04-21T23:18:23.399041+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:14:02.368659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.500362+0000","last_change":"2024-04-21T23:16:00.427781+0000","last_active":"2024-04-21T23:18:23.500362+0000","last_peered":"2024-04-21T23:18:23.500362+0000","last_clean":"2024-04-21T23:18:23.500362+0000","last_became_active":"2024-04-21T23:16:00.427244+0000","last_became_peered":"2024-04-21T23:16:00.427244+0000","last_unstale":"2024-04-21T23:18:23.500362+0000","last_undegraded":"2024-04-21T23:18:23.500362+0000","last_fullsized":"2024-04-21T23:18:23.500362+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:21:30.548865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.498142+0000","last_change":"2024-04-21T23:15:55.405687+0000","last_active":"2024-04-21T23:18:23.498142+0000","last_peered":"2024-04-21T23:18:23.498142+0000","last_clean":"2024-04-21T23:18:23.498142+0000","last_became_active":"2024-04-21T23:15:55.405379+0000","last_became_peered":"2024-04-21T23:15:55.405379+0000","last_unstale":"2024-04-21T23:18:23.498142+0000","last_undegraded":"2024-04-21T23:18:23.498142+0000","last_fullsized":"2024-04-21T23:18:23.498142+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:12:44.192092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"92'4","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.399301+0000","last_change":"2024-04-21T23:15:56.348989+0000","last_active":"2024-04-21T23:18:23.399301+0000","last_peered":"2024-04-21T23:18:23.399301+0000","last_clean":"2024-04-21T23:18:23.399301+0000","last_became_active":"2024-04-21T23:15:56.348039+0000","last_became_peered":"2024-04-21T23:15:56.348039+0000","last_unstale":"2024-04-21T23:18:23.399301+0000","last_undegraded":"2024-04-21T23:18:23.399301+0000","last_fullsized":"2024-04-21T23:18:23.399301+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:39:15.577567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.688857+0000","last_change":"2024-04-21T23:16:00.434557+0000","last_active":"2024-04-21T23:18:23.688857+0000","last_peered":"2024-04-21T23:18:23.688857+0000","last_clean":"2024-04-21T23:18:23.688857+0000","last_became_active":"2024-04-21T23:16:00.433737+0000","last_became_peered":"2024-04-21T23:16:00.433737+0000","last_unstale":"2024-04-21T23:18:23.688857+0000","last_undegraded":"2024-04-21T23:18:23.688857+0000","last_fullsized":"2024-04-21T23:18:23.688857+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:38:00.295054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"90'1","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.662784+0000","last_change":"2024-04-21T23:15:55.396792+0000","last_active":"2024-04-21T23:18:23.662784+0000","last_peered":"2024-04-21T23:18:23.662784+0000","last_clean":"2024-04-21T23:18:23.662784+0000","last_became_active":"2024-04-21T23:15:55.395740+0000","last_became_peered":"2024-04-21T23:15:55.395740+0000","last_unstale":"2024-04-21T23:18:23.662784+0000","last_undegraded":"2024-04-21T23:18:23.662784+0000","last_fullsized":"2024-04-21T23:18:23.662784+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:39:22.366310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.399163+0000","last_change":"2024-04-21T23:15:56.346372+0000","last_active":"2024-04-21T23:18:23.399163+0000","last_peered":"2024-04-21T23:18:23.399163+0000","last_clean":"2024-04-21T23:18:23.399163+0000","last_became_active":"2024-04-21T23:15:56.346047+0000","last_became_peered":"2024-04-21T23:15:56.346047+0000","last_unstale":"2024-04-21T23:18:23.399163+0000","last_undegraded":"2024-04-21T23:18:23.399163+0000","last_fullsized":"2024-04-21T23:18:23.399163+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:32:29.684227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.689210+0000","last_change":"2024-04-21T23:16:00.434747+0000","last_active":"2024-04-21T23:18:23.689210+0000","last_peered":"2024-04-21T23:18:23.689210+0000","last_clean":"2024-04-21T23:18:23.689210+0000","last_became_active":"2024-04-21T23:16:00.433950+0000","last_became_peered":"2024-04-21T23:16:00.433950+0000","last_unstale":"2024-04-21T23:18:23.689210+0000","last_undegraded":"2024-04-21T23:18:23.689210+0000","last_fullsized":"2024-04-21T23:18:23.689210+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:02:10.562135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"92'8","reported_seq":46,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.507442+0000","last_change":"2024-04-21T23:16:52.206022+0000","last_active":"2024-04-21T23:18:23.507442+0000","last_peered":"2024-04-21T23:18:23.507442+0000","last_clean":"2024-04-21T23:18:23.507442+0000","last_became_active":"2024-04-21T23:16:52.205469+0000","last_became_peered":"2024-04-21T23:16:52.205469+0000","last_unstale":"2024-04-21T23:18:23.507442+0000","last_undegraded":"2024-04-21T23:18:23.507442+0000","last_fullsized":"2024-04-21T23:18:23.507442+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:34:46.264003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,8],"acting":[5,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"94'6","reported_seq":162,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.688568+0000","last_change":"2024-04-21T23:15:55.394900+0000","last_active":"2024-04-21T23:18:23.688568+0000","last_peered":"2024-04-21T23:18:23.688568+0000","last_clean":"2024-04-21T23:18:23.688568+0000","last_became_active":"2024-04-21T23:15:55.393556+0000","last_became_peered":"2024-04-21T23:15:55.393556+0000","last_unstale":"2024-04-21T23:18:23.688568+0000","last_undegraded":"2024-04-21T23:18:23.688568+0000","last_fullsized":"2024-04-21T23:18:23.688568+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:16:24.691930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":122,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":26,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.662759+0000","last_change":"2024-04-21T23:16:00.443338+0000","last_active":"2024-04-21T23:18:23.662759+0000","last_peered":"2024-04-21T23:18:23.662759+0000","last_clean":"2024-04-21T23:18:23.662759+0000","last_became_active":"2024-04-21T23:16:00.442855+0000","last_became_peered":"2024-04-21T23:16:00.442855+0000","last_unstale":"2024-04-21T23:18:23.662759+0000","last_undegraded":"2024-04-21T23:18:23.662759+0000","last_fullsized":"2024-04-21T23:18:23.662759+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:04:36.763027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.500267+0000","last_change":"2024-04-21T23:15:56.356788+0000","last_active":"2024-04-21T23:18:23.500267+0000","last_peered":"2024-04-21T23:18:23.500267+0000","last_clean":"2024-04-21T23:18:23.500267+0000","last_became_active":"2024-04-21T23:15:56.356474+0000","last_became_peered":"2024-04-21T23:15:56.356474+0000","last_unstale":"2024-04-21T23:18:23.500267+0000","last_undegraded":"2024-04-21T23:18:23.500267+0000","last_fullsized":"2024-04-21T23:18:23.500267+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:34:33.700897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"90'3","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.660276+0000","last_change":"2024-04-21T23:15:55.395398+0000","last_active":"2024-04-21T23:18:23.660276+0000","last_peered":"2024-04-21T23:18:23.660276+0000","last_clean":"2024-04-21T23:18:23.660276+0000","last_became_active":"2024-04-21T23:15:55.394978+0000","last_became_peered":"2024-04-21T23:15:55.394978+0000","last_unstale":"2024-04-21T23:18:23.660276+0000","last_undegraded":"2024-04-21T23:18:23.660276+0000","last_fullsized":"2024-04-21T23:18:23.660276+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:07:01.266885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":368,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.662639+0000","last_change":"2024-04-21T23:14:42.877959+0000","last_active":"2024-04-21T23:18:23.662639+0000","last_peered":"2024-04-21T23:18:23.662639+0000","last_clean":"2024-04-21T23:18:23.662639+0000","last_became_active":"2024-04-21T23:14:42.877384+0000","last_became_peered":"2024-04-21T23:14:42.877384+0000","last_unstale":"2024-04-21T23:18:23.662639+0000","last_undegraded":"2024-04-21T23:18:23.662639+0000","last_fullsized":"2024-04-21T23:18:23.662639+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:11:15.009349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:11:15.009349+0000","last_clean_scrub_stamp":"2024-04-21T23:11:15.009349+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:18:55.621052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.507306+0000","last_change":"2024-04-21T23:16:00.444576+0000","last_active":"2024-04-21T23:18:23.507306+0000","last_peered":"2024-04-21T23:18:23.507306+0000","last_clean":"2024-04-21T23:18:23.507306+0000","last_became_active":"2024-04-21T23:16:00.444153+0000","last_became_peered":"2024-04-21T23:16:00.444153+0000","last_unstale":"2024-04-21T23:18:23.507306+0000","last_undegraded":"2024-04-21T23:18:23.507306+0000","last_fullsized":"2024-04-21T23:18:23.507306+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:11:48.040383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"92'5","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.688448+0000","last_change":"2024-04-21T23:15:56.349870+0000","last_active":"2024-04-21T23:18:23.688448+0000","last_peered":"2024-04-21T23:18:23.688448+0000","last_clean":"2024-04-21T23:18:23.688448+0000","last_became_active":"2024-04-21T23:15:56.349535+0000","last_became_peered":"2024-04-21T23:15:56.349535+0000","last_unstale":"2024-04-21T23:18:23.688448+0000","last_undegraded":"2024-04-21T23:18:23.688448+0000","last_fullsized":"2024-04-21T23:18:23.688448+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:12:50.702556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"90'3","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.662748+0000","last_change":"2024-04-21T23:15:55.402406+0000","last_active":"2024-04-21T23:18:23.662748+0000","last_peered":"2024-04-21T23:18:23.662748+0000","last_clean":"2024-04-21T23:18:23.662748+0000","last_became_active":"2024-04-21T23:15:55.402075+0000","last_became_peered":"2024-04-21T23:15:55.402075+0000","last_unstale":"2024-04-21T23:18:23.662748+0000","last_undegraded":"2024-04-21T23:18:23.662748+0000","last_fullsized":"2024-04-21T23:18:23.662748+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:44:18.013031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.398753+0000","last_change":"2024-04-21T23:16:00.439206+0000","last_active":"2024-04-21T23:18:23.398753+0000","last_peered":"2024-04-21T23:18:23.398753+0000","last_clean":"2024-04-21T23:18:23.398753+0000","last_became_active":"2024-04-21T23:16:00.438601+0000","last_became_peered":"2024-04-21T23:16:00.438601+0000","last_unstale":"2024-04-21T23:18:23.398753+0000","last_undegraded":"2024-04-21T23:18:23.398753+0000","last_fullsized":"2024-04-21T23:18:23.398753+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:58:46.771456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"90'1","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.500582+0000","last_change":"2024-04-21T23:15:55.401154+0000","last_active":"2024-04-21T23:18:23.500582+0000","last_peered":"2024-04-21T23:18:23.500582+0000","last_clean":"2024-04-21T23:18:23.500582+0000","last_became_active":"2024-04-21T23:15:55.400579+0000","last_became_peered":"2024-04-21T23:15:55.400579+0000","last_unstale":"2024-04-21T23:18:23.500582+0000","last_undegraded":"2024-04-21T23:18:23.500582+0000","last_fullsized":"2024-04-21T23:18:23.500582+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:20:03.879660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.689222+0000","last_change":"2024-04-21T23:15:56.349837+0000","last_active":"2024-04-21T23:18:23.689222+0000","last_peered":"2024-04-21T23:18:23.689222+0000","last_clean":"2024-04-21T23:18:23.689222+0000","last_became_active":"2024-04-21T23:15:56.349466+0000","last_became_peered":"2024-04-21T23:15:56.349466+0000","last_unstale":"2024-04-21T23:18:23.689222+0000","last_undegraded":"2024-04-21T23:18:23.689222+0000","last_fullsized":"2024-04-21T23:18:23.689222+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:15:32.675770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.500545+0000","last_change":"2024-04-21T23:16:00.432599+0000","last_active":"2024-04-21T23:18:23.500545+0000","last_peered":"2024-04-21T23:18:23.500545+0000","last_clean":"2024-04-21T23:18:23.500545+0000","last_became_active":"2024-04-21T23:16:00.432194+0000","last_became_peered":"2024-04-21T23:16:00.432194+0000","last_unstale":"2024-04-21T23:18:23.500545+0000","last_undegraded":"2024-04-21T23:18:23.500545+0000","last_fullsized":"2024-04-21T23:18:23.500545+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:10:21.354338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"95'44","reported_seq":101,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.615014+0000","last_change":"2024-04-21T23:15:55.400328+0000","last_active":"2024-04-21T23:18:23.615014+0000","last_peered":"2024-04-21T23:18:23.615014+0000","last_clean":"2024-04-21T23:18:23.615014+0000","last_became_active":"2024-04-21T23:15:55.399991+0000","last_became_peered":"2024-04-21T23:15:55.399991+0000","last_unstale":"2024-04-21T23:18:23.615014+0000","last_undegraded":"2024-04-21T23:18:23.615014+0000","last_fullsized":"2024-04-21T23:18:23.615014+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:58:54.480720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":78,"num_write_kb":836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"92'5","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.501203+0000","last_change":"2024-04-21T23:15:56.358698+0000","last_active":"2024-04-21T23:18:23.501203+0000","last_peered":"2024-04-21T23:18:23.501203+0000","last_clean":"2024-04-21T23:18:23.501203+0000","last_became_active":"2024-04-21T23:15:56.358364+0000","last_became_peered":"2024-04-21T23:15:56.358364+0000","last_unstale":"2024-04-21T23:18:23.501203+0000","last_undegraded":"2024-04-21T23:18:23.501203+0000","last_fullsized":"2024-04-21T23:18:23.501203+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:34:15.281142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.395058+0000","last_change":"2024-04-21T23:16:00.432237+0000","last_active":"2024-04-21T23:18:23.395058+0000","last_peered":"2024-04-21T23:18:23.395058+0000","last_clean":"2024-04-21T23:18:23.395058+0000","last_became_active":"2024-04-21T23:16:00.431745+0000","last_became_peered":"2024-04-21T23:16:00.431745+0000","last_unstale":"2024-04-21T23:18:23.395058+0000","last_undegraded":"2024-04-21T23:18:23.395058+0000","last_fullsized":"2024-04-21T23:18:23.395058+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:25:00.477154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"90'4","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.661315+0000","last_change":"2024-04-21T23:15:55.398667+0000","last_active":"2024-04-21T23:18:23.661315+0000","last_peered":"2024-04-21T23:18:23.661315+0000","last_clean":"2024-04-21T23:18:23.661315+0000","last_became_active":"2024-04-21T23:15:55.398290+0000","last_became_peered":"2024-04-21T23:15:55.398290+0000","last_unstale":"2024-04-21T23:18:23.661315+0000","last_undegraded":"2024-04-21T23:18:23.661315+0000","last_fullsized":"2024-04-21T23:18:23.661315+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:57:58.992615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.832119+0000","last_change":"2024-04-21T23:15:56.359518+0000","last_active":"2024-04-21T23:18:23.832119+0000","last_peered":"2024-04-21T23:18:23.832119+0000","last_clean":"2024-04-21T23:18:23.832119+0000","last_became_active":"2024-04-21T23:15:56.358970+0000","last_became_peered":"2024-04-21T23:15:56.358970+0000","last_unstale":"2024-04-21T23:18:23.832119+0000","last_undegraded":"2024-04-21T23:18:23.832119+0000","last_fullsized":"2024-04-21T23:18:23.832119+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:41:22.113877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.660034+0000","last_change":"2024-04-21T23:16:00.443380+0000","last_active":"2024-04-21T23:18:23.660034+0000","last_peered":"2024-04-21T23:18:23.660034+0000","last_clean":"2024-04-21T23:18:23.660034+0000","last_became_active":"2024-04-21T23:16:00.442896+0000","last_became_peered":"2024-04-21T23:16:00.442896+0000","last_unstale":"2024-04-21T23:18:23.660034+0000","last_undegraded":"2024-04-21T23:18:23.660034+0000","last_fullsized":"2024-04-21T23:18:23.660034+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:51:00.089766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"90'2","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.660516+0000","last_change":"2024-04-21T23:15:55.395287+0000","last_active":"2024-04-21T23:18:23.660516+0000","last_peered":"2024-04-21T23:18:23.660516+0000","last_clean":"2024-04-21T23:18:23.660516+0000","last_became_active":"2024-04-21T23:15:55.394908+0000","last_became_peered":"2024-04-21T23:15:55.394908+0000","last_unstale":"2024-04-21T23:18:23.660516+0000","last_undegraded":"2024-04-21T23:18:23.660516+0000","last_fullsized":"2024-04-21T23:18:23.660516+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:57:29.915581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"92'2","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.500115+0000","last_change":"2024-04-21T23:15:56.348016+0000","last_active":"2024-04-21T23:18:23.500115+0000","last_peered":"2024-04-21T23:18:23.500115+0000","last_clean":"2024-04-21T23:18:23.500115+0000","last_became_active":"2024-04-21T23:15:56.347630+0000","last_became_peered":"2024-04-21T23:15:56.347630+0000","last_unstale":"2024-04-21T23:18:23.500115+0000","last_undegraded":"2024-04-21T23:18:23.500115+0000","last_fullsized":"2024-04-21T23:18:23.500115+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:17:27.372880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.500678+0000","last_change":"2024-04-21T23:17:52.925060+0000","last_active":"2024-04-21T23:18:23.500678+0000","last_peered":"2024-04-21T23:18:23.500678+0000","last_clean":"2024-04-21T23:18:23.500678+0000","last_became_active":"2024-04-21T23:17:52.924560+0000","last_became_peered":"2024-04-21T23:17:52.924560+0000","last_unstale":"2024-04-21T23:18:23.500678+0000","last_undegraded":"2024-04-21T23:18:23.500678+0000","last_fullsized":"2024-04-21T23:18:23.500678+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:24:19.265334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,5],"acting":[8,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"90'3","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.500520+0000","last_change":"2024-04-21T23:15:55.400859+0000","last_active":"2024-04-21T23:18:23.500520+0000","last_peered":"2024-04-21T23:18:23.500520+0000","last_clean":"2024-04-21T23:18:23.500520+0000","last_became_active":"2024-04-21T23:15:55.400460+0000","last_became_peered":"2024-04-21T23:15:55.400460+0000","last_unstale":"2024-04-21T23:18:23.500520+0000","last_undegraded":"2024-04-21T23:18:23.500520+0000","last_fullsized":"2024-04-21T23:18:23.500520+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:31:14.113476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.398661+0000","last_change":"2024-04-21T23:15:56.350354+0000","last_active":"2024-04-21T23:18:23.398661+0000","last_peered":"2024-04-21T23:18:23.398661+0000","last_clean":"2024-04-21T23:18:23.398661+0000","last_became_active":"2024-04-21T23:15:56.349954+0000","last_became_peered":"2024-04-21T23:15:56.349954+0000","last_unstale":"2024-04-21T23:18:23.398661+0000","last_undegraded":"2024-04-21T23:18:23.398661+0000","last_fullsized":"2024-04-21T23:18:23.398661+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:42:59.165610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":26,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.660123+0000","last_change":"2024-04-21T23:16:00.444829+0000","last_active":"2024-04-21T23:18:23.660123+0000","last_peered":"2024-04-21T23:18:23.660123+0000","last_clean":"2024-04-21T23:18:23.660123+0000","last_became_active":"2024-04-21T23:16:00.444365+0000","last_became_peered":"2024-04-21T23:16:00.444365+0000","last_unstale":"2024-04-21T23:18:23.660123+0000","last_undegraded":"2024-04-21T23:18:23.660123+0000","last_fullsized":"2024-04-21T23:18:23.660123+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:23:16.747684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"90'1","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.395250+0000","last_change":"2024-04-21T23:15:55.397855+0000","last_active":"2024-04-21T23:18:23.395250+0000","last_peered":"2024-04-21T23:18:23.395250+0000","last_clean":"2024-04-21T23:18:23.395250+0000","last_became_active":"2024-04-21T23:15:55.397498+0000","last_became_peered":"2024-04-21T23:15:55.397498+0000","last_unstale":"2024-04-21T23:18:23.395250+0000","last_undegraded":"2024-04-21T23:18:23.395250+0000","last_fullsized":"2024-04-21T23:18:23.395250+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:45:44.680350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.500943+0000","last_change":"2024-04-21T23:15:56.353023+0000","last_active":"2024-04-21T23:18:23.500943+0000","last_peered":"2024-04-21T23:18:23.500943+0000","last_clean":"2024-04-21T23:18:23.500943+0000","last_became_active":"2024-04-21T23:15:56.352209+0000","last_became_peered":"2024-04-21T23:15:56.352209+0000","last_unstale":"2024-04-21T23:18:23.500943+0000","last_undegraded":"2024-04-21T23:18:23.500943+0000","last_fullsized":"2024-04-21T23:18:23.500943+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:20:58.075627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.660994+0000","last_change":"2024-04-21T23:16:00.429258+0000","last_active":"2024-04-21T23:18:23.660994+0000","last_peered":"2024-04-21T23:18:23.660994+0000","last_clean":"2024-04-21T23:18:23.660994+0000","last_became_active":"2024-04-21T23:16:00.428856+0000","last_became_peered":"2024-04-21T23:16:00.428856+0000","last_unstale":"2024-04-21T23:18:23.660994+0000","last_undegraded":"2024-04-21T23:18:23.660994+0000","last_fullsized":"2024-04-21T23:18:23.660994+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:07:44.608370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"90'4","reported_seq":23,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.398659+0000","last_change":"2024-04-21T23:16:52.209766+0000","last_active":"2024-04-21T23:18:23.398659+0000","last_peered":"2024-04-21T23:18:23.398659+0000","last_clean":"2024-04-21T23:18:23.398659+0000","last_became_active":"2024-04-21T23:16:52.204654+0000","last_became_peered":"2024-04-21T23:16:52.204654+0000","last_unstale":"2024-04-21T23:18:23.398659+0000","last_undegraded":"2024-04-21T23:18:23.398659+0000","last_fullsized":"2024-04-21T23:18:23.398659+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:40:41.230460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"92'5","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.662982+0000","last_change":"2024-04-21T23:15:56.352024+0000","last_active":"2024-04-21T23:18:23.662982+0000","last_peered":"2024-04-21T23:18:23.662982+0000","last_clean":"2024-04-21T23:18:23.662982+0000","last_became_active":"2024-04-21T23:15:56.351052+0000","last_became_peered":"2024-04-21T23:15:56.351052+0000","last_unstale":"2024-04-21T23:18:23.662982+0000","last_undegraded":"2024-04-21T23:18:23.662982+0000","last_fullsized":"2024-04-21T23:18:23.662982+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:36:42.777601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.660850+0000","last_change":"2024-04-21T23:16:00.432371+0000","last_active":"2024-04-21T23:18:23.660850+0000","last_peered":"2024-04-21T23:18:23.660850+0000","last_clean":"2024-04-21T23:18:23.660850+0000","last_became_active":"2024-04-21T23:16:00.431828+0000","last_became_peered":"2024-04-21T23:16:00.431828+0000","last_unstale":"2024-04-21T23:18:23.660850+0000","last_undegraded":"2024-04-21T23:18:23.660850+0000","last_fullsized":"2024-04-21T23:18:23.660850+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:02:06.500075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"90'3","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.395651+0000","last_change":"2024-04-21T23:15:55.407018+0000","last_active":"2024-04-21T23:18:23.395651+0000","last_peered":"2024-04-21T23:18:23.395651+0000","last_clean":"2024-04-21T23:18:23.395651+0000","last_became_active":"2024-04-21T23:15:55.406709+0000","last_became_peered":"2024-04-21T23:15:55.406709+0000","last_unstale":"2024-04-21T23:18:23.395651+0000","last_undegraded":"2024-04-21T23:18:23.395651+0000","last_fullsized":"2024-04-21T23:18:23.395651+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:10:40.138470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.663535+0000","last_change":"2024-04-21T23:15:56.358569+0000","last_active":"2024-04-21T23:18:23.663535+0000","last_peered":"2024-04-21T23:18:23.663535+0000","last_clean":"2024-04-21T23:18:23.663535+0000","last_became_active":"2024-04-21T23:15:56.357533+0000","last_became_peered":"2024-04-21T23:15:56.357533+0000","last_unstale":"2024-04-21T23:18:23.663535+0000","last_undegraded":"2024-04-21T23:18:23.663535+0000","last_fullsized":"2024-04-21T23:18:23.663535+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:59:58.525340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.832026+0000","last_change":"2024-04-21T23:16:00.443981+0000","last_active":"2024-04-21T23:18:23.832026+0000","last_peered":"2024-04-21T23:18:23.832026+0000","last_clean":"2024-04-21T23:18:23.832026+0000","last_became_active":"2024-04-21T23:16:00.443368+0000","last_became_peered":"2024-04-21T23:16:00.443368+0000","last_unstale":"2024-04-21T23:18:23.832026+0000","last_undegraded":"2024-04-21T23:18:23.832026+0000","last_fullsized":"2024-04-21T23:18:23.832026+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:56:48.776540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"95'39","reported_seq":81,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.612653+0000","last_change":"2024-04-21T23:15:55.397334+0000","last_active":"2024-04-21T23:18:23.612653+0000","last_peered":"2024-04-21T23:18:23.612653+0000","last_clean":"2024-04-21T23:18:23.612653+0000","last_became_active":"2024-04-21T23:15:55.396745+0000","last_became_peered":"2024-04-21T23:15:55.396745+0000","last_unstale":"2024-04-21T23:18:23.612653+0000","last_undegraded":"2024-04-21T23:18:23.612653+0000","last_fullsized":"2024-04-21T23:18:23.612653+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:56:21.042187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.663027+0000","last_change":"2024-04-21T23:15:56.351012+0000","last_active":"2024-04-21T23:18:23.663027+0000","last_peered":"2024-04-21T23:18:23.663027+0000","last_clean":"2024-04-21T23:18:23.663027+0000","last_became_active":"2024-04-21T23:15:56.350042+0000","last_became_peered":"2024-04-21T23:15:56.350042+0000","last_unstale":"2024-04-21T23:18:23.663027+0000","last_undegraded":"2024-04-21T23:18:23.663027+0000","last_fullsized":"2024-04-21T23:18:23.663027+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:47:02.629870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.507013+0000","last_change":"2024-04-21T23:16:00.428775+0000","last_active":"2024-04-21T23:18:23.507013+0000","last_peered":"2024-04-21T23:18:23.507013+0000","last_clean":"2024-04-21T23:18:23.507013+0000","last_became_active":"2024-04-21T23:16:00.428278+0000","last_became_peered":"2024-04-21T23:16:00.428278+0000","last_unstale":"2024-04-21T23:18:23.507013+0000","last_undegraded":"2024-04-21T23:18:23.507013+0000","last_fullsized":"2024-04-21T23:18:23.507013+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:59.394266+0000","last_clean_scrub_stamp":"2024-04-21T23:15:59.394266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:24:04.397387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.689442+0000","last_change":"2024-04-21T23:15:55.395524+0000","last_active":"2024-04-21T23:18:23.689442+0000","last_peered":"2024-04-21T23:18:23.689442+0000","last_clean":"2024-04-21T23:18:23.689442+0000","last_became_active":"2024-04-21T23:15:55.395220+0000","last_became_peered":"2024-04-21T23:15:55.395220+0000","last_unstale":"2024-04-21T23:18:23.689442+0000","last_undegraded":"2024-04-21T23:18:23.689442+0000","last_fullsized":"2024-04-21T23:18:23.689442+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:54.372717+0000","last_clean_scrub_stamp":"2024-04-21T23:15:54.372717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:01:37.517620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.663388+0000","last_change":"2024-04-21T23:15:56.352128+0000","last_active":"2024-04-21T23:18:23.663388+0000","last_peered":"2024-04-21T23:18:23.663388+0000","last_clean":"2024-04-21T23:18:23.663388+0000","last_became_active":"2024-04-21T23:15:56.349349+0000","last_became_peered":"2024-04-21T23:15:56.349349+0000","last_unstale":"2024-04-21T23:18:23.663388+0000","last_undegraded":"2024-04-21T23:18:23.663388+0000","last_fullsized":"2024-04-21T23:18:23.663388+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:12:23.247084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.398793+0000","last_change":"2024-04-21T23:15:56.350830+0000","last_active":"2024-04-21T23:18:23.398793+0000","last_peered":"2024-04-21T23:18:23.398793+0000","last_clean":"2024-04-21T23:18:23.398793+0000","last_became_active":"2024-04-21T23:15:56.350385+0000","last_became_peered":"2024-04-21T23:15:56.350385+0000","last_unstale":"2024-04-21T23:18:23.398793+0000","last_undegraded":"2024-04-21T23:18:23.398793+0000","last_fullsized":"2024-04-21T23:18:23.398793+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:18:48.242375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"92'12","reported_seq":44,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.395348+0000","last_change":"2024-04-21T23:15:56.350947+0000","last_active":"2024-04-21T23:18:23.395348+0000","last_peered":"2024-04-21T23:18:23.395348+0000","last_clean":"2024-04-21T23:18:23.395348+0000","last_became_active":"2024-04-21T23:15:56.350383+0000","last_became_peered":"2024-04-21T23:15:56.350383+0000","last_unstale":"2024-04-21T23:18:23.395348+0000","last_undegraded":"2024-04-21T23:18:23.395348+0000","last_fullsized":"2024-04-21T23:18:23.395348+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:07:28.879822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.395154+0000","last_change":"2024-04-21T23:15:56.354559+0000","last_active":"2024-04-21T23:18:23.395154+0000","last_peered":"2024-04-21T23:18:23.395154+0000","last_clean":"2024-04-21T23:18:23.395154+0000","last_became_active":"2024-04-21T23:15:56.354205+0000","last_became_peered":"2024-04-21T23:15:56.354205+0000","last_unstale":"2024-04-21T23:18:23.395154+0000","last_undegraded":"2024-04-21T23:18:23.395154+0000","last_fullsized":"2024-04-21T23:18:23.395154+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:33:50.157971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.832253+0000","last_change":"2024-04-21T23:15:56.358395+0000","last_active":"2024-04-21T23:18:23.832253+0000","last_peered":"2024-04-21T23:18:23.832253+0000","last_clean":"2024-04-21T23:18:23.832253+0000","last_became_active":"2024-04-21T23:15:56.358077+0000","last_became_peered":"2024-04-21T23:15:56.358077+0000","last_unstale":"2024-04-21T23:18:23.832253+0000","last_undegraded":"2024-04-21T23:18:23.832253+0000","last_fullsized":"2024-04-21T23:18:23.832253+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:12:43.124588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"92'4","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.689166+0000","last_change":"2024-04-21T23:15:56.347476+0000","last_active":"2024-04-21T23:18:23.689166+0000","last_peered":"2024-04-21T23:18:23.689166+0000","last_clean":"2024-04-21T23:18:23.689166+0000","last_became_active":"2024-04-21T23:15:56.347097+0000","last_became_peered":"2024-04-21T23:15:56.347097+0000","last_unstale":"2024-04-21T23:18:23.689166+0000","last_undegraded":"2024-04-21T23:18:23.689166+0000","last_fullsized":"2024-04-21T23:18:23.689166+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:11:58.715526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.399175+0000","last_change":"2024-04-21T23:15:56.346413+0000","last_active":"2024-04-21T23:18:23.399175+0000","last_peered":"2024-04-21T23:18:23.399175+0000","last_clean":"2024-04-21T23:18:23.399175+0000","last_became_active":"2024-04-21T23:15:56.346056+0000","last_became_peered":"2024-04-21T23:15:56.346056+0000","last_unstale":"2024-04-21T23:18:23.399175+0000","last_undegraded":"2024-04-21T23:18:23.399175+0000","last_fullsized":"2024-04-21T23:18:23.399175+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:47:16.340323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"92'15","reported_seq":47,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.395818+0000","last_change":"2024-04-21T23:15:56.354876+0000","last_active":"2024-04-21T23:18:23.395818+0000","last_peered":"2024-04-21T23:18:23.395818+0000","last_clean":"2024-04-21T23:18:23.395818+0000","last_became_active":"2024-04-21T23:15:56.354525+0000","last_became_peered":"2024-04-21T23:15:56.354525+0000","last_unstale":"2024-04-21T23:18:23.395818+0000","last_undegraded":"2024-04-21T23:18:23.395818+0000","last_fullsized":"2024-04-21T23:18:23.395818+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:09:23.662664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"92'2","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.501341+0000","last_change":"2024-04-21T23:15:56.358456+0000","last_active":"2024-04-21T23:18:23.501341+0000","last_peered":"2024-04-21T23:18:23.501341+0000","last_clean":"2024-04-21T23:18:23.501341+0000","last_became_active":"2024-04-21T23:15:56.358060+0000","last_became_peered":"2024-04-21T23:15:56.358060+0000","last_unstale":"2024-04-21T23:18:23.501341+0000","last_undegraded":"2024-04-21T23:18:23.501341+0000","last_fullsized":"2024-04-21T23:18:23.501341+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:47:58.928438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.507107+0000","last_change":"2024-04-21T23:15:56.352870+0000","last_active":"2024-04-21T23:18:23.507107+0000","last_peered":"2024-04-21T23:18:23.507107+0000","last_clean":"2024-04-21T23:18:23.507107+0000","last_became_active":"2024-04-21T23:15:56.352267+0000","last_became_peered":"2024-04-21T23:15:56.352267+0000","last_unstale":"2024-04-21T23:18:23.507107+0000","last_undegraded":"2024-04-21T23:18:23.507107+0000","last_fullsized":"2024-04-21T23:18:23.507107+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:32:22.244773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.500521+0000","last_change":"2024-04-21T23:15:56.357468+0000","last_active":"2024-04-21T23:18:23.500521+0000","last_peered":"2024-04-21T23:18:23.500521+0000","last_clean":"2024-04-21T23:18:23.500521+0000","last_became_active":"2024-04-21T23:15:56.357104+0000","last_became_peered":"2024-04-21T23:15:56.357104+0000","last_unstale":"2024-04-21T23:18:23.500521+0000","last_undegraded":"2024-04-21T23:18:23.500521+0000","last_fullsized":"2024-04-21T23:18:23.500521+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:37:36.884036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"92'10","reported_seq":48,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.661120+0000","last_change":"2024-04-21T23:16:52.203142+0000","last_active":"2024-04-21T23:18:23.661120+0000","last_peered":"2024-04-21T23:18:23.661120+0000","last_clean":"2024-04-21T23:18:23.661120+0000","last_became_active":"2024-04-21T23:16:52.202558+0000","last_became_peered":"2024-04-21T23:16:52.202558+0000","last_unstale":"2024-04-21T23:18:23.661120+0000","last_undegraded":"2024-04-21T23:18:23.661120+0000","last_fullsized":"2024-04-21T23:18:23.661120+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:29:05.428123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,8],"acting":[1,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"92'11","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.688864+0000","last_change":"2024-04-21T23:15:56.351901+0000","last_active":"2024-04-21T23:18:23.688864+0000","last_peered":"2024-04-21T23:18:23.688864+0000","last_clean":"2024-04-21T23:18:23.688864+0000","last_became_active":"2024-04-21T23:15:56.351545+0000","last_became_peered":"2024-04-21T23:15:56.351545+0000","last_unstale":"2024-04-21T23:18:23.688864+0000","last_undegraded":"2024-04-21T23:18:23.688864+0000","last_fullsized":"2024-04-21T23:18:23.688864+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:37:20.855631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:18:23.398840+0000","last_change":"2024-04-21T23:15:56.355773+0000","last_active":"2024-04-21T23:18:23.398840+0000","last_peered":"2024-04-21T23:18:23.398840+0000","last_clean":"2024-04-21T23:18:23.398840+0000","last_became_active":"2024-04-21T23:15:56.355269+0000","last_became_peered":"2024-04-21T23:15:56.355269+0000","last_unstale":"2024-04-21T23:18:23.398840+0000","last_undegraded":"2024-04-21T23:18:23.398840+0000","last_fullsized":"2024-04-21T23:18:23.398840+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:15:55.374858+0000","last_clean_scrub_stamp":"2024-04-21T23:15:55.374858+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:04:54.540817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":597867,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":854,"num_read_kb":536,"num_write":288,"num_write_kb":1440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":112,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2027520,"data_stored":1793601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":228,"ondisk_log_size":228,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351018,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1093632,"data_stored":536470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612656,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37268,"kb_used_data":1872,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93707884,"statfs":{"total":95995035648,"available":95956873216,"internally_reserved":0,"allocated":1916928,"data_stored":1324276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808885,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36804,"kb_used_data":1464,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708348,"statfs":{"total":95995035648,"available":95957348352,"internally_reserved":0,"allocated":1499136,"data_stored":933906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070522,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28472,"kb_used_data":1452,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716680,"statfs":{"total":95995035648,"available":95965880320,"internally_reserved":0,"allocated":1486848,"data_stored":929618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332159,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28676,"kb_used_data":1656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716476,"statfs":{"total":95995035648,"available":95965671424,"internally_reserved":0,"allocated":1695744,"data_stored":1134045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495684,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1081344,"data_stored":531770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574942,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28148,"kb_used_data":1128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717004,"statfs":{"total":95995035648,"available":95966212096,"internally_reserved":0,"allocated":1155072,"data_stored":599108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738466,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28100,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717052,"statfs":{"total":95995035648,"available":95966261248,"internally_reserved":0,"allocated":1105920,"data_stored":533158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411417,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28168,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716984,"statfs":{"total":95995035648,"available":95966191616,"internally_reserved":0,"allocated":1175552,"data_stored":595362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182483,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36920,"kb_used_data":1584,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708232,"statfs":{"total":95995035648,"available":95957229568,"internally_reserved":0,"allocated":1622016,"data_stored":1056780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920847,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1077248,"data_stored":527336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757321,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37132,"kb_used_data":1796,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708020,"statfs":{"total":95995035648,"available":95957012480,"internally_reserved":0,"allocated":1839104,"data_stored":1259611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":67428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":69169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":544768,"data_stored":525100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":133467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":393842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":394034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":202228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-21T23:18:29.231 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T23:18:29.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.369+0000 7feaa54c9640 1 -- 172.21.15.40:0/1047690445 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feaa01066b0 msgr2=0x7feaa0106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:29.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.369+0000 7feaa54c9640 1 --2- 172.21.15.40:0/1047690445 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feaa01066b0 0x7feaa0106a90 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fea90009930 tx=0x7fea9002bc30 comp rx=0 tx=0).stop 2024-04-21T23:18:29.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.369+0000 7feaa54c9640 1 -- 172.21.15.40:0/1047690445 shutdown_connections 2024-04-21T23:18:29.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.369+0000 7feaa54c9640 1 --2- 172.21.15.40:0/1047690445 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feaa0100610 0x7feaa0100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:29.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.369+0000 7feaa54c9640 1 --2- 172.21.15.40:0/1047690445 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feaa0100fd0 0x7feaa010d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:29.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.369+0000 7feaa54c9640 1 --2- 172.21.15.40:0/1047690445 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feaa01066b0 0x7feaa0106a90 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:29.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.369+0000 7feaa54c9640 1 -- 172.21.15.40:0/1047690445 >> 172.21.15.40:0/1047690445 conn(0x7feaa00769c0 msgr2=0x7feaa0076dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:29.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.369+0000 7feaa54c9640 1 -- 172.21.15.40:0/1047690445 shutdown_connections 2024-04-21T23:18:29.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.369+0000 7feaa54c9640 1 -- 172.21.15.40:0/1047690445 wait complete. 2024-04-21T23:18:29.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.369+0000 7feaa54c9640 1 Processor -- start 2024-04-21T23:18:29.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.370+0000 7feaa54c9640 1 -- start start 2024-04-21T23:18:29.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.370+0000 7feaa54c9640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feaa0100610 0x7feaa0198df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:29.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.370+0000 7feaa54c9640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feaa0100fd0 0x7feaa0199330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:29.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.370+0000 7feaa54c9640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feaa01066b0 0x7feaa01a3e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:29.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.370+0000 7feaa54c9640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7feaa0110a30 con 0x7feaa0100fd0 2024-04-21T23:18:29.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.370+0000 7feaa54c9640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7feaa01108b0 con 0x7feaa0100610 2024-04-21T23:18:29.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.370+0000 7feaa54c9640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7feaa0110bb0 con 0x7feaa01066b0 2024-04-21T23:18:29.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.370+0000 7fea97fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feaa0100fd0 0x7feaa0199330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:29.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.370+0000 7fea97fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feaa0100fd0 0x7feaa0199330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32852/0 (socket says 172.21.15.40:32852) 2024-04-21T23:18:29.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.370+0000 7fea97fff640 1 -- 172.21.15.40:0/1597023666 learned_addr learned my addr 172.21.15.40:0/1597023666 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:29.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.370+0000 7fea9effd640 1 --2- 172.21.15.40:0/1597023666 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feaa0100610 0x7feaa0198df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:29.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.370+0000 7fea97fff640 1 -- 172.21.15.40:0/1597023666 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feaa01066b0 msgr2=0x7feaa01a3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:29.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.370+0000 7fea97fff640 1 --2- 172.21.15.40:0/1597023666 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feaa01066b0 0x7feaa01a3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:29.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.370+0000 7fea97fff640 1 -- 172.21.15.40:0/1597023666 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feaa0100610 msgr2=0x7feaa0198df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:29.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.370+0000 7fea97fff640 1 --2- 172.21.15.40:0/1597023666 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feaa0100610 0x7feaa0198df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:29.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.370+0000 7fea97fff640 1 -- 172.21.15.40:0/1597023666 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feaa01a43b0 con 0x7feaa0100fd0 2024-04-21T23:18:29.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.370+0000 7fea97fff640 1 --2- 172.21.15.40:0/1597023666 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feaa0100fd0 0x7feaa0199330 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fea8800d950 tx=0x7fea8800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:29.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.370+0000 7fea9cff9640 1 -- 172.21.15.40:0/1597023666 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea88017070 con 0x7feaa0100fd0 2024-04-21T23:18:29.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.370+0000 7feaa54c9640 1 -- 172.21.15.40:0/1597023666 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feaa01a4580 con 0x7feaa0100fd0 2024-04-21T23:18:29.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.370+0000 7fea9cff9640 1 -- 172.21.15.40:0/1597023666 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea88009c30 con 0x7feaa0100fd0 2024-04-21T23:18:29.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.370+0000 7fea9cff9640 1 -- 172.21.15.40:0/1597023666 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea8800f6b0 con 0x7feaa0100fd0 2024-04-21T23:18:29.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.370+0000 7feaa54c9640 1 -- 172.21.15.40:0/1597023666 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feaa01a4ad0 con 0x7feaa0100fd0 2024-04-21T23:18:29.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.371+0000 7feaa54c9640 1 -- 172.21.15.40:0/1597023666 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feaa0101d50 con 0x7feaa0100fd0 2024-04-21T23:18:29.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.372+0000 7fea9cff9640 1 -- 172.21.15.40:0/1597023666 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7fea880109d0 con 0x7feaa0100fd0 2024-04-21T23:18:29.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.373+0000 7fea9cff9640 1 --2- 172.21.15.40:0/1597023666 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fea8c0a2e20 0x7fea8c0a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:29.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.373+0000 7fea9cff9640 1 -- 172.21.15.40:0/1597023666 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7fea880cf7d0 con 0x7feaa0100fd0 2024-04-21T23:18:29.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.373+0000 7fea9effd640 1 --2- 172.21.15.40:0/1597023666 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fea8c0a2e20 0x7fea8c0a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:29.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.374+0000 7fea9effd640 1 --2- 172.21.15.40:0/1597023666 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fea8c0a2e20 0x7fea8c0a52e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fea90009900 tx=0x7fea90005fd0 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:29.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.378+0000 7fea9cff9640 1 -- 172.21.15.40:0/1597023666 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fea88009de0 con 0x7feaa0100fd0 2024-04-21T23:18:29.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:28 smithi174 ceph-mon[31409]: Updating smithi040:/var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-21T23:18:29.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:28 smithi174 ceph-mon[31409]: Updating smithi111:/var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-21T23:18:29.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:28 smithi174 ceph-mon[31409]: Updating smithi174:/var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-21T23:18:29.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:28 smithi174 ceph-mon[31409]: Updating smithi040:/etc/ceph/ceph.client.admin.keyring 2024-04-21T23:18:29.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:28 smithi174 ceph-mon[31409]: Updating smithi111:/etc/ceph/ceph.client.admin.keyring 2024-04-21T23:18:29.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:28 smithi174 ceph-mon[31409]: Updating smithi174:/etc/ceph/ceph.client.admin.keyring 2024-04-21T23:18:29.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:28 smithi174 ceph-mon[31409]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:18:29.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:28 smithi174 ceph-mon[31409]: Updating smithi040:/var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T23:18:29.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:28 smithi174 ceph-mon[31409]: Updating smithi111:/var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T23:18:29.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:28 smithi174 ceph-mon[31409]: Updating smithi174:/var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T23:18:29.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:28 smithi174 ceph-mon[31409]: mgrmap e24: x(active, since 4s), standbys: y, z 2024-04-21T23:18:29.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:28 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:18:29.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:28 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:18:29.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:28 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:18:29.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:28 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:18:29.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:28 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/1320803449' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:18:29.446 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:28 smithi111 ceph-mon[31085]: Updating smithi040:/var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-21T23:18:29.446 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:28 smithi111 ceph-mon[31085]: Updating smithi111:/var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-21T23:18:29.446 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:28 smithi111 ceph-mon[31085]: Updating smithi174:/var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-21T23:18:29.446 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:28 smithi111 ceph-mon[31085]: Updating smithi040:/etc/ceph/ceph.client.admin.keyring 2024-04-21T23:18:29.446 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:28 smithi111 ceph-mon[31085]: Updating smithi111:/etc/ceph/ceph.client.admin.keyring 2024-04-21T23:18:29.446 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:28 smithi111 ceph-mon[31085]: Updating smithi174:/etc/ceph/ceph.client.admin.keyring 2024-04-21T23:18:29.446 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:28 smithi111 ceph-mon[31085]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:18:29.446 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:28 smithi111 ceph-mon[31085]: Updating smithi040:/var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T23:18:29.447 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:28 smithi111 ceph-mon[31085]: Updating smithi111:/var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T23:18:29.447 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:28 smithi111 ceph-mon[31085]: Updating smithi174:/var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T23:18:29.447 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:28 smithi111 ceph-mon[31085]: mgrmap e24: x(active, since 4s), standbys: y, z 2024-04-21T23:18:29.447 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:28 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:18:29.447 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:28 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:18:29.447 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:28 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:18:29.447 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:28 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:18:29.447 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:28 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/1320803449' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:18:29.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.567+0000 7feaa54c9640 1 -- 172.21.15.40:0/1597023666 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7feaa01085f0 con 0x7feaa0100fd0 2024-04-21T23:18:29.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.568+0000 7fea9cff9640 1 -- 172.21.15.40:0/1597023666 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) ==== 74+0+25231 (secure 0 0 0) 0x7fea8809bdf0 con 0x7feaa0100fd0 2024-04-21T23:18:29.568 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:18:29.568 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":95,"fsid":"d814e4dc-0033-11ef-bc93-c7b262605968","created":"2024-04-21T23:07:52.659207+0000","modified":"2024-04-21T23:18:23.389706+0000","last_up_change":"2024-04-21T23:15:05.308821+0000","last_in_change":"2024-04-21T23:14:45.447491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:11:14.824775+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T23:15:54.322571+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T23:15:55.164493+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T23:15:58.838266+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"0e8d692a-22fe-4a87-8d99-9151f96cbb02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6803","nonce":2009758784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6805","nonce":2009758784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6809","nonce":2009758784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6807","nonce":2009758784}]},"public_addr":"172.21.15.40:6803/2009758784","cluster_addr":"172.21.15.40:6805/2009758784","heartbeat_back_addr":"172.21.15.40:6809/2009758784","heartbeat_front_addr":"172.21.15.40:6807/2009758784","state":["exists","up"]},{"osd":1,"uuid":"fe5fe11b-0e84-4359-83a7-1ba45105755a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6801","nonce":1763903836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6803","nonce":1763903836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6807","nonce":1763903836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6805","nonce":1763903836}]},"public_addr":"172.21.15.111:6801/1763903836","cluster_addr":"172.21.15.111:6803/1763903836","heartbeat_back_addr":"172.21.15.111:6807/1763903836","heartbeat_front_addr":"172.21.15.111:6805/1763903836","state":["exists","up"]},{"osd":2,"uuid":"813ad735-5cc2-4da0-ae16-6db238395fbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6801","nonce":439695825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6803","nonce":439695825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6807","nonce":439695825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6805","nonce":439695825}]},"public_addr":"172.21.15.174:6801/439695825","cluster_addr":"172.21.15.174:6803/439695825","heartbeat_back_addr":"172.21.15.174:6807/439695825","heartbeat_front_addr":"172.21.15.174:6805/439695825","state":["exists","up"]},{"osd":3,"uuid":"09becb03-b7ce-4ffc-917d-b7900ec37e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6811","nonce":3454312436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6813","nonce":3454312436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6817","nonce":3454312436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6815","nonce":3454312436}]},"public_addr":"172.21.15.40:6811/3454312436","cluster_addr":"172.21.15.40:6813/3454312436","heartbeat_back_addr":"172.21.15.40:6817/3454312436","heartbeat_front_addr":"172.21.15.40:6815/3454312436","state":["exists","up"]},{"osd":4,"uuid":"46eabfd7-d88a-4efd-94eb-04354d3a9d7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6809","nonce":1810866070}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6811","nonce":1810866070}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6815","nonce":1810866070}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6813","nonce":1810866070}]},"public_addr":"172.21.15.111:6809/1810866070","cluster_addr":"172.21.15.111:6811/1810866070","heartbeat_back_addr":"172.21.15.111:6815/1810866070","heartbeat_front_addr":"172.21.15.111:6813/1810866070","state":["exists","up"]},{"osd":5,"uuid":"6f73b582-6c65-41db-aab0-9b360f83d5c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6809","nonce":3752986711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6811","nonce":3752986711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6815","nonce":3752986711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6813","nonce":3752986711}]},"public_addr":"172.21.15.174:6809/3752986711","cluster_addr":"172.21.15.174:6811/3752986711","heartbeat_back_addr":"172.21.15.174:6815/3752986711","heartbeat_front_addr":"172.21.15.174:6813/3752986711","state":["exists","up"]},{"osd":6,"uuid":"39f305f0-12d2-4732-aaa0-cd66700af538","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6819","nonce":1618855427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6821","nonce":1618855427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6825","nonce":1618855427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6823","nonce":1618855427}]},"public_addr":"172.21.15.40:6819/1618855427","cluster_addr":"172.21.15.40:6821/1618855427","heartbeat_back_addr":"172.21.15.40:6825/1618855427","heartbeat_front_addr":"172.21.15.40:6823/1618855427","state":["exists","up"]},{"osd":7,"uuid":"769f2468-5102-490b-89f8-d0444d3169a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6817","nonce":927054162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6819","nonce":927054162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6823","nonce":927054162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6821","nonce":927054162}]},"public_addr":"172.21.15.111:6817/927054162","cluster_addr":"172.21.15.111:6819/927054162","heartbeat_back_addr":"172.21.15.111:6823/927054162","heartbeat_front_addr":"172.21.15.111:6821/927054162","state":["exists","up"]},{"osd":8,"uuid":"1a444e80-007d-473b-b972-4c814a813c22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6817","nonce":3817848746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6819","nonce":3817848746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6823","nonce":3817848746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6821","nonce":3817848746}]},"public_addr":"172.21.15.174:6817/3817848746","cluster_addr":"172.21.15.174:6819/3817848746","heartbeat_back_addr":"172.21.15.174:6823/3817848746","heartbeat_front_addr":"172.21.15.174:6821/3817848746","state":["exists","up"]},{"osd":9,"uuid":"9a6b6a56-6ae5-49fe-9e5b-cb17d686e0dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6827","nonce":1649446531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6829","nonce":1649446531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6833","nonce":1649446531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6831","nonce":1649446531}]},"public_addr":"172.21.15.40:6827/1649446531","cluster_addr":"172.21.15.40:6829/1649446531","heartbeat_back_addr":"172.21.15.40:6833/1649446531","heartbeat_front_addr":"172.21.15.40:6831/1649446531","state":["exists","up"]},{"osd":10,"uuid":"a70038ba-e2ad-4170-85cc-ea0470a2edef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6825","nonce":1227538935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6827","nonce":1227538935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6831","nonce":1227538935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6829","nonce":1227538935}]},"public_addr":"172.21.15.111:6825/1227538935","cluster_addr":"172.21.15.111:6827/1227538935","heartbeat_back_addr":"172.21.15.111:6831/1227538935","heartbeat_front_addr":"172.21.15.111:6829/1227538935","state":["exists","up"]},{"osd":11,"uuid":"781532bb-7a5e-44e3-8eb7-b584b0054666","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6825","nonce":2045865553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6827","nonce":2045865553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6831","nonce":2045865553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6829","nonce":2045865553}]},"public_addr":"172.21.15.174:6825/2045865553","cluster_addr":"172.21.15.174:6827/2045865553","heartbeat_back_addr":"172.21.15.174:6831/2045865553","heartbeat_front_addr":"172.21.15.174:6829/2045865553","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:10:24.761290+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:10:48.565428+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:11:12.307818+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:11:37.951529+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:02.832523+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:27.568879+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:52.997707+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:13:18.937274+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:13:44.708657+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:14:10.833855+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:14:37.078272+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:15:03.878076+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":1}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.2a","mappings":[{"from":3,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"4.9","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":10}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:6801/2500760788":"2024-04-22T23:18:23.389674+0000","172.21.15.40:6800/2500760788":"2024-04-22T23:18:23.389674+0000","172.21.15.40:0/2742002992":"2024-04-22T23:18:23.389674+0000","172.21.15.40:0/2494672676":"2024-04-22T23:18:23.389674+0000","172.21.15.40:6801/526554319":"2024-04-22T23:08:50.305770+0000","172.21.15.40:6800/526554319":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/3178763471":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/4045911364":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/276300080":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/3689649814":"2024-04-22T23:08:23.743030+0000","172.21.15.40:0/2241116715":"2024-04-22T23:08:23.743030+0000","172.21.15.40:0/1406242364":"2024-04-22T23:18:23.389674+0000","172.21.15.40:0/1737227818":"2024-04-22T23:18:23.389674+0000","172.21.15.40:6801/1479040326":"2024-04-22T23:08:23.743030+0000","172.21.15.40:6800/1479040326":"2024-04-22T23:08:23.743030+0000","172.21.15.40:0/681298099":"2024-04-22T23:08:23.743030+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T23:18:29.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.571+0000 7feaa54c9640 1 -- 172.21.15.40:0/1597023666 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fea8c0a2e20 msgr2=0x7fea8c0a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:29.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.571+0000 7feaa54c9640 1 --2- 172.21.15.40:0/1597023666 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fea8c0a2e20 0x7fea8c0a52e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fea90009900 tx=0x7fea90005fd0 comp rx=0 tx=0).stop 2024-04-21T23:18:29.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.571+0000 7feaa54c9640 1 -- 172.21.15.40:0/1597023666 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feaa0100fd0 msgr2=0x7feaa0199330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:29.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.571+0000 7feaa54c9640 1 --2- 172.21.15.40:0/1597023666 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feaa0100fd0 0x7feaa0199330 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fea8800d950 tx=0x7fea8800de10 comp rx=0 tx=0).stop 2024-04-21T23:18:29.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.571+0000 7feaa54c9640 1 -- 172.21.15.40:0/1597023666 shutdown_connections 2024-04-21T23:18:29.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.571+0000 7feaa54c9640 1 --2- 172.21.15.40:0/1597023666 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fea8c0a2e20 0x7fea8c0a52e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:29.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.571+0000 7feaa54c9640 1 --2- 172.21.15.40:0/1597023666 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feaa01066b0 0x7feaa01a3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:29.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.571+0000 7feaa54c9640 1 --2- 172.21.15.40:0/1597023666 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feaa0100fd0 0x7feaa0199330 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:29.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.571+0000 7feaa54c9640 1 --2- 172.21.15.40:0/1597023666 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feaa0100610 0x7feaa0198df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:29.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.571+0000 7feaa54c9640 1 -- 172.21.15.40:0/1597023666 >> 172.21.15.40:0/1597023666 conn(0x7feaa00769c0 msgr2=0x7feaa00fef60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:29.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.571+0000 7feaa54c9640 1 -- 172.21.15.40:0/1597023666 shutdown_connections 2024-04-21T23:18:29.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.571+0000 7feaa54c9640 1 -- 172.21.15.40:0/1597023666 wait complete. 2024-04-21T23:18:29.581 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T23:11:14.824775+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T23:15:54.322571+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T23:15:55.164493+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T23:15:58.838266+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T23:18:29.582 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T23:18:29.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.687+0000 7f75ce45f640 1 -- 172.21.15.40:0/4269751689 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75c8101710 msgr2=0x7f75c8101b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:29.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.687+0000 7f75ce45f640 1 --2- 172.21.15.40:0/4269751689 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75c8101710 0x7f75c8101b90 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f75b8009930 tx=0x7f75b802bc30 comp rx=0 tx=0).stop 2024-04-21T23:18:29.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.688+0000 7f75ce45f640 1 -- 172.21.15.40:0/4269751689 shutdown_connections 2024-04-21T23:18:29.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.688+0000 7f75ce45f640 1 --2- 172.21.15.40:0/4269751689 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f75c81020d0 0x7f75c810e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:29.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.688+0000 7f75ce45f640 1 --2- 172.21.15.40:0/4269751689 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75c8101710 0x7f75c8101b90 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:29.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.688+0000 7f75ce45f640 1 --2- 172.21.15.40:0/4269751689 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75c8107720 0x7f75c8107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:29.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.688+0000 7f75ce45f640 1 -- 172.21.15.40:0/4269751689 >> 172.21.15.40:0/4269751689 conn(0x7f75c8078b90 msgr2=0x7f75c8078fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:29.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.688+0000 7f75ce45f640 1 -- 172.21.15.40:0/4269751689 shutdown_connections 2024-04-21T23:18:29.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.688+0000 7f75ce45f640 1 -- 172.21.15.40:0/4269751689 wait complete. 2024-04-21T23:18:29.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.688+0000 7f75ce45f640 1 Processor -- start 2024-04-21T23:18:29.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.688+0000 7f75ce45f640 1 -- start start 2024-04-21T23:18:29.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.688+0000 7f75ce45f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f75c8101710 0x7f75c81980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:29.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.688+0000 7f75ce45f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75c81020d0 0x7f75c8198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:29.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.689+0000 7f75ce45f640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75c8107720 0x7f75c819c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:29.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.689+0000 7f75ce45f640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f75c8111b30 con 0x7f75c81020d0 2024-04-21T23:18:29.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.689+0000 7f75ce45f640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f75c81119b0 con 0x7f75c8107720 2024-04-21T23:18:29.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.689+0000 7f75ce45f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f75c8111cb0 con 0x7f75c8101710 2024-04-21T23:18:29.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.689+0000 7f75c77fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75c81020d0 0x7f75c8198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:29.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.689+0000 7f75c77fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75c81020d0 0x7f75c8198630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32860/0 (socket says 172.21.15.40:32860) 2024-04-21T23:18:29.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.689+0000 7f75c77fe640 1 -- 172.21.15.40:0/1272101549 learned_addr learned my addr 172.21.15.40:0/1272101549 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:29.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.689+0000 7f75cc9fc640 1 --2- 172.21.15.40:0/1272101549 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75c8107720 0x7f75c819c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:29.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.689+0000 7f75c7fff640 1 --2- 172.21.15.40:0/1272101549 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f75c8101710 0x7f75c81980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:29.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.689+0000 7f75c77fe640 1 -- 172.21.15.40:0/1272101549 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f75c8101710 msgr2=0x7f75c81980f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:29.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.689+0000 7f75c77fe640 1 --2- 172.21.15.40:0/1272101549 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f75c8101710 0x7f75c81980f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:29.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.689+0000 7f75c77fe640 1 -- 172.21.15.40:0/1272101549 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75c8107720 msgr2=0x7f75c819c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:29.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.689+0000 7f75c77fe640 1 --2- 172.21.15.40:0/1272101549 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75c8107720 0x7f75c819c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:29.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.689+0000 7f75c77fe640 1 -- 172.21.15.40:0/1272101549 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75c819d0e0 con 0x7f75c81020d0 2024-04-21T23:18:29.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.689+0000 7f75cc9fc640 1 --2- 172.21.15.40:0/1272101549 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75c8107720 0x7f75c819c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:18:29.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.689+0000 7f75c77fe640 1 --2- 172.21.15.40:0/1272101549 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75c81020d0 0x7f75c8198630 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f75b8009230 tx=0x7f75b802cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:29.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.689+0000 7f75c57fa640 1 -- 172.21.15.40:0/1272101549 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75b8041070 con 0x7f75c81020d0 2024-04-21T23:18:29.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.689+0000 7f75ce45f640 1 -- 172.21.15.40:0/1272101549 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75c819d370 con 0x7f75c81020d0 2024-04-21T23:18:29.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.689+0000 7f75c57fa640 1 -- 172.21.15.40:0/1272101549 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75b8004450 con 0x7f75c81020d0 2024-04-21T23:18:29.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.689+0000 7f75c57fa640 1 -- 172.21.15.40:0/1272101549 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75b8034440 con 0x7f75c81020d0 2024-04-21T23:18:29.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.689+0000 7f75ce45f640 1 -- 172.21.15.40:0/1272101549 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75c81a4c60 con 0x7f75c81020d0 2024-04-21T23:18:29.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.690+0000 7f75ce45f640 1 -- 172.21.15.40:0/1272101549 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75c80689b0 con 0x7f75c81020d0 2024-04-21T23:18:29.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.692+0000 7f75c57fa640 1 -- 172.21.15.40:0/1272101549 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7f75b8044070 con 0x7f75c81020d0 2024-04-21T23:18:29.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.692+0000 7f75c57fa640 1 --2- 172.21.15.40:0/1272101549 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f75b00a2e20 0x7f75b00a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:29.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.692+0000 7f75c57fa640 1 -- 172.21.15.40:0/1272101549 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f75b80f1870 con 0x7f75c81020d0 2024-04-21T23:18:29.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.692+0000 7f75c7fff640 1 --2- 172.21.15.40:0/1272101549 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f75b00a2e20 0x7f75b00a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:29.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.693+0000 7f75c7fff640 1 --2- 172.21.15.40:0/1272101549 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f75b00a2e20 0x7f75b00a52e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f75bc005fd0 tx=0x7f75bc005850 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:29.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.696+0000 7f75c57fa640 1 -- 172.21.15.40:0/1272101549 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f75b80bdfa0 con 0x7f75c81020d0 2024-04-21T23:18:29.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.881+0000 7f75ce45f640 1 -- 172.21.15.40:0/1272101549 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f75c8109cd0 con 0x7f75c81020d0 2024-04-21T23:18:29.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.882+0000 7f75c57fa640 1 -- 172.21.15.40:0/1272101549 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v95) ==== 93+0+10 (secure 0 0 0) 0x7f75b80bd8a0 con 0x7f75c81020d0 2024-04-21T23:18:29.881 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-04-21T23:18:29.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.883+0000 7f75ce45f640 1 -- 172.21.15.40:0/1272101549 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f75b00a2e20 msgr2=0x7f75b00a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:29.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.883+0000 7f75ce45f640 1 --2- 172.21.15.40:0/1272101549 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f75b00a2e20 0x7f75b00a52e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f75bc005fd0 tx=0x7f75bc005850 comp rx=0 tx=0).stop 2024-04-21T23:18:29.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.883+0000 7f75ce45f640 1 -- 172.21.15.40:0/1272101549 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75c81020d0 msgr2=0x7f75c8198630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:29.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.883+0000 7f75ce45f640 1 --2- 172.21.15.40:0/1272101549 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75c81020d0 0x7f75c8198630 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f75b8009230 tx=0x7f75b802cc70 comp rx=0 tx=0).stop 2024-04-21T23:18:29.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.883+0000 7f75ce45f640 1 -- 172.21.15.40:0/1272101549 shutdown_connections 2024-04-21T23:18:29.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.883+0000 7f75ce45f640 1 --2- 172.21.15.40:0/1272101549 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f75b00a2e20 0x7f75b00a52e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:29.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.883+0000 7f75ce45f640 1 --2- 172.21.15.40:0/1272101549 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75c8107720 0x7f75c819c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:29.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.884+0000 7f75ce45f640 1 --2- 172.21.15.40:0/1272101549 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75c81020d0 0x7f75c8198630 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:29.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.884+0000 7f75ce45f640 1 --2- 172.21.15.40:0/1272101549 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f75c8101710 0x7f75c81980f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:29.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.884+0000 7f75ce45f640 1 -- 172.21.15.40:0/1272101549 >> 172.21.15.40:0/1272101549 conn(0x7f75c8078b90 msgr2=0x7f75c80ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:29.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.884+0000 7f75ce45f640 1 -- 172.21.15.40:0/1272101549 shutdown_connections 2024-04-21T23:18:29.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:29.884+0000 7f75ce45f640 1 -- 172.21.15.40:0/1272101549 wait complete. 2024-04-21T23:18:29.894 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T23:18:30.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.001+0000 7f5896040640 1 -- 172.21.15.40:0/3667344866 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58901020d0 msgr2=0x7f589010e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:30.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.001+0000 7f5896040640 1 --2- 172.21.15.40:0/3667344866 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58901020d0 0x7f589010e5e0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f5888009cd0 tx=0x7f588802c0f0 comp rx=0 tx=0).stop 2024-04-21T23:18:30.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.001+0000 7f5896040640 1 -- 172.21.15.40:0/3667344866 shutdown_connections 2024-04-21T23:18:30.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.001+0000 7f5896040640 1 --2- 172.21.15.40:0/3667344866 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5890101710 0x7f5890101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:30.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.001+0000 7f5896040640 1 --2- 172.21.15.40:0/3667344866 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58901020d0 0x7f589010e5e0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:30.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.001+0000 7f5896040640 1 --2- 172.21.15.40:0/3667344866 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5890107720 0x7f5890107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:30.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.001+0000 7f5896040640 1 -- 172.21.15.40:0/3667344866 >> 172.21.15.40:0/3667344866 conn(0x7f58900fd180 msgr2=0x7f58900ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:30.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.001+0000 7f5896040640 1 -- 172.21.15.40:0/3667344866 shutdown_connections 2024-04-21T23:18:30.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.001+0000 7f5896040640 1 -- 172.21.15.40:0/3667344866 wait complete. 2024-04-21T23:18:30.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.001+0000 7f5896040640 1 Processor -- start 2024-04-21T23:18:30.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.001+0000 7f5896040640 1 -- start start 2024-04-21T23:18:30.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.002+0000 7f5896040640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5890101710 0x7f5890073a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:30.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.002+0000 7f5896040640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58901020d0 0x7f5890073f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:30.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.002+0000 7f5896040640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5890107720 0x7f58900744c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:30.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.002+0000 7f5896040640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5890111b30 con 0x7f58901020d0 2024-04-21T23:18:30.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.002+0000 7f5896040640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f58901119b0 con 0x7f5890107720 2024-04-21T23:18:30.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.002+0000 7f5896040640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5890111cb0 con 0x7f5890101710 2024-04-21T23:18:30.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.002+0000 7f588effd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58901020d0 0x7f5890073f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:30.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.002+0000 7f588effd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58901020d0 0x7f5890073f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32878/0 (socket says 172.21.15.40:32878) 2024-04-21T23:18:30.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.002+0000 7f588effd640 1 -- 172.21.15.40:0/2787906670 learned_addr learned my addr 172.21.15.40:0/2787906670 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:30.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.002+0000 7f588f7fe640 1 --2- 172.21.15.40:0/2787906670 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5890101710 0x7f5890073a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:30.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.002+0000 7f588ffff640 1 --2- 172.21.15.40:0/2787906670 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5890107720 0x7f58900744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:30.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.002+0000 7f588effd640 1 -- 172.21.15.40:0/2787906670 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5890101710 msgr2=0x7f5890073a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:30.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.002+0000 7f588effd640 1 --2- 172.21.15.40:0/2787906670 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5890101710 0x7f5890073a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:30.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.002+0000 7f588effd640 1 -- 172.21.15.40:0/2787906670 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5890107720 msgr2=0x7f58900744c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:30.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.002+0000 7f588effd640 1 --2- 172.21.15.40:0/2787906670 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5890107720 0x7f58900744c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:30.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.002+0000 7f588effd640 1 -- 172.21.15.40:0/2787906670 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58901a6510 con 0x7f58901020d0 2024-04-21T23:18:30.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.002+0000 7f588effd640 1 --2- 172.21.15.40:0/2787906670 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58901020d0 0x7f5890073f80 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f588400e9f0 tx=0x7f588400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:30.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.002+0000 7f588cff9640 1 -- 172.21.15.40:0/2787906670 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f588400cd80 con 0x7f58901020d0 2024-04-21T23:18:30.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.002+0000 7f5896040640 1 -- 172.21.15.40:0/2787906670 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58901a6800 con 0x7f58901020d0 2024-04-21T23:18:30.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.002+0000 7f588cff9640 1 -- 172.21.15.40:0/2787906670 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5884010910 con 0x7f58901020d0 2024-04-21T23:18:30.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.002+0000 7f588cff9640 1 -- 172.21.15.40:0/2787906670 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58840188f0 con 0x7f58901020d0 2024-04-21T23:18:30.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.003+0000 7f5896040640 1 -- 172.21.15.40:0/2787906670 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58901a6da0 con 0x7f58901020d0 2024-04-21T23:18:30.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.003+0000 7f5896040640 1 -- 172.21.15.40:0/2787906670 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58900689b0 con 0x7f58901020d0 2024-04-21T23:18:30.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.005+0000 7f588cff9640 1 -- 172.21.15.40:0/2787906670 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7f5884010ac0 con 0x7f58901020d0 2024-04-21T23:18:30.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.006+0000 7f588cff9640 1 --2- 172.21.15.40:0/2787906670 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f58780a2e20 0x7f58780a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:30.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.006+0000 7f588f7fe640 1 --2- 172.21.15.40:0/2787906670 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f58780a2e20 0x7f58780a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:30.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.006+0000 7f588cff9640 1 -- 172.21.15.40:0/2787906670 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f58840d03a0 con 0x7f58901020d0 2024-04-21T23:18:30.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.006+0000 7f588f7fe640 1 --2- 172.21.15.40:0/2787906670 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f58780a2e20 0x7f58780a52e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5880005fd0 tx=0x7f588000b510 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:30.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.010+0000 7f588cff9640 1 -- 172.21.15.40:0/2787906670 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f588409c9c0 con 0x7f58901020d0 2024-04-21T23:18:30.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:29 smithi040 ceph-mon[27984]: from='client.25214 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T23:18:30.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:29 smithi040 ceph-mon[27984]: from='client.15615 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T23:18:30.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:29 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/1597023666' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:18:30.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:29 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/1272101549' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:18:30.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.196+0000 7f5896040640 1 -- 172.21.15.40:0/2787906670 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f5890109c80 con 0x7f58901020d0 2024-04-21T23:18:30.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.197+0000 7f588cff9640 1 -- 172.21.15.40:0/2787906670 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v95) ==== 104+0+11 (secure 0 0 0) 0x7f588409c2c0 con 0x7f58901020d0 2024-04-21T23:18:30.196 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-21T23:18:30.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.199+0000 7f5896040640 1 -- 172.21.15.40:0/2787906670 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f58780a2e20 msgr2=0x7f58780a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:30.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.199+0000 7f5896040640 1 --2- 172.21.15.40:0/2787906670 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f58780a2e20 0x7f58780a52e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5880005fd0 tx=0x7f588000b510 comp rx=0 tx=0).stop 2024-04-21T23:18:30.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.199+0000 7f5896040640 1 -- 172.21.15.40:0/2787906670 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58901020d0 msgr2=0x7f5890073f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:30.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.199+0000 7f5896040640 1 --2- 172.21.15.40:0/2787906670 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58901020d0 0x7f5890073f80 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f588400e9f0 tx=0x7f588400eeb0 comp rx=0 tx=0).stop 2024-04-21T23:18:30.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.199+0000 7f5896040640 1 -- 172.21.15.40:0/2787906670 shutdown_connections 2024-04-21T23:18:30.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.199+0000 7f5896040640 1 --2- 172.21.15.40:0/2787906670 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f58780a2e20 0x7f58780a52e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:30.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.199+0000 7f5896040640 1 --2- 172.21.15.40:0/2787906670 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5890107720 0x7f58900744c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:30.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.199+0000 7f5896040640 1 --2- 172.21.15.40:0/2787906670 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58901020d0 0x7f5890073f80 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:30.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.199+0000 7f5896040640 1 --2- 172.21.15.40:0/2787906670 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5890101710 0x7f5890073a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:30.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.199+0000 7f5896040640 1 -- 172.21.15.40:0/2787906670 >> 172.21.15.40:0/2787906670 conn(0x7f58900fd180 msgr2=0x7f58900fd870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:30.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.199+0000 7f5896040640 1 -- 172.21.15.40:0/2787906670 shutdown_connections 2024-04-21T23:18:30.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.199+0000 7f5896040640 1 -- 172.21.15.40:0/2787906670 wait complete. 2024-04-21T23:18:30.210 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T23:18:30.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.316+0000 7f336c0c9640 1 -- 172.21.15.40:0/2343827362 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33641066b0 msgr2=0x7f3364106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:30.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.316+0000 7f336c0c9640 1 --2- 172.21.15.40:0/2343827362 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33641066b0 0x7f3364106a90 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f3360009930 tx=0x7f336002bc20 comp rx=0 tx=0).stop 2024-04-21T23:18:30.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.316+0000 7f336c0c9640 1 -- 172.21.15.40:0/2343827362 shutdown_connections 2024-04-21T23:18:30.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.316+0000 7f336c0c9640 1 --2- 172.21.15.40:0/2343827362 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3364100610 0x7f3364100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:30.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.316+0000 7f336c0c9640 1 --2- 172.21.15.40:0/2343827362 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3364100fd0 0x7f336410d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:30.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.316+0000 7f336c0c9640 1 --2- 172.21.15.40:0/2343827362 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33641066b0 0x7f3364106a90 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:30.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.317+0000 7f336c0c9640 1 -- 172.21.15.40:0/2343827362 >> 172.21.15.40:0/2343827362 conn(0x7f3364077740 msgr2=0x7f3364077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:30.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.317+0000 7f336c0c9640 1 -- 172.21.15.40:0/2343827362 shutdown_connections 2024-04-21T23:18:30.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.317+0000 7f336c0c9640 1 -- 172.21.15.40:0/2343827362 wait complete. 2024-04-21T23:18:30.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.317+0000 7f336c0c9640 1 Processor -- start 2024-04-21T23:18:30.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.317+0000 7f336c0c9640 1 -- start start 2024-04-21T23:18:30.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.317+0000 7f336c0c9640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3364100610 0x7f336419a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:30.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.317+0000 7f336c0c9640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3364100fd0 0x7f336419a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:30.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.317+0000 7f336c0c9640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f33641066b0 0x7f336419ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:30.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.317+0000 7f336c0c9640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3364111b80 con 0x7f3364100fd0 2024-04-21T23:18:30.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.317+0000 7f336c0c9640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3364111a00 con 0x7f33641066b0 2024-04-21T23:18:30.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.317+0000 7f336c0c9640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3364111d00 con 0x7f3364100610 2024-04-21T23:18:30.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.317+0000 7f3369664640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3364100fd0 0x7f336419a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:30.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.317+0000 7f3369664640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3364100fd0 0x7f336419a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32898/0 (socket says 172.21.15.40:32898) 2024-04-21T23:18:30.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.318+0000 7f3369664640 1 -- 172.21.15.40:0/1073554669 learned_addr learned my addr 172.21.15.40:0/1073554669 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:30.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.318+0000 7f336a666640 1 --2- 172.21.15.40:0/1073554669 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f33641066b0 0x7f336419ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:30.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.318+0000 7f3369e65640 1 --2- 172.21.15.40:0/1073554669 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3364100610 0x7f336419a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:30.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.318+0000 7f3369664640 1 -- 172.21.15.40:0/1073554669 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3364100610 msgr2=0x7f336419a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:30.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.318+0000 7f3369664640 1 --2- 172.21.15.40:0/1073554669 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3364100610 0x7f336419a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:30.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.318+0000 7f3369664640 1 -- 172.21.15.40:0/1073554669 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f33641066b0 msgr2=0x7f336419ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:30.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.318+0000 7f3369664640 1 --2- 172.21.15.40:0/1073554669 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f33641066b0 0x7f336419ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:30.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.318+0000 7f3369664640 1 -- 172.21.15.40:0/1073554669 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f336419f190 con 0x7f3364100fd0 2024-04-21T23:18:30.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.318+0000 7f336a666640 1 --2- 172.21.15.40:0/1073554669 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f33641066b0 0x7f336419ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:18:30.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.318+0000 7f3369664640 1 --2- 172.21.15.40:0/1073554669 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3364100fd0 0x7f336419a6e0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f335400d950 tx=0x7f335400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:30.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.318+0000 7f335affd640 1 -- 172.21.15.40:0/1073554669 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3354017030 con 0x7f3364100fd0 2024-04-21T23:18:30.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.318+0000 7f335affd640 1 -- 172.21.15.40:0/1073554669 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3354004020 con 0x7f3364100fd0 2024-04-21T23:18:30.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.318+0000 7f336c0c9640 1 -- 172.21.15.40:0/1073554669 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f336419f480 con 0x7f3364100fd0 2024-04-21T23:18:30.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.318+0000 7f335affd640 1 -- 172.21.15.40:0/1073554669 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33540047a0 con 0x7f3364100fd0 2024-04-21T23:18:30.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.318+0000 7f336c0c9640 1 -- 172.21.15.40:0/1073554669 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33641a6d10 con 0x7f3364100fd0 2024-04-21T23:18:30.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.319+0000 7f336c0c9640 1 -- 172.21.15.40:0/1073554669 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3364068560 con 0x7f3364100fd0 2024-04-21T23:18:30.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.320+0000 7f335affd640 1 -- 172.21.15.40:0/1073554669 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7f33540041d0 con 0x7f3364100fd0 2024-04-21T23:18:30.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.321+0000 7f335affd640 1 --2- 172.21.15.40:0/1073554669 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f33500a2e20 0x7f33500a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:30.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.321+0000 7f3369e65640 1 --2- 172.21.15.40:0/1073554669 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f33500a2e20 0x7f33500a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:30.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.321+0000 7f335affd640 1 -- 172.21.15.40:0/1073554669 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f33540cf600 con 0x7f3364100fd0 2024-04-21T23:18:30.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.321+0000 7f3369e65640 1 --2- 172.21.15.40:0/1073554669 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f33500a2e20 0x7f33500a52e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3360036040 tx=0x7f3360005fd0 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:30.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.325+0000 7f335affd640 1 -- 172.21.15.40:0/1073554669 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f335409bbc0 con 0x7f3364100fd0 2024-04-21T23:18:30.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:29 smithi174 ceph-mon[31409]: from='client.25214 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T23:18:30.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:29 smithi174 ceph-mon[31409]: from='client.15615 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T23:18:30.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:29 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/1597023666' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:18:30.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:29 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/1272101549' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:18:30.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:29 smithi111 ceph-mon[31085]: from='client.25214 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T23:18:30.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:29 smithi111 ceph-mon[31085]: from='client.15615 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T23:18:30.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:29 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/1597023666' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:18:30.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:29 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/1272101549' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:18:30.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.509+0000 7f336c0c9640 1 -- 172.21.15.40:0/1073554669 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f33641a7010 con 0x7f3364100fd0 2024-04-21T23:18:30.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.509+0000 7f335affd640 1 -- 172.21.15.40:0/1073554669 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v95) ==== 100+0+11 (secure 0 0 0) 0x7f335409b4c0 con 0x7f3364100fd0 2024-04-21T23:18:30.508 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-04-21T23:18:30.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.511+0000 7f336c0c9640 1 -- 172.21.15.40:0/1073554669 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f33500a2e20 msgr2=0x7f33500a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:30.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.511+0000 7f336c0c9640 1 --2- 172.21.15.40:0/1073554669 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f33500a2e20 0x7f33500a52e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3360036040 tx=0x7f3360005fd0 comp rx=0 tx=0).stop 2024-04-21T23:18:30.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.511+0000 7f336c0c9640 1 -- 172.21.15.40:0/1073554669 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3364100fd0 msgr2=0x7f336419a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:30.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.511+0000 7f336c0c9640 1 --2- 172.21.15.40:0/1073554669 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3364100fd0 0x7f336419a6e0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f335400d950 tx=0x7f335400de10 comp rx=0 tx=0).stop 2024-04-21T23:18:30.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.511+0000 7f336c0c9640 1 -- 172.21.15.40:0/1073554669 shutdown_connections 2024-04-21T23:18:30.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.511+0000 7f336c0c9640 1 --2- 172.21.15.40:0/1073554669 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f33500a2e20 0x7f33500a52e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:30.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.511+0000 7f336c0c9640 1 --2- 172.21.15.40:0/1073554669 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f33641066b0 0x7f336419ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:30.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.511+0000 7f336c0c9640 1 --2- 172.21.15.40:0/1073554669 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3364100fd0 0x7f336419a6e0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:30.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.511+0000 7f336c0c9640 1 --2- 172.21.15.40:0/1073554669 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3364100610 0x7f336419a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:30.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.511+0000 7f336c0c9640 1 -- 172.21.15.40:0/1073554669 >> 172.21.15.40:0/1073554669 conn(0x7f3364077740 msgr2=0x7f33640fee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:30.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.511+0000 7f336c0c9640 1 -- 172.21.15.40:0/1073554669 shutdown_connections 2024-04-21T23:18:30.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.511+0000 7f336c0c9640 1 -- 172.21.15.40:0/1073554669 wait complete. 2024-04-21T23:18:30.521 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T23:18:30.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.627+0000 7f24d0bde640 1 -- 172.21.15.40:0/116345710 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24cc1066b0 msgr2=0x7f24cc106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:30.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.627+0000 7f24d0bde640 1 --2- 172.21.15.40:0/116345710 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24cc1066b0 0x7f24cc106a90 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f24bc009930 tx=0x7f24bc02bc30 comp rx=0 tx=0).stop 2024-04-21T23:18:30.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.627+0000 7f24d0bde640 1 -- 172.21.15.40:0/116345710 shutdown_connections 2024-04-21T23:18:30.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.627+0000 7f24d0bde640 1 --2- 172.21.15.40:0/116345710 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24cc100fd0 0x7f24cc10d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:30.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.627+0000 7f24d0bde640 1 --2- 172.21.15.40:0/116345710 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f24cc100610 0x7f24cc100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:30.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.627+0000 7f24d0bde640 1 --2- 172.21.15.40:0/116345710 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24cc1066b0 0x7f24cc106a90 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:30.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.627+0000 7f24d0bde640 1 -- 172.21.15.40:0/116345710 >> 172.21.15.40:0/116345710 conn(0x7f24cc077740 msgr2=0x7f24cc077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:30.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.627+0000 7f24d0bde640 1 -- 172.21.15.40:0/116345710 shutdown_connections 2024-04-21T23:18:30.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.627+0000 7f24d0bde640 1 -- 172.21.15.40:0/116345710 wait complete. 2024-04-21T23:18:30.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.627+0000 7f24d0bde640 1 Processor -- start 2024-04-21T23:18:30.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.627+0000 7f24d0bde640 1 -- start start 2024-04-21T23:18:30.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.628+0000 7f24d0bde640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24cc100610 0x7f24cc198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:30.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.628+0000 7f24d0bde640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f24cc100fd0 0x7f24cc198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:30.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.628+0000 7f24d0bde640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24cc1066b0 0x7f24cc19c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:30.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.628+0000 7f24d0bde640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f24cc110a30 con 0x7f24cc1066b0 2024-04-21T23:18:30.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.628+0000 7f24d0bde640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f24cc1108b0 con 0x7f24cc100610 2024-04-21T23:18:30.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.628+0000 7f24d0bde640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f24cc110bb0 con 0x7f24cc100fd0 2024-04-21T23:18:30.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.628+0000 7f24cad9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24cc1066b0 0x7f24cc19c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:30.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.628+0000 7f24cad9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24cc1066b0 0x7f24cc19c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32914/0 (socket says 172.21.15.40:32914) 2024-04-21T23:18:30.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.628+0000 7f24cad9d640 1 -- 172.21.15.40:0/4010835270 learned_addr learned my addr 172.21.15.40:0/4010835270 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:30.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.628+0000 7f24ca59c640 1 --2- 172.21.15.40:0/4010835270 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24cc100610 0x7f24cc198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:30.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.628+0000 7f24c9d9b640 1 --2- 172.21.15.40:0/4010835270 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f24cc100fd0 0x7f24cc198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:30.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.628+0000 7f24cad9d640 1 -- 172.21.15.40:0/4010835270 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f24cc100fd0 msgr2=0x7f24cc198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:30.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.628+0000 7f24cad9d640 1 --2- 172.21.15.40:0/4010835270 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f24cc100fd0 0x7f24cc198580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:30.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.628+0000 7f24cad9d640 1 -- 172.21.15.40:0/4010835270 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24cc100610 msgr2=0x7f24cc198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:30.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.628+0000 7f24cad9d640 1 --2- 172.21.15.40:0/4010835270 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24cc100610 0x7f24cc198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:30.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.628+0000 7f24cad9d640 1 -- 172.21.15.40:0/4010835270 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24cc19cfe0 con 0x7f24cc1066b0 2024-04-21T23:18:30.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.628+0000 7f24cad9d640 1 --2- 172.21.15.40:0/4010835270 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24cc1066b0 0x7f24cc19c8b0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f24c400eaa0 tx=0x7f24c400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:30.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.628+0000 7f24bb7fe640 1 -- 172.21.15.40:0/4010835270 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24c400cdb0 con 0x7f24cc1066b0 2024-04-21T23:18:30.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.628+0000 7f24bb7fe640 1 -- 172.21.15.40:0/4010835270 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24c4005020 con 0x7f24cc1066b0 2024-04-21T23:18:30.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.628+0000 7f24d0bde640 1 -- 172.21.15.40:0/4010835270 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24cc19d2d0 con 0x7f24cc1066b0 2024-04-21T23:18:30.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.628+0000 7f24bb7fe640 1 -- 172.21.15.40:0/4010835270 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24c40116b0 con 0x7f24cc1066b0 2024-04-21T23:18:30.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.629+0000 7f24d0bde640 1 -- 172.21.15.40:0/4010835270 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24cc1a4b60 con 0x7f24cc1066b0 2024-04-21T23:18:30.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.630+0000 7f24d0bde640 1 -- 172.21.15.40:0/4010835270 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24cc068560 con 0x7f24cc1066b0 2024-04-21T23:18:30.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.631+0000 7f24bb7fe640 1 -- 172.21.15.40:0/4010835270 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7f24c4005560 con 0x7f24cc1066b0 2024-04-21T23:18:30.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.632+0000 7f24bb7fe640 1 --2- 172.21.15.40:0/4010835270 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f24b40a2e20 0x7f24b40a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:30.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.632+0000 7f24ca59c640 1 --2- 172.21.15.40:0/4010835270 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f24b40a2e20 0x7f24b40a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:30.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.632+0000 7f24bb7fe640 1 -- 172.21.15.40:0/4010835270 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f24c40cfd00 con 0x7f24cc1066b0 2024-04-21T23:18:30.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.632+0000 7f24ca59c640 1 --2- 172.21.15.40:0/4010835270 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f24b40a2e20 0x7f24b40a52e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f24bc000c00 tx=0x7f24bc005fd0 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:30.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.636+0000 7f24bb7fe640 1 -- 172.21.15.40:0/4010835270 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f24c4016030 con 0x7f24cc1066b0 2024-04-21T23:18:30.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.825+0000 7f24d0bde640 1 -- 172.21.15.40:0/4010835270 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f24cc1a4e00 con 0x7f24cc1066b0 2024-04-21T23:18:30.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.825+0000 7f24bb7fe640 1 -- 172.21.15.40:0/4010835270 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v95) ==== 103+0+11 (secure 0 0 0) 0x7f24c409c320 con 0x7f24cc1066b0 2024-04-21T23:18:30.824 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-21T23:18:30.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.827+0000 7f24d0bde640 1 -- 172.21.15.40:0/4010835270 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f24b40a2e20 msgr2=0x7f24b40a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:30.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.827+0000 7f24d0bde640 1 --2- 172.21.15.40:0/4010835270 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f24b40a2e20 0x7f24b40a52e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f24bc000c00 tx=0x7f24bc005fd0 comp rx=0 tx=0).stop 2024-04-21T23:18:30.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.827+0000 7f24d0bde640 1 -- 172.21.15.40:0/4010835270 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24cc1066b0 msgr2=0x7f24cc19c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:30.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.827+0000 7f24d0bde640 1 --2- 172.21.15.40:0/4010835270 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24cc1066b0 0x7f24cc19c8b0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f24c400eaa0 tx=0x7f24c400ef60 comp rx=0 tx=0).stop 2024-04-21T23:18:30.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.827+0000 7f24d0bde640 1 -- 172.21.15.40:0/4010835270 shutdown_connections 2024-04-21T23:18:30.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.827+0000 7f24d0bde640 1 --2- 172.21.15.40:0/4010835270 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f24b40a2e20 0x7f24b40a52e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:30.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.827+0000 7f24d0bde640 1 --2- 172.21.15.40:0/4010835270 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24cc1066b0 0x7f24cc19c8b0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:30.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.827+0000 7f24d0bde640 1 --2- 172.21.15.40:0/4010835270 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f24cc100fd0 0x7f24cc198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:30.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.827+0000 7f24d0bde640 1 --2- 172.21.15.40:0/4010835270 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24cc100610 0x7f24cc198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:30.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.827+0000 7f24d0bde640 1 -- 172.21.15.40:0/4010835270 >> 172.21.15.40:0/4010835270 conn(0x7f24cc077740 msgr2=0x7f24cc0ff8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:30.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.827+0000 7f24d0bde640 1 -- 172.21.15.40:0/4010835270 shutdown_connections 2024-04-21T23:18:30.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.827+0000 7f24d0bde640 1 -- 172.21.15.40:0/4010835270 wait complete. 2024-04-21T23:18:30.837 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:18:30.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.940+0000 7fbba9ef4640 1 -- 172.21.15.40:0/2767223250 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbba4100fd0 msgr2=0x7fbba410d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:30.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.940+0000 7fbba9ef4640 1 --2- 172.21.15.40:0/2767223250 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbba4100fd0 0x7fbba410d4e0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fbb9c009cd0 tx=0x7fbb9c02c0f0 comp rx=0 tx=0).stop 2024-04-21T23:18:30.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.940+0000 7fbba9ef4640 1 -- 172.21.15.40:0/2767223250 shutdown_connections 2024-04-21T23:18:30.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.940+0000 7fbba9ef4640 1 --2- 172.21.15.40:0/2767223250 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbba4100610 0x7fbba4100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:30.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.940+0000 7fbba9ef4640 1 --2- 172.21.15.40:0/2767223250 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbba4100fd0 0x7fbba410d4e0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:30.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.940+0000 7fbba9ef4640 1 --2- 172.21.15.40:0/2767223250 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbba41066b0 0x7fbba4106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:30.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.940+0000 7fbba9ef4640 1 -- 172.21.15.40:0/2767223250 >> 172.21.15.40:0/2767223250 conn(0x7fbba40769c0 msgr2=0x7fbba4076dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:30.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.940+0000 7fbba9ef4640 1 -- 172.21.15.40:0/2767223250 shutdown_connections 2024-04-21T23:18:30.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.940+0000 7fbba9ef4640 1 -- 172.21.15.40:0/2767223250 wait complete. 2024-04-21T23:18:30.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.940+0000 7fbba9ef4640 1 Processor -- start 2024-04-21T23:18:30.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.940+0000 7fbba9ef4640 1 -- start start 2024-04-21T23:18:30.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.941+0000 7fbba9ef4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbba4100610 0x7fbba4197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:30.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.941+0000 7fbba9ef4640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbba4100fd0 0x7fbba4198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:30.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.941+0000 7fbba9ef4640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbba41066b0 0x7fbba419c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:30.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.941+0000 7fbba9ef4640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fbba41109f0 con 0x7fbba4100fd0 2024-04-21T23:18:30.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.941+0000 7fbba9ef4640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fbba4110870 con 0x7fbba41066b0 2024-04-21T23:18:30.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.941+0000 7fbba9ef4640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fbba4110b70 con 0x7fbba4100610 2024-04-21T23:18:30.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.941+0000 7fbba2ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbba4100fd0 0x7fbba4198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:30.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.941+0000 7fbba2ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbba4100fd0 0x7fbba4198400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32938/0 (socket says 172.21.15.40:32938) 2024-04-21T23:18:30.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.941+0000 7fbba2ffd640 1 -- 172.21.15.40:0/151147968 learned_addr learned my addr 172.21.15.40:0/151147968 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:30.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.941+0000 7fbba37fe640 1 --2- 172.21.15.40:0/151147968 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbba4100610 0x7fbba4197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:30.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.941+0000 7fbba3fff640 1 --2- 172.21.15.40:0/151147968 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbba41066b0 0x7fbba419c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:30.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.941+0000 7fbba2ffd640 1 -- 172.21.15.40:0/151147968 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbba4100610 msgr2=0x7fbba4197ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:30.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.941+0000 7fbba2ffd640 1 --2- 172.21.15.40:0/151147968 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbba4100610 0x7fbba4197ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:30.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.941+0000 7fbba2ffd640 1 -- 172.21.15.40:0/151147968 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbba41066b0 msgr2=0x7fbba419c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:30.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.941+0000 7fbba2ffd640 1 --2- 172.21.15.40:0/151147968 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbba41066b0 0x7fbba419c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:30.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.941+0000 7fbba2ffd640 1 -- 172.21.15.40:0/151147968 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbba419ceb0 con 0x7fbba4100fd0 2024-04-21T23:18:30.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.941+0000 7fbba3fff640 1 --2- 172.21.15.40:0/151147968 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbba41066b0 0x7fbba419c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:18:30.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.941+0000 7fbba2ffd640 1 --2- 172.21.15.40:0/151147968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbba4100fd0 0x7fbba4198400 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fbb9800c890 tx=0x7fbb9800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:30.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.941+0000 7fbba0ff9640 1 -- 172.21.15.40:0/151147968 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb98004220 con 0x7fbba4100fd0 2024-04-21T23:18:30.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.942+0000 7fbba9ef4640 1 -- 172.21.15.40:0/151147968 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbba419d1a0 con 0x7fbba4100fd0 2024-04-21T23:18:30.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.942+0000 7fbba0ff9640 1 -- 172.21.15.40:0/151147968 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb980043c0 con 0x7fbba4100fd0 2024-04-21T23:18:30.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.942+0000 7fbba0ff9640 1 -- 172.21.15.40:0/151147968 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb980049e0 con 0x7fbba4100fd0 2024-04-21T23:18:30.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.942+0000 7fbba9ef4640 1 -- 172.21.15.40:0/151147968 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbba41a4a30 con 0x7fbba4100fd0 2024-04-21T23:18:30.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.942+0000 7fbba9ef4640 1 -- 172.21.15.40:0/151147968 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbba4101d50 con 0x7fbba4100fd0 2024-04-21T23:18:30.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.944+0000 7fbba0ff9640 1 -- 172.21.15.40:0/151147968 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7fbb980079b0 con 0x7fbba4100fd0 2024-04-21T23:18:30.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.944+0000 7fbba0ff9640 1 --2- 172.21.15.40:0/151147968 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fbb8c0a2e20 0x7fbb8c0a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:30.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.944+0000 7fbba0ff9640 1 -- 172.21.15.40:0/151147968 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7fbb980cfb70 con 0x7fbba4100fd0 2024-04-21T23:18:30.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.944+0000 7fbba37fe640 1 --2- 172.21.15.40:0/151147968 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fbb8c0a2e20 0x7fbb8c0a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:30.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.945+0000 7fbba37fe640 1 --2- 172.21.15.40:0/151147968 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fbb8c0a2e20 0x7fbb8c0a52e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fbba4101750 tx=0x7fbb94005f90 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:30.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:30.948+0000 7fbba0ff9640 1 -- 172.21.15.40:0/151147968 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fbb98014030 con 0x7fbba4100fd0 2024-04-21T23:18:31.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.173+0000 7fbba9ef4640 1 -- 172.21.15.40:0/151147968 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbba40689b0 con 0x7fbba4100fd0 2024-04-21T23:18:31.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.173+0000 7fbba0ff9640 1 -- 172.21.15.40:0/151147968 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11882 (secure 0 0 0) 0x7fbb9809c130 con 0x7fbba4100fd0 2024-04-21T23:18:31.172 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:18:31.173 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/1571948830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1571948830},{"type":"v1","addr":"172.21.15.40:6835","nonce":1571948830}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/969356642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":969356642},{"type":"v1","addr":"172.21.15.40:6837","nonce":969356642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6839/3025057845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":3025057845},{"type":"v1","addr":"172.21.15.40:6839","nonce":3025057845}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6841/3350488804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":3350488804},{"type":"v1","addr":"172.21.15.40:6841","nonce":3350488804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/3438493203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":3438493203},{"type":"v1","addr":"172.21.15.174:6833","nonce":3438493203}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/3105752678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":3105752678},{"type":"v1","addr":"172.21.15.111:6833","nonce":3105752678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T23:15:55.742233+0000","modified":"2024-04-21T23:18:21.879184+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24521,"mds_1":24502,"mds_2":24494},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24461":{"gid":24461,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6835/2704462092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2704462092},{"type":"v1","addr":"172.21.15.111:6835","nonce":2704462092}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/3905733045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3905733045},{"type":"v1","addr":"172.21.15.174:6835","nonce":3905733045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6837/465579746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":465579746},{"type":"v1","addr":"172.21.15.174:6837","nonce":465579746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/2703764182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":2703764182},{"type":"v1","addr":"172.21.15.111:6837","nonce":2703764182}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24502":{"gid":24502,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.174:6839/2467509274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":2467509274},{"type":"v1","addr":"172.21.15.174:6839","nonce":2467509274}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3712235548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3712235548},{"type":"v1","addr":"172.21.15.111:6839","nonce":3712235548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24521,24494,24502]},"id":1}]} 2024-04-21T23:18:31.173 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 25 2024-04-21T23:18:31.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.175+0000 7fbba9ef4640 1 -- 172.21.15.40:0/151147968 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fbb8c0a2e20 msgr2=0x7fbb8c0a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:31.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.175+0000 7fbba9ef4640 1 --2- 172.21.15.40:0/151147968 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fbb8c0a2e20 0x7fbb8c0a52e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fbba4101750 tx=0x7fbb94005f90 comp rx=0 tx=0).stop 2024-04-21T23:18:31.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.175+0000 7fbba9ef4640 1 -- 172.21.15.40:0/151147968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbba4100fd0 msgr2=0x7fbba4198400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:31.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.175+0000 7fbba9ef4640 1 --2- 172.21.15.40:0/151147968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbba4100fd0 0x7fbba4198400 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fbb9800c890 tx=0x7fbb9800cd50 comp rx=0 tx=0).stop 2024-04-21T23:18:31.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.175+0000 7fbba9ef4640 1 -- 172.21.15.40:0/151147968 shutdown_connections 2024-04-21T23:18:31.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.175+0000 7fbba9ef4640 1 --2- 172.21.15.40:0/151147968 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fbb8c0a2e20 0x7fbb8c0a52e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.175+0000 7fbba9ef4640 1 --2- 172.21.15.40:0/151147968 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbba41066b0 0x7fbba419c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.175+0000 7fbba9ef4640 1 --2- 172.21.15.40:0/151147968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbba4100fd0 0x7fbba4198400 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.175+0000 7fbba9ef4640 1 --2- 172.21.15.40:0/151147968 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbba4100610 0x7fbba4197ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.175+0000 7fbba9ef4640 1 -- 172.21.15.40:0/151147968 >> 172.21.15.40:0/151147968 conn(0x7fbba40769c0 msgr2=0x7fbba40ff540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:31.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.175+0000 7fbba9ef4640 1 -- 172.21.15.40:0/151147968 shutdown_connections 2024-04-21T23:18:31.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.175+0000 7fbba9ef4640 1 -- 172.21.15.40:0/151147968 wait complete. 2024-04-21T23:18:31.181 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:30 smithi040 ceph-mon[27984]: from='client.25226 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T23:18:31.181 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:30 smithi040 ceph-mon[27984]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 615 B/s wr, 26 op/s 2024-04-21T23:18:31.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:30 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/2787906670' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T23:18:31.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:30 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/1073554669' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T23:18:31.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:30 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/4010835270' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T23:18:31.186 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T23:18:31.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.289+0000 7f382b583640 1 -- 172.21.15.40:0/1418829326 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3824102a20 msgr2=0x7f3824102e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:31.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.289+0000 7f382b583640 1 --2- 172.21.15.40:0/1418829326 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3824102a20 0x7f3824102e00 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f3820009900 tx=0x7f382002bcd0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.289+0000 7f382b583640 1 -- 172.21.15.40:0/1418829326 shutdown_connections 2024-04-21T23:18:31.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.289+0000 7f382b583640 1 --2- 172.21.15.40:0/1418829326 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3824103340 0x7f382410b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.289+0000 7f382b583640 1 --2- 172.21.15.40:0/1418829326 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f382410bd60 0x7f382410e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.289+0000 7f382b583640 1 --2- 172.21.15.40:0/1418829326 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3824102a20 0x7f3824102e00 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.289+0000 7f382b583640 1 -- 172.21.15.40:0/1418829326 >> 172.21.15.40:0/1418829326 conn(0x7f38240777a0 msgr2=0x7f3824077bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:31.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.290+0000 7f382b583640 1 -- 172.21.15.40:0/1418829326 shutdown_connections 2024-04-21T23:18:31.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.290+0000 7f382b583640 1 -- 172.21.15.40:0/1418829326 wait complete. 2024-04-21T23:18:31.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.290+0000 7f382b583640 1 Processor -- start 2024-04-21T23:18:31.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.290+0000 7f382b583640 1 -- start start 2024-04-21T23:18:31.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.290+0000 7f382b583640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3824102a20 0x7f3824100670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:31.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.290+0000 7f382b583640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3824103340 0x7f3824100bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:31.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.290+0000 7f382b583640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f382410bd60 0x7f38241010f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:31.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.290+0000 7f382b583640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3824111b80 con 0x7f3824103340 2024-04-21T23:18:31.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.290+0000 7f382b583640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3824111a00 con 0x7f382410bd60 2024-04-21T23:18:31.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.290+0000 7f382b583640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3824111d00 con 0x7f3824102a20 2024-04-21T23:18:31.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.290+0000 7f3828b1e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3824103340 0x7f3824100bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:31.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.290+0000 7f3828b1e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3824103340 0x7f3824100bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32956/0 (socket says 172.21.15.40:32956) 2024-04-21T23:18:31.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.290+0000 7f3828b1e640 1 -- 172.21.15.40:0/958670969 learned_addr learned my addr 172.21.15.40:0/958670969 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:31.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.291+0000 7f3829b20640 1 --2- 172.21.15.40:0/958670969 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f382410bd60 0x7f38241010f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:31.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.291+0000 7f382931f640 1 --2- 172.21.15.40:0/958670969 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3824102a20 0x7f3824100670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:31.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.291+0000 7f3828b1e640 1 -- 172.21.15.40:0/958670969 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3824102a20 msgr2=0x7f3824100670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:31.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.291+0000 7f3828b1e640 1 --2- 172.21.15.40:0/958670969 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3824102a20 0x7f3824100670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.291+0000 7f3828b1e640 1 -- 172.21.15.40:0/958670969 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f382410bd60 msgr2=0x7f38241010f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:31.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.291+0000 7f3828b1e640 1 --2- 172.21.15.40:0/958670969 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f382410bd60 0x7f38241010f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.291+0000 7f3828b1e640 1 -- 172.21.15.40:0/958670969 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38241a6540 con 0x7f3824103340 2024-04-21T23:18:31.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.291+0000 7f3828b1e640 1 --2- 172.21.15.40:0/958670969 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3824103340 0x7f3824100bb0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f381400c920 tx=0x7f381400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:31.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.291+0000 7f381a7fc640 1 -- 172.21.15.40:0/958670969 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3814007c70 con 0x7f3824103340 2024-04-21T23:18:31.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.291+0000 7f381a7fc640 1 -- 172.21.15.40:0/958670969 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3814007e10 con 0x7f3824103340 2024-04-21T23:18:31.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.291+0000 7f381a7fc640 1 -- 172.21.15.40:0/958670969 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3814018890 con 0x7f3824103340 2024-04-21T23:18:31.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.291+0000 7f382b583640 1 -- 172.21.15.40:0/958670969 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38241a6830 con 0x7f3824103340 2024-04-21T23:18:31.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.291+0000 7f382b583640 1 -- 172.21.15.40:0/958670969 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38241a6dd0 con 0x7f3824103340 2024-04-21T23:18:31.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.292+0000 7f382b583640 1 -- 172.21.15.40:0/958670969 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38240689b0 con 0x7f3824103340 2024-04-21T23:18:31.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.293+0000 7f381a7fc640 1 -- 172.21.15.40:0/958670969 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7f3814018a70 con 0x7f3824103340 2024-04-21T23:18:31.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.294+0000 7f381a7fc640 1 --2- 172.21.15.40:0/958670969 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f38100a2ea0 0x7f38100a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:31.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.294+0000 7f381a7fc640 1 -- 172.21.15.40:0/958670969 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f38140cfc80 con 0x7f3824103340 2024-04-21T23:18:31.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.294+0000 7f382931f640 1 --2- 172.21.15.40:0/958670969 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f38100a2ea0 0x7f38100a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:31.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.294+0000 7f382931f640 1 --2- 172.21.15.40:0/958670969 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f38100a2ea0 0x7f38100a5360 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3820036040 tx=0x7f3820005fd0 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:31.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.298+0000 7f381a7fc640 1 -- 172.21.15.40:0/958670969 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f3814014030 con 0x7f3824103340 2024-04-21T23:18:31.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:30 smithi174 ceph-mon[31409]: from='client.25226 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T23:18:31.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:30 smithi174 ceph-mon[31409]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 615 B/s wr, 26 op/s 2024-04-21T23:18:31.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:30 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/2787906670' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T23:18:31.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:30 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/1073554669' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T23:18:31.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:30 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/4010835270' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T23:18:31.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:30 smithi111 ceph-mon[31085]: from='client.25226 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T23:18:31.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:30 smithi111 ceph-mon[31085]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 615 B/s wr, 26 op/s 2024-04-21T23:18:31.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:30 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/2787906670' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T23:18:31.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:30 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/1073554669' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T23:18:31.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:30 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/4010835270' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T23:18:31.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.484+0000 7f382b583640 1 -- 172.21.15.40:0/958670969 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3824106ee0 con 0x7f3824103340 2024-04-21T23:18:31.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.486+0000 7f381a7fc640 1 -- 172.21.15.40:0/958670969 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) ==== 74+0+25231 (secure 0 0 0) 0x7f381409c2a0 con 0x7f3824103340 2024-04-21T23:18:31.485 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:18:31.485 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":95,"fsid":"d814e4dc-0033-11ef-bc93-c7b262605968","created":"2024-04-21T23:07:52.659207+0000","modified":"2024-04-21T23:18:23.389706+0000","last_up_change":"2024-04-21T23:15:05.308821+0000","last_in_change":"2024-04-21T23:14:45.447491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:11:14.824775+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T23:15:54.322571+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T23:15:55.164493+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T23:15:58.838266+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"0e8d692a-22fe-4a87-8d99-9151f96cbb02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6803","nonce":2009758784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6805","nonce":2009758784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6809","nonce":2009758784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6807","nonce":2009758784}]},"public_addr":"172.21.15.40:6803/2009758784","cluster_addr":"172.21.15.40:6805/2009758784","heartbeat_back_addr":"172.21.15.40:6809/2009758784","heartbeat_front_addr":"172.21.15.40:6807/2009758784","state":["exists","up"]},{"osd":1,"uuid":"fe5fe11b-0e84-4359-83a7-1ba45105755a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6801","nonce":1763903836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6803","nonce":1763903836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6807","nonce":1763903836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6805","nonce":1763903836}]},"public_addr":"172.21.15.111:6801/1763903836","cluster_addr":"172.21.15.111:6803/1763903836","heartbeat_back_addr":"172.21.15.111:6807/1763903836","heartbeat_front_addr":"172.21.15.111:6805/1763903836","state":["exists","up"]},{"osd":2,"uuid":"813ad735-5cc2-4da0-ae16-6db238395fbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6801","nonce":439695825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6803","nonce":439695825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6807","nonce":439695825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6805","nonce":439695825}]},"public_addr":"172.21.15.174:6801/439695825","cluster_addr":"172.21.15.174:6803/439695825","heartbeat_back_addr":"172.21.15.174:6807/439695825","heartbeat_front_addr":"172.21.15.174:6805/439695825","state":["exists","up"]},{"osd":3,"uuid":"09becb03-b7ce-4ffc-917d-b7900ec37e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6811","nonce":3454312436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6813","nonce":3454312436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6817","nonce":3454312436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6815","nonce":3454312436}]},"public_addr":"172.21.15.40:6811/3454312436","cluster_addr":"172.21.15.40:6813/3454312436","heartbeat_back_addr":"172.21.15.40:6817/3454312436","heartbeat_front_addr":"172.21.15.40:6815/3454312436","state":["exists","up"]},{"osd":4,"uuid":"46eabfd7-d88a-4efd-94eb-04354d3a9d7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6809","nonce":1810866070}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6811","nonce":1810866070}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6815","nonce":1810866070}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6813","nonce":1810866070}]},"public_addr":"172.21.15.111:6809/1810866070","cluster_addr":"172.21.15.111:6811/1810866070","heartbeat_back_addr":"172.21.15.111:6815/1810866070","heartbeat_front_addr":"172.21.15.111:6813/1810866070","state":["exists","up"]},{"osd":5,"uuid":"6f73b582-6c65-41db-aab0-9b360f83d5c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6809","nonce":3752986711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6811","nonce":3752986711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6815","nonce":3752986711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6813","nonce":3752986711}]},"public_addr":"172.21.15.174:6809/3752986711","cluster_addr":"172.21.15.174:6811/3752986711","heartbeat_back_addr":"172.21.15.174:6815/3752986711","heartbeat_front_addr":"172.21.15.174:6813/3752986711","state":["exists","up"]},{"osd":6,"uuid":"39f305f0-12d2-4732-aaa0-cd66700af538","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6819","nonce":1618855427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6821","nonce":1618855427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6825","nonce":1618855427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6823","nonce":1618855427}]},"public_addr":"172.21.15.40:6819/1618855427","cluster_addr":"172.21.15.40:6821/1618855427","heartbeat_back_addr":"172.21.15.40:6825/1618855427","heartbeat_front_addr":"172.21.15.40:6823/1618855427","state":["exists","up"]},{"osd":7,"uuid":"769f2468-5102-490b-89f8-d0444d3169a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6817","nonce":927054162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6819","nonce":927054162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6823","nonce":927054162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6821","nonce":927054162}]},"public_addr":"172.21.15.111:6817/927054162","cluster_addr":"172.21.15.111:6819/927054162","heartbeat_back_addr":"172.21.15.111:6823/927054162","heartbeat_front_addr":"172.21.15.111:6821/927054162","state":["exists","up"]},{"osd":8,"uuid":"1a444e80-007d-473b-b972-4c814a813c22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6817","nonce":3817848746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6819","nonce":3817848746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6823","nonce":3817848746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6821","nonce":3817848746}]},"public_addr":"172.21.15.174:6817/3817848746","cluster_addr":"172.21.15.174:6819/3817848746","heartbeat_back_addr":"172.21.15.174:6823/3817848746","heartbeat_front_addr":"172.21.15.174:6821/3817848746","state":["exists","up"]},{"osd":9,"uuid":"9a6b6a56-6ae5-49fe-9e5b-cb17d686e0dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6827","nonce":1649446531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6829","nonce":1649446531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6833","nonce":1649446531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6831","nonce":1649446531}]},"public_addr":"172.21.15.40:6827/1649446531","cluster_addr":"172.21.15.40:6829/1649446531","heartbeat_back_addr":"172.21.15.40:6833/1649446531","heartbeat_front_addr":"172.21.15.40:6831/1649446531","state":["exists","up"]},{"osd":10,"uuid":"a70038ba-e2ad-4170-85cc-ea0470a2edef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6825","nonce":1227538935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6827","nonce":1227538935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6831","nonce":1227538935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6829","nonce":1227538935}]},"public_addr":"172.21.15.111:6825/1227538935","cluster_addr":"172.21.15.111:6827/1227538935","heartbeat_back_addr":"172.21.15.111:6831/1227538935","heartbeat_front_addr":"172.21.15.111:6829/1227538935","state":["exists","up"]},{"osd":11,"uuid":"781532bb-7a5e-44e3-8eb7-b584b0054666","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6825","nonce":2045865553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6827","nonce":2045865553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6831","nonce":2045865553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6829","nonce":2045865553}]},"public_addr":"172.21.15.174:6825/2045865553","cluster_addr":"172.21.15.174:6827/2045865553","heartbeat_back_addr":"172.21.15.174:6831/2045865553","heartbeat_front_addr":"172.21.15.174:6829/2045865553","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:10:24.761290+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:10:48.565428+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:11:12.307818+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:11:37.951529+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:02.832523+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:27.568879+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:52.997707+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:13:18.937274+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:13:44.708657+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:14:10.833855+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:14:37.078272+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:15:03.878076+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":1}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.2a","mappings":[{"from":3,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"4.9","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":10}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:6801/2500760788":"2024-04-22T23:18:23.389674+0000","172.21.15.40:6800/2500760788":"2024-04-22T23:18:23.389674+0000","172.21.15.40:0/2742002992":"2024-04-22T23:18:23.389674+0000","172.21.15.40:0/2494672676":"2024-04-22T23:18:23.389674+0000","172.21.15.40:6801/526554319":"2024-04-22T23:08:50.305770+0000","172.21.15.40:6800/526554319":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/3178763471":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/4045911364":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/276300080":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/3689649814":"2024-04-22T23:08:23.743030+0000","172.21.15.40:0/2241116715":"2024-04-22T23:08:23.743030+0000","172.21.15.40:0/1406242364":"2024-04-22T23:18:23.389674+0000","172.21.15.40:0/1737227818":"2024-04-22T23:18:23.389674+0000","172.21.15.40:6801/1479040326":"2024-04-22T23:08:23.743030+0000","172.21.15.40:6800/1479040326":"2024-04-22T23:08:23.743030+0000","172.21.15.40:0/681298099":"2024-04-22T23:08:23.743030+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T23:18:31.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.488+0000 7f382b583640 1 -- 172.21.15.40:0/958670969 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f38100a2ea0 msgr2=0x7f38100a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:31.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.488+0000 7f382b583640 1 --2- 172.21.15.40:0/958670969 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f38100a2ea0 0x7f38100a5360 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3820036040 tx=0x7f3820005fd0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.488+0000 7f382b583640 1 -- 172.21.15.40:0/958670969 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3824103340 msgr2=0x7f3824100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:31.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.488+0000 7f382b583640 1 --2- 172.21.15.40:0/958670969 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3824103340 0x7f3824100bb0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f381400c920 tx=0x7f381400cde0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.488+0000 7f382b583640 1 -- 172.21.15.40:0/958670969 shutdown_connections 2024-04-21T23:18:31.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.488+0000 7f382b583640 1 --2- 172.21.15.40:0/958670969 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f38100a2ea0 0x7f38100a5360 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.488+0000 7f382b583640 1 --2- 172.21.15.40:0/958670969 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f382410bd60 0x7f38241010f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.488+0000 7f382b583640 1 --2- 172.21.15.40:0/958670969 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3824103340 0x7f3824100bb0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.488+0000 7f382b583640 1 --2- 172.21.15.40:0/958670969 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3824102a20 0x7f3824100670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.488+0000 7f382b583640 1 -- 172.21.15.40:0/958670969 >> 172.21.15.40:0/958670969 conn(0x7f38240777a0 msgr2=0x7f38240fd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:31.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.488+0000 7f382b583640 1 -- 172.21.15.40:0/958670969 shutdown_connections 2024-04-21T23:18:31.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.488+0000 7f382b583640 1 -- 172.21.15.40:0/958670969 wait complete. 2024-04-21T23:18:31.499 DEBUG:tasks.fwd_scrub:Yielding 2024-04-21T23:18:31.499 INFO:teuthology.run_tasks:Running task exec... 2024-04-21T23:18:31.510 INFO:teuthology.task.exec:Executing custom commands... 2024-04-21T23:18:31.510 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi040.front.sepia.ceph.com 2024-04-21T23:18:31.510 DEBUG:teuthology.orchestra.run.smithi040:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-04-21T23:18:31.511 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-04-21T23:18:31.511 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T23:18:31.511 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:18:31.511 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T23:18:31.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.617+0000 7f532a6c2640 1 -- 172.21.15.40:0/3862644630 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53241029d0 msgr2=0x7f5324102db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:31.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.617+0000 7f532a6c2640 1 --2- 172.21.15.40:0/3862644630 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53241029d0 0x7f5324102db0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f5314009900 tx=0x7f531402bd60 comp rx=0 tx=0).stop 2024-04-21T23:18:31.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.617+0000 7f532a6c2640 1 -- 172.21.15.40:0/3862644630 shutdown_connections 2024-04-21T23:18:31.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.617+0000 7f532a6c2640 1 --2- 172.21.15.40:0/3862644630 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f53241032f0 0x7f532410b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.617+0000 7f532a6c2640 1 --2- 172.21.15.40:0/3862644630 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f532410bd10 0x7f532410e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.617+0000 7f532a6c2640 1 --2- 172.21.15.40:0/3862644630 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53241029d0 0x7f5324102db0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.617+0000 7f532a6c2640 1 -- 172.21.15.40:0/3862644630 >> 172.21.15.40:0/3862644630 conn(0x7f53240777c0 msgr2=0x7f5324077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:31.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.617+0000 7f532a6c2640 1 -- 172.21.15.40:0/3862644630 shutdown_connections 2024-04-21T23:18:31.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.617+0000 7f532a6c2640 1 -- 172.21.15.40:0/3862644630 wait complete. 2024-04-21T23:18:31.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.618+0000 7f532a6c2640 1 Processor -- start 2024-04-21T23:18:31.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.618+0000 7f532a6c2640 1 -- start start 2024-04-21T23:18:31.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.618+0000 7f6b233e0640 1 -- 172.21.15.40:0/1894173580 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b1c1066b0 msgr2=0x7f6b1c106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:31.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.618+0000 7f6b233e0640 1 --2- 172.21.15.40:0/1894173580 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b1c1066b0 0x7f6b1c106a90 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f6b18009930 tx=0x7f6b1802bc30 comp rx=0 tx=0).stop 2024-04-21T23:18:31.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.618+0000 7f6b233e0640 1 -- 172.21.15.40:0/1894173580 shutdown_connections 2024-04-21T23:18:31.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.618+0000 7f6b233e0640 1 --2- 172.21.15.40:0/1894173580 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6b1c100610 0x7f6b1c100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.618+0000 7f6b233e0640 1 --2- 172.21.15.40:0/1894173580 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b1c100fd0 0x7f6b1c10d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.618+0000 7f6b233e0640 1 --2- 172.21.15.40:0/1894173580 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b1c1066b0 0x7f6b1c106a90 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.618+0000 7f6b233e0640 1 -- 172.21.15.40:0/1894173580 >> 172.21.15.40:0/1894173580 conn(0x7f6b1c077740 msgr2=0x7f6b1c077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:31.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.618+0000 7f6b233e0640 1 -- 172.21.15.40:0/1894173580 shutdown_connections 2024-04-21T23:18:31.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.618+0000 7f6b233e0640 1 -- 172.21.15.40:0/1894173580 wait complete. 2024-04-21T23:18:31.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.618+0000 7f532a6c2640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53241029d0 0x7f5324197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:31.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.618+0000 7f532a6c2640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f53241032f0 0x7f53241984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:31.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.618+0000 7f532a6c2640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f532410bd10 0x7f532419c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:31.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.618+0000 7f532a6c2640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5324111b60 con 0x7f53241029d0 2024-04-21T23:18:31.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.618+0000 7f532a6c2640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f53241119e0 con 0x7f53241032f0 2024-04-21T23:18:31.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.618+0000 7f532a6c2640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5324111ce0 con 0x7f532410bd10 2024-04-21T23:18:31.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.618+0000 7f5323fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53241029d0 0x7f5324197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:31.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.619+0000 7f5323fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53241029d0 0x7f5324197fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32976/0 (socket says 172.21.15.40:32976) 2024-04-21T23:18:31.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.619+0000 7f5323fff640 1 -- 172.21.15.40:0/2332982767 learned_addr learned my addr 172.21.15.40:0/2332982767 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:31.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.619+0000 7f53237fe640 1 --2- 172.21.15.40:0/2332982767 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f53241032f0 0x7f53241984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:31.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.619+0000 7f5328c5f640 1 --2- 172.21.15.40:0/2332982767 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f532410bd10 0x7f532419c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:31.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.619+0000 7f5323fff640 1 -- 172.21.15.40:0/2332982767 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f532410bd10 msgr2=0x7f532419c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:31.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.619+0000 7f5323fff640 1 --2- 172.21.15.40:0/2332982767 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f532410bd10 0x7f532419c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.619+0000 7f5323fff640 1 -- 172.21.15.40:0/2332982767 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f53241032f0 msgr2=0x7f53241984e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:31.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.619+0000 7f5323fff640 1 --2- 172.21.15.40:0/2332982767 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f53241032f0 0x7f53241984e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.619+0000 7f5323fff640 1 -- 172.21.15.40:0/2332982767 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f532419cf40 con 0x7f53241029d0 2024-04-21T23:18:31.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.619+0000 7f6b233e0640 1 Processor -- start 2024-04-21T23:18:31.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.619+0000 7f6b233e0640 1 -- start start 2024-04-21T23:18:31.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.619+0000 7f5323fff640 1 --2- 172.21.15.40:0/2332982767 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53241029d0 0x7f5324197fa0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f5314009a20 tx=0x7f531402dd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:31.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.619+0000 7f53217fa640 1 -- 172.21.15.40:0/2332982767 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5314043020 con 0x7f53241029d0 2024-04-21T23:18:31.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.619+0000 7f6b233e0640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b1c100610 0x7f6b1c197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:31.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.619+0000 7f6b233e0640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6b1c100fd0 0x7f6b1c1984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:31.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.619+0000 7f53217fa640 1 -- 172.21.15.40:0/2332982767 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53140054e0 con 0x7f53241029d0 2024-04-21T23:18:31.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.619+0000 7f6b233e0640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b1c1066b0 0x7f6b1c19c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:31.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.619+0000 7f6b233e0640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6b1c110a30 con 0x7f6b1c100610 2024-04-21T23:18:31.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.619+0000 7f6b233e0640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f6b1c1108b0 con 0x7f6b1c100fd0 2024-04-21T23:18:31.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.619+0000 7f6b233e0640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6b1c110bb0 con 0x7f6b1c1066b0 2024-04-21T23:18:31.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.619+0000 7f6b2117c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b1c100610 0x7f6b1c197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:31.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.619+0000 7f532a6c2640 1 -- 172.21.15.40:0/2332982767 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f532419d1d0 con 0x7f53241029d0 2024-04-21T23:18:31.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.619+0000 7f53217fa640 1 -- 172.21.15.40:0/2332982767 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5314003bd0 con 0x7f53241029d0 2024-04-21T23:18:31.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.619+0000 7f6b2117c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b1c100610 0x7f6b1c197f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32984/0 (socket says 172.21.15.40:32984) 2024-04-21T23:18:31.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.619+0000 7f6b2117c640 1 -- 172.21.15.40:0/3598297006 learned_addr learned my addr 172.21.15.40:0/3598297006 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:31.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.619+0000 7f6b2097b640 1 --2- 172.21.15.40:0/3598297006 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6b1c100fd0 0x7f6b1c1984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:31.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.619+0000 7f532a6c2640 1 -- 172.21.15.40:0/2332982767 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5324100550 con 0x7f53241029d0 2024-04-21T23:18:31.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.620+0000 7f6b2117c640 1 -- 172.21.15.40:0/3598297006 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b1c1066b0 msgr2=0x7f6b1c19c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:31.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.620+0000 7f6b2117c640 1 --2- 172.21.15.40:0/3598297006 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b1c1066b0 0x7f6b1c19c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.620+0000 7f6b2117c640 1 -- 172.21.15.40:0/3598297006 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6b1c100fd0 msgr2=0x7f6b1c1984c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:31.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.620+0000 7f6b2117c640 1 --2- 172.21.15.40:0/3598297006 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6b1c100fd0 0x7f6b1c1984c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.620+0000 7f6b2117c640 1 -- 172.21.15.40:0/3598297006 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b1c19cf70 con 0x7f6b1c100610 2024-04-21T23:18:31.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.620+0000 7f6b2097b640 1 --2- 172.21.15.40:0/3598297006 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6b1c100fd0 0x7f6b1c1984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:18:31.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.620+0000 7f6b2117c640 1 --2- 172.21.15.40:0/3598297006 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b1c100610 0x7f6b1c197f80 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f6b18009230 tx=0x7f6b1802cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:31.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.620+0000 7f6b127fc640 1 -- 172.21.15.40:0/3598297006 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b18041070 con 0x7f6b1c100610 2024-04-21T23:18:31.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.620+0000 7f6b233e0640 1 -- 172.21.15.40:0/3598297006 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b1c19d200 con 0x7f6b1c100610 2024-04-21T23:18:31.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.620+0000 7f6b127fc640 1 -- 172.21.15.40:0/3598297006 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b18004450 con 0x7f6b1c100610 2024-04-21T23:18:31.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.620+0000 7f6b127fc640 1 -- 172.21.15.40:0/3598297006 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b18003e50 con 0x7f6b1c100610 2024-04-21T23:18:31.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.620+0000 7f6b233e0640 1 -- 172.21.15.40:0/3598297006 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b1c1a4af0 con 0x7f6b1c100610 2024-04-21T23:18:31.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.621+0000 7f532a6c2640 1 -- 172.21.15.40:0/2332982767 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f532404f500 con 0x7f53241029d0 2024-04-21T23:18:31.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.622+0000 7f53217fa640 1 -- 172.21.15.40:0/2332982767 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7f5314046020 con 0x7f53241029d0 2024-04-21T23:18:31.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.623+0000 7f53217fa640 1 --2- 172.21.15.40:0/2332982767 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f530c0a7350 0x7f530c0a9810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:31.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.623+0000 7f53237fe640 1 --2- 172.21.15.40:0/2332982767 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f530c0a7350 0x7f530c0a9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:31.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.623+0000 7f53217fa640 1 -- 172.21.15.40:0/2332982767 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f53140f1ec0 con 0x7f53241029d0 2024-04-21T23:18:31.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.623+0000 7f53237fe640 1 --2- 172.21.15.40:0/2332982767 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f530c0a7350 0x7f530c0a9810 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5324100010 tx=0x7f531800b410 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:31.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.624+0000 7f6b127fc640 1 -- 172.21.15.40:0/3598297006 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7f6b18044070 con 0x7f6b1c100610 2024-04-21T23:18:31.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.624+0000 7f6b25fc5740 1 Processor -- start 2024-04-21T23:18:31.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.624+0000 7f6b25fc5740 1 -- start start 2024-04-21T23:18:31.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.624+0000 7f6b25fc5740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e9eb270720 0x55e9eb26f470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:31.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.624+0000 7f6b25fc5740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e9eb26fa40 0x55e9eb26fec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:31.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.624+0000 7f6b25fc5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e9eb278db0 0x55e9eb27b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:31.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.624+0000 7f6b25fc5740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55e9eb1d3170 con 0x55e9eb26fa40 2024-04-21T23:18:31.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.624+0000 7f6b25fc5740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55e9eb17b7c0 con 0x55e9eb270720 2024-04-21T23:18:31.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.624+0000 7f6b25fc5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55e9eb1cf130 con 0x55e9eb278db0 2024-04-21T23:18:31.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.624+0000 7f6b2117c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e9eb26fa40 0x55e9eb26fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:31.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.624+0000 7f6b2117c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e9eb26fa40 0x55e9eb26fec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32986/0 (socket says 172.21.15.40:32986) 2024-04-21T23:18:31.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.624+0000 7f6b2117c640 1 -- 172.21.15.40:0/652421857 learned_addr learned my addr 172.21.15.40:0/652421857 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:31.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.624+0000 7f6b2097b640 1 --2- 172.21.15.40:0/652421857 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e9eb278db0 0x55e9eb27b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:31.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.624+0000 7f6b127fc640 1 --2- 172.21.15.40:0/3598297006 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f6b080a2fc0 0x7f6b080a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:31.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.624+0000 7f6b2197d640 1 --2- 172.21.15.40:0/652421857 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e9eb270720 0x55e9eb26f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:31.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.625+0000 7f6b2117c640 1 -- 172.21.15.40:0/652421857 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e9eb278db0 msgr2=0x55e9eb27b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:31.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.625+0000 7f6b2117c640 1 --2- 172.21.15.40:0/652421857 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e9eb278db0 0x55e9eb27b1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.625+0000 7f6b2117c640 1 -- 172.21.15.40:0/652421857 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e9eb270720 msgr2=0x55e9eb26f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:31.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.625+0000 7f6b2117c640 1 --2- 172.21.15.40:0/652421857 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e9eb270720 0x55e9eb26f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.625+0000 7f6b2117c640 1 -- 172.21.15.40:0/652421857 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e9eb18ce60 con 0x55e9eb26fa40 2024-04-21T23:18:31.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.625+0000 7f6b2197d640 1 --2- 172.21.15.40:0/3598297006 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f6b080a2fc0 0x7f6b080a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:31.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.625+0000 7f6b127fc640 1 -- 172.21.15.40:0/3598297006 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f6b180f1590 con 0x7f6b1c100610 2024-04-21T23:18:31.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.625+0000 7f6b2117c640 1 --2- 172.21.15.40:0/652421857 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e9eb26fa40 0x55e9eb26fec0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f6b180ee9d0 tx=0x7f6b18004ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bedfed3dc3241ee7 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:31.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.625+0000 7f6b2197d640 1 --2- 172.21.15.40:0/3598297006 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f6b080a2fc0 0x7f6b080a5480 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f6b14009270 tx=0x7f6b14004a40 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:31.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.628+0000 7f6b077fe640 1 -- 172.21.15.40:0/652421857 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b180bd100 con 0x55e9eb26fa40 2024-04-21T23:18:31.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.628+0000 7f53217fa640 1 -- 172.21.15.40:0/2332982767 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f531404d020 con 0x7f53241029d0 2024-04-21T23:18:31.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.628+0000 7f6b077fe640 1 -- 172.21.15.40:0/652421857 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b180bdcc0 con 0x55e9eb26fa40 2024-04-21T23:18:31.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.628+0000 7f6b077fe640 1 -- 172.21.15.40:0/652421857 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b180bcf20 con 0x55e9eb26fa40 2024-04-21T23:18:31.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.629+0000 7f6b25fc5740 1 -- 172.21.15.40:0/652421857 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e9eb26fa40 msgr2=0x55e9eb26fec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:31.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.629+0000 7f6b25fc5740 1 --2- 172.21.15.40:0/652421857 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e9eb26fa40 0x55e9eb26fec0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f6b180ee9d0 tx=0x7f6b18004ee0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.629+0000 7f6b25fc5740 1 -- 172.21.15.40:0/652421857 shutdown_connections 2024-04-21T23:18:31.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.629+0000 7f6b25fc5740 1 --2- 172.21.15.40:0/652421857 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e9eb278db0 0x55e9eb27b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.629+0000 7f6b25fc5740 1 --2- 172.21.15.40:0/652421857 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e9eb26fa40 0x55e9eb26fec0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.629+0000 7f6b25fc5740 1 --2- 172.21.15.40:0/652421857 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e9eb270720 0x55e9eb26f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.629+0000 7f6b25fc5740 1 -- 172.21.15.40:0/652421857 >> 172.21.15.40:0/652421857 conn(0x55e9eb2364f0 msgr2=0x55e9eb26a490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:31.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.629+0000 7f6b25fc5740 1 -- 172.21.15.40:0/652421857 shutdown_connections 2024-04-21T23:18:31.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.629+0000 7f6b25fc5740 1 -- 172.21.15.40:0/652421857 wait complete. 2024-04-21T23:18:31.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.629+0000 7f6b25fc5740 1 -- start start 2024-04-21T23:18:31.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.629+0000 7f6b25fc5740 1 Processor -- start 2024-04-21T23:18:31.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.630+0000 7f6b25fc5740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e9eb270720 0x55e9eb275d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:31.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.630+0000 7f6b25fc5740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e9eb278db0 0x55e9eb276240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:31.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.630+0000 7f6b25fc5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e9eb276780 0x55e9eb284dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:31.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.630+0000 7f6b25fc5740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55e9eb27b9e0 con 0x55e9eb270720 2024-04-21T23:18:31.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.630+0000 7f6b25fc5740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55e9eb18ed40 con 0x55e9eb278db0 2024-04-21T23:18:31.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.630+0000 7f6b25fc5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55e9eb27bb60 con 0x55e9eb276780 2024-04-21T23:18:31.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.630+0000 7f6b25fc5740 10 client.0 _ll_register_callbacks cb 0x55e9eb1ca820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:18:31.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.630+0000 7f6b2117c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e9eb270720 0x55e9eb275d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:31.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.630+0000 7f6b2117c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e9eb270720 0x55e9eb275d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32998/0 (socket says 172.21.15.40:32998) 2024-04-21T23:18:31.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.630+0000 7f6b2117c640 1 -- 172.21.15.40:0/3623965178 learned_addr learned my addr 172.21.15.40:0/3623965178 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:31.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.630+0000 7f6b2197d640 1 --2- 172.21.15.40:0/3623965178 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e9eb276780 0x55e9eb284dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:31.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.630+0000 7f6b2097b640 1 --2- 172.21.15.40:0/3623965178 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e9eb278db0 0x55e9eb276240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:31.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.630+0000 7f6b2117c640 1 -- 172.21.15.40:0/3623965178 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e9eb276780 msgr2=0x55e9eb284dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:31.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.630+0000 7f6b2117c640 1 --2- 172.21.15.40:0/3623965178 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e9eb276780 0x55e9eb284dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.630+0000 7f6b2117c640 1 -- 172.21.15.40:0/3623965178 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e9eb278db0 msgr2=0x55e9eb276240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:31.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.630+0000 7f6b2117c640 1 --2- 172.21.15.40:0/3623965178 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e9eb278db0 0x55e9eb276240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.630+0000 7f6b2117c640 1 -- 172.21.15.40:0/3623965178 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e9eafaf7e0 con 0x55e9eb270720 2024-04-21T23:18:31.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.630+0000 7f6b2117c640 1 --2- 172.21.15.40:0/3623965178 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e9eb270720 0x55e9eb275d00 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f6b180bde70 tx=0x7f6b18009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f11f822cd2d5d4b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:31.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.630+0000 7f6b04ff9640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-21T23:18:31.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.630+0000 7f6b04ff9640 1 -- 172.21.15.40:0/3623965178 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b180bcf20 con 0x55e9eb270720 2024-04-21T23:18:31.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.631+0000 7f6b04ff9640 1 -- 172.21.15.40:0/3623965178 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f6b180d17a0 con 0x55e9eb270720 2024-04-21T23:18:31.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.631+0000 7f6b25fc5740 1 -- 172.21.15.40:0/3623965178 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6b180bdcc0 con 0x55e9eb270720 2024-04-21T23:18:31.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.631+0000 7f6b04ff9640 1 -- 172.21.15.40:0/3623965178 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f6b180bd100 con 0x55e9eb270720 2024-04-21T23:18:31.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.631+0000 7f6b25fc5740 10 client.15681 fetch_fsmap learned FSMap version 25 2024-04-21T23:18:31.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.631+0000 7f6b25fc5740 1 -- 172.21.15.40:0/3623965178 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55e9eb00ff70 con 0x55e9eb270720 2024-04-21T23:18:31.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.631+0000 7f6b04ff9640 1 -- 172.21.15.40:0/3623965178 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b180bdcc0 con 0x55e9eb270720 2024-04-21T23:18:31.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.631+0000 7f6b04ff9640 1 -- 172.21.15.40:0/3623965178 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f6b18097ce0 con 0x55e9eb270720 2024-04-21T23:18:31.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.631+0000 7f6b25fc5740 10 client.15681 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:18:31.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.631+0000 7f6b25fc5740 10 client.15681 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:18:31.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.631+0000 7f6b25fc5740 20 client.15681 populate_metadata read hostname 'smithi040' 2024-04-21T23:18:31.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.631+0000 7f6b25fc5740 1 --2- 172.21.15.40:0/3623965178 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55e9eb288e00 0x55e9eb28b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:31.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.631+0000 7f6b25fc5740 4 client.15681 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:18:31.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.631+0000 7f6b25fc5740 1 -- 172.21.15.40:0/3623965178 --> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e9eb1a7450 con 0x55e9eb288e00 2024-04-21T23:18:31.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.631+0000 7f6b2097b640 1 --2- 172.21.15.40:0/3623965178 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55e9eb288e00 0x55e9eb28b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:31.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.632+0000 7f6b2097b640 1 --2- 172.21.15.40:0/3623965178 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55e9eb288e00 0x55e9eb28b1f0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=24416163efbcfb65 server_cookie=5b6ca54e86005891 in_seq=0 out_seq=0 2024-04-21T23:18:31.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.632+0000 7f6b04ff9640 10 client.15681 ms_handle_connect on v2:172.21.15.111:6838/3712235548 2024-04-21T23:18:31.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.633+0000 7f6b04ff9640 1 -- 172.21.15.40:0/3623965178 <== mds.0 v2:172.21.15.111:6838/3712235548 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e9eb1a7450 con 0x55e9eb288e00 2024-04-21T23:18:31.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.633+0000 7f6b04ff9640 10 client.15681 handle_command_reply: tid=0 2024-04-21T23:18:31.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.633+0000 7f6b25fc5740 1 client.15681 shutdown 2024-04-21T23:18:31.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.633+0000 7f6b25fc5740 1 -- 172.21.15.40:0/3623965178 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e9eb270720 msgr2=0x55e9eb275d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:31.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.633+0000 7f6b25fc5740 1 --2- 172.21.15.40:0/3623965178 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e9eb270720 0x55e9eb275d00 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f6b180bde70 tx=0x7f6b18009260 comp rx=0 tx=0).stop 2024-04-21T23:18:31.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.633+0000 7f6b25fc5740 1 -- 172.21.15.40:0/3623965178 shutdown_connections 2024-04-21T23:18:31.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.633+0000 7f6b25fc5740 1 --2- 172.21.15.40:0/3623965178 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55e9eb288e00 0x55e9eb28b1f0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.633+0000 7f6b25fc5740 1 --2- 172.21.15.40:0/3623965178 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e9eb278db0 0x55e9eb276240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.633+0000 7f6b04ff9640 0 client.15681 ms_handle_reset on v2:172.21.15.111:6838/3712235548 2024-04-21T23:18:31.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.633+0000 7f6b25fc5740 1 --2- 172.21.15.40:0/3623965178 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e9eb276780 0x55e9eb284dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.633+0000 7f6b25fc5740 1 --2- 172.21.15.40:0/3623965178 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e9eb270720 0x55e9eb275d00 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.633+0000 7f6b25fc5740 1 -- 172.21.15.40:0/3623965178 >> 172.21.15.40:0/3623965178 conn(0x55e9eb2364f0 msgr2=0x55e9eb26c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:31.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.633+0000 7f6b25fc5740 1 -- 172.21.15.40:0/3623965178 shutdown_connections 2024-04-21T23:18:31.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.633+0000 7f6b25fc5740 1 -- 172.21.15.40:0/3623965178 wait complete. 2024-04-21T23:18:31.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.633+0000 7f6b25fc5740 20 client.15681 trim_cache size 0 max 16384 2024-04-21T23:18:31.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.651+0000 7f6b25fc5740 1 Processor -- start 2024-04-21T23:18:31.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.651+0000 7f6b25fc5740 1 -- start start 2024-04-21T23:18:31.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.651+0000 7f6b25fc5740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e9eb270720 0x55e9eb269370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:31.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.651+0000 7f6b25fc5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e9eb276780 0x55e9eb2909d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:31.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.651+0000 7f6b25fc5740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e9eb278db0 0x55e9eb290f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:31.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.651+0000 7f6b25fc5740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6b1c1b1490 con 0x55e9eb270720 2024-04-21T23:18:31.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.651+0000 7f6b25fc5740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f6b1c1aed60 con 0x55e9eb278db0 2024-04-21T23:18:31.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.651+0000 7f6b25fc5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6b1c1acbb0 con 0x55e9eb276780 2024-04-21T23:18:31.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.651+0000 7f6b2117c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e9eb270720 0x55e9eb269370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:31.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.651+0000 7f6b2117c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e9eb270720 0x55e9eb269370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33004/0 (socket says 172.21.15.40:33004) 2024-04-21T23:18:31.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.651+0000 7f6b2117c640 1 -- 172.21.15.40:0/3986987614 learned_addr learned my addr 172.21.15.40:0/3986987614 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:31.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.651+0000 7f6b2097b640 1 --2- 172.21.15.40:0/3986987614 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e9eb276780 0x55e9eb2909d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:31.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.651+0000 7f6b2117c640 1 -- 172.21.15.40:0/3986987614 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e9eb276780 msgr2=0x55e9eb2909d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:31.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.651+0000 7f6b2117c640 1 --2- 172.21.15.40:0/3986987614 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e9eb276780 0x55e9eb2909d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.651+0000 7f6b2117c640 1 -- 172.21.15.40:0/3986987614 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e9eb278db0 msgr2=0x55e9eb290f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:31.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.651+0000 7f6b2117c640 1 --2- 172.21.15.40:0/3986987614 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e9eb278db0 0x55e9eb290f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.651+0000 7f6b2117c640 1 -- 172.21.15.40:0/3986987614 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e9eaea7170 con 0x55e9eb270720 2024-04-21T23:18:31.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.652+0000 7f6b2117c640 1 --2- 172.21.15.40:0/3986987614 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e9eb270720 0x55e9eb269370 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f6b180b9f90 tx=0x7f6b180f1890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e9f40b898292a9b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:31.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.652+0000 7f6b06ffd640 1 -- 172.21.15.40:0/3986987614 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b180bcf20 con 0x55e9eb270720 2024-04-21T23:18:31.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.652+0000 7f6b06ffd640 1 -- 172.21.15.40:0/3986987614 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b180bdcc0 con 0x55e9eb270720 2024-04-21T23:18:31.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.652+0000 7f6b06ffd640 1 -- 172.21.15.40:0/3986987614 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b180bd390 con 0x55e9eb270720 2024-04-21T23:18:31.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.652+0000 7f6b25fc5740 1 -- 172.21.15.40:0/3986987614 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e9eb270720 msgr2=0x55e9eb269370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:31.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.652+0000 7f6b25fc5740 1 --2- 172.21.15.40:0/3986987614 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e9eb270720 0x55e9eb269370 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f6b180b9f90 tx=0x7f6b180f1890 comp rx=0 tx=0).stop 2024-04-21T23:18:31.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.653+0000 7f6b25fc5740 1 -- 172.21.15.40:0/3986987614 shutdown_connections 2024-04-21T23:18:31.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.653+0000 7f6b25fc5740 1 --2- 172.21.15.40:0/3986987614 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e9eb278db0 0x55e9eb290f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.653+0000 7f6b25fc5740 1 --2- 172.21.15.40:0/3986987614 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e9eb276780 0x55e9eb2909d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.653+0000 7f6b25fc5740 1 --2- 172.21.15.40:0/3986987614 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e9eb270720 0x55e9eb269370 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.653+0000 7f6b25fc5740 1 -- 172.21.15.40:0/3986987614 >> 172.21.15.40:0/3986987614 conn(0x55e9eb2364f0 msgr2=0x55e9eb268c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:31.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.653+0000 7f6b25fc5740 1 -- 172.21.15.40:0/3986987614 shutdown_connections 2024-04-21T23:18:31.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.653+0000 7f6b25fc5740 1 -- 172.21.15.40:0/3986987614 wait complete. 2024-04-21T23:18:31.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.653+0000 7f6b25fc5740 1 -- start start 2024-04-21T23:18:31.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.653+0000 7f6b25fc5740 1 Processor -- start 2024-04-21T23:18:31.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.653+0000 7f6b25fc5740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e9eb270720 0x55e9eb280ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:31.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.653+0000 7f6b25fc5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e9eb276780 0x55e9eb27dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:31.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.653+0000 7f6b25fc5740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e9eb278db0 0x55e9eb27e200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:31.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.653+0000 7f6b25fc5740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6b1c1ad220 con 0x55e9eb270720 2024-04-21T23:18:31.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.653+0000 7f6b25fc5740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f6b1c1ae690 con 0x55e9eb278db0 2024-04-21T23:18:31.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.653+0000 7f6b25fc5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6b1c1ad8f0 con 0x55e9eb276780 2024-04-21T23:18:31.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.653+0000 7f6b25fc5740 10 client.0 _ll_register_callbacks cb 0x7f6b0c00bae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:18:31.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.653+0000 7f6b2117c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e9eb270720 0x55e9eb280ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:31.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.653+0000 7f6b2117c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e9eb270720 0x55e9eb280ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33020/0 (socket says 172.21.15.40:33020) 2024-04-21T23:18:31.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.653+0000 7f6b2117c640 1 -- 172.21.15.40:0/328277513 learned_addr learned my addr 172.21.15.40:0/328277513 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:31.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.654+0000 7f6b2197d640 1 --2- 172.21.15.40:0/328277513 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e9eb278db0 0x55e9eb27e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:31.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.654+0000 7f6b2097b640 1 --2- 172.21.15.40:0/328277513 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e9eb276780 0x55e9eb27dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:31.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.654+0000 7f6b2117c640 1 -- 172.21.15.40:0/328277513 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e9eb276780 msgr2=0x55e9eb27dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:31.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.654+0000 7f6b2117c640 1 --2- 172.21.15.40:0/328277513 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e9eb276780 0x55e9eb27dcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.654+0000 7f6b2117c640 1 -- 172.21.15.40:0/328277513 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e9eb278db0 msgr2=0x55e9eb27e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:31.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.654+0000 7f6b2117c640 1 --2- 172.21.15.40:0/328277513 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e9eb278db0 0x55e9eb27e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.654+0000 7f6b2117c640 1 -- 172.21.15.40:0/328277513 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e9eb27e740 con 0x55e9eb270720 2024-04-21T23:18:31.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.654+0000 7f6b2197d640 1 --2- 172.21.15.40:0/328277513 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e9eb278db0 0x55e9eb27e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:18:31.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.654+0000 7f6b2117c640 1 --2- 172.21.15.40:0/328277513 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e9eb270720 0x55e9eb280ee0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f6b18006010 tx=0x7f6b18004ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ba8db860726bd69 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:31.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.654+0000 7f6b04ff9640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-21T23:18:31.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.654+0000 7f6b04ff9640 1 -- 172.21.15.40:0/328277513 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b180bd390 con 0x55e9eb270720 2024-04-21T23:18:31.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.654+0000 7f6b04ff9640 1 -- 172.21.15.40:0/328277513 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f6b180bd820 con 0x55e9eb270720 2024-04-21T23:18:31.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.654+0000 7f6b25fc5740 1 -- 172.21.15.40:0/328277513 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6b180bdcc0 con 0x55e9eb270720 2024-04-21T23:18:31.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.654+0000 7f6b04ff9640 1 -- 172.21.15.40:0/328277513 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f6b180bcf20 con 0x55e9eb270720 2024-04-21T23:18:31.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.654+0000 7f6b25fc5740 10 client.15687 fetch_fsmap learned FSMap version 25 2024-04-21T23:18:31.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.654+0000 7f6b25fc5740 1 -- 172.21.15.40:0/328277513 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55e9eb27e740 con 0x55e9eb270720 2024-04-21T23:18:31.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.654+0000 7f6b04ff9640 1 -- 172.21.15.40:0/328277513 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b180bdcc0 con 0x55e9eb270720 2024-04-21T23:18:31.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.655+0000 7f6b04ff9640 1 -- 172.21.15.40:0/328277513 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f6b180f3b60 con 0x55e9eb270720 2024-04-21T23:18:31.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.655+0000 7f6b25fc5740 10 client.15687 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:18:31.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.655+0000 7f6b25fc5740 10 client.15687 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:18:31.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.655+0000 7f6b25fc5740 20 client.15687 populate_metadata read hostname 'smithi040' 2024-04-21T23:18:31.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.655+0000 7f6b25fc5740 1 --2- 172.21.15.40:0/328277513 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55e9eb2f02a0 0x55e9eb2f2690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:31.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.655+0000 7f6b25fc5740 4 client.15687 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T23:18:31.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.655+0000 7f6b25fc5740 1 -- 172.21.15.40:0/328277513 --> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55e9eb1e06b0 con 0x55e9eb2f02a0 2024-04-21T23:18:31.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.655+0000 7f6b2097b640 1 --2- 172.21.15.40:0/328277513 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55e9eb2f02a0 0x55e9eb2f2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:31.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.655+0000 7f6b04ff9640 10 client.15687 ms_handle_connect on v2:172.21.15.111:6838/3712235548 2024-04-21T23:18:31.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.655+0000 7f6b2097b640 1 --2- 172.21.15.40:0/328277513 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55e9eb2f02a0 0x55e9eb2f2690 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc525737d27250ac server_cookie=8e7638d92d45d766 in_seq=0 out_seq=0 2024-04-21T23:18:31.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.657+0000 7f6b04ff9640 1 -- 172.21.15.40:0/328277513 <== mds.0 v2:172.21.15.111:6838/3712235548 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55e9eb1e06b0 con 0x55e9eb2f02a0 2024-04-21T23:18:31.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.657+0000 7f6b04ff9640 10 client.15687 handle_command_reply: tid=0 2024-04-21T23:18:31.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.657+0000 7f6b25fc5740 1 client.15687 shutdown 2024-04-21T23:18:31.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.657+0000 7f6b25fc5740 1 -- 172.21.15.40:0/328277513 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e9eb270720 msgr2=0x55e9eb280ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:31.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.657+0000 7f6b25fc5740 1 --2- 172.21.15.40:0/328277513 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e9eb270720 0x55e9eb280ee0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f6b18006010 tx=0x7f6b18004ee0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.657+0000 7f6b25fc5740 1 -- 172.21.15.40:0/328277513 shutdown_connections 2024-04-21T23:18:31.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.657+0000 7f6b25fc5740 1 --2- 172.21.15.40:0/328277513 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55e9eb2f02a0 0x55e9eb2f2690 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.657+0000 7f6b25fc5740 1 --2- 172.21.15.40:0/328277513 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e9eb278db0 0x55e9eb27e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.657+0000 7f6b25fc5740 1 --2- 172.21.15.40:0/328277513 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e9eb276780 0x55e9eb27dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.657+0000 7f6b25fc5740 1 --2- 172.21.15.40:0/328277513 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e9eb270720 0x55e9eb280ee0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.657+0000 7f6b25fc5740 1 -- 172.21.15.40:0/328277513 >> 172.21.15.40:0/328277513 conn(0x55e9eb2364f0 msgr2=0x55e9eb2694f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:31.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.657+0000 7f6b04ff9640 0 client.15687 ms_handle_reset on v2:172.21.15.111:6838/3712235548 2024-04-21T23:18:31.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.657+0000 7f6b25fc5740 1 -- 172.21.15.40:0/328277513 shutdown_connections 2024-04-21T23:18:31.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.657+0000 7f6b25fc5740 1 -- 172.21.15.40:0/328277513 wait complete. 2024-04-21T23:18:31.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.657+0000 7f6b25fc5740 20 client.15687 trim_cache size 0 max 16384 2024-04-21T23:18:31.661 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-21T23:18:31.661 INFO:teuthology.orchestra.run.smithi040.stdout: "return_code": 0, 2024-04-21T23:18:31.661 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_tag": "93a7401c-e7f6-410e-b5a8-413666fa5f20", 2024-04-21T23:18:31.661 INFO:teuthology.orchestra.run.smithi040.stdout: "mode": "asynchronous" 2024-04-21T23:18:31.662 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-21T23:18:31.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.658+0000 7f6b233e0640 1 -- 172.21.15.40:0/3598297006 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f6b080a2fc0 msgr2=0x7f6b080a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:31.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.658+0000 7f6b233e0640 1 --2- 172.21.15.40:0/3598297006 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f6b080a2fc0 0x7f6b080a5480 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f6b14009270 tx=0x7f6b14004a40 comp rx=0 tx=0).stop 2024-04-21T23:18:31.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.658+0000 7f6b233e0640 1 -- 172.21.15.40:0/3598297006 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b1c100610 msgr2=0x7f6b1c197f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:31.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.658+0000 7f6b233e0640 1 --2- 172.21.15.40:0/3598297006 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b1c100610 0x7f6b1c197f80 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f6b18009230 tx=0x7f6b1802cbf0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.658+0000 7f6b233e0640 1 -- 172.21.15.40:0/3598297006 shutdown_connections 2024-04-21T23:18:31.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.658+0000 7f6b233e0640 1 --2- 172.21.15.40:0/3598297006 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f6b080a2fc0 0x7f6b080a5480 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.658+0000 7f6b233e0640 1 --2- 172.21.15.40:0/3598297006 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6b1c100fd0 0x7f6b1c1984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.658+0000 7f6b233e0640 1 --2- 172.21.15.40:0/3598297006 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b1c1066b0 0x7f6b1c19c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.658+0000 7f6b233e0640 1 --2- 172.21.15.40:0/3598297006 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b1c100610 0x7f6b1c197f80 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.658+0000 7f6b233e0640 1 -- 172.21.15.40:0/3598297006 >> 172.21.15.40:0/3598297006 conn(0x7f6b1c077740 msgr2=0x7f6b1c104790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:31.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.658+0000 7f6b233e0640 1 -- 172.21.15.40:0/3598297006 shutdown_connections 2024-04-21T23:18:31.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.658+0000 7f6b233e0640 1 -- 172.21.15.40:0/3598297006 wait complete. 2024-04-21T23:18:31.668 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T23:18:31.668 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T23:18:31.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.774+0000 7f58863c8640 1 -- 172.21.15.40:0/153207252 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f588010bd00 msgr2=0x7f588010e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:31.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.774+0000 7f58863c8640 1 --2- 172.21.15.40:0/153207252 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f588010bd00 0x7f588010e1a0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f5878009ca0 tx=0x7f587802c230 comp rx=0 tx=0).stop 2024-04-21T23:18:31.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.774+0000 7f58863c8640 1 -- 172.21.15.40:0/153207252 shutdown_connections 2024-04-21T23:18:31.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.774+0000 7f58863c8640 1 --2- 172.21.15.40:0/153207252 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f58801032e0 0x7f588010b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.774+0000 7f58863c8640 1 --2- 172.21.15.40:0/153207252 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f588010bd00 0x7f588010e1a0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.774+0000 7f58863c8640 1 --2- 172.21.15.40:0/153207252 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f58801029c0 0x7f5880102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.774+0000 7f58863c8640 1 -- 172.21.15.40:0/153207252 >> 172.21.15.40:0/153207252 conn(0x7f5880077740 msgr2=0x7f5880077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:31.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.774+0000 7f58863c8640 1 -- 172.21.15.40:0/153207252 shutdown_connections 2024-04-21T23:18:31.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.774+0000 7f58863c8640 1 -- 172.21.15.40:0/153207252 wait complete. 2024-04-21T23:18:31.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.774+0000 7f58863c8640 1 Processor -- start 2024-04-21T23:18:31.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.774+0000 7f58863c8640 1 -- start start 2024-04-21T23:18:31.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.775+0000 7f58863c8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f58801029c0 0x7f5880100610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:31.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.775+0000 7f58863c8640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58801032e0 0x7f5880100b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:31.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.775+0000 7f58863c8640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f588010bd00 0x7f5880101090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:31.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.775+0000 7f58863c8640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5880111b00 con 0x7f58801032e0 2024-04-21T23:18:31.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.775+0000 7f58863c8640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f5880111980 con 0x7f588010bd00 2024-04-21T23:18:31.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.775+0000 7f58863c8640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5880111c80 con 0x7f58801029c0 2024-04-21T23:18:31.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.775+0000 7f587f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58801032e0 0x7f5880100b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:31.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.775+0000 7f587f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58801032e0 0x7f5880100b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51132/0 (socket says 172.21.15.40:51132) 2024-04-21T23:18:31.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.775+0000 7f587f7fe640 1 -- 172.21.15.40:0/3164480273 learned_addr learned my addr 172.21.15.40:0/3164480273 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:31.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.775+0000 7f587ffff640 1 --2- 172.21.15.40:0/3164480273 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f58801029c0 0x7f5880100610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:31.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.775+0000 7f5884965640 1 --2- 172.21.15.40:0/3164480273 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f588010bd00 0x7f5880101090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:31.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.775+0000 7f587f7fe640 1 -- 172.21.15.40:0/3164480273 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f58801029c0 msgr2=0x7f5880100610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:31.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.775+0000 7f587f7fe640 1 --2- 172.21.15.40:0/3164480273 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f58801029c0 0x7f5880100610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.775+0000 7f587f7fe640 1 -- 172.21.15.40:0/3164480273 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f588010bd00 msgr2=0x7f5880101090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:31.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.775+0000 7f587f7fe640 1 --2- 172.21.15.40:0/3164480273 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f588010bd00 0x7f5880101090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.775+0000 7f587f7fe640 1 -- 172.21.15.40:0/3164480273 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58801a64a0 con 0x7f58801032e0 2024-04-21T23:18:31.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.775+0000 7f587ffff640 1 --2- 172.21.15.40:0/3164480273 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f58801029c0 0x7f5880100610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:18:31.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.775+0000 7f587f7fe640 1 --2- 172.21.15.40:0/3164480273 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58801032e0 0x7f5880100b50 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f587000d950 tx=0x7f587000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:31.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.776+0000 7f587d7fa640 1 -- 172.21.15.40:0/3164480273 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5870017030 con 0x7f58801032e0 2024-04-21T23:18:31.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.776+0000 7f58863c8640 1 -- 172.21.15.40:0/3164480273 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58801a6790 con 0x7f58801032e0 2024-04-21T23:18:31.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.776+0000 7f587d7fa640 1 -- 172.21.15.40:0/3164480273 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5870004020 con 0x7f58801032e0 2024-04-21T23:18:31.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.776+0000 7f587d7fa640 1 -- 172.21.15.40:0/3164480273 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58700047a0 con 0x7f58801032e0 2024-04-21T23:18:31.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.776+0000 7f58863c8640 1 -- 172.21.15.40:0/3164480273 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58801a6ce0 con 0x7f58801032e0 2024-04-21T23:18:31.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.778+0000 7f587d7fa640 1 -- 172.21.15.40:0/3164480273 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7f58700041d0 con 0x7f58801032e0 2024-04-21T23:18:31.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.778+0000 7f587d7fa640 1 --2- 172.21.15.40:0/3164480273 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f586c0a2e50 0x7f586c0a5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:31.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.778+0000 7f587ffff640 1 --2- 172.21.15.40:0/3164480273 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f586c0a2e50 0x7f586c0a5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:31.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.778+0000 7f587d7fa640 1 -- 172.21.15.40:0/3164480273 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f58700cf4b0 con 0x7f58801032e0 2024-04-21T23:18:31.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.779+0000 7f587ffff640 1 --2- 172.21.15.40:0/3164480273 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f586c0a2e50 0x7f586c0a5310 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5868009c20 tx=0x7f5868005f90 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:31.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.779+0000 7f5888fa8740 1 Processor -- start 2024-04-21T23:18:31.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.779+0000 7f5888fa8740 1 -- start start 2024-04-21T23:18:31.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.779+0000 7f5888fa8740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555aed6fcf30 0x555aed6fd310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:31.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.779+0000 7f5888fa8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555aed6fd850 0x555aed705ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:31.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.779+0000 7f5888fa8740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555aed706430 0x555aed708820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:31.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.779+0000 7f5888fa8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x555aed661560 con 0x555aed6fd850 2024-04-21T23:18:31.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.779+0000 7f5888fa8740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x555aed609730 con 0x555aed706430 2024-04-21T23:18:31.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.779+0000 7f5888fa8740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x555aed65d520 con 0x555aed6fcf30 2024-04-21T23:18:31.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.780+0000 7f587f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555aed6fd850 0x555aed705ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:31.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.780+0000 7f587f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555aed6fd850 0x555aed705ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51136/0 (socket says 172.21.15.40:51136) 2024-04-21T23:18:31.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.780+0000 7f587f7fe640 1 -- 172.21.15.40:0/928265207 learned_addr learned my addr 172.21.15.40:0/928265207 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:31.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.780+0000 7f587ffff640 1 --2- 172.21.15.40:0/928265207 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555aed6fcf30 0x555aed6fd310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:31.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.780+0000 7f587f7fe640 1 -- 172.21.15.40:0/928265207 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555aed6fcf30 msgr2=0x555aed6fd310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:31.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.780+0000 7f587f7fe640 1 --2- 172.21.15.40:0/928265207 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555aed6fcf30 0x555aed6fd310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.780+0000 7f5884965640 1 --2- 172.21.15.40:0/928265207 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555aed706430 0x555aed708820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:31.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.780+0000 7f587f7fe640 1 -- 172.21.15.40:0/928265207 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555aed706430 msgr2=0x555aed708820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:31.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.780+0000 7f587f7fe640 1 --2- 172.21.15.40:0/928265207 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555aed706430 0x555aed708820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.780+0000 7f587f7fe640 1 -- 172.21.15.40:0/928265207 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555aed43f7d0 con 0x555aed6fd850 2024-04-21T23:18:31.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.780+0000 7f5884965640 1 --2- 172.21.15.40:0/928265207 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555aed706430 0x555aed708820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:18:31.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.780+0000 7f587f7fe640 1 --2- 172.21.15.40:0/928265207 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555aed6fd850 0x555aed705ef0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f5870005fd0 tx=0x7f587009af10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7477c09d723005bd server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:31.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.780+0000 7f58767fc640 1 -- 172.21.15.40:0/928265207 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f587009b3d0 con 0x555aed6fd850 2024-04-21T23:18:31.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.780+0000 7f58767fc640 1 -- 172.21.15.40:0/928265207 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f587009f780 con 0x555aed6fd850 2024-04-21T23:18:31.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.780+0000 7f58767fc640 1 -- 172.21.15.40:0/928265207 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58700a3df0 con 0x555aed6fd850 2024-04-21T23:18:31.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.781+0000 7f5888fa8740 1 -- 172.21.15.40:0/928265207 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555aed6fd850 msgr2=0x555aed705ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:31.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.781+0000 7f5888fa8740 1 --2- 172.21.15.40:0/928265207 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555aed6fd850 0x555aed705ef0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f5870005fd0 tx=0x7f587009af10 comp rx=0 tx=0).stop 2024-04-21T23:18:31.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.781+0000 7f5888fa8740 1 -- 172.21.15.40:0/928265207 shutdown_connections 2024-04-21T23:18:31.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.781+0000 7f5888fa8740 1 --2- 172.21.15.40:0/928265207 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555aed706430 0x555aed708820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.781+0000 7f5888fa8740 1 --2- 172.21.15.40:0/928265207 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555aed6fd850 0x555aed705ef0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.781+0000 7f5888fa8740 1 --2- 172.21.15.40:0/928265207 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555aed6fcf30 0x555aed6fd310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.781+0000 7f5888fa8740 1 -- 172.21.15.40:0/928265207 >> 172.21.15.40:0/928265207 conn(0x555aed6f7660 msgr2=0x555aed6f7a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:31.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.781+0000 7f5888fa8740 1 -- 172.21.15.40:0/928265207 shutdown_connections 2024-04-21T23:18:31.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.781+0000 7f5888fa8740 1 -- 172.21.15.40:0/928265207 wait complete. 2024-04-21T23:18:31.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.781+0000 7f5888fa8740 1 -- start start 2024-04-21T23:18:31.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.782+0000 7f5888fa8740 1 Processor -- start 2024-04-21T23:18:31.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.782+0000 7f5888fa8740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555aed6fcf30 0x555aed703570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:31.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.782+0000 7f5888fa8740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555aed6fd850 0x555aed703ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:31.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.782+0000 7f5888fa8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555aed706430 0x555aed703ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:31.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.782+0000 7f5888fa8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x555aed7091c0 con 0x555aed706430 2024-04-21T23:18:31.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.782+0000 7f5888fa8740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x555aed61ce00 con 0x555aed6fcf30 2024-04-21T23:18:31.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.782+0000 7f5888fa8740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x555aed709340 con 0x555aed6fd850 2024-04-21T23:18:31.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.782+0000 7f5888fa8740 10 client.0 _ll_register_callbacks cb 0x555aed657eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:18:31.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.782+0000 7f5884965640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555aed706430 0x555aed703ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:31.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.782+0000 7f5884965640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555aed706430 0x555aed703ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51148/0 (socket says 172.21.15.40:51148) 2024-04-21T23:18:31.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.782+0000 7f5884965640 1 -- 172.21.15.40:0/2136327533 learned_addr learned my addr 172.21.15.40:0/2136327533 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:31.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.782+0000 7f587f7fe640 1 --2- 172.21.15.40:0/2136327533 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555aed6fd850 0x555aed703ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:31.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.782+0000 7f587ffff640 1 --2- 172.21.15.40:0/2136327533 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555aed6fcf30 0x555aed703570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:31.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.782+0000 7f5884965640 1 -- 172.21.15.40:0/2136327533 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555aed6fd850 msgr2=0x555aed703ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:31.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.782+0000 7f5884965640 1 --2- 172.21.15.40:0/2136327533 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555aed6fd850 0x555aed703ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.782+0000 7f5884965640 1 -- 172.21.15.40:0/2136327533 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555aed6fcf30 msgr2=0x555aed703570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:31.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.782+0000 7f5884965640 1 --2- 172.21.15.40:0/2136327533 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555aed6fcf30 0x555aed703570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.782+0000 7f5884965640 1 -- 172.21.15.40:0/2136327533 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555aed3753f0 con 0x555aed706430 2024-04-21T23:18:31.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.782+0000 7f587f7fe640 1 --2- 172.21.15.40:0/2136327533 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555aed6fd850 0x555aed703ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:18:31.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.782+0000 7f5884965640 1 --2- 172.21.15.40:0/2136327533 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555aed706430 0x555aed703ff0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f5878004af0 tx=0x7f5878003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=624b791288c723eb server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:31.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.782+0000 7f585f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-21T23:18:31.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.783+0000 7f585f7fe640 1 -- 172.21.15.40:0/2136327533 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5878043020 con 0x555aed706430 2024-04-21T23:18:31.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.783+0000 7f585f7fe640 1 -- 172.21.15.40:0/2136327533 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f5878032720 con 0x555aed706430 2024-04-21T23:18:31.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.783+0000 7f5888fa8740 1 -- 172.21.15.40:0/2136327533 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f587009f780 con 0x555aed706430 2024-04-21T23:18:31.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.783+0000 7f585f7fe640 1 -- 172.21.15.40:0/2136327533 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f587804e050 con 0x555aed706430 2024-04-21T23:18:31.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.783+0000 7f5888fa8740 10 client.15699 fetch_fsmap learned FSMap version 25 2024-04-21T23:18:31.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.783+0000 7f5888fa8740 1 -- 172.21.15.40:0/2136327533 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x555aed3350f0 con 0x555aed706430 2024-04-21T23:18:31.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.783+0000 7f585f7fe640 1 -- 172.21.15.40:0/2136327533 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f587009f780 con 0x555aed706430 2024-04-21T23:18:31.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.783+0000 7f585f7fe640 1 -- 172.21.15.40:0/2136327533 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f58780046c0 con 0x555aed706430 2024-04-21T23:18:31.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.783+0000 7f5888fa8740 10 client.15699 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:18:31.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.783+0000 7f5888fa8740 10 client.15699 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:18:31.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.783+0000 7f5888fa8740 20 client.15699 populate_metadata read hostname 'smithi040' 2024-04-21T23:18:31.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.783+0000 7f5888fa8740 1 --2- 172.21.15.40:0/2136327533 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x555aed7167d0 0x555aed718bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:31.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.783+0000 7f5888fa8740 4 client.15699 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:18:31.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.783+0000 7f5888fa8740 1 -- 172.21.15.40:0/2136327533 --> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555aed66eaa0 con 0x555aed7167d0 2024-04-21T23:18:31.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.784+0000 7f587ffff640 1 --2- 172.21.15.40:0/2136327533 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x555aed7167d0 0x555aed718bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:31.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.784+0000 7f585f7fe640 10 client.15699 ms_handle_connect on v2:172.21.15.111:6838/3712235548 2024-04-21T23:18:31.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.784+0000 7f587ffff640 1 --2- 172.21.15.40:0/2136327533 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x555aed7167d0 0x555aed718bc0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b95ddcf41d35c782 server_cookie=31aef0f2e1deb581 in_seq=0 out_seq=0 2024-04-21T23:18:31.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.785+0000 7f585f7fe640 1 -- 172.21.15.40:0/2136327533 <== mds.0 v2:172.21.15.111:6838/3712235548 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555aed66eaa0 con 0x555aed7167d0 2024-04-21T23:18:31.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.785+0000 7f585f7fe640 10 client.15699 handle_command_reply: tid=0 2024-04-21T23:18:31.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.785+0000 7f5888fa8740 1 client.15699 shutdown 2024-04-21T23:18:31.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.785+0000 7f5888fa8740 1 -- 172.21.15.40:0/2136327533 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555aed706430 msgr2=0x555aed703ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:31.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.785+0000 7f5888fa8740 1 --2- 172.21.15.40:0/2136327533 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555aed706430 0x555aed703ff0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f5878004af0 tx=0x7f5878003ba0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.785+0000 7f5888fa8740 1 -- 172.21.15.40:0/2136327533 shutdown_connections 2024-04-21T23:18:31.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.785+0000 7f5888fa8740 1 --2- 172.21.15.40:0/2136327533 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x555aed7167d0 0x555aed718bc0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.785+0000 7f5888fa8740 1 --2- 172.21.15.40:0/2136327533 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555aed706430 0x555aed703ff0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.785+0000 7f5888fa8740 1 --2- 172.21.15.40:0/2136327533 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555aed6fd850 0x555aed703ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.785+0000 7f585f7fe640 0 client.15699 ms_handle_reset on v2:172.21.15.111:6838/3712235548 2024-04-21T23:18:31.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.785+0000 7f5888fa8740 1 --2- 172.21.15.40:0/2136327533 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555aed6fcf30 0x555aed703570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.785+0000 7f5888fa8740 1 -- 172.21.15.40:0/2136327533 >> 172.21.15.40:0/2136327533 conn(0x555aed6f7660 msgr2=0x555aed6f93b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:31.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.785+0000 7f5888fa8740 1 -- 172.21.15.40:0/2136327533 shutdown_connections 2024-04-21T23:18:31.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.785+0000 7f5888fa8740 1 -- 172.21.15.40:0/2136327533 wait complete. 2024-04-21T23:18:31.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.785+0000 7f5888fa8740 20 client.15699 trim_cache size 0 max 16384 2024-04-21T23:18:31.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.803+0000 7f5888fa8740 1 Processor -- start 2024-04-21T23:18:31.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.803+0000 7f5888fa8740 1 -- start start 2024-04-21T23:18:31.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.803+0000 7f5888fa8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555aed6fcf30 0x555aed712080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:31.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.803+0000 7f5888fa8740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555aed6fd850 0x555aed7024b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:31.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.803+0000 7f5888fa8740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555aed706430 0x555aed703e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:31.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.803+0000 7f5888fa8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f58801b3770 con 0x555aed6fcf30 2024-04-21T23:18:31.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.803+0000 7f5888fa8740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f58801b1040 con 0x555aed706430 2024-04-21T23:18:31.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.803+0000 7f5888fa8740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f58801aee90 con 0x555aed6fd850 2024-04-21T23:18:31.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.803+0000 7f587ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555aed6fcf30 0x555aed712080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:31.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.803+0000 7f587ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555aed6fcf30 0x555aed712080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51162/0 (socket says 172.21.15.40:51162) 2024-04-21T23:18:31.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.803+0000 7f587ffff640 1 -- 172.21.15.40:0/1234646205 learned_addr learned my addr 172.21.15.40:0/1234646205 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:31.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.803+0000 7f5884965640 1 --2- 172.21.15.40:0/1234646205 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555aed706430 0x555aed703e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:31.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.803+0000 7f587f7fe640 1 --2- 172.21.15.40:0/1234646205 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555aed6fd850 0x555aed7024b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:31.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.803+0000 7f587ffff640 1 -- 172.21.15.40:0/1234646205 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555aed6fd850 msgr2=0x555aed7024b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:31.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.803+0000 7f587ffff640 1 --2- 172.21.15.40:0/1234646205 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555aed6fd850 0x555aed7024b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.803+0000 7f587ffff640 1 -- 172.21.15.40:0/1234646205 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555aed706430 msgr2=0x555aed703e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:31.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.803+0000 7f587ffff640 1 --2- 172.21.15.40:0/1234646205 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555aed706430 0x555aed703e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.803+0000 7f587ffff640 1 -- 172.21.15.40:0/1234646205 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555aed6f2cf0 con 0x555aed6fcf30 2024-04-21T23:18:31.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.803+0000 7f5884965640 1 --2- 172.21.15.40:0/1234646205 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555aed706430 0x555aed703e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:18:31.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.803+0000 7f587ffff640 1 --2- 172.21.15.40:0/1234646205 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555aed6fcf30 0x555aed712080 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x555aed719400 tx=0x7f5868014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3e9b4b03907a452 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:31.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.804+0000 7f5875ffb640 1 -- 172.21.15.40:0/1234646205 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f586802d070 con 0x555aed6fcf30 2024-04-21T23:18:31.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.804+0000 7f5875ffb640 1 -- 172.21.15.40:0/1234646205 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5868027870 con 0x555aed6fcf30 2024-04-21T23:18:31.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.804+0000 7f5875ffb640 1 -- 172.21.15.40:0/1234646205 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f586801e670 con 0x555aed6fcf30 2024-04-21T23:18:31.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.804+0000 7f5888fa8740 1 -- 172.21.15.40:0/1234646205 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555aed6fcf30 msgr2=0x555aed712080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:31.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.804+0000 7f5888fa8740 1 --2- 172.21.15.40:0/1234646205 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555aed6fcf30 0x555aed712080 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x555aed719400 tx=0x7f5868014000 comp rx=0 tx=0).stop 2024-04-21T23:18:31.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.804+0000 7f5888fa8740 1 -- 172.21.15.40:0/1234646205 shutdown_connections 2024-04-21T23:18:31.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.804+0000 7f5888fa8740 1 --2- 172.21.15.40:0/1234646205 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555aed706430 0x555aed703e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.804+0000 7f5888fa8740 1 --2- 172.21.15.40:0/1234646205 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555aed6fd850 0x555aed7024b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.804+0000 7f5888fa8740 1 --2- 172.21.15.40:0/1234646205 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555aed6fcf30 0x555aed712080 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.804+0000 7f5888fa8740 1 -- 172.21.15.40:0/1234646205 >> 172.21.15.40:0/1234646205 conn(0x555aed6f7660 msgr2=0x555aed6f66e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:31.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.804+0000 7f5888fa8740 1 -- 172.21.15.40:0/1234646205 shutdown_connections 2024-04-21T23:18:31.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.804+0000 7f5888fa8740 1 -- 172.21.15.40:0/1234646205 wait complete. 2024-04-21T23:18:31.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.805+0000 7f5888fa8740 1 -- start start 2024-04-21T23:18:31.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.805+0000 7f5888fa8740 1 Processor -- start 2024-04-21T23:18:31.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.805+0000 7f5888fa8740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555aed6fcf30 0x555aed70e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:31.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.805+0000 7f5888fa8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555aed6fd850 0x555aed6ffec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:31.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.805+0000 7f5888fa8740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555aed706430 0x555aed700400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:31.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.805+0000 7f5888fa8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f58801af500 con 0x555aed6fd850 2024-04-21T23:18:31.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.805+0000 7f5888fa8740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f58801b0970 con 0x555aed6fcf30 2024-04-21T23:18:31.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.805+0000 7f5888fa8740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f58801afbd0 con 0x555aed706430 2024-04-21T23:18:31.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.805+0000 7f5888fa8740 10 client.0 _ll_register_callbacks cb 0x7f5878037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:18:31.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.805+0000 7f587f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555aed6fd850 0x555aed6ffec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:31.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.805+0000 7f587f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555aed6fd850 0x555aed6ffec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51168/0 (socket says 172.21.15.40:51168) 2024-04-21T23:18:31.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.805+0000 7f587f7fe640 1 -- 172.21.15.40:0/4182064501 learned_addr learned my addr 172.21.15.40:0/4182064501 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:31.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.805+0000 7f587ffff640 1 --2- 172.21.15.40:0/4182064501 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555aed6fcf30 0x555aed70e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:31.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.805+0000 7f5884965640 1 --2- 172.21.15.40:0/4182064501 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555aed706430 0x555aed700400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:31.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.806+0000 7f587f7fe640 1 -- 172.21.15.40:0/4182064501 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555aed706430 msgr2=0x555aed700400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:31.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.806+0000 7f587f7fe640 1 --2- 172.21.15.40:0/4182064501 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555aed706430 0x555aed700400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.806+0000 7f587f7fe640 1 -- 172.21.15.40:0/4182064501 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555aed6fcf30 msgr2=0x555aed70e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:31.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.806+0000 7f587f7fe640 1 --2- 172.21.15.40:0/4182064501 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555aed6fcf30 0x555aed70e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.806+0000 7f587f7fe640 1 -- 172.21.15.40:0/4182064501 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555aed700b90 con 0x555aed6fd850 2024-04-21T23:18:31.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.806+0000 7f587f7fe640 1 --2- 172.21.15.40:0/4182064501 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555aed6fd850 0x555aed6ffec0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f5870097830 tx=0x7f58700998e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94d72bcda19f361e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:31.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.806+0000 7f585f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-21T23:18:31.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.806+0000 7f585f7fe640 1 -- 172.21.15.40:0/4182064501 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f587009f780 con 0x555aed6fd850 2024-04-21T23:18:31.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.806+0000 7f585f7fe640 1 -- 172.21.15.40:0/4182064501 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f58700b41b0 con 0x555aed6fd850 2024-04-21T23:18:31.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.806+0000 7f5888fa8740 1 -- 172.21.15.40:0/4182064501 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5868027870 con 0x555aed6fd850 2024-04-21T23:18:31.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.806+0000 7f585f7fe640 1 -- 172.21.15.40:0/4182064501 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f58700a4040 con 0x555aed6fd850 2024-04-21T23:18:31.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.806+0000 7f5888fa8740 10 client.15705 fetch_fsmap learned FSMap version 25 2024-04-21T23:18:31.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.806+0000 7f5888fa8740 1 -- 172.21.15.40:0/4182064501 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x555aed77cd30 con 0x555aed6fd850 2024-04-21T23:18:31.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.806+0000 7f585f7fe640 1 -- 172.21.15.40:0/4182064501 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5868027870 con 0x555aed6fd850 2024-04-21T23:18:31.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.806+0000 7f585f7fe640 1 -- 172.21.15.40:0/4182064501 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f58700cc9b0 con 0x555aed6fd850 2024-04-21T23:18:31.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.807+0000 7f5888fa8740 10 client.15705 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:18:31.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.807+0000 7f5888fa8740 10 client.15705 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:18:31.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.807+0000 7f5888fa8740 20 client.15705 populate_metadata read hostname 'smithi040' 2024-04-21T23:18:31.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.807+0000 7f5888fa8740 1 --2- 172.21.15.40:0/4182064501 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x555aed77d070 0x555aed77f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:31.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.807+0000 7f5888fa8740 4 client.15705 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T23:18:31.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.807+0000 7f5888fa8740 1 -- 172.21.15.40:0/4182064501 --> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] -- command(tid 0: {"prefix": "scrub status"}) -- 0x555aed668df0 con 0x555aed77d070 2024-04-21T23:18:31.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.807+0000 7f587ffff640 1 --2- 172.21.15.40:0/4182064501 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x555aed77d070 0x555aed77f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:31.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.807+0000 7f587ffff640 1 --2- 172.21.15.40:0/4182064501 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x555aed77d070 0x555aed77f460 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=859aecbc65eef6ee server_cookie=8b24aeba7e064bae in_seq=0 out_seq=0 2024-04-21T23:18:31.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.807+0000 7f585f7fe640 10 client.15705 ms_handle_connect on v2:172.21.15.111:6838/3712235548 2024-04-21T23:18:31.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.807+0000 7f585f7fe640 1 -- 172.21.15.40:0/4182064501 <== mds.0 v2:172.21.15.111:6838/3712235548 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x555aed668df0 con 0x555aed77d070 2024-04-21T23:18:31.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.807+0000 7f585f7fe640 10 client.15705 handle_command_reply: tid=0 2024-04-21T23:18:31.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.808+0000 7f5888fa8740 1 client.15705 shutdown 2024-04-21T23:18:31.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.808+0000 7f5888fa8740 1 -- 172.21.15.40:0/4182064501 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555aed6fd850 msgr2=0x555aed6ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:31.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.808+0000 7f5888fa8740 1 --2- 172.21.15.40:0/4182064501 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555aed6fd850 0x555aed6ffec0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f5870097830 tx=0x7f58700998e0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.808+0000 7f5888fa8740 1 -- 172.21.15.40:0/4182064501 shutdown_connections 2024-04-21T23:18:31.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.808+0000 7f5888fa8740 1 --2- 172.21.15.40:0/4182064501 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x555aed77d070 0x555aed77f460 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.808+0000 7f5888fa8740 1 --2- 172.21.15.40:0/4182064501 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555aed706430 0x555aed700400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.808+0000 7f5888fa8740 1 --2- 172.21.15.40:0/4182064501 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555aed6fd850 0x555aed6ffec0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.808+0000 7f585f7fe640 0 client.15705 ms_handle_reset on v2:172.21.15.111:6838/3712235548 2024-04-21T23:18:31.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.808+0000 7f5888fa8740 1 --2- 172.21.15.40:0/4182064501 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555aed6fcf30 0x555aed70e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.808+0000 7f5888fa8740 1 -- 172.21.15.40:0/4182064501 >> 172.21.15.40:0/4182064501 conn(0x555aed6f7660 msgr2=0x555aed6f89e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:31.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.808+0000 7f5888fa8740 1 -- 172.21.15.40:0/4182064501 shutdown_connections 2024-04-21T23:18:31.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.808+0000 7f5888fa8740 1 -- 172.21.15.40:0/4182064501 wait complete. 2024-04-21T23:18:31.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.808+0000 7f5888fa8740 20 client.15705 trim_cache size 0 max 16384 2024-04-21T23:18:31.816 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-21T23:18:31.816 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-21T23:18:31.816 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": { 2024-04-21T23:18:31.816 INFO:teuthology.orchestra.run.smithi040.stdout: "93a7401c-e7f6-410e-b5a8-413666fa5f20": { 2024-04-21T23:18:31.816 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/", 2024-04-21T23:18:31.817 INFO:teuthology.orchestra.run.smithi040.stdout: "tag": "93a7401c-e7f6-410e-b5a8-413666fa5f20", 2024-04-21T23:18:31.817 INFO:teuthology.orchestra.run.smithi040.stdout: "options": "recursive,force" 2024-04-21T23:18:31.817 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:18:31.817 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:18:31.817 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-21T23:18:31.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.808+0000 7f58863c8640 1 -- 172.21.15.40:0/3164480273 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f586c0a2e50 msgr2=0x7f586c0a5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:31.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.808+0000 7f58863c8640 1 --2- 172.21.15.40:0/3164480273 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f586c0a2e50 0x7f586c0a5310 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5868009c20 tx=0x7f5868005f90 comp rx=0 tx=0).stop 2024-04-21T23:18:31.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.808+0000 7f58863c8640 1 -- 172.21.15.40:0/3164480273 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58801032e0 msgr2=0x7f5880100b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:31.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.808+0000 7f58863c8640 1 --2- 172.21.15.40:0/3164480273 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58801032e0 0x7f5880100b50 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f587000d950 tx=0x7f587000de10 comp rx=0 tx=0).stop 2024-04-21T23:18:31.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.809+0000 7f58863c8640 1 -- 172.21.15.40:0/3164480273 shutdown_connections 2024-04-21T23:18:31.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.809+0000 7f58863c8640 1 --2- 172.21.15.40:0/3164480273 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f586c0a2e50 0x7f586c0a5310 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.809+0000 7f58863c8640 1 --2- 172.21.15.40:0/3164480273 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f588010bd00 0x7f5880101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.809+0000 7f58863c8640 1 --2- 172.21.15.40:0/3164480273 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58801032e0 0x7f5880100b50 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.809+0000 7f58863c8640 1 --2- 172.21.15.40:0/3164480273 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f58801029c0 0x7f5880100610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.809+0000 7f58863c8640 1 -- 172.21.15.40:0/3164480273 >> 172.21.15.40:0/3164480273 conn(0x7f5880077740 msgr2=0x7f58800fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:31.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.809+0000 7f58863c8640 1 -- 172.21.15.40:0/3164480273 shutdown_connections 2024-04-21T23:18:31.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.809+0000 7f58863c8640 1 -- 172.21.15.40:0/3164480273 wait complete. 2024-04-21T23:18:31.819 INFO:tasks.cephfs.filesystem:scrub status for tag:93a7401c-e7f6-410e-b5a8-413666fa5f20 - {'path': '/', 'tag': '93a7401c-e7f6-410e-b5a8-413666fa5f20', 'options': 'recursive,force'} 2024-04-21T23:18:31.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.845+0000 7f532a6c2640 1 -- 172.21.15.40:0/2332982767 --> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f53240fde20 con 0x7f530c0a7350 2024-04-21T23:18:31.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.911+0000 7f53217fa640 1 -- 172.21.15.40:0/2332982767 <== mgr.25111 v2:172.21.15.40:6800/478611259 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f53240fde20 con 0x7f530c0a7350 2024-04-21T23:18:31.911 INFO:teuthology.orchestra.run.smithi040.stdout:Schedule set for path / 2024-04-21T23:18:31.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.913+0000 7f532a6c2640 1 -- 172.21.15.40:0/2332982767 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f530c0a7350 msgr2=0x7f530c0a9810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:31.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.913+0000 7f532a6c2640 1 --2- 172.21.15.40:0/2332982767 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f530c0a7350 0x7f530c0a9810 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5324100010 tx=0x7f531800b410 comp rx=0 tx=0).stop 2024-04-21T23:18:31.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.913+0000 7f532a6c2640 1 -- 172.21.15.40:0/2332982767 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53241029d0 msgr2=0x7f5324197fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:31.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.913+0000 7f532a6c2640 1 --2- 172.21.15.40:0/2332982767 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53241029d0 0x7f5324197fa0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f5314009a20 tx=0x7f531402dd30 comp rx=0 tx=0).stop 2024-04-21T23:18:31.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.914+0000 7f532a6c2640 1 -- 172.21.15.40:0/2332982767 shutdown_connections 2024-04-21T23:18:31.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.914+0000 7f532a6c2640 1 --2- 172.21.15.40:0/2332982767 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f530c0a7350 0x7f530c0a9810 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.914+0000 7f532a6c2640 1 --2- 172.21.15.40:0/2332982767 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f53241032f0 0x7f53241984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.914+0000 7f532a6c2640 1 --2- 172.21.15.40:0/2332982767 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f532410bd10 0x7f532419c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.914+0000 7f532a6c2640 1 --2- 172.21.15.40:0/2332982767 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53241029d0 0x7f5324197fa0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:31.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.914+0000 7f532a6c2640 1 -- 172.21.15.40:0/2332982767 >> 172.21.15.40:0/2332982767 conn(0x7f53240777c0 msgr2=0x7f53240fd710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:31.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.914+0000 7f532a6c2640 1 -- 172.21.15.40:0/2332982767 shutdown_connections 2024-04-21T23:18:31.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:31.914+0000 7f532a6c2640 1 -- 172.21.15.40:0/2332982767 wait complete. 2024-04-21T23:18:31.923 DEBUG:teuthology.orchestra.run.smithi040:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-04-21T23:18:32.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.070+0000 7f7bf2efb640 1 -- 172.21.15.40:0/1955006243 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7bec1029f0 msgr2=0x7f7bec102dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:32.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.070+0000 7f7bf2efb640 1 --2- 172.21.15.40:0/1955006243 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7bec1029f0 0x7f7bec102dd0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f7be0009900 tx=0x7f7be002bcd0 comp rx=0 tx=0).stop 2024-04-21T23:18:32.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.070+0000 7f7bf2efb640 1 -- 172.21.15.40:0/1955006243 shutdown_connections 2024-04-21T23:18:32.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.070+0000 7f7bf2efb640 1 --2- 172.21.15.40:0/1955006243 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7bec103310 0x7f7bec10b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:32.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.070+0000 7f7bf2efb640 1 --2- 172.21.15.40:0/1955006243 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7bec10bdc0 0x7f7bec10e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:32.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.070+0000 7f7bf2efb640 1 --2- 172.21.15.40:0/1955006243 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7bec1029f0 0x7f7bec102dd0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:32.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.070+0000 7f7bf2efb640 1 -- 172.21.15.40:0/1955006243 >> 172.21.15.40:0/1955006243 conn(0x7f7bec0769c0 msgr2=0x7f7bec076dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:32.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.070+0000 7f7bf2efb640 1 -- 172.21.15.40:0/1955006243 shutdown_connections 2024-04-21T23:18:32.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.070+0000 7f7bf2efb640 1 -- 172.21.15.40:0/1955006243 wait complete. 2024-04-21T23:18:32.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.070+0000 7f7bf2efb640 1 Processor -- start 2024-04-21T23:18:32.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.070+0000 7f7bf2efb640 1 -- start start 2024-04-21T23:18:32.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.071+0000 7f7bf2efb640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7bec1029f0 0x7f7bec197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:32.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.071+0000 7f7bf2efb640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7bec103310 0x7f7bec1984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:32.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.071+0000 7f7bf2efb640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7bec10bdc0 0x7f7bec19c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:32.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.071+0000 7f7bf2efb640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7bec111ba0 con 0x7f7bec10bdc0 2024-04-21T23:18:32.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.071+0000 7f7bf2efb640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f7bec111a20 con 0x7f7bec103310 2024-04-21T23:18:32.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.071+0000 7f7bf2efb640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7bec111d20 con 0x7f7bec1029f0 2024-04-21T23:18:32.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.071+0000 7f7bf1498640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7bec10bdc0 0x7f7bec19c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:32.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.071+0000 7f7bf1498640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7bec10bdc0 0x7f7bec19c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51182/0 (socket says 172.21.15.40:51182) 2024-04-21T23:18:32.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.071+0000 7f7bf1498640 1 -- 172.21.15.40:0/743489723 learned_addr learned my addr 172.21.15.40:0/743489723 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:32.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.071+0000 7f7bf0c97640 1 --2- 172.21.15.40:0/743489723 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7bec1029f0 0x7f7bec197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:32.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.071+0000 7f7bebfff640 1 --2- 172.21.15.40:0/743489723 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7bec103310 0x7f7bec1984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:32.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.071+0000 7f7bf1498640 1 -- 172.21.15.40:0/743489723 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7bec1029f0 msgr2=0x7f7bec197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:32.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.071+0000 7f7bf1498640 1 --2- 172.21.15.40:0/743489723 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7bec1029f0 0x7f7bec197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:32.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.071+0000 7f7bf1498640 1 -- 172.21.15.40:0/743489723 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7bec103310 msgr2=0x7f7bec1984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:32.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.071+0000 7f7bf1498640 1 --2- 172.21.15.40:0/743489723 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7bec103310 0x7f7bec1984a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:32.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.071+0000 7f7bf1498640 1 -- 172.21.15.40:0/743489723 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bec19cf50 con 0x7f7bec10bdc0 2024-04-21T23:18:32.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.071+0000 7f7bf1498640 1 --2- 172.21.15.40:0/743489723 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7bec10bdc0 0x7f7bec19c820 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f7be400e9d0 tx=0x7f7be400ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:32.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.071+0000 7f7be9ffb640 1 -- 172.21.15.40:0/743489723 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7be400cd80 con 0x7f7bec10bdc0 2024-04-21T23:18:32.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.072+0000 7f7be9ffb640 1 -- 172.21.15.40:0/743489723 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7be4005020 con 0x7f7bec10bdc0 2024-04-21T23:18:32.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.072+0000 7f7bf2efb640 1 -- 172.21.15.40:0/743489723 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bec19d240 con 0x7f7bec10bdc0 2024-04-21T23:18:32.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.072+0000 7f7be9ffb640 1 -- 172.21.15.40:0/743489723 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7be40116d0 con 0x7f7bec10bdc0 2024-04-21T23:18:32.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.072+0000 7f7bf2efb640 1 -- 172.21.15.40:0/743489723 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bec1a4ad0 con 0x7f7bec10bdc0 2024-04-21T23:18:32.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.073+0000 7f7bf2efb640 1 -- 172.21.15.40:0/743489723 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7bec04f500 con 0x7f7bec10bdc0 2024-04-21T23:18:32.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.074+0000 7f7be9ffb640 1 -- 172.21.15.40:0/743489723 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7f7be40051d0 con 0x7f7bec10bdc0 2024-04-21T23:18:32.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.075+0000 7f7be9ffb640 1 --2- 172.21.15.40:0/743489723 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f7bdc0a2ec0 0x7f7bdc0a5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:32.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.075+0000 7f7bf0c97640 1 --2- 172.21.15.40:0/743489723 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f7bdc0a2ec0 0x7f7bdc0a5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:32.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.075+0000 7f7be9ffb640 1 -- 172.21.15.40:0/743489723 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f7be40cff00 con 0x7f7bec10bdc0 2024-04-21T23:18:32.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.075+0000 7f7bf0c97640 1 --2- 172.21.15.40:0/743489723 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f7bdc0a2ec0 0x7f7bdc0a5380 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7be0005ab0 tx=0x7f7be0005a20 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:32.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.079+0000 7f7be9ffb640 1 -- 172.21.15.40:0/743489723 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f7be4016030 con 0x7f7bec10bdc0 2024-04-21T23:18:32.181 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:31 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/151147968' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:18:32.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:31 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/958670969' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:18:32.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.287+0000 7f7bf2efb640 1 -- 172.21.15.40:0/743489723 --> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) -- 0x7f7bec0fdb40 con 0x7f7bdc0a2ec0 2024-04-21T23:18:32.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.348+0000 7f7be9ffb640 1 -- 172.21.15.40:0/743489723 <== mgr.25111 v2:172.21.15.40:6800/478611259 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+26 (secure 0 0 0) 0x7f7bec0fdb40 con 0x7f7bdc0a2ec0 2024-04-21T23:18:32.348 INFO:teuthology.orchestra.run.smithi040.stdout:Retention added to path / 2024-04-21T23:18:32.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.350+0000 7f7bf2efb640 1 -- 172.21.15.40:0/743489723 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f7bdc0a2ec0 msgr2=0x7f7bdc0a5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:32.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.350+0000 7f7bf2efb640 1 --2- 172.21.15.40:0/743489723 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f7bdc0a2ec0 0x7f7bdc0a5380 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7be0005ab0 tx=0x7f7be0005a20 comp rx=0 tx=0).stop 2024-04-21T23:18:32.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.350+0000 7f7bf2efb640 1 -- 172.21.15.40:0/743489723 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7bec10bdc0 msgr2=0x7f7bec19c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:32.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.350+0000 7f7bf2efb640 1 --2- 172.21.15.40:0/743489723 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7bec10bdc0 0x7f7bec19c820 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f7be400e9d0 tx=0x7f7be400ee90 comp rx=0 tx=0).stop 2024-04-21T23:18:32.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.350+0000 7f7bf2efb640 1 -- 172.21.15.40:0/743489723 shutdown_connections 2024-04-21T23:18:32.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.350+0000 7f7bf2efb640 1 --2- 172.21.15.40:0/743489723 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f7bdc0a2ec0 0x7f7bdc0a5380 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:32.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.350+0000 7f7bf2efb640 1 --2- 172.21.15.40:0/743489723 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7bec103310 0x7f7bec1984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:32.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.350+0000 7f7bf2efb640 1 --2- 172.21.15.40:0/743489723 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7bec10bdc0 0x7f7bec19c820 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:32.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.350+0000 7f7bf2efb640 1 --2- 172.21.15.40:0/743489723 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7bec1029f0 0x7f7bec197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:32.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.350+0000 7f7bf2efb640 1 -- 172.21.15.40:0/743489723 >> 172.21.15.40:0/743489723 conn(0x7f7bec0769c0 msgr2=0x7f7bec0fd430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:32.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.351+0000 7f7bf2efb640 1 -- 172.21.15.40:0/743489723 shutdown_connections 2024-04-21T23:18:32.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.351+0000 7f7bf2efb640 1 -- 172.21.15.40:0/743489723 wait complete. 2024-04-21T23:18:32.360 DEBUG:teuthology.orchestra.run.smithi040:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-04-21T23:18:32.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:31 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/151147968' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:18:32.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:31 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/958670969' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:18:32.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:31 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/151147968' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:18:32.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:31 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/958670969' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:18:32.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.466+0000 7fb6769bf640 1 -- 172.21.15.40:0/485748973 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb67010bd40 msgr2=0x7fb67010e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:32.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.466+0000 7fb6769bf640 1 --2- 172.21.15.40:0/485748973 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb67010bd40 0x7fb67010e1c0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fb668009ca0 tx=0x7fb66802c180 comp rx=0 tx=0).stop 2024-04-21T23:18:32.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.466+0000 7fb6769bf640 1 -- 172.21.15.40:0/485748973 shutdown_connections 2024-04-21T23:18:32.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.466+0000 7fb6769bf640 1 --2- 172.21.15.40:0/485748973 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb670103290 0x7fb67010b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:32.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.466+0000 7fb6769bf640 1 --2- 172.21.15.40:0/485748973 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb67010bd40 0x7fb67010e1c0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:32.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.466+0000 7fb6769bf640 1 --2- 172.21.15.40:0/485748973 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb670102970 0x7fb670102d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:32.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.466+0000 7fb6769bf640 1 -- 172.21.15.40:0/485748973 >> 172.21.15.40:0/485748973 conn(0x7fb6700777c0 msgr2=0x7fb670077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:32.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.466+0000 7fb6769bf640 1 -- 172.21.15.40:0/485748973 shutdown_connections 2024-04-21T23:18:32.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.466+0000 7fb6769bf640 1 -- 172.21.15.40:0/485748973 wait complete. 2024-04-21T23:18:32.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.466+0000 7fb6769bf640 1 Processor -- start 2024-04-21T23:18:32.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.466+0000 7fb6769bf640 1 -- start start 2024-04-21T23:18:32.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.467+0000 7fb6769bf640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb670102970 0x7fb670197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:32.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.467+0000 7fb6769bf640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb670103290 0x7fb670198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:32.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.467+0000 7fb6769bf640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb67010bd40 0x7fb67019c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:32.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.467+0000 7fb6769bf640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb670111b20 con 0x7fb670102970 2024-04-21T23:18:32.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.467+0000 7fb6769bf640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb6701119a0 con 0x7fb670103290 2024-04-21T23:18:32.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.467+0000 7fb6769bf640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb670111ca0 con 0x7fb67010bd40 2024-04-21T23:18:32.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.467+0000 7fb66ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb670102970 0x7fb670197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:32.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.467+0000 7fb66ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb670102970 0x7fb670197f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51194/0 (socket says 172.21.15.40:51194) 2024-04-21T23:18:32.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.467+0000 7fb66ffff640 1 -- 172.21.15.40:0/4113487038 learned_addr learned my addr 172.21.15.40:0/4113487038 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:32.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.467+0000 7fb66f7fe640 1 --2- 172.21.15.40:0/4113487038 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb670103290 0x7fb670198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:32.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.467+0000 7fb674f5c640 1 --2- 172.21.15.40:0/4113487038 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb67010bd40 0x7fb67019c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:32.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.467+0000 7fb66ffff640 1 -- 172.21.15.40:0/4113487038 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb67010bd40 msgr2=0x7fb67019c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:32.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.467+0000 7fb66ffff640 1 --2- 172.21.15.40:0/4113487038 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb67010bd40 0x7fb67019c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:32.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.467+0000 7fb66ffff640 1 -- 172.21.15.40:0/4113487038 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb670103290 msgr2=0x7fb670198440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:32.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.467+0000 7fb66ffff640 1 --2- 172.21.15.40:0/4113487038 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb670103290 0x7fb670198440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:32.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.467+0000 7fb66ffff640 1 -- 172.21.15.40:0/4113487038 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb67019cef0 con 0x7fb670102970 2024-04-21T23:18:32.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.467+0000 7fb674f5c640 1 --2- 172.21.15.40:0/4113487038 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb67010bd40 0x7fb67019c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:18:32.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.467+0000 7fb66ffff640 1 --2- 172.21.15.40:0/4113487038 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb670102970 0x7fb670197f00 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fb66000c920 tx=0x7fb66000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:32.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.467+0000 7fb66d7fa640 1 -- 172.21.15.40:0/4113487038 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb660007c70 con 0x7fb670102970 2024-04-21T23:18:32.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.467+0000 7fb66d7fa640 1 -- 172.21.15.40:0/4113487038 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb660007e10 con 0x7fb670102970 2024-04-21T23:18:32.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.467+0000 7fb66d7fa640 1 -- 172.21.15.40:0/4113487038 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb660018890 con 0x7fb670102970 2024-04-21T23:18:32.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.468+0000 7fb6769bf640 1 -- 172.21.15.40:0/4113487038 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb67019d1e0 con 0x7fb670102970 2024-04-21T23:18:32.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.468+0000 7fb6769bf640 1 -- 172.21.15.40:0/4113487038 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6701a4a70 con 0x7fb670102970 2024-04-21T23:18:32.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.469+0000 7fb6769bf640 1 -- 172.21.15.40:0/4113487038 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb67004f500 con 0x7fb670102970 2024-04-21T23:18:32.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.470+0000 7fb66d7fa640 1 -- 172.21.15.40:0/4113487038 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7fb660018a50 con 0x7fb670102970 2024-04-21T23:18:32.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.471+0000 7fb66d7fa640 1 --2- 172.21.15.40:0/4113487038 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fb6580a2ec0 0x7fb6580a5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:32.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.471+0000 7fb66f7fe640 1 --2- 172.21.15.40:0/4113487038 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fb6580a2ec0 0x7fb6580a5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:32.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.471+0000 7fb66d7fa640 1 -- 172.21.15.40:0/4113487038 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7fb6600cff40 con 0x7fb670102970 2024-04-21T23:18:32.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.471+0000 7fb66f7fe640 1 --2- 172.21.15.40:0/4113487038 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fb6580a2ec0 0x7fb6580a5380 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb664005fd0 tx=0x7fb664005850 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:32.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.474+0000 7fb66d7fa640 1 -- 172.21.15.40:0/4113487038 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fb660014030 con 0x7fb670102970 2024-04-21T23:18:32.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.682+0000 7fb6769bf640 1 -- 172.21.15.40:0/4113487038 --> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7fb6700fdca0 con 0x7fb6580a2ec0 2024-04-21T23:18:32.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.690+0000 7fb66d7fa640 1 -- 172.21.15.40:0/4113487038 <== mgr.25111 v2:172.21.15.40:6800/478611259 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+298 (secure 0 0 0) 0x7fb6700fdca0 con 0x7fb6580a2ec0 2024-04-21T23:18:32.689 INFO:teuthology.orchestra.run.smithi040.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-04-21T00:00:00", "created": "2024-04-21T23:18:31", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-04-21T23:18:32.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.692+0000 7fb6769bf640 1 -- 172.21.15.40:0/4113487038 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fb6580a2ec0 msgr2=0x7fb6580a5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:32.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.692+0000 7fb6769bf640 1 --2- 172.21.15.40:0/4113487038 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fb6580a2ec0 0x7fb6580a5380 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb664005fd0 tx=0x7fb664005850 comp rx=0 tx=0).stop 2024-04-21T23:18:32.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.692+0000 7fb6769bf640 1 -- 172.21.15.40:0/4113487038 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb670102970 msgr2=0x7fb670197f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:32.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.692+0000 7fb6769bf640 1 --2- 172.21.15.40:0/4113487038 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb670102970 0x7fb670197f00 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fb66000c920 tx=0x7fb66000cde0 comp rx=0 tx=0).stop 2024-04-21T23:18:32.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.692+0000 7fb6769bf640 1 -- 172.21.15.40:0/4113487038 shutdown_connections 2024-04-21T23:18:32.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.692+0000 7fb6769bf640 1 --2- 172.21.15.40:0/4113487038 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fb6580a2ec0 0x7fb6580a5380 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:32.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.692+0000 7fb6769bf640 1 --2- 172.21.15.40:0/4113487038 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb670103290 0x7fb670198440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:32.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.692+0000 7fb6769bf640 1 --2- 172.21.15.40:0/4113487038 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb67010bd40 0x7fb67019c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:32.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.692+0000 7fb6769bf640 1 --2- 172.21.15.40:0/4113487038 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb670102970 0x7fb670197f00 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:32.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.692+0000 7fb6769bf640 1 -- 172.21.15.40:0/4113487038 >> 172.21.15.40:0/4113487038 conn(0x7fb6700777c0 msgr2=0x7fb6700fd590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:32.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.692+0000 7fb6769bf640 1 -- 172.21.15.40:0/4113487038 shutdown_connections 2024-04-21T23:18:32.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.692+0000 7fb6769bf640 1 -- 172.21.15.40:0/4113487038 wait complete. 2024-04-21T23:18:32.702 DEBUG:teuthology.orchestra.run.smithi040:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-04-21T23:18:32.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.807+0000 7f7cd2be5640 1 -- 172.21.15.40:0/823897703 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ccc1029f0 msgr2=0x7f7ccc102dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:32.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.807+0000 7f7cd2be5640 1 --2- 172.21.15.40:0/823897703 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ccc1029f0 0x7f7ccc102dd0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f7cc0009900 tx=0x7f7cc002bcd0 comp rx=0 tx=0).stop 2024-04-21T23:18:32.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.807+0000 7f7cd2be5640 1 -- 172.21.15.40:0/823897703 shutdown_connections 2024-04-21T23:18:32.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.807+0000 7f7cd2be5640 1 --2- 172.21.15.40:0/823897703 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ccc103310 0x7f7ccc10b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:32.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.807+0000 7f7cd2be5640 1 --2- 172.21.15.40:0/823897703 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7ccc10bdc0 0x7f7ccc10e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:32.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.807+0000 7f7cd2be5640 1 --2- 172.21.15.40:0/823897703 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ccc1029f0 0x7f7ccc102dd0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:32.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.807+0000 7f7cd2be5640 1 -- 172.21.15.40:0/823897703 >> 172.21.15.40:0/823897703 conn(0x7f7ccc0769c0 msgr2=0x7f7ccc076dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:32.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.807+0000 7f7cd2be5640 1 -- 172.21.15.40:0/823897703 shutdown_connections 2024-04-21T23:18:32.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.807+0000 7f7cd2be5640 1 -- 172.21.15.40:0/823897703 wait complete. 2024-04-21T23:18:32.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.808+0000 7f7cd2be5640 1 Processor -- start 2024-04-21T23:18:32.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.808+0000 7f7cd2be5640 1 -- start start 2024-04-21T23:18:32.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.808+0000 7f7cd2be5640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ccc1029f0 0x7f7ccc197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:32.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.808+0000 7f7cd2be5640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7ccc103310 0x7f7ccc198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:32.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.808+0000 7f7cd2be5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ccc10bdc0 0x7f7ccc19c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:32.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.808+0000 7f7cd2be5640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7ccc111c00 con 0x7f7ccc10bdc0 2024-04-21T23:18:32.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.808+0000 7f7cd2be5640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f7ccc111a80 con 0x7f7ccc1029f0 2024-04-21T23:18:32.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.808+0000 7f7cd2be5640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7ccc111d80 con 0x7f7ccc103310 2024-04-21T23:18:32.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.808+0000 7f7cd1182640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ccc10bdc0 0x7f7ccc19c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:32.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.808+0000 7f7cd1182640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ccc10bdc0 0x7f7ccc19c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51202/0 (socket says 172.21.15.40:51202) 2024-04-21T23:18:32.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.808+0000 7f7cd1182640 1 -- 172.21.15.40:0/3811804045 learned_addr learned my addr 172.21.15.40:0/3811804045 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:32.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.808+0000 7f7cd0981640 1 --2- 172.21.15.40:0/3811804045 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ccc1029f0 0x7f7ccc197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:32.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.808+0000 7f7ccbfff640 1 --2- 172.21.15.40:0/3811804045 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7ccc103310 0x7f7ccc198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:32.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.808+0000 7f7cd1182640 1 -- 172.21.15.40:0/3811804045 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7ccc103310 msgr2=0x7f7ccc198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:32.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.808+0000 7f7cd1182640 1 --2- 172.21.15.40:0/3811804045 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7ccc103310 0x7f7ccc198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:32.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.808+0000 7f7cd1182640 1 -- 172.21.15.40:0/3811804045 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ccc1029f0 msgr2=0x7f7ccc197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:32.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.808+0000 7f7cd1182640 1 --2- 172.21.15.40:0/3811804045 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ccc1029f0 0x7f7ccc197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:32.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.808+0000 7f7cd1182640 1 -- 172.21.15.40:0/3811804045 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ccc19cfb0 con 0x7f7ccc10bdc0 2024-04-21T23:18:32.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.808+0000 7f7cd0981640 1 --2- 172.21.15.40:0/3811804045 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ccc1029f0 0x7f7ccc197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:18:32.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.808+0000 7f7ccbfff640 1 --2- 172.21.15.40:0/3811804045 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7ccc103310 0x7f7ccc198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:18:32.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.809+0000 7f7cd1182640 1 --2- 172.21.15.40:0/3811804045 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ccc10bdc0 0x7f7ccc19c880 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f7cc400b9e0 tx=0x7f7cc400bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:32.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.809+0000 7f7cc9ffb640 1 -- 172.21.15.40:0/3811804045 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cc4019070 con 0x7f7ccc10bdc0 2024-04-21T23:18:32.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.809+0000 7f7cc9ffb640 1 -- 172.21.15.40:0/3811804045 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7cc4014440 con 0x7f7ccc10bdc0 2024-04-21T23:18:32.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.809+0000 7f7cd2be5640 1 -- 172.21.15.40:0/3811804045 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ccc19d2a0 con 0x7f7ccc10bdc0 2024-04-21T23:18:32.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.809+0000 7f7cc9ffb640 1 -- 172.21.15.40:0/3811804045 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cc40125e0 con 0x7f7ccc10bdc0 2024-04-21T23:18:32.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.809+0000 7f7cd2be5640 1 -- 172.21.15.40:0/3811804045 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ccc1a4b30 con 0x7f7ccc10bdc0 2024-04-21T23:18:32.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.810+0000 7f7cd2be5640 1 -- 172.21.15.40:0/3811804045 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ccc04f500 con 0x7f7ccc10bdc0 2024-04-21T23:18:32.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.811+0000 7f7cc9ffb640 1 -- 172.21.15.40:0/3811804045 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7f7cc401c070 con 0x7f7ccc10bdc0 2024-04-21T23:18:32.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.812+0000 7f7cc9ffb640 1 --2- 172.21.15.40:0/3811804045 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f7cbc0a2ec0 0x7f7cbc0a5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:32.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.812+0000 7f7cd0981640 1 --2- 172.21.15.40:0/3811804045 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f7cbc0a2ec0 0x7f7cbc0a5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:32.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.812+0000 7f7cc9ffb640 1 -- 172.21.15.40:0/3811804045 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f7cc40cf070 con 0x7f7ccc10bdc0 2024-04-21T23:18:32.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.812+0000 7f7cd0981640 1 --2- 172.21.15.40:0/3811804045 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f7cbc0a2ec0 0x7f7cbc0a5380 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7cc0006010 tx=0x7f7cc0005a20 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:32.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:32.816+0000 7f7cc9ffb640 1 -- 172.21.15.40:0/3811804045 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f7cc409b7a0 con 0x7f7ccc10bdc0 2024-04-21T23:18:33.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:33.023+0000 7f7cd2be5640 1 -- 172.21.15.40:0/3811804045 --> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f7ccc0fdb20 con 0x7f7cbc0a2ec0 2024-04-21T23:18:33.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:33.031+0000 7f7cc9ffb640 1 -- 172.21.15.40:0/3811804045 <== mgr.25111 v2:172.21.15.40:6800/478611259 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f7ccc0fdb20 con 0x7f7cbc0a2ec0 2024-04-21T23:18:33.030 INFO:teuthology.orchestra.run.smithi040.stdout:/ 1m 6m3h 2024-04-21T23:18:33.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:33.033+0000 7f7cd2be5640 1 -- 172.21.15.40:0/3811804045 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f7cbc0a2ec0 msgr2=0x7f7cbc0a5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:33.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:33.033+0000 7f7cd2be5640 1 --2- 172.21.15.40:0/3811804045 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f7cbc0a2ec0 0x7f7cbc0a5380 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7cc0006010 tx=0x7f7cc0005a20 comp rx=0 tx=0).stop 2024-04-21T23:18:33.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:33.033+0000 7f7cd2be5640 1 -- 172.21.15.40:0/3811804045 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ccc10bdc0 msgr2=0x7f7ccc19c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:33.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:33.033+0000 7f7cd2be5640 1 --2- 172.21.15.40:0/3811804045 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ccc10bdc0 0x7f7ccc19c880 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f7cc400b9e0 tx=0x7f7cc400bea0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:33.033+0000 7f7cd2be5640 1 -- 172.21.15.40:0/3811804045 shutdown_connections 2024-04-21T23:18:33.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:33.033+0000 7f7cd2be5640 1 --2- 172.21.15.40:0/3811804045 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f7cbc0a2ec0 0x7f7cbc0a5380 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:33.033+0000 7f7cd2be5640 1 --2- 172.21.15.40:0/3811804045 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ccc10bdc0 0x7f7ccc19c880 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:33.033+0000 7f7cd2be5640 1 --2- 172.21.15.40:0/3811804045 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7ccc103310 0x7f7ccc198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:33.033+0000 7f7cd2be5640 1 --2- 172.21.15.40:0/3811804045 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ccc1029f0 0x7f7ccc197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:33.033+0000 7f7cd2be5640 1 -- 172.21.15.40:0/3811804045 >> 172.21.15.40:0/3811804045 conn(0x7f7ccc0769c0 msgr2=0x7f7ccc0fd410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:33.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:33.033+0000 7f7cd2be5640 1 -- 172.21.15.40:0/3811804045 shutdown_connections 2024-04-21T23:18:33.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:18:33.033+0000 7f7cd2be5640 1 -- 172.21.15.40:0/3811804045 wait complete. 2024-04-21T23:18:33.043 DEBUG:teuthology.orchestra.run.smithi040:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-04-21T23:18:33.069 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-04-21T23:18:33.080 INFO:teuthology.run_tasks:Running task background_exec... 2024-04-21T23:18:33.108 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi040.front.sepia.ceph.com 2024-04-21T23:18:33.109 DEBUG:teuthology.orchestra.run.smithi040:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-04-21T23:18:33.111 INFO:teuthology.run_tasks:Running task workunit... 2024-04-21T23:18:33.150 INFO:tasks.workunit:Pulling workunits from ref 430e09df97c8fc7dc2b2ae424f68ed11366c540f 2024-04-21T23:18:33.151 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-21T23:18:33.151 INFO:tasks.workunit:timeout=3h 2024-04-21T23:18:33.151 INFO:tasks.workunit:cleanup=True 2024-04-21T23:18:33.151 DEBUG:teuthology.orchestra.run.smithi040:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-21T23:18:33.167 INFO:teuthology.orchestra.run.smithi040.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-04-21T23:18:33.167 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-04-21T23:18:33.167 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 73h/115d Inode: 1099511628283 Links: 2 2024-04-21T23:18:33.167 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T23:18:33.167 INFO:teuthology.orchestra.run.smithi040.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-04-21T23:18:33.167 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2024-04-21 23:16:23.922472157 +0000 2024-04-21T23:18:33.167 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2024-04-21 23:16:23.922472157 +0000 2024-04-21T23:18:33.168 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2024-04-21 23:18:21.182175211 +0000 2024-04-21T23:18:33.168 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: 2024-04-21 23:16:23.922472157 +0000 2024-04-21T23:18:33.168 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-04-21T23:18:33.168 DEBUG:teuthology.orchestra.run.smithi040:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-04-21T23:18:33.240 DEBUG:teuthology.orchestra.run.smithi040:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 430e09df97c8fc7dc2b2ae424f68ed11366c540f 2024-04-21T23:18:33.295 INFO:tasks.workunit.client.0.smithi040.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-21T23:18:33.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:32 smithi040 ceph-mon[27984]: pgmap v7: 129 pgs: 129 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 1.3 KiB/s wr, 23 op/s 2024-04-21T23:18:33.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:32 smithi040 ceph-mon[27984]: scrub queued for path: / 2024-04-21T23:18:33.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:32 smithi040 ceph-mon[27984]: scrub summary: idle+waiting paths [/] 2024-04-21T23:18:33.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:32 smithi040 ceph-mon[27984]: scrub summary: active paths [/] 2024-04-21T23:18:33.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:32 smithi040 ceph-mon[27984]: scrub summary: idle+waiting paths [/] 2024-04-21T23:18:33.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:32 smithi040 ceph-mon[27984]: from='client.15672 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:18:33.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:32 smithi174 ceph-mon[31409]: pgmap v7: 129 pgs: 129 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 1.3 KiB/s wr, 23 op/s 2024-04-21T23:18:33.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:32 smithi174 ceph-mon[31409]: scrub queued for path: / 2024-04-21T23:18:33.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:32 smithi174 ceph-mon[31409]: scrub summary: idle+waiting paths [/] 2024-04-21T23:18:33.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:32 smithi174 ceph-mon[31409]: scrub summary: active paths [/] 2024-04-21T23:18:33.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:32 smithi174 ceph-mon[31409]: scrub summary: idle+waiting paths [/] 2024-04-21T23:18:33.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:32 smithi174 ceph-mon[31409]: from='client.15672 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:18:33.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:32 smithi111 ceph-mon[31085]: pgmap v7: 129 pgs: 129 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 1.3 KiB/s wr, 23 op/s 2024-04-21T23:18:33.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:32 smithi111 ceph-mon[31085]: scrub queued for path: / 2024-04-21T23:18:33.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:32 smithi111 ceph-mon[31085]: scrub summary: idle+waiting paths [/] 2024-04-21T23:18:33.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:32 smithi111 ceph-mon[31085]: scrub summary: active paths [/] 2024-04-21T23:18:33.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:32 smithi111 ceph-mon[31085]: scrub summary: idle+waiting paths [/] 2024-04-21T23:18:33.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:32 smithi111 ceph-mon[31085]: from='client.15672 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:18:34.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:34 smithi040 ceph-mon[27984]: from='client.15711 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:18:34.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:34 smithi040 ceph-mon[27984]: from='client.15717 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:18:34.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:34 smithi040 ceph-mon[27984]: from='client.15723 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:18:34.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:34 smithi174 ceph-mon[31409]: from='client.15711 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:18:34.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:34 smithi174 ceph-mon[31409]: from='client.15717 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:18:34.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:34 smithi174 ceph-mon[31409]: from='client.15723 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:18:34.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:34 smithi111 ceph-mon[31085]: from='client.15711 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:18:34.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:34 smithi111 ceph-mon[31085]: from='client.15717 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:18:34.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:34 smithi111 ceph-mon[31085]: from='client.15723 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:18:35.345 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:35 smithi174 ceph-mon[31409]: pgmap v8: 129 pgs: 129 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 1 KiB/s wr, 18 op/s 2024-04-21T23:18:35.345 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:35 smithi111 ceph-mon[31085]: pgmap v8: 129 pgs: 129 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 1 KiB/s wr, 18 op/s 2024-04-21T23:18:35.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:35 smithi040 ceph-mon[27984]: pgmap v8: 129 pgs: 129 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 1 KiB/s wr, 18 op/s 2024-04-21T23:18:37.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:37 smithi040 ceph-mon[27984]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 85 KiB/s wr, 53 op/s 2024-04-21T23:18:37.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:37 smithi111 ceph-mon[31085]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 85 KiB/s wr, 53 op/s 2024-04-21T23:18:37.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:37 smithi174 ceph-mon[31409]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 85 KiB/s wr, 53 op/s 2024-04-21T23:18:39.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:39 smithi040 ceph-mon[27984]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 78 KiB/s wr, 48 op/s 2024-04-21T23:18:39.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:39 smithi174 ceph-mon[31409]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 78 KiB/s wr, 48 op/s 2024-04-21T23:18:39.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:39 smithi111 ceph-mon[31085]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 78 KiB/s wr, 48 op/s 2024-04-21T23:18:41.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:41 smithi040 ceph-mon[27984]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 78 KiB/s wr, 50 op/s 2024-04-21T23:18:41.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:41 smithi174 ceph-mon[31409]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 78 KiB/s wr, 50 op/s 2024-04-21T23:18:41.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:41 smithi111 ceph-mon[31085]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 78 KiB/s wr, 50 op/s 2024-04-21T23:18:43.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:43 smithi040 ceph-mon[27984]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 40 op/s 2024-04-21T23:18:43.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:43 smithi174 ceph-mon[31409]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 40 op/s 2024-04-21T23:18:43.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:43 smithi111 ceph-mon[31085]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 40 op/s 2024-04-21T23:18:45.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:45 smithi040 ceph-mon[27984]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 77 KiB/s wr, 37 op/s 2024-04-21T23:18:45.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:45 smithi040 ceph-mon[27984]: scrub summary: idle 2024-04-21T23:18:45.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:45 smithi174 ceph-mon[31409]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 77 KiB/s wr, 37 op/s 2024-04-21T23:18:45.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:45 smithi174 ceph-mon[31409]: scrub summary: idle 2024-04-21T23:18:45.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:45 smithi111 ceph-mon[31085]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 77 KiB/s wr, 37 op/s 2024-04-21T23:18:45.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:45 smithi111 ceph-mon[31085]: scrub summary: idle 2024-04-21T23:18:46.254 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.254+0000 7fb303397640 1 -- 172.21.15.40:0/661933909 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2fc1029d0 msgr2=0x7fb2fc102db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:46.254 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.254+0000 7fb303397640 1 --2- 172.21.15.40:0/661933909 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2fc1029d0 0x7fb2fc102db0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fb2f8009900 tx=0x7fb2f802bd60 comp rx=0 tx=0).stop 2024-04-21T23:18:46.254 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.254+0000 7fb303397640 1 -- 172.21.15.40:0/661933909 shutdown_connections 2024-04-21T23:18:46.254 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.254+0000 7fb303397640 1 --2- 172.21.15.40:0/661933909 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2fc10bd10 0x7fb2fc10e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:46.254 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.254+0000 7fb303397640 1 --2- 172.21.15.40:0/661933909 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb2fc1032f0 0x7fb2fc10b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:46.254 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.254+0000 7fb303397640 1 --2- 172.21.15.40:0/661933909 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2fc1029d0 0x7fb2fc102db0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:46.254 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.254+0000 7fb303397640 1 -- 172.21.15.40:0/661933909 >> 172.21.15.40:0/661933909 conn(0x7fb2fc0777c0 msgr2=0x7fb2fc077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:46.254 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.254+0000 7fb303397640 1 -- 172.21.15.40:0/661933909 shutdown_connections 2024-04-21T23:18:46.254 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.254+0000 7fb303397640 1 -- 172.21.15.40:0/661933909 wait complete. 2024-04-21T23:18:46.254 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.255+0000 7fb303397640 1 Processor -- start 2024-04-21T23:18:46.254 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.255+0000 7fb303397640 1 -- start start 2024-04-21T23:18:46.256 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.255+0000 7fb303397640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2fc1029d0 0x7fb2fc100620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:46.256 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.255+0000 7fb303397640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb2fc1032f0 0x7fb2fc100b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:46.257 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.255+0000 7fb303397640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2fc10bd10 0x7fb2fc1010a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:46.257 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.255+0000 7fb303397640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb2fc111b60 con 0x7fb2fc1029d0 2024-04-21T23:18:46.257 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.255+0000 7fb303397640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb2fc1119e0 con 0x7fb2fc10bd10 2024-04-21T23:18:46.257 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.255+0000 7fb303397640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb2fc111ce0 con 0x7fb2fc1032f0 2024-04-21T23:18:46.257 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.255+0000 7fb301133640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2fc1029d0 0x7fb2fc100620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:46.257 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.255+0000 7fb301133640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2fc1029d0 0x7fb2fc100620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44092/0 (socket says 172.21.15.40:44092) 2024-04-21T23:18:46.257 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.255+0000 7fb301133640 1 -- 172.21.15.40:0/3411550284 learned_addr learned my addr 172.21.15.40:0/3411550284 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:46.257 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.255+0000 7fb301934640 1 --2- 172.21.15.40:0/3411550284 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2fc10bd10 0x7fb2fc1010a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:46.257 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.255+0000 7fb300932640 1 --2- 172.21.15.40:0/3411550284 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb2fc1032f0 0x7fb2fc100b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:46.257 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.256+0000 7fb301133640 1 -- 172.21.15.40:0/3411550284 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb2fc1032f0 msgr2=0x7fb2fc100b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:46.257 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.256+0000 7fb301133640 1 --2- 172.21.15.40:0/3411550284 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb2fc1032f0 0x7fb2fc100b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:46.257 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.256+0000 7fb301133640 1 -- 172.21.15.40:0/3411550284 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2fc10bd10 msgr2=0x7fb2fc1010a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:46.257 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.256+0000 7fb301133640 1 --2- 172.21.15.40:0/3411550284 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2fc10bd10 0x7fb2fc1010a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:46.258 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.256+0000 7fb301133640 1 -- 172.21.15.40:0/3411550284 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2fc1a64a0 con 0x7fb2fc1029d0 2024-04-21T23:18:46.258 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.256+0000 7fb300932640 1 --2- 172.21.15.40:0/3411550284 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb2fc1032f0 0x7fb2fc100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:18:46.258 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.256+0000 7fb301133640 1 --2- 172.21.15.40:0/3411550284 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2fc1029d0 0x7fb2fc100620 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fb2f8009a20 tx=0x7fb2f802dd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:46.258 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.256+0000 7fb2f27fc640 1 -- 172.21.15.40:0/3411550284 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2f8043020 con 0x7fb2fc1029d0 2024-04-21T23:18:46.258 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.256+0000 7fb303397640 1 -- 172.21.15.40:0/3411550284 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2fc1a6730 con 0x7fb2fc1029d0 2024-04-21T23:18:46.258 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.256+0000 7fb2f27fc640 1 -- 172.21.15.40:0/3411550284 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2f80054e0 con 0x7fb2fc1029d0 2024-04-21T23:18:46.258 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.256+0000 7fb2f27fc640 1 -- 172.21.15.40:0/3411550284 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2f8003bd0 con 0x7fb2fc1029d0 2024-04-21T23:18:46.258 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.256+0000 7fb303397640 1 -- 172.21.15.40:0/3411550284 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2fc1a6c80 con 0x7fb2fc1029d0 2024-04-21T23:18:46.264 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.259+0000 7fb2f27fc640 1 -- 172.21.15.40:0/3411550284 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7fb2f8046020 con 0x7fb2fc1029d0 2024-04-21T23:18:46.264 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.259+0000 7fb305f7d740 1 Processor -- start 2024-04-21T23:18:46.264 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.260+0000 7fb305f7d740 1 -- start start 2024-04-21T23:18:46.264 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.260+0000 7fb305f7d740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d887e12b20 0x55d887e12f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:46.264 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.260+0000 7fb305f7d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d887e13440 0x55d887e1bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:46.264 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.260+0000 7fb305f7d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d887e1c210 0x55d887e1e600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:46.264 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.260+0000 7fb305f7d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55d887d76b70 con 0x55d887e13440 2024-04-21T23:18:46.265 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.260+0000 7fb305f7d740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55d887d1f470 con 0x55d887e12b20 2024-04-21T23:18:46.265 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.260+0000 7fb305f7d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55d887d72b30 con 0x55d887e1c210 2024-04-21T23:18:46.265 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.260+0000 7fb301133640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d887e13440 0x55d887e1bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:46.265 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.260+0000 7fb2f27fc640 1 --2- 172.21.15.40:0/3411550284 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fb2e80a2ef0 0x7fb2e80a53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:46.265 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.260+0000 7fb301133640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d887e13440 0x55d887e1bcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44108/0 (socket says 172.21.15.40:44108) 2024-04-21T23:18:46.265 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.260+0000 7fb301133640 1 -- 172.21.15.40:0/1407047003 learned_addr learned my addr 172.21.15.40:0/1407047003 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:46.265 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.260+0000 7fb300932640 1 --2- 172.21.15.40:0/1407047003 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d887e1c210 0x55d887e1e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:46.265 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.260+0000 7fb301934640 1 --2- 172.21.15.40:0/3411550284 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fb2e80a2ef0 0x7fb2e80a53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:46.265 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.260+0000 7fb2f27fc640 1 -- 172.21.15.40:0/3411550284 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7fb2f80f1b00 con 0x7fb2fc1029d0 2024-04-21T23:18:46.265 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.260+0000 7fb301934640 1 --2- 172.21.15.40:0/1407047003 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d887e12b20 0x55d887e12f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:46.265 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.260+0000 7fb301133640 1 -- 172.21.15.40:0/1407047003 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d887e1c210 msgr2=0x55d887e1e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:46.265 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.260+0000 7fb301133640 1 --2- 172.21.15.40:0/1407047003 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d887e1c210 0x55d887e1e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:46.265 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.260+0000 7fb301133640 1 -- 172.21.15.40:0/1407047003 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d887e12b20 msgr2=0x55d887e12f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:46.265 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.260+0000 7fb301133640 1 --2- 172.21.15.40:0/1407047003 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d887e12b20 0x55d887e12f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:46.266 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.260+0000 7fb301133640 1 -- 172.21.15.40:0/1407047003 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d887a8b210 con 0x55d887e13440 2024-04-21T23:18:46.266 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.260+0000 7fb301934640 1 --2- 172.21.15.40:0/1407047003 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d887e12b20 0x55d887e12f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:18:46.266 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.261+0000 7fb301133640 1 --2- 172.21.15.40:0/1407047003 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d887e13440 0x55d887e1bcd0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fb2f80eee60 tx=0x7fb2f802bd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=301300cc1ee75394 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:46.266 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.261+0000 7fb301934640 1 --2- 172.21.15.40:0/3411550284 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fb2e80a2ef0 0x7fb2e80a53b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb2f4005fd0 tx=0x7fb2f4009b70 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:46.266 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.261+0000 7fb2e77fe640 1 -- 172.21.15.40:0/1407047003 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2f80bd430 con 0x55d887e13440 2024-04-21T23:18:46.266 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.261+0000 7fb2e77fe640 1 -- 172.21.15.40:0/1407047003 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2f8005690 con 0x55d887e13440 2024-04-21T23:18:46.266 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.261+0000 7fb2e77fe640 1 -- 172.21.15.40:0/1407047003 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2f80043c0 con 0x55d887e13440 2024-04-21T23:18:46.266 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.262+0000 7fb305f7d740 1 -- 172.21.15.40:0/1407047003 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d887e13440 msgr2=0x55d887e1bcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:46.266 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.262+0000 7fb305f7d740 1 --2- 172.21.15.40:0/1407047003 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d887e13440 0x55d887e1bcd0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fb2f80eee60 tx=0x7fb2f802bd60 comp rx=0 tx=0).stop 2024-04-21T23:18:46.266 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.262+0000 7fb305f7d740 1 -- 172.21.15.40:0/1407047003 shutdown_connections 2024-04-21T23:18:46.266 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.262+0000 7fb305f7d740 1 --2- 172.21.15.40:0/1407047003 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d887e1c210 0x55d887e1e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:46.266 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.262+0000 7fb305f7d740 1 --2- 172.21.15.40:0/1407047003 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d887e13440 0x55d887e1bcd0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:46.266 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.262+0000 7fb305f7d740 1 --2- 172.21.15.40:0/1407047003 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d887e12b20 0x55d887e12f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:46.267 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.262+0000 7fb305f7d740 1 -- 172.21.15.40:0/1407047003 >> 172.21.15.40:0/1407047003 conn(0x55d887e0d440 msgr2=0x55d887e0d820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:46.267 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.262+0000 7fb305f7d740 1 -- 172.21.15.40:0/1407047003 shutdown_connections 2024-04-21T23:18:46.267 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.262+0000 7fb305f7d740 1 -- 172.21.15.40:0/1407047003 wait complete. 2024-04-21T23:18:46.267 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.262+0000 7fb305f7d740 1 -- start start 2024-04-21T23:18:46.267 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.262+0000 7fb305f7d740 1 Processor -- start 2024-04-21T23:18:46.267 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.263+0000 7fb305f7d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d887e12b20 0x55d887e191f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:46.267 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.263+0000 7fb305f7d740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d887e13440 0x55d887e19730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:46.267 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.263+0000 7fb305f7d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d887e1c210 0x55d887e19c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:46.267 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.263+0000 7fb305f7d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55d887e1ec00 con 0x55d887e12b20 2024-04-21T23:18:46.267 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.263+0000 7fb305f7d740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55d887d32a10 con 0x55d887e13440 2024-04-21T23:18:46.267 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.263+0000 7fb305f7d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55d887e1ed80 con 0x55d887e1c210 2024-04-21T23:18:46.267 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.263+0000 7fb305f7d740 10 client.0 _ll_register_callbacks cb 0x55d887d6db60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:18:46.267 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.263+0000 7fb301133640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d887e12b20 0x55d887e191f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:46.267 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.263+0000 7fb301133640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d887e12b20 0x55d887e191f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44120/0 (socket says 172.21.15.40:44120) 2024-04-21T23:18:46.267 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.263+0000 7fb301133640 1 -- 172.21.15.40:0/3294584029 learned_addr learned my addr 172.21.15.40:0/3294584029 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:46.267 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.263+0000 7fb300932640 1 --2- 172.21.15.40:0/3294584029 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d887e13440 0x55d887e19730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:46.268 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.263+0000 7fb301934640 1 --2- 172.21.15.40:0/3294584029 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d887e1c210 0x55d887e19c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:46.268 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.263+0000 7fb301133640 1 -- 172.21.15.40:0/3294584029 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d887e1c210 msgr2=0x55d887e19c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:46.268 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.263+0000 7fb301133640 1 --2- 172.21.15.40:0/3294584029 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d887e1c210 0x55d887e19c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:46.268 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.263+0000 7fb301133640 1 -- 172.21.15.40:0/3294584029 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d887e13440 msgr2=0x55d887e19730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:46.268 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.263+0000 7fb301133640 1 --2- 172.21.15.40:0/3294584029 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d887e13440 0x55d887e19730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:46.268 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.263+0000 7fb301133640 1 -- 172.21.15.40:0/3294584029 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d887b57620 con 0x55d887e12b20 2024-04-21T23:18:46.268 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.263+0000 7fb301133640 1 --2- 172.21.15.40:0/3294584029 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d887e12b20 0x55d887e191f0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fb2f80b9f30 tx=0x7fb2f80c6740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3e5caa95724bb57 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:46.268 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.263+0000 7fb2e4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-21T23:18:46.268 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.263+0000 7fb2e4ff9640 1 -- 172.21.15.40:0/3294584029 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2f80c1e80 con 0x55d887e12b20 2024-04-21T23:18:46.268 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.263+0000 7fb2e4ff9640 1 -- 172.21.15.40:0/3294584029 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7fb2f80043c0 con 0x55d887e12b20 2024-04-21T23:18:46.268 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.263+0000 7fb305f7d740 1 -- 172.21.15.40:0/3294584029 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d887d840b0 con 0x55d887e12b20 2024-04-21T23:18:46.268 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.264+0000 7fb2e4ff9640 1 -- 172.21.15.40:0/3294584029 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fb2f80db050 con 0x55d887e12b20 2024-04-21T23:18:46.268 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.264+0000 7fb305f7d740 10 client.15735 fetch_fsmap learned FSMap version 25 2024-04-21T23:18:46.268 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.264+0000 7fb305f7d740 1 -- 172.21.15.40:0/3294584029 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55d887e1ec00 con 0x55d887e12b20 2024-04-21T23:18:46.268 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.264+0000 7fb2e4ff9640 1 -- 172.21.15.40:0/3294584029 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55d887d840b0 con 0x55d887e12b20 2024-04-21T23:18:46.269 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.264+0000 7fb2e4ff9640 1 -- 172.21.15.40:0/3294584029 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fb2f80dc240 con 0x55d887e12b20 2024-04-21T23:18:46.269 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.264+0000 7fb305f7d740 10 client.15735 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:18:46.269 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.264+0000 7fb305f7d740 10 client.15735 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:18:46.269 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.264+0000 7fb305f7d740 20 client.15735 populate_metadata read hostname 'smithi040' 2024-04-21T23:18:46.269 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.264+0000 7fb305f7d740 1 --2- 172.21.15.40:0/3294584029 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55d887e2bf70 0x55d887e2e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:46.269 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.264+0000 7fb305f7d740 4 client.15735 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:18:46.269 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.264+0000 7fb305f7d740 1 -- 172.21.15.40:0/3294584029 --> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d887d7e400 con 0x55d887e2bf70 2024-04-21T23:18:46.270 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.264+0000 7fb300932640 1 --2- 172.21.15.40:0/3294584029 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55d887e2bf70 0x55d887e2e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:46.270 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.265+0000 7fb2e4ff9640 10 client.15735 ms_handle_connect on v2:172.21.15.111:6838/3712235548 2024-04-21T23:18:46.270 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.265+0000 7fb300932640 1 --2- 172.21.15.40:0/3294584029 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55d887e2bf70 0x55d887e2e3c0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=107f84fb05e93421 server_cookie=6a74f4711ee6311b in_seq=0 out_seq=0 2024-04-21T23:18:46.271 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.265+0000 7fb2e4ff9640 1 -- 172.21.15.40:0/3294584029 <== mds.0 v2:172.21.15.111:6838/3712235548 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d887d7e400 con 0x55d887e2bf70 2024-04-21T23:18:46.271 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.265+0000 7fb2e4ff9640 10 client.15735 handle_command_reply: tid=0 2024-04-21T23:18:46.271 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.266+0000 7fb305f7d740 1 client.15735 shutdown 2024-04-21T23:18:46.271 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.266+0000 7fb305f7d740 1 -- 172.21.15.40:0/3294584029 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d887e12b20 msgr2=0x55d887e191f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:46.271 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.266+0000 7fb305f7d740 1 --2- 172.21.15.40:0/3294584029 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d887e12b20 0x55d887e191f0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fb2f80b9f30 tx=0x7fb2f80c6740 comp rx=0 tx=0).stop 2024-04-21T23:18:46.271 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.266+0000 7fb305f7d740 1 -- 172.21.15.40:0/3294584029 shutdown_connections 2024-04-21T23:18:46.271 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.266+0000 7fb305f7d740 1 --2- 172.21.15.40:0/3294584029 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55d887e2bf70 0x55d887e2e3c0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:46.271 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.266+0000 7fb305f7d740 1 --2- 172.21.15.40:0/3294584029 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d887e13440 0x55d887e19730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:46.271 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.266+0000 7fb305f7d740 1 --2- 172.21.15.40:0/3294584029 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d887e1c210 0x55d887e19c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:46.271 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.266+0000 7fb305f7d740 1 --2- 172.21.15.40:0/3294584029 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d887e12b20 0x55d887e191f0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:46.272 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.266+0000 7fb2e4ff9640 0 client.15735 ms_handle_reset on v2:172.21.15.111:6838/3712235548 2024-04-21T23:18:46.272 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.266+0000 7fb305f7d740 1 -- 172.21.15.40:0/3294584029 >> 172.21.15.40:0/3294584029 conn(0x55d887e0d440 msgr2=0x55d887e0f190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:46.272 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.266+0000 7fb305f7d740 1 -- 172.21.15.40:0/3294584029 shutdown_connections 2024-04-21T23:18:46.272 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.266+0000 7fb305f7d740 1 -- 172.21.15.40:0/3294584029 wait complete. 2024-04-21T23:18:46.272 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.266+0000 7fb305f7d740 20 client.15735 trim_cache size 0 max 16384 2024-04-21T23:18:46.282 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.281+0000 7fb305f7d740 1 Processor -- start 2024-04-21T23:18:46.283 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.281+0000 7fb305f7d740 1 -- start start 2024-04-21T23:18:46.283 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.281+0000 7fb305f7d740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d887e12b20 0x55d887e1b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:46.283 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.281+0000 7fb305f7d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d887e13440 0x55d887e27a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:46.283 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.281+0000 7fb305f7d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d887e1c210 0x55d887e0efa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:46.283 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.281+0000 7fb305f7d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb2fc1b3690 con 0x55d887e1c210 2024-04-21T23:18:46.283 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.281+0000 7fb305f7d740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb2fc1b0f60 con 0x55d887e12b20 2024-04-21T23:18:46.283 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.281+0000 7fb305f7d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb2fc1aedb0 con 0x55d887e13440 2024-04-21T23:18:46.283 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.282+0000 7fb301934640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d887e1c210 0x55d887e0efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:46.283 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.282+0000 7fb301934640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d887e1c210 0x55d887e0efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44124/0 (socket says 172.21.15.40:44124) 2024-04-21T23:18:46.283 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.282+0000 7fb301934640 1 -- 172.21.15.40:0/3144645623 learned_addr learned my addr 172.21.15.40:0/3144645623 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:46.283 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.282+0000 7fb301133640 1 --2- 172.21.15.40:0/3144645623 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d887e12b20 0x55d887e1b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:46.284 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.282+0000 7fb300932640 1 --2- 172.21.15.40:0/3144645623 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d887e13440 0x55d887e27a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:46.284 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.282+0000 7fb301934640 1 -- 172.21.15.40:0/3144645623 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d887e13440 msgr2=0x55d887e27a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:46.284 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.282+0000 7fb301934640 1 --2- 172.21.15.40:0/3144645623 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d887e13440 0x55d887e27a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:46.284 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.282+0000 7fb301934640 1 -- 172.21.15.40:0/3144645623 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d887e12b20 msgr2=0x55d887e1b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:46.284 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.282+0000 7fb301934640 1 --2- 172.21.15.40:0/3144645623 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d887e12b20 0x55d887e1b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:46.284 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.282+0000 7fb301934640 1 -- 172.21.15.40:0/3144645623 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d887e17740 con 0x55d887e1c210 2024-04-21T23:18:46.284 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.282+0000 7fb301934640 1 --2- 172.21.15.40:0/3144645623 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d887e1c210 0x55d887e0efa0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fb2f4009260 tx=0x7fb2f4009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d7ac2ba9722bd6b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:46.284 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.282+0000 7fb2e6ffd640 1 -- 172.21.15.40:0/3144645623 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2f4003e00 con 0x55d887e1c210 2024-04-21T23:18:46.284 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.282+0000 7fb2e6ffd640 1 -- 172.21.15.40:0/3144645623 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2f4004530 con 0x55d887e1c210 2024-04-21T23:18:46.284 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.282+0000 7fb2e6ffd640 1 -- 172.21.15.40:0/3144645623 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2f4020c90 con 0x55d887e1c210 2024-04-21T23:18:46.284 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.283+0000 7fb305f7d740 1 -- 172.21.15.40:0/3144645623 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d887e1c210 msgr2=0x55d887e0efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:46.284 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.283+0000 7fb305f7d740 1 --2- 172.21.15.40:0/3144645623 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d887e1c210 0x55d887e0efa0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fb2f4009260 tx=0x7fb2f4009500 comp rx=0 tx=0).stop 2024-04-21T23:18:46.284 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.283+0000 7fb305f7d740 1 -- 172.21.15.40:0/3144645623 shutdown_connections 2024-04-21T23:18:46.284 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.283+0000 7fb305f7d740 1 --2- 172.21.15.40:0/3144645623 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d887e1c210 0x55d887e0efa0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:46.285 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.283+0000 7fb305f7d740 1 --2- 172.21.15.40:0/3144645623 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d887e13440 0x55d887e27a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:46.285 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.283+0000 7fb305f7d740 1 --2- 172.21.15.40:0/3144645623 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d887e12b20 0x55d887e1b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:46.285 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.283+0000 7fb305f7d740 1 -- 172.21.15.40:0/3144645623 >> 172.21.15.40:0/3144645623 conn(0x55d887e0d440 msgr2=0x55d887e0e890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:46.285 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.283+0000 7fb305f7d740 1 -- 172.21.15.40:0/3144645623 shutdown_connections 2024-04-21T23:18:46.285 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.283+0000 7fb305f7d740 1 -- 172.21.15.40:0/3144645623 wait complete. 2024-04-21T23:18:46.285 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.283+0000 7fb305f7d740 1 -- start start 2024-04-21T23:18:46.285 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.283+0000 7fb305f7d740 1 Processor -- start 2024-04-21T23:18:46.288 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.284+0000 7fb305f7d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d887e12b20 0x55d887e23e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:46.288 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.284+0000 7fb305f7d740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d887e13440 0x55d887e24350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:46.288 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.284+0000 7fb305f7d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d887e1c210 0x55d887e20de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:46.288 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.284+0000 7fb305f7d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb2fc1af420 con 0x55d887e12b20 2024-04-21T23:18:46.288 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.284+0000 7fb305f7d740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb2fc1b0890 con 0x55d887e13440 2024-04-21T23:18:46.288 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.284+0000 7fb305f7d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb2fc1afaf0 con 0x55d887e1c210 2024-04-21T23:18:46.288 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.284+0000 7fb305f7d740 10 client.0 _ll_register_callbacks cb 0x7fb2f80bbf40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:18:46.288 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.284+0000 7fb301133640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d887e12b20 0x55d887e23e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:46.288 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.284+0000 7fb301133640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d887e12b20 0x55d887e23e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44138/0 (socket says 172.21.15.40:44138) 2024-04-21T23:18:46.288 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.284+0000 7fb301133640 1 -- 172.21.15.40:0/3484257995 learned_addr learned my addr 172.21.15.40:0/3484257995 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:46.288 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.284+0000 7fb301934640 1 --2- 172.21.15.40:0/3484257995 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d887e1c210 0x55d887e20de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:46.289 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.284+0000 7fb301133640 1 -- 172.21.15.40:0/3484257995 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d887e1c210 msgr2=0x55d887e20de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:46.289 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.284+0000 7fb300932640 1 --2- 172.21.15.40:0/3484257995 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d887e13440 0x55d887e24350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:46.289 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.284+0000 7fb301133640 1 --2- 172.21.15.40:0/3484257995 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d887e1c210 0x55d887e20de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:46.289 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.284+0000 7fb301133640 1 -- 172.21.15.40:0/3484257995 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d887e13440 msgr2=0x55d887e24350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:46.289 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.284+0000 7fb301133640 1 --2- 172.21.15.40:0/3484257995 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d887e13440 0x55d887e24350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:46.289 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.284+0000 7fb301133640 1 -- 172.21.15.40:0/3484257995 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d887e21320 con 0x55d887e12b20 2024-04-21T23:18:46.289 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.284+0000 7fb300932640 1 --2- 172.21.15.40:0/3484257995 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d887e13440 0x55d887e24350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:18:46.289 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.284+0000 7fb301133640 1 --2- 172.21.15.40:0/3484257995 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d887e12b20 0x55d887e23e10 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fb2f80bd3c0 tx=0x7fb2f80045f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=638a534b8e9d64ac server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:46.289 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.284+0000 7fb2e4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-21T23:18:46.289 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.285+0000 7fb2e4ff9640 1 -- 172.21.15.40:0/3484257995 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2f80b9d70 con 0x55d887e12b20 2024-04-21T23:18:46.289 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.285+0000 7fb2e4ff9640 1 -- 172.21.15.40:0/3484257995 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7fb2f80c64c0 con 0x55d887e12b20 2024-04-21T23:18:46.289 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.285+0000 7fb305f7d740 1 -- 172.21.15.40:0/3484257995 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb2f4004530 con 0x55d887e12b20 2024-04-21T23:18:46.289 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.285+0000 7fb2e4ff9640 1 -- 172.21.15.40:0/3484257995 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fb2f80be1d0 con 0x55d887e12b20 2024-04-21T23:18:46.289 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.285+0000 7fb305f7d740 10 client.15741 fetch_fsmap learned FSMap version 25 2024-04-21T23:18:46.290 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.285+0000 7fb305f7d740 1 -- 172.21.15.40:0/3484257995 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55d887e21320 con 0x55d887e12b20 2024-04-21T23:18:46.290 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.285+0000 7fb2e4ff9640 1 -- 172.21.15.40:0/3484257995 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2f4004530 con 0x55d887e12b20 2024-04-21T23:18:46.290 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.285+0000 7fb2e4ff9640 1 -- 172.21.15.40:0/3484257995 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fb2f80bc990 con 0x55d887e12b20 2024-04-21T23:18:46.290 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.285+0000 7fb305f7d740 10 client.15741 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:18:46.290 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.285+0000 7fb305f7d740 10 client.15741 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:18:46.290 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.285+0000 7fb305f7d740 20 client.15741 populate_metadata read hostname 'smithi040' 2024-04-21T23:18:46.290 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.285+0000 7fb305f7d740 1 --2- 172.21.15.40:0/3484257995 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55d887e93060 0x55d887e95450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:46.290 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.285+0000 7fb305f7d740 4 client.15741 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T23:18:46.290 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.285+0000 7fb305f7d740 1 -- 172.21.15.40:0/3484257995 --> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d887cfc7a0 con 0x55d887e93060 2024-04-21T23:18:46.290 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.285+0000 7fb300932640 1 --2- 172.21.15.40:0/3484257995 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55d887e93060 0x55d887e95450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:46.290 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.286+0000 7fb300932640 1 --2- 172.21.15.40:0/3484257995 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55d887e93060 0x55d887e95450 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=55b08381185264c8 server_cookie=7061ca3286be059e in_seq=0 out_seq=0 2024-04-21T23:18:46.290 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.286+0000 7fb2e4ff9640 10 client.15741 ms_handle_connect on v2:172.21.15.111:6838/3712235548 2024-04-21T23:18:46.301 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.299+0000 7fb2e4ff9640 1 -- 172.21.15.40:0/3484257995 <== mds.0 v2:172.21.15.111:6838/3712235548 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d887cfc7a0 con 0x55d887e93060 2024-04-21T23:18:46.301 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.299+0000 7fb2e4ff9640 10 client.15741 handle_command_reply: tid=0 2024-04-21T23:18:46.301 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.299+0000 7fb305f7d740 1 client.15741 shutdown 2024-04-21T23:18:46.302 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.299+0000 7fb305f7d740 1 -- 172.21.15.40:0/3484257995 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d887e12b20 msgr2=0x55d887e23e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:46.302 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.299+0000 7fb305f7d740 1 --2- 172.21.15.40:0/3484257995 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d887e12b20 0x55d887e23e10 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fb2f80bd3c0 tx=0x7fb2f80045f0 comp rx=0 tx=0).stop 2024-04-21T23:18:46.302 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.299+0000 7fb305f7d740 1 -- 172.21.15.40:0/3484257995 shutdown_connections 2024-04-21T23:18:46.302 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.299+0000 7fb305f7d740 1 --2- 172.21.15.40:0/3484257995 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55d887e93060 0x55d887e95450 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:46.302 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.299+0000 7fb305f7d740 1 --2- 172.21.15.40:0/3484257995 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d887e13440 0x55d887e24350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:46.302 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.299+0000 7fb2e4ff9640 0 client.15741 ms_handle_reset on v2:172.21.15.111:6838/3712235548 2024-04-21T23:18:46.302 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.299+0000 7fb305f7d740 1 --2- 172.21.15.40:0/3484257995 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d887e1c210 0x55d887e20de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:46.302 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.299+0000 7fb305f7d740 1 --2- 172.21.15.40:0/3484257995 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d887e12b20 0x55d887e23e10 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:46.302 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.299+0000 7fb305f7d740 1 -- 172.21.15.40:0/3484257995 >> 172.21.15.40:0/3484257995 conn(0x55d887e0d440 msgr2=0x55d887e0c4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:46.302 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.299+0000 7fb305f7d740 1 -- 172.21.15.40:0/3484257995 shutdown_connections 2024-04-21T23:18:46.302 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.299+0000 7fb305f7d740 1 -- 172.21.15.40:0/3484257995 wait complete. 2024-04-21T23:18:46.302 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.299+0000 7fb305f7d740 20 client.15741 trim_cache size 0 max 16384 2024-04-21T23:18:46.303 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:{ 2024-04-21T23:18:46.303 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "message": "", 2024-04-21T23:18:46.303 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "return_code": 0 2024-04-21T23:18:46.303 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:} 2024-04-21T23:18:46.304 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.300+0000 7fb303397640 1 -- 172.21.15.40:0/3411550284 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fb2e80a2ef0 msgr2=0x7fb2e80a53b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:46.304 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.300+0000 7fb303397640 1 --2- 172.21.15.40:0/3411550284 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fb2e80a2ef0 0x7fb2e80a53b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb2f4005fd0 tx=0x7fb2f4009b70 comp rx=0 tx=0).stop 2024-04-21T23:18:46.304 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.300+0000 7fb303397640 1 -- 172.21.15.40:0/3411550284 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2fc1029d0 msgr2=0x7fb2fc100620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:46.304 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.300+0000 7fb303397640 1 --2- 172.21.15.40:0/3411550284 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2fc1029d0 0x7fb2fc100620 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fb2f8009a20 tx=0x7fb2f802dd30 comp rx=0 tx=0).stop 2024-04-21T23:18:46.304 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.300+0000 7fb303397640 1 -- 172.21.15.40:0/3411550284 shutdown_connections 2024-04-21T23:18:46.304 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.300+0000 7fb303397640 1 --2- 172.21.15.40:0/3411550284 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fb2e80a2ef0 0x7fb2e80a53b0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:46.304 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.300+0000 7fb303397640 1 --2- 172.21.15.40:0/3411550284 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2fc10bd10 0x7fb2fc1010a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:46.304 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.300+0000 7fb303397640 1 --2- 172.21.15.40:0/3411550284 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb2fc1032f0 0x7fb2fc100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:46.304 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.300+0000 7fb303397640 1 --2- 172.21.15.40:0/3411550284 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2fc1029d0 0x7fb2fc100620 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:46.304 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.300+0000 7fb303397640 1 -- 172.21.15.40:0/3411550284 >> 172.21.15.40:0/3411550284 conn(0x7fb2fc0777c0 msgr2=0x7fb2fc0fd710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:46.305 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.300+0000 7fb303397640 1 -- 172.21.15.40:0/3411550284 shutdown_connections 2024-04-21T23:18:46.305 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:46.300+0000 7fb303397640 1 -- 172.21.15.40:0/3411550284 wait complete. 2024-04-21T23:18:47.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:47 smithi040 ceph-mon[27984]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 39 op/s 2024-04-21T23:18:47.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:47 smithi174 ceph-mon[31409]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 39 op/s 2024-04-21T23:18:47.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:47 smithi111 ceph-mon[31085]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 39 op/s 2024-04-21T23:18:49.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:49 smithi040 ceph-mon[27984]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:18:49.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:49 smithi174 ceph-mon[31409]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:18:49.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:49 smithi111 ceph-mon[31085]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:18:51.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:51 smithi040 ceph-mon[27984]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-04-21T23:18:51.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:51 smithi174 ceph-mon[31409]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-04-21T23:18:51.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:51 smithi111 ceph-mon[31085]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-04-21T23:18:53.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:53 smithi040 ceph-mon[27984]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.6 KiB/s wr, 10 op/s 2024-04-21T23:18:53.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:53 smithi174 ceph-mon[31409]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.6 KiB/s wr, 10 op/s 2024-04-21T23:18:53.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:53 smithi111 ceph-mon[31085]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.6 KiB/s wr, 10 op/s 2024-04-21T23:18:55.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:55 smithi040 ceph-mon[27984]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.6 KiB/s wr, 9 op/s 2024-04-21T23:18:55.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:55 smithi174 ceph-mon[31409]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.6 KiB/s wr, 9 op/s 2024-04-21T23:18:55.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:55 smithi111 ceph-mon[31085]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.6 KiB/s wr, 9 op/s 2024-04-21T23:18:57.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:57 smithi040 ceph-mon[27984]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.6 KiB/s wr, 12 op/s 2024-04-21T23:18:57.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:57 smithi174 ceph-mon[31409]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.6 KiB/s wr, 12 op/s 2024-04-21T23:18:57.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:57 smithi111 ceph-mon[31085]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.6 KiB/s wr, 12 op/s 2024-04-21T23:18:59.399 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.400+0000 7fb501815640 1 -- 172.21.15.40:0/2814172073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4fc1020e0 msgr2=0x7fb4fc10e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:59.399 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.400+0000 7fb501815640 1 --2- 172.21.15.40:0/2814172073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4fc1020e0 0x7fb4fc10e5f0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fb4f4009cd0 tx=0x7fb4f402c0f0 comp rx=0 tx=0).stop 2024-04-21T23:18:59.400 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.400+0000 7fb501815640 1 -- 172.21.15.40:0/2814172073 shutdown_connections 2024-04-21T23:18:59.400 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.400+0000 7fb501815640 1 --2- 172.21.15.40:0/2814172073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4fc1020e0 0x7fb4fc10e5f0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:59.400 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.400+0000 7fb501815640 1 --2- 172.21.15.40:0/2814172073 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb4fc101720 0x7fb4fc101ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:59.400 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.400+0000 7fb501815640 1 --2- 172.21.15.40:0/2814172073 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4fc107730 0x7fb4fc107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:59.400 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.400+0000 7fb501815640 1 -- 172.21.15.40:0/2814172073 >> 172.21.15.40:0/2814172073 conn(0x7fb4fc0fd1f0 msgr2=0x7fb4fc0ff610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:59.400 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.400+0000 7fb501815640 1 -- 172.21.15.40:0/2814172073 shutdown_connections 2024-04-21T23:18:59.400 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.400+0000 7fb501815640 1 -- 172.21.15.40:0/2814172073 wait complete. 2024-04-21T23:18:59.400 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.400+0000 7fb501815640 1 Processor -- start 2024-04-21T23:18:59.402 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.401+0000 7fb501815640 1 -- start start 2024-04-21T23:18:59.402 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.401+0000 7fb501815640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb4fc101720 0x7fb4fc197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:59.402 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.401+0000 7fb501815640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4fc1020e0 0x7fb4fc198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:59.402 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.401+0000 7fb501815640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4fc107730 0x7fb4fc19c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:59.402 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.401+0000 7fb501815640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb4fc111b40 con 0x7fb4fc107730 2024-04-21T23:18:59.402 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.401+0000 7fb501815640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb4fc1119c0 con 0x7fb4fc1020e0 2024-04-21T23:18:59.402 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.401+0000 7fb501815640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb4fc111cc0 con 0x7fb4fc101720 2024-04-21T23:18:59.402 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.401+0000 7fb4fb7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4fc107730 0x7fb4fc19c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:59.403 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.401+0000 7fb4fb7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4fc107730 0x7fb4fc19c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46632/0 (socket says 172.21.15.40:46632) 2024-04-21T23:18:59.403 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.401+0000 7fb4fb7fe640 1 -- 172.21.15.40:0/3049896923 learned_addr learned my addr 172.21.15.40:0/3049896923 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:59.403 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.401+0000 7fb4fa7fc640 1 --2- 172.21.15.40:0/3049896923 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4fc1020e0 0x7fb4fc198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:59.403 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.401+0000 7fb4faffd640 1 --2- 172.21.15.40:0/3049896923 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb4fc101720 0x7fb4fc197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:59.403 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.401+0000 7fb4fb7fe640 1 -- 172.21.15.40:0/3049896923 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb4fc101720 msgr2=0x7fb4fc197f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:59.403 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.401+0000 7fb4fb7fe640 1 --2- 172.21.15.40:0/3049896923 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb4fc101720 0x7fb4fc197f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:59.403 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.401+0000 7fb4fb7fe640 1 -- 172.21.15.40:0/3049896923 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4fc1020e0 msgr2=0x7fb4fc198460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:59.403 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.401+0000 7fb4fb7fe640 1 --2- 172.21.15.40:0/3049896923 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4fc1020e0 0x7fb4fc198460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:59.403 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.401+0000 7fb4fb7fe640 1 -- 172.21.15.40:0/3049896923 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4fc19cf10 con 0x7fb4fc107730 2024-04-21T23:18:59.403 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.402+0000 7fb4fb7fe640 1 --2- 172.21.15.40:0/3049896923 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4fc107730 0x7fb4fc19c7e0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fb4f4005e30 tx=0x7fb4f4032ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:59.403 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.402+0000 7fb500813640 1 -- 172.21.15.40:0/3049896923 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4f4041020 con 0x7fb4fc107730 2024-04-21T23:18:59.403 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.402+0000 7fb500813640 1 -- 172.21.15.40:0/3049896923 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb4f4034cd0 con 0x7fb4fc107730 2024-04-21T23:18:59.403 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.402+0000 7fb501815640 1 -- 172.21.15.40:0/3049896923 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4fc19d1a0 con 0x7fb4fc107730 2024-04-21T23:18:59.404 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.402+0000 7fb500813640 1 -- 172.21.15.40:0/3049896923 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4f4039040 con 0x7fb4fc107730 2024-04-21T23:18:59.404 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.402+0000 7fb501815640 1 -- 172.21.15.40:0/3049896923 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4fc1a4a90 con 0x7fb4fc107730 2024-04-21T23:18:59.406 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.405+0000 7fb500813640 1 -- 172.21.15.40:0/3049896923 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7fb4f4044020 con 0x7fb4fc107730 2024-04-21T23:18:59.407 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.405+0000 7fb504402740 1 Processor -- start 2024-04-21T23:18:59.407 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.405+0000 7fb504402740 1 -- start start 2024-04-21T23:18:59.407 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.405+0000 7fb500813640 1 --2- 172.21.15.40:0/3049896923 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fb4e40a2f40 0x7fb4e40a5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:59.407 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.405+0000 7fb504402740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565053b23640 0x565053b23a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:59.407 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.405+0000 7fb504402740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565053b23f60 0x565053b310e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:59.407 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.405+0000 7fb504402740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565053b31620 0x565053b33a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:59.407 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.405+0000 7fb504402740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x565053a87b70 con 0x565053b31620 2024-04-21T23:18:59.407 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.405+0000 7fb504402740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x565053a30470 con 0x565053b23f60 2024-04-21T23:18:59.407 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.405+0000 7fb504402740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x565053a83b30 con 0x565053b23640 2024-04-21T23:18:59.407 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.406+0000 7fb500813640 1 -- 172.21.15.40:0/3049896923 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7fb4f40f1790 con 0x7fb4fc107730 2024-04-21T23:18:59.407 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.406+0000 7fb4fb7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565053b31620 0x565053b33a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:59.407 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.406+0000 7fb4fa7fc640 1 --2- 172.21.15.40:0/3049896923 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fb4e40a2f40 0x7fb4e40a5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:59.407 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.406+0000 7fb4faffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565053b23640 0x565053b23a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:59.407 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.406+0000 7fb4fb7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565053b31620 0x565053b33a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46636/0 (socket says 172.21.15.40:46636) 2024-04-21T23:18:59.408 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.406+0000 7fb4fb7fe640 1 -- 172.21.15.40:0/686856323 learned_addr learned my addr 172.21.15.40:0/686856323 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:59.408 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.406+0000 7fb4fa7fc640 1 --2- 172.21.15.40:0/686856323 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565053b23f60 0x565053b310e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:59.408 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.406+0000 7fb4fb7fe640 1 -- 172.21.15.40:0/686856323 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565053b23640 msgr2=0x565053b23a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:59.408 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.406+0000 7fb4fb7fe640 1 --2- 172.21.15.40:0/686856323 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565053b23640 0x565053b23a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:59.408 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.406+0000 7fb4fb7fe640 1 -- 172.21.15.40:0/686856323 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565053b23f60 msgr2=0x565053b310e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:59.408 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.406+0000 7fb4fb7fe640 1 --2- 172.21.15.40:0/686856323 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565053b23f60 0x565053b310e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:59.408 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.406+0000 7fb4fb7fe640 1 -- 172.21.15.40:0/686856323 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56505379c210 con 0x565053b31620 2024-04-21T23:18:59.408 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.406+0000 7fb4fa7fc640 1 --2- 172.21.15.40:0/3049896923 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fb4e40a2f40 0x7fb4e40a5400 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb4f000b620 tx=0x7fb4f000bb50 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:59.408 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.406+0000 7fb4fb7fe640 1 --2- 172.21.15.40:0/686856323 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565053b31620 0x565053b33a10 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fb4f4009bf0 tx=0x7fb4f4003d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30f533b3a05681e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:59.409 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.406+0000 7fb4e1ffb640 1 -- 172.21.15.40:0/686856323 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4f403d5d0 con 0x565053b31620 2024-04-21T23:18:59.409 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.407+0000 7fb4e1ffb640 1 -- 172.21.15.40:0/686856323 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb4f40bdec0 con 0x565053b31620 2024-04-21T23:18:59.409 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.407+0000 7fb4e1ffb640 1 -- 172.21.15.40:0/686856323 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4f403d3f0 con 0x565053b31620 2024-04-21T23:18:59.409 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.407+0000 7fb504402740 1 -- 172.21.15.40:0/686856323 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565053b31620 msgr2=0x565053b33a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:59.409 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.407+0000 7fb504402740 1 --2- 172.21.15.40:0/686856323 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565053b31620 0x565053b33a10 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fb4f4009bf0 tx=0x7fb4f4003d10 comp rx=0 tx=0).stop 2024-04-21T23:18:59.409 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.407+0000 7fb504402740 1 -- 172.21.15.40:0/686856323 shutdown_connections 2024-04-21T23:18:59.409 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.407+0000 7fb504402740 1 --2- 172.21.15.40:0/686856323 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565053b23f60 0x565053b310e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:59.409 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.407+0000 7fb504402740 1 --2- 172.21.15.40:0/686856323 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565053b31620 0x565053b33a10 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:59.409 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.407+0000 7fb504402740 1 --2- 172.21.15.40:0/686856323 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565053b23640 0x565053b23a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:59.409 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.407+0000 7fb504402740 1 -- 172.21.15.40:0/686856323 >> 172.21.15.40:0/686856323 conn(0x565053b1e440 msgr2=0x565053b1e820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:59.409 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.407+0000 7fb504402740 1 -- 172.21.15.40:0/686856323 shutdown_connections 2024-04-21T23:18:59.409 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.407+0000 7fb504402740 1 -- 172.21.15.40:0/686856323 wait complete. 2024-04-21T23:18:59.409 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.408+0000 7fb504402740 1 -- start start 2024-04-21T23:18:59.414 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.408+0000 7fb504402740 1 Processor -- start 2024-04-21T23:18:59.414 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.408+0000 7fb504402740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565053b23640 0x565053b2e580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:59.414 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.408+0000 7fb504402740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565053b23f60 0x565053b2eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:59.414 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.408+0000 7fb504402740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565053b31620 0x565053b2f000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:59.414 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.408+0000 7fb504402740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x565053b33fa0 con 0x565053b31620 2024-04-21T23:18:59.414 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.408+0000 7fb504402740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x565053a43a10 con 0x565053b23f60 2024-04-21T23:18:59.414 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.408+0000 7fb504402740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x565053b34120 con 0x565053b23640 2024-04-21T23:18:59.414 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.408+0000 7fb504402740 10 client.0 _ll_register_callbacks cb 0x565053a7eb60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:18:59.414 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.409+0000 7fb4fb7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565053b31620 0x565053b2f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:59.414 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.409+0000 7fb4faffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565053b23640 0x565053b2e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:59.414 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.409+0000 7fb4fa7fc640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565053b23f60 0x565053b2eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:59.414 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.409+0000 7fb4fb7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565053b31620 0x565053b2f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46644/0 (socket says 172.21.15.40:46644) 2024-04-21T23:18:59.415 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.409+0000 7fb4faffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565053b23640 0x565053b2e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.40:47070/0 (socket says 172.21.15.40:47070) 2024-04-21T23:18:59.415 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.409+0000 7fb4fb7fe640 1 -- 172.21.15.40:0/1370644564 learned_addr learned my addr 172.21.15.40:0/1370644564 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:59.415 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.409+0000 7fb4fb7fe640 1 -- 172.21.15.40:0/1370644564 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565053b23640 msgr2=0x565053b2e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:59.415 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.409+0000 7fb4fb7fe640 1 --2- 172.21.15.40:0/1370644564 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565053b23640 0x565053b2e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:59.415 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.409+0000 7fb4fb7fe640 1 -- 172.21.15.40:0/1370644564 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565053b23f60 msgr2=0x565053b2eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:59.415 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.409+0000 7fb4fb7fe640 1 --2- 172.21.15.40:0/1370644564 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565053b23f60 0x565053b2eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:59.415 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.409+0000 7fb4fb7fe640 1 -- 172.21.15.40:0/1370644564 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565053868620 con 0x565053b31620 2024-04-21T23:18:59.415 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.409+0000 7fb4fa7fc640 1 --2- 172.21.15.40:0/1370644564 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565053b23f60 0x565053b2eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:18:59.415 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.409+0000 7fb4fb7fe640 1 --2- 172.21.15.40:0/1370644564 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565053b31620 0x565053b2f000 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fb4f40ef130 tx=0x7fb4f4009990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c8b5c7d709e49e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:59.415 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.409+0000 7fb4daffd640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-21T23:18:59.415 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.409+0000 7fb4daffd640 1 -- 172.21.15.40:0/1370644564 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4f40db060 con 0x565053b31620 2024-04-21T23:18:59.415 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.409+0000 7fb4daffd640 1 -- 172.21.15.40:0/1370644564 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7fb4f40f1590 con 0x565053b31620 2024-04-21T23:18:59.415 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.409+0000 7fb504402740 1 -- 172.21.15.40:0/1370644564 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb4f40bdec0 con 0x565053b31620 2024-04-21T23:18:59.415 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.409+0000 7fb4daffd640 1 -- 172.21.15.40:0/1370644564 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fb4f403d3f0 con 0x565053b31620 2024-04-21T23:18:59.416 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.409+0000 7fb504402740 10 client.15753 fetch_fsmap learned FSMap version 25 2024-04-21T23:18:59.416 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.409+0000 7fb504402740 1 -- 172.21.15.40:0/1370644564 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x565053b33fa0 con 0x565053b31620 2024-04-21T23:18:59.416 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.410+0000 7fb4daffd640 1 -- 172.21.15.40:0/1370644564 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb4f40bdec0 con 0x565053b31620 2024-04-21T23:18:59.416 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.410+0000 7fb4daffd640 1 -- 172.21.15.40:0/1370644564 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fb4f40d1ce0 con 0x565053b31620 2024-04-21T23:18:59.416 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.410+0000 7fb504402740 10 client.15753 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:18:59.416 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.410+0000 7fb504402740 10 client.15753 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:18:59.416 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.410+0000 7fb504402740 20 client.15753 populate_metadata read hostname 'smithi040' 2024-04-21T23:18:59.416 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.410+0000 7fb504402740 1 --2- 172.21.15.40:0/1370644564 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x565053b412e0 0x565053b436d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:59.416 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.410+0000 7fb504402740 4 client.15753 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:18:59.416 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.410+0000 7fb504402740 1 -- 172.21.15.40:0/1370644564 --> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x565053a950b0 con 0x565053b412e0 2024-04-21T23:18:59.416 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.410+0000 7fb4faffd640 1 --2- 172.21.15.40:0/1370644564 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x565053b412e0 0x565053b436d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:59.416 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.411+0000 7fb4faffd640 1 --2- 172.21.15.40:0/1370644564 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x565053b412e0 0x565053b436d0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=72f9528ed152c884 server_cookie=b78d9e2106dd6202 in_seq=0 out_seq=0 2024-04-21T23:18:59.416 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.411+0000 7fb4daffd640 10 client.15753 ms_handle_connect on v2:172.21.15.111:6838/3712235548 2024-04-21T23:18:59.416 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.411+0000 7fb4daffd640 1 -- 172.21.15.40:0/1370644564 <== mds.0 v2:172.21.15.111:6838/3712235548 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x565053a950b0 con 0x565053b412e0 2024-04-21T23:18:59.417 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.411+0000 7fb4daffd640 10 client.15753 handle_command_reply: tid=0 2024-04-21T23:18:59.417 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.411+0000 7fb504402740 1 client.15753 shutdown 2024-04-21T23:18:59.417 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.412+0000 7fb504402740 1 -- 172.21.15.40:0/1370644564 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565053b31620 msgr2=0x565053b2f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:59.417 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.412+0000 7fb504402740 1 --2- 172.21.15.40:0/1370644564 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565053b31620 0x565053b2f000 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fb4f40ef130 tx=0x7fb4f4009990 comp rx=0 tx=0).stop 2024-04-21T23:18:59.417 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.412+0000 7fb504402740 1 -- 172.21.15.40:0/1370644564 shutdown_connections 2024-04-21T23:18:59.417 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.412+0000 7fb504402740 1 --2- 172.21.15.40:0/1370644564 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x565053b412e0 0x565053b436d0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:59.417 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.412+0000 7fb504402740 1 --2- 172.21.15.40:0/1370644564 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565053b23f60 0x565053b2eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:59.417 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.412+0000 7fb504402740 1 --2- 172.21.15.40:0/1370644564 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565053b31620 0x565053b2f000 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:59.417 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.412+0000 7fb504402740 1 --2- 172.21.15.40:0/1370644564 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565053b23640 0x565053b2e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:59.417 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.412+0000 7fb504402740 1 -- 172.21.15.40:0/1370644564 >> 172.21.15.40:0/1370644564 conn(0x565053b1e440 msgr2=0x565053b20190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:59.417 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.412+0000 7fb4daffd640 0 client.15753 ms_handle_reset on v2:172.21.15.111:6838/3712235548 2024-04-21T23:18:59.417 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.412+0000 7fb504402740 1 -- 172.21.15.40:0/1370644564 shutdown_connections 2024-04-21T23:18:59.417 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.412+0000 7fb504402740 1 -- 172.21.15.40:0/1370644564 wait complete. 2024-04-21T23:18:59.417 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.412+0000 7fb504402740 20 client.15753 trim_cache size 0 max 16384 2024-04-21T23:18:59.429 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.428+0000 7fb504402740 1 Processor -- start 2024-04-21T23:18:59.429 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.428+0000 7fb504402740 1 -- start start 2024-04-21T23:18:59.430 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.428+0000 7fb504402740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565053b23640 0x565053b3cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:59.430 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.428+0000 7fb504402740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565053b23f60 0x565053b19450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:59.430 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.428+0000 7fb504402740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565053b31620 0x565053b2ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:59.430 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.428+0000 7fb504402740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb4fc1af280 con 0x565053b23640 2024-04-21T23:18:59.430 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.428+0000 7fb504402740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb4fc1acb50 con 0x565053b31620 2024-04-21T23:18:59.430 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.428+0000 7fb504402740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb4fc1aa9a0 con 0x565053b23f60 2024-04-21T23:18:59.430 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.428+0000 7fb4faffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565053b23640 0x565053b3cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:59.430 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.428+0000 7fb4faffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565053b23640 0x565053b3cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46656/0 (socket says 172.21.15.40:46656) 2024-04-21T23:18:59.430 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.428+0000 7fb4faffd640 1 -- 172.21.15.40:0/1900265757 learned_addr learned my addr 172.21.15.40:0/1900265757 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:59.430 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.428+0000 7fb4fb7fe640 1 --2- 172.21.15.40:0/1900265757 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565053b31620 0x565053b2ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:59.430 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.428+0000 7fb4fa7fc640 1 --2- 172.21.15.40:0/1900265757 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565053b23f60 0x565053b19450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:59.430 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.428+0000 7fb4faffd640 1 -- 172.21.15.40:0/1900265757 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565053b23f60 msgr2=0x565053b19450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:59.430 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.429+0000 7fb4faffd640 1 --2- 172.21.15.40:0/1900265757 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565053b23f60 0x565053b19450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:59.431 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.429+0000 7fb4faffd640 1 -- 172.21.15.40:0/1900265757 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565053b31620 msgr2=0x565053b2ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:59.431 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.429+0000 7fb4faffd640 1 --2- 172.21.15.40:0/1900265757 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565053b31620 0x565053b2ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:59.431 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.429+0000 7fb4faffd640 1 -- 172.21.15.40:0/1900265757 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565053b2d550 con 0x565053b23640 2024-04-21T23:18:59.431 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.429+0000 7fb4fa7fc640 1 --2- 172.21.15.40:0/1900265757 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565053b23f60 0x565053b19450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:18:59.431 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.429+0000 7fb4faffd640 1 --2- 172.21.15.40:0/1900265757 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565053b23640 0x565053b3cdd0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fb4ec000c00 tx=0x7fb4ec004410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55ae1aed5c1cafa3 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:59.431 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.429+0000 7fb4e17fa640 1 -- 172.21.15.40:0/1900265757 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4ec003de0 con 0x565053b23640 2024-04-21T23:18:59.431 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.429+0000 7fb4e17fa640 1 -- 172.21.15.40:0/1900265757 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb4ec022070 con 0x565053b23640 2024-04-21T23:18:59.431 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.429+0000 7fb4e17fa640 1 -- 172.21.15.40:0/1900265757 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4ec011dd0 con 0x565053b23640 2024-04-21T23:18:59.431 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.430+0000 7fb504402740 1 -- 172.21.15.40:0/1900265757 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565053b23640 msgr2=0x565053b3cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:59.431 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.430+0000 7fb504402740 1 --2- 172.21.15.40:0/1900265757 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565053b23640 0x565053b3cdd0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fb4ec000c00 tx=0x7fb4ec004410 comp rx=0 tx=0).stop 2024-04-21T23:18:59.431 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.430+0000 7fb504402740 1 -- 172.21.15.40:0/1900265757 shutdown_connections 2024-04-21T23:18:59.431 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.430+0000 7fb504402740 1 --2- 172.21.15.40:0/1900265757 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565053b31620 0x565053b2ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:59.431 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.430+0000 7fb504402740 1 --2- 172.21.15.40:0/1900265757 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565053b23f60 0x565053b19450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:59.431 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.430+0000 7fb504402740 1 --2- 172.21.15.40:0/1900265757 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565053b23640 0x565053b3cdd0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:59.431 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.430+0000 7fb504402740 1 -- 172.21.15.40:0/1900265757 >> 172.21.15.40:0/1900265757 conn(0x565053b1e440 msgr2=0x565053b200f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:59.432 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.430+0000 7fb504402740 1 -- 172.21.15.40:0/1900265757 shutdown_connections 2024-04-21T23:18:59.432 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.430+0000 7fb504402740 1 -- 172.21.15.40:0/1900265757 wait complete. 2024-04-21T23:18:59.432 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.430+0000 7fb504402740 1 -- start start 2024-04-21T23:18:59.432 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.430+0000 7fb504402740 1 Processor -- start 2024-04-21T23:18:59.434 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.430+0000 7fb504402740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565053b23640 0x565053b2b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:59.435 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.430+0000 7fb504402740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565053b23f60 0x565053b2b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:59.435 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.431+0000 7fb504402740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565053b31620 0x565053b2be20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:59.435 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.431+0000 7fb504402740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb4fc1ab010 con 0x565053b31620 2024-04-21T23:18:59.435 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.431+0000 7fb504402740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb4fc1ac480 con 0x565053b23f60 2024-04-21T23:18:59.435 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.431+0000 7fb504402740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb4fc1ab6e0 con 0x565053b23640 2024-04-21T23:18:59.435 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.431+0000 7fb504402740 10 client.0 _ll_register_callbacks cb 0x7fb4f40b93a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:18:59.435 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.431+0000 7fb4fb7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565053b31620 0x565053b2be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:59.435 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.431+0000 7fb4fb7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565053b31620 0x565053b2be20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46668/0 (socket says 172.21.15.40:46668) 2024-04-21T23:18:59.435 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.431+0000 7fb4fb7fe640 1 -- 172.21.15.40:0/3241048982 learned_addr learned my addr 172.21.15.40:0/3241048982 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:18:59.435 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.431+0000 7fb4fa7fc640 1 --2- 172.21.15.40:0/3241048982 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565053b23f60 0x565053b2b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:59.435 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.431+0000 7fb4faffd640 1 --2- 172.21.15.40:0/3241048982 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565053b23640 0x565053b2b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:59.435 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.431+0000 7fb4fb7fe640 1 -- 172.21.15.40:0/3241048982 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565053b23640 msgr2=0x565053b2b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:59.435 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.431+0000 7fb4fb7fe640 1 --2- 172.21.15.40:0/3241048982 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565053b23640 0x565053b2b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:59.436 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.431+0000 7fb4fb7fe640 1 -- 172.21.15.40:0/3241048982 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565053b23f60 msgr2=0x565053b2b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:59.436 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.431+0000 7fb4fb7fe640 1 --2- 172.21.15.40:0/3241048982 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565053b23f60 0x565053b2b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:59.436 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.431+0000 7fb4fb7fe640 1 -- 172.21.15.40:0/3241048982 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565053b43a40 con 0x565053b31620 2024-04-21T23:18:59.436 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.431+0000 7fb4fb7fe640 1 --2- 172.21.15.40:0/3241048982 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565053b31620 0x565053b2be20 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fb4f40ba140 tx=0x7fb4f4032480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df8a5b6d6acaebf0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:59.436 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.431+0000 7fb4daffd640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-21T23:18:59.436 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.431+0000 7fb4daffd640 1 -- 172.21.15.40:0/3241048982 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4f40db070 con 0x565053b31620 2024-04-21T23:18:59.436 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.431+0000 7fb4daffd640 1 -- 172.21.15.40:0/3241048982 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7fb4f40f1590 con 0x565053b31620 2024-04-21T23:18:59.436 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.431+0000 7fb504402740 1 -- 172.21.15.40:0/3241048982 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb4ec022070 con 0x565053b31620 2024-04-21T23:18:59.436 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.432+0000 7fb4daffd640 1 -- 172.21.15.40:0/3241048982 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fb4f40eea00 con 0x565053b31620 2024-04-21T23:18:59.436 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.432+0000 7fb504402740 10 client.15759 fetch_fsmap learned FSMap version 25 2024-04-21T23:18:59.436 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.432+0000 7fb504402740 1 -- 172.21.15.40:0/3241048982 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x565053b43a40 con 0x565053b31620 2024-04-21T23:18:59.436 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.432+0000 7fb4daffd640 1 -- 172.21.15.40:0/3241048982 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb4ec022070 con 0x565053b31620 2024-04-21T23:18:59.436 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.432+0000 7fb4daffd640 1 -- 172.21.15.40:0/3241048982 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fb4f40d1ce0 con 0x565053b31620 2024-04-21T23:18:59.436 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.432+0000 7fb504402740 10 client.15759 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:18:59.437 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.432+0000 7fb504402740 10 client.15759 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:18:59.437 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.432+0000 7fb504402740 20 client.15759 populate_metadata read hostname 'smithi040' 2024-04-21T23:18:59.437 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.432+0000 7fb504402740 1 --2- 172.21.15.40:0/3241048982 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x565053ba7ae0 0x565053ba9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:59.437 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.432+0000 7fb504402740 4 client.15759 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T23:18:59.437 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.432+0000 7fb504402740 1 -- 172.21.15.40:0/3241048982 --> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] -- command(tid 0: {"prefix": "flush journal"}) -- 0x565053a8f400 con 0x565053ba7ae0 2024-04-21T23:18:59.437 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.432+0000 7fb4faffd640 1 --2- 172.21.15.40:0/3241048982 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x565053ba7ae0 0x565053ba9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:59.437 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:18:59 smithi040 ceph-mon[27984]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.6 KiB/s wr, 9 op/s 2024-04-21T23:18:59.438 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.433+0000 7fb4faffd640 1 --2- 172.21.15.40:0/3241048982 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x565053ba7ae0 0x565053ba9ed0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d954aa9525c73f22 server_cookie=2cd6492a9d629713 in_seq=0 out_seq=0 2024-04-21T23:18:59.438 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.433+0000 7fb4daffd640 10 client.15759 ms_handle_connect on v2:172.21.15.111:6838/3712235548 2024-04-21T23:18:59.438 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.436+0000 7fb4daffd640 1 -- 172.21.15.40:0/3241048982 <== mds.0 v2:172.21.15.111:6838/3712235548 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x565053a8f400 con 0x565053ba7ae0 2024-04-21T23:18:59.438 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.436+0000 7fb4daffd640 10 client.15759 handle_command_reply: tid=0 2024-04-21T23:18:59.439 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.436+0000 7fb504402740 1 client.15759 shutdown 2024-04-21T23:18:59.439 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.437+0000 7fb504402740 1 -- 172.21.15.40:0/3241048982 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565053b31620 msgr2=0x565053b2be20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:59.439 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.437+0000 7fb504402740 1 --2- 172.21.15.40:0/3241048982 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565053b31620 0x565053b2be20 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fb4f40ba140 tx=0x7fb4f4032480 comp rx=0 tx=0).stop 2024-04-21T23:18:59.439 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.437+0000 7fb504402740 1 -- 172.21.15.40:0/3241048982 shutdown_connections 2024-04-21T23:18:59.439 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.437+0000 7fb504402740 1 --2- 172.21.15.40:0/3241048982 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x565053ba7ae0 0x565053ba9ed0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:59.439 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.437+0000 7fb504402740 1 --2- 172.21.15.40:0/3241048982 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565053b23f60 0x565053b2b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:59.439 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.437+0000 7fb504402740 1 --2- 172.21.15.40:0/3241048982 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565053b31620 0x565053b2be20 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:59.439 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.437+0000 7fb4daffd640 0 client.15759 ms_handle_reset on v2:172.21.15.111:6838/3712235548 2024-04-21T23:18:59.439 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.437+0000 7fb504402740 1 --2- 172.21.15.40:0/3241048982 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565053b23640 0x565053b2b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:59.439 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.437+0000 7fb504402740 1 -- 172.21.15.40:0/3241048982 >> 172.21.15.40:0/3241048982 conn(0x565053b1e440 msgr2=0x565053b1f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:59.439 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.437+0000 7fb504402740 1 -- 172.21.15.40:0/3241048982 shutdown_connections 2024-04-21T23:18:59.439 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.437+0000 7fb504402740 1 -- 172.21.15.40:0/3241048982 wait complete. 2024-04-21T23:18:59.439 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.437+0000 7fb504402740 20 client.15759 trim_cache size 0 max 16384 2024-04-21T23:18:59.440 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:{ 2024-04-21T23:18:59.440 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "message": "", 2024-04-21T23:18:59.440 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "return_code": 0 2024-04-21T23:18:59.440 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:} 2024-04-21T23:18:59.441 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.437+0000 7fb501815640 1 -- 172.21.15.40:0/3049896923 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fb4e40a2f40 msgr2=0x7fb4e40a5400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:59.441 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.437+0000 7fb501815640 1 --2- 172.21.15.40:0/3049896923 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fb4e40a2f40 0x7fb4e40a5400 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb4f000b620 tx=0x7fb4f000bb50 comp rx=0 tx=0).stop 2024-04-21T23:18:59.441 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.437+0000 7fb501815640 1 -- 172.21.15.40:0/3049896923 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4fc107730 msgr2=0x7fb4fc19c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:59.441 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.437+0000 7fb501815640 1 --2- 172.21.15.40:0/3049896923 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4fc107730 0x7fb4fc19c7e0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fb4f4005e30 tx=0x7fb4f4032ea0 comp rx=0 tx=0).stop 2024-04-21T23:18:59.441 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.438+0000 7fb501815640 1 -- 172.21.15.40:0/3049896923 shutdown_connections 2024-04-21T23:18:59.441 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.438+0000 7fb501815640 1 --2- 172.21.15.40:0/3049896923 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fb4e40a2f40 0x7fb4e40a5400 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:59.441 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.438+0000 7fb501815640 1 --2- 172.21.15.40:0/3049896923 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4fc1020e0 0x7fb4fc198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:59.441 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.438+0000 7fb501815640 1 --2- 172.21.15.40:0/3049896923 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4fc107730 0x7fb4fc19c7e0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:59.441 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.438+0000 7fb501815640 1 --2- 172.21.15.40:0/3049896923 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb4fc101720 0x7fb4fc197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:59.441 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.438+0000 7fb501815640 1 -- 172.21.15.40:0/3049896923 >> 172.21.15.40:0/3049896923 conn(0x7fb4fc0fd1f0 msgr2=0x7fb4fc0fd790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:59.441 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.438+0000 7fb501815640 1 -- 172.21.15.40:0/3049896923 shutdown_connections 2024-04-21T23:18:59.442 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:18:59.438+0000 7fb501815640 1 -- 172.21.15.40:0/3049896923 wait complete. 2024-04-21T23:18:59.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:18:59 smithi174 ceph-mon[31409]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.6 KiB/s wr, 9 op/s 2024-04-21T23:18:59.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:18:59 smithi111 ceph-mon[31085]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.6 KiB/s wr, 9 op/s 2024-04-21T23:19:00.682 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:19:00 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:19:00.361+0000 7f2e5a6e2700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T23:19:00.682 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:19:00 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:19:00.361+0000 7f2e5a6e2700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T23:19:00.682 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:19:00 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:19:00.361+0000 7f2e5a6e2700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T23:19:00.682 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:19:00 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:19:00.361+0000 7f2e5a6e2700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T23:19:00.682 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:19:00 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:19:00.361+0000 7f2e5a6e2700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T23:19:01.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:19:01 smithi040 ceph-mon[27984]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2024-04-21T23:19:01.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:19:01 smithi174 ceph-mon[31409]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2024-04-21T23:19:01.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:19:01 smithi111 ceph-mon[31085]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2024-04-21T23:19:01.820 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T23:19:01.820 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T23:19:01.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.941+0000 7f63c898c640 1 -- 172.21.15.40:0/521616469 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63c4100610 msgr2=0x7f63c4100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:01.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.941+0000 7f63c898c640 1 --2- 172.21.15.40:0/521616469 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63c4100610 0x7f63c4100a90 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f63b8009930 tx=0x7f63b802bc30 comp rx=0 tx=0).stop 2024-04-21T23:19:01.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.941+0000 7f63c898c640 1 -- 172.21.15.40:0/521616469 shutdown_connections 2024-04-21T23:19:01.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.941+0000 7f63c898c640 1 --2- 172.21.15.40:0/521616469 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63c4100fd0 0x7f63c410d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:01.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.941+0000 7f63c898c640 1 --2- 172.21.15.40:0/521616469 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63c4100610 0x7f63c4100a90 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:01.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.941+0000 7f63c898c640 1 --2- 172.21.15.40:0/521616469 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f63c41066b0 0x7f63c4106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:01.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.941+0000 7f63c898c640 1 -- 172.21.15.40:0/521616469 >> 172.21.15.40:0/521616469 conn(0x7f63c4077740 msgr2=0x7f63c4077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:01.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.941+0000 7f63c898c640 1 -- 172.21.15.40:0/521616469 shutdown_connections 2024-04-21T23:19:01.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.941+0000 7f63c898c640 1 -- 172.21.15.40:0/521616469 wait complete. 2024-04-21T23:19:01.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.942+0000 7f63c898c640 1 Processor -- start 2024-04-21T23:19:01.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.942+0000 7f63c898c640 1 -- start start 2024-04-21T23:19:01.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.942+0000 7f63c898c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63c4100610 0x7f63c4197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:01.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.942+0000 7f63c898c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63c4100fd0 0x7f63c4198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:01.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.942+0000 7f63c898c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f63c41066b0 0x7f63c419c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:01.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.942+0000 7f63c898c640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f63c41109f0 con 0x7f63c4100610 2024-04-21T23:19:01.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.942+0000 7f63c898c640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f63c4110870 con 0x7f63c4100fd0 2024-04-21T23:19:01.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.942+0000 7f63c898c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f63c4110b70 con 0x7f63c41066b0 2024-04-21T23:19:01.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.942+0000 7f63c259c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63c4100610 0x7f63c4197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:01.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.942+0000 7f63c259c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63c4100610 0x7f63c4197fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47416/0 (socket says 172.21.15.40:47416) 2024-04-21T23:19:01.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.942+0000 7f63c259c640 1 -- 172.21.15.40:0/822304289 learned_addr learned my addr 172.21.15.40:0/822304289 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:01.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.942+0000 7f63c2d9d640 1 --2- 172.21.15.40:0/822304289 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f63c41066b0 0x7f63c419c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:01.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.942+0000 7f63c259c640 1 -- 172.21.15.40:0/822304289 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f63c41066b0 msgr2=0x7f63c419c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:01.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.942+0000 7f63c259c640 1 --2- 172.21.15.40:0/822304289 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f63c41066b0 0x7f63c419c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:01.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.942+0000 7f63c259c640 1 -- 172.21.15.40:0/822304289 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63c4100fd0 msgr2=0x7f63c4198520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:01.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.942+0000 7f63c259c640 1 --2- 172.21.15.40:0/822304289 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63c4100fd0 0x7f63c4198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:01.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.942+0000 7f63c259c640 1 -- 172.21.15.40:0/822304289 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63c419cfd0 con 0x7f63c4100610 2024-04-21T23:19:01.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.942+0000 7f63c2d9d640 1 --2- 172.21.15.40:0/822304289 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f63c41066b0 0x7f63c419c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:19:01.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.943+0000 7f63c259c640 1 --2- 172.21.15.40:0/822304289 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63c4100610 0x7f63c4197fe0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f63bc00d950 tx=0x7f63bc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:01.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.943+0000 7f63b37fe640 1 -- 172.21.15.40:0/822304289 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63bc017070 con 0x7f63c4100610 2024-04-21T23:19:01.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.943+0000 7f63b37fe640 1 -- 172.21.15.40:0/822304289 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f63bc009c30 con 0x7f63c4100610 2024-04-21T23:19:01.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.943+0000 7f63c898c640 1 -- 172.21.15.40:0/822304289 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63c419d2c0 con 0x7f63c4100610 2024-04-21T23:19:01.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.943+0000 7f63b37fe640 1 -- 172.21.15.40:0/822304289 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63bc00f600 con 0x7f63c4100610 2024-04-21T23:19:01.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.943+0000 7f63c898c640 1 -- 172.21.15.40:0/822304289 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63c41a4b50 con 0x7f63c4100610 2024-04-21T23:19:01.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.945+0000 7f63b37fe640 1 -- 172.21.15.40:0/822304289 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7f63bc01a070 con 0x7f63c4100610 2024-04-21T23:19:01.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.945+0000 7f63cb57b740 1 Processor -- start 2024-04-21T23:19:01.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.946+0000 7f63cb57b740 1 -- start start 2024-04-21T23:19:01.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.946+0000 7f63cb57b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55696b707f30 0x55696b708310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:01.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.946+0000 7f63cb57b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55696b708850 0x55696b710ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:01.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.946+0000 7f63cb57b740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55696b711430 0x55696b713820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:01.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.946+0000 7f63cb57b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55696b66c560 con 0x55696b708850 2024-04-21T23:19:01.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.946+0000 7f63cb57b740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55696b614730 con 0x55696b711430 2024-04-21T23:19:01.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.946+0000 7f63cb57b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55696b668520 con 0x55696b707f30 2024-04-21T23:19:01.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.946+0000 7f63b37fe640 1 --2- 172.21.15.40:0/822304289 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f63ac0a2e20 0x7f63ac0a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:01.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.946+0000 7f63c259c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55696b708850 0x55696b710ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:01.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.946+0000 7f63c259c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55696b708850 0x55696b710ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47428/0 (socket says 172.21.15.40:47428) 2024-04-21T23:19:01.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.946+0000 7f63c259c640 1 -- 172.21.15.40:0/69439391 learned_addr learned my addr 172.21.15.40:0/69439391 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:01.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.946+0000 7f63c2d9d640 1 --2- 172.21.15.40:0/69439391 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55696b707f30 0x55696b708310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:01.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.946+0000 7f63c2d9d640 1 --2- 172.21.15.40:0/822304289 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f63ac0a2e20 0x7f63ac0a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:01.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.946+0000 7f63c1d9b640 1 --2- 172.21.15.40:0/69439391 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55696b711430 0x55696b713820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:01.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.946+0000 7f63b37fe640 1 -- 172.21.15.40:0/822304289 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f63bc0cf570 con 0x7f63c4100610 2024-04-21T23:19:01.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.946+0000 7f63c259c640 1 -- 172.21.15.40:0/69439391 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55696b707f30 msgr2=0x55696b708310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:01.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.946+0000 7f63c259c640 1 --2- 172.21.15.40:0/69439391 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55696b707f30 0x55696b708310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:01.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.946+0000 7f63c259c640 1 -- 172.21.15.40:0/69439391 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55696b711430 msgr2=0x55696b713820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:01.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.946+0000 7f63c259c640 1 --2- 172.21.15.40:0/69439391 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55696b711430 0x55696b713820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:01.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.946+0000 7f63c259c640 1 -- 172.21.15.40:0/69439391 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55696b44a7d0 con 0x55696b708850 2024-04-21T23:19:01.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.947+0000 7f63c259c640 1 --2- 172.21.15.40:0/69439391 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55696b708850 0x55696b710ef0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f63bc0ba0d0 tx=0x7f63bc09b100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42faf74749161ad3 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:01.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.947+0000 7f63c2d9d640 1 --2- 172.21.15.40:0/822304289 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f63ac0a2e20 0x7f63ac0a52e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f63b4005fd0 tx=0x7f63b4009440 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:01.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.947+0000 7f63b0ff9640 1 -- 172.21.15.40:0/69439391 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63bc021070 con 0x55696b708850 2024-04-21T23:19:01.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.947+0000 7f63b0ff9640 1 -- 172.21.15.40:0/69439391 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f63bc09bb90 con 0x55696b708850 2024-04-21T23:19:01.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.947+0000 7f63b0ff9640 1 -- 172.21.15.40:0/69439391 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63bc0bb360 con 0x55696b708850 2024-04-21T23:19:01.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.948+0000 7f63cb57b740 1 -- 172.21.15.40:0/69439391 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55696b708850 msgr2=0x55696b710ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:01.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.948+0000 7f63cb57b740 1 --2- 172.21.15.40:0/69439391 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55696b708850 0x55696b710ef0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f63bc0ba0d0 tx=0x7f63bc09b100 comp rx=0 tx=0).stop 2024-04-21T23:19:01.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.948+0000 7f63cb57b740 1 -- 172.21.15.40:0/69439391 shutdown_connections 2024-04-21T23:19:01.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.948+0000 7f63cb57b740 1 --2- 172.21.15.40:0/69439391 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55696b711430 0x55696b713820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:01.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.948+0000 7f63cb57b740 1 --2- 172.21.15.40:0/69439391 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55696b708850 0x55696b710ef0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:01.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.948+0000 7f63cb57b740 1 --2- 172.21.15.40:0/69439391 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55696b707f30 0x55696b708310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:01.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.948+0000 7f63cb57b740 1 -- 172.21.15.40:0/69439391 >> 172.21.15.40:0/69439391 conn(0x55696b702660 msgr2=0x55696b702a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:01.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.948+0000 7f63cb57b740 1 -- 172.21.15.40:0/69439391 shutdown_connections 2024-04-21T23:19:01.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.948+0000 7f63cb57b740 1 -- 172.21.15.40:0/69439391 wait complete. 2024-04-21T23:19:01.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.948+0000 7f63cb57b740 1 -- start start 2024-04-21T23:19:01.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.948+0000 7f63cb57b740 1 Processor -- start 2024-04-21T23:19:01.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.949+0000 7f63cb57b740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55696b707f30 0x55696b70e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:01.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.949+0000 7f63cb57b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55696b708850 0x55696b70ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:01.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.949+0000 7f63cb57b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55696b711430 0x55696b70ef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:01.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.949+0000 7f63cb57b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55696b7141c0 con 0x55696b711430 2024-04-21T23:19:01.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.949+0000 7f63cb57b740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55696b627e00 con 0x55696b707f30 2024-04-21T23:19:01.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.949+0000 7f63cb57b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55696b714340 con 0x55696b708850 2024-04-21T23:19:01.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.949+0000 7f63cb57b740 10 client.0 _ll_register_callbacks cb 0x55696b662eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:19:01.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.949+0000 7f63c2d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55696b711430 0x55696b70ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:01.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.949+0000 7f63c2d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55696b711430 0x55696b70ef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47442/0 (socket says 172.21.15.40:47442) 2024-04-21T23:19:01.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.949+0000 7f63c2d9d640 1 -- 172.21.15.40:0/2540432518 learned_addr learned my addr 172.21.15.40:0/2540432518 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:01.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.949+0000 7f63c259c640 1 --2- 172.21.15.40:0/2540432518 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55696b707f30 0x55696b70e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:01.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.949+0000 7f63c1d9b640 1 --2- 172.21.15.40:0/2540432518 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55696b708850 0x55696b70ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:01.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.949+0000 7f63c2d9d640 1 -- 172.21.15.40:0/2540432518 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55696b708850 msgr2=0x55696b70ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:01.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.949+0000 7f63c2d9d640 1 --2- 172.21.15.40:0/2540432518 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55696b708850 0x55696b70ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:01.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.949+0000 7f63c2d9d640 1 -- 172.21.15.40:0/2540432518 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55696b707f30 msgr2=0x55696b70e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:01.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.949+0000 7f63c2d9d640 1 --2- 172.21.15.40:0/2540432518 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55696b707f30 0x55696b70e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:01.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.949+0000 7f63c2d9d640 1 -- 172.21.15.40:0/2540432518 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55696b3803f0 con 0x55696b711430 2024-04-21T23:19:01.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.949+0000 7f63c1d9b640 1 --2- 172.21.15.40:0/2540432518 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55696b708850 0x55696b70ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:19:01.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.949+0000 7f63c2d9d640 1 --2- 172.21.15.40:0/2540432518 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55696b711430 0x55696b70ef70 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f63b4009a60 tx=0x7f63b4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d16d23d762f93f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:01.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.949+0000 7f63a9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-21T23:19:01.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.949+0000 7f63a9ffb640 1 -- 172.21.15.40:0/2540432518 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63b4023070 con 0x55696b711430 2024-04-21T23:19:01.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.949+0000 7f63a9ffb640 1 -- 172.21.15.40:0/2540432518 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f63b401f400 con 0x55696b711430 2024-04-21T23:19:01.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.949+0000 7f63cb57b740 1 -- 172.21.15.40:0/2540432518 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f63bc09bb90 con 0x55696b711430 2024-04-21T23:19:01.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.950+0000 7f63a9ffb640 1 -- 172.21.15.40:0/2540432518 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f63b4003e40 con 0x55696b711430 2024-04-21T23:19:01.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.950+0000 7f63cb57b740 10 client.15777 fetch_fsmap learned FSMap version 25 2024-04-21T23:19:01.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.950+0000 7f63cb57b740 1 -- 172.21.15.40:0/2540432518 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55696b3400f0 con 0x55696b711430 2024-04-21T23:19:01.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.950+0000 7f63a9ffb640 1 -- 172.21.15.40:0/2540432518 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f63bc09bb90 con 0x55696b711430 2024-04-21T23:19:01.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.950+0000 7f63a9ffb640 1 -- 172.21.15.40:0/2540432518 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f63b400f040 con 0x55696b711430 2024-04-21T23:19:01.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.950+0000 7f63cb57b740 10 client.15777 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:19:01.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.950+0000 7f63cb57b740 10 client.15777 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:19:01.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.950+0000 7f63cb57b740 20 client.15777 populate_metadata read hostname 'smithi040' 2024-04-21T23:19:01.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.950+0000 7f63cb57b740 1 --2- 172.21.15.40:0/2540432518 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55696b721840 0x55696b723c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:01.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.950+0000 7f63cb57b740 4 client.15777 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:19:01.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.950+0000 7f63cb57b740 1 -- 172.21.15.40:0/2540432518 --> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55696b679aa0 con 0x55696b721840 2024-04-21T23:19:01.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.950+0000 7f63c259c640 1 --2- 172.21.15.40:0/2540432518 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55696b721840 0x55696b723c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:01.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.951+0000 7f63c259c640 1 --2- 172.21.15.40:0/2540432518 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55696b721840 0x55696b723c50 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2f7863cad3de6dac server_cookie=47f7b69a100ce6ef in_seq=0 out_seq=0 2024-04-21T23:19:01.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.951+0000 7f63a9ffb640 10 client.15777 ms_handle_connect on v2:172.21.15.111:6838/3712235548 2024-04-21T23:19:01.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.951+0000 7f63a9ffb640 1 -- 172.21.15.40:0/2540432518 <== mds.0 v2:172.21.15.111:6838/3712235548 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55696b679aa0 con 0x55696b721840 2024-04-21T23:19:01.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.951+0000 7f63a9ffb640 10 client.15777 handle_command_reply: tid=0 2024-04-21T23:19:01.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.952+0000 7f63cb57b740 1 client.15777 shutdown 2024-04-21T23:19:01.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.952+0000 7f63cb57b740 1 -- 172.21.15.40:0/2540432518 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55696b711430 msgr2=0x55696b70ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:01.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.952+0000 7f63cb57b740 1 --2- 172.21.15.40:0/2540432518 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55696b711430 0x55696b70ef70 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f63b4009a60 tx=0x7f63b4004a40 comp rx=0 tx=0).stop 2024-04-21T23:19:01.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.952+0000 7f63cb57b740 1 -- 172.21.15.40:0/2540432518 shutdown_connections 2024-04-21T23:19:01.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.952+0000 7f63cb57b740 1 --2- 172.21.15.40:0/2540432518 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55696b721840 0x55696b723c50 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:01.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.952+0000 7f63cb57b740 1 --2- 172.21.15.40:0/2540432518 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55696b711430 0x55696b70ef70 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:01.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.952+0000 7f63cb57b740 1 --2- 172.21.15.40:0/2540432518 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55696b708850 0x55696b70ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:01.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.952+0000 7f63cb57b740 1 --2- 172.21.15.40:0/2540432518 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55696b707f30 0x55696b70e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:01.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.952+0000 7f63cb57b740 1 -- 172.21.15.40:0/2540432518 >> 172.21.15.40:0/2540432518 conn(0x55696b702660 msgr2=0x55696b7043b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:01.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.952+0000 7f63a9ffb640 0 client.15777 ms_handle_reset on v2:172.21.15.111:6838/3712235548 2024-04-21T23:19:01.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.952+0000 7f63cb57b740 1 -- 172.21.15.40:0/2540432518 shutdown_connections 2024-04-21T23:19:01.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.952+0000 7f63cb57b740 1 -- 172.21.15.40:0/2540432518 wait complete. 2024-04-21T23:19:01.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.952+0000 7f63cb57b740 20 client.15777 trim_cache size 0 max 16384 2024-04-21T23:19:01.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.972+0000 7f63cb57b740 1 Processor -- start 2024-04-21T23:19:01.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.972+0000 7f63cb57b740 1 -- start start 2024-04-21T23:19:01.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.972+0000 7f63cb57b740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55696b707f30 0x55696b710c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:01.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.972+0000 7f63cb57b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55696b708850 0x55696b71d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:01.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.972+0000 7f63cb57b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55696b711430 0x55696b729520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:01.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.972+0000 7f63cb57b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f63c41b1650 con 0x55696b708850 2024-04-21T23:19:01.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.972+0000 7f63cb57b740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f63c41aef20 con 0x55696b707f30 2024-04-21T23:19:01.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.972+0000 7f63cb57b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f63c41acd70 con 0x55696b711430 2024-04-21T23:19:01.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.972+0000 7f63c1d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55696b708850 0x55696b71d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:01.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.972+0000 7f63c1d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55696b708850 0x55696b71d080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47456/0 (socket says 172.21.15.40:47456) 2024-04-21T23:19:01.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.972+0000 7f63c1d9b640 1 -- 172.21.15.40:0/1550647110 learned_addr learned my addr 172.21.15.40:0/1550647110 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:01.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.973+0000 7f63c259c640 1 --2- 172.21.15.40:0/1550647110 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55696b707f30 0x55696b710c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:01.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.973+0000 7f63c2d9d640 1 --2- 172.21.15.40:0/1550647110 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55696b711430 0x55696b729520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:01.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.973+0000 7f63c1d9b640 1 -- 172.21.15.40:0/1550647110 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55696b711430 msgr2=0x55696b729520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:01.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.973+0000 7f63c1d9b640 1 --2- 172.21.15.40:0/1550647110 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55696b711430 0x55696b729520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:01.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.973+0000 7f63c1d9b640 1 -- 172.21.15.40:0/1550647110 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55696b707f30 msgr2=0x55696b710c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:01.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.973+0000 7f63c1d9b640 1 --2- 172.21.15.40:0/1550647110 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55696b707f30 0x55696b710c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:01.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.973+0000 7f63c1d9b640 1 -- 172.21.15.40:0/1550647110 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55696b71d5c0 con 0x55696b708850 2024-04-21T23:19:01.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.973+0000 7f63c259c640 1 --2- 172.21.15.40:0/1550647110 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55696b707f30 0x55696b710c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:19:01.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.973+0000 7f63c2d9d640 1 --2- 172.21.15.40:0/1550647110 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55696b711430 0x55696b729520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:19:01.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.973+0000 7f63c1d9b640 1 --2- 172.21.15.40:0/1550647110 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55696b708850 0x55696b71d080 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f63c4105cc0 tx=0x7f63b8005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7c35017b0748ab0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:01.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.973+0000 7f63abfff640 1 -- 172.21.15.40:0/1550647110 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63b802c880 con 0x55696b708850 2024-04-21T23:19:01.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.973+0000 7f63abfff640 1 -- 172.21.15.40:0/1550647110 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f63b802ca20 con 0x55696b708850 2024-04-21T23:19:01.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.973+0000 7f63abfff640 1 -- 172.21.15.40:0/1550647110 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63b8004920 con 0x55696b708850 2024-04-21T23:19:01.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.974+0000 7f63cb57b740 1 -- 172.21.15.40:0/1550647110 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55696b708850 msgr2=0x55696b71d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:01.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.974+0000 7f63cb57b740 1 --2- 172.21.15.40:0/1550647110 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55696b708850 0x55696b71d080 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f63c4105cc0 tx=0x7f63b8005d50 comp rx=0 tx=0).stop 2024-04-21T23:19:01.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.974+0000 7f63cb57b740 1 -- 172.21.15.40:0/1550647110 shutdown_connections 2024-04-21T23:19:01.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.974+0000 7f63cb57b740 1 --2- 172.21.15.40:0/1550647110 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55696b711430 0x55696b729520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:01.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.974+0000 7f63cb57b740 1 --2- 172.21.15.40:0/1550647110 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55696b708850 0x55696b71d080 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:01.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.974+0000 7f63cb57b740 1 --2- 172.21.15.40:0/1550647110 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55696b707f30 0x55696b710c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:01.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.974+0000 7f63cb57b740 1 -- 172.21.15.40:0/1550647110 >> 172.21.15.40:0/1550647110 conn(0x55696b702660 msgr2=0x55696b7016e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:01.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.974+0000 7f63cb57b740 1 -- 172.21.15.40:0/1550647110 shutdown_connections 2024-04-21T23:19:01.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.974+0000 7f63cb57b740 1 -- 172.21.15.40:0/1550647110 wait complete. 2024-04-21T23:19:01.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.974+0000 7f63cb57b740 1 -- start start 2024-04-21T23:19:01.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.974+0000 7f63cb57b740 1 Processor -- start 2024-04-21T23:19:01.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.974+0000 7f63cb57b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55696b707f30 0x55696b70b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:01.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.974+0000 7f63cb57b740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55696b708850 0x55696b70bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:01.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.974+0000 7f63cb57b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55696b711430 0x55696b786d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:01.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.974+0000 7f63cb57b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f63c41ad3e0 con 0x55696b711430 2024-04-21T23:19:01.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.974+0000 7f63cb57b740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f63c41ae850 con 0x55696b708850 2024-04-21T23:19:01.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.974+0000 7f63cb57b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f63c41adab0 con 0x55696b707f30 2024-04-21T23:19:01.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.974+0000 7f63cb57b740 10 client.0 _ll_register_callbacks cb 0x7f63b4012470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:19:01.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.975+0000 7f63c2d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55696b711430 0x55696b786d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:01.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.975+0000 7f63c2d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55696b711430 0x55696b786d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47470/0 (socket says 172.21.15.40:47470) 2024-04-21T23:19:01.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.975+0000 7f63c2d9d640 1 -- 172.21.15.40:0/3961548435 learned_addr learned my addr 172.21.15.40:0/3961548435 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:01.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.975+0000 7f63c1d9b640 1 --2- 172.21.15.40:0/3961548435 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55696b708850 0x55696b70bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:01.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.975+0000 7f63c259c640 1 --2- 172.21.15.40:0/3961548435 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55696b707f30 0x55696b70b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:01.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.975+0000 7f63c2d9d640 1 -- 172.21.15.40:0/3961548435 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55696b707f30 msgr2=0x55696b70b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:01.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.975+0000 7f63c2d9d640 1 --2- 172.21.15.40:0/3961548435 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55696b707f30 0x55696b70b5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:01.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.975+0000 7f63c2d9d640 1 -- 172.21.15.40:0/3961548435 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55696b708850 msgr2=0x55696b70bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:01.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.975+0000 7f63c2d9d640 1 --2- 172.21.15.40:0/3961548435 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55696b708850 0x55696b70bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:01.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.975+0000 7f63c2d9d640 1 -- 172.21.15.40:0/3961548435 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55696b7872d0 con 0x55696b711430 2024-04-21T23:19:01.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.975+0000 7f63c2d9d640 1 --2- 172.21.15.40:0/3961548435 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55696b711430 0x55696b786d90 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f63b4009a30 tx=0x7f63b4011f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=662b9c99d9bf81ea server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:01.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.975+0000 7f63a9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-21T23:19:01.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.975+0000 7f63a9ffb640 1 -- 172.21.15.40:0/3961548435 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63b403a060 con 0x55696b711430 2024-04-21T23:19:01.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.975+0000 7f63a9ffb640 1 -- 172.21.15.40:0/3961548435 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f63b40115e0 con 0x55696b711430 2024-04-21T23:19:01.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.975+0000 7f63cb57b740 1 -- 172.21.15.40:0/3961548435 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f63b802ca20 con 0x55696b711430 2024-04-21T23:19:01.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.975+0000 7f63a9ffb640 1 -- 172.21.15.40:0/3961548435 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f63b4041050 con 0x55696b711430 2024-04-21T23:19:01.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.975+0000 7f63cb57b740 10 client.15783 fetch_fsmap learned FSMap version 25 2024-04-21T23:19:01.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.975+0000 7f63cb57b740 1 -- 172.21.15.40:0/3961548435 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55696b7883f0 con 0x55696b711430 2024-04-21T23:19:01.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.976+0000 7f63a9ffb640 1 -- 172.21.15.40:0/3961548435 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f63b802ca20 con 0x55696b711430 2024-04-21T23:19:01.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.976+0000 7f63a9ffb640 1 -- 172.21.15.40:0/3961548435 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f63b4028580 con 0x55696b711430 2024-04-21T23:19:01.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.976+0000 7f63cb57b740 10 client.15783 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:19:01.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.976+0000 7f63cb57b740 10 client.15783 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:19:01.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.976+0000 7f63cb57b740 20 client.15783 populate_metadata read hostname 'smithi040' 2024-04-21T23:19:01.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.976+0000 7f63cb57b740 1 --2- 172.21.15.40:0/3961548435 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55696b788b70 0x55696b78af90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:01.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.976+0000 7f63cb57b740 4 client.15783 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T23:19:01.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.976+0000 7f63cb57b740 1 -- 172.21.15.40:0/3961548435 --> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55696b673df0 con 0x55696b788b70 2024-04-21T23:19:01.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.976+0000 7f63c259c640 1 --2- 172.21.15.40:0/3961548435 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55696b788b70 0x55696b78af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:01.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.976+0000 7f63a9ffb640 10 client.15783 ms_handle_connect on v2:172.21.15.111:6838/3712235548 2024-04-21T23:19:01.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.976+0000 7f63c259c640 1 --2- 172.21.15.40:0/3961548435 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55696b788b70 0x55696b78af90 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dacc10f72baefef7 server_cookie=6f8fa883c2c3fcd6 in_seq=0 out_seq=0 2024-04-21T23:19:01.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.977+0000 7f63a9ffb640 1 -- 172.21.15.40:0/3961548435 <== mds.0 v2:172.21.15.111:6838/3712235548 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55696b673df0 con 0x55696b788b70 2024-04-21T23:19:01.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.977+0000 7f63a9ffb640 10 client.15783 handle_command_reply: tid=0 2024-04-21T23:19:01.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.977+0000 7f63cb57b740 1 client.15783 shutdown 2024-04-21T23:19:01.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.977+0000 7f63cb57b740 1 -- 172.21.15.40:0/3961548435 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55696b711430 msgr2=0x55696b786d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:01.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.977+0000 7f63cb57b740 1 --2- 172.21.15.40:0/3961548435 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55696b711430 0x55696b786d90 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f63b4009a30 tx=0x7f63b4011f70 comp rx=0 tx=0).stop 2024-04-21T23:19:01.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.977+0000 7f63cb57b740 1 -- 172.21.15.40:0/3961548435 shutdown_connections 2024-04-21T23:19:01.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.977+0000 7f63cb57b740 1 --2- 172.21.15.40:0/3961548435 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55696b788b70 0x55696b78af90 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:01.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.977+0000 7f63cb57b740 1 --2- 172.21.15.40:0/3961548435 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55696b708850 0x55696b70bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:01.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.977+0000 7f63cb57b740 1 --2- 172.21.15.40:0/3961548435 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55696b711430 0x55696b786d90 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:01.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.977+0000 7f63cb57b740 1 --2- 172.21.15.40:0/3961548435 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55696b707f30 0x55696b70b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:01.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.977+0000 7f63a9ffb640 0 client.15783 ms_handle_reset on v2:172.21.15.111:6838/3712235548 2024-04-21T23:19:01.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.977+0000 7f63cb57b740 1 -- 172.21.15.40:0/3961548435 >> 172.21.15.40:0/3961548435 conn(0x55696b702660 msgr2=0x55696b70e4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:01.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.977+0000 7f63cb57b740 1 -- 172.21.15.40:0/3961548435 shutdown_connections 2024-04-21T23:19:01.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.977+0000 7f63cb57b740 1 -- 172.21.15.40:0/3961548435 wait complete. 2024-04-21T23:19:01.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.977+0000 7f63cb57b740 20 client.15783 trim_cache size 0 max 16384 2024-04-21T23:19:01.983 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-21T23:19:01.983 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "no active scrubs running", 2024-04-21T23:19:01.983 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": {} 2024-04-21T23:19:01.983 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-21T23:19:01.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.977+0000 7f63c898c640 1 -- 172.21.15.40:0/822304289 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f63ac0a2e20 msgr2=0x7f63ac0a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:01.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.977+0000 7f63c898c640 1 --2- 172.21.15.40:0/822304289 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f63ac0a2e20 0x7f63ac0a52e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f63b4005fd0 tx=0x7f63b4009440 comp rx=0 tx=0).stop 2024-04-21T23:19:01.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.977+0000 7f63c898c640 1 -- 172.21.15.40:0/822304289 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63c4100610 msgr2=0x7f63c4197fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:01.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.977+0000 7f63c898c640 1 --2- 172.21.15.40:0/822304289 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63c4100610 0x7f63c4197fe0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f63bc00d950 tx=0x7f63bc00de10 comp rx=0 tx=0).stop 2024-04-21T23:19:01.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.978+0000 7f63c898c640 1 -- 172.21.15.40:0/822304289 shutdown_connections 2024-04-21T23:19:01.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.978+0000 7f63c898c640 1 --2- 172.21.15.40:0/822304289 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f63ac0a2e20 0x7f63ac0a52e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:01.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.978+0000 7f63c898c640 1 --2- 172.21.15.40:0/822304289 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63c4100fd0 0x7f63c4198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:01.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.978+0000 7f63c898c640 1 --2- 172.21.15.40:0/822304289 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f63c41066b0 0x7f63c419c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:01.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.978+0000 7f63c898c640 1 --2- 172.21.15.40:0/822304289 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63c4100610 0x7f63c4197fe0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:01.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.978+0000 7f63c898c640 1 -- 172.21.15.40:0/822304289 >> 172.21.15.40:0/822304289 conn(0x7f63c4077740 msgr2=0x7f63c40ff8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:01.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.978+0000 7f63c898c640 1 -- 172.21.15.40:0/822304289 shutdown_connections 2024-04-21T23:19:01.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:01.978+0000 7f63c898c640 1 -- 172.21.15.40:0/822304289 wait complete. 2024-04-21T23:19:01.988 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T23:19:01.988 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:19:02.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.100+0000 7f7a6354f640 1 -- 172.21.15.40:0/4064038573 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a5c1032f0 msgr2=0x7f7a5c10b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:02.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.100+0000 7f7a6354f640 1 --2- 172.21.15.40:0/4064038573 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a5c1032f0 0x7f7a5c10b7d0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f7a4c009900 tx=0x7f7a4c02bd60 comp rx=0 tx=0).stop 2024-04-21T23:19:02.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.100+0000 7f7a6354f640 1 -- 172.21.15.40:0/4064038573 shutdown_connections 2024-04-21T23:19:02.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.100+0000 7f7a6354f640 1 --2- 172.21.15.40:0/4064038573 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7a5c10bd10 0x7f7a5c10e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.100+0000 7f7a6354f640 1 --2- 172.21.15.40:0/4064038573 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a5c1032f0 0x7f7a5c10b7d0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.100+0000 7f7a6354f640 1 --2- 172.21.15.40:0/4064038573 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7a5c1029d0 0x7f7a5c102db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.100+0000 7f7a6354f640 1 -- 172.21.15.40:0/4064038573 >> 172.21.15.40:0/4064038573 conn(0x7f7a5c077740 msgr2=0x7f7a5c077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:02.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.100+0000 7f7a6354f640 1 -- 172.21.15.40:0/4064038573 shutdown_connections 2024-04-21T23:19:02.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.100+0000 7f7a6354f640 1 -- 172.21.15.40:0/4064038573 wait complete. 2024-04-21T23:19:02.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.101+0000 7f7a6354f640 1 Processor -- start 2024-04-21T23:19:02.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.101+0000 7f7a6354f640 1 -- start start 2024-04-21T23:19:02.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.101+0000 7f7a6354f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7a5c1029d0 0x7f7a5c100620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:02.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.101+0000 7f7a6354f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a5c1032f0 0x7f7a5c100b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:02.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.101+0000 7f7a6354f640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7a5c10bd10 0x7f7a5c1010a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:02.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.101+0000 7f7a6354f640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7a5c111b60 con 0x7f7a5c1032f0 2024-04-21T23:19:02.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.101+0000 7f7a6354f640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f7a5c1119e0 con 0x7f7a5c10bd10 2024-04-21T23:19:02.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.101+0000 7f7a6354f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7a5c111ce0 con 0x7f7a5c1029d0 2024-04-21T23:19:02.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.101+0000 7f7a60aea640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a5c1032f0 0x7f7a5c100b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:02.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.101+0000 7f7a60aea640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a5c1032f0 0x7f7a5c100b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47492/0 (socket says 172.21.15.40:47492) 2024-04-21T23:19:02.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.101+0000 7f7a60aea640 1 -- 172.21.15.40:0/2834389560 learned_addr learned my addr 172.21.15.40:0/2834389560 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:02.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.101+0000 7f7a61aec640 1 --2- 172.21.15.40:0/2834389560 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7a5c10bd10 0x7f7a5c1010a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:02.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.101+0000 7f7a60aea640 1 -- 172.21.15.40:0/2834389560 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7a5c1029d0 msgr2=0x7f7a5c100620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:02.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.101+0000 7f7a60aea640 1 --2- 172.21.15.40:0/2834389560 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7a5c1029d0 0x7f7a5c100620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.101+0000 7f7a60aea640 1 -- 172.21.15.40:0/2834389560 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7a5c10bd10 msgr2=0x7f7a5c1010a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:02.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.101+0000 7f7a60aea640 1 --2- 172.21.15.40:0/2834389560 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7a5c10bd10 0x7f7a5c1010a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.101+0000 7f7a60aea640 1 -- 172.21.15.40:0/2834389560 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a5c1a64a0 con 0x7f7a5c1032f0 2024-04-21T23:19:02.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.102+0000 7f7a60aea640 1 --2- 172.21.15.40:0/2834389560 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a5c1032f0 0x7f7a5c100b60 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f7a4c009a20 tx=0x7f7a4c02df60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:02.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.102+0000 7f7a527fc640 1 -- 172.21.15.40:0/2834389560 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a4c043020 con 0x7f7a5c1032f0 2024-04-21T23:19:02.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.102+0000 7f7a527fc640 1 -- 172.21.15.40:0/2834389560 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a4c0054e0 con 0x7f7a5c1032f0 2024-04-21T23:19:02.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.102+0000 7f7a6354f640 1 -- 172.21.15.40:0/2834389560 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a5c1a6730 con 0x7f7a5c1032f0 2024-04-21T23:19:02.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.102+0000 7f7a527fc640 1 -- 172.21.15.40:0/2834389560 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a4c003ce0 con 0x7f7a5c1032f0 2024-04-21T23:19:02.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.102+0000 7f7a6354f640 1 -- 172.21.15.40:0/2834389560 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a5c1a6cd0 con 0x7f7a5c1032f0 2024-04-21T23:19:02.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.103+0000 7f7a6354f640 1 -- 172.21.15.40:0/2834389560 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a5c068560 con 0x7f7a5c1032f0 2024-04-21T23:19:02.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.105+0000 7f7a527fc640 1 -- 172.21.15.40:0/2834389560 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 25) ==== 138117+0+0 (secure 0 0 0) 0x7f7a4c046020 con 0x7f7a5c1032f0 2024-04-21T23:19:02.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.105+0000 7f7a527fc640 1 --2- 172.21.15.40:0/2834389560 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f7a480a2f60 0x7f7a480a5420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:02.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.105+0000 7f7a612eb640 1 --2- 172.21.15.40:0/2834389560 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f7a480a2f60 0x7f7a480a5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:02.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.105+0000 7f7a527fc640 1 -- 172.21.15.40:0/2834389560 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f7a4c0f1be0 con 0x7f7a5c1032f0 2024-04-21T23:19:02.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.106+0000 7f7a612eb640 1 --2- 172.21.15.40:0/2834389560 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f7a480a2f60 0x7f7a480a5420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7a58005fd0 tx=0x7f7a58005850 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:02.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.110+0000 7f7a527fc640 1 -- 172.21.15.40:0/2834389560 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f7a4c04d020 con 0x7f7a5c1032f0 2024-04-21T23:19:02.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.346+0000 7f7a6354f640 1 -- 172.21.15.40:0/2834389560 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7a5c1a6ea0 con 0x7f7a5c1032f0 2024-04-21T23:19:02.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.347+0000 7f7a527fc640 1 -- 172.21.15.40:0/2834389560 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11882 (secure 0 0 0) 0x7f7a4c0be3b0 con 0x7f7a5c1032f0 2024-04-21T23:19:02.346 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:19:02.346 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/1571948830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1571948830},{"type":"v1","addr":"172.21.15.40:6835","nonce":1571948830}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/969356642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":969356642},{"type":"v1","addr":"172.21.15.40:6837","nonce":969356642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6839/3025057845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":3025057845},{"type":"v1","addr":"172.21.15.40:6839","nonce":3025057845}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6841/3350488804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":3350488804},{"type":"v1","addr":"172.21.15.40:6841","nonce":3350488804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/3438493203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":3438493203},{"type":"v1","addr":"172.21.15.174:6833","nonce":3438493203}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/3105752678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":3105752678},{"type":"v1","addr":"172.21.15.111:6833","nonce":3105752678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T23:15:55.742233+0000","modified":"2024-04-21T23:18:21.879184+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24521,"mds_1":24502,"mds_2":24494},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24461":{"gid":24461,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6835/2704462092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2704462092},{"type":"v1","addr":"172.21.15.111:6835","nonce":2704462092}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/3905733045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3905733045},{"type":"v1","addr":"172.21.15.174:6835","nonce":3905733045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6837/465579746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":465579746},{"type":"v1","addr":"172.21.15.174:6837","nonce":465579746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/2703764182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":2703764182},{"type":"v1","addr":"172.21.15.111:6837","nonce":2703764182}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24502":{"gid":24502,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.174:6839/2467509274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":2467509274},{"type":"v1","addr":"172.21.15.174:6839","nonce":2467509274}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3712235548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3712235548},{"type":"v1","addr":"172.21.15.111:6839","nonce":3712235548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24521,24494,24502]},"id":1}]} 2024-04-21T23:19:02.347 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 25 2024-04-21T23:19:02.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.349+0000 7f7a6354f640 1 -- 172.21.15.40:0/2834389560 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f7a480a2f60 msgr2=0x7f7a480a5420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:02.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.349+0000 7f7a6354f640 1 --2- 172.21.15.40:0/2834389560 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f7a480a2f60 0x7f7a480a5420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7a58005fd0 tx=0x7f7a58005850 comp rx=0 tx=0).stop 2024-04-21T23:19:02.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.349+0000 7f7a6354f640 1 -- 172.21.15.40:0/2834389560 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a5c1032f0 msgr2=0x7f7a5c100b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:02.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.349+0000 7f7a6354f640 1 --2- 172.21.15.40:0/2834389560 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a5c1032f0 0x7f7a5c100b60 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f7a4c009a20 tx=0x7f7a4c02df60 comp rx=0 tx=0).stop 2024-04-21T23:19:02.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.349+0000 7f7a6354f640 1 -- 172.21.15.40:0/2834389560 shutdown_connections 2024-04-21T23:19:02.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.349+0000 7f7a6354f640 1 --2- 172.21.15.40:0/2834389560 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f7a480a2f60 0x7f7a480a5420 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.349+0000 7f7a6354f640 1 --2- 172.21.15.40:0/2834389560 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7a5c10bd10 0x7f7a5c1010a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.349+0000 7f7a6354f640 1 --2- 172.21.15.40:0/2834389560 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a5c1032f0 0x7f7a5c100b60 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.349+0000 7f7a6354f640 1 --2- 172.21.15.40:0/2834389560 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7a5c1029d0 0x7f7a5c100620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.349+0000 7f7a6354f640 1 -- 172.21.15.40:0/2834389560 >> 172.21.15.40:0/2834389560 conn(0x7f7a5c077740 msgr2=0x7f7a5c0fd710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:02.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.349+0000 7f7a6354f640 1 -- 172.21.15.40:0/2834389560 shutdown_connections 2024-04-21T23:19:02.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.349+0000 7f7a6354f640 1 -- 172.21.15.40:0/2834389560 wait complete. 2024-04-21T23:19:02.359 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-21T23:19:02.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.466+0000 7ffbf9d75640 1 -- 172.21.15.40:0/4208142905 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffbf406a780 msgr2=0x7ffbf410e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:02.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.466+0000 7ffbf9d75640 1 --2- 172.21.15.40:0/4208142905 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffbf406a780 0x7ffbf410e5f0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7ffbec009cd0 tx=0x7ffbec02c0f0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.467+0000 7ffbf9d75640 1 -- 172.21.15.40:0/4208142905 shutdown_connections 2024-04-21T23:19:02.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.467+0000 7ffbf9d75640 1 --2- 172.21.15.40:0/4208142905 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffbf4069dc0 0x7ffbf406a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.467+0000 7ffbf9d75640 1 --2- 172.21.15.40:0/4208142905 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffbf406a780 0x7ffbf410e5f0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.467+0000 7ffbf9d75640 1 --2- 172.21.15.40:0/4208142905 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffbf4107730 0x7ffbf4107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.467+0000 7ffbf9d75640 1 -- 172.21.15.40:0/4208142905 >> 172.21.15.40:0/4208142905 conn(0x7ffbf40776a0 msgr2=0x7ffbf4077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:02.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.467+0000 7ffbf9d75640 1 -- 172.21.15.40:0/4208142905 shutdown_connections 2024-04-21T23:19:02.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.467+0000 7ffbf9d75640 1 -- 172.21.15.40:0/4208142905 wait complete. 2024-04-21T23:19:02.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.467+0000 7ffbf9d75640 1 Processor -- start 2024-04-21T23:19:02.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.467+0000 7ffbf9d75640 1 -- start start 2024-04-21T23:19:02.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.468+0000 7ffbf9d75640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffbf4069dc0 0x7ffbf4197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:02.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.468+0000 7ffbf9d75640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffbf406a780 0x7ffbf4198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:02.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.468+0000 7ffbf9d75640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffbf4107730 0x7ffbf419c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:02.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.468+0000 7ffbf9d75640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ffbf4111b40 con 0x7ffbf406a780 2024-04-21T23:19:02.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.468+0000 7ffbf9d75640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ffbf41119c0 con 0x7ffbf4069dc0 2024-04-21T23:19:02.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.468+0000 7ffbf9d75640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ffbf4111cc0 con 0x7ffbf4107730 2024-04-21T23:19:02.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.468+0000 7ffbeaffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffbf406a780 0x7ffbf4198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:02.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.468+0000 7ffbeaffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffbf406a780 0x7ffbf4198420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47508/0 (socket says 172.21.15.40:47508) 2024-04-21T23:19:02.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.468+0000 7ffbeaffd640 1 -- 172.21.15.40:0/3190265097 learned_addr learned my addr 172.21.15.40:0/3190265097 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:02.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.468+0000 7ffbf37fe640 1 --2- 172.21.15.40:0/3190265097 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffbf4069dc0 0x7ffbf4197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:02.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.468+0000 7ffbf3fff640 1 --2- 172.21.15.40:0/3190265097 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffbf4107730 0x7ffbf419c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:02.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.468+0000 7ffbeaffd640 1 -- 172.21.15.40:0/3190265097 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffbf4107730 msgr2=0x7ffbf419c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:02.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.468+0000 7ffbeaffd640 1 --2- 172.21.15.40:0/3190265097 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffbf4107730 0x7ffbf419c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.468+0000 7ffbeaffd640 1 -- 172.21.15.40:0/3190265097 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffbf4069dc0 msgr2=0x7ffbf4197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:02.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.468+0000 7ffbeaffd640 1 --2- 172.21.15.40:0/3190265097 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffbf4069dc0 0x7ffbf4197ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.468+0000 7ffbeaffd640 1 -- 172.21.15.40:0/3190265097 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbf419ced0 con 0x7ffbf406a780 2024-04-21T23:19:02.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.468+0000 7ffbeaffd640 1 --2- 172.21.15.40:0/3190265097 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffbf406a780 0x7ffbf4198420 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc00c920 tx=0x7ffbdc00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:02.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.468+0000 7ffbf17fa640 1 -- 172.21.15.40:0/3190265097 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbdc007c70 con 0x7ffbf406a780 2024-04-21T23:19:02.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.469+0000 7ffbf9d75640 1 -- 172.21.15.40:0/3190265097 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbf419d1c0 con 0x7ffbf406a780 2024-04-21T23:19:02.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.469+0000 7ffbf17fa640 1 -- 172.21.15.40:0/3190265097 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffbdc007e10 con 0x7ffbf406a780 2024-04-21T23:19:02.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.469+0000 7ffbf17fa640 1 -- 172.21.15.40:0/3190265097 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbdc018890 con 0x7ffbf406a780 2024-04-21T23:19:02.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.469+0000 7ffbf9d75640 1 -- 172.21.15.40:0/3190265097 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbf41a4a50 con 0x7ffbf406a780 2024-04-21T23:19:02.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.471+0000 7ffbf17fa640 1 -- 172.21.15.40:0/3190265097 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 25) ==== 138117+0+0 (secure 0 0 0) 0x7ffbdc018b10 con 0x7ffbf406a780 2024-04-21T23:19:02.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.471+0000 7ffbfca35740 1 Processor -- start 2024-04-21T23:19:02.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.471+0000 7ffbfca35740 1 -- start start 2024-04-21T23:19:02.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.472+0000 7ffbf17fa640 1 --2- 172.21.15.40:0/3190265097 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7ffbe00a2eb0 0x7ffbe00a5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:02.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.472+0000 7ffbfca35740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec58c32d50 0x55ec58c33130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:02.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.472+0000 7ffbfca35740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ec58c33670 0x55ec58c403f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:02.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.472+0000 7ffbfca35740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ec58c40930 0x55ec58c42d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:02.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.472+0000 7ffbfca35740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55ec58b97530 con 0x55ec58c33670 2024-04-21T23:19:02.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.472+0000 7ffbfca35740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55ec58b3f830 con 0x55ec58c40930 2024-04-21T23:19:02.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.472+0000 7ffbfca35740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55ec58b934f0 con 0x55ec58c32d50 2024-04-21T23:19:02.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.472+0000 7ffbf3fff640 1 --2- 172.21.15.40:0/3190265097 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7ffbe00a2eb0 0x7ffbe00a5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:02.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.472+0000 7ffbf17fa640 1 -- 172.21.15.40:0/3190265097 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7ffbdc0cfc30 con 0x7ffbf406a780 2024-04-21T23:19:02.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.472+0000 7ffbeaffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ec58c33670 0x55ec58c403f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:02.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.472+0000 7ffbeaffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ec58c33670 0x55ec58c403f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47524/0 (socket says 172.21.15.40:47524) 2024-04-21T23:19:02.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.472+0000 7ffbeaffd640 1 -- 172.21.15.40:0/1472377007 learned_addr learned my addr 172.21.15.40:0/1472377007 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:02.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.472+0000 7ffbf37fe640 1 --2- 172.21.15.40:0/1472377007 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec58c32d50 0x55ec58c33130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:02.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.472+0000 7ffbf3fff640 1 --2- 172.21.15.40:0/1472377007 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ec58c40930 0x55ec58c42d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:02.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.472+0000 7ffbf37fe640 1 -- 172.21.15.40:0/1472377007 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ec58c40930 msgr2=0x55ec58c42d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:02.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.472+0000 7ffbf37fe640 1 --2- 172.21.15.40:0/1472377007 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ec58c40930 0x55ec58c42d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.472+0000 7ffbf37fe640 1 -- 172.21.15.40:0/1472377007 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ec58c33670 msgr2=0x55ec58c403f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:02.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.472+0000 7ffbf37fe640 1 --2- 172.21.15.40:0/1472377007 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ec58c33670 0x55ec58c403f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.472+0000 7ffbf37fe640 1 -- 172.21.15.40:0/1472377007 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ec58b632c0 con 0x55ec58c32d50 2024-04-21T23:19:02.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.472+0000 7ffbeaffd640 1 --2- 172.21.15.40:0/1472377007 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ec58c33670 0x55ec58c403f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:19:02.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.472+0000 7ffbf3fff640 1 --2- 172.21.15.40:0/3190265097 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7ffbe00a2eb0 0x7ffbe00a5370 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ffbec006010 tx=0x7ffbec032c90 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:02.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.473+0000 7ffbf37fe640 1 --2- 172.21.15.40:0/1472377007 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec58c32d50 0x55ec58c33130 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ffbe400aa40 tx=0x7ffbe400af00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b7a64b212e86a5c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:02.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.473+0000 7ffbe9ffb640 1 -- 172.21.15.40:0/1472377007 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbe4004020 con 0x55ec58c32d50 2024-04-21T23:19:02.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.473+0000 7ffbe9ffb640 1 -- 172.21.15.40:0/1472377007 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffbe40041c0 con 0x55ec58c32d50 2024-04-21T23:19:02.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.473+0000 7ffbe9ffb640 1 -- 172.21.15.40:0/1472377007 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbe40187b0 con 0x55ec58c32d50 2024-04-21T23:19:02.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.474+0000 7ffbfca35740 1 -- 172.21.15.40:0/1472377007 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec58c32d50 msgr2=0x55ec58c33130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:02.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.474+0000 7ffbfca35740 1 --2- 172.21.15.40:0/1472377007 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec58c32d50 0x55ec58c33130 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ffbe400aa40 tx=0x7ffbe400af00 comp rx=0 tx=0).stop 2024-04-21T23:19:02.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.474+0000 7ffbfca35740 1 -- 172.21.15.40:0/1472377007 shutdown_connections 2024-04-21T23:19:02.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.474+0000 7ffbfca35740 1 --2- 172.21.15.40:0/1472377007 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ec58c40930 0x55ec58c42d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.474+0000 7ffbfca35740 1 --2- 172.21.15.40:0/1472377007 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ec58c33670 0x55ec58c403f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.474+0000 7ffbfca35740 1 --2- 172.21.15.40:0/1472377007 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec58c32d50 0x55ec58c33130 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.474+0000 7ffbfca35740 1 -- 172.21.15.40:0/1472377007 >> 172.21.15.40:0/1472377007 conn(0x55ec58c2d750 msgr2=0x55ec58c2db30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:02.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.474+0000 7ffbfca35740 1 -- 172.21.15.40:0/1472377007 shutdown_connections 2024-04-21T23:19:02.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.474+0000 7ffbfca35740 1 -- 172.21.15.40:0/1472377007 wait complete. 2024-04-21T23:19:02.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.474+0000 7ffbfca35740 1 -- start start 2024-04-21T23:19:02.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.474+0000 7ffbfca35740 1 Processor -- start 2024-04-21T23:19:02.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.475+0000 7ffbfca35740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec58c32d50 0x55ec58c402e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:02.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.475+0000 7ffbfca35740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ec58c33670 0x55ec58c3d670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:02.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.475+0000 7ffbfca35740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ec58c40930 0x55ec58c3dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:02.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.475+0000 7ffbfca35740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55ec58c436f0 con 0x55ec58c40930 2024-04-21T23:19:02.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.475+0000 7ffbfca35740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55ec58b52f10 con 0x55ec58c33670 2024-04-21T23:19:02.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.475+0000 7ffbfca35740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55ec58c43870 con 0x55ec58c32d50 2024-04-21T23:19:02.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.475+0000 7ffbfca35740 10 client.0 _ll_register_callbacks cb 0x55ec588aabe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:19:02.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.475+0000 7ffbf3fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ec58c40930 0x55ec58c3dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:02.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.475+0000 7ffbf3fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ec58c40930 0x55ec58c3dc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47528/0 (socket says 172.21.15.40:47528) 2024-04-21T23:19:02.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.475+0000 7ffbf3fff640 1 -- 172.21.15.40:0/548860432 learned_addr learned my addr 172.21.15.40:0/548860432 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:02.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.475+0000 7ffbf37fe640 1 --2- 172.21.15.40:0/548860432 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec58c32d50 0x55ec58c402e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:02.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.475+0000 7ffbeaffd640 1 --2- 172.21.15.40:0/548860432 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ec58c33670 0x55ec58c3d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:02.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.475+0000 7ffbf3fff640 1 -- 172.21.15.40:0/548860432 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec58c32d50 msgr2=0x55ec58c402e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:02.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.475+0000 7ffbf3fff640 1 --2- 172.21.15.40:0/548860432 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec58c32d50 0x55ec58c402e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.475+0000 7ffbf3fff640 1 -- 172.21.15.40:0/548860432 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ec58c33670 msgr2=0x55ec58c3d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:02.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.475+0000 7ffbf3fff640 1 --2- 172.21.15.40:0/548860432 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ec58c33670 0x55ec58c3d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.475+0000 7ffbf3fff640 1 -- 172.21.15.40:0/548860432 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ec58b24c80 con 0x55ec58c40930 2024-04-21T23:19:02.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.476+0000 7ffbf3fff640 1 --2- 172.21.15.40:0/548860432 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ec58c40930 0x55ec58c3dc10 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7ffbec02dfd0 tx=0x7ffbec02c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dce657a9b9754b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:02.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.476+0000 7ffbd2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-21T23:19:02.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.476+0000 7ffbd2ffd640 1 -- 172.21.15.40:0/548860432 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbec051020 con 0x55ec58c40930 2024-04-21T23:19:02.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.476+0000 7ffbd2ffd640 1 -- 172.21.15.40:0/548860432 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7ffbec004240 con 0x55ec58c40930 2024-04-21T23:19:02.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.476+0000 7ffbfca35740 1 -- 172.21.15.40:0/548860432 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffbe40041c0 con 0x55ec58c40930 2024-04-21T23:19:02.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.476+0000 7ffbd2ffd640 1 -- 172.21.15.40:0/548860432 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7ffbec05c050 con 0x55ec58c40930 2024-04-21T23:19:02.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.476+0000 7ffbfca35740 10 client.15801 fetch_fsmap learned FSMap version 25 2024-04-21T23:19:02.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.476+0000 7ffbfca35740 1 -- 172.21.15.40:0/548860432 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55ec588ab3f0 con 0x55ec58c40930 2024-04-21T23:19:02.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.476+0000 7ffbd2ffd640 1 -- 172.21.15.40:0/548860432 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffbe40041c0 con 0x55ec58c40930 2024-04-21T23:19:02.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.476+0000 7ffbd2ffd640 1 -- 172.21.15.40:0/548860432 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7ffbec04d930 con 0x55ec58c40930 2024-04-21T23:19:02.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.476+0000 7ffbfca35740 10 client.15801 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:19:02.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.476+0000 7ffbfca35740 10 client.15801 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-21T23:19:02.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.476+0000 7ffbfca35740 20 client.15801 populate_metadata read hostname 'smithi040' 2024-04-21T23:19:02.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.476+0000 7ffbfca35740 1 --2- 172.21.15.40:0/548860432 >> [v2:172.21.15.111:6836/2703764182,v1:172.21.15.111:6837/2703764182] conn(0x55ec58c50a20 0x55ec58c52e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:02.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.476+0000 7ffbfca35740 4 client.15801 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:19:02.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.477+0000 7ffbfca35740 1 -- 172.21.15.40:0/548860432 --> [v2:172.21.15.111:6836/2703764182,v1:172.21.15.111:6837/2703764182] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ec58ba4a70 con 0x55ec58c50a20 2024-04-21T23:19:02.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.477+0000 7ffbf37fe640 1 --2- 172.21.15.40:0/548860432 >> [v2:172.21.15.111:6836/2703764182,v1:172.21.15.111:6837/2703764182] conn(0x55ec58c50a20 0x55ec58c52e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:02.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.477+0000 7ffbf37fe640 1 --2- 172.21.15.40:0/548860432 >> [v2:172.21.15.111:6836/2703764182,v1:172.21.15.111:6837/2703764182] conn(0x55ec58c50a20 0x55ec58c52e50 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9affda693abb8988 server_cookie=6e4bca492a6d8d7b in_seq=0 out_seq=0 2024-04-21T23:19:02.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.477+0000 7ffbd2ffd640 10 client.15801 ms_handle_connect on v2:172.21.15.111:6836/2703764182 2024-04-21T23:19:02.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.478+0000 7ffbd2ffd640 1 -- 172.21.15.40:0/548860432 <== mds.2 v2:172.21.15.111:6836/2703764182 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ec58ba4a70 con 0x55ec58c50a20 2024-04-21T23:19:02.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.478+0000 7ffbd2ffd640 10 client.15801 handle_command_reply: tid=0 2024-04-21T23:19:02.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.478+0000 7ffbfca35740 1 client.15801 shutdown 2024-04-21T23:19:02.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.478+0000 7ffbfca35740 1 -- 172.21.15.40:0/548860432 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ec58c40930 msgr2=0x55ec58c3dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:02.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.478+0000 7ffbfca35740 1 --2- 172.21.15.40:0/548860432 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ec58c40930 0x55ec58c3dc10 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7ffbec02dfd0 tx=0x7ffbec02c0f0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.478+0000 7ffbfca35740 1 -- 172.21.15.40:0/548860432 shutdown_connections 2024-04-21T23:19:02.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.478+0000 7ffbfca35740 1 --2- 172.21.15.40:0/548860432 >> [v2:172.21.15.111:6836/2703764182,v1:172.21.15.111:6837/2703764182] conn(0x55ec58c50a20 0x55ec58c52e50 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.478+0000 7ffbfca35740 1 --2- 172.21.15.40:0/548860432 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ec58c33670 0x55ec58c3d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.478+0000 7ffbfca35740 1 --2- 172.21.15.40:0/548860432 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ec58c40930 0x55ec58c3dc10 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.478+0000 7ffbd2ffd640 0 client.15801 ms_handle_reset on v2:172.21.15.111:6836/2703764182 2024-04-21T23:19:02.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.478+0000 7ffbfca35740 1 --2- 172.21.15.40:0/548860432 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec58c32d50 0x55ec58c402e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.478+0000 7ffbfca35740 1 -- 172.21.15.40:0/548860432 >> 172.21.15.40:0/548860432 conn(0x55ec58c2d750 msgr2=0x55ec58c2f4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:02.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.478+0000 7ffbfca35740 1 -- 172.21.15.40:0/548860432 shutdown_connections 2024-04-21T23:19:02.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.479+0000 7ffbfca35740 1 -- 172.21.15.40:0/548860432 wait complete. 2024-04-21T23:19:02.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.479+0000 7ffbfca35740 20 client.15801 trim_cache size 0 max 16384 2024-04-21T23:19:02.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.494+0000 7ffbfca35740 1 Processor -- start 2024-04-21T23:19:02.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.494+0000 7ffbfca35740 1 -- start start 2024-04-21T23:19:02.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.494+0000 7ffbfca35740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ec58c32d50 0x55ec58c4c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:02.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.494+0000 7ffbfca35740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ec58c33670 0x55ec58c3fda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:02.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.494+0000 7ffbfca35740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec58c40930 0x55ec58c402e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:02.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.494+0000 7ffbfca35740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ffbf41af2a0 con 0x55ec58c33670 2024-04-21T23:19:02.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.494+0000 7ffbfca35740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ffbf41acb70 con 0x55ec58c32d50 2024-04-21T23:19:02.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.494+0000 7ffbfca35740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ffbf41aa9c0 con 0x55ec58c40930 2024-04-21T23:19:02.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.494+0000 7ffbeaffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ec58c33670 0x55ec58c3fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:02.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.494+0000 7ffbeaffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ec58c33670 0x55ec58c3fda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47530/0 (socket says 172.21.15.40:47530) 2024-04-21T23:19:02.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.494+0000 7ffbeaffd640 1 -- 172.21.15.40:0/4258439929 learned_addr learned my addr 172.21.15.40:0/4258439929 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:02.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.494+0000 7ffbf37fe640 1 --2- 172.21.15.40:0/4258439929 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ec58c32d50 0x55ec58c4c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:02.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.494+0000 7ffbf3fff640 1 --2- 172.21.15.40:0/4258439929 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec58c40930 0x55ec58c402e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:02.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.494+0000 7ffbeaffd640 1 -- 172.21.15.40:0/4258439929 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec58c40930 msgr2=0x55ec58c402e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:02.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.494+0000 7ffbeaffd640 1 --2- 172.21.15.40:0/4258439929 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec58c40930 0x55ec58c402e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.494+0000 7ffbeaffd640 1 -- 172.21.15.40:0/4258439929 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ec58c32d50 msgr2=0x55ec58c4c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:02.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.494+0000 7ffbeaffd640 1 --2- 172.21.15.40:0/4258439929 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ec58c32d50 0x55ec58c4c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.494+0000 7ffbeaffd640 1 -- 172.21.15.40:0/4258439929 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ec58c334a0 con 0x55ec58c33670 2024-04-21T23:19:02.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.494+0000 7ffbf3fff640 1 --2- 172.21.15.40:0/4258439929 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec58c40930 0x55ec58c402e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:19:02.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.495+0000 7ffbeaffd640 1 --2- 172.21.15.40:0/4258439929 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ec58c33670 0x55ec58c3fda0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc097fc0 tx=0x7ffbdc00a5e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec3f614b391f4525 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:02.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.495+0000 7ffbe97fa640 1 -- 172.21.15.40:0/4258439929 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbdc09bb60 con 0x55ec58c33670 2024-04-21T23:19:02.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.495+0000 7ffbe97fa640 1 -- 172.21.15.40:0/4258439929 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffbdc09ff10 con 0x55ec58c33670 2024-04-21T23:19:02.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.495+0000 7ffbe97fa640 1 -- 172.21.15.40:0/4258439929 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbdc0a4340 con 0x55ec58c33670 2024-04-21T23:19:02.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.495+0000 7ffbfca35740 1 -- 172.21.15.40:0/4258439929 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ec58c33670 msgr2=0x55ec58c3fda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:02.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.495+0000 7ffbfca35740 1 --2- 172.21.15.40:0/4258439929 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ec58c33670 0x55ec58c3fda0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc097fc0 tx=0x7ffbdc00a5e0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.495+0000 7ffbfca35740 1 -- 172.21.15.40:0/4258439929 shutdown_connections 2024-04-21T23:19:02.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.495+0000 7ffbfca35740 1 --2- 172.21.15.40:0/4258439929 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec58c40930 0x55ec58c402e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.495+0000 7ffbfca35740 1 --2- 172.21.15.40:0/4258439929 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ec58c33670 0x55ec58c3fda0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.495+0000 7ffbfca35740 1 --2- 172.21.15.40:0/4258439929 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ec58c32d50 0x55ec58c4c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.495+0000 7ffbfca35740 1 -- 172.21.15.40:0/4258439929 >> 172.21.15.40:0/4258439929 conn(0x55ec58c2d750 msgr2=0x55ec58c38c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:02.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.496+0000 7ffbfca35740 1 -- 172.21.15.40:0/4258439929 shutdown_connections 2024-04-21T23:19:02.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.496+0000 7ffbfca35740 1 -- 172.21.15.40:0/4258439929 wait complete. 2024-04-21T23:19:02.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.496+0000 7ffbfca35740 1 -- start start 2024-04-21T23:19:02.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.496+0000 7ffbfca35740 1 Processor -- start 2024-04-21T23:19:02.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.496+0000 7ffbfca35740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec58c32d50 0x55ec58c48ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:02.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.496+0000 7ffbfca35740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ec58c33670 0x55ec58c3a070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:02.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.497+0000 7ffbfca35740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ec58c40930 0x55ec58c3a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:02.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.497+0000 7ffbfca35740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ffbf41ab030 con 0x55ec58c40930 2024-04-21T23:19:02.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.497+0000 7ffbfca35740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ffbf41ac4a0 con 0x55ec58c33670 2024-04-21T23:19:02.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.497+0000 7ffbfca35740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ffbf41ab700 con 0x55ec58c32d50 2024-04-21T23:19:02.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.497+0000 7ffbfca35740 10 client.0 _ll_register_callbacks cb 0x7ffbec037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:19:02.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.497+0000 7ffbf3fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ec58c40930 0x55ec58c3a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:02.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.497+0000 7ffbf3fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ec58c40930 0x55ec58c3a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47536/0 (socket says 172.21.15.40:47536) 2024-04-21T23:19:02.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.497+0000 7ffbf3fff640 1 -- 172.21.15.40:0/1144556291 learned_addr learned my addr 172.21.15.40:0/1144556291 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:02.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.497+0000 7ffbeaffd640 1 --2- 172.21.15.40:0/1144556291 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ec58c33670 0x55ec58c3a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:02.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.497+0000 7ffbf37fe640 1 --2- 172.21.15.40:0/1144556291 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec58c32d50 0x55ec58c48ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:02.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.497+0000 7ffbf3fff640 1 -- 172.21.15.40:0/1144556291 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec58c32d50 msgr2=0x55ec58c48ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:02.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.497+0000 7ffbf3fff640 1 --2- 172.21.15.40:0/1144556291 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec58c32d50 0x55ec58c48ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.497+0000 7ffbf3fff640 1 -- 172.21.15.40:0/1144556291 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ec58c33670 msgr2=0x55ec58c3a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:02.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.497+0000 7ffbf3fff640 1 --2- 172.21.15.40:0/1144556291 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ec58c33670 0x55ec58c3a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.497+0000 7ffbf3fff640 1 -- 172.21.15.40:0/1144556291 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ec589757d0 con 0x55ec58c40930 2024-04-21T23:19:02.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.497+0000 7ffbf37fe640 1 --2- 172.21.15.40:0/1144556291 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec58c32d50 0x55ec58c48ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:19:02.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.497+0000 7ffbf3fff640 1 --2- 172.21.15.40:0/1144556291 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ec58c40930 0x55ec58c3a5f0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7ffbec009600 tx=0x7ffbec005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e391e98972b299d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:02.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.497+0000 7ffbd2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-21T23:19:02.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.497+0000 7ffbd2ffd640 1 -- 172.21.15.40:0/1144556291 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbec065060 con 0x55ec58c40930 2024-04-21T23:19:02.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.497+0000 7ffbd2ffd640 1 -- 172.21.15.40:0/1144556291 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7ffbec04d880 con 0x55ec58c40930 2024-04-21T23:19:02.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.497+0000 7ffbfca35740 1 -- 172.21.15.40:0/1144556291 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffbdc09ff10 con 0x55ec58c40930 2024-04-21T23:19:02.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.498+0000 7ffbd2ffd640 1 -- 172.21.15.40:0/1144556291 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7ffbec06c050 con 0x55ec58c40930 2024-04-21T23:19:02.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.498+0000 7ffbfca35740 10 client.15807 fetch_fsmap learned FSMap version 25 2024-04-21T23:19:02.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.498+0000 7ffbfca35740 1 -- 172.21.15.40:0/1144556291 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55ec58cb6dd0 con 0x55ec58c40930 2024-04-21T23:19:02.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.498+0000 7ffbd2ffd640 1 -- 172.21.15.40:0/1144556291 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffbdc09ff10 con 0x55ec58c40930 2024-04-21T23:19:02.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.498+0000 7ffbd2ffd640 1 -- 172.21.15.40:0/1144556291 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7ffbec043600 con 0x55ec58c40930 2024-04-21T23:19:02.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.498+0000 7ffbfca35740 10 client.15807 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:19:02.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.498+0000 7ffbfca35740 10 client.15807 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-21T23:19:02.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.498+0000 7ffbfca35740 20 client.15807 populate_metadata read hostname 'smithi040' 2024-04-21T23:19:02.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.498+0000 7ffbfca35740 1 --2- 172.21.15.40:0/1144556291 >> [v2:172.21.15.111:6836/2703764182,v1:172.21.15.111:6837/2703764182] conn(0x55ec58cb74f0 0x55ec58cb98e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:02.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.498+0000 7ffbfca35740 4 client.15807 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T23:19:02.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.498+0000 7ffbfca35740 1 -- 172.21.15.40:0/1144556291 --> [v2:172.21.15.111:6836/2703764182,v1:172.21.15.111:6837/2703764182] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ec58b9edc0 con 0x55ec58cb74f0 2024-04-21T23:19:02.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.498+0000 7ffbf37fe640 1 --2- 172.21.15.40:0/1144556291 >> [v2:172.21.15.111:6836/2703764182,v1:172.21.15.111:6837/2703764182] conn(0x55ec58cb74f0 0x55ec58cb98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:02.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.499+0000 7ffbf37fe640 1 --2- 172.21.15.40:0/1144556291 >> [v2:172.21.15.111:6836/2703764182,v1:172.21.15.111:6837/2703764182] conn(0x55ec58cb74f0 0x55ec58cb98e0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=82aab14c1e13758b server_cookie=a978d0ffcfe1e361 in_seq=0 out_seq=0 2024-04-21T23:19:02.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.499+0000 7ffbd2ffd640 10 client.15807 ms_handle_connect on v2:172.21.15.111:6836/2703764182 2024-04-21T23:19:02.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.499+0000 7ffbd2ffd640 1 -- 172.21.15.40:0/1144556291 <== mds.2 v2:172.21.15.111:6836/2703764182 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ec58b9edc0 con 0x55ec58cb74f0 2024-04-21T23:19:02.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.499+0000 7ffbd2ffd640 10 client.15807 handle_command_reply: tid=0 2024-04-21T23:19:02.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.499+0000 7ffbfca35740 1 client.15807 shutdown 2024-04-21T23:19:02.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.499+0000 7ffbfca35740 1 -- 172.21.15.40:0/1144556291 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ec58c40930 msgr2=0x55ec58c3a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:02.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.499+0000 7ffbfca35740 1 --2- 172.21.15.40:0/1144556291 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ec58c40930 0x55ec58c3a5f0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7ffbec009600 tx=0x7ffbec005ef0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.499+0000 7ffbfca35740 1 -- 172.21.15.40:0/1144556291 shutdown_connections 2024-04-21T23:19:02.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.499+0000 7ffbfca35740 1 --2- 172.21.15.40:0/1144556291 >> [v2:172.21.15.111:6836/2703764182,v1:172.21.15.111:6837/2703764182] conn(0x55ec58cb74f0 0x55ec58cb98e0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.499+0000 7ffbfca35740 1 --2- 172.21.15.40:0/1144556291 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ec58c33670 0x55ec58c3a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.499+0000 7ffbfca35740 1 --2- 172.21.15.40:0/1144556291 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ec58c40930 0x55ec58c3a5f0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.499+0000 7ffbd2ffd640 0 client.15807 ms_handle_reset on v2:172.21.15.111:6836/2703764182 2024-04-21T23:19:02.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.499+0000 7ffbfca35740 1 --2- 172.21.15.40:0/1144556291 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec58c32d50 0x55ec58c48ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.499+0000 7ffbfca35740 1 -- 172.21.15.40:0/1144556291 >> 172.21.15.40:0/1144556291 conn(0x55ec58c2d750 msgr2=0x55ec58c2bfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:02.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.499+0000 7ffbfca35740 1 -- 172.21.15.40:0/1144556291 shutdown_connections 2024-04-21T23:19:02.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.499+0000 7ffbfca35740 1 -- 172.21.15.40:0/1144556291 wait complete. 2024-04-21T23:19:02.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.499+0000 7ffbfca35740 20 client.15807 trim_cache size 0 max 16384 2024-04-21T23:19:02.507 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-04-21T23:19:02.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.500+0000 7ffbf9d75640 1 -- 172.21.15.40:0/3190265097 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7ffbe00a2eb0 msgr2=0x7ffbe00a5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:02.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.500+0000 7ffbf9d75640 1 --2- 172.21.15.40:0/3190265097 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7ffbe00a2eb0 0x7ffbe00a5370 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ffbec006010 tx=0x7ffbec032c90 comp rx=0 tx=0).stop 2024-04-21T23:19:02.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.500+0000 7ffbf9d75640 1 -- 172.21.15.40:0/3190265097 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffbf406a780 msgr2=0x7ffbf4198420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:02.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.500+0000 7ffbf9d75640 1 --2- 172.21.15.40:0/3190265097 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffbf406a780 0x7ffbf4198420 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc00c920 tx=0x7ffbdc00cde0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.500+0000 7ffbf9d75640 1 -- 172.21.15.40:0/3190265097 shutdown_connections 2024-04-21T23:19:02.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.500+0000 7ffbf9d75640 1 --2- 172.21.15.40:0/3190265097 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7ffbe00a2eb0 0x7ffbe00a5370 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.500+0000 7ffbf9d75640 1 --2- 172.21.15.40:0/3190265097 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffbf4107730 0x7ffbf419c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.500+0000 7ffbf9d75640 1 --2- 172.21.15.40:0/3190265097 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffbf406a780 0x7ffbf4198420 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.500+0000 7ffbf9d75640 1 --2- 172.21.15.40:0/3190265097 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffbf4069dc0 0x7ffbf4197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.500+0000 7ffbf9d75640 1 -- 172.21.15.40:0/3190265097 >> 172.21.15.40:0/3190265097 conn(0x7ffbf40776a0 msgr2=0x7ffbf40fe5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:02.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.500+0000 7ffbf9d75640 1 -- 172.21.15.40:0/3190265097 shutdown_connections 2024-04-21T23:19:02.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.500+0000 7ffbf9d75640 1 -- 172.21.15.40:0/3190265097 wait complete. 2024-04-21T23:19:02.511 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-21T23:19:02.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.628+0000 7fe9bc020640 1 -- 172.21.15.40:0/2644687792 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9b4102150 msgr2=0x7fe9b410e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:02.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.628+0000 7fe9bc020640 1 --2- 172.21.15.40:0/2644687792 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9b4102150 0x7fe9b410e660 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac009cd0 tx=0x7fe9ac02c0f0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.628+0000 7fe9bc020640 1 -- 172.21.15.40:0/2644687792 shutdown_connections 2024-04-21T23:19:02.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.628+0000 7fe9bc020640 1 --2- 172.21.15.40:0/2644687792 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9b4102150 0x7fe9b410e660 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.628+0000 7fe9bc020640 1 --2- 172.21.15.40:0/2644687792 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe9b4101790 0x7fe9b4101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.628+0000 7fe9bc020640 1 --2- 172.21.15.40:0/2644687792 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe9b41077a0 0x7fe9b4107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.628+0000 7fe9bc020640 1 -- 172.21.15.40:0/2644687792 >> 172.21.15.40:0/2644687792 conn(0x7fe9b40fd2a0 msgr2=0x7fe9b40ff6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:02.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.628+0000 7fe9bc020640 1 -- 172.21.15.40:0/2644687792 shutdown_connections 2024-04-21T23:19:02.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.628+0000 7fe9bc020640 1 -- 172.21.15.40:0/2644687792 wait complete. 2024-04-21T23:19:02.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.629+0000 7fe9bc020640 1 Processor -- start 2024-04-21T23:19:02.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.629+0000 7fe9bc020640 1 -- start start 2024-04-21T23:19:02.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.629+0000 7fe9bc020640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe9b4101790 0x7fe9b419a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:02.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.629+0000 7fe9bc020640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9b4102150 0x7fe9b419a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:02.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.629+0000 7fe9bc020640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe9b41077a0 0x7fe9b419eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:02.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.629+0000 7fe9bc020640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe9b4111bb0 con 0x7fe9b4102150 2024-04-21T23:19:02.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.629+0000 7fe9bc020640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe9b4111a30 con 0x7fe9b41077a0 2024-04-21T23:19:02.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.629+0000 7fe9bc020640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe9b4111d30 con 0x7fe9b4101790 2024-04-21T23:19:02.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.629+0000 7fe9b95bb640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9b4102150 0x7fe9b419a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:02.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.630+0000 7fe9b95bb640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9b4102150 0x7fe9b419a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47546/0 (socket says 172.21.15.40:47546) 2024-04-21T23:19:02.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.630+0000 7fe9b95bb640 1 -- 172.21.15.40:0/3642121833 learned_addr learned my addr 172.21.15.40:0/3642121833 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:02.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.630+0000 7fe9ba5bd640 1 --2- 172.21.15.40:0/3642121833 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe9b41077a0 0x7fe9b419eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:02.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.630+0000 7fe9b9dbc640 1 --2- 172.21.15.40:0/3642121833 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe9b4101790 0x7fe9b419a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:02.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.630+0000 7fe9b95bb640 1 -- 172.21.15.40:0/3642121833 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe9b4101790 msgr2=0x7fe9b419a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:02.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.630+0000 7fe9b95bb640 1 --2- 172.21.15.40:0/3642121833 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe9b4101790 0x7fe9b419a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.630+0000 7fe9b95bb640 1 -- 172.21.15.40:0/3642121833 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe9b41077a0 msgr2=0x7fe9b419eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:02.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.630+0000 7fe9b95bb640 1 --2- 172.21.15.40:0/3642121833 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe9b41077a0 0x7fe9b419eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.630+0000 7fe9b95bb640 1 -- 172.21.15.40:0/3642121833 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9b419f2c0 con 0x7fe9b4102150 2024-04-21T23:19:02.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.630+0000 7fe9b95bb640 1 --2- 172.21.15.40:0/3642121833 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9b4102150 0x7fe9b419a810 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fe9a400c970 tx=0x7fe9a400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:02.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.630+0000 7fe9aaffd640 1 -- 172.21.15.40:0/3642121833 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9a4007bb0 con 0x7fe9b4102150 2024-04-21T23:19:02.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.630+0000 7fe9aaffd640 1 -- 172.21.15.40:0/3642121833 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9a4007d50 con 0x7fe9b4102150 2024-04-21T23:19:02.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.630+0000 7fe9bc020640 1 -- 172.21.15.40:0/3642121833 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9b419f5b0 con 0x7fe9b4102150 2024-04-21T23:19:02.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.630+0000 7fe9aaffd640 1 -- 172.21.15.40:0/3642121833 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9a400f450 con 0x7fe9b4102150 2024-04-21T23:19:02.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.630+0000 7fe9bc020640 1 -- 172.21.15.40:0/3642121833 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9b41a6e40 con 0x7fe9b4102150 2024-04-21T23:19:02.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.633+0000 7fe9aaffd640 1 -- 172.21.15.40:0/3642121833 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 25) ==== 138117+0+0 (secure 0 0 0) 0x7fe9a4005020 con 0x7fe9b4102150 2024-04-21T23:19:02.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.633+0000 7fe9bec0a740 1 Processor -- start 2024-04-21T23:19:02.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.633+0000 7fe9bec0a740 1 -- start start 2024-04-21T23:19:02.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.633+0000 7fe9bec0a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55638adb9020 0x55638adb9400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:02.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.633+0000 7fe9bec0a740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55638adb9940 0x55638adc1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:02.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.633+0000 7fe9bec0a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55638adc2520 0x55638adc4910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:02.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.633+0000 7fe9bec0a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55638ad1d530 con 0x55638adb9020 2024-04-21T23:19:02.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.633+0000 7fe9bec0a740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55638acc5830 con 0x55638adb9940 2024-04-21T23:19:02.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.633+0000 7fe9bec0a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55638ad194f0 con 0x55638adc2520 2024-04-21T23:19:02.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.633+0000 7fe9aaffd640 1 --2- 172.21.15.40:0/3642121833 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fe9a00a2eb0 0x7fe9a00a5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:02.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.633+0000 7fe9ba5bd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55638adb9020 0x55638adb9400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:02.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.633+0000 7fe9ba5bd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55638adb9020 0x55638adb9400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47560/0 (socket says 172.21.15.40:47560) 2024-04-21T23:19:02.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.633+0000 7fe9ba5bd640 1 -- 172.21.15.40:0/23787063 learned_addr learned my addr 172.21.15.40:0/23787063 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:02.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.634+0000 7fe9b9dbc640 1 --2- 172.21.15.40:0/23787063 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55638adb9940 0x55638adc1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:02.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.634+0000 7fe9aaffd640 1 -- 172.21.15.40:0/3642121833 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7fe9a40cf820 con 0x7fe9b4102150 2024-04-21T23:19:02.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.634+0000 7fe9b95bb640 1 --2- 172.21.15.40:0/23787063 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55638adc2520 0x55638adc4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:02.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.634+0000 7fe9ba5bd640 1 --2- 172.21.15.40:0/3642121833 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fe9a00a2eb0 0x7fe9a00a5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:02.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.634+0000 7fe9ba5bd640 1 -- 172.21.15.40:0/23787063 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55638adc2520 msgr2=0x55638adc4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:02.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.634+0000 7fe9ba5bd640 1 --2- 172.21.15.40:0/23787063 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55638adc2520 0x55638adc4910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.634+0000 7fe9ba5bd640 1 -- 172.21.15.40:0/23787063 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55638adb9940 msgr2=0x55638adc1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:02.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.634+0000 7fe9ba5bd640 1 --2- 172.21.15.40:0/23787063 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55638adb9940 0x55638adc1fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.634+0000 7fe9ba5bd640 1 -- 172.21.15.40:0/23787063 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55638ace92c0 con 0x55638adb9020 2024-04-21T23:19:02.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.634+0000 7fe9ba5bd640 1 --2- 172.21.15.40:0/23787063 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55638adb9020 0x55638adb9400 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac009a60 tx=0x7fe9ac032c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbbd3a0cb96a8e96 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:02.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.634+0000 7fe9ba5bd640 1 --2- 172.21.15.40:0/3642121833 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fe9a00a2eb0 0x7fe9a00a5370 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac047000 tx=0x7fe9ac048000 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:02.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.634+0000 7fe99ffff640 1 -- 172.21.15.40:0/23787063 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9ac050070 con 0x55638adb9020 2024-04-21T23:19:02.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.634+0000 7fe99ffff640 1 -- 172.21.15.40:0/23787063 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9ac003b70 con 0x55638adb9020 2024-04-21T23:19:02.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.634+0000 7fe99ffff640 1 -- 172.21.15.40:0/23787063 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9ac039040 con 0x55638adb9020 2024-04-21T23:19:02.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.635+0000 7fe9bec0a740 1 -- 172.21.15.40:0/23787063 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55638adb9020 msgr2=0x55638adb9400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:02.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.635+0000 7fe9bec0a740 1 --2- 172.21.15.40:0/23787063 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55638adb9020 0x55638adb9400 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac009a60 tx=0x7fe9ac032c90 comp rx=0 tx=0).stop 2024-04-21T23:19:02.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.635+0000 7fe9bec0a740 1 -- 172.21.15.40:0/23787063 shutdown_connections 2024-04-21T23:19:02.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.635+0000 7fe9bec0a740 1 --2- 172.21.15.40:0/23787063 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55638adb9940 0x55638adc1fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.635+0000 7fe9bec0a740 1 --2- 172.21.15.40:0/23787063 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55638adc2520 0x55638adc4910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.635+0000 7fe9bec0a740 1 --2- 172.21.15.40:0/23787063 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55638adb9020 0x55638adb9400 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.636+0000 7fe9bec0a740 1 -- 172.21.15.40:0/23787063 >> 172.21.15.40:0/23787063 conn(0x55638adb3750 msgr2=0x55638adb3b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:02.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.636+0000 7fe9bec0a740 1 -- 172.21.15.40:0/23787063 shutdown_connections 2024-04-21T23:19:02.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.636+0000 7fe9bec0a740 1 -- 172.21.15.40:0/23787063 wait complete. 2024-04-21T23:19:02.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.636+0000 7fe9bec0a740 1 -- start start 2024-04-21T23:19:02.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.636+0000 7fe9bec0a740 1 Processor -- start 2024-04-21T23:19:02.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.636+0000 7fe9bec0a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55638adb9020 0x55638adc1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:02.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.636+0000 7fe9bec0a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55638adb9940 0x55638adbf270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:02.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.636+0000 7fe9bec0a740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55638adc2520 0x55638adbf880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:02.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.636+0000 7fe9bec0a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55638adc52b0 con 0x55638adb9940 2024-04-21T23:19:02.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.636+0000 7fe9bec0a740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55638acd8f10 con 0x55638adc2520 2024-04-21T23:19:02.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.636+0000 7fe9bec0a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55638adc5430 con 0x55638adb9020 2024-04-21T23:19:02.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.637+0000 7fe9bec0a740 10 client.0 _ll_register_callbacks cb 0x55638aa30be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:19:02.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.637+0000 7fe9b95bb640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55638adb9940 0x55638adbf270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:02.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.637+0000 7fe9b95bb640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55638adb9940 0x55638adbf270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47562/0 (socket says 172.21.15.40:47562) 2024-04-21T23:19:02.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.637+0000 7fe9b95bb640 1 -- 172.21.15.40:0/1913360505 learned_addr learned my addr 172.21.15.40:0/1913360505 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:02.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.637+0000 7fe9b9dbc640 1 --2- 172.21.15.40:0/1913360505 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55638adb9020 0x55638adc1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:02.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.637+0000 7fe9ba5bd640 1 --2- 172.21.15.40:0/1913360505 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55638adc2520 0x55638adbf880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:02.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.637+0000 7fe9b95bb640 1 -- 172.21.15.40:0/1913360505 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55638adb9020 msgr2=0x55638adc1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:02.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.637+0000 7fe9b95bb640 1 --2- 172.21.15.40:0/1913360505 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55638adb9020 0x55638adc1ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.637+0000 7fe9b95bb640 1 -- 172.21.15.40:0/1913360505 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55638adc2520 msgr2=0x55638adbf880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:02.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.637+0000 7fe9b95bb640 1 --2- 172.21.15.40:0/1913360505 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55638adc2520 0x55638adbf880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.637+0000 7fe9b95bb640 1 -- 172.21.15.40:0/1913360505 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55638acaac80 con 0x55638adb9940 2024-04-21T23:19:02.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.637+0000 7fe9ba5bd640 1 --2- 172.21.15.40:0/1913360505 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55638adc2520 0x55638adbf880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:19:02.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.637+0000 7fe9b95bb640 1 --2- 172.21.15.40:0/1913360505 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55638adb9940 0x55638adbf270 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fe9a4097c60 tx=0x7fe9a409b160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de00dc7d1d1768fd server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:02.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.637+0000 7fe99d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-21T23:19:02.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.637+0000 7fe99d7fa640 1 -- 172.21.15.40:0/1913360505 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9a40b4060 con 0x55638adb9940 2024-04-21T23:19:02.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.637+0000 7fe99d7fa640 1 -- 172.21.15.40:0/1913360505 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7fe9a4005310 con 0x55638adb9940 2024-04-21T23:19:02.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.637+0000 7fe9bec0a740 1 -- 172.21.15.40:0/1913360505 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe9ac003b70 con 0x55638adb9940 2024-04-21T23:19:02.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.637+0000 7fe99d7fa640 1 -- 172.21.15.40:0/1913360505 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fe9a40b6e60 con 0x55638adb9940 2024-04-21T23:19:02.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.637+0000 7fe9bec0a740 10 client.15819 fetch_fsmap learned FSMap version 25 2024-04-21T23:19:02.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.637+0000 7fe9bec0a740 1 -- 172.21.15.40:0/1913360505 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55638aa313f0 con 0x55638adb9940 2024-04-21T23:19:02.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.638+0000 7fe99d7fa640 1 -- 172.21.15.40:0/1913360505 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9ac003b70 con 0x55638adb9940 2024-04-21T23:19:02.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.638+0000 7fe99d7fa640 1 -- 172.21.15.40:0/1913360505 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fe9a40a3fc0 con 0x55638adb9940 2024-04-21T23:19:02.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.638+0000 7fe9bec0a740 10 client.15819 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:19:02.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.638+0000 7fe9bec0a740 10 client.15819 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T23:19:02.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.638+0000 7fe9bec0a740 20 client.15819 populate_metadata read hostname 'smithi040' 2024-04-21T23:19:02.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.638+0000 7fe9bec0a740 1 --2- 172.21.15.40:0/1913360505 >> [v2:172.21.15.174:6838/2467509274,v1:172.21.15.174:6839/2467509274] conn(0x55638add2550 0x55638add4940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:02.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.638+0000 7fe9bec0a740 4 client.15819 mds_command: new command op to 24502 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:19:02.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.638+0000 7fe9bec0a740 1 -- 172.21.15.40:0/1913360505 --> [v2:172.21.15.174:6838/2467509274,v1:172.21.15.174:6839/2467509274] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55638ad2aa70 con 0x55638add2550 2024-04-21T23:19:02.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.638+0000 7fe9b9dbc640 1 --2- 172.21.15.40:0/1913360505 >> [v2:172.21.15.174:6838/2467509274,v1:172.21.15.174:6839/2467509274] conn(0x55638add2550 0x55638add4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:02.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.639+0000 7fe9b9dbc640 1 --2- 172.21.15.40:0/1913360505 >> [v2:172.21.15.174:6838/2467509274,v1:172.21.15.174:6839/2467509274] conn(0x55638add2550 0x55638add4940 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=bc0adbbfe99ba0d server_cookie=dd31487628570922 in_seq=0 out_seq=0 2024-04-21T23:19:02.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.639+0000 7fe99d7fa640 10 client.15819 ms_handle_connect on v2:172.21.15.174:6838/2467509274 2024-04-21T23:19:02.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.639+0000 7fe99d7fa640 1 -- 172.21.15.40:0/1913360505 <== mds.1 v2:172.21.15.174:6838/2467509274 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55638ad2aa70 con 0x55638add2550 2024-04-21T23:19:02.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.639+0000 7fe99d7fa640 10 client.15819 handle_command_reply: tid=0 2024-04-21T23:19:02.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.639+0000 7fe9bec0a740 1 client.15819 shutdown 2024-04-21T23:19:02.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.639+0000 7fe9bec0a740 1 -- 172.21.15.40:0/1913360505 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55638adb9940 msgr2=0x55638adbf270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:02.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.639+0000 7fe9bec0a740 1 --2- 172.21.15.40:0/1913360505 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55638adb9940 0x55638adbf270 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fe9a4097c60 tx=0x7fe9a409b160 comp rx=0 tx=0).stop 2024-04-21T23:19:02.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.640+0000 7fe9bec0a740 1 -- 172.21.15.40:0/1913360505 shutdown_connections 2024-04-21T23:19:02.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.640+0000 7fe9bec0a740 1 --2- 172.21.15.40:0/1913360505 >> [v2:172.21.15.174:6838/2467509274,v1:172.21.15.174:6839/2467509274] conn(0x55638add2550 0x55638add4940 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.640+0000 7fe9bec0a740 1 --2- 172.21.15.40:0/1913360505 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55638adc2520 0x55638adbf880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.640+0000 7fe9bec0a740 1 --2- 172.21.15.40:0/1913360505 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55638adb9940 0x55638adbf270 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.640+0000 7fe9bec0a740 1 --2- 172.21.15.40:0/1913360505 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55638adb9020 0x55638adc1ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.640+0000 7fe99d7fa640 0 client.15819 ms_handle_reset on v2:172.21.15.174:6838/2467509274 2024-04-21T23:19:02.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.640+0000 7fe9bec0a740 1 -- 172.21.15.40:0/1913360505 >> 172.21.15.40:0/1913360505 conn(0x55638adb3750 msgr2=0x55638adb54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:02.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.640+0000 7fe9bec0a740 1 -- 172.21.15.40:0/1913360505 shutdown_connections 2024-04-21T23:19:02.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.640+0000 7fe9bec0a740 1 -- 172.21.15.40:0/1913360505 wait complete. 2024-04-21T23:19:02.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.640+0000 7fe9bec0a740 20 client.15819 trim_cache size 0 max 16384 2024-04-21T23:19:02.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.653+0000 7fe9bec0a740 1 Processor -- start 2024-04-21T23:19:02.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.653+0000 7fe9bec0a740 1 -- start start 2024-04-21T23:19:02.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.653+0000 7fe9bec0a740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55638adb9020 0x55638adce1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:02.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.653+0000 7fe9bec0a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55638adb9940 0x55638adb2850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:02.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.653+0000 7fe9bec0a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55638adc2520 0x55638adc19a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:02.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.653+0000 7fe9bec0a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe9b41b37a0 con 0x55638adc2520 2024-04-21T23:19:02.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.653+0000 7fe9bec0a740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe9b41b1070 con 0x55638adb9020 2024-04-21T23:19:02.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.653+0000 7fe9bec0a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe9b41aeec0 con 0x55638adb9940 2024-04-21T23:19:02.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.654+0000 7fe9ba5bd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55638adc2520 0x55638adc19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:02.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.654+0000 7fe9ba5bd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55638adc2520 0x55638adc19a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47568/0 (socket says 172.21.15.40:47568) 2024-04-21T23:19:02.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.654+0000 7fe9ba5bd640 1 -- 172.21.15.40:0/925338747 learned_addr learned my addr 172.21.15.40:0/925338747 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:02.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.654+0000 7fe9b95bb640 1 --2- 172.21.15.40:0/925338747 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55638adb9940 0x55638adb2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:02.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.654+0000 7fe9b9dbc640 1 --2- 172.21.15.40:0/925338747 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55638adb9020 0x55638adce1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:02.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.654+0000 7fe9ba5bd640 1 -- 172.21.15.40:0/925338747 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55638adb9940 msgr2=0x55638adb2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:02.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.654+0000 7fe9ba5bd640 1 --2- 172.21.15.40:0/925338747 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55638adb9940 0x55638adb2850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.654+0000 7fe9ba5bd640 1 -- 172.21.15.40:0/925338747 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55638adb9020 msgr2=0x55638adce1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:02.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.654+0000 7fe9ba5bd640 1 --2- 172.21.15.40:0/925338747 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55638adb9020 0x55638adce1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.654+0000 7fe9ba5bd640 1 -- 172.21.15.40:0/925338747 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55638aafb7d0 con 0x55638adc2520 2024-04-21T23:19:02.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.654+0000 7fe9ba5bd640 1 --2- 172.21.15.40:0/925338747 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55638adc2520 0x55638adc19a0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac02c6d0 tx=0x7fe9ac0048c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c24d92755c618c9d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:02.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.654+0000 7fe99f7fe640 1 -- 172.21.15.40:0/925338747 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9ac039040 con 0x55638adc2520 2024-04-21T23:19:02.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.654+0000 7fe99f7fe640 1 -- 172.21.15.40:0/925338747 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9ac059070 con 0x55638adc2520 2024-04-21T23:19:02.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.654+0000 7fe99f7fe640 1 -- 172.21.15.40:0/925338747 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9ac050040 con 0x55638adc2520 2024-04-21T23:19:02.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.655+0000 7fe9bec0a740 1 -- 172.21.15.40:0/925338747 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55638adc2520 msgr2=0x55638adc19a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:02.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.655+0000 7fe9bec0a740 1 --2- 172.21.15.40:0/925338747 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55638adc2520 0x55638adc19a0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac02c6d0 tx=0x7fe9ac0048c0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.655+0000 7fe9bec0a740 1 -- 172.21.15.40:0/925338747 shutdown_connections 2024-04-21T23:19:02.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.655+0000 7fe9bec0a740 1 --2- 172.21.15.40:0/925338747 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55638adc2520 0x55638adc19a0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.655+0000 7fe9bec0a740 1 --2- 172.21.15.40:0/925338747 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55638adb9940 0x55638adb2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.655+0000 7fe9bec0a740 1 --2- 172.21.15.40:0/925338747 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55638adb9020 0x55638adce1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.655+0000 7fe9bec0a740 1 -- 172.21.15.40:0/925338747 >> 172.21.15.40:0/925338747 conn(0x55638adb3750 msgr2=0x55638adbe2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:02.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.655+0000 7fe9bec0a740 1 -- 172.21.15.40:0/925338747 shutdown_connections 2024-04-21T23:19:02.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.655+0000 7fe9bec0a740 1 -- 172.21.15.40:0/925338747 wait complete. 2024-04-21T23:19:02.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.655+0000 7fe9bec0a740 1 -- start start 2024-04-21T23:19:02.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.655+0000 7fe9bec0a740 1 Processor -- start 2024-04-21T23:19:02.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.656+0000 7fe9bec0a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55638adb9020 0x55638adca610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:02.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.656+0000 7fe9bec0a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55638adb9940 0x55638adcab50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:02.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.656+0000 7fe9bec0a740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55638adc2520 0x55638adc5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:02.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.656+0000 7fe9bec0a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe9b41af530 con 0x55638adb9940 2024-04-21T23:19:02.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.656+0000 7fe9bec0a740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe9b41b09a0 con 0x55638adc2520 2024-04-21T23:19:02.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.656+0000 7fe9bec0a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe9b41afc00 con 0x55638adb9020 2024-04-21T23:19:02.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.656+0000 7fe9bec0a740 10 client.0 _ll_register_callbacks cb 0x7fe9a40058c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:19:02.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.656+0000 7fe9b95bb640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55638adb9940 0x55638adcab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:02.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.656+0000 7fe9b95bb640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55638adb9940 0x55638adcab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47584/0 (socket says 172.21.15.40:47584) 2024-04-21T23:19:02.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.656+0000 7fe9b95bb640 1 -- 172.21.15.40:0/2815089113 learned_addr learned my addr 172.21.15.40:0/2815089113 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:02.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.656+0000 7fe9b9dbc640 1 --2- 172.21.15.40:0/2815089113 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55638adb9020 0x55638adca610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:02.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.656+0000 7fe9ba5bd640 1 --2- 172.21.15.40:0/2815089113 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55638adc2520 0x55638adc5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:02.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.656+0000 7fe9b95bb640 1 -- 172.21.15.40:0/2815089113 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55638adb9020 msgr2=0x55638adca610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:02.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.656+0000 7fe9b95bb640 1 --2- 172.21.15.40:0/2815089113 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55638adb9020 0x55638adca610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.656+0000 7fe9b95bb640 1 -- 172.21.15.40:0/2815089113 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55638adc2520 msgr2=0x55638adc5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:02.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.656+0000 7fe9b95bb640 1 --2- 172.21.15.40:0/2815089113 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55638adc2520 0x55638adc5ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.656+0000 7fe9b95bb640 1 -- 172.21.15.40:0/2815089113 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55638adc62b0 con 0x55638adb9940 2024-04-21T23:19:02.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.656+0000 7fe9ba5bd640 1 --2- 172.21.15.40:0/2815089113 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55638adc2520 0x55638adc5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:19:02.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.656+0000 7fe9b9dbc640 1 --2- 172.21.15.40:0/2815089113 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55638adb9020 0x55638adca610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:19:02.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.656+0000 7fe9b95bb640 1 --2- 172.21.15.40:0/2815089113 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55638adb9940 0x55638adcab50 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fe9a4007590 tx=0x7fe9a4099d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=164aeeb449cfb632 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:02.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.656+0000 7fe99d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-21T23:19:02.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.657+0000 7fe99d7fa640 1 -- 172.21.15.40:0/2815089113 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9a401d040 con 0x55638adb9940 2024-04-21T23:19:02.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.657+0000 7fe99d7fa640 1 -- 172.21.15.40:0/2815089113 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7fe9a40a3f30 con 0x55638adb9940 2024-04-21T23:19:02.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.657+0000 7fe9bec0a740 1 -- 172.21.15.40:0/2815089113 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe9ac059070 con 0x55638adb9940 2024-04-21T23:19:02.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.657+0000 7fe99d7fa640 1 -- 172.21.15.40:0/2815089113 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fe9a409b5d0 con 0x55638adb9940 2024-04-21T23:19:02.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.657+0000 7fe9bec0a740 10 client.15825 fetch_fsmap learned FSMap version 25 2024-04-21T23:19:02.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.657+0000 7fe9bec0a740 1 -- 172.21.15.40:0/2815089113 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55638ae38650 con 0x55638adb9940 2024-04-21T23:19:02.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.657+0000 7fe99d7fa640 1 -- 172.21.15.40:0/2815089113 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9ac059070 con 0x55638adb9940 2024-04-21T23:19:02.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.657+0000 7fe99d7fa640 1 -- 172.21.15.40:0/2815089113 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fe9a40ccc90 con 0x55638adb9940 2024-04-21T23:19:02.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.657+0000 7fe9bec0a740 10 client.15825 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:19:02.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.657+0000 7fe9bec0a740 10 client.15825 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T23:19:02.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.657+0000 7fe9bec0a740 20 client.15825 populate_metadata read hostname 'smithi040' 2024-04-21T23:19:02.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.657+0000 7fe9bec0a740 1 --2- 172.21.15.40:0/2815089113 >> [v2:172.21.15.174:6838/2467509274,v1:172.21.15.174:6839/2467509274] conn(0x55638ae38dd0 0x55638ae3b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:02.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.657+0000 7fe9bec0a740 4 client.15825 mds_command: new command op to 24502 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T23:19:02.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.657+0000 7fe9bec0a740 1 -- 172.21.15.40:0/2815089113 --> [v2:172.21.15.174:6838/2467509274,v1:172.21.15.174:6839/2467509274] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55638ad24dc0 con 0x55638ae38dd0 2024-04-21T23:19:02.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.658+0000 7fe9b9dbc640 1 --2- 172.21.15.40:0/2815089113 >> [v2:172.21.15.174:6838/2467509274,v1:172.21.15.174:6839/2467509274] conn(0x55638ae38dd0 0x55638ae3b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:02.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.658+0000 7fe99d7fa640 10 client.15825 ms_handle_connect on v2:172.21.15.174:6838/2467509274 2024-04-21T23:19:02.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.658+0000 7fe9b9dbc640 1 --2- 172.21.15.40:0/2815089113 >> [v2:172.21.15.174:6838/2467509274,v1:172.21.15.174:6839/2467509274] conn(0x55638ae38dd0 0x55638ae3b1f0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d616620b56a7645 server_cookie=62e4483166ef9ed8 in_seq=0 out_seq=0 2024-04-21T23:19:02.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.658+0000 7fe99d7fa640 1 -- 172.21.15.40:0/2815089113 <== mds.1 v2:172.21.15.174:6838/2467509274 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55638ad24dc0 con 0x55638ae38dd0 2024-04-21T23:19:02.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.658+0000 7fe99d7fa640 10 client.15825 handle_command_reply: tid=0 2024-04-21T23:19:02.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.658+0000 7fe9bec0a740 1 client.15825 shutdown 2024-04-21T23:19:02.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.658+0000 7fe9bec0a740 1 -- 172.21.15.40:0/2815089113 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55638adb9940 msgr2=0x55638adcab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:02.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.658+0000 7fe9bec0a740 1 --2- 172.21.15.40:0/2815089113 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55638adb9940 0x55638adcab50 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fe9a4007590 tx=0x7fe9a4099d10 comp rx=0 tx=0).stop 2024-04-21T23:19:02.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.658+0000 7fe9bec0a740 1 -- 172.21.15.40:0/2815089113 shutdown_connections 2024-04-21T23:19:02.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.658+0000 7fe9bec0a740 1 --2- 172.21.15.40:0/2815089113 >> [v2:172.21.15.174:6838/2467509274,v1:172.21.15.174:6839/2467509274] conn(0x55638ae38dd0 0x55638ae3b1f0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.658+0000 7fe9bec0a740 1 --2- 172.21.15.40:0/2815089113 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55638adc2520 0x55638adc5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.658+0000 7fe9bec0a740 1 --2- 172.21.15.40:0/2815089113 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55638adb9940 0x55638adcab50 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.658+0000 7fe9bec0a740 1 --2- 172.21.15.40:0/2815089113 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55638adb9020 0x55638adca610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.658+0000 7fe99d7fa640 0 client.15825 ms_handle_reset on v2:172.21.15.174:6838/2467509274 2024-04-21T23:19:02.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.658+0000 7fe9bec0a740 1 -- 172.21.15.40:0/2815089113 >> 172.21.15.40:0/2815089113 conn(0x55638adb3750 msgr2=0x55638adb2850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:02.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.658+0000 7fe9bec0a740 1 -- 172.21.15.40:0/2815089113 shutdown_connections 2024-04-21T23:19:02.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.658+0000 7fe9bec0a740 1 -- 172.21.15.40:0/2815089113 wait complete. 2024-04-21T23:19:02.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.659+0000 7fe9bec0a740 20 client.15825 trim_cache size 0 max 16384 2024-04-21T23:19:02.666 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-04-21T23:19:02.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.659+0000 7fe9bc020640 1 -- 172.21.15.40:0/3642121833 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fe9a00a2eb0 msgr2=0x7fe9a00a5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:02.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.659+0000 7fe9bc020640 1 --2- 172.21.15.40:0/3642121833 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fe9a00a2eb0 0x7fe9a00a5370 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac047000 tx=0x7fe9ac048000 comp rx=0 tx=0).stop 2024-04-21T23:19:02.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.659+0000 7fe9bc020640 1 -- 172.21.15.40:0/3642121833 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9b4102150 msgr2=0x7fe9b419a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:02.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.659+0000 7fe9bc020640 1 --2- 172.21.15.40:0/3642121833 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9b4102150 0x7fe9b419a810 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fe9a400c970 tx=0x7fe9a400ce30 comp rx=0 tx=0).stop 2024-04-21T23:19:02.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.659+0000 7fe9bc020640 1 -- 172.21.15.40:0/3642121833 shutdown_connections 2024-04-21T23:19:02.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.659+0000 7fe9bc020640 1 --2- 172.21.15.40:0/3642121833 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fe9a00a2eb0 0x7fe9a00a5370 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.659+0000 7fe9bc020640 1 --2- 172.21.15.40:0/3642121833 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe9b41077a0 0x7fe9b419eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.659+0000 7fe9bc020640 1 --2- 172.21.15.40:0/3642121833 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9b4102150 0x7fe9b419a810 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.659+0000 7fe9bc020640 1 --2- 172.21.15.40:0/3642121833 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe9b4101790 0x7fe9b419a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.659+0000 7fe9bc020640 1 -- 172.21.15.40:0/3642121833 >> 172.21.15.40:0/3642121833 conn(0x7fe9b40fd2a0 msgr2=0x7fe9b410cfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:02.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.659+0000 7fe9bc020640 1 -- 172.21.15.40:0/3642121833 shutdown_connections 2024-04-21T23:19:02.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.659+0000 7fe9bc020640 1 -- 172.21.15.40:0/3642121833 wait complete. 2024-04-21T23:19:02.672 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T23:19:02.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.780+0000 7ff9e4688640 1 -- 172.21.15.40:0/2070536060 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9dc10bd00 msgr2=0x7ff9dc10e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:02.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.780+0000 7ff9e4688640 1 --2- 172.21.15.40:0/2070536060 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9dc10bd00 0x7ff9dc10e1a0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7ff9d4009ca0 tx=0x7ff9d402c230 comp rx=0 tx=0).stop 2024-04-21T23:19:02.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.780+0000 7ff9e4688640 1 -- 172.21.15.40:0/2070536060 shutdown_connections 2024-04-21T23:19:02.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.780+0000 7ff9e4688640 1 --2- 172.21.15.40:0/2070536060 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9dc10bd00 0x7ff9dc10e1a0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.780+0000 7ff9e4688640 1 --2- 172.21.15.40:0/2070536060 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff9dc1032e0 0x7ff9dc10b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.781+0000 7ff9e4688640 1 --2- 172.21.15.40:0/2070536060 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff9dc1029c0 0x7ff9dc102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.781+0000 7ff9e4688640 1 -- 172.21.15.40:0/2070536060 >> 172.21.15.40:0/2070536060 conn(0x7ff9dc077740 msgr2=0x7ff9dc077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:02.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.781+0000 7ff9e4688640 1 -- 172.21.15.40:0/2070536060 shutdown_connections 2024-04-21T23:19:02.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.781+0000 7ff9e4688640 1 -- 172.21.15.40:0/2070536060 wait complete. 2024-04-21T23:19:02.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.781+0000 7ff9e4688640 1 Processor -- start 2024-04-21T23:19:02.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.781+0000 7ff9e4688640 1 -- start start 2024-04-21T23:19:02.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.781+0000 7ff9e4688640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff9dc1029c0 0x7ff9dc198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:02.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.781+0000 7ff9e4688640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff9dc1032e0 0x7ff9dc198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:02.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.781+0000 7ff9e4688640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9dc10bd00 0x7ff9dc19c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:02.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.781+0000 7ff9e4688640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff9dc111b00 con 0x7ff9dc10bd00 2024-04-21T23:19:02.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.781+0000 7ff9e4688640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff9dc111980 con 0x7ff9dc1029c0 2024-04-21T23:19:02.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.781+0000 7ff9e4688640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff9dc111c80 con 0x7ff9dc1032e0 2024-04-21T23:19:02.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.782+0000 7ff9e2424640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff9dc1029c0 0x7ff9dc198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:02.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.782+0000 7ff9e2424640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff9dc1029c0 0x7ff9dc198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.40:45956/0 (socket says 172.21.15.40:45956) 2024-04-21T23:19:02.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.782+0000 7ff9e2c25640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9dc10bd00 0x7ff9dc19c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:02.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.782+0000 7ff9e2424640 1 -- 172.21.15.40:0/3099887061 learned_addr learned my addr 172.21.15.40:0/3099887061 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:02.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.782+0000 7ff9e1c23640 1 --2- 172.21.15.40:0/3099887061 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff9dc1032e0 0x7ff9dc198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:02.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.782+0000 7ff9e2424640 1 -- 172.21.15.40:0/3099887061 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff9dc1032e0 msgr2=0x7ff9dc198540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:02.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.782+0000 7ff9e2424640 1 --2- 172.21.15.40:0/3099887061 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff9dc1032e0 0x7ff9dc198540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.782+0000 7ff9e2424640 1 -- 172.21.15.40:0/3099887061 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9dc10bd00 msgr2=0x7ff9dc19c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:02.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.782+0000 7ff9e2424640 1 --2- 172.21.15.40:0/3099887061 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9dc10bd00 0x7ff9dc19c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.782+0000 7ff9e2424640 1 -- 172.21.15.40:0/3099887061 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9dc19cff0 con 0x7ff9dc1029c0 2024-04-21T23:19:02.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.782+0000 7ff9e2c25640 1 --2- 172.21.15.40:0/3099887061 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9dc10bd00 0x7ff9dc19c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:19:02.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.782+0000 7ff9e2424640 1 --2- 172.21.15.40:0/3099887061 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff9dc1029c0 0x7ff9dc198000 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff9d800e9f0 tx=0x7ff9d800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:02.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.782+0000 7ff9d37fe640 1 -- 172.21.15.40:0/3099887061 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9d800cd80 con 0x7ff9dc1029c0 2024-04-21T23:19:02.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.782+0000 7ff9e4688640 1 -- 172.21.15.40:0/3099887061 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9dc19d2e0 con 0x7ff9dc1029c0 2024-04-21T23:19:02.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.782+0000 7ff9d37fe640 1 -- 172.21.15.40:0/3099887061 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff9d8010910 con 0x7ff9dc1029c0 2024-04-21T23:19:02.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.782+0000 7ff9d37fe640 1 -- 172.21.15.40:0/3099887061 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9d80188f0 con 0x7ff9dc1029c0 2024-04-21T23:19:02.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.782+0000 7ff9e4688640 1 -- 172.21.15.40:0/3099887061 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9dc1a4b70 con 0x7ff9dc1029c0 2024-04-21T23:19:02.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.785+0000 7ff9e7277740 1 Processor -- start 2024-04-21T23:19:02.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.785+0000 7ff9e7277740 1 -- start start 2024-04-21T23:19:02.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.785+0000 7ff9d37fe640 1 -- 172.21.15.40:0/3099887061 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) ==== 138117+0+0 (secure 0 0 0) 0x7ff9d8010ac0 con 0x7ff9dc1029c0 2024-04-21T23:19:02.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.785+0000 7ff9e7277740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5575c7349020 0x5575c7349400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:02.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.785+0000 7ff9e7277740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575c7349940 0x5575c7351fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:02.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.785+0000 7ff9e7277740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5575c7352520 0x5575c7354910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:02.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.785+0000 7ff9e7277740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5575c72ad530 con 0x5575c7349940 2024-04-21T23:19:02.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.785+0000 7ff9e7277740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5575c7255830 con 0x5575c7349020 2024-04-21T23:19:02.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.785+0000 7ff9e7277740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5575c72a94f0 con 0x5575c7352520 2024-04-21T23:19:02.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.786+0000 7ff9e2424640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575c7349940 0x5575c7351fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:02.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.786+0000 7ff9e2424640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575c7349940 0x5575c7351fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47612/0 (socket says 172.21.15.40:47612) 2024-04-21T23:19:02.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.786+0000 7ff9e2424640 1 -- 172.21.15.40:0/2845248580 learned_addr learned my addr 172.21.15.40:0/2845248580 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:02.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.786+0000 7ff9e2c25640 1 --2- 172.21.15.40:0/2845248580 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5575c7349020 0x5575c7349400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:02.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.786+0000 7ff9e1c23640 1 --2- 172.21.15.40:0/2845248580 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5575c7352520 0x5575c7354910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:02.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.786+0000 7ff9e2424640 1 -- 172.21.15.40:0/2845248580 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5575c7352520 msgr2=0x5575c7354910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:02.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.786+0000 7ff9e2424640 1 --2- 172.21.15.40:0/2845248580 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5575c7352520 0x5575c7354910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.786+0000 7ff9e2424640 1 -- 172.21.15.40:0/2845248580 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5575c7349020 msgr2=0x5575c7349400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:02.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.786+0000 7ff9e2424640 1 --2- 172.21.15.40:0/2845248580 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5575c7349020 0x5575c7349400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.786+0000 7ff9e2424640 1 -- 172.21.15.40:0/2845248580 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5575c72792c0 con 0x5575c7349940 2024-04-21T23:19:02.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.786+0000 7ff9e1c23640 1 --2- 172.21.15.40:0/2845248580 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5575c7352520 0x5575c7354910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:19:02.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.786+0000 7ff9d37fe640 1 --2- 172.21.15.40:0/3099887061 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7ff9c80a2eb0 0x7ff9c80a5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:02.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.786+0000 7ff9e2424640 1 --2- 172.21.15.40:0/2845248580 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575c7349940 0x5575c7351fe0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7ff9d80ccfa0 tx=0x7ff9d80d8570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27fcfee58591cb28 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:02.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.786+0000 7ff9e2c25640 1 --2- 172.21.15.40:0/3099887061 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7ff9c80a2eb0 0x7ff9c80a5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:02.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.786+0000 7ff9d37fe640 1 -- 172.21.15.40:0/3099887061 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7ff9d80cfea0 con 0x7ff9dc1029c0 2024-04-21T23:19:02.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.787+0000 7ff9d0ff9640 1 -- 172.21.15.40:0/2845248580 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9d8098120 con 0x5575c7349940 2024-04-21T23:19:02.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.787+0000 7ff9d0ff9640 1 -- 172.21.15.40:0/2845248580 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff9d809be80 con 0x5575c7349940 2024-04-21T23:19:02.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.787+0000 7ff9d0ff9640 1 -- 172.21.15.40:0/2845248580 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9d809b7e0 con 0x5575c7349940 2024-04-21T23:19:02.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.787+0000 7ff9e2c25640 1 --2- 172.21.15.40:0/3099887061 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7ff9c80a2eb0 0x7ff9c80a5370 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff9d4004af0 tx=0x7ff9d4005c50 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:02.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.787+0000 7ff9e7277740 1 -- 172.21.15.40:0/2845248580 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575c7349940 msgr2=0x5575c7351fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:02.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.787+0000 7ff9e7277740 1 --2- 172.21.15.40:0/2845248580 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575c7349940 0x5575c7351fe0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7ff9d80ccfa0 tx=0x7ff9d80d8570 comp rx=0 tx=0).stop 2024-04-21T23:19:02.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.787+0000 7ff9e7277740 1 -- 172.21.15.40:0/2845248580 shutdown_connections 2024-04-21T23:19:02.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.787+0000 7ff9e7277740 1 --2- 172.21.15.40:0/2845248580 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5575c7352520 0x5575c7354910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.787+0000 7ff9e7277740 1 --2- 172.21.15.40:0/2845248580 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575c7349940 0x5575c7351fe0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.787+0000 7ff9e7277740 1 --2- 172.21.15.40:0/2845248580 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5575c7349020 0x5575c7349400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.787+0000 7ff9e7277740 1 -- 172.21.15.40:0/2845248580 >> 172.21.15.40:0/2845248580 conn(0x5575c7343750 msgr2=0x5575c7343b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:02.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.788+0000 7ff9e7277740 1 -- 172.21.15.40:0/2845248580 shutdown_connections 2024-04-21T23:19:02.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.788+0000 7ff9e7277740 1 -- 172.21.15.40:0/2845248580 wait complete. 2024-04-21T23:19:02.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.788+0000 7ff9e7277740 1 -- start start 2024-04-21T23:19:02.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.788+0000 7ff9e7277740 1 Processor -- start 2024-04-21T23:19:02.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.788+0000 7ff9e7277740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575c7349020 0x5575c7351e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:02.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.788+0000 7ff9e7277740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5575c7349940 0x5575c734f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:02.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.788+0000 7ff9e7277740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5575c7352520 0x5575c734f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:02.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.788+0000 7ff9e7277740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5575c7355310 con 0x5575c7349020 2024-04-21T23:19:02.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.788+0000 7ff9e7277740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5575c7268f10 con 0x5575c7352520 2024-04-21T23:19:02.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.788+0000 7ff9e7277740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5575c7355490 con 0x5575c7349940 2024-04-21T23:19:02.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.788+0000 7ff9e7277740 10 client.0 _ll_register_callbacks cb 0x5575c6fc0be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:19:02.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.788+0000 7ff9e2424640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575c7349020 0x5575c7351e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:02.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.788+0000 7ff9e2424640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575c7349020 0x5575c7351e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47616/0 (socket says 172.21.15.40:47616) 2024-04-21T23:19:02.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.788+0000 7ff9e2424640 1 -- 172.21.15.40:0/1608561475 learned_addr learned my addr 172.21.15.40:0/1608561475 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:02.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.788+0000 7ff9e2c25640 1 --2- 172.21.15.40:0/1608561475 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5575c7352520 0x5575c734f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:02.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.788+0000 7ff9e1c23640 1 --2- 172.21.15.40:0/1608561475 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5575c7349940 0x5575c734f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:02.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.789+0000 7ff9e2424640 1 -- 172.21.15.40:0/1608561475 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5575c7349940 msgr2=0x5575c734f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:02.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.789+0000 7ff9e2424640 1 --2- 172.21.15.40:0/1608561475 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5575c7349940 0x5575c734f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.789+0000 7ff9e2424640 1 -- 172.21.15.40:0/1608561475 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5575c7352520 msgr2=0x5575c734f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:02.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.789+0000 7ff9e2424640 1 --2- 172.21.15.40:0/1608561475 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5575c7352520 0x5575c734f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.789+0000 7ff9e2424640 1 -- 172.21.15.40:0/1608561475 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5575c723ac80 con 0x5575c7349020 2024-04-21T23:19:02.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.789+0000 7ff9e1c23640 1 --2- 172.21.15.40:0/1608561475 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5575c7349940 0x5575c734f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:19:02.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.789+0000 7ff9e2424640 1 --2- 172.21.15.40:0/1608561475 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575c7349020 0x5575c7351e40 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7ff9d8009230 tx=0x7ff9d8009700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d19eb7c6cfef5c08 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:02.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.789+0000 7ff9c5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-21T23:19:02.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.789+0000 7ff9c5ffb640 1 -- 172.21.15.40:0/1608561475 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9d809b9c0 con 0x5575c7349020 2024-04-21T23:19:02.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.789+0000 7ff9e7277740 1 -- 172.21.15.40:0/1608561475 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff9d809be80 con 0x5575c7349020 2024-04-21T23:19:02.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.789+0000 7ff9c5ffb640 1 -- 172.21.15.40:0/1608561475 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7ff9d809c0e0 con 0x5575c7349020 2024-04-21T23:19:02.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.789+0000 7ff9c5ffb640 1 -- 172.21.15.40:0/1608561475 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7ff9d809bc50 con 0x5575c7349020 2024-04-21T23:19:02.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.789+0000 7ff9e7277740 10 client.15837 fetch_fsmap learned FSMap version 25 2024-04-21T23:19:02.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.789+0000 7ff9e7277740 1 -- 172.21.15.40:0/1608561475 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5575c6fc13f0 con 0x5575c7349020 2024-04-21T23:19:02.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.789+0000 7ff9c5ffb640 1 -- 172.21.15.40:0/1608561475 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff9d809be80 con 0x5575c7349020 2024-04-21T23:19:02.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.790+0000 7ff9c5ffb640 1 -- 172.21.15.40:0/1608561475 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7ff9d80b1070 con 0x5575c7349020 2024-04-21T23:19:02.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.790+0000 7ff9e7277740 10 client.15837 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:19:02.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.790+0000 7ff9e7277740 10 client.15837 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:19:02.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.790+0000 7ff9e7277740 20 client.15837 populate_metadata read hostname 'smithi040' 2024-04-21T23:19:02.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.790+0000 7ff9e7277740 1 --2- 172.21.15.40:0/1608561475 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x5575c73623f0 0x5575c73647e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:02.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.790+0000 7ff9e7277740 4 client.15837 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:19:02.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.790+0000 7ff9e7277740 1 -- 172.21.15.40:0/1608561475 --> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5575c72baa70 con 0x5575c73623f0 2024-04-21T23:19:02.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.790+0000 7ff9e1c23640 1 --2- 172.21.15.40:0/1608561475 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x5575c73623f0 0x5575c73647e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:02.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.790+0000 7ff9e1c23640 1 --2- 172.21.15.40:0/1608561475 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x5575c73623f0 0x5575c73647e0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=92ea86ee7ad96eaf server_cookie=d2718c6a6fb07ea3 in_seq=0 out_seq=0 2024-04-21T23:19:02.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.790+0000 7ff9c5ffb640 10 client.15837 ms_handle_connect on v2:172.21.15.111:6838/3712235548 2024-04-21T23:19:02.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.791+0000 7ff9c5ffb640 1 -- 172.21.15.40:0/1608561475 <== mds.0 v2:172.21.15.111:6838/3712235548 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5575c72baa70 con 0x5575c73623f0 2024-04-21T23:19:02.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.791+0000 7ff9c5ffb640 10 client.15837 handle_command_reply: tid=0 2024-04-21T23:19:02.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.791+0000 7ff9e7277740 1 client.15837 shutdown 2024-04-21T23:19:02.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.791+0000 7ff9e7277740 1 -- 172.21.15.40:0/1608561475 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575c7349020 msgr2=0x5575c7351e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:02.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.791+0000 7ff9e7277740 1 --2- 172.21.15.40:0/1608561475 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575c7349020 0x5575c7351e40 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7ff9d8009230 tx=0x7ff9d8009700 comp rx=0 tx=0).stop 2024-04-21T23:19:02.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.791+0000 7ff9e7277740 1 -- 172.21.15.40:0/1608561475 shutdown_connections 2024-04-21T23:19:02.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.791+0000 7ff9e7277740 1 --2- 172.21.15.40:0/1608561475 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x5575c73623f0 0x5575c73647e0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.791+0000 7ff9e7277740 1 --2- 172.21.15.40:0/1608561475 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5575c7352520 0x5575c734f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.791+0000 7ff9e7277740 1 --2- 172.21.15.40:0/1608561475 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5575c7349940 0x5575c734f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.791+0000 7ff9e7277740 1 --2- 172.21.15.40:0/1608561475 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575c7349020 0x5575c7351e40 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.791+0000 7ff9c5ffb640 0 client.15837 ms_handle_reset on v2:172.21.15.111:6838/3712235548 2024-04-21T23:19:02.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.791+0000 7ff9e7277740 1 -- 172.21.15.40:0/1608561475 >> 172.21.15.40:0/1608561475 conn(0x5575c7343750 msgr2=0x5575c73454a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:02.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.791+0000 7ff9e7277740 1 -- 172.21.15.40:0/1608561475 shutdown_connections 2024-04-21T23:19:02.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.791+0000 7ff9e7277740 1 -- 172.21.15.40:0/1608561475 wait complete. 2024-04-21T23:19:02.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.792+0000 7ff9e7277740 20 client.15837 trim_cache size 0 max 16384 2024-04-21T23:19:02.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.807+0000 7ff9e7277740 1 Processor -- start 2024-04-21T23:19:02.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.807+0000 7ff9e7277740 1 -- start start 2024-04-21T23:19:02.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.807+0000 7ff9e7277740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5575c7349020 0x5575c735e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:02.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.807+0000 7ff9e7277740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575c7349940 0x5575c7342a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:02.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.807+0000 7ff9e7277740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5575c7352520 0x5575c7351970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:02.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.807+0000 7ff9e7277740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff9dc1b1560 con 0x5575c7349940 2024-04-21T23:19:02.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.807+0000 7ff9e7277740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff9dc1aee30 con 0x5575c7352520 2024-04-21T23:19:02.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.807+0000 7ff9e7277740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff9dc1acc80 con 0x5575c7349020 2024-04-21T23:19:02.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.807+0000 7ff9e1c23640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575c7349940 0x5575c7342a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:02.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.807+0000 7ff9e1c23640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575c7349940 0x5575c7342a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47618/0 (socket says 172.21.15.40:47618) 2024-04-21T23:19:02.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.807+0000 7ff9e1c23640 1 -- 172.21.15.40:0/689507439 learned_addr learned my addr 172.21.15.40:0/689507439 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:02.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.807+0000 7ff9e2424640 1 --2- 172.21.15.40:0/689507439 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5575c7349020 0x5575c735e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:02.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.807+0000 7ff9e2c25640 1 --2- 172.21.15.40:0/689507439 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5575c7352520 0x5575c7351970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:02.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.807+0000 7ff9e1c23640 1 -- 172.21.15.40:0/689507439 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5575c7349020 msgr2=0x5575c735e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:02.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.807+0000 7ff9e1c23640 1 --2- 172.21.15.40:0/689507439 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5575c7349020 0x5575c735e230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.807+0000 7ff9e1c23640 1 -- 172.21.15.40:0/689507439 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5575c7352520 msgr2=0x5575c7351970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:02.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.807+0000 7ff9e1c23640 1 --2- 172.21.15.40:0/689507439 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5575c7352520 0x5575c7351970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.807+0000 7ff9e1c23640 1 -- 172.21.15.40:0/689507439 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5575c708b7d0 con 0x5575c7349940 2024-04-21T23:19:02.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.808+0000 7ff9e1c23640 1 --2- 172.21.15.40:0/689507439 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575c7349940 0x5575c7342a90 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x5575c7364d50 tx=0x7ff9cc005b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fffaaca8c4d43c8f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:02.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.808+0000 7ff9c7fff640 1 -- 172.21.15.40:0/689507439 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9cc0142d0 con 0x5575c7349940 2024-04-21T23:19:02.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.808+0000 7ff9c7fff640 1 -- 172.21.15.40:0/689507439 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff9cc0035f0 con 0x5575c7349940 2024-04-21T23:19:02.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.808+0000 7ff9c7fff640 1 -- 172.21.15.40:0/689507439 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9cc01e280 con 0x5575c7349940 2024-04-21T23:19:02.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.809+0000 7ff9e7277740 1 -- 172.21.15.40:0/689507439 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575c7349940 msgr2=0x5575c7342a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:02.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.809+0000 7ff9e7277740 1 --2- 172.21.15.40:0/689507439 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575c7349940 0x5575c7342a90 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x5575c7364d50 tx=0x7ff9cc005b20 comp rx=0 tx=0).stop 2024-04-21T23:19:02.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.809+0000 7ff9e7277740 1 -- 172.21.15.40:0/689507439 shutdown_connections 2024-04-21T23:19:02.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.809+0000 7ff9e7277740 1 --2- 172.21.15.40:0/689507439 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5575c7352520 0x5575c7351970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.809+0000 7ff9e7277740 1 --2- 172.21.15.40:0/689507439 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575c7349940 0x5575c7342a90 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.809+0000 7ff9e7277740 1 --2- 172.21.15.40:0/689507439 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5575c7349020 0x5575c735e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.809+0000 7ff9e7277740 1 -- 172.21.15.40:0/689507439 >> 172.21.15.40:0/689507439 conn(0x5575c7343750 msgr2=0x5575c7345210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:02.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.809+0000 7ff9e7277740 1 -- 172.21.15.40:0/689507439 shutdown_connections 2024-04-21T23:19:02.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.809+0000 7ff9e7277740 1 -- 172.21.15.40:0/689507439 wait complete. 2024-04-21T23:19:02.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.809+0000 7ff9e7277740 1 -- start start 2024-04-21T23:19:02.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.809+0000 7ff9e7277740 1 Processor -- start 2024-04-21T23:19:02.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.810+0000 7ff9e7277740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5575c7349020 0x5575c735aa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:02.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.810+0000 7ff9e7277740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575c7349940 0x5575c734bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:02.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.810+0000 7ff9e7277740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5575c7352520 0x5575c734c070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:02.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.810+0000 7ff9e7277740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff9dc1ad2f0 con 0x5575c7349940 2024-04-21T23:19:02.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.810+0000 7ff9e7277740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff9dc1ae760 con 0x5575c7349020 2024-04-21T23:19:02.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.810+0000 7ff9e7277740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff9dc1ad9c0 con 0x5575c7352520 2024-04-21T23:19:02.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.810+0000 7ff9e7277740 10 client.0 _ll_register_callbacks cb 0x7ff9d8005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:19:02.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.810+0000 7ff9e1c23640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575c7349940 0x5575c734bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:02.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.810+0000 7ff9e1c23640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575c7349940 0x5575c734bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47622/0 (socket says 172.21.15.40:47622) 2024-04-21T23:19:02.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.810+0000 7ff9e1c23640 1 -- 172.21.15.40:0/2293158820 learned_addr learned my addr 172.21.15.40:0/2293158820 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:02.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.810+0000 7ff9e2424640 1 --2- 172.21.15.40:0/2293158820 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5575c7349020 0x5575c735aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:02.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.810+0000 7ff9e2c25640 1 --2- 172.21.15.40:0/2293158820 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5575c7352520 0x5575c734c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:02.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.810+0000 7ff9e1c23640 1 -- 172.21.15.40:0/2293158820 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5575c7352520 msgr2=0x5575c734c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:02.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.810+0000 7ff9e1c23640 1 --2- 172.21.15.40:0/2293158820 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5575c7352520 0x5575c734c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.810+0000 7ff9e1c23640 1 -- 172.21.15.40:0/2293158820 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5575c7349020 msgr2=0x5575c735aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:02.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.810+0000 7ff9e1c23640 1 --2- 172.21.15.40:0/2293158820 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5575c7349020 0x5575c735aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.810+0000 7ff9e1c23640 1 -- 172.21.15.40:0/2293158820 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5575c734c820 con 0x5575c7349940 2024-04-21T23:19:02.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.810+0000 7ff9e1c23640 1 --2- 172.21.15.40:0/2293158820 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575c7349940 0x5575c734bb30 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc021040 tx=0x7ff9cc0140d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d070c5cdf5f8616 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:02.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.810+0000 7ff9c5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-21T23:19:02.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.810+0000 7ff9c5ffb640 1 -- 172.21.15.40:0/2293158820 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9cc02c070 con 0x5575c7349940 2024-04-21T23:19:02.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.810+0000 7ff9c5ffb640 1 -- 172.21.15.40:0/2293158820 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7ff9cc0037a0 con 0x5575c7349940 2024-04-21T23:19:02.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.810+0000 7ff9e7277740 1 -- 172.21.15.40:0/2293158820 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff9cc0035f0 con 0x5575c7349940 2024-04-21T23:19:02.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.811+0000 7ff9c5ffb640 1 -- 172.21.15.40:0/2293158820 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7ff9cc0039a0 con 0x5575c7349940 2024-04-21T23:19:02.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.811+0000 7ff9e7277740 10 client.15843 fetch_fsmap learned FSMap version 25 2024-04-21T23:19:02.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.811+0000 7ff9e7277740 1 -- 172.21.15.40:0/2293158820 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5575c73c85a0 con 0x5575c7349940 2024-04-21T23:19:02.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.811+0000 7ff9c5ffb640 1 -- 172.21.15.40:0/2293158820 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff9cc0035f0 con 0x5575c7349940 2024-04-21T23:19:02.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.811+0000 7ff9c5ffb640 1 -- 172.21.15.40:0/2293158820 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7ff9cc012c50 con 0x5575c7349940 2024-04-21T23:19:02.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.811+0000 7ff9e7277740 10 client.15843 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:19:02.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.811+0000 7ff9e7277740 10 client.15843 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:19:02.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.811+0000 7ff9e7277740 20 client.15843 populate_metadata read hostname 'smithi040' 2024-04-21T23:19:02.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.811+0000 7ff9e7277740 1 --2- 172.21.15.40:0/2293158820 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x5575c73c8cf0 0x5575c73cb110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:02.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.811+0000 7ff9e7277740 4 client.15843 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T23:19:02.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.811+0000 7ff9e7277740 1 -- 172.21.15.40:0/2293158820 --> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5575c72b4dc0 con 0x5575c73c8cf0 2024-04-21T23:19:02.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.811+0000 7ff9e2424640 1 --2- 172.21.15.40:0/2293158820 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x5575c73c8cf0 0x5575c73cb110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:02.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.812+0000 7ff9e2424640 1 --2- 172.21.15.40:0/2293158820 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x5575c73c8cf0 0x5575c73cb110 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4fba42355bb9dc58 server_cookie=7f52985891d75f93 in_seq=0 out_seq=0 2024-04-21T23:19:02.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.812+0000 7ff9c5ffb640 10 client.15843 ms_handle_connect on v2:172.21.15.111:6838/3712235548 2024-04-21T23:19:02.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.812+0000 7ff9c5ffb640 1 -- 172.21.15.40:0/2293158820 <== mds.0 v2:172.21.15.111:6838/3712235548 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5575c72b4dc0 con 0x5575c73c8cf0 2024-04-21T23:19:02.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.812+0000 7ff9c5ffb640 10 client.15843 handle_command_reply: tid=0 2024-04-21T23:19:02.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.812+0000 7ff9e7277740 1 client.15843 shutdown 2024-04-21T23:19:02.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.812+0000 7ff9e7277740 1 -- 172.21.15.40:0/2293158820 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575c7349940 msgr2=0x5575c734bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:02.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.812+0000 7ff9e7277740 1 --2- 172.21.15.40:0/2293158820 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575c7349940 0x5575c734bb30 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc021040 tx=0x7ff9cc0140d0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.812+0000 7ff9e7277740 1 -- 172.21.15.40:0/2293158820 shutdown_connections 2024-04-21T23:19:02.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.812+0000 7ff9e7277740 1 --2- 172.21.15.40:0/2293158820 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x5575c73c8cf0 0x5575c73cb110 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.812+0000 7ff9e7277740 1 --2- 172.21.15.40:0/2293158820 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5575c7352520 0x5575c734c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.812+0000 7ff9e7277740 1 --2- 172.21.15.40:0/2293158820 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575c7349940 0x5575c734bb30 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.812+0000 7ff9e7277740 1 --2- 172.21.15.40:0/2293158820 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5575c7349020 0x5575c735aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.812+0000 7ff9e7277740 1 -- 172.21.15.40:0/2293158820 >> 172.21.15.40:0/2293158820 conn(0x5575c7343750 msgr2=0x5575c735d220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:02.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.812+0000 7ff9c5ffb640 0 client.15843 ms_handle_reset on v2:172.21.15.111:6838/3712235548 2024-04-21T23:19:02.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.812+0000 7ff9e7277740 1 -- 172.21.15.40:0/2293158820 shutdown_connections 2024-04-21T23:19:02.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.812+0000 7ff9e7277740 1 -- 172.21.15.40:0/2293158820 wait complete. 2024-04-21T23:19:02.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.812+0000 7ff9e7277740 20 client.15843 trim_cache size 0 max 16384 2024-04-21T23:19:02.818 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-04-21T23:19:02.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.813+0000 7ff9e4688640 1 -- 172.21.15.40:0/3099887061 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7ff9c80a2eb0 msgr2=0x7ff9c80a5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:02.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.813+0000 7ff9e4688640 1 --2- 172.21.15.40:0/3099887061 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7ff9c80a2eb0 0x7ff9c80a5370 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff9d4004af0 tx=0x7ff9d4005c50 comp rx=0 tx=0).stop 2024-04-21T23:19:02.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.813+0000 7ff9e4688640 1 -- 172.21.15.40:0/3099887061 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff9dc1029c0 msgr2=0x7ff9dc198000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:02.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.813+0000 7ff9e4688640 1 --2- 172.21.15.40:0/3099887061 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff9dc1029c0 0x7ff9dc198000 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff9d800e9f0 tx=0x7ff9d800eeb0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.813+0000 7ff9e4688640 1 -- 172.21.15.40:0/3099887061 shutdown_connections 2024-04-21T23:19:02.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.813+0000 7ff9e4688640 1 --2- 172.21.15.40:0/3099887061 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7ff9c80a2eb0 0x7ff9c80a5370 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.813+0000 7ff9e4688640 1 --2- 172.21.15.40:0/3099887061 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9dc10bd00 0x7ff9dc19c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.813+0000 7ff9e4688640 1 --2- 172.21.15.40:0/3099887061 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff9dc1032e0 0x7ff9dc198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.813+0000 7ff9e4688640 1 --2- 172.21.15.40:0/3099887061 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff9dc1029c0 0x7ff9dc198000 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:02.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.813+0000 7ff9e4688640 1 -- 172.21.15.40:0/3099887061 >> 172.21.15.40:0/3099887061 conn(0x7ff9dc077740 msgr2=0x7ff9dc0fd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:02.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.813+0000 7ff9e4688640 1 -- 172.21.15.40:0/3099887061 shutdown_connections 2024-04-21T23:19:02.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:02.813+0000 7ff9e4688640 1 -- 172.21.15.40:0/3099887061 wait complete. 2024-04-21T23:19:03.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:19:03 smithi040 ceph-mon[27984]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-21T23:19:03.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:19:03 smithi040 ceph-mon[27984]: mgrmap e25: x(active, since 38s), standbys: y, z 2024-04-21T23:19:03.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:19:03 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/2834389560' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:19:03.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:19:03 smithi174 ceph-mon[31409]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-21T23:19:03.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:19:03 smithi174 ceph-mon[31409]: mgrmap e25: x(active, since 38s), standbys: y, z 2024-04-21T23:19:03.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:19:03 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/2834389560' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:19:03.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:19:03 smithi111 ceph-mon[31085]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-21T23:19:03.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:19:03 smithi111 ceph-mon[31085]: mgrmap e25: x(active, since 38s), standbys: y, z 2024-04-21T23:19:03.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:19:03 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/2834389560' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:19:03.823 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T23:19:03.824 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:19:03.824 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T23:19:03.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.934+0000 7f2f22e68640 1 -- 172.21.15.40:0/3037403956 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f1c1066b0 msgr2=0x7f2f1c106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:03.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.934+0000 7f2f22e68640 1 --2- 172.21.15.40:0/3037403956 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f1c1066b0 0x7f2f1c106a90 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f2f10009930 tx=0x7f2f1002bc30 comp rx=0 tx=0).stop 2024-04-21T23:19:03.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.934+0000 7f2f22e68640 1 -- 172.21.15.40:0/3037403956 shutdown_connections 2024-04-21T23:19:03.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.934+0000 7f2f22e68640 1 --2- 172.21.15.40:0/3037403956 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f1c100610 0x7f2f1c100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:03.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.934+0000 7f2f22e68640 1 --2- 172.21.15.40:0/3037403956 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2f1c100fd0 0x7f2f1c10d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:03.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.934+0000 7f2f22e68640 1 --2- 172.21.15.40:0/3037403956 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f1c1066b0 0x7f2f1c106a90 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:03.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.934+0000 7f2f22e68640 1 -- 172.21.15.40:0/3037403956 >> 172.21.15.40:0/3037403956 conn(0x7f2f1c077740 msgr2=0x7f2f1c077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:03.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.934+0000 7f2f22e68640 1 -- 172.21.15.40:0/3037403956 shutdown_connections 2024-04-21T23:19:03.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.934+0000 7f2f22e68640 1 -- 172.21.15.40:0/3037403956 wait complete. 2024-04-21T23:19:03.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.934+0000 7f2f22e68640 1 Processor -- start 2024-04-21T23:19:03.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.935+0000 7f2f22e68640 1 -- start start 2024-04-21T23:19:03.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.935+0000 7f2f22e68640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f1c100610 0x7f2f1c198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:03.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.935+0000 7f2f22e68640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f1c100fd0 0x7f2f1c198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:03.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.935+0000 7f2f22e68640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2f1c1066b0 0x7f2f1c19c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:03.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.935+0000 7f2f22e68640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2f1c110a80 con 0x7f2f1c100610 2024-04-21T23:19:03.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.935+0000 7f2f22e68640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f2f1c110900 con 0x7f2f1c100fd0 2024-04-21T23:19:03.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.935+0000 7f2f22e68640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2f1c110c00 con 0x7f2f1c1066b0 2024-04-21T23:19:03.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.935+0000 7f2f20c04640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f1c100610 0x7f2f1c198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:03.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.935+0000 7f2f20c04640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f1c100610 0x7f2f1c198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47632/0 (socket says 172.21.15.40:47632) 2024-04-21T23:19:03.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.935+0000 7f2f20c04640 1 -- 172.21.15.40:0/1690987125 learned_addr learned my addr 172.21.15.40:0/1690987125 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:03.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.935+0000 7f2f1bfff640 1 --2- 172.21.15.40:0/1690987125 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f1c100fd0 0x7f2f1c198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:03.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.935+0000 7f2f21405640 1 --2- 172.21.15.40:0/1690987125 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2f1c1066b0 0x7f2f1c19c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:03.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.935+0000 7f2f20c04640 1 -- 172.21.15.40:0/1690987125 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2f1c1066b0 msgr2=0x7f2f1c19c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:03.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.935+0000 7f2f20c04640 1 --2- 172.21.15.40:0/1690987125 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2f1c1066b0 0x7f2f1c19c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:03.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.935+0000 7f2f20c04640 1 -- 172.21.15.40:0/1690987125 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f1c100fd0 msgr2=0x7f2f1c198540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:03.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.935+0000 7f2f20c04640 1 --2- 172.21.15.40:0/1690987125 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f1c100fd0 0x7f2f1c198540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:03.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.935+0000 7f2f20c04640 1 -- 172.21.15.40:0/1690987125 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f1c19cff0 con 0x7f2f1c100610 2024-04-21T23:19:03.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.935+0000 7f2f1bfff640 1 --2- 172.21.15.40:0/1690987125 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f1c100fd0 0x7f2f1c198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:19:03.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.935+0000 7f2f21405640 1 --2- 172.21.15.40:0/1690987125 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2f1c1066b0 0x7f2f1c19c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:19:03.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.935+0000 7f2f20c04640 1 --2- 172.21.15.40:0/1690987125 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f1c100610 0x7f2f1c198000 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f2f10009900 tx=0x7f2f10004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:03.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.936+0000 7f2f19ffb640 1 -- 172.21.15.40:0/1690987125 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f10043020 con 0x7f2f1c100610 2024-04-21T23:19:03.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.936+0000 7f2f22e68640 1 -- 172.21.15.40:0/1690987125 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f1c19d280 con 0x7f2f1c100610 2024-04-21T23:19:03.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.936+0000 7f2f19ffb640 1 -- 172.21.15.40:0/1690987125 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f100054e0 con 0x7f2f1c100610 2024-04-21T23:19:03.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.936+0000 7f2f19ffb640 1 -- 172.21.15.40:0/1690987125 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f10003bb0 con 0x7f2f1c100610 2024-04-21T23:19:03.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.936+0000 7f2f22e68640 1 -- 172.21.15.40:0/1690987125 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f1c1a4b70 con 0x7f2f1c100610 2024-04-21T23:19:03.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.938+0000 7f2f19ffb640 1 -- 172.21.15.40:0/1690987125 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 25) ==== 138117+0+0 (secure 0 0 0) 0x7f2f10046020 con 0x7f2f1c100610 2024-04-21T23:19:03.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.939+0000 7f2f25a45740 1 Processor -- start 2024-04-21T23:19:03.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.939+0000 7f2f25a45740 1 -- start start 2024-04-21T23:19:03.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.939+0000 7f2f19ffb640 1 --2- 172.21.15.40:0/1690987125 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f2f0c0a2eb0 0x7f2f0c0a5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:03.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.939+0000 7f2f25a45740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5575fe48c720 0x5575fe48b470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:03.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.939+0000 7f2f25a45740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575fe48ba40 0x5575fe48bec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:03.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.939+0000 7f2f25a45740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5575fe494db0 0x5575fe4971a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:03.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.939+0000 7f2f19ffb640 1 -- 172.21.15.40:0/1690987125 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f2f100f1a80 con 0x7f2f1c100610 2024-04-21T23:19:03.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.939+0000 7f2f21405640 1 --2- 172.21.15.40:0/1690987125 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f2f0c0a2eb0 0x7f2f0c0a5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:03.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.939+0000 7f2f25a45740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5575fe3ef170 con 0x5575fe48ba40 2024-04-21T23:19:03.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.939+0000 7f2f25a45740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5575fe3977c0 con 0x5575fe48c720 2024-04-21T23:19:03.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.939+0000 7f2f25a45740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5575fe3eb130 con 0x5575fe494db0 2024-04-21T23:19:03.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.939+0000 7f2f1bfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575fe48ba40 0x5575fe48bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:03.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.939+0000 7f2f1bfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575fe48ba40 0x5575fe48bec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47646/0 (socket says 172.21.15.40:47646) 2024-04-21T23:19:03.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.939+0000 7f2f1bfff640 1 -- 172.21.15.40:0/3873238122 learned_addr learned my addr 172.21.15.40:0/3873238122 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:03.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.939+0000 7f2f20c04640 1 --2- 172.21.15.40:0/3873238122 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5575fe48c720 0x5575fe48b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:03.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.939+0000 7f2f1bfff640 1 -- 172.21.15.40:0/3873238122 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5575fe494db0 msgr2=0x5575fe4971a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:03.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.939+0000 7f2f1bfff640 1 --2- 172.21.15.40:0/3873238122 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5575fe494db0 0x5575fe4971a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:03.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.939+0000 7f2f1bfff640 1 -- 172.21.15.40:0/3873238122 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5575fe48c720 msgr2=0x5575fe48b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:03.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.939+0000 7f2f1bfff640 1 --2- 172.21.15.40:0/3873238122 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5575fe48c720 0x5575fe48b470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:03.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.939+0000 7f2f1bfff640 1 -- 172.21.15.40:0/3873238122 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5575fe3a8e60 con 0x5575fe48ba40 2024-04-21T23:19:03.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.940+0000 7f2f21405640 1 --2- 172.21.15.40:0/1690987125 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f2f0c0a2eb0 0x7f2f0c0a5370 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2f14005fd0 tx=0x7f2f1400f730 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:03.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.940+0000 7f2f1bfff640 1 --2- 172.21.15.40:0/3873238122 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575fe48ba40 0x5575fe48bec0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f2f08009b80 tx=0x7f2f08010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2243be4211f16b86 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:03.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.940+0000 7f2f06ffd640 1 -- 172.21.15.40:0/3873238122 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f08004030 con 0x5575fe48ba40 2024-04-21T23:19:03.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.940+0000 7f2f06ffd640 1 -- 172.21.15.40:0/3873238122 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f080041d0 con 0x5575fe48ba40 2024-04-21T23:19:03.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.940+0000 7f2f06ffd640 1 -- 172.21.15.40:0/3873238122 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f080126b0 con 0x5575fe48ba40 2024-04-21T23:19:03.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.941+0000 7f2f25a45740 1 -- 172.21.15.40:0/3873238122 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575fe48ba40 msgr2=0x5575fe48bec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:03.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.941+0000 7f2f25a45740 1 --2- 172.21.15.40:0/3873238122 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575fe48ba40 0x5575fe48bec0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f2f08009b80 tx=0x7f2f08010740 comp rx=0 tx=0).stop 2024-04-21T23:19:03.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.941+0000 7f2f25a45740 1 -- 172.21.15.40:0/3873238122 shutdown_connections 2024-04-21T23:19:03.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.941+0000 7f2f25a45740 1 --2- 172.21.15.40:0/3873238122 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5575fe494db0 0x5575fe4971a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:03.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.941+0000 7f2f25a45740 1 --2- 172.21.15.40:0/3873238122 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575fe48ba40 0x5575fe48bec0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:03.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.941+0000 7f2f25a45740 1 --2- 172.21.15.40:0/3873238122 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5575fe48c720 0x5575fe48b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:03.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.941+0000 7f2f25a45740 1 -- 172.21.15.40:0/3873238122 >> 172.21.15.40:0/3873238122 conn(0x5575fe4524f0 msgr2=0x5575fe486490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:03.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.941+0000 7f2f25a45740 1 -- 172.21.15.40:0/3873238122 shutdown_connections 2024-04-21T23:19:03.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.941+0000 7f2f25a45740 1 -- 172.21.15.40:0/3873238122 wait complete. 2024-04-21T23:19:03.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.941+0000 7f2f25a45740 1 -- start start 2024-04-21T23:19:03.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.941+0000 7f2f25a45740 1 Processor -- start 2024-04-21T23:19:03.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.941+0000 7f2f25a45740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5575fe48ba40 0x5575fe491dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:03.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.941+0000 7f2f25a45740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5575fe48c720 0x5575fe492300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:03.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.942+0000 7f2f25a45740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575fe494db0 0x5575fe492840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:03.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.942+0000 7f2f25a45740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5575fe497c20 con 0x5575fe494db0 2024-04-21T23:19:03.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.942+0000 7f2f25a45740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5575fe3aad40 con 0x5575fe48ba40 2024-04-21T23:19:03.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.942+0000 7f2f25a45740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5575fe497da0 con 0x5575fe48c720 2024-04-21T23:19:03.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.942+0000 7f2f25a45740 10 client.0 _ll_register_callbacks cb 0x5575fe3e6820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:19:03.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.942+0000 7f2f21405640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575fe494db0 0x5575fe492840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:03.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.942+0000 7f2f21405640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575fe494db0 0x5575fe492840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47656/0 (socket says 172.21.15.40:47656) 2024-04-21T23:19:03.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.942+0000 7f2f21405640 1 -- 172.21.15.40:0/1766644950 learned_addr learned my addr 172.21.15.40:0/1766644950 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:03.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.942+0000 7f2f20c04640 1 --2- 172.21.15.40:0/1766644950 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5575fe48ba40 0x5575fe491dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:03.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.942+0000 7f2f1bfff640 1 --2- 172.21.15.40:0/1766644950 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5575fe48c720 0x5575fe492300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:03.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.942+0000 7f2f21405640 1 -- 172.21.15.40:0/1766644950 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5575fe48c720 msgr2=0x5575fe492300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:03.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.942+0000 7f2f21405640 1 --2- 172.21.15.40:0/1766644950 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5575fe48c720 0x5575fe492300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:03.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.942+0000 7f2f21405640 1 -- 172.21.15.40:0/1766644950 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5575fe48ba40 msgr2=0x5575fe491dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:03.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.942+0000 7f2f21405640 1 --2- 172.21.15.40:0/1766644950 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5575fe48ba40 0x5575fe491dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:03.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.942+0000 7f2f21405640 1 -- 172.21.15.40:0/1766644950 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5575fe1cb7e0 con 0x5575fe494db0 2024-04-21T23:19:03.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.942+0000 7f2f20c04640 1 --2- 172.21.15.40:0/1766644950 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5575fe48ba40 0x5575fe491dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:19:03.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.942+0000 7f2f21405640 1 --2- 172.21.15.40:0/1766644950 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575fe494db0 0x5575fe492840 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f2f14011800 tx=0x7f2f14004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4950a9f8784c9d22 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:03.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.942+0000 7f2efbfff640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-21T23:19:03.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.942+0000 7f2efbfff640 1 -- 172.21.15.40:0/1766644950 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f14010b50 con 0x5575fe494db0 2024-04-21T23:19:03.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.942+0000 7f2efbfff640 1 -- 172.21.15.40:0/1766644950 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f2f14010cf0 con 0x5575fe494db0 2024-04-21T23:19:03.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.942+0000 7f2f25a45740 1 -- 172.21.15.40:0/1766644950 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2f080041d0 con 0x5575fe494db0 2024-04-21T23:19:03.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.943+0000 7f2efbfff640 1 -- 172.21.15.40:0/1766644950 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f2f1402d050 con 0x5575fe494db0 2024-04-21T23:19:03.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.943+0000 7f2f25a45740 10 client.15855 fetch_fsmap learned FSMap version 25 2024-04-21T23:19:03.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.943+0000 7f2f25a45740 1 -- 172.21.15.40:0/1766644950 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5575fe22bf70 con 0x5575fe494db0 2024-04-21T23:19:03.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.943+0000 7f2efbfff640 1 -- 172.21.15.40:0/1766644950 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f080041d0 con 0x5575fe494db0 2024-04-21T23:19:03.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.943+0000 7f2efbfff640 1 -- 172.21.15.40:0/1766644950 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f2f140285b0 con 0x5575fe494db0 2024-04-21T23:19:03.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.943+0000 7f2f25a45740 10 client.15855 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:19:03.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.943+0000 7f2f25a45740 10 client.15855 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:19:03.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.943+0000 7f2f25a45740 20 client.15855 populate_metadata read hostname 'smithi040' 2024-04-21T23:19:03.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.943+0000 7f2f25a45740 1 --2- 172.21.15.40:0/1766644950 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x5575fe4a4fa0 0x5575fe4a73f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:03.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.943+0000 7f2f25a45740 4 client.15855 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:19:03.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.943+0000 7f2f25a45740 1 -- 172.21.15.40:0/1766644950 --> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5575fe3c3450 con 0x5575fe4a4fa0 2024-04-21T23:19:03.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.944+0000 7f2f20c04640 1 --2- 172.21.15.40:0/1766644950 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x5575fe4a4fa0 0x5575fe4a73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:03.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.944+0000 7f2f20c04640 1 --2- 172.21.15.40:0/1766644950 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x5575fe4a4fa0 0x5575fe4a73f0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=737d000dd92aecc1 server_cookie=ee339c08c3c73a6d in_seq=0 out_seq=0 2024-04-21T23:19:03.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.944+0000 7f2efbfff640 10 client.15855 ms_handle_connect on v2:172.21.15.111:6838/3712235548 2024-04-21T23:19:03.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.945+0000 7f2efbfff640 1 -- 172.21.15.40:0/1766644950 <== mds.0 v2:172.21.15.111:6838/3712235548 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5575fe3c3450 con 0x5575fe4a4fa0 2024-04-21T23:19:03.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.945+0000 7f2efbfff640 10 client.15855 handle_command_reply: tid=0 2024-04-21T23:19:03.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.945+0000 7f2f25a45740 1 client.15855 shutdown 2024-04-21T23:19:03.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.945+0000 7f2f25a45740 1 -- 172.21.15.40:0/1766644950 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575fe494db0 msgr2=0x5575fe492840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:03.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.945+0000 7f2f25a45740 1 --2- 172.21.15.40:0/1766644950 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575fe494db0 0x5575fe492840 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f2f14011800 tx=0x7f2f14004ab0 comp rx=0 tx=0).stop 2024-04-21T23:19:03.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.945+0000 7f2f25a45740 1 -- 172.21.15.40:0/1766644950 shutdown_connections 2024-04-21T23:19:03.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.945+0000 7f2f25a45740 1 --2- 172.21.15.40:0/1766644950 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x5575fe4a4fa0 0x5575fe4a73f0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:03.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.945+0000 7f2f25a45740 1 --2- 172.21.15.40:0/1766644950 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575fe494db0 0x5575fe492840 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:03.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.945+0000 7f2efbfff640 0 client.15855 ms_handle_reset on v2:172.21.15.111:6838/3712235548 2024-04-21T23:19:03.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.945+0000 7f2f25a45740 1 --2- 172.21.15.40:0/1766644950 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5575fe48c720 0x5575fe492300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:03.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.945+0000 7f2f25a45740 1 --2- 172.21.15.40:0/1766644950 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5575fe48ba40 0x5575fe491dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:03.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.945+0000 7f2f25a45740 1 -- 172.21.15.40:0/1766644950 >> 172.21.15.40:0/1766644950 conn(0x5575fe4524f0 msgr2=0x5575fe4881e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:03.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.945+0000 7f2f25a45740 1 -- 172.21.15.40:0/1766644950 shutdown_connections 2024-04-21T23:19:03.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.945+0000 7f2f25a45740 1 -- 172.21.15.40:0/1766644950 wait complete. 2024-04-21T23:19:03.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.945+0000 7f2f25a45740 20 client.15855 trim_cache size 0 max 16384 2024-04-21T23:19:03.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.963+0000 7f2f25a45740 1 Processor -- start 2024-04-21T23:19:03.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.963+0000 7f2f25a45740 1 -- start start 2024-04-21T23:19:03.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.963+0000 7f2f25a45740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5575fe48ba40 0x5575fe4944f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:03.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.963+0000 7f2f25a45740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575fe48c720 0x5575fe4a0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:03.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.963+0000 7f2f25a45740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5575fe494db0 0x5575fe4accc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:03.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.963+0000 7f2f25a45740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2f1c1b1600 con 0x5575fe48c720 2024-04-21T23:19:03.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.963+0000 7f2f25a45740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f2f1c1aeed0 con 0x5575fe48ba40 2024-04-21T23:19:03.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.963+0000 7f2f25a45740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2f1c1acd20 con 0x5575fe494db0 2024-04-21T23:19:03.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.963+0000 7f2f1bfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575fe48c720 0x5575fe4a0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:03.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.963+0000 7f2f1bfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575fe48c720 0x5575fe4a0b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47664/0 (socket says 172.21.15.40:47664) 2024-04-21T23:19:03.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.963+0000 7f2f1bfff640 1 -- 172.21.15.40:0/247704580 learned_addr learned my addr 172.21.15.40:0/247704580 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:03.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.963+0000 7f2f20c04640 1 --2- 172.21.15.40:0/247704580 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5575fe48ba40 0x5575fe4944f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:03.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.963+0000 7f2f21405640 1 --2- 172.21.15.40:0/247704580 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5575fe494db0 0x5575fe4accc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:03.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.963+0000 7f2f1bfff640 1 -- 172.21.15.40:0/247704580 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5575fe494db0 msgr2=0x5575fe4accc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:03.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.963+0000 7f2f1bfff640 1 --2- 172.21.15.40:0/247704580 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5575fe494db0 0x5575fe4accc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:03.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.963+0000 7f2f1bfff640 1 -- 172.21.15.40:0/247704580 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5575fe48ba40 msgr2=0x5575fe4944f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:03.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.963+0000 7f2f1bfff640 1 --2- 172.21.15.40:0/247704580 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5575fe48ba40 0x5575fe4944f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:03.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.963+0000 7f2f1bfff640 1 -- 172.21.15.40:0/247704580 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5575fe0c3170 con 0x5575fe48c720 2024-04-21T23:19:03.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.964+0000 7f2f1bfff640 1 --2- 172.21.15.40:0/247704580 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575fe48c720 0x5575fe4a0b50 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f2f08009b80 tx=0x7f2f08005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=860459aff72951e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:03.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.964+0000 7f2f067fc640 1 -- 172.21.15.40:0/247704580 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f080052c0 con 0x5575fe48c720 2024-04-21T23:19:03.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.964+0000 7f2f067fc640 1 -- 172.21.15.40:0/247704580 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f08003ad0 con 0x5575fe48c720 2024-04-21T23:19:03.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.964+0000 7f2f067fc640 1 -- 172.21.15.40:0/247704580 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f08012ad0 con 0x5575fe48c720 2024-04-21T23:19:03.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.964+0000 7f2f25a45740 1 -- 172.21.15.40:0/247704580 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575fe48c720 msgr2=0x5575fe4a0b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:03.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.964+0000 7f2f25a45740 1 --2- 172.21.15.40:0/247704580 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575fe48c720 0x5575fe4a0b50 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f2f08009b80 tx=0x7f2f08005850 comp rx=0 tx=0).stop 2024-04-21T23:19:03.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.964+0000 7f2f25a45740 1 -- 172.21.15.40:0/247704580 shutdown_connections 2024-04-21T23:19:03.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.964+0000 7f2f25a45740 1 --2- 172.21.15.40:0/247704580 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5575fe494db0 0x5575fe4accc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:03.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.964+0000 7f2f25a45740 1 --2- 172.21.15.40:0/247704580 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575fe48c720 0x5575fe4a0b50 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:03.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.964+0000 7f2f25a45740 1 --2- 172.21.15.40:0/247704580 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5575fe48ba40 0x5575fe4944f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:03.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.964+0000 7f2f25a45740 1 -- 172.21.15.40:0/247704580 >> 172.21.15.40:0/247704580 conn(0x5575fe4524f0 msgr2=0x5575fe488170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:03.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.965+0000 7f2f25a45740 1 -- 172.21.15.40:0/247704580 shutdown_connections 2024-04-21T23:19:03.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.965+0000 7f2f25a45740 1 -- 172.21.15.40:0/247704580 wait complete. 2024-04-21T23:19:03.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.965+0000 7f2f25a45740 1 -- start start 2024-04-21T23:19:03.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.965+0000 7f2f25a45740 1 Processor -- start 2024-04-21T23:19:03.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.965+0000 7f2f25a45740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5575fe48ba40 0x5575fe49d060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:03.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.965+0000 7f2f25a45740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5575fe48c720 0x5575fe49d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:03.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.965+0000 7f2f25a45740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575fe494db0 0x5575fe48e030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:03.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.965+0000 7f2f25a45740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2f1c1ad390 con 0x5575fe494db0 2024-04-21T23:19:03.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.965+0000 7f2f25a45740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f2f1c1ae800 con 0x5575fe48ba40 2024-04-21T23:19:03.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.965+0000 7f2f25a45740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2f1c1ada60 con 0x5575fe48c720 2024-04-21T23:19:03.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.965+0000 7f2f25a45740 10 client.0 _ll_register_callbacks cb 0x7f2f1400fe00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:19:03.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.965+0000 7f2f21405640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575fe494db0 0x5575fe48e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:03.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.965+0000 7f2f21405640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575fe494db0 0x5575fe48e030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47676/0 (socket says 172.21.15.40:47676) 2024-04-21T23:19:03.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.965+0000 7f2f21405640 1 -- 172.21.15.40:0/3154384490 learned_addr learned my addr 172.21.15.40:0/3154384490 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:03.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.966+0000 7f2f20c04640 1 --2- 172.21.15.40:0/3154384490 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5575fe48ba40 0x5575fe49d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:03.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.966+0000 7f2f1bfff640 1 --2- 172.21.15.40:0/3154384490 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5575fe48c720 0x5575fe49d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:03.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.966+0000 7f2f21405640 1 -- 172.21.15.40:0/3154384490 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5575fe48c720 msgr2=0x5575fe49d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:03.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.966+0000 7f2f21405640 1 --2- 172.21.15.40:0/3154384490 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5575fe48c720 0x5575fe49d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:03.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.966+0000 7f2f21405640 1 -- 172.21.15.40:0/3154384490 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5575fe48ba40 msgr2=0x5575fe49d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:03.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.966+0000 7f2f21405640 1 --2- 172.21.15.40:0/3154384490 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5575fe48ba40 0x5575fe49d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:03.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.966+0000 7f2f21405640 1 -- 172.21.15.40:0/3154384490 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5575fe48e830 con 0x5575fe494db0 2024-04-21T23:19:03.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.966+0000 7f2f21405640 1 --2- 172.21.15.40:0/3154384490 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575fe494db0 0x5575fe48e030 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f2f14000f80 tx=0x7f2f14014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9b0aef5769ceeb7 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:03.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.966+0000 7f2efbfff640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-21T23:19:03.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.966+0000 7f2efbfff640 1 -- 172.21.15.40:0/3154384490 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f14037070 con 0x5575fe494db0 2024-04-21T23:19:03.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.966+0000 7f2efbfff640 1 -- 172.21.15.40:0/3154384490 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f2f14010bb0 con 0x5575fe494db0 2024-04-21T23:19:03.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.966+0000 7f2f25a45740 1 -- 172.21.15.40:0/3154384490 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2f08003ad0 con 0x5575fe494db0 2024-04-21T23:19:03.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.966+0000 7f2efbfff640 1 -- 172.21.15.40:0/3154384490 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f2f14011430 con 0x5575fe494db0 2024-04-21T23:19:03.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.966+0000 7f2f25a45740 10 client.15861 fetch_fsmap learned FSMap version 25 2024-04-21T23:19:03.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.966+0000 7f2f25a45740 1 -- 172.21.15.40:0/3154384490 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5575fe48fb30 con 0x5575fe494db0 2024-04-21T23:19:03.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.966+0000 7f2efbfff640 1 -- 172.21.15.40:0/3154384490 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f08003ad0 con 0x5575fe494db0 2024-04-21T23:19:03.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.967+0000 7f2efbfff640 1 -- 172.21.15.40:0/3154384490 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f2f1401f450 con 0x5575fe494db0 2024-04-21T23:19:03.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.967+0000 7f2f25a45740 10 client.15861 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:19:03.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.967+0000 7f2f25a45740 10 client.15861 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:19:03.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.967+0000 7f2f25a45740 20 client.15861 populate_metadata read hostname 'smithi040' 2024-04-21T23:19:03.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.967+0000 7f2f25a45740 1 --2- 172.21.15.40:0/3154384490 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x5575fe50c6f0 0x5575fe50eb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:03.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.967+0000 7f2f25a45740 4 client.15861 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T23:19:03.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.967+0000 7f2f25a45740 1 -- 172.21.15.40:0/3154384490 --> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5575fe3fc6b0 con 0x5575fe50c6f0 2024-04-21T23:19:03.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.967+0000 7f2f20c04640 1 --2- 172.21.15.40:0/3154384490 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x5575fe50c6f0 0x5575fe50eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:03.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.967+0000 7f2f20c04640 1 --2- 172.21.15.40:0/3154384490 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x5575fe50c6f0 0x5575fe50eb10 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9362a12492faa4c6 server_cookie=989734577b804fdb in_seq=0 out_seq=0 2024-04-21T23:19:03.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.967+0000 7f2efbfff640 10 client.15861 ms_handle_connect on v2:172.21.15.111:6838/3712235548 2024-04-21T23:19:03.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.968+0000 7f2efbfff640 1 -- 172.21.15.40:0/3154384490 <== mds.0 v2:172.21.15.111:6838/3712235548 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5575fe3fc6b0 con 0x5575fe50c6f0 2024-04-21T23:19:03.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.968+0000 7f2efbfff640 10 client.15861 handle_command_reply: tid=0 2024-04-21T23:19:03.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.968+0000 7f2f25a45740 1 client.15861 shutdown 2024-04-21T23:19:03.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.969+0000 7f2f25a45740 1 -- 172.21.15.40:0/3154384490 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575fe494db0 msgr2=0x5575fe48e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:03.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.969+0000 7f2f25a45740 1 --2- 172.21.15.40:0/3154384490 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575fe494db0 0x5575fe48e030 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f2f14000f80 tx=0x7f2f14014000 comp rx=0 tx=0).stop 2024-04-21T23:19:03.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.969+0000 7f2f25a45740 1 -- 172.21.15.40:0/3154384490 shutdown_connections 2024-04-21T23:19:03.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.969+0000 7f2f25a45740 1 --2- 172.21.15.40:0/3154384490 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x5575fe50c6f0 0x5575fe50eb10 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:03.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.969+0000 7f2f25a45740 1 --2- 172.21.15.40:0/3154384490 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575fe494db0 0x5575fe48e030 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:03.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.969+0000 7f2efbfff640 0 client.15861 ms_handle_reset on v2:172.21.15.111:6838/3712235548 2024-04-21T23:19:03.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.969+0000 7f2f25a45740 1 --2- 172.21.15.40:0/3154384490 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5575fe48c720 0x5575fe49d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:03.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.969+0000 7f2f25a45740 1 --2- 172.21.15.40:0/3154384490 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5575fe48ba40 0x5575fe49d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:03.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.969+0000 7f2f25a45740 1 -- 172.21.15.40:0/3154384490 >> 172.21.15.40:0/3154384490 conn(0x5575fe4524f0 msgr2=0x5575fe491d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:03.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.969+0000 7f2f25a45740 1 -- 172.21.15.40:0/3154384490 shutdown_connections 2024-04-21T23:19:03.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.969+0000 7f2f25a45740 1 -- 172.21.15.40:0/3154384490 wait complete. 2024-04-21T23:19:03.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.969+0000 7f2f25a45740 20 client.15861 trim_cache size 0 max 16384 2024-04-21T23:19:03.974 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-21T23:19:03.974 INFO:teuthology.orchestra.run.smithi040.stdout: "return_code": 0, 2024-04-21T23:19:03.974 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_tag": "dc2287ba-9ce6-429a-8944-a9bd5517a108", 2024-04-21T23:19:03.974 INFO:teuthology.orchestra.run.smithi040.stdout: "mode": "asynchronous" 2024-04-21T23:19:03.974 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-21T23:19:03.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.970+0000 7f2f22e68640 1 -- 172.21.15.40:0/1690987125 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f2f0c0a2eb0 msgr2=0x7f2f0c0a5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:03.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.970+0000 7f2f22e68640 1 --2- 172.21.15.40:0/1690987125 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f2f0c0a2eb0 0x7f2f0c0a5370 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2f14005fd0 tx=0x7f2f1400f730 comp rx=0 tx=0).stop 2024-04-21T23:19:03.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.970+0000 7f2f22e68640 1 -- 172.21.15.40:0/1690987125 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f1c100610 msgr2=0x7f2f1c198000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:03.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.970+0000 7f2f22e68640 1 --2- 172.21.15.40:0/1690987125 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f1c100610 0x7f2f1c198000 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f2f10009900 tx=0x7f2f10004360 comp rx=0 tx=0).stop 2024-04-21T23:19:03.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.970+0000 7f2f22e68640 1 -- 172.21.15.40:0/1690987125 shutdown_connections 2024-04-21T23:19:03.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.970+0000 7f2f22e68640 1 --2- 172.21.15.40:0/1690987125 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f2f0c0a2eb0 0x7f2f0c0a5370 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:03.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.970+0000 7f2f22e68640 1 --2- 172.21.15.40:0/1690987125 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f1c100fd0 0x7f2f1c198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:03.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.970+0000 7f2f22e68640 1 --2- 172.21.15.40:0/1690987125 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2f1c1066b0 0x7f2f1c19c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:03.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.970+0000 7f2f22e68640 1 --2- 172.21.15.40:0/1690987125 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f1c100610 0x7f2f1c198000 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:03.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.970+0000 7f2f22e68640 1 -- 172.21.15.40:0/1690987125 >> 172.21.15.40:0/1690987125 conn(0x7f2f1c077740 msgr2=0x7f2f1c068b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:03.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.970+0000 7f2f22e68640 1 -- 172.21.15.40:0/1690987125 shutdown_connections 2024-04-21T23:19:03.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:03.970+0000 7f2f22e68640 1 -- 172.21.15.40:0/1690987125 wait complete. 2024-04-21T23:19:03.980 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T23:19:03.980 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T23:19:04.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.090+0000 7fa8e4313640 1 -- 172.21.15.40:0/998777157 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa8dc106630 msgr2=0x7fa8dc106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:04.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.090+0000 7fa8e4313640 1 --2- 172.21.15.40:0/998777157 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa8dc106630 0x7fa8dc106a10 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fa8d8009930 tx=0x7fa8d802bc30 comp rx=0 tx=0).stop 2024-04-21T23:19:04.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.090+0000 7fa8e4313640 1 -- 172.21.15.40:0/998777157 shutdown_connections 2024-04-21T23:19:04.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.090+0000 7fa8e4313640 1 --2- 172.21.15.40:0/998777157 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8dc100fe0 0x7fa8dc10d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:04.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.090+0000 7fa8e4313640 1 --2- 172.21.15.40:0/998777157 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa8dc100620 0x7fa8dc100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:04.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.090+0000 7fa8e4313640 1 --2- 172.21.15.40:0/998777157 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa8dc106630 0x7fa8dc106a10 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:04.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.090+0000 7fa8e4313640 1 -- 172.21.15.40:0/998777157 >> 172.21.15.40:0/998777157 conn(0x7fa8dc0776a0 msgr2=0x7fa8dc077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:04.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.090+0000 7fa8e4313640 1 -- 172.21.15.40:0/998777157 shutdown_connections 2024-04-21T23:19:04.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.090+0000 7fa8e4313640 1 -- 172.21.15.40:0/998777157 wait complete. 2024-04-21T23:19:04.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.090+0000 7fa8e4313640 1 Processor -- start 2024-04-21T23:19:04.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.090+0000 7fa8e4313640 1 -- start start 2024-04-21T23:19:04.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.091+0000 7fa8e4313640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa8dc100620 0x7fa8dc198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:04.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.091+0000 7fa8e4313640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8dc100fe0 0x7fa8dc198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:04.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.091+0000 7fa8e4313640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa8dc106630 0x7fa8dc19c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:04.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.091+0000 7fa8e4313640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa8dc111b40 con 0x7fa8dc100620 2024-04-21T23:19:04.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.091+0000 7fa8e4313640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fa8dc1119c0 con 0x7fa8dc100fe0 2024-04-21T23:19:04.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.091+0000 7fa8e4313640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa8dc111cc0 con 0x7fa8dc106630 2024-04-21T23:19:04.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.091+0000 7fa8e20af640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa8dc100620 0x7fa8dc198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:04.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.091+0000 7fa8e20af640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa8dc100620 0x7fa8dc198020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47688/0 (socket says 172.21.15.40:47688) 2024-04-21T23:19:04.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.091+0000 7fa8e20af640 1 -- 172.21.15.40:0/487021454 learned_addr learned my addr 172.21.15.40:0/487021454 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:04.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.091+0000 7fa8e18ae640 1 --2- 172.21.15.40:0/487021454 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8dc100fe0 0x7fa8dc198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:04.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.091+0000 7fa8e20af640 1 -- 172.21.15.40:0/487021454 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa8dc106630 msgr2=0x7fa8dc19c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:04.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.091+0000 7fa8e28b0640 1 --2- 172.21.15.40:0/487021454 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa8dc106630 0x7fa8dc19c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:04.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.091+0000 7fa8e20af640 1 --2- 172.21.15.40:0/487021454 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa8dc106630 0x7fa8dc19c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:04.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.091+0000 7fa8e20af640 1 -- 172.21.15.40:0/487021454 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8dc100fe0 msgr2=0x7fa8dc198560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:04.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.091+0000 7fa8e20af640 1 --2- 172.21.15.40:0/487021454 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8dc100fe0 0x7fa8dc198560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:04.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.091+0000 7fa8e20af640 1 -- 172.21.15.40:0/487021454 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8dc19d010 con 0x7fa8dc100620 2024-04-21T23:19:04.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.091+0000 7fa8e18ae640 1 --2- 172.21.15.40:0/487021454 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8dc100fe0 0x7fa8dc198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:19:04.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.091+0000 7fa8e28b0640 1 --2- 172.21.15.40:0/487021454 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa8dc106630 0x7fa8dc19c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:19:04.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.091+0000 7fa8e20af640 1 --2- 172.21.15.40:0/487021454 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa8dc100620 0x7fa8dc198020 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fa8d8009230 tx=0x7fa8d802cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:04.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.091+0000 7fa8d37fe640 1 -- 172.21.15.40:0/487021454 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8d8041070 con 0x7fa8dc100620 2024-04-21T23:19:04.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.092+0000 7fa8e4313640 1 -- 172.21.15.40:0/487021454 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8dc19d2a0 con 0x7fa8dc100620 2024-04-21T23:19:04.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.092+0000 7fa8d37fe640 1 -- 172.21.15.40:0/487021454 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8d8004450 con 0x7fa8dc100620 2024-04-21T23:19:04.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.092+0000 7fa8d37fe640 1 -- 172.21.15.40:0/487021454 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8d8034440 con 0x7fa8dc100620 2024-04-21T23:19:04.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.092+0000 7fa8e4313640 1 -- 172.21.15.40:0/487021454 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8dc1a4b90 con 0x7fa8dc100620 2024-04-21T23:19:04.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.094+0000 7fa8d37fe640 1 -- 172.21.15.40:0/487021454 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 25) ==== 138117+0+0 (secure 0 0 0) 0x7fa8d8044070 con 0x7fa8dc100620 2024-04-21T23:19:04.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.094+0000 7fa8e6fda740 1 Processor -- start 2024-04-21T23:19:04.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.094+0000 7fa8e6fda740 1 -- start start 2024-04-21T23:19:04.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.095+0000 7fa8e6fda740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558d4cc16f30 0x558d4cc17310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:04.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.095+0000 7fa8e6fda740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558d4cc17850 0x558d4cc1fef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:04.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.095+0000 7fa8e6fda740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d4cc20430 0x558d4cc22820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:04.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.095+0000 7fa8e6fda740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x558d4cb7b560 con 0x558d4cc20430 2024-04-21T23:19:04.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.095+0000 7fa8e6fda740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x558d4cb23730 con 0x558d4cc17850 2024-04-21T23:19:04.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.095+0000 7fa8e6fda740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x558d4cb77520 con 0x558d4cc16f30 2024-04-21T23:19:04.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.095+0000 7fa8d37fe640 1 --2- 172.21.15.40:0/487021454 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fa8c80a2eb0 0x7fa8c80a5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:04.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.095+0000 7fa8e18ae640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d4cc20430 0x558d4cc22820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:04.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.095+0000 7fa8e28b0640 1 --2- 172.21.15.40:0/487021454 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fa8c80a2eb0 0x7fa8c80a5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:04.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.095+0000 7fa8d37fe640 1 -- 172.21.15.40:0/487021454 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7fa8d80f14b0 con 0x7fa8dc100620 2024-04-21T23:19:04.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.095+0000 7fa8e18ae640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d4cc20430 0x558d4cc22820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47704/0 (socket says 172.21.15.40:47704) 2024-04-21T23:19:04.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.095+0000 7fa8e18ae640 1 -- 172.21.15.40:0/3938157451 learned_addr learned my addr 172.21.15.40:0/3938157451 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:04.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.095+0000 7fa8e20af640 1 --2- 172.21.15.40:0/3938157451 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558d4cc17850 0x558d4cc1fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:04.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.095+0000 7fa8e28b0640 1 --2- 172.21.15.40:0/3938157451 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558d4cc16f30 0x558d4cc17310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:04.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.095+0000 7fa8e18ae640 1 -- 172.21.15.40:0/3938157451 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558d4cc16f30 msgr2=0x558d4cc17310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:04.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.095+0000 7fa8e18ae640 1 --2- 172.21.15.40:0/3938157451 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558d4cc16f30 0x558d4cc17310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:04.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.095+0000 7fa8e18ae640 1 -- 172.21.15.40:0/3938157451 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558d4cc17850 msgr2=0x558d4cc1fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:04.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.095+0000 7fa8e18ae640 1 --2- 172.21.15.40:0/3938157451 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558d4cc17850 0x558d4cc1fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:04.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.095+0000 7fa8e18ae640 1 -- 172.21.15.40:0/3938157451 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558d4c9597d0 con 0x558d4cc20430 2024-04-21T23:19:04.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.095+0000 7fa8e20af640 1 --2- 172.21.15.40:0/3938157451 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558d4cc17850 0x558d4cc1fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:19:04.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.095+0000 7fa8e18ae640 1 --2- 172.21.15.40:0/3938157451 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d4cc20430 0x558d4cc22820 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc00bd30 tx=0x7fa8cc00f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f872a8ae7d79deec server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:04.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.095+0000 7fa8e28b0640 1 --2- 172.21.15.40:0/487021454 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fa8c80a2eb0 0x7fa8c80a5370 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa8d4005fd0 tx=0x7fa8d40094c0 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:04.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.096+0000 7fa8d0bfe640 1 -- 172.21.15.40:0/3938157451 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8cc015070 con 0x558d4cc20430 2024-04-21T23:19:04.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.096+0000 7fa8d0bfe640 1 -- 172.21.15.40:0/3938157451 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8cc00be20 con 0x558d4cc20430 2024-04-21T23:19:04.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.096+0000 7fa8d0bfe640 1 -- 172.21.15.40:0/3938157451 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8cc0119b0 con 0x558d4cc20430 2024-04-21T23:19:04.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.096+0000 7fa8e6fda740 1 -- 172.21.15.40:0/3938157451 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d4cc20430 msgr2=0x558d4cc22820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:04.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.096+0000 7fa8e6fda740 1 --2- 172.21.15.40:0/3938157451 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d4cc20430 0x558d4cc22820 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc00bd30 tx=0x7fa8cc00f730 comp rx=0 tx=0).stop 2024-04-21T23:19:04.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.096+0000 7fa8e6fda740 1 -- 172.21.15.40:0/3938157451 shutdown_connections 2024-04-21T23:19:04.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.096+0000 7fa8e6fda740 1 --2- 172.21.15.40:0/3938157451 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558d4cc17850 0x558d4cc1fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:04.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.096+0000 7fa8e6fda740 1 --2- 172.21.15.40:0/3938157451 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d4cc20430 0x558d4cc22820 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:04.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.097+0000 7fa8e6fda740 1 --2- 172.21.15.40:0/3938157451 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558d4cc16f30 0x558d4cc17310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:04.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.097+0000 7fa8e6fda740 1 -- 172.21.15.40:0/3938157451 >> 172.21.15.40:0/3938157451 conn(0x558d4cc11660 msgr2=0x558d4cc11a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:04.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.097+0000 7fa8e6fda740 1 -- 172.21.15.40:0/3938157451 shutdown_connections 2024-04-21T23:19:04.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.097+0000 7fa8e6fda740 1 -- 172.21.15.40:0/3938157451 wait complete. 2024-04-21T23:19:04.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.097+0000 7fa8e6fda740 1 -- start start 2024-04-21T23:19:04.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.097+0000 7fa8e6fda740 1 Processor -- start 2024-04-21T23:19:04.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.097+0000 7fa8e6fda740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558d4cc16f30 0x558d4cc1d480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:04.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.097+0000 7fa8e6fda740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d4cc17850 0x558d4cc1d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:04.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.097+0000 7fa8e6fda740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558d4cc20430 0x558d4cc1df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:04.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.097+0000 7fa8e6fda740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x558d4cc23220 con 0x558d4cc17850 2024-04-21T23:19:04.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.097+0000 7fa8e6fda740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x558d4cb36e00 con 0x558d4cc16f30 2024-04-21T23:19:04.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.097+0000 7fa8e6fda740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x558d4cc233a0 con 0x558d4cc20430 2024-04-21T23:19:04.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.097+0000 7fa8e6fda740 10 client.0 _ll_register_callbacks cb 0x558d4cb71eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:19:04.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.097+0000 7fa8e18ae640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d4cc17850 0x558d4cc1d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:04.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.097+0000 7fa8e18ae640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d4cc17850 0x558d4cc1d9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47710/0 (socket says 172.21.15.40:47710) 2024-04-21T23:19:04.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.097+0000 7fa8e18ae640 1 -- 172.21.15.40:0/657059634 learned_addr learned my addr 172.21.15.40:0/657059634 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:04.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.098+0000 7fa8e20af640 1 --2- 172.21.15.40:0/657059634 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558d4cc16f30 0x558d4cc1d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:04.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.098+0000 7fa8e28b0640 1 --2- 172.21.15.40:0/657059634 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558d4cc20430 0x558d4cc1df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:04.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.098+0000 7fa8e18ae640 1 -- 172.21.15.40:0/657059634 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558d4cc20430 msgr2=0x558d4cc1df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:04.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.098+0000 7fa8e18ae640 1 --2- 172.21.15.40:0/657059634 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558d4cc20430 0x558d4cc1df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:04.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.098+0000 7fa8e18ae640 1 -- 172.21.15.40:0/657059634 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558d4cc16f30 msgr2=0x558d4cc1d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:04.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.098+0000 7fa8e18ae640 1 --2- 172.21.15.40:0/657059634 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558d4cc16f30 0x558d4cc1d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:04.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.098+0000 7fa8e18ae640 1 -- 172.21.15.40:0/657059634 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558d4c88f3f0 con 0x558d4cc17850 2024-04-21T23:19:04.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.098+0000 7fa8e28b0640 1 --2- 172.21.15.40:0/657059634 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558d4cc20430 0x558d4cc1df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:19:04.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.098+0000 7fa8e18ae640 1 --2- 172.21.15.40:0/657059634 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d4cc17850 0x558d4cc1d9c0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc00bac0 tx=0x7fa8cc005040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=135f50785fbb41db server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:04.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.098+0000 7fa8c5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-21T23:19:04.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.098+0000 7fa8c5ffb640 1 -- 172.21.15.40:0/657059634 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8cc028060 con 0x558d4cc17850 2024-04-21T23:19:04.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.098+0000 7fa8c5ffb640 1 -- 172.21.15.40:0/657059634 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7fa8cc005530 con 0x558d4cc17850 2024-04-21T23:19:04.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.098+0000 7fa8e6fda740 1 -- 172.21.15.40:0/657059634 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa8cc00be20 con 0x558d4cc17850 2024-04-21T23:19:04.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.098+0000 7fa8c5ffb640 1 -- 172.21.15.40:0/657059634 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fa8cc02f050 con 0x558d4cc17850 2024-04-21T23:19:04.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.098+0000 7fa8e6fda740 10 client.15873 fetch_fsmap learned FSMap version 25 2024-04-21T23:19:04.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.098+0000 7fa8e6fda740 1 -- 172.21.15.40:0/657059634 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x558d4c84f0f0 con 0x558d4cc17850 2024-04-21T23:19:04.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.098+0000 7fa8c5ffb640 1 -- 172.21.15.40:0/657059634 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8cc00be20 con 0x558d4cc17850 2024-04-21T23:19:04.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.099+0000 7fa8c5ffb640 1 -- 172.21.15.40:0/657059634 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fa8cc004c10 con 0x558d4cc17850 2024-04-21T23:19:04.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.099+0000 7fa8e6fda740 10 client.15873 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:19:04.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.099+0000 7fa8e6fda740 10 client.15873 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:19:04.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.099+0000 7fa8e6fda740 20 client.15873 populate_metadata read hostname 'smithi040' 2024-04-21T23:19:04.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.099+0000 7fa8e6fda740 1 --2- 172.21.15.40:0/657059634 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x558d4cc30770 0x558d4cc32b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:04.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.099+0000 7fa8e6fda740 4 client.15873 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:19:04.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.099+0000 7fa8e6fda740 1 -- 172.21.15.40:0/657059634 --> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558d4cb88aa0 con 0x558d4cc30770 2024-04-21T23:19:04.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.099+0000 7fa8e20af640 1 --2- 172.21.15.40:0/657059634 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x558d4cc30770 0x558d4cc32b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:04.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.100+0000 7fa8c5ffb640 10 client.15873 ms_handle_connect on v2:172.21.15.111:6838/3712235548 2024-04-21T23:19:04.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.100+0000 7fa8e20af640 1 --2- 172.21.15.40:0/657059634 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x558d4cc30770 0x558d4cc32b60 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9f4ad94ed8d8acc6 server_cookie=efb1e29b5064db74 in_seq=0 out_seq=0 2024-04-21T23:19:04.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.100+0000 7fa8c5ffb640 1 -- 172.21.15.40:0/657059634 <== mds.0 v2:172.21.15.111:6838/3712235548 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558d4cb88aa0 con 0x558d4cc30770 2024-04-21T23:19:04.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.100+0000 7fa8c5ffb640 10 client.15873 handle_command_reply: tid=0 2024-04-21T23:19:04.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.100+0000 7fa8e6fda740 1 client.15873 shutdown 2024-04-21T23:19:04.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.100+0000 7fa8e6fda740 1 -- 172.21.15.40:0/657059634 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d4cc17850 msgr2=0x558d4cc1d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:04.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.100+0000 7fa8e6fda740 1 --2- 172.21.15.40:0/657059634 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d4cc17850 0x558d4cc1d9c0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc00bac0 tx=0x7fa8cc005040 comp rx=0 tx=0).stop 2024-04-21T23:19:04.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.100+0000 7fa8e6fda740 1 -- 172.21.15.40:0/657059634 shutdown_connections 2024-04-21T23:19:04.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.100+0000 7fa8e6fda740 1 --2- 172.21.15.40:0/657059634 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x558d4cc30770 0x558d4cc32b60 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:04.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.100+0000 7fa8c5ffb640 0 client.15873 ms_handle_reset on v2:172.21.15.111:6838/3712235548 2024-04-21T23:19:04.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.100+0000 7fa8e6fda740 1 --2- 172.21.15.40:0/657059634 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558d4cc20430 0x558d4cc1df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:04.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.100+0000 7fa8e6fda740 1 --2- 172.21.15.40:0/657059634 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d4cc17850 0x558d4cc1d9c0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:04.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.100+0000 7fa8e6fda740 1 --2- 172.21.15.40:0/657059634 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558d4cc16f30 0x558d4cc1d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:04.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.100+0000 7fa8e6fda740 1 -- 172.21.15.40:0/657059634 >> 172.21.15.40:0/657059634 conn(0x558d4cc11660 msgr2=0x558d4cc133b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:04.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.101+0000 7fa8e6fda740 1 -- 172.21.15.40:0/657059634 shutdown_connections 2024-04-21T23:19:04.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.101+0000 7fa8e6fda740 1 -- 172.21.15.40:0/657059634 wait complete. 2024-04-21T23:19:04.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.101+0000 7fa8e6fda740 20 client.15873 trim_cache size 0 max 16384 2024-04-21T23:19:04.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.120+0000 7fa8e6fda740 1 Processor -- start 2024-04-21T23:19:04.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.120+0000 7fa8e6fda740 1 -- start start 2024-04-21T23:19:04.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.120+0000 7fa8e6fda740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d4cc16f30 0x558d4cc1fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:04.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.120+0000 7fa8e6fda740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558d4cc17850 0x558d4cc2c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:04.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.120+0000 7fa8e6fda740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558d4cc20430 0x558d4cc382a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:04.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.120+0000 7fa8e6fda740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa8dc1b1590 con 0x558d4cc16f30 2024-04-21T23:19:04.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.120+0000 7fa8e6fda740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fa8dc1aee60 con 0x558d4cc20430 2024-04-21T23:19:04.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.120+0000 7fa8e6fda740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa8dc1accb0 con 0x558d4cc17850 2024-04-21T23:19:04.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.120+0000 7fa8e20af640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d4cc16f30 0x558d4cc1fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:04.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.120+0000 7fa8e20af640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d4cc16f30 0x558d4cc1fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47720/0 (socket says 172.21.15.40:47720) 2024-04-21T23:19:04.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.120+0000 7fa8e20af640 1 -- 172.21.15.40:0/3638514386 learned_addr learned my addr 172.21.15.40:0/3638514386 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:04.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.120+0000 7fa8e28b0640 1 --2- 172.21.15.40:0/3638514386 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558d4cc20430 0x558d4cc382a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:04.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.120+0000 7fa8e18ae640 1 --2- 172.21.15.40:0/3638514386 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558d4cc17850 0x558d4cc2c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:04.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.120+0000 7fa8e20af640 1 -- 172.21.15.40:0/3638514386 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558d4cc17850 msgr2=0x558d4cc2c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:04.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.120+0000 7fa8e20af640 1 --2- 172.21.15.40:0/3638514386 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558d4cc17850 0x558d4cc2c0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:04.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.120+0000 7fa8e20af640 1 -- 172.21.15.40:0/3638514386 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558d4cc20430 msgr2=0x558d4cc382a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:04.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.120+0000 7fa8e20af640 1 --2- 172.21.15.40:0/3638514386 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558d4cc20430 0x558d4cc382a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:04.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.120+0000 7fa8e20af640 1 -- 172.21.15.40:0/3638514386 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558d4cc23100 con 0x558d4cc16f30 2024-04-21T23:19:04.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.120+0000 7fa8e18ae640 1 --2- 172.21.15.40:0/3638514386 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558d4cc17850 0x558d4cc2c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:19:04.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.121+0000 7fa8e20af640 1 --2- 172.21.15.40:0/3638514386 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d4cc16f30 0x558d4cc1fbb0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fa8d80bd520 tx=0x7fa8d80f0040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7d758b1d6a58338 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:04.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.121+0000 7fa8c7fff640 1 -- 172.21.15.40:0/3638514386 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8d804b040 con 0x558d4cc16f30 2024-04-21T23:19:04.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.121+0000 7fa8c7fff640 1 -- 172.21.15.40:0/3638514386 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8d80bdc90 con 0x558d4cc16f30 2024-04-21T23:19:04.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.121+0000 7fa8c7fff640 1 -- 172.21.15.40:0/3638514386 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8d80b9c00 con 0x558d4cc16f30 2024-04-21T23:19:04.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.122+0000 7fa8e6fda740 1 -- 172.21.15.40:0/3638514386 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d4cc16f30 msgr2=0x558d4cc1fbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:04.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.122+0000 7fa8e6fda740 1 --2- 172.21.15.40:0/3638514386 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d4cc16f30 0x558d4cc1fbb0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fa8d80bd520 tx=0x7fa8d80f0040 comp rx=0 tx=0).stop 2024-04-21T23:19:04.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.122+0000 7fa8e6fda740 1 -- 172.21.15.40:0/3638514386 shutdown_connections 2024-04-21T23:19:04.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.122+0000 7fa8e6fda740 1 --2- 172.21.15.40:0/3638514386 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558d4cc20430 0x558d4cc382a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:04.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.122+0000 7fa8e6fda740 1 --2- 172.21.15.40:0/3638514386 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558d4cc17850 0x558d4cc2c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:04.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.122+0000 7fa8e6fda740 1 --2- 172.21.15.40:0/3638514386 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d4cc16f30 0x558d4cc1fbb0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:04.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.122+0000 7fa8e6fda740 1 -- 172.21.15.40:0/3638514386 >> 172.21.15.40:0/3638514386 conn(0x558d4cc11660 msgr2=0x558d4cc11a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:04.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.122+0000 7fa8e6fda740 1 -- 172.21.15.40:0/3638514386 shutdown_connections 2024-04-21T23:19:04.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.122+0000 7fa8e6fda740 1 -- 172.21.15.40:0/3638514386 wait complete. 2024-04-21T23:19:04.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.122+0000 7fa8e6fda740 1 -- start start 2024-04-21T23:19:04.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.122+0000 7fa8e6fda740 1 Processor -- start 2024-04-21T23:19:04.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.122+0000 7fa8e6fda740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558d4cc16f30 0x558d4cc285f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:04.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.122+0000 7fa8e6fda740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d4cc17850 0x558d4cc28b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:04.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.122+0000 7fa8e6fda740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558d4cc20430 0x558d4cc19c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:04.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.122+0000 7fa8e6fda740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa8dc1ad320 con 0x558d4cc17850 2024-04-21T23:19:04.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.122+0000 7fa8e6fda740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fa8dc1ae790 con 0x558d4cc16f30 2024-04-21T23:19:04.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.122+0000 7fa8e6fda740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa8dc1ad9f0 con 0x558d4cc20430 2024-04-21T23:19:04.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.122+0000 7fa8e6fda740 10 client.0 _ll_register_callbacks cb 0x7fa8d40135c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:19:04.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.122+0000 7fa8e18ae640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d4cc17850 0x558d4cc28b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:04.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.122+0000 7fa8e18ae640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d4cc17850 0x558d4cc28b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47730/0 (socket says 172.21.15.40:47730) 2024-04-21T23:19:04.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.122+0000 7fa8e18ae640 1 -- 172.21.15.40:0/1550797929 learned_addr learned my addr 172.21.15.40:0/1550797929 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:04.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.123+0000 7fa8e20af640 1 --2- 172.21.15.40:0/1550797929 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558d4cc16f30 0x558d4cc285f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:04.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.123+0000 7fa8e28b0640 1 --2- 172.21.15.40:0/1550797929 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558d4cc20430 0x558d4cc19c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:04.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.123+0000 7fa8e18ae640 1 -- 172.21.15.40:0/1550797929 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558d4cc20430 msgr2=0x558d4cc19c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:04.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.123+0000 7fa8e18ae640 1 --2- 172.21.15.40:0/1550797929 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558d4cc20430 0x558d4cc19c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:04.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.123+0000 7fa8e18ae640 1 -- 172.21.15.40:0/1550797929 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558d4cc16f30 msgr2=0x558d4cc285f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:04.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.123+0000 7fa8e18ae640 1 --2- 172.21.15.40:0/1550797929 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558d4cc16f30 0x558d4cc285f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:04.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.123+0000 7fa8e18ae640 1 -- 172.21.15.40:0/1550797929 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558d4cc1a490 con 0x558d4cc17850 2024-04-21T23:19:04.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.123+0000 7fa8e28b0640 1 --2- 172.21.15.40:0/1550797929 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558d4cc20430 0x558d4cc19c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:19:04.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.123+0000 7fa8e18ae640 1 --2- 172.21.15.40:0/1550797929 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d4cc17850 0x558d4cc28b30 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x558d4cc1ea50 tx=0x7fa8cc011440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7011cbb380637796 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:04.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.123+0000 7fa8c5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-21T23:19:04.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.123+0000 7fa8c5ffb640 1 -- 172.21.15.40:0/1550797929 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8cc028040 con 0x558d4cc17850 2024-04-21T23:19:04.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.123+0000 7fa8c5ffb640 1 -- 172.21.15.40:0/1550797929 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7fa8cc01a650 con 0x558d4cc17850 2024-04-21T23:19:04.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.123+0000 7fa8e6fda740 1 -- 172.21.15.40:0/1550797929 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa8d80bdc90 con 0x558d4cc17850 2024-04-21T23:19:04.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.123+0000 7fa8c5ffb640 1 -- 172.21.15.40:0/1550797929 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fa8cc015070 con 0x558d4cc17850 2024-04-21T23:19:04.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.123+0000 7fa8e6fda740 10 client.15879 fetch_fsmap learned FSMap version 25 2024-04-21T23:19:04.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.123+0000 7fa8e6fda740 1 -- 172.21.15.40:0/1550797929 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x558d4cc96730 con 0x558d4cc17850 2024-04-21T23:19:04.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.124+0000 7fa8c5ffb640 1 -- 172.21.15.40:0/1550797929 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8d80bdc90 con 0x558d4cc17850 2024-04-21T23:19:04.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.124+0000 7fa8c5ffb640 1 -- 172.21.15.40:0/1550797929 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fa8cc004710 con 0x558d4cc17850 2024-04-21T23:19:04.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.124+0000 7fa8e6fda740 10 client.15879 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:19:04.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.124+0000 7fa8e6fda740 10 client.15879 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:19:04.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.124+0000 7fa8e6fda740 20 client.15879 populate_metadata read hostname 'smithi040' 2024-04-21T23:19:04.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.124+0000 7fa8e6fda740 1 --2- 172.21.15.40:0/1550797929 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x558d4cc30770 0x558d4cc98e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:04.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.124+0000 7fa8e6fda740 4 client.15879 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T23:19:04.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.124+0000 7fa8e6fda740 1 -- 172.21.15.40:0/1550797929 --> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558d4cb82df0 con 0x558d4cc30770 2024-04-21T23:19:04.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.124+0000 7fa8e20af640 1 --2- 172.21.15.40:0/1550797929 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x558d4cc30770 0x558d4cc98e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:04.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.124+0000 7fa8e20af640 1 --2- 172.21.15.40:0/1550797929 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x558d4cc30770 0x558d4cc98e80 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=75640fcaa302f0b1 server_cookie=4841a46ffa202917 in_seq=0 out_seq=0 2024-04-21T23:19:04.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.124+0000 7fa8c5ffb640 10 client.15879 ms_handle_connect on v2:172.21.15.111:6838/3712235548 2024-04-21T23:19:04.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.125+0000 7fa8c5ffb640 1 -- 172.21.15.40:0/1550797929 <== mds.0 v2:172.21.15.111:6838/3712235548 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x558d4cb82df0 con 0x558d4cc30770 2024-04-21T23:19:04.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.125+0000 7fa8c5ffb640 10 client.15879 handle_command_reply: tid=0 2024-04-21T23:19:04.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.125+0000 7fa8e6fda740 1 client.15879 shutdown 2024-04-21T23:19:04.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.125+0000 7fa8e6fda740 1 -- 172.21.15.40:0/1550797929 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d4cc17850 msgr2=0x558d4cc28b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:04.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.125+0000 7fa8e6fda740 1 --2- 172.21.15.40:0/1550797929 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d4cc17850 0x558d4cc28b30 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x558d4cc1ea50 tx=0x7fa8cc011440 comp rx=0 tx=0).stop 2024-04-21T23:19:04.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.125+0000 7fa8e6fda740 1 -- 172.21.15.40:0/1550797929 shutdown_connections 2024-04-21T23:19:04.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.125+0000 7fa8e6fda740 1 --2- 172.21.15.40:0/1550797929 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x558d4cc30770 0x558d4cc98e80 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:04.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.125+0000 7fa8e6fda740 1 --2- 172.21.15.40:0/1550797929 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558d4cc20430 0x558d4cc19c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:04.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.125+0000 7fa8e6fda740 1 --2- 172.21.15.40:0/1550797929 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d4cc17850 0x558d4cc28b30 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:04.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.125+0000 7fa8c5ffb640 0 client.15879 ms_handle_reset on v2:172.21.15.111:6838/3712235548 2024-04-21T23:19:04.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.125+0000 7fa8e6fda740 1 --2- 172.21.15.40:0/1550797929 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558d4cc16f30 0x558d4cc285f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:04.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.125+0000 7fa8e6fda740 1 -- 172.21.15.40:0/1550797929 >> 172.21.15.40:0/1550797929 conn(0x558d4cc11660 msgr2=0x558d4cc1d420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:04.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.125+0000 7fa8e6fda740 1 -- 172.21.15.40:0/1550797929 shutdown_connections 2024-04-21T23:19:04.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.125+0000 7fa8e6fda740 1 -- 172.21.15.40:0/1550797929 wait complete. 2024-04-21T23:19:04.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.125+0000 7fa8e6fda740 20 client.15879 trim_cache size 0 max 16384 2024-04-21T23:19:04.130 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-21T23:19:04.130 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-21T23:19:04.130 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": { 2024-04-21T23:19:04.130 INFO:teuthology.orchestra.run.smithi040.stdout: "dc2287ba-9ce6-429a-8944-a9bd5517a108": { 2024-04-21T23:19:04.131 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/", 2024-04-21T23:19:04.131 INFO:teuthology.orchestra.run.smithi040.stdout: "tag": "dc2287ba-9ce6-429a-8944-a9bd5517a108", 2024-04-21T23:19:04.131 INFO:teuthology.orchestra.run.smithi040.stdout: "options": "recursive,force" 2024-04-21T23:19:04.131 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:19:04.131 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:19:04.131 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-21T23:19:04.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.125+0000 7fa8e4313640 1 -- 172.21.15.40:0/487021454 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fa8c80a2eb0 msgr2=0x7fa8c80a5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:04.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.125+0000 7fa8e4313640 1 --2- 172.21.15.40:0/487021454 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fa8c80a2eb0 0x7fa8c80a5370 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa8d4005fd0 tx=0x7fa8d40094c0 comp rx=0 tx=0).stop 2024-04-21T23:19:04.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.125+0000 7fa8e4313640 1 -- 172.21.15.40:0/487021454 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa8dc100620 msgr2=0x7fa8dc198020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:04.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.125+0000 7fa8e4313640 1 --2- 172.21.15.40:0/487021454 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa8dc100620 0x7fa8dc198020 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fa8d8009230 tx=0x7fa8d802cc70 comp rx=0 tx=0).stop 2024-04-21T23:19:04.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.125+0000 7fa8e4313640 1 -- 172.21.15.40:0/487021454 shutdown_connections 2024-04-21T23:19:04.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.125+0000 7fa8e4313640 1 --2- 172.21.15.40:0/487021454 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fa8c80a2eb0 0x7fa8c80a5370 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:04.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.125+0000 7fa8e4313640 1 --2- 172.21.15.40:0/487021454 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8dc100fe0 0x7fa8dc198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:04.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.125+0000 7fa8e4313640 1 --2- 172.21.15.40:0/487021454 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa8dc106630 0x7fa8dc19c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:04.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.125+0000 7fa8e4313640 1 --2- 172.21.15.40:0/487021454 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa8dc100620 0x7fa8dc198020 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:04.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.126+0000 7fa8e4313640 1 -- 172.21.15.40:0/487021454 >> 172.21.15.40:0/487021454 conn(0x7fa8dc0776a0 msgr2=0x7fa8dc103190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:04.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.126+0000 7fa8e4313640 1 -- 172.21.15.40:0/487021454 shutdown_connections 2024-04-21T23:19:04.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:04.126+0000 7fa8e4313640 1 -- 172.21.15.40:0/487021454 wait complete. 2024-04-21T23:19:04.134 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:19:04 smithi040 ceph-mon[27984]: scrub queued for path: / 2024-04-21T23:19:04.134 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:19:04 smithi040 ceph-mon[27984]: scrub summary: idle+waiting paths [/] 2024-04-21T23:19:04.134 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:19:04 smithi040 ceph-mon[27984]: scrub summary: active paths [/] 2024-04-21T23:19:04.134 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:19:04 smithi040 ceph-mon[27984]: scrub summary: idle+waiting paths [/] 2024-04-21T23:19:04.135 INFO:tasks.cephfs.filesystem:scrub status for tag:dc2287ba-9ce6-429a-8944-a9bd5517a108 - {'path': '/', 'tag': 'dc2287ba-9ce6-429a-8944-a9bd5517a108', 'options': 'recursive,force'} 2024-04-21T23:19:04.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:19:04 smithi174 ceph-mon[31409]: scrub queued for path: / 2024-04-21T23:19:04.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:19:04 smithi174 ceph-mon[31409]: scrub summary: idle+waiting paths [/] 2024-04-21T23:19:04.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:19:04 smithi174 ceph-mon[31409]: scrub summary: active paths [/] 2024-04-21T23:19:04.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:19:04 smithi174 ceph-mon[31409]: scrub summary: idle+waiting paths [/] 2024-04-21T23:19:04.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:19:04 smithi111 ceph-mon[31085]: scrub queued for path: / 2024-04-21T23:19:04.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:19:04 smithi111 ceph-mon[31085]: scrub summary: idle+waiting paths [/] 2024-04-21T23:19:04.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:19:04 smithi111 ceph-mon[31085]: scrub summary: active paths [/] 2024-04-21T23:19:04.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:19:04 smithi111 ceph-mon[31085]: scrub summary: idle+waiting paths [/] 2024-04-21T23:19:05.344 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:19:05 smithi174 ceph-mon[31409]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:19:05.344 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:19:05 smithi174 ceph-mon[31409]: scrub summary: idle 2024-04-21T23:19:05.345 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:19:05 smithi111 ceph-mon[31085]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:19:05.345 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:19:05 smithi111 ceph-mon[31085]: scrub summary: idle 2024-04-21T23:19:05.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:19:05 smithi040 ceph-mon[27984]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:19:05.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:19:05 smithi040 ceph-mon[27984]: scrub summary: idle 2024-04-21T23:19:07.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:19:07 smithi040 ceph-mon[27984]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-04-21T23:19:07.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:19:07 smithi174 ceph-mon[31409]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-04-21T23:19:07.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:19:07 smithi111 ceph-mon[31085]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-04-21T23:19:09.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:19:09 smithi040 ceph-mon[27984]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 21 op/s 2024-04-21T23:19:09.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:19:09 smithi174 ceph-mon[31409]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 21 op/s 2024-04-21T23:19:09.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:19:09 smithi111 ceph-mon[31085]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 21 op/s 2024-04-21T23:19:11.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:19:11 smithi040 ceph-mon[27984]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-04-21T23:19:11.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:19:11 smithi174 ceph-mon[31409]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-04-21T23:19:11.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:19:11 smithi111 ceph-mon[31085]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-04-21T23:19:12.534 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.532+0000 7f828c3d7640 1 -- 172.21.15.40:0/2536408198 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8284100670 msgr2=0x7f8284100af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:12.534 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.532+0000 7f828c3d7640 1 --2- 172.21.15.40:0/2536408198 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8284100670 0x7f8284100af0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f8274009930 tx=0x7f827402bc30 comp rx=0 tx=0).stop 2024-04-21T23:19:12.534 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.532+0000 7f828c3d7640 1 -- 172.21.15.40:0/2536408198 shutdown_connections 2024-04-21T23:19:12.534 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.532+0000 7f828c3d7640 1 --2- 172.21.15.40:0/2536408198 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8284101030 0x7f828410d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:12.534 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.532+0000 7f828c3d7640 1 --2- 172.21.15.40:0/2536408198 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8284100670 0x7f8284100af0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:12.534 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.533+0000 7f828c3d7640 1 --2- 172.21.15.40:0/2536408198 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8284106630 0x7f8284106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:12.534 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.533+0000 7f828c3d7640 1 -- 172.21.15.40:0/2536408198 >> 172.21.15.40:0/2536408198 conn(0x7f82840777c0 msgr2=0x7f8284077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:12.534 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.533+0000 7f828c3d7640 1 -- 172.21.15.40:0/2536408198 shutdown_connections 2024-04-21T23:19:12.534 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.533+0000 7f828c3d7640 1 -- 172.21.15.40:0/2536408198 wait complete. 2024-04-21T23:19:12.535 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.533+0000 7f828c3d7640 1 Processor -- start 2024-04-21T23:19:12.535 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.533+0000 7f828c3d7640 1 -- start start 2024-04-21T23:19:12.535 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.533+0000 7f828c3d7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8284100670 0x7f8284198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:12.535 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.533+0000 7f828c3d7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8284101030 0x7f82841985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:12.535 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.533+0000 7f828c3d7640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8284106630 0x7f828419c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:12.535 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.534+0000 7f828c3d7640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8284110a40 con 0x7f8284100670 2024-04-21T23:19:12.535 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.534+0000 7f828c3d7640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f82841108c0 con 0x7f8284106630 2024-04-21T23:19:12.535 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.534+0000 7f828c3d7640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8284110bc0 con 0x7f8284101030 2024-04-21T23:19:12.535 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.534+0000 7f828a173640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8284100670 0x7f8284198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:12.535 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.534+0000 7f828a173640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8284100670 0x7f8284198070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39262/0 (socket says 172.21.15.40:39262) 2024-04-21T23:19:12.535 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.534+0000 7f828a173640 1 -- 172.21.15.40:0/2484000753 learned_addr learned my addr 172.21.15.40:0/2484000753 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:12.535 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.534+0000 7f828a974640 1 --2- 172.21.15.40:0/2484000753 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8284106630 0x7f828419c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:12.535 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.534+0000 7f8289972640 1 --2- 172.21.15.40:0/2484000753 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8284101030 0x7f82841985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:12.536 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.534+0000 7f828a173640 1 -- 172.21.15.40:0/2484000753 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8284101030 msgr2=0x7f82841985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:12.536 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.534+0000 7f828a173640 1 --2- 172.21.15.40:0/2484000753 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8284101030 0x7f82841985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:12.536 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.534+0000 7f828a173640 1 -- 172.21.15.40:0/2484000753 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8284106630 msgr2=0x7f828419c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:12.536 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.534+0000 7f828a173640 1 --2- 172.21.15.40:0/2484000753 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8284106630 0x7f828419c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:12.536 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.534+0000 7f828a173640 1 -- 172.21.15.40:0/2484000753 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f828419d010 con 0x7f8284100670 2024-04-21T23:19:12.536 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.534+0000 7f828a173640 1 --2- 172.21.15.40:0/2484000753 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8284100670 0x7f8284198070 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f828000c890 tx=0x7f828000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:12.536 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.534+0000 7f827b7fe640 1 -- 172.21.15.40:0/2484000753 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8280004220 con 0x7f8284100670 2024-04-21T23:19:12.536 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.534+0000 7f828c3d7640 1 -- 172.21.15.40:0/2484000753 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f828419d300 con 0x7f8284100670 2024-04-21T23:19:12.536 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.534+0000 7f827b7fe640 1 -- 172.21.15.40:0/2484000753 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f82800043c0 con 0x7f8284100670 2024-04-21T23:19:12.536 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.534+0000 7f828c3d7640 1 -- 172.21.15.40:0/2484000753 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82841a4b90 con 0x7f8284100670 2024-04-21T23:19:12.536 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.534+0000 7f827b7fe640 1 -- 172.21.15.40:0/2484000753 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8280004a00 con 0x7f8284100670 2024-04-21T23:19:12.537 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.537+0000 7f827b7fe640 1 -- 172.21.15.40:0/2484000753 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 25) ==== 138117+0+0 (secure 0 0 0) 0x7f82800079b0 con 0x7f8284100670 2024-04-21T23:19:12.537 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.538+0000 7f827b7fe640 1 --2- 172.21.15.40:0/2484000753 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f82700a2f50 0x7f82700a5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:12.537 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.538+0000 7f827b7fe640 1 -- 172.21.15.40:0/2484000753 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f82800cfb70 con 0x7f8284100670 2024-04-21T23:19:12.537 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.538+0000 7f8289972640 1 --2- 172.21.15.40:0/2484000753 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f82700a2f50 0x7f82700a5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:12.539 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.539+0000 7f8289972640 1 --2- 172.21.15.40:0/2484000753 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f82700a2f50 0x7f82700a5410 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8274006010 tx=0x7f8274000bc0 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:12.539 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.539+0000 7f828efc1740 1 Processor -- start 2024-04-21T23:19:12.540 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.539+0000 7f828efc1740 1 -- start start 2024-04-21T23:19:12.540 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.539+0000 7f828efc1740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cfd41f2b20 0x55cfd41f2f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:12.540 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.539+0000 7f828efc1740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cfd41f3440 0x55cfd41fbcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:12.540 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.539+0000 7f828efc1740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cfd41fc210 0x55cfd41fe600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:12.540 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.539+0000 7f828efc1740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55cfd4156b70 con 0x55cfd41fc210 2024-04-21T23:19:12.540 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.539+0000 7f828efc1740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55cfd40ff470 con 0x55cfd41f3440 2024-04-21T23:19:12.540 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.539+0000 7f828efc1740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55cfd4152b30 con 0x55cfd41f2b20 2024-04-21T23:19:12.540 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.539+0000 7f828a974640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cfd41fc210 0x55cfd41fe600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:12.540 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.540+0000 7f828a974640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cfd41fc210 0x55cfd41fe600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39276/0 (socket says 172.21.15.40:39276) 2024-04-21T23:19:12.540 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.540+0000 7f828a974640 1 -- 172.21.15.40:0/4141127862 learned_addr learned my addr 172.21.15.40:0/4141127862 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:12.540 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.540+0000 7f8289972640 1 --2- 172.21.15.40:0/4141127862 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cfd41f3440 0x55cfd41fbcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:12.540 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.540+0000 7f828a173640 1 --2- 172.21.15.40:0/4141127862 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cfd41f2b20 0x55cfd41f2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:12.540 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.540+0000 7f828a974640 1 -- 172.21.15.40:0/4141127862 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cfd41f2b20 msgr2=0x55cfd41f2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:12.540 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.540+0000 7f828a974640 1 --2- 172.21.15.40:0/4141127862 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cfd41f2b20 0x55cfd41f2f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:12.541 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.540+0000 7f828a974640 1 -- 172.21.15.40:0/4141127862 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cfd41f3440 msgr2=0x55cfd41fbcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:12.541 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.540+0000 7f828a974640 1 --2- 172.21.15.40:0/4141127862 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cfd41f3440 0x55cfd41fbcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:12.541 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.540+0000 7f828a974640 1 -- 172.21.15.40:0/4141127862 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cfd3e6b210 con 0x55cfd41fc210 2024-04-21T23:19:12.541 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.540+0000 7f828a974640 1 --2- 172.21.15.40:0/4141127862 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cfd41fc210 0x55cfd41fe600 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f827c008dc0 tx=0x7f827c00e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdac9e0a28c8e4e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:12.541 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.540+0000 7f8278bfe640 1 -- 172.21.15.40:0/4141127862 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f827c014070 con 0x55cfd41fc210 2024-04-21T23:19:12.541 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.540+0000 7f8278bfe640 1 -- 172.21.15.40:0/4141127862 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f827c003b70 con 0x55cfd41fc210 2024-04-21T23:19:12.541 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.540+0000 7f8278bfe640 1 -- 172.21.15.40:0/4141127862 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f827c0107e0 con 0x55cfd41fc210 2024-04-21T23:19:12.545 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.541+0000 7f828efc1740 1 -- 172.21.15.40:0/4141127862 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cfd41fc210 msgr2=0x55cfd41fe600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:12.546 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.541+0000 7f828efc1740 1 --2- 172.21.15.40:0/4141127862 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cfd41fc210 0x55cfd41fe600 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f827c008dc0 tx=0x7f827c00e720 comp rx=0 tx=0).stop 2024-04-21T23:19:12.546 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.541+0000 7f828efc1740 1 -- 172.21.15.40:0/4141127862 shutdown_connections 2024-04-21T23:19:12.546 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.541+0000 7f828efc1740 1 --2- 172.21.15.40:0/4141127862 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cfd41f3440 0x55cfd41fbcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:12.546 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.541+0000 7f828efc1740 1 --2- 172.21.15.40:0/4141127862 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cfd41fc210 0x55cfd41fe600 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:12.546 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.541+0000 7f828efc1740 1 --2- 172.21.15.40:0/4141127862 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cfd41f2b20 0x55cfd41f2f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:12.546 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.541+0000 7f828efc1740 1 -- 172.21.15.40:0/4141127862 >> 172.21.15.40:0/4141127862 conn(0x55cfd41ed440 msgr2=0x55cfd41ed820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:12.546 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.541+0000 7f828efc1740 1 -- 172.21.15.40:0/4141127862 shutdown_connections 2024-04-21T23:19:12.546 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.542+0000 7f828efc1740 1 -- 172.21.15.40:0/4141127862 wait complete. 2024-04-21T23:19:12.546 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.542+0000 7f828efc1740 1 -- start start 2024-04-21T23:19:12.546 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.542+0000 7f828efc1740 1 Processor -- start 2024-04-21T23:19:12.546 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.542+0000 7f828efc1740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cfd41f2b20 0x55cfd41f91f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:12.546 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.542+0000 7f828efc1740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cfd41f3440 0x55cfd41f9730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:12.546 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.542+0000 7f828efc1740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cfd41fc210 0x55cfd41f9c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:12.547 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.542+0000 7f828efc1740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55cfd41fee40 con 0x55cfd41f2b20 2024-04-21T23:19:12.547 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.542+0000 7f828efc1740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55cfd4112a10 con 0x55cfd41fc210 2024-04-21T23:19:12.547 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.542+0000 7f828efc1740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55cfd41fefc0 con 0x55cfd41f3440 2024-04-21T23:19:12.547 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.542+0000 7f828efc1740 10 client.0 _ll_register_callbacks cb 0x55cfd414db60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:19:12.547 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.542+0000 7f828a173640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cfd41f2b20 0x55cfd41f91f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:12.547 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.542+0000 7f828a173640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cfd41f2b20 0x55cfd41f91f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39292/0 (socket says 172.21.15.40:39292) 2024-04-21T23:19:12.547 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.542+0000 7f828a173640 1 -- 172.21.15.40:0/848079532 learned_addr learned my addr 172.21.15.40:0/848079532 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:12.547 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.543+0000 7f8289972640 1 --2- 172.21.15.40:0/848079532 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cfd41f3440 0x55cfd41f9730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:12.547 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.543+0000 7f828a974640 1 --2- 172.21.15.40:0/848079532 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cfd41fc210 0x55cfd41f9c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:12.547 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.543+0000 7f828a173640 1 -- 172.21.15.40:0/848079532 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cfd41f3440 msgr2=0x55cfd41f9730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:12.547 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.543+0000 7f828a173640 1 --2- 172.21.15.40:0/848079532 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cfd41f3440 0x55cfd41f9730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:12.547 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.543+0000 7f828a173640 1 -- 172.21.15.40:0/848079532 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cfd41fc210 msgr2=0x55cfd41f9c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:12.547 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.543+0000 7f828a173640 1 --2- 172.21.15.40:0/848079532 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cfd41fc210 0x55cfd41f9c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:12.548 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.543+0000 7f828a173640 1 -- 172.21.15.40:0/848079532 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cfd3f37620 con 0x55cfd41f2b20 2024-04-21T23:19:12.548 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.543+0000 7f8289972640 1 --2- 172.21.15.40:0/848079532 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cfd41f3440 0x55cfd41f9730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:19:12.548 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.543+0000 7f828a974640 1 --2- 172.21.15.40:0/848079532 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cfd41fc210 0x55cfd41f9c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:19:12.548 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.543+0000 7f828a173640 1 --2- 172.21.15.40:0/848079532 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cfd41f2b20 0x55cfd41f91f0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f8280005fd0 tx=0x7f828009c260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a701a5341e21c79 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:12.548 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.543+0000 7f826dffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-21T23:19:12.548 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.543+0000 7f826dffb640 1 -- 172.21.15.40:0/848079532 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f828001d040 con 0x55cfd41f2b20 2024-04-21T23:19:12.548 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.543+0000 7f826dffb640 1 -- 172.21.15.40:0/848079532 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f82800a4480 con 0x55cfd41f2b20 2024-04-21T23:19:12.548 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.543+0000 7f828efc1740 1 -- 172.21.15.40:0/848079532 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f827c003b70 con 0x55cfd41f2b20 2024-04-21T23:19:12.548 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.543+0000 7f826dffb640 1 -- 172.21.15.40:0/848079532 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f82800a4710 con 0x55cfd41f2b20 2024-04-21T23:19:12.548 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.543+0000 7f828efc1740 10 client.15891 fetch_fsmap learned FSMap version 25 2024-04-21T23:19:12.548 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.543+0000 7f828efc1740 1 -- 172.21.15.40:0/848079532 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55cfd420b090 con 0x55cfd41f2b20 2024-04-21T23:19:12.548 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.543+0000 7f826dffb640 1 -- 172.21.15.40:0/848079532 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f827c003b70 con 0x55cfd41f2b20 2024-04-21T23:19:12.548 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.544+0000 7f826dffb640 1 -- 172.21.15.40:0/848079532 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f8280097980 con 0x55cfd41f2b20 2024-04-21T23:19:12.548 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.544+0000 7f828efc1740 10 client.15891 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:19:12.549 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.544+0000 7f828efc1740 10 client.15891 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:19:12.549 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.544+0000 7f828efc1740 20 client.15891 populate_metadata read hostname 'smithi040' 2024-04-21T23:19:12.549 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.544+0000 7f828efc1740 1 --2- 172.21.15.40:0/848079532 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55cfd420c550 0x55cfd420e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:12.549 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.544+0000 7f828efc1740 4 client.15891 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:19:12.549 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.544+0000 7f828efc1740 1 -- 172.21.15.40:0/848079532 --> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cfd41640b0 con 0x55cfd420c550 2024-04-21T23:19:12.549 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.544+0000 7f8289972640 1 --2- 172.21.15.40:0/848079532 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55cfd420c550 0x55cfd420e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:12.549 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.545+0000 7f8289972640 1 --2- 172.21.15.40:0/848079532 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55cfd420c550 0x55cfd420e960 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b2fd3a9752313b15 server_cookie=1e37e762d790ec44 in_seq=0 out_seq=0 2024-04-21T23:19:12.549 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.545+0000 7f826dffb640 10 client.15891 ms_handle_connect on v2:172.21.15.111:6838/3712235548 2024-04-21T23:19:12.549 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.545+0000 7f826dffb640 1 -- 172.21.15.40:0/848079532 <== mds.0 v2:172.21.15.111:6838/3712235548 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cfd41640b0 con 0x55cfd420c550 2024-04-21T23:19:12.549 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.545+0000 7f826dffb640 10 client.15891 handle_command_reply: tid=0 2024-04-21T23:19:12.549 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.545+0000 7f828efc1740 1 client.15891 shutdown 2024-04-21T23:19:12.549 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.546+0000 7f828efc1740 1 -- 172.21.15.40:0/848079532 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cfd41f2b20 msgr2=0x55cfd41f91f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:12.549 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.546+0000 7f828efc1740 1 --2- 172.21.15.40:0/848079532 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cfd41f2b20 0x55cfd41f91f0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f8280005fd0 tx=0x7f828009c260 comp rx=0 tx=0).stop 2024-04-21T23:19:12.550 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.546+0000 7f828efc1740 1 -- 172.21.15.40:0/848079532 shutdown_connections 2024-04-21T23:19:12.550 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.546+0000 7f828efc1740 1 --2- 172.21.15.40:0/848079532 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55cfd420c550 0x55cfd420e960 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:12.550 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.546+0000 7f828efc1740 1 --2- 172.21.15.40:0/848079532 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cfd41fc210 0x55cfd41f9c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:12.550 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.546+0000 7f826dffb640 0 client.15891 ms_handle_reset on v2:172.21.15.111:6838/3712235548 2024-04-21T23:19:12.550 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.546+0000 7f828efc1740 1 --2- 172.21.15.40:0/848079532 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cfd41f3440 0x55cfd41f9730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:12.550 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.546+0000 7f828efc1740 1 --2- 172.21.15.40:0/848079532 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cfd41f2b20 0x55cfd41f91f0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:12.550 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.546+0000 7f828efc1740 1 -- 172.21.15.40:0/848079532 >> 172.21.15.40:0/848079532 conn(0x55cfd41ed440 msgr2=0x55cfd41ef190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:12.550 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.546+0000 7f828efc1740 1 -- 172.21.15.40:0/848079532 shutdown_connections 2024-04-21T23:19:12.550 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.546+0000 7f828efc1740 1 -- 172.21.15.40:0/848079532 wait complete. 2024-04-21T23:19:12.550 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.546+0000 7f828efc1740 20 client.15891 trim_cache size 0 max 16384 2024-04-21T23:19:12.561 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.561+0000 7f828efc1740 1 Processor -- start 2024-04-21T23:19:12.561 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.561+0000 7f828efc1740 1 -- start start 2024-04-21T23:19:12.561 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.561+0000 7f828efc1740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cfd41f2b20 0x55cfd41fb920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:12.562 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.561+0000 7f828efc1740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cfd41f3440 0x55cfd4207d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:12.562 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.561+0000 7f828efc1740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cfd41fc210 0x55cfd4214290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:12.562 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.561+0000 7f828efc1740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f82841b15f0 con 0x55cfd41fc210 2024-04-21T23:19:12.562 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.561+0000 7f828efc1740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f82841aeec0 con 0x55cfd41f3440 2024-04-21T23:19:12.562 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.561+0000 7f828efc1740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f82841acd10 con 0x55cfd41f2b20 2024-04-21T23:19:12.562 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.562+0000 7f828a974640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cfd41fc210 0x55cfd4214290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:12.562 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.562+0000 7f828a974640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cfd41fc210 0x55cfd4214290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39304/0 (socket says 172.21.15.40:39304) 2024-04-21T23:19:12.562 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.562+0000 7f828a974640 1 -- 172.21.15.40:0/576100439 learned_addr learned my addr 172.21.15.40:0/576100439 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:12.562 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.562+0000 7f8289972640 1 --2- 172.21.15.40:0/576100439 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cfd41f3440 0x55cfd4207d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:12.562 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.562+0000 7f828a173640 1 --2- 172.21.15.40:0/576100439 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cfd41f2b20 0x55cfd41fb920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:12.562 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.562+0000 7f828a974640 1 -- 172.21.15.40:0/576100439 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cfd41f2b20 msgr2=0x55cfd41fb920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:12.562 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.562+0000 7f828a974640 1 --2- 172.21.15.40:0/576100439 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cfd41f2b20 0x55cfd41fb920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:12.563 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.562+0000 7f828a974640 1 -- 172.21.15.40:0/576100439 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cfd41f3440 msgr2=0x55cfd4207d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:12.563 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.562+0000 7f828a974640 1 --2- 172.21.15.40:0/576100439 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cfd41f3440 0x55cfd4207d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:12.563 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.562+0000 7f828a974640 1 -- 172.21.15.40:0/576100439 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cfd41ed1a0 con 0x55cfd41fc210 2024-04-21T23:19:12.563 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.562+0000 7f828a974640 1 --2- 172.21.15.40:0/576100439 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cfd41fc210 0x55cfd4214290 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f827c008ce0 tx=0x7f827c008520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=335948b3080354b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:12.563 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.562+0000 7f826ffff640 1 -- 172.21.15.40:0/576100439 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f827c026070 con 0x55cfd41fc210 2024-04-21T23:19:12.563 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.562+0000 7f826ffff640 1 -- 172.21.15.40:0/576100439 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f827c0089d0 con 0x55cfd41fc210 2024-04-21T23:19:12.563 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.562+0000 7f826ffff640 1 -- 172.21.15.40:0/576100439 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f827c014040 con 0x55cfd41fc210 2024-04-21T23:19:12.566 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.563+0000 7f828efc1740 1 -- 172.21.15.40:0/576100439 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cfd41fc210 msgr2=0x55cfd4214290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:12.566 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.563+0000 7f828efc1740 1 --2- 172.21.15.40:0/576100439 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cfd41fc210 0x55cfd4214290 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f827c008ce0 tx=0x7f827c008520 comp rx=0 tx=0).stop 2024-04-21T23:19:12.566 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.563+0000 7f828efc1740 1 -- 172.21.15.40:0/576100439 shutdown_connections 2024-04-21T23:19:12.567 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.563+0000 7f828efc1740 1 --2- 172.21.15.40:0/576100439 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cfd41f3440 0x55cfd4207d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:12.567 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.563+0000 7f828efc1740 1 --2- 172.21.15.40:0/576100439 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cfd41fc210 0x55cfd4214290 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:12.567 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.563+0000 7f828efc1740 1 --2- 172.21.15.40:0/576100439 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cfd41f2b20 0x55cfd41fb920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:12.567 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.563+0000 7f828efc1740 1 -- 172.21.15.40:0/576100439 >> 172.21.15.40:0/576100439 conn(0x55cfd41ed440 msgr2=0x55cfd41ef120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:12.567 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.564+0000 7f828efc1740 1 -- 172.21.15.40:0/576100439 shutdown_connections 2024-04-21T23:19:12.567 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.564+0000 7f828efc1740 1 -- 172.21.15.40:0/576100439 wait complete. 2024-04-21T23:19:12.567 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.564+0000 7f828efc1740 1 -- start start 2024-04-21T23:19:12.567 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.564+0000 7f828efc1740 1 Processor -- start 2024-04-21T23:19:12.567 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.564+0000 7f828efc1740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cfd41f2b20 0x55cfd41f6ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:12.567 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.564+0000 7f828efc1740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cfd41f3440 0x55cfd41f5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:12.567 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.564+0000 7f828efc1740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cfd41fc210 0x55cfd41f59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:12.567 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.564+0000 7f828efc1740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f82841ad380 con 0x55cfd41f2b20 2024-04-21T23:19:12.568 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.564+0000 7f828efc1740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f82841ae7f0 con 0x55cfd41fc210 2024-04-21T23:19:12.568 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.564+0000 7f828efc1740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f82841ada50 con 0x55cfd41f3440 2024-04-21T23:19:12.568 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.565+0000 7f828efc1740 10 client.0 _ll_register_callbacks cb 0x7f828009b870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:19:12.568 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.565+0000 7f828a173640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cfd41f2b20 0x55cfd41f6ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:12.568 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.565+0000 7f828a173640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cfd41f2b20 0x55cfd41f6ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39306/0 (socket says 172.21.15.40:39306) 2024-04-21T23:19:12.568 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.565+0000 7f828a173640 1 -- 172.21.15.40:0/105803246 learned_addr learned my addr 172.21.15.40:0/105803246 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:12.568 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.565+0000 7f828a974640 1 --2- 172.21.15.40:0/105803246 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cfd41fc210 0x55cfd41f59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:12.568 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.565+0000 7f8289972640 1 --2- 172.21.15.40:0/105803246 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cfd41f3440 0x55cfd41f5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:12.568 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.565+0000 7f828a173640 1 -- 172.21.15.40:0/105803246 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cfd41f3440 msgr2=0x55cfd41f5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:12.568 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.565+0000 7f828a173640 1 --2- 172.21.15.40:0/105803246 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cfd41f3440 0x55cfd41f5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:12.568 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.565+0000 7f828a173640 1 -- 172.21.15.40:0/105803246 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cfd41fc210 msgr2=0x55cfd41f59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:12.568 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.565+0000 7f828a173640 1 --2- 172.21.15.40:0/105803246 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cfd41fc210 0x55cfd41f59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:12.568 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.565+0000 7f828a173640 1 -- 172.21.15.40:0/105803246 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cfd41f61a0 con 0x55cfd41f2b20 2024-04-21T23:19:12.569 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.565+0000 7f828a173640 1 --2- 172.21.15.40:0/105803246 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cfd41f2b20 0x55cfd41f6ca0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f828009ba30 tx=0x7f8280005300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32338f22e526ffe9 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:12.569 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.565+0000 7f826dffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-21T23:19:12.569 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.565+0000 7f826dffb640 1 -- 172.21.15.40:0/105803246 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f828001d040 con 0x55cfd41f2b20 2024-04-21T23:19:12.569 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.565+0000 7f826dffb640 1 -- 172.21.15.40:0/105803246 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f828009aa30 con 0x55cfd41f2b20 2024-04-21T23:19:12.569 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.565+0000 7f828efc1740 1 -- 172.21.15.40:0/105803246 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f827c0089d0 con 0x55cfd41f2b20 2024-04-21T23:19:12.569 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.566+0000 7f826dffb640 1 -- 172.21.15.40:0/105803246 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f82800ba4a0 con 0x55cfd41f2b20 2024-04-21T23:19:12.569 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.566+0000 7f828efc1740 10 client.15897 fetch_fsmap learned FSMap version 25 2024-04-21T23:19:12.569 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.566+0000 7f828efc1740 1 -- 172.21.15.40:0/105803246 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55cfd4277a80 con 0x55cfd41f2b20 2024-04-21T23:19:12.569 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.566+0000 7f826dffb640 1 -- 172.21.15.40:0/105803246 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f827c0089d0 con 0x55cfd41f2b20 2024-04-21T23:19:12.569 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.566+0000 7f826dffb640 1 -- 172.21.15.40:0/105803246 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f8280097980 con 0x55cfd41f2b20 2024-04-21T23:19:12.569 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.566+0000 7f828efc1740 10 client.15897 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:19:12.569 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.566+0000 7f828efc1740 10 client.15897 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:19:12.569 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.566+0000 7f828efc1740 20 client.15897 populate_metadata read hostname 'smithi040' 2024-04-21T23:19:12.569 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.566+0000 7f828efc1740 1 --2- 172.21.15.40:0/105803246 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55cfd420c550 0x55cfd427a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:12.570 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.566+0000 7f828efc1740 4 client.15897 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T23:19:12.570 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.566+0000 7f828efc1740 1 -- 172.21.15.40:0/105803246 --> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55cfd415e400 con 0x55cfd420c550 2024-04-21T23:19:12.570 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.566+0000 7f8289972640 1 --2- 172.21.15.40:0/105803246 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55cfd420c550 0x55cfd427a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:12.570 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.567+0000 7f826dffb640 10 client.15897 ms_handle_connect on v2:172.21.15.111:6838/3712235548 2024-04-21T23:19:12.570 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.567+0000 7f8289972640 1 --2- 172.21.15.40:0/105803246 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55cfd420c550 0x55cfd427a1e0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae399562541955e0 server_cookie=3bcfc79b9d3547bf in_seq=0 out_seq=0 2024-04-21T23:19:12.574 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.575+0000 7f826dffb640 1 -- 172.21.15.40:0/105803246 <== mds.0 v2:172.21.15.111:6838/3712235548 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55cfd415e400 con 0x55cfd420c550 2024-04-21T23:19:12.574 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.575+0000 7f826dffb640 10 client.15897 handle_command_reply: tid=0 2024-04-21T23:19:12.574 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.575+0000 7f828efc1740 1 client.15897 shutdown 2024-04-21T23:19:12.574 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.575+0000 7f828efc1740 1 -- 172.21.15.40:0/105803246 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cfd41f2b20 msgr2=0x55cfd41f6ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:12.574 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.575+0000 7f828efc1740 1 --2- 172.21.15.40:0/105803246 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cfd41f2b20 0x55cfd41f6ca0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f828009ba30 tx=0x7f8280005300 comp rx=0 tx=0).stop 2024-04-21T23:19:12.575 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.575+0000 7f828efc1740 1 -- 172.21.15.40:0/105803246 shutdown_connections 2024-04-21T23:19:12.575 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.575+0000 7f828efc1740 1 --2- 172.21.15.40:0/105803246 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55cfd420c550 0x55cfd427a1e0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:12.575 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.575+0000 7f828efc1740 1 --2- 172.21.15.40:0/105803246 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cfd41fc210 0x55cfd41f59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:12.575 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.575+0000 7f828efc1740 1 --2- 172.21.15.40:0/105803246 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cfd41f3440 0x55cfd41f5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:12.575 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.575+0000 7f826dffb640 0 client.15897 ms_handle_reset on v2:172.21.15.111:6838/3712235548 2024-04-21T23:19:12.575 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.575+0000 7f828efc1740 1 --2- 172.21.15.40:0/105803246 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cfd41f2b20 0x55cfd41f6ca0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:12.575 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.575+0000 7f828efc1740 1 -- 172.21.15.40:0/105803246 >> 172.21.15.40:0/105803246 conn(0x55cfd41ed440 msgr2=0x55cfd41f91b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:12.575 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.575+0000 7f828efc1740 1 -- 172.21.15.40:0/105803246 shutdown_connections 2024-04-21T23:19:12.575 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.575+0000 7f828efc1740 1 -- 172.21.15.40:0/105803246 wait complete. 2024-04-21T23:19:12.575 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.575+0000 7f828efc1740 20 client.15897 trim_cache size 0 max 16384 2024-04-21T23:19:12.576 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:{ 2024-04-21T23:19:12.576 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "message": "", 2024-04-21T23:19:12.576 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "return_code": 0 2024-04-21T23:19:12.576 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:} 2024-04-21T23:19:12.576 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.576+0000 7f828c3d7640 1 -- 172.21.15.40:0/2484000753 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f82700a2f50 msgr2=0x7f82700a5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:12.577 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.576+0000 7f828c3d7640 1 --2- 172.21.15.40:0/2484000753 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f82700a2f50 0x7f82700a5410 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8274006010 tx=0x7f8274000bc0 comp rx=0 tx=0).stop 2024-04-21T23:19:12.577 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.576+0000 7f828c3d7640 1 -- 172.21.15.40:0/2484000753 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8284100670 msgr2=0x7f8284198070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:12.577 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.576+0000 7f828c3d7640 1 --2- 172.21.15.40:0/2484000753 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8284100670 0x7f8284198070 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f828000c890 tx=0x7f828000cd50 comp rx=0 tx=0).stop 2024-04-21T23:19:12.577 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.576+0000 7f828c3d7640 1 -- 172.21.15.40:0/2484000753 shutdown_connections 2024-04-21T23:19:12.577 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.576+0000 7f828c3d7640 1 --2- 172.21.15.40:0/2484000753 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f82700a2f50 0x7f82700a5410 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:12.577 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.576+0000 7f828c3d7640 1 --2- 172.21.15.40:0/2484000753 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8284106630 0x7f828419c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:12.577 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.576+0000 7f828c3d7640 1 --2- 172.21.15.40:0/2484000753 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8284101030 0x7f82841985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:12.577 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.576+0000 7f828c3d7640 1 --2- 172.21.15.40:0/2484000753 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8284100670 0x7f8284198070 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:12.577 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.576+0000 7f828c3d7640 1 -- 172.21.15.40:0/2484000753 >> 172.21.15.40:0/2484000753 conn(0x7f82840777c0 msgr2=0x7f82840691d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:12.577 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.576+0000 7f828c3d7640 1 -- 172.21.15.40:0/2484000753 shutdown_connections 2024-04-21T23:19:12.577 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:12.576+0000 7f828c3d7640 1 -- 172.21.15.40:0/2484000753 wait complete. 2024-04-21T23:19:13.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:19:13 smithi040 ceph-mon[27984]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 22 op/s 2024-04-21T23:19:13.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:19:13 smithi174 ceph-mon[31409]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 22 op/s 2024-04-21T23:19:13.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:19:13 smithi111 ceph-mon[31085]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 22 op/s 2024-04-21T23:19:15.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:19:15 smithi040 ceph-mon[27984]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 22 op/s 2024-04-21T23:19:15.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:19:15 smithi174 ceph-mon[31409]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 22 op/s 2024-04-21T23:19:15.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:19:15 smithi111 ceph-mon[31085]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 22 op/s 2024-04-21T23:19:17.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:19:17 smithi040 ceph-mon[27984]: pgmap v29: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 25 KiB/s wr, 26 op/s 2024-04-21T23:19:17.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:19:17 smithi174 ceph-mon[31409]: pgmap v29: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 25 KiB/s wr, 26 op/s 2024-04-21T23:19:17.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:19:17 smithi111 ceph-mon[31085]: pgmap v29: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 25 KiB/s wr, 26 op/s 2024-04-21T23:19:19.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:19:19 smithi040 ceph-mon[27984]: pgmap v30: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-04-21T23:19:19.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:19:19 smithi174 ceph-mon[31409]: pgmap v30: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-04-21T23:19:19.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:19:19 smithi111 ceph-mon[31085]: pgmap v30: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-04-21T23:19:21.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:19:21 smithi040 ceph-mon[27984]: pgmap v31: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.2 KiB/s wr, 10 op/s 2024-04-21T23:19:21.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:19:21 smithi174 ceph-mon[31409]: pgmap v31: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.2 KiB/s wr, 10 op/s 2024-04-21T23:19:21.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:19:21 smithi111 ceph-mon[31085]: pgmap v31: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.2 KiB/s wr, 10 op/s 2024-04-21T23:19:23.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:19:23 smithi040 ceph-mon[27984]: pgmap v32: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.3 KiB/s wr, 7 op/s 2024-04-21T23:19:23.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:19:23 smithi174 ceph-mon[31409]: pgmap v32: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.3 KiB/s wr, 7 op/s 2024-04-21T23:19:23.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:19:23 smithi111 ceph-mon[31085]: pgmap v32: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.3 KiB/s wr, 7 op/s 2024-04-21T23:19:25.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:19:25 smithi040 ceph-mon[27984]: pgmap v33: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.3 KiB/s wr, 7 op/s 2024-04-21T23:19:25.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:19:25 smithi174 ceph-mon[31409]: pgmap v33: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.3 KiB/s wr, 7 op/s 2024-04-21T23:19:25.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:19:25 smithi111 ceph-mon[31085]: pgmap v33: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.3 KiB/s wr, 7 op/s 2024-04-21T23:19:25.669 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.667+0000 7f608bbf7640 1 -- 172.21.15.40:0/4030515841 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60841032f0 msgr2=0x7f608410b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:25.669 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.667+0000 7f608bbf7640 1 --2- 172.21.15.40:0/4030515841 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60841032f0 0x7f608410b7d0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f6074009900 tx=0x7f607402bd60 comp rx=0 tx=0).stop 2024-04-21T23:19:25.669 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.667+0000 7f608bbf7640 1 -- 172.21.15.40:0/4030515841 shutdown_connections 2024-04-21T23:19:25.670 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.667+0000 7f608bbf7640 1 --2- 172.21.15.40:0/4030515841 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f608410bd10 0x7f608410e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:25.670 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.667+0000 7f608bbf7640 1 --2- 172.21.15.40:0/4030515841 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60841032f0 0x7f608410b7d0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:25.670 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.667+0000 7f608bbf7640 1 --2- 172.21.15.40:0/4030515841 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f60841029d0 0x7f6084102db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:25.670 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.667+0000 7f608bbf7640 1 -- 172.21.15.40:0/4030515841 >> 172.21.15.40:0/4030515841 conn(0x7f60840777c0 msgr2=0x7f6084077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:25.670 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.667+0000 7f608bbf7640 1 -- 172.21.15.40:0/4030515841 shutdown_connections 2024-04-21T23:19:25.670 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.667+0000 7f608bbf7640 1 -- 172.21.15.40:0/4030515841 wait complete. 2024-04-21T23:19:25.670 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.668+0000 7f608bbf7640 1 Processor -- start 2024-04-21T23:19:25.670 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.668+0000 7f608bbf7640 1 -- start start 2024-04-21T23:19:25.670 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.668+0000 7f608bbf7640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60841029d0 0x7f6084197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:25.670 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.668+0000 7f608bbf7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f60841032f0 0x7f6084198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:25.671 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.669+0000 7f608bbf7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f608410bd10 0x7f608419c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:25.671 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.669+0000 7f608bbf7640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6084111b60 con 0x7f608410bd10 2024-04-21T23:19:25.671 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.669+0000 7f608bbf7640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f60841119e0 con 0x7f60841029d0 2024-04-21T23:19:25.671 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.669+0000 7f608bbf7640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6084111ce0 con 0x7f60841032f0 2024-04-21T23:19:25.671 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.669+0000 7f608a194640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f608410bd10 0x7f608419c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:25.671 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.669+0000 7f608a194640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f608410bd10 0x7f608419c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37268/0 (socket says 172.21.15.40:37268) 2024-04-21T23:19:25.671 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.669+0000 7f608a194640 1 -- 172.21.15.40:0/3355119027 learned_addr learned my addr 172.21.15.40:0/3355119027 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:25.671 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.669+0000 7f6089993640 1 --2- 172.21.15.40:0/3355119027 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60841029d0 0x7f6084197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:25.671 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.669+0000 7f6089192640 1 --2- 172.21.15.40:0/3355119027 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f60841032f0 0x7f6084198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:25.671 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.669+0000 7f608a194640 1 -- 172.21.15.40:0/3355119027 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f60841032f0 msgr2=0x7f6084198520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:25.671 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.669+0000 7f608a194640 1 --2- 172.21.15.40:0/3355119027 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f60841032f0 0x7f6084198520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:25.671 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.669+0000 7f608a194640 1 -- 172.21.15.40:0/3355119027 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60841029d0 msgr2=0x7f6084197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:25.672 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.669+0000 7f608a194640 1 --2- 172.21.15.40:0/3355119027 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60841029d0 0x7f6084197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:25.672 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.669+0000 7f608a194640 1 -- 172.21.15.40:0/3355119027 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f608419cf80 con 0x7f608410bd10 2024-04-21T23:19:25.672 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.669+0000 7f608a194640 1 --2- 172.21.15.40:0/3355119027 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f608410bd10 0x7f608419c850 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f607c00eaa0 tx=0x7f607c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:25.672 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.670+0000 7f607affd640 1 -- 172.21.15.40:0/3355119027 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f607c00cdb0 con 0x7f608410bd10 2024-04-21T23:19:25.672 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.670+0000 7f607affd640 1 -- 172.21.15.40:0/3355119027 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f607c005020 con 0x7f608410bd10 2024-04-21T23:19:25.672 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.670+0000 7f608bbf7640 1 -- 172.21.15.40:0/3355119027 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f608419d270 con 0x7f608410bd10 2024-04-21T23:19:25.672 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.670+0000 7f607affd640 1 -- 172.21.15.40:0/3355119027 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f607c0116b0 con 0x7f608410bd10 2024-04-21T23:19:25.672 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.670+0000 7f608bbf7640 1 -- 172.21.15.40:0/3355119027 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60841a4b00 con 0x7f608410bd10 2024-04-21T23:19:25.676 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.672+0000 7f608e7e5740 1 Processor -- start 2024-04-21T23:19:25.676 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.673+0000 7f607affd640 1 -- 172.21.15.40:0/3355119027 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 25) ==== 138117+0+0 (secure 0 0 0) 0x7f607c005560 con 0x7f608410bd10 2024-04-21T23:19:25.676 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.673+0000 7f608e7e5740 1 -- start start 2024-04-21T23:19:25.676 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.674+0000 7f608e7e5740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55860e1d6b20 0x55860e1d6f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:25.676 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.674+0000 7f608e7e5740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55860e1d7440 0x55860e1dfcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:25.676 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.674+0000 7f608e7e5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55860e1e0210 0x55860e1e2600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:25.676 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.674+0000 7f608e7e5740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55860e13ab70 con 0x55860e1d7440 2024-04-21T23:19:25.676 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.674+0000 7f608e7e5740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55860e0e3470 con 0x55860e1d6b20 2024-04-21T23:19:25.677 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.674+0000 7f608e7e5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55860e136b30 con 0x55860e1e0210 2024-04-21T23:19:25.677 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.674+0000 7f6089993640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55860e1d7440 0x55860e1dfcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:25.677 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.674+0000 7f6089993640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55860e1d7440 0x55860e1dfcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37272/0 (socket says 172.21.15.40:37272) 2024-04-21T23:19:25.677 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.674+0000 7f6089993640 1 -- 172.21.15.40:0/1232619735 learned_addr learned my addr 172.21.15.40:0/1232619735 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:25.677 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.674+0000 7f6089192640 1 --2- 172.21.15.40:0/1232619735 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55860e1e0210 0x55860e1e2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:25.677 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.674+0000 7f6089192640 1 --2- 172.21.15.40:0/1232619735 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55860e1d6b20 0x55860e1d6f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:25.677 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.674+0000 7f607affd640 1 --2- 172.21.15.40:0/3355119027 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f60700a30a0 0x7f60700a5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:25.677 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.674+0000 7f6089993640 1 -- 172.21.15.40:0/1232619735 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55860e1e0210 msgr2=0x55860e1e2600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:25.677 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.674+0000 7f608a194640 1 --2- 172.21.15.40:0/3355119027 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f60700a30a0 0x7f60700a5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:25.677 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.674+0000 7f6089993640 1 --2- 172.21.15.40:0/1232619735 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55860e1e0210 0x55860e1e2600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:25.677 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.675+0000 7f6089993640 1 -- 172.21.15.40:0/1232619735 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55860e1d6b20 msgr2=0x55860e1d6f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:25.678 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.675+0000 7f6089993640 1 --2- 172.21.15.40:0/1232619735 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55860e1d6b20 0x55860e1d6f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:25.678 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.675+0000 7f6089993640 1 -- 172.21.15.40:0/1232619735 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55860de4f210 con 0x55860e1d7440 2024-04-21T23:19:25.678 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.675+0000 7f6089192640 1 --2- 172.21.15.40:0/1232619735 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55860e1e0210 0x55860e1e2600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:19:25.678 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.675+0000 7f607affd640 1 -- 172.21.15.40:0/3355119027 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f607c0cf800 con 0x7f608410bd10 2024-04-21T23:19:25.678 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.675+0000 7f6089993640 1 --2- 172.21.15.40:0/1232619735 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55860e1d7440 0x55860e1dfcd0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f608000ac10 tx=0x7f608000f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c409cf85b00aa6aa server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:25.678 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.675+0000 7f608a194640 1 --2- 172.21.15.40:0/3355119027 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f60700a30a0 0x7f60700a5560 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f607c0cc9c0 tx=0x7f607c0b6040 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:25.678 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.675+0000 7f606ffff640 1 -- 172.21.15.40:0/1232619735 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f608001d070 con 0x55860e1d7440 2024-04-21T23:19:25.678 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.675+0000 7f606ffff640 1 -- 172.21.15.40:0/1232619735 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6080018b20 con 0x55860e1d7440 2024-04-21T23:19:25.678 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.675+0000 7f606ffff640 1 -- 172.21.15.40:0/1232619735 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6080010af0 con 0x55860e1d7440 2024-04-21T23:19:25.678 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.676+0000 7f608e7e5740 1 -- 172.21.15.40:0/1232619735 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55860e1d7440 msgr2=0x55860e1dfcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:25.678 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.676+0000 7f608e7e5740 1 --2- 172.21.15.40:0/1232619735 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55860e1d7440 0x55860e1dfcd0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f608000ac10 tx=0x7f608000f790 comp rx=0 tx=0).stop 2024-04-21T23:19:25.679 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.676+0000 7f608e7e5740 1 -- 172.21.15.40:0/1232619735 shutdown_connections 2024-04-21T23:19:25.679 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.676+0000 7f608e7e5740 1 --2- 172.21.15.40:0/1232619735 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55860e1e0210 0x55860e1e2600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:25.679 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.676+0000 7f608e7e5740 1 --2- 172.21.15.40:0/1232619735 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55860e1d7440 0x55860e1dfcd0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:25.679 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.676+0000 7f608e7e5740 1 --2- 172.21.15.40:0/1232619735 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55860e1d6b20 0x55860e1d6f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:25.679 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.676+0000 7f608e7e5740 1 -- 172.21.15.40:0/1232619735 >> 172.21.15.40:0/1232619735 conn(0x55860e1d1440 msgr2=0x55860e1d1820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:25.679 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.676+0000 7f608e7e5740 1 -- 172.21.15.40:0/1232619735 shutdown_connections 2024-04-21T23:19:25.679 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.676+0000 7f608e7e5740 1 -- 172.21.15.40:0/1232619735 wait complete. 2024-04-21T23:19:25.679 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.677+0000 7f608e7e5740 1 -- start start 2024-04-21T23:19:25.679 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.677+0000 7f608e7e5740 1 Processor -- start 2024-04-21T23:19:25.684 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.677+0000 7f608e7e5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55860e1d6b20 0x55860e1dd300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:25.684 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.677+0000 7f608e7e5740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55860e1e0210 0x55860e1dd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:25.684 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.677+0000 7f608e7e5740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55860e1ddd80 0x55860e1ec270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:25.684 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.677+0000 7f608e7e5740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55860e1e2e40 con 0x55860e1e0210 2024-04-21T23:19:25.684 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.677+0000 7f608e7e5740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55860e0f6a10 con 0x55860e1ddd80 2024-04-21T23:19:25.684 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.677+0000 7f608e7e5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55860e1e2fc0 con 0x55860e1d6b20 2024-04-21T23:19:25.684 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.677+0000 7f608e7e5740 10 client.0 _ll_register_callbacks cb 0x55860e131b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:19:25.684 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.677+0000 7f6089993640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55860e1e0210 0x55860e1dd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:25.684 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.677+0000 7f6089993640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55860e1e0210 0x55860e1dd840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37274/0 (socket says 172.21.15.40:37274) 2024-04-21T23:19:25.684 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.677+0000 7f6089993640 1 -- 172.21.15.40:0/1997451078 learned_addr learned my addr 172.21.15.40:0/1997451078 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:25.685 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.677+0000 7f6089192640 1 --2- 172.21.15.40:0/1997451078 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55860e1d6b20 0x55860e1dd300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:25.685 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.677+0000 7f6089192640 1 --2- 172.21.15.40:0/1997451078 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55860e1ddd80 0x55860e1ec270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:25.685 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.678+0000 7f6089192640 1 -- 172.21.15.40:0/1997451078 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55860e1d6b20 msgr2=0x55860e1dd300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:25.685 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.678+0000 7f6089192640 1 --2- 172.21.15.40:0/1997451078 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55860e1d6b20 0x55860e1dd300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:25.685 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.678+0000 7f6089192640 1 -- 172.21.15.40:0/1997451078 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55860e1e0210 msgr2=0x55860e1dd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:25.685 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.678+0000 7f6089192640 1 --2- 172.21.15.40:0/1997451078 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55860e1e0210 0x55860e1dd840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:25.685 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.678+0000 7f6089192640 1 -- 172.21.15.40:0/1997451078 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55860df1b620 con 0x55860e1ddd80 2024-04-21T23:19:25.685 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.678+0000 7f6089993640 1 --2- 172.21.15.40:0/1997451078 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55860e1e0210 0x55860e1dd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:19:25.685 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.678+0000 7f6089192640 1 --2- 172.21.15.40:0/1997451078 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55860e1ddd80 0x55860e1ec270 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f607402c2a0 tx=0x7f6074000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f9809841ed199b24 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:25.685 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.678+0000 7f606d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-21T23:19:25.685 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.678+0000 7f606d7fa640 1 -- 172.21.15.40:0/1997451078 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f607402ce60 con 0x55860e1ddd80 2024-04-21T23:19:25.686 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.678+0000 7f608e7e5740 1 -- 172.21.15.40:0/1997451078 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6080018b20 con 0x55860e1ddd80 2024-04-21T23:19:25.686 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.678+0000 7f606d7fa640 1 -- 172.21.15.40:0/1997451078 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f60740037f0 con 0x55860e1ddd80 2024-04-21T23:19:25.686 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.678+0000 7f606d7fa640 1 -- 172.21.15.40:0/1997451078 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f60740039f0 con 0x55860e1ddd80 2024-04-21T23:19:25.686 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.678+0000 7f608e7e5740 10 client.25454 fetch_fsmap learned FSMap version 25 2024-04-21T23:19:25.686 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.678+0000 7f608e7e5740 1 -- 172.21.15.40:0/1997451078 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55860e1ef650 con 0x55860e1ddd80 2024-04-21T23:19:25.686 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.679+0000 7f606d7fa640 1 -- 172.21.15.40:0/1997451078 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6080018b20 con 0x55860e1ddd80 2024-04-21T23:19:25.686 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.679+0000 7f606d7fa640 1 -- 172.21.15.40:0/1997451078 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f607404d070 con 0x55860e1ddd80 2024-04-21T23:19:25.686 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.679+0000 7f608e7e5740 10 client.25454 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:19:25.686 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.679+0000 7f608e7e5740 10 client.25454 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:19:25.686 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.679+0000 7f608e7e5740 20 client.25454 populate_metadata read hostname 'smithi040' 2024-04-21T23:19:25.686 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.679+0000 7f608e7e5740 1 --2- 172.21.15.40:0/1997451078 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55860e1f0ac0 0x55860e1f2eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:25.686 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.679+0000 7f608e7e5740 4 client.25454 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:19:25.687 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.679+0000 7f608e7e5740 1 -- 172.21.15.40:0/1997451078 --> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55860e1480b0 con 0x55860e1f0ac0 2024-04-21T23:19:25.687 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.679+0000 7f6089993640 1 --2- 172.21.15.40:0/1997451078 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55860e1f0ac0 0x55860e1f2eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:25.687 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.680+0000 7f606d7fa640 10 client.25454 ms_handle_connect on v2:172.21.15.111:6838/3712235548 2024-04-21T23:19:25.687 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.680+0000 7f6089993640 1 --2- 172.21.15.40:0/1997451078 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55860e1f0ac0 0x55860e1f2eb0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cd01aa319c95e995 server_cookie=4baa599c8b1a50f3 in_seq=0 out_seq=0 2024-04-21T23:19:25.687 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.680+0000 7f606d7fa640 1 -- 172.21.15.40:0/1997451078 <== mds.0 v2:172.21.15.111:6838/3712235548 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55860e1480b0 con 0x55860e1f0ac0 2024-04-21T23:19:25.687 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.680+0000 7f606d7fa640 10 client.25454 handle_command_reply: tid=0 2024-04-21T23:19:25.687 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.680+0000 7f608e7e5740 1 client.25454 shutdown 2024-04-21T23:19:25.687 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.681+0000 7f608e7e5740 1 -- 172.21.15.40:0/1997451078 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55860e1ddd80 msgr2=0x55860e1ec270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:25.687 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.681+0000 7f608e7e5740 1 --2- 172.21.15.40:0/1997451078 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55860e1ddd80 0x55860e1ec270 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f607402c2a0 tx=0x7f6074000bc0 comp rx=0 tx=0).stop 2024-04-21T23:19:25.687 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.681+0000 7f608e7e5740 1 -- 172.21.15.40:0/1997451078 shutdown_connections 2024-04-21T23:19:25.687 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.681+0000 7f608e7e5740 1 --2- 172.21.15.40:0/1997451078 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55860e1f0ac0 0x55860e1f2eb0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:25.687 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.681+0000 7f608e7e5740 1 --2- 172.21.15.40:0/1997451078 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55860e1ddd80 0x55860e1ec270 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:25.688 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.681+0000 7f608e7e5740 1 --2- 172.21.15.40:0/1997451078 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55860e1e0210 0x55860e1dd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:25.688 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.681+0000 7f606d7fa640 0 client.25454 ms_handle_reset on v2:172.21.15.111:6838/3712235548 2024-04-21T23:19:25.688 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.681+0000 7f608e7e5740 1 --2- 172.21.15.40:0/1997451078 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55860e1d6b20 0x55860e1dd300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:25.688 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.681+0000 7f608e7e5740 1 -- 172.21.15.40:0/1997451078 >> 172.21.15.40:0/1997451078 conn(0x55860e1d1440 msgr2=0x55860e1d3190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:25.688 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.681+0000 7f608e7e5740 1 -- 172.21.15.40:0/1997451078 shutdown_connections 2024-04-21T23:19:25.688 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.681+0000 7f608e7e5740 1 -- 172.21.15.40:0/1997451078 wait complete. 2024-04-21T23:19:25.688 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.681+0000 7f608e7e5740 20 client.25454 trim_cache size 0 max 16384 2024-04-21T23:19:25.696 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.696+0000 7f608e7e5740 1 Processor -- start 2024-04-21T23:19:25.696 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.696+0000 7f608e7e5740 1 -- start start 2024-04-21T23:19:25.696 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.696+0000 7f608e7e5740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55860e1d6b20 0x55860e1cc450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:25.697 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.696+0000 7f608e7e5740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55860e1ddd80 0x55860e1f8700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:25.697 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.696+0000 7f608e7e5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55860e1e0210 0x55860e1f8c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:25.697 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.696+0000 7f608e7e5740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f60841b14e0 con 0x55860e1ddd80 2024-04-21T23:19:25.697 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.696+0000 7f608e7e5740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f60841aedb0 con 0x55860e1d6b20 2024-04-21T23:19:25.697 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.696+0000 7f608e7e5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f60841acc00 con 0x55860e1e0210 2024-04-21T23:19:25.697 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.696+0000 7f6089993640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55860e1ddd80 0x55860e1f8700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:25.697 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.696+0000 7f6089993640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55860e1ddd80 0x55860e1f8700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37276/0 (socket says 172.21.15.40:37276) 2024-04-21T23:19:25.697 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.696+0000 7f6089993640 1 -- 172.21.15.40:0/105973821 learned_addr learned my addr 172.21.15.40:0/105973821 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:25.697 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.697+0000 7f6089192640 1 --2- 172.21.15.40:0/105973821 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55860e1d6b20 0x55860e1cc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:25.697 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.697+0000 7f6089192640 1 --2- 172.21.15.40:0/105973821 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55860e1e0210 0x55860e1f8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:25.697 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.697+0000 7f6089993640 1 -- 172.21.15.40:0/105973821 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55860e1e0210 msgr2=0x55860e1f8c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:25.697 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.697+0000 7f6089993640 1 --2- 172.21.15.40:0/105973821 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55860e1e0210 0x55860e1f8c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:25.697 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.697+0000 7f6089993640 1 -- 172.21.15.40:0/105973821 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55860e1d6b20 msgr2=0x55860e1cc450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:25.698 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.697+0000 7f6089993640 1 --2- 172.21.15.40:0/105973821 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55860e1d6b20 0x55860e1cc450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:25.698 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.697+0000 7f6089993640 1 -- 172.21.15.40:0/105973821 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55860e1dc2b0 con 0x55860e1ddd80 2024-04-21T23:19:25.698 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.697+0000 7f6089192640 1 --2- 172.21.15.40:0/105973821 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55860e1d6b20 0x55860e1cc450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:19:25.698 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.697+0000 7f6089993640 1 --2- 172.21.15.40:0/105973821 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55860e1ddd80 0x55860e1f8700 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f608000ac10 tx=0x7f608000a2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3469bb49c167ba74 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:25.698 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.697+0000 7f606f7fe640 1 -- 172.21.15.40:0/105973821 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f608001d040 con 0x55860e1ddd80 2024-04-21T23:19:25.698 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.697+0000 7f606f7fe640 1 -- 172.21.15.40:0/105973821 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6080030070 con 0x55860e1ddd80 2024-04-21T23:19:25.698 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.697+0000 7f606f7fe640 1 -- 172.21.15.40:0/105973821 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6080004660 con 0x55860e1ddd80 2024-04-21T23:19:25.703 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.698+0000 7f608e7e5740 1 -- 172.21.15.40:0/105973821 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55860e1ddd80 msgr2=0x55860e1f8700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:25.703 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.698+0000 7f608e7e5740 1 --2- 172.21.15.40:0/105973821 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55860e1ddd80 0x55860e1f8700 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f608000ac10 tx=0x7f608000a2a0 comp rx=0 tx=0).stop 2024-04-21T23:19:25.703 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.698+0000 7f608e7e5740 1 -- 172.21.15.40:0/105973821 shutdown_connections 2024-04-21T23:19:25.703 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.698+0000 7f608e7e5740 1 --2- 172.21.15.40:0/105973821 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55860e1e0210 0x55860e1f8c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:25.703 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.698+0000 7f608e7e5740 1 --2- 172.21.15.40:0/105973821 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55860e1ddd80 0x55860e1f8700 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:25.703 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.698+0000 7f608e7e5740 1 --2- 172.21.15.40:0/105973821 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55860e1d6b20 0x55860e1cc450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:25.703 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.698+0000 7f608e7e5740 1 -- 172.21.15.40:0/105973821 >> 172.21.15.40:0/105973821 conn(0x55860e1d1440 msgr2=0x55860e1d30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:25.703 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.698+0000 7f608e7e5740 1 -- 172.21.15.40:0/105973821 shutdown_connections 2024-04-21T23:19:25.703 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.699+0000 7f608e7e5740 1 -- 172.21.15.40:0/105973821 wait complete. 2024-04-21T23:19:25.703 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.699+0000 7f608e7e5740 1 -- start start 2024-04-21T23:19:25.703 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.699+0000 7f608e7e5740 1 Processor -- start 2024-04-21T23:19:25.704 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.700+0000 7f608e7e5740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55860e1d6b20 0x55860e1e8210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:25.704 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.700+0000 7f608e7e5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55860e1ddd80 0x55860e1e8750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:25.704 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.700+0000 7f608e7e5740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55860e1e0210 0x55860e1d9b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:25.704 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.700+0000 7f608e7e5740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f60841ad270 con 0x55860e1e0210 2024-04-21T23:19:25.704 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.700+0000 7f608e7e5740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f60841ae6e0 con 0x55860e1d6b20 2024-04-21T23:19:25.704 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.700+0000 7f608e7e5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f60841ad940 con 0x55860e1ddd80 2024-04-21T23:19:25.704 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.700+0000 7f6089192640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55860e1e0210 0x55860e1d9b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:25.704 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.700+0000 7f6089993640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55860e1ddd80 0x55860e1e8750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:25.704 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.700+0000 7f6089192640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55860e1e0210 0x55860e1d9b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37292/0 (socket says 172.21.15.40:37292) 2024-04-21T23:19:25.704 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.700+0000 7f6089192640 1 -- 172.21.15.40:0/2270484193 learned_addr learned my addr 172.21.15.40:0/2270484193 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:25.704 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.700+0000 7f6089192640 1 --2- 172.21.15.40:0/2270484193 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55860e1d6b20 0x55860e1e8210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:25.704 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.700+0000 7f608e7e5740 10 client.0 _ll_register_callbacks cb 0x7f607402caf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:19:25.704 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.701+0000 7f6089192640 1 -- 172.21.15.40:0/2270484193 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55860e1ddd80 msgr2=0x55860e1e8750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:25.704 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.701+0000 7f6089192640 1 --2- 172.21.15.40:0/2270484193 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55860e1ddd80 0x55860e1e8750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:25.705 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.701+0000 7f6089192640 1 -- 172.21.15.40:0/2270484193 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55860e1d6b20 msgr2=0x55860e1e8210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:25.705 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.701+0000 7f6089192640 1 --2- 172.21.15.40:0/2270484193 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55860e1d6b20 0x55860e1e8210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:25.705 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.701+0000 7f6089192640 1 -- 172.21.15.40:0/2270484193 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55860e1da360 con 0x55860e1e0210 2024-04-21T23:19:25.705 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.701+0000 7f6089192640 1 --2- 172.21.15.40:0/2270484193 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55860e1e0210 0x55860e1d9b30 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f607402cc20 tx=0x7f607402c700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87258a7746ba13c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:25.705 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.701+0000 7f606d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-21T23:19:25.705 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.701+0000 7f606d7fa640 1 -- 172.21.15.40:0/2270484193 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f607405c060 con 0x55860e1e0210 2024-04-21T23:19:25.705 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.701+0000 7f606d7fa640 1 -- 172.21.15.40:0/2270484193 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f6074003d90 con 0x55860e1e0210 2024-04-21T23:19:25.705 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.701+0000 7f608e7e5740 1 -- 172.21.15.40:0/2270484193 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6080030070 con 0x55860e1e0210 2024-04-21T23:19:25.705 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.701+0000 7f606d7fa640 1 -- 172.21.15.40:0/2270484193 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f6074043450 con 0x55860e1e0210 2024-04-21T23:19:25.705 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.701+0000 7f608e7e5740 10 client.15915 fetch_fsmap learned FSMap version 25 2024-04-21T23:19:25.705 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.702+0000 7f608e7e5740 1 -- 172.21.15.40:0/2270484193 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55860e256b30 con 0x55860e1e0210 2024-04-21T23:19:25.705 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.702+0000 7f606d7fa640 1 -- 172.21.15.40:0/2270484193 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6080030070 con 0x55860e1e0210 2024-04-21T23:19:25.706 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.703+0000 7f606d7fa640 1 -- 172.21.15.40:0/2270484193 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f607404d040 con 0x55860e1e0210 2024-04-21T23:19:25.706 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.703+0000 7f608e7e5740 10 client.15915 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:19:25.706 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.703+0000 7f608e7e5740 10 client.15915 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:19:25.706 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.703+0000 7f608e7e5740 20 client.15915 populate_metadata read hostname 'smithi040' 2024-04-21T23:19:25.706 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.703+0000 7f608e7e5740 1 --2- 172.21.15.40:0/2270484193 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55860e2571b0 0x55860e2595a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:25.706 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.703+0000 7f608e7e5740 4 client.15915 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T23:19:25.706 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.703+0000 7f608e7e5740 1 -- 172.21.15.40:0/2270484193 --> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55860e142400 con 0x55860e2571b0 2024-04-21T23:19:25.706 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.704+0000 7f6089993640 1 --2- 172.21.15.40:0/2270484193 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55860e2571b0 0x55860e2595a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:25.708 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.704+0000 7f606d7fa640 10 client.15915 ms_handle_connect on v2:172.21.15.111:6838/3712235548 2024-04-21T23:19:25.708 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.704+0000 7f6089993640 1 --2- 172.21.15.40:0/2270484193 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55860e2571b0 0x55860e2595a0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5f5420181abf38f7 server_cookie=d1164c44597c8043 in_seq=0 out_seq=0 2024-04-21T23:19:25.708 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.708+0000 7f606d7fa640 1 -- 172.21.15.40:0/2270484193 <== mds.0 v2:172.21.15.111:6838/3712235548 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55860e142400 con 0x55860e2571b0 2024-04-21T23:19:25.708 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.708+0000 7f606d7fa640 10 client.15915 handle_command_reply: tid=0 2024-04-21T23:19:25.708 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.708+0000 7f608e7e5740 1 client.15915 shutdown 2024-04-21T23:19:25.708 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.708+0000 7f608e7e5740 1 -- 172.21.15.40:0/2270484193 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55860e1e0210 msgr2=0x55860e1d9b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:25.708 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.708+0000 7f608e7e5740 1 --2- 172.21.15.40:0/2270484193 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55860e1e0210 0x55860e1d9b30 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f607402cc20 tx=0x7f607402c700 comp rx=0 tx=0).stop 2024-04-21T23:19:25.708 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.708+0000 7f608e7e5740 1 -- 172.21.15.40:0/2270484193 shutdown_connections 2024-04-21T23:19:25.708 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.708+0000 7f608e7e5740 1 --2- 172.21.15.40:0/2270484193 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55860e2571b0 0x55860e2595a0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:25.708 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.708+0000 7f608e7e5740 1 --2- 172.21.15.40:0/2270484193 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55860e1e0210 0x55860e1d9b30 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:25.708 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.708+0000 7f608e7e5740 1 --2- 172.21.15.40:0/2270484193 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55860e1ddd80 0x55860e1e8750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:25.709 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.708+0000 7f606d7fa640 0 client.15915 ms_handle_reset on v2:172.21.15.111:6838/3712235548 2024-04-21T23:19:25.709 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.708+0000 7f608e7e5740 1 --2- 172.21.15.40:0/2270484193 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55860e1d6b20 0x55860e1e8210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:25.709 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.708+0000 7f608e7e5740 1 -- 172.21.15.40:0/2270484193 >> 172.21.15.40:0/2270484193 conn(0x55860e1d1440 msgr2=0x55860e1dcb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:25.709 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.708+0000 7f608e7e5740 1 -- 172.21.15.40:0/2270484193 shutdown_connections 2024-04-21T23:19:25.709 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.708+0000 7f608e7e5740 1 -- 172.21.15.40:0/2270484193 wait complete. 2024-04-21T23:19:25.709 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.708+0000 7f608e7e5740 20 client.15915 trim_cache size 0 max 16384 2024-04-21T23:19:25.709 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:{ 2024-04-21T23:19:25.709 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "message": "", 2024-04-21T23:19:25.709 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "return_code": 0 2024-04-21T23:19:25.710 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:} 2024-04-21T23:19:25.710 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.709+0000 7f608bbf7640 1 -- 172.21.15.40:0/3355119027 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f60700a30a0 msgr2=0x7f60700a5560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:25.710 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.709+0000 7f608bbf7640 1 --2- 172.21.15.40:0/3355119027 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f60700a30a0 0x7f60700a5560 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f607c0cc9c0 tx=0x7f607c0b6040 comp rx=0 tx=0).stop 2024-04-21T23:19:25.711 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.709+0000 7f608bbf7640 1 -- 172.21.15.40:0/3355119027 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f608410bd10 msgr2=0x7f608419c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:25.711 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.709+0000 7f608bbf7640 1 --2- 172.21.15.40:0/3355119027 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f608410bd10 0x7f608419c850 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f607c00eaa0 tx=0x7f607c00ef60 comp rx=0 tx=0).stop 2024-04-21T23:19:25.711 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.709+0000 7f608bbf7640 1 -- 172.21.15.40:0/3355119027 shutdown_connections 2024-04-21T23:19:25.711 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.709+0000 7f608bbf7640 1 --2- 172.21.15.40:0/3355119027 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f60700a30a0 0x7f60700a5560 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:25.711 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.709+0000 7f608bbf7640 1 --2- 172.21.15.40:0/3355119027 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f608410bd10 0x7f608419c850 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:25.711 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.709+0000 7f608bbf7640 1 --2- 172.21.15.40:0/3355119027 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f60841032f0 0x7f6084198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:25.711 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.709+0000 7f608bbf7640 1 --2- 172.21.15.40:0/3355119027 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60841029d0 0x7f6084197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:25.711 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.709+0000 7f608bbf7640 1 -- 172.21.15.40:0/3355119027 >> 172.21.15.40:0/3355119027 conn(0x7f60840777c0 msgr2=0x7f60840fd710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:25.711 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.709+0000 7f608bbf7640 1 -- 172.21.15.40:0/3355119027 shutdown_connections 2024-04-21T23:19:25.711 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:25.709+0000 7f608bbf7640 1 -- 172.21.15.40:0/3355119027 wait complete. 2024-04-21T23:19:27.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:19:27 smithi040 ceph-mon[27984]: pgmap v34: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.3 KiB/s wr, 9 op/s 2024-04-21T23:19:27.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:19:27 smithi174 ceph-mon[31409]: pgmap v34: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.3 KiB/s wr, 9 op/s 2024-04-21T23:19:27.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:19:27 smithi111 ceph-mon[31085]: pgmap v34: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.3 KiB/s wr, 9 op/s 2024-04-21T23:19:29.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:19:29 smithi040 ceph-mon[27984]: pgmap v35: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:19:29.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:19:29 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:19:29.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:19:29 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:19:29.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:19:29 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:19:29.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:19:29 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:19:29.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:19:29 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:19:29.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:19:29 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:19:29.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:19:29 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:19:29.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:19:29 smithi174 ceph-mon[31409]: pgmap v35: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:19:29.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:19:29 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:19:29.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:19:29 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:19:29.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:19:29 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:19:29.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:19:29 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:19:29.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:19:29 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:19:29.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:19:29 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:19:29.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:19:29 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:19:29.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:19:29 smithi111 ceph-mon[31085]: pgmap v35: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:19:29.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:19:29 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:19:29.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:19:29 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:19:29.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:19:29 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:19:29.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:19:29 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:19:29.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:19:29 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:19:29.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:19:29 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:19:29.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:19:29 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:19:30.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:19:30 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:19:30.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:19:30 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:19:30.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:19:30 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:19:30.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:19:30 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:19:30.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:19:30 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:19:30.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:19:30 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:19:30.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:19:30 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:19:30.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:19:30 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:19:30.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:19:30 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:19:31.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:19:31 smithi040 ceph-mon[27984]: pgmap v36: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-21T23:19:31.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:19:31 smithi174 ceph-mon[31409]: pgmap v36: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-21T23:19:31.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:19:31 smithi111 ceph-mon[31085]: pgmap v36: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-21T23:19:33.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:19:33 smithi040 ceph-mon[27984]: pgmap v37: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-21T23:19:33.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:19:33 smithi174 ceph-mon[31409]: pgmap v37: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-21T23:19:33.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:19:33 smithi111 ceph-mon[31085]: pgmap v37: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-21T23:19:34.136 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T23:19:34.136 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T23:19:34.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.246+0000 7f931800e640 1 -- 172.21.15.40:0/1027702967 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f931010bd00 msgr2=0x7f931010e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:34.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.246+0000 7f931800e640 1 --2- 172.21.15.40:0/1027702967 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f931010bd00 0x7f931010e1a0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f9308009ca0 tx=0x7f930802c230 comp rx=0 tx=0).stop 2024-04-21T23:19:34.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.246+0000 7f931800e640 1 -- 172.21.15.40:0/1027702967 shutdown_connections 2024-04-21T23:19:34.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.246+0000 7f931800e640 1 --2- 172.21.15.40:0/1027702967 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f931010bd00 0x7f931010e1a0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.246+0000 7f931800e640 1 --2- 172.21.15.40:0/1027702967 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f93101032e0 0x7f931010b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.246+0000 7f931800e640 1 --2- 172.21.15.40:0/1027702967 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93101029c0 0x7f9310102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.246+0000 7f931800e640 1 -- 172.21.15.40:0/1027702967 >> 172.21.15.40:0/1027702967 conn(0x7f9310077740 msgr2=0x7f9310077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:34.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.246+0000 7f931800e640 1 -- 172.21.15.40:0/1027702967 shutdown_connections 2024-04-21T23:19:34.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.246+0000 7f931800e640 1 -- 172.21.15.40:0/1027702967 wait complete. 2024-04-21T23:19:34.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.247+0000 7f931800e640 1 Processor -- start 2024-04-21T23:19:34.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.247+0000 7f931800e640 1 -- start start 2024-04-21T23:19:34.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.248+0000 7f931800e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f93101029c0 0x7f9310198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:34.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.248+0000 7f931800e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93101032e0 0x7f9310198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:34.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.248+0000 7f931800e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f931010bd00 0x7f931019c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:34.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.248+0000 7f931800e640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9310111b00 con 0x7f931010bd00 2024-04-21T23:19:34.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.248+0000 7f931800e640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f9310111980 con 0x7f93101032e0 2024-04-21T23:19:34.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.248+0000 7f931800e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9310111c80 con 0x7f93101029c0 2024-04-21T23:19:34.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.248+0000 7f9315daa640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f93101029c0 0x7f9310198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:34.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.248+0000 7f93165ab640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f931010bd00 0x7f931019c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:34.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.248+0000 7f9315daa640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f93101029c0 0x7f9310198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.40:34648/0 (socket says 172.21.15.40:34648) 2024-04-21T23:19:34.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.248+0000 7f93155a9640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93101032e0 0x7f9310198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:34.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.248+0000 7f9315daa640 1 -- 172.21.15.40:0/2077034741 learned_addr learned my addr 172.21.15.40:0/2077034741 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:34.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.248+0000 7f9315daa640 1 -- 172.21.15.40:0/2077034741 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93101032e0 msgr2=0x7f9310198540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:34.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.248+0000 7f9315daa640 1 --2- 172.21.15.40:0/2077034741 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93101032e0 0x7f9310198540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.248+0000 7f9315daa640 1 -- 172.21.15.40:0/2077034741 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f931010bd00 msgr2=0x7f931019c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:34.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.248+0000 7f9315daa640 1 --2- 172.21.15.40:0/2077034741 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f931010bd00 0x7f931019c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.248+0000 7f9315daa640 1 -- 172.21.15.40:0/2077034741 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f931019cff0 con 0x7f93101029c0 2024-04-21T23:19:34.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.248+0000 7f93165ab640 1 --2- 172.21.15.40:0/2077034741 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f931010bd00 0x7f931019c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:19:34.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.249+0000 7f9315daa640 1 --2- 172.21.15.40:0/2077034741 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f93101029c0 0x7f9310198000 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f930c00e9f0 tx=0x7f930c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:34.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.249+0000 7f9306ffd640 1 -- 172.21.15.40:0/2077034741 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f930c00cd80 con 0x7f93101029c0 2024-04-21T23:19:34.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.249+0000 7f931800e640 1 -- 172.21.15.40:0/2077034741 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f931019d2e0 con 0x7f93101029c0 2024-04-21T23:19:34.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.249+0000 7f9306ffd640 1 -- 172.21.15.40:0/2077034741 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f930c010910 con 0x7f93101029c0 2024-04-21T23:19:34.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.249+0000 7f931800e640 1 -- 172.21.15.40:0/2077034741 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9310100540 con 0x7f93101029c0 2024-04-21T23:19:34.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.249+0000 7f9306ffd640 1 -- 172.21.15.40:0/2077034741 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f930c0188f0 con 0x7f93101029c0 2024-04-21T23:19:34.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.251+0000 7f9306ffd640 1 -- 172.21.15.40:0/2077034741 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 25) ==== 138117+0+0 (secure 0 0 0) 0x7f930c018bb0 con 0x7f93101029c0 2024-04-21T23:19:34.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.252+0000 7f9306ffd640 1 --2- 172.21.15.40:0/2077034741 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f92fc0a2f30 0x7f92fc0a53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:34.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.252+0000 7f9306ffd640 1 -- 172.21.15.40:0/2077034741 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f930c0cfcc0 con 0x7f93101029c0 2024-04-21T23:19:34.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.252+0000 7f93155a9640 1 --2- 172.21.15.40:0/2077034741 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f92fc0a2f30 0x7f92fc0a53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:34.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.252+0000 7f931abfc740 1 Processor -- start 2024-04-21T23:19:34.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.252+0000 7f931abfc740 1 -- start start 2024-04-21T23:19:34.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.253+0000 7f93155a9640 1 --2- 172.21.15.40:0/2077034741 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f92fc0a2f30 0x7f92fc0a53f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f93100fe570 tx=0x7f9300005f90 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:34.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.253+0000 7f931abfc740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5568ebc65f30 0x5568ebc66310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:34.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.253+0000 7f931abfc740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568ebc66850 0x5568ebc6eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:34.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.253+0000 7f931abfc740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5568ebc6f430 0x5568ebc71820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:34.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.253+0000 7f931abfc740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5568ebbca560 con 0x5568ebc66850 2024-04-21T23:19:34.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.253+0000 7f931abfc740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5568ebb72730 con 0x5568ebc6f430 2024-04-21T23:19:34.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.253+0000 7f93155a9640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568ebc66850 0x5568ebc6eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:34.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.253+0000 7f931abfc740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5568ebbc6520 con 0x5568ebc65f30 2024-04-21T23:19:34.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.253+0000 7f93155a9640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568ebc66850 0x5568ebc6eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48122/0 (socket says 172.21.15.40:48122) 2024-04-21T23:19:34.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.253+0000 7f9315daa640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5568ebc65f30 0x5568ebc66310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:34.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.253+0000 7f93155a9640 1 -- 172.21.15.40:0/2528982256 learned_addr learned my addr 172.21.15.40:0/2528982256 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:34.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.253+0000 7f93165ab640 1 --2- 172.21.15.40:0/2528982256 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5568ebc6f430 0x5568ebc71820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:34.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.254+0000 7f9315daa640 1 -- 172.21.15.40:0/2528982256 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5568ebc6f430 msgr2=0x5568ebc71820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:34.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.254+0000 7f9315daa640 1 --2- 172.21.15.40:0/2528982256 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5568ebc6f430 0x5568ebc71820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.254+0000 7f9315daa640 1 -- 172.21.15.40:0/2528982256 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568ebc66850 msgr2=0x5568ebc6eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:34.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.254+0000 7f9315daa640 1 --2- 172.21.15.40:0/2528982256 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568ebc66850 0x5568ebc6eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.254+0000 7f9315daa640 1 -- 172.21.15.40:0/2528982256 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5568eb9a87d0 con 0x5568ebc65f30 2024-04-21T23:19:34.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.254+0000 7f93155a9640 1 --2- 172.21.15.40:0/2528982256 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568ebc66850 0x5568ebc6eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:19:34.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.254+0000 7f9315daa640 1 --2- 172.21.15.40:0/2528982256 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5568ebc65f30 0x5568ebc66310 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f930c0cce20 tx=0x7f930c09bca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=11f3fcd10ae65b7b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:34.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.254+0000 7f92fbfff640 1 -- 172.21.15.40:0/2528982256 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f930c099fa0 con 0x5568ebc65f30 2024-04-21T23:19:34.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.254+0000 7f92fbfff640 1 -- 172.21.15.40:0/2528982256 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f930c098310 con 0x5568ebc65f30 2024-04-21T23:19:34.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.254+0000 7f92fbfff640 1 -- 172.21.15.40:0/2528982256 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f930c0979f0 con 0x5568ebc65f30 2024-04-21T23:19:34.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.255+0000 7f931abfc740 1 -- 172.21.15.40:0/2528982256 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5568ebc65f30 msgr2=0x5568ebc66310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:34.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.255+0000 7f931abfc740 1 --2- 172.21.15.40:0/2528982256 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5568ebc65f30 0x5568ebc66310 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f930c0cce20 tx=0x7f930c09bca0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.255+0000 7f931abfc740 1 -- 172.21.15.40:0/2528982256 shutdown_connections 2024-04-21T23:19:34.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.255+0000 7f931abfc740 1 --2- 172.21.15.40:0/2528982256 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5568ebc6f430 0x5568ebc71820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.255+0000 7f931abfc740 1 --2- 172.21.15.40:0/2528982256 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568ebc66850 0x5568ebc6eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.255+0000 7f931abfc740 1 --2- 172.21.15.40:0/2528982256 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5568ebc65f30 0x5568ebc66310 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.255+0000 7f931abfc740 1 -- 172.21.15.40:0/2528982256 >> 172.21.15.40:0/2528982256 conn(0x5568ebc60660 msgr2=0x5568ebc60a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:34.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.256+0000 7f931abfc740 1 -- 172.21.15.40:0/2528982256 shutdown_connections 2024-04-21T23:19:34.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.256+0000 7f931abfc740 1 -- 172.21.15.40:0/2528982256 wait complete. 2024-04-21T23:19:34.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.256+0000 7f931abfc740 1 -- start start 2024-04-21T23:19:34.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.256+0000 7f931abfc740 1 Processor -- start 2024-04-21T23:19:34.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.257+0000 7f931abfc740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5568ebc65f30 0x5568ebc6c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:34.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.257+0000 7f931abfc740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5568ebc66850 0x5568ebc6c970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:34.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.257+0000 7f931abfc740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568ebc6f430 0x5568ebc6ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:34.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.257+0000 7f93165ab640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568ebc6f430 0x5568ebc6ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:34.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.257+0000 7f93165ab640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568ebc6f430 0x5568ebc6ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48126/0 (socket says 172.21.15.40:48126) 2024-04-21T23:19:34.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.257+0000 7f93165ab640 1 -- 172.21.15.40:0/3992583050 learned_addr learned my addr 172.21.15.40:0/3992583050 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:34.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.257+0000 7f9315daa640 1 --2- 172.21.15.40:0/3992583050 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5568ebc65f30 0x5568ebc6c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:34.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.257+0000 7f931abfc740 1 -- 172.21.15.40:0/3992583050 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5568ebc72220 con 0x5568ebc6f430 2024-04-21T23:19:34.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.257+0000 7f931abfc740 1 -- 172.21.15.40:0/3992583050 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5568ebb85e00 con 0x5568ebc66850 2024-04-21T23:19:34.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.257+0000 7f931abfc740 1 -- 172.21.15.40:0/3992583050 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5568ebc723a0 con 0x5568ebc65f30 2024-04-21T23:19:34.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.257+0000 7f93155a9640 1 --2- 172.21.15.40:0/3992583050 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5568ebc66850 0x5568ebc6c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:34.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.258+0000 7f93165ab640 1 -- 172.21.15.40:0/3992583050 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5568ebc65f30 msgr2=0x5568ebc6c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:34.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.258+0000 7f93165ab640 1 --2- 172.21.15.40:0/3992583050 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5568ebc65f30 0x5568ebc6c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.258+0000 7f93165ab640 1 -- 172.21.15.40:0/3992583050 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5568ebc66850 msgr2=0x5568ebc6c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:34.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.258+0000 7f93165ab640 1 --2- 172.21.15.40:0/3992583050 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5568ebc66850 0x5568ebc6c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.258+0000 7f93165ab640 1 -- 172.21.15.40:0/3992583050 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5568eb8de3f0 con 0x5568ebc6f430 2024-04-21T23:19:34.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.258+0000 7f93155a9640 1 --2- 172.21.15.40:0/3992583050 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5568ebc66850 0x5568ebc6c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:19:34.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.258+0000 7f9315daa640 1 --2- 172.21.15.40:0/3992583050 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5568ebc65f30 0x5568ebc6c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:19:34.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.258+0000 7f931abfc740 10 client.0 _ll_register_callbacks cb 0x5568ebbc0eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:19:34.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.258+0000 7f93165ab640 1 --2- 172.21.15.40:0/3992583050 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568ebc6f430 0x5568ebc6ceb0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f9308009ca0 tx=0x7f9308032950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4aa9515a3993e532 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:34.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.258+0000 7f92f97fa640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-21T23:19:34.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.258+0000 7f931abfc740 1 -- 172.21.15.40:0/3992583050 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f930c098310 con 0x5568ebc6f430 2024-04-21T23:19:34.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.258+0000 7f92f97fa640 1 -- 172.21.15.40:0/3992583050 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93080098c0 con 0x5568ebc6f430 2024-04-21T23:19:34.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.258+0000 7f92f97fa640 1 -- 172.21.15.40:0/3992583050 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f9308032cd0 con 0x5568ebc6f430 2024-04-21T23:19:34.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.258+0000 7f92f97fa640 1 -- 172.21.15.40:0/3992583050 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f9308004040 con 0x5568ebc6f430 2024-04-21T23:19:34.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.259+0000 7f931abfc740 10 client.15927 fetch_fsmap learned FSMap version 25 2024-04-21T23:19:34.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.259+0000 7f931abfc740 1 -- 172.21.15.40:0/3992583050 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({fsmap=25}) -- 0x5568eb89e0f0 con 0x5568ebc6f430 2024-04-21T23:19:34.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.259+0000 7f92f97fa640 1 -- 172.21.15.40:0/3992583050 <== mon.0 v2:172.21.15.40:3300/0 4 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f930804f030 con 0x5568ebc6f430 2024-04-21T23:19:34.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.259+0000 7f931abfc740 10 client.15927 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:19:34.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.259+0000 7f931abfc740 10 client.15927 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:19:34.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.259+0000 7f931abfc740 20 client.15927 populate_metadata read hostname 'smithi040' 2024-04-21T23:19:34.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.259+0000 7f931abfc740 1 --2- 172.21.15.40:0/3992583050 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x5568ebc7f580 0x5568ebc81a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:34.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.259+0000 7f931abfc740 4 client.15927 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:19:34.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.259+0000 7f931abfc740 1 -- 172.21.15.40:0/3992583050 --> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f930c098310 con 0x5568ebc7f580 2024-04-21T23:19:34.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.259+0000 7f9315daa640 1 --2- 172.21.15.40:0/3992583050 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x5568ebc7f580 0x5568ebc81a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:34.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.260+0000 7f9315daa640 1 --2- 172.21.15.40:0/3992583050 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x5568ebc7f580 0x5568ebc81a40 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a6bf401411157a03 server_cookie=adaf4db9e975651d in_seq=0 out_seq=0 2024-04-21T23:19:34.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.260+0000 7f92f97fa640 10 client.15927 ms_handle_connect on v2:172.21.15.111:6838/3712235548 2024-04-21T23:19:34.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.261+0000 7f92f97fa640 1 -- 172.21.15.40:0/3992583050 <== mds.0 v2:172.21.15.111:6838/3712235548 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f930c098310 con 0x5568ebc7f580 2024-04-21T23:19:34.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.261+0000 7f92f97fa640 10 client.15927 handle_command_reply: tid=0 2024-04-21T23:19:34.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.261+0000 7f931abfc740 1 client.15927 shutdown 2024-04-21T23:19:34.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.261+0000 7f931abfc740 1 -- 172.21.15.40:0/3992583050 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568ebc6f430 msgr2=0x5568ebc6ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:34.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.261+0000 7f931abfc740 1 --2- 172.21.15.40:0/3992583050 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568ebc6f430 0x5568ebc6ceb0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f9308009ca0 tx=0x7f9308032950 comp rx=0 tx=0).stop 2024-04-21T23:19:34.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.261+0000 7f931abfc740 1 -- 172.21.15.40:0/3992583050 shutdown_connections 2024-04-21T23:19:34.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.261+0000 7f931abfc740 1 --2- 172.21.15.40:0/3992583050 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x5568ebc7f580 0x5568ebc81a40 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.261+0000 7f92f97fa640 0 client.15927 ms_handle_reset on v2:172.21.15.111:6838/3712235548 2024-04-21T23:19:34.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.261+0000 7f931abfc740 1 --2- 172.21.15.40:0/3992583050 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5568ebc66850 0x5568ebc6c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.261+0000 7f931abfc740 1 --2- 172.21.15.40:0/3992583050 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568ebc6f430 0x5568ebc6ceb0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.261+0000 7f931abfc740 1 --2- 172.21.15.40:0/3992583050 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5568ebc65f30 0x5568ebc6c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.261+0000 7f931abfc740 1 -- 172.21.15.40:0/3992583050 >> 172.21.15.40:0/3992583050 conn(0x5568ebc60660 msgr2=0x5568ebc623b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:34.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.261+0000 7f931abfc740 1 -- 172.21.15.40:0/3992583050 shutdown_connections 2024-04-21T23:19:34.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.261+0000 7f931abfc740 1 -- 172.21.15.40:0/3992583050 wait complete. 2024-04-21T23:19:34.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.261+0000 7f931abfc740 20 client.15927 trim_cache size 0 max 16384 2024-04-21T23:19:34.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.297+0000 7f931abfc740 1 Processor -- start 2024-04-21T23:19:34.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.297+0000 7f931abfc740 1 -- start start 2024-04-21T23:19:34.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.297+0000 7f931abfc740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5568ebc65f30 0x5568ebc7b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:34.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.297+0000 7f931abfc740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5568ebc66850 0x5568ebc6ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:34.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.297+0000 7f931abfc740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568ebc6f430 0x5568ebc621a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:34.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.297+0000 7f9315daa640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5568ebc65f30 0x5568ebc7b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:34.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.297+0000 7f9315daa640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5568ebc65f30 0x5568ebc7b0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.40:34680/0 (socket says 172.21.15.40:34680) 2024-04-21T23:19:34.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.297+0000 7f93165ab640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568ebc6f430 0x5568ebc621a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:34.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.297+0000 7f9315daa640 1 -- 172.21.15.40:0/1232698390 learned_addr learned my addr 172.21.15.40:0/1232698390 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:34.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.297+0000 7f931abfc740 1 -- 172.21.15.40:0/1232698390 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f92f40127a0 con 0x5568ebc6f430 2024-04-21T23:19:34.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.297+0000 7f93155a9640 1 --2- 172.21.15.40:0/1232698390 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5568ebc66850 0x5568ebc6ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:34.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.297+0000 7f931abfc740 1 -- 172.21.15.40:0/1232698390 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f92f4010070 con 0x5568ebc66850 2024-04-21T23:19:34.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.297+0000 7f931abfc740 1 -- 172.21.15.40:0/1232698390 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f92f400dec0 con 0x5568ebc65f30 2024-04-21T23:19:34.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.298+0000 7f93165ab640 1 -- 172.21.15.40:0/1232698390 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5568ebc65f30 msgr2=0x5568ebc7b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:34.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.298+0000 7f93165ab640 1 --2- 172.21.15.40:0/1232698390 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5568ebc65f30 0x5568ebc7b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.298+0000 7f93165ab640 1 -- 172.21.15.40:0/1232698390 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5568ebc66850 msgr2=0x5568ebc6ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:34.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.298+0000 7f93165ab640 1 --2- 172.21.15.40:0/1232698390 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5568ebc66850 0x5568ebc6ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.298+0000 7f93165ab640 1 -- 172.21.15.40:0/1232698390 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5568ebc71820 con 0x5568ebc6f430 2024-04-21T23:19:34.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.298+0000 7f93165ab640 1 --2- 172.21.15.40:0/1232698390 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568ebc6f430 0x5568ebc621a0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f930802f020 tx=0x7f930803f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c10f134a855f1e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:34.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.298+0000 7f92fb7fe640 1 -- 172.21.15.40:0/1232698390 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93080098c0 con 0x5568ebc6f430 2024-04-21T23:19:34.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.298+0000 7f92fb7fe640 1 -- 172.21.15.40:0/1232698390 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9308004530 con 0x5568ebc6f430 2024-04-21T23:19:34.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.298+0000 7f92fb7fe640 1 -- 172.21.15.40:0/1232698390 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f930803ce40 con 0x5568ebc6f430 2024-04-21T23:19:34.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.299+0000 7f931abfc740 1 -- 172.21.15.40:0/1232698390 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568ebc6f430 msgr2=0x5568ebc621a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:34.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.299+0000 7f931abfc740 1 --2- 172.21.15.40:0/1232698390 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568ebc6f430 0x5568ebc621a0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f930802f020 tx=0x7f930803f000 comp rx=0 tx=0).stop 2024-04-21T23:19:34.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.299+0000 7f931abfc740 1 -- 172.21.15.40:0/1232698390 shutdown_connections 2024-04-21T23:19:34.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.299+0000 7f931abfc740 1 --2- 172.21.15.40:0/1232698390 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5568ebc66850 0x5568ebc6ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.299+0000 7f931abfc740 1 --2- 172.21.15.40:0/1232698390 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568ebc6f430 0x5568ebc621a0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.299+0000 7f931abfc740 1 --2- 172.21.15.40:0/1232698390 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5568ebc65f30 0x5568ebc7b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.299+0000 7f931abfc740 1 -- 172.21.15.40:0/1232698390 >> 172.21.15.40:0/1232698390 conn(0x5568ebc60660 msgr2=0x5568ebc61a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:34.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.299+0000 7f931abfc740 1 -- 172.21.15.40:0/1232698390 shutdown_connections 2024-04-21T23:19:34.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.299+0000 7f931abfc740 1 -- 172.21.15.40:0/1232698390 wait complete. 2024-04-21T23:19:34.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.300+0000 7f931abfc740 1 -- start start 2024-04-21T23:19:34.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.300+0000 7f931abfc740 1 Processor -- start 2024-04-21T23:19:34.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.300+0000 7f931abfc740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5568ebc65f30 0x5568ebc69330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:34.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.300+0000 7f931abfc740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5568ebc66850 0x5568ebc69870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:34.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.300+0000 7f931abfc740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568ebc6f430 0x5568ebc69db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:34.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.300+0000 7f931abfc740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f92f400e530 con 0x5568ebc6f430 2024-04-21T23:19:34.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.300+0000 7f931abfc740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f92f400f9a0 con 0x5568ebc65f30 2024-04-21T23:19:34.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.300+0000 7f931abfc740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f92f400ec00 con 0x5568ebc66850 2024-04-21T23:19:34.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.301+0000 7f931abfc740 10 client.0 _ll_register_callbacks cb 0x7f9300005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:19:34.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.301+0000 7f9315daa640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5568ebc65f30 0x5568ebc69330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:34.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.301+0000 7f9315daa640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5568ebc65f30 0x5568ebc69330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.40:53742/0 (socket says 172.21.15.40:53742) 2024-04-21T23:19:34.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.301+0000 7f9315daa640 1 -- 172.21.15.40:0/1217138360 learned_addr learned my addr 172.21.15.40:0/1217138360 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:34.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.301+0000 7f93165ab640 1 --2- 172.21.15.40:0/1217138360 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568ebc6f430 0x5568ebc69db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:34.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.301+0000 7f93155a9640 1 --2- 172.21.15.40:0/1217138360 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5568ebc66850 0x5568ebc69870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:34.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.301+0000 7f9315daa640 1 -- 172.21.15.40:0/1217138360 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5568ebc66850 msgr2=0x5568ebc69870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:34.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.301+0000 7f9315daa640 1 --2- 172.21.15.40:0/1217138360 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5568ebc66850 0x5568ebc69870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.301+0000 7f9315daa640 1 -- 172.21.15.40:0/1217138360 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568ebc6f430 msgr2=0x5568ebc69db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:34.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.301+0000 7f9315daa640 1 --2- 172.21.15.40:0/1217138360 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568ebc6f430 0x5568ebc69db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.301+0000 7f9315daa640 1 -- 172.21.15.40:0/1217138360 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5568ebce4dd0 con 0x5568ebc65f30 2024-04-21T23:19:34.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.301+0000 7f93155a9640 1 --2- 172.21.15.40:0/1217138360 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5568ebc66850 0x5568ebc69870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:19:34.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.301+0000 7f93165ab640 1 --2- 172.21.15.40:0/1217138360 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568ebc6f430 0x5568ebc69db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:19:34.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.301+0000 7f9315daa640 1 --2- 172.21.15.40:0/1217138360 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5568ebc65f30 0x5568ebc69330 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f930c098670 tx=0x7f930c005f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=26605d51679c58b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:34.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.301+0000 7f92f97fa640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-21T23:19:34.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.301+0000 7f92f97fa640 1 -- 172.21.15.40:0/1217138360 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f930c0b9060 con 0x5568ebc65f30 2024-04-21T23:19:34.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.302+0000 7f92f97fa640 1 -- 172.21.15.40:0/1217138360 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f930c098310 con 0x5568ebc65f30 2024-04-21T23:19:34.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.302+0000 7f931abfc740 1 -- 172.21.15.40:0/1217138360 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9308004530 con 0x5568ebc65f30 2024-04-21T23:19:34.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.302+0000 7f92f97fa640 1 -- 172.21.15.40:0/1217138360 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f930c09b0f0 con 0x5568ebc65f30 2024-04-21T23:19:34.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.302+0000 7f931abfc740 10 client.25475 fetch_fsmap learned FSMap version 25 2024-04-21T23:19:34.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.302+0000 7f931abfc740 1 -- 172.21.15.40:0/1217138360 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5568ebce60a0 con 0x5568ebc65f30 2024-04-21T23:19:34.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.302+0000 7f92f97fa640 1 -- 172.21.15.40:0/1217138360 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f930c0ae070 con 0x5568ebc65f30 2024-04-21T23:19:34.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.302+0000 7f92f97fa640 1 -- 172.21.15.40:0/1217138360 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f930c010ce0 con 0x5568ebc65f30 2024-04-21T23:19:34.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.302+0000 7f931abfc740 10 client.25475 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:19:34.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.302+0000 7f931abfc740 10 client.25475 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:19:34.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.302+0000 7f931abfc740 20 client.25475 populate_metadata read hostname 'smithi040' 2024-04-21T23:19:34.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.302+0000 7f931abfc740 1 --2- 172.21.15.40:0/1217138360 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x5568ebc7f580 0x5568ebce8340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:34.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.302+0000 7f931abfc740 4 client.25475 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T23:19:34.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.302+0000 7f931abfc740 1 -- 172.21.15.40:0/1217138360 --> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f9308004530 con 0x5568ebc7f580 2024-04-21T23:19:34.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.302+0000 7f93155a9640 1 --2- 172.21.15.40:0/1217138360 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x5568ebc7f580 0x5568ebce8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:34.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.303+0000 7f93155a9640 1 --2- 172.21.15.40:0/1217138360 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x5568ebc7f580 0x5568ebce8340 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f3858185b5e2ceaa server_cookie=d7e34381b3407784 in_seq=0 out_seq=0 2024-04-21T23:19:34.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.303+0000 7f92f97fa640 10 client.25475 ms_handle_connect on v2:172.21.15.111:6838/3712235548 2024-04-21T23:19:34.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.303+0000 7f92f97fa640 1 -- 172.21.15.40:0/1217138360 <== mds.0 v2:172.21.15.111:6838/3712235548 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7f9308004530 con 0x5568ebc7f580 2024-04-21T23:19:34.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.303+0000 7f92f97fa640 10 client.25475 handle_command_reply: tid=0 2024-04-21T23:19:34.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.303+0000 7f931abfc740 1 client.25475 shutdown 2024-04-21T23:19:34.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.303+0000 7f931abfc740 1 -- 172.21.15.40:0/1217138360 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5568ebc65f30 msgr2=0x5568ebc69330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:34.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.303+0000 7f931abfc740 1 --2- 172.21.15.40:0/1217138360 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5568ebc65f30 0x5568ebc69330 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f930c098670 tx=0x7f930c005f10 comp rx=0 tx=0).stop 2024-04-21T23:19:34.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.303+0000 7f931abfc740 1 -- 172.21.15.40:0/1217138360 shutdown_connections 2024-04-21T23:19:34.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.303+0000 7f931abfc740 1 --2- 172.21.15.40:0/1217138360 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x5568ebc7f580 0x5568ebce8340 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.303+0000 7f92f97fa640 0 client.25475 ms_handle_reset on v2:172.21.15.111:6838/3712235548 2024-04-21T23:19:34.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.303+0000 7f931abfc740 1 --2- 172.21.15.40:0/1217138360 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568ebc6f430 0x5568ebc69db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.303+0000 7f931abfc740 1 --2- 172.21.15.40:0/1217138360 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5568ebc66850 0x5568ebc69870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.303+0000 7f931abfc740 1 --2- 172.21.15.40:0/1217138360 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5568ebc65f30 0x5568ebc69330 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.303+0000 7f931abfc740 1 -- 172.21.15.40:0/1217138360 >> 172.21.15.40:0/1217138360 conn(0x5568ebc60660 msgr2=0x5568ebc62320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:34.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.303+0000 7f931abfc740 1 -- 172.21.15.40:0/1217138360 shutdown_connections 2024-04-21T23:19:34.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.304+0000 7f931abfc740 1 -- 172.21.15.40:0/1217138360 wait complete. 2024-04-21T23:19:34.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.304+0000 7f931abfc740 20 client.25475 trim_cache size 0 max 16384 2024-04-21T23:19:34.308 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-21T23:19:34.308 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "no active scrubs running", 2024-04-21T23:19:34.308 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": {} 2024-04-21T23:19:34.308 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-21T23:19:34.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.304+0000 7f931800e640 1 -- 172.21.15.40:0/2077034741 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f92fc0a2f30 msgr2=0x7f92fc0a53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:34.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.304+0000 7f931800e640 1 --2- 172.21.15.40:0/2077034741 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f92fc0a2f30 0x7f92fc0a53f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f93100fe570 tx=0x7f9300005f90 comp rx=0 tx=0).stop 2024-04-21T23:19:34.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.304+0000 7f931800e640 1 -- 172.21.15.40:0/2077034741 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f93101029c0 msgr2=0x7f9310198000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:34.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.304+0000 7f931800e640 1 --2- 172.21.15.40:0/2077034741 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f93101029c0 0x7f9310198000 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f930c00e9f0 tx=0x7f930c00eeb0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.304+0000 7f931800e640 1 -- 172.21.15.40:0/2077034741 shutdown_connections 2024-04-21T23:19:34.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.304+0000 7f931800e640 1 --2- 172.21.15.40:0/2077034741 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f92fc0a2f30 0x7f92fc0a53f0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.304+0000 7f931800e640 1 --2- 172.21.15.40:0/2077034741 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93101032e0 0x7f9310198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.304+0000 7f931800e640 1 --2- 172.21.15.40:0/2077034741 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f931010bd00 0x7f931019c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.304+0000 7f931800e640 1 --2- 172.21.15.40:0/2077034741 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f93101029c0 0x7f9310198000 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.304+0000 7f931800e640 1 -- 172.21.15.40:0/2077034741 >> 172.21.15.40:0/2077034741 conn(0x7f9310077740 msgr2=0x7f93100fd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:34.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.304+0000 7f931800e640 1 -- 172.21.15.40:0/2077034741 shutdown_connections 2024-04-21T23:19:34.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.304+0000 7f931800e640 1 -- 172.21.15.40:0/2077034741 wait complete. 2024-04-21T23:19:34.317 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T23:19:34.318 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:19:34.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.423+0000 7f5319bc9640 1 -- 172.21.15.40:0/1238008392 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53141042f0 msgr2=0x7f5314110800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:34.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.423+0000 7f5319bc9640 1 --2- 172.21.15.40:0/1238008392 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53141042f0 0x7f5314110800 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f530c009cd0 tx=0x7f530c02c0f0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.423+0000 7f5319bc9640 1 -- 172.21.15.40:0/1238008392 shutdown_connections 2024-04-21T23:19:34.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.423+0000 7f5319bc9640 1 --2- 172.21.15.40:0/1238008392 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5314103930 0x7f5314103db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.423+0000 7f5319bc9640 1 --2- 172.21.15.40:0/1238008392 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53141042f0 0x7f5314110800 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.423+0000 7f5319bc9640 1 --2- 172.21.15.40:0/1238008392 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5314109940 0x7f5314109d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.423+0000 7f5319bc9640 1 -- 172.21.15.40:0/1238008392 >> 172.21.15.40:0/1238008392 conn(0x7f53140ff400 msgr2=0x7f5314101820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:34.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.423+0000 7f5319bc9640 1 -- 172.21.15.40:0/1238008392 shutdown_connections 2024-04-21T23:19:34.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.423+0000 7f5319bc9640 1 -- 172.21.15.40:0/1238008392 wait complete. 2024-04-21T23:19:34.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.424+0000 7f5319bc9640 1 Processor -- start 2024-04-21T23:19:34.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.424+0000 7f5319bc9640 1 -- start start 2024-04-21T23:19:34.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.424+0000 7f5319bc9640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5314103930 0x7f531419a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:34.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.424+0000 7f5319bc9640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f53141042f0 0x7f531419a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:34.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.424+0000 7f5319bc9640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5314109940 0x7f531419eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:34.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.424+0000 7f5319bc9640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5314113d50 con 0x7f5314109940 2024-04-21T23:19:34.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.424+0000 7f5319bc9640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f5314113bd0 con 0x7f5314103930 2024-04-21T23:19:34.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.424+0000 7f5319bc9640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5314113ed0 con 0x7f53141042f0 2024-04-21T23:19:34.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.424+0000 7f5313fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5314109940 0x7f531419eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:34.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.424+0000 7f5313fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5314109940 0x7f531419eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48162/0 (socket says 172.21.15.40:48162) 2024-04-21T23:19:34.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.424+0000 7f5313fff640 1 -- 172.21.15.40:0/3448120976 learned_addr learned my addr 172.21.15.40:0/3448120976 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:34.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.424+0000 7f53137fe640 1 --2- 172.21.15.40:0/3448120976 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5314103930 0x7f531419a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:34.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.425+0000 7f5312ffd640 1 --2- 172.21.15.40:0/3448120976 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f53141042f0 0x7f531419a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:34.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.425+0000 7f5313fff640 1 -- 172.21.15.40:0/3448120976 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f53141042f0 msgr2=0x7f531419a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:34.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.425+0000 7f5313fff640 1 --2- 172.21.15.40:0/3448120976 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f53141042f0 0x7f531419a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.425+0000 7f5313fff640 1 -- 172.21.15.40:0/3448120976 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5314103930 msgr2=0x7f531419a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:34.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.425+0000 7f5313fff640 1 --2- 172.21.15.40:0/3448120976 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5314103930 0x7f531419a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.425+0000 7f5313fff640 1 -- 172.21.15.40:0/3448120976 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f531419f230 con 0x7f5314109940 2024-04-21T23:19:34.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.425+0000 7f5313fff640 1 --2- 172.21.15.40:0/3448120976 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5314109940 0x7f531419eb00 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f530c0348f0 tx=0x7f530c032c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:34.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.425+0000 7f5310ff9640 1 -- 172.21.15.40:0/3448120976 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f530c041020 con 0x7f5314109940 2024-04-21T23:19:34.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.425+0000 7f5310ff9640 1 -- 172.21.15.40:0/3448120976 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f530c032e80 con 0x7f5314109940 2024-04-21T23:19:34.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.425+0000 7f5310ff9640 1 -- 172.21.15.40:0/3448120976 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f530c039040 con 0x7f5314109940 2024-04-21T23:19:34.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.425+0000 7f5319bc9640 1 -- 172.21.15.40:0/3448120976 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f531419f4c0 con 0x7f5314109940 2024-04-21T23:19:34.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.425+0000 7f5319bc9640 1 -- 172.21.15.40:0/3448120976 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53141a6db0 con 0x7f5314109940 2024-04-21T23:19:34.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.427+0000 7f5319bc9640 1 -- 172.21.15.40:0/3448120976 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5314068560 con 0x7f5314109940 2024-04-21T23:19:34.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.428+0000 7f5310ff9640 1 -- 172.21.15.40:0/3448120976 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 25) ==== 138117+0+0 (secure 0 0 0) 0x7f530c044020 con 0x7f5314109940 2024-04-21T23:19:34.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.429+0000 7f5310ff9640 1 --2- 172.21.15.40:0/3448120976 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f52fc0a2f30 0x7f52fc0a53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:34.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.429+0000 7f5310ff9640 1 -- 172.21.15.40:0/3448120976 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f530c0f1bf0 con 0x7f5314109940 2024-04-21T23:19:34.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.430+0000 7f53137fe640 1 --2- 172.21.15.40:0/3448120976 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f52fc0a2f30 0x7f52fc0a53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:34.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.430+0000 7f53137fe640 1 --2- 172.21.15.40:0/3448120976 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f52fc0a2f30 0x7f52fc0a53f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5304009b50 tx=0x7f5304005f90 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:34.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.433+0000 7f5310ff9640 1 -- 172.21.15.40:0/3448120976 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f530c04b020 con 0x7f5314109940 2024-04-21T23:19:34.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.662+0000 7f5319bc9640 1 -- 172.21.15.40:0/3448120976 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f531419b590 con 0x7f5314109940 2024-04-21T23:19:34.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.663+0000 7f5310ff9640 1 -- 172.21.15.40:0/3448120976 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11882 (secure 0 0 0) 0x7f531419b590 con 0x7f5314109940 2024-04-21T23:19:34.662 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:19:34.662 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/1571948830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1571948830},{"type":"v1","addr":"172.21.15.40:6835","nonce":1571948830}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/969356642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":969356642},{"type":"v1","addr":"172.21.15.40:6837","nonce":969356642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6839/3025057845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":3025057845},{"type":"v1","addr":"172.21.15.40:6839","nonce":3025057845}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6841/3350488804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":3350488804},{"type":"v1","addr":"172.21.15.40:6841","nonce":3350488804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/3438493203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":3438493203},{"type":"v1","addr":"172.21.15.174:6833","nonce":3438493203}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/3105752678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":3105752678},{"type":"v1","addr":"172.21.15.111:6833","nonce":3105752678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T23:15:55.742233+0000","modified":"2024-04-21T23:18:21.879184+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24521,"mds_1":24502,"mds_2":24494},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24461":{"gid":24461,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6835/2704462092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2704462092},{"type":"v1","addr":"172.21.15.111:6835","nonce":2704462092}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/3905733045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3905733045},{"type":"v1","addr":"172.21.15.174:6835","nonce":3905733045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6837/465579746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":465579746},{"type":"v1","addr":"172.21.15.174:6837","nonce":465579746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/2703764182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":2703764182},{"type":"v1","addr":"172.21.15.111:6837","nonce":2703764182}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24502":{"gid":24502,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.174:6839/2467509274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":2467509274},{"type":"v1","addr":"172.21.15.174:6839","nonce":2467509274}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3712235548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3712235548},{"type":"v1","addr":"172.21.15.111:6839","nonce":3712235548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24521,24494,24502]},"id":1}]} 2024-04-21T23:19:34.663 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 25 2024-04-21T23:19:34.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.665+0000 7f5319bc9640 1 -- 172.21.15.40:0/3448120976 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f52fc0a2f30 msgr2=0x7f52fc0a53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:34.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.665+0000 7f5319bc9640 1 --2- 172.21.15.40:0/3448120976 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f52fc0a2f30 0x7f52fc0a53f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5304009b50 tx=0x7f5304005f90 comp rx=0 tx=0).stop 2024-04-21T23:19:34.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.665+0000 7f5319bc9640 1 -- 172.21.15.40:0/3448120976 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5314109940 msgr2=0x7f531419eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:34.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.665+0000 7f5319bc9640 1 --2- 172.21.15.40:0/3448120976 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5314109940 0x7f531419eb00 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f530c0348f0 tx=0x7f530c032c90 comp rx=0 tx=0).stop 2024-04-21T23:19:34.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.665+0000 7f5319bc9640 1 -- 172.21.15.40:0/3448120976 shutdown_connections 2024-04-21T23:19:34.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.665+0000 7f5319bc9640 1 --2- 172.21.15.40:0/3448120976 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f52fc0a2f30 0x7f52fc0a53f0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.665+0000 7f5319bc9640 1 --2- 172.21.15.40:0/3448120976 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5314109940 0x7f531419eb00 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.665+0000 7f5319bc9640 1 --2- 172.21.15.40:0/3448120976 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f53141042f0 0x7f531419a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.665+0000 7f5319bc9640 1 --2- 172.21.15.40:0/3448120976 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5314103930 0x7f531419a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.665+0000 7f5319bc9640 1 -- 172.21.15.40:0/3448120976 >> 172.21.15.40:0/3448120976 conn(0x7f53140ff400 msgr2=0x7f53140ffc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:34.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.666+0000 7f5319bc9640 1 -- 172.21.15.40:0/3448120976 shutdown_connections 2024-04-21T23:19:34.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.666+0000 7f5319bc9640 1 -- 172.21.15.40:0/3448120976 wait complete. 2024-04-21T23:19:34.676 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-21T23:19:34.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.783+0000 7f4e5dbf7640 1 -- 172.21.15.40:0/2126223391 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e58100610 msgr2=0x7f4e58100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:34.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.783+0000 7f4e5dbf7640 1 --2- 172.21.15.40:0/2126223391 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e58100610 0x7f4e58100a90 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f4e4c009930 tx=0x7f4e4c02bc30 comp rx=0 tx=0).stop 2024-04-21T23:19:34.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.783+0000 7f4e5dbf7640 1 -- 172.21.15.40:0/2126223391 shutdown_connections 2024-04-21T23:19:34.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.783+0000 7f4e5dbf7640 1 --2- 172.21.15.40:0/2126223391 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e58100fd0 0x7f4e5810d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.783+0000 7f4e5dbf7640 1 --2- 172.21.15.40:0/2126223391 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e58100610 0x7f4e58100a90 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.783+0000 7f4e5dbf7640 1 --2- 172.21.15.40:0/2126223391 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4e581066b0 0x7f4e58106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.783+0000 7f4e5dbf7640 1 -- 172.21.15.40:0/2126223391 >> 172.21.15.40:0/2126223391 conn(0x7f4e58077740 msgr2=0x7f4e58077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:34.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.783+0000 7f4e5dbf7640 1 -- 172.21.15.40:0/2126223391 shutdown_connections 2024-04-21T23:19:34.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.783+0000 7f4e5dbf7640 1 -- 172.21.15.40:0/2126223391 wait complete. 2024-04-21T23:19:34.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.784+0000 7f4e5dbf7640 1 Processor -- start 2024-04-21T23:19:34.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.784+0000 7f4e5dbf7640 1 -- start start 2024-04-21T23:19:34.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.784+0000 7f4e5dbf7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e58100610 0x7f4e58196f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:34.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.784+0000 7f4e5dbf7640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e58100fd0 0x7f4e58197490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:34.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.784+0000 7f4e5dbf7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4e581066b0 0x7f4e5819b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:34.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.784+0000 7f4e5dbf7640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4e581109f0 con 0x7f4e58100610 2024-04-21T23:19:34.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.784+0000 7f4e5dbf7640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f4e58110870 con 0x7f4e58100fd0 2024-04-21T23:19:34.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.784+0000 7f4e5dbf7640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4e58110b70 con 0x7f4e581066b0 2024-04-21T23:19:34.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.784+0000 7f4e577fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e58100610 0x7f4e58196f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:34.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.784+0000 7f4e577fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e58100610 0x7f4e58196f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48176/0 (socket says 172.21.15.40:48176) 2024-04-21T23:19:34.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.784+0000 7f4e577fe640 1 -- 172.21.15.40:0/4040574625 learned_addr learned my addr 172.21.15.40:0/4040574625 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:34.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.785+0000 7f4e57fff640 1 --2- 172.21.15.40:0/4040574625 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4e581066b0 0x7f4e5819b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:34.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.785+0000 7f4e577fe640 1 -- 172.21.15.40:0/4040574625 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4e581066b0 msgr2=0x7f4e5819b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:34.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.785+0000 7f4e577fe640 1 --2- 172.21.15.40:0/4040574625 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4e581066b0 0x7f4e5819b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.785+0000 7f4e577fe640 1 -- 172.21.15.40:0/4040574625 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e58100fd0 msgr2=0x7f4e58197490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:34.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.785+0000 7f4e56ffd640 1 --2- 172.21.15.40:0/4040574625 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e58100fd0 0x7f4e58197490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:34.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.785+0000 7f4e577fe640 1 --2- 172.21.15.40:0/4040574625 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e58100fd0 0x7f4e58197490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.785+0000 7f4e577fe640 1 -- 172.21.15.40:0/4040574625 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e5819bec0 con 0x7f4e58100610 2024-04-21T23:19:34.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.785+0000 7f4e56ffd640 1 --2- 172.21.15.40:0/4040574625 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e58100fd0 0x7f4e58197490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:19:34.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.785+0000 7f4e577fe640 1 --2- 172.21.15.40:0/4040574625 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e58100610 0x7f4e58196f50 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f4e4800e9c0 tx=0x7f4e4800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:34.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.785+0000 7f4e54ff9640 1 -- 172.21.15.40:0/4040574625 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e4800cde0 con 0x7f4e58100610 2024-04-21T23:19:34.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.785+0000 7f4e54ff9640 1 -- 172.21.15.40:0/4040574625 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e48010910 con 0x7f4e58100610 2024-04-21T23:19:34.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.785+0000 7f4e54ff9640 1 -- 172.21.15.40:0/4040574625 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e480189f0 con 0x7f4e58100610 2024-04-21T23:19:34.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.785+0000 7f4e5dbf7640 1 -- 172.21.15.40:0/4040574625 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e5819c1b0 con 0x7f4e58100610 2024-04-21T23:19:34.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.786+0000 7f4e5dbf7640 1 -- 172.21.15.40:0/4040574625 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e580688d0 con 0x7f4e58100610 2024-04-21T23:19:34.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.788+0000 7f4e54ff9640 1 -- 172.21.15.40:0/4040574625 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 25) ==== 138117+0+0 (secure 0 0 0) 0x7f4e4801f020 con 0x7f4e58100610 2024-04-21T23:19:34.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.789+0000 7f4e607e6740 1 Processor -- start 2024-04-21T23:19:34.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.789+0000 7f4e607e6740 1 -- start start 2024-04-21T23:19:34.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.789+0000 7f4e607e6740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560cb55a6020 0x560cb55a6400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:34.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.789+0000 7f4e607e6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560cb55a6940 0x560cb55aefe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:34.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.789+0000 7f4e607e6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560cb55af520 0x560cb55b1910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:34.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.789+0000 7f4e607e6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x560cb550a530 con 0x560cb55a6940 2024-04-21T23:19:34.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.789+0000 7f4e607e6740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x560cb54b2830 con 0x560cb55a6020 2024-04-21T23:19:34.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.789+0000 7f4e607e6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x560cb55064f0 con 0x560cb55af520 2024-04-21T23:19:34.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.789+0000 7f4e577fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560cb55a6940 0x560cb55aefe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:34.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.789+0000 7f4e577fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560cb55a6940 0x560cb55aefe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48184/0 (socket says 172.21.15.40:48184) 2024-04-21T23:19:34.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.789+0000 7f4e577fe640 1 -- 172.21.15.40:0/4137489130 learned_addr learned my addr 172.21.15.40:0/4137489130 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:34.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.789+0000 7f4e57fff640 1 --2- 172.21.15.40:0/4137489130 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560cb55a6020 0x560cb55a6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:34.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.790+0000 7f4e54ff9640 1 --2- 172.21.15.40:0/4040574625 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f4e400a2eb0 0x7f4e400a5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:34.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.790+0000 7f4e56ffd640 1 --2- 172.21.15.40:0/4137489130 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560cb55af520 0x560cb55b1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:34.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.790+0000 7f4e57fff640 1 --2- 172.21.15.40:0/4040574625 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f4e400a2eb0 0x7f4e400a5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:34.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.790+0000 7f4e54ff9640 1 -- 172.21.15.40:0/4040574625 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f4e480cfd20 con 0x7f4e58100610 2024-04-21T23:19:34.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.790+0000 7f4e577fe640 1 -- 172.21.15.40:0/4137489130 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560cb55af520 msgr2=0x560cb55b1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:34.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.790+0000 7f4e577fe640 1 --2- 172.21.15.40:0/4137489130 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560cb55af520 0x560cb55b1910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.790+0000 7f4e577fe640 1 -- 172.21.15.40:0/4137489130 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560cb55a6020 msgr2=0x560cb55a6400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:34.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.790+0000 7f4e577fe640 1 --2- 172.21.15.40:0/4137489130 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560cb55a6020 0x560cb55a6400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.790+0000 7f4e577fe640 1 -- 172.21.15.40:0/4137489130 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560cb54d62c0 con 0x560cb55a6940 2024-04-21T23:19:34.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.790+0000 7f4e57fff640 1 --2- 172.21.15.40:0/4137489130 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560cb55a6020 0x560cb55a6400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:19:34.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.790+0000 7f4e577fe640 1 --2- 172.21.15.40:0/4137489130 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560cb55a6940 0x560cb55aefe0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f4e480ccf70 tx=0x7f4e4800efb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=117da550d9ea9002 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:34.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.791+0000 7f4e3dffb640 1 -- 172.21.15.40:0/4137489130 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e48097ef0 con 0x560cb55a6940 2024-04-21T23:19:34.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.791+0000 7f4e57fff640 1 --2- 172.21.15.40:0/4040574625 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f4e400a2eb0 0x7f4e400a5370 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4e50009fd0 tx=0x7f4e50004ab0 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:34.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.791+0000 7f4e3dffb640 1 -- 172.21.15.40:0/4137489130 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e4809c350 con 0x560cb55a6940 2024-04-21T23:19:34.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.791+0000 7f4e3dffb640 1 -- 172.21.15.40:0/4137489130 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e4809b5b0 con 0x560cb55a6940 2024-04-21T23:19:34.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.791+0000 7f4e607e6740 1 -- 172.21.15.40:0/4137489130 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560cb55a6940 msgr2=0x560cb55aefe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:34.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.791+0000 7f4e607e6740 1 --2- 172.21.15.40:0/4137489130 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560cb55a6940 0x560cb55aefe0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f4e480ccf70 tx=0x7f4e4800efb0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.791+0000 7f4e607e6740 1 -- 172.21.15.40:0/4137489130 shutdown_connections 2024-04-21T23:19:34.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.791+0000 7f4e607e6740 1 --2- 172.21.15.40:0/4137489130 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560cb55af520 0x560cb55b1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.791+0000 7f4e607e6740 1 --2- 172.21.15.40:0/4137489130 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560cb55a6940 0x560cb55aefe0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.791+0000 7f4e607e6740 1 --2- 172.21.15.40:0/4137489130 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560cb55a6020 0x560cb55a6400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.791+0000 7f4e607e6740 1 -- 172.21.15.40:0/4137489130 >> 172.21.15.40:0/4137489130 conn(0x560cb55a0750 msgr2=0x560cb55a0b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:34.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.791+0000 7f4e607e6740 1 -- 172.21.15.40:0/4137489130 shutdown_connections 2024-04-21T23:19:34.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.791+0000 7f4e607e6740 1 -- 172.21.15.40:0/4137489130 wait complete. 2024-04-21T23:19:34.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.792+0000 7f4e607e6740 1 -- start start 2024-04-21T23:19:34.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.792+0000 7f4e607e6740 1 Processor -- start 2024-04-21T23:19:34.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.792+0000 7f4e607e6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560cb55a6020 0x560cb55aef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:34.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.792+0000 7f4e607e6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560cb55a6940 0x560cb55ac2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:34.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.792+0000 7f4e607e6740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560cb55af520 0x560cb55ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:34.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.793+0000 7f4e607e6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x560cb55b22b0 con 0x560cb55a6940 2024-04-21T23:19:34.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.793+0000 7f4e607e6740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x560cb54c5f10 con 0x560cb55af520 2024-04-21T23:19:34.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.793+0000 7f4e607e6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x560cb55b2430 con 0x560cb55a6020 2024-04-21T23:19:34.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.793+0000 7f4e56ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560cb55a6940 0x560cb55ac2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:34.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.793+0000 7f4e607e6740 10 client.0 _ll_register_callbacks cb 0x560cb521dbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:19:34.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.793+0000 7f4e56ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560cb55a6940 0x560cb55ac2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48196/0 (socket says 172.21.15.40:48196) 2024-04-21T23:19:34.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.793+0000 7f4e56ffd640 1 -- 172.21.15.40:0/2377812309 learned_addr learned my addr 172.21.15.40:0/2377812309 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:34.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.793+0000 7f4e577fe640 1 --2- 172.21.15.40:0/2377812309 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560cb55a6020 0x560cb55aef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:34.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.793+0000 7f4e57fff640 1 --2- 172.21.15.40:0/2377812309 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560cb55af520 0x560cb55ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:34.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.793+0000 7f4e577fe640 1 -- 172.21.15.40:0/2377812309 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560cb55af520 msgr2=0x560cb55ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:34.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.793+0000 7f4e577fe640 1 --2- 172.21.15.40:0/2377812309 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560cb55af520 0x560cb55ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.793+0000 7f4e577fe640 1 -- 172.21.15.40:0/2377812309 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560cb55a6940 msgr2=0x560cb55ac2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:34.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.793+0000 7f4e577fe640 1 --2- 172.21.15.40:0/2377812309 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560cb55a6940 0x560cb55ac2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.793+0000 7f4e577fe640 1 -- 172.21.15.40:0/2377812309 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560cb5497c80 con 0x560cb55a6020 2024-04-21T23:19:34.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.793+0000 7f4e56ffd640 1 --2- 172.21.15.40:0/2377812309 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560cb55a6940 0x560cb55ac2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:19:34.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.793+0000 7f4e57fff640 1 --2- 172.21.15.40:0/2377812309 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560cb55af520 0x560cb55ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:19:34.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.793+0000 7f4e577fe640 1 --2- 172.21.15.40:0/2377812309 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560cb55a6020 0x560cb55aef30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x560cb55a5af0 tx=0x7f4e48098460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=54411b94abf05372 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:34.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.794+0000 7f4e36ffd640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-04-21T23:19:34.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.794+0000 7f4e36ffd640 1 -- 172.21.15.40:0/2377812309 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e480b1070 con 0x560cb55a6020 2024-04-21T23:19:34.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.794+0000 7f4e607e6740 1 -- 172.21.15.40:0/2377812309 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4e4809c350 con 0x560cb55a6020 2024-04-21T23:19:34.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.794+0000 7f4e36ffd640 1 -- 172.21.15.40:0/2377812309 <== mon.1 v2:172.21.15.174:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f4e48018dc0 con 0x560cb55a6020 2024-04-21T23:19:34.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.794+0000 7f4e36ffd640 1 -- 172.21.15.40:0/2377812309 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f4e4809b790 con 0x560cb55a6020 2024-04-21T23:19:34.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.794+0000 7f4e607e6740 10 client.25483 fetch_fsmap learned FSMap version 25 2024-04-21T23:19:34.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.794+0000 7f4e607e6740 1 -- 172.21.15.40:0/2377812309 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x560cb521e3f0 con 0x560cb55a6020 2024-04-21T23:19:34.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.794+0000 7f4e36ffd640 1 -- 172.21.15.40:0/2377812309 <== mon.1 v2:172.21.15.174:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e4809c350 con 0x560cb55a6020 2024-04-21T23:19:34.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.794+0000 7f4e36ffd640 1 -- 172.21.15.40:0/2377812309 <== mon.1 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f4e4809b230 con 0x560cb55a6020 2024-04-21T23:19:34.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.794+0000 7f4e607e6740 10 client.25483 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:19:34.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.794+0000 7f4e607e6740 10 client.25483 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-21T23:19:34.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.794+0000 7f4e607e6740 20 client.25483 populate_metadata read hostname 'smithi040' 2024-04-21T23:19:34.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.794+0000 7f4e607e6740 1 --2- 172.21.15.40:0/2377812309 >> [v2:172.21.15.111:6836/2703764182,v1:172.21.15.111:6837/2703764182] conn(0x560cb55bf460 0x560cb55c1850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:34.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.794+0000 7f4e607e6740 4 client.25483 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:19:34.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.794+0000 7f4e607e6740 1 -- 172.21.15.40:0/2377812309 --> [v2:172.21.15.111:6836/2703764182,v1:172.21.15.111:6837/2703764182] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560cb5517a70 con 0x560cb55bf460 2024-04-21T23:19:34.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.795+0000 7f4e56ffd640 1 --2- 172.21.15.40:0/2377812309 >> [v2:172.21.15.111:6836/2703764182,v1:172.21.15.111:6837/2703764182] conn(0x560cb55bf460 0x560cb55c1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:34.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.795+0000 7f4e56ffd640 1 --2- 172.21.15.40:0/2377812309 >> [v2:172.21.15.111:6836/2703764182,v1:172.21.15.111:6837/2703764182] conn(0x560cb55bf460 0x560cb55c1850 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f1fb4b168a14025a server_cookie=cd65d248fa0d8747 in_seq=0 out_seq=0 2024-04-21T23:19:34.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.795+0000 7f4e36ffd640 10 client.25483 ms_handle_connect on v2:172.21.15.111:6836/2703764182 2024-04-21T23:19:34.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.796+0000 7f4e36ffd640 1 -- 172.21.15.40:0/2377812309 <== mds.2 v2:172.21.15.111:6836/2703764182 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560cb5517a70 con 0x560cb55bf460 2024-04-21T23:19:34.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.796+0000 7f4e36ffd640 10 client.25483 handle_command_reply: tid=0 2024-04-21T23:19:34.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.796+0000 7f4e607e6740 1 client.25483 shutdown 2024-04-21T23:19:34.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.796+0000 7f4e607e6740 1 -- 172.21.15.40:0/2377812309 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560cb55a6020 msgr2=0x560cb55aef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:34.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.796+0000 7f4e607e6740 1 --2- 172.21.15.40:0/2377812309 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560cb55a6020 0x560cb55aef30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x560cb55a5af0 tx=0x7f4e48098460 comp rx=0 tx=0).stop 2024-04-21T23:19:34.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.796+0000 7f4e607e6740 1 -- 172.21.15.40:0/2377812309 shutdown_connections 2024-04-21T23:19:34.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.796+0000 7f4e607e6740 1 --2- 172.21.15.40:0/2377812309 >> [v2:172.21.15.111:6836/2703764182,v1:172.21.15.111:6837/2703764182] conn(0x560cb55bf460 0x560cb55c1850 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.796+0000 7f4e607e6740 1 --2- 172.21.15.40:0/2377812309 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560cb55af520 0x560cb55ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.796+0000 7f4e607e6740 1 --2- 172.21.15.40:0/2377812309 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560cb55a6940 0x560cb55ac2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.796+0000 7f4e36ffd640 0 client.25483 ms_handle_reset on v2:172.21.15.111:6836/2703764182 2024-04-21T23:19:34.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.796+0000 7f4e607e6740 1 --2- 172.21.15.40:0/2377812309 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560cb55a6020 0x560cb55aef30 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.796+0000 7f4e607e6740 1 -- 172.21.15.40:0/2377812309 >> 172.21.15.40:0/2377812309 conn(0x560cb55a0750 msgr2=0x560cb55a24a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:34.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.796+0000 7f4e607e6740 1 -- 172.21.15.40:0/2377812309 shutdown_connections 2024-04-21T23:19:34.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.796+0000 7f4e607e6740 1 -- 172.21.15.40:0/2377812309 wait complete. 2024-04-21T23:19:34.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.797+0000 7f4e607e6740 20 client.25483 trim_cache size 0 max 16384 2024-04-21T23:19:34.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.810+0000 7f4e607e6740 1 Processor -- start 2024-04-21T23:19:34.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.810+0000 7f4e607e6740 1 -- start start 2024-04-21T23:19:34.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.810+0000 7f4e607e6740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560cb55a6020 0x560cb55a2290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:34.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.810+0000 7f4e607e6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560cb55a6940 0x560cb55ae9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:34.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.810+0000 7f4e607e6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560cb55af520 0x560cb55aef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:34.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.810+0000 7f4e607e6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4e581b3840 con 0x560cb55a6940 2024-04-21T23:19:34.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.810+0000 7f4e607e6740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f4e581b1110 con 0x560cb55a6020 2024-04-21T23:19:34.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.810+0000 7f4e607e6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4e581aef60 con 0x560cb55af520 2024-04-21T23:19:34.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.810+0000 7f4e56ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560cb55a6940 0x560cb55ae9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:34.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.810+0000 7f4e56ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560cb55a6940 0x560cb55ae9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48210/0 (socket says 172.21.15.40:48210) 2024-04-21T23:19:34.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.810+0000 7f4e56ffd640 1 -- 172.21.15.40:0/1276560794 learned_addr learned my addr 172.21.15.40:0/1276560794 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:34.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.811+0000 7f4e577fe640 1 --2- 172.21.15.40:0/1276560794 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560cb55a6020 0x560cb55a2290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:34.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.811+0000 7f4e57fff640 1 --2- 172.21.15.40:0/1276560794 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560cb55af520 0x560cb55aef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:34.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.811+0000 7f4e56ffd640 1 -- 172.21.15.40:0/1276560794 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560cb55af520 msgr2=0x560cb55aef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:34.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.811+0000 7f4e56ffd640 1 --2- 172.21.15.40:0/1276560794 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560cb55af520 0x560cb55aef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.811+0000 7f4e56ffd640 1 -- 172.21.15.40:0/1276560794 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560cb55a6020 msgr2=0x560cb55a2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:34.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.811+0000 7f4e56ffd640 1 --2- 172.21.15.40:0/1276560794 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560cb55a6020 0x560cb55a2290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.811+0000 7f4e56ffd640 1 -- 172.21.15.40:0/1276560794 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560cb55acc40 con 0x560cb55a6940 2024-04-21T23:19:34.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.811+0000 7f4e57fff640 1 --2- 172.21.15.40:0/1276560794 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560cb55af520 0x560cb55aef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:19:34.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.811+0000 7f4e577fe640 1 --2- 172.21.15.40:0/1276560794 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560cb55a6020 0x560cb55a2290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:19:34.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.811+0000 7f4e56ffd640 1 --2- 172.21.15.40:0/1276560794 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560cb55a6940 0x560cb55ae9f0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f4e4c036040 tx=0x7f4e4c005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51ec73adad8c9348 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:34.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.811+0000 7f4e3d7fa640 1 -- 172.21.15.40:0/1276560794 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e4c02cd70 con 0x560cb55a6940 2024-04-21T23:19:34.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.811+0000 7f4e3d7fa640 1 -- 172.21.15.40:0/1276560794 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e4c045bc0 con 0x560cb55a6940 2024-04-21T23:19:34.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.811+0000 7f4e3d7fa640 1 -- 172.21.15.40:0/1276560794 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e4c0052a0 con 0x560cb55a6940 2024-04-21T23:19:34.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.812+0000 7f4e607e6740 1 -- 172.21.15.40:0/1276560794 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560cb55a6940 msgr2=0x560cb55ae9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:34.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.812+0000 7f4e607e6740 1 --2- 172.21.15.40:0/1276560794 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560cb55a6940 0x560cb55ae9f0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f4e4c036040 tx=0x7f4e4c005a20 comp rx=0 tx=0).stop 2024-04-21T23:19:34.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.812+0000 7f4e607e6740 1 -- 172.21.15.40:0/1276560794 shutdown_connections 2024-04-21T23:19:34.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.812+0000 7f4e607e6740 1 --2- 172.21.15.40:0/1276560794 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560cb55af520 0x560cb55aef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.812+0000 7f4e607e6740 1 --2- 172.21.15.40:0/1276560794 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560cb55a6940 0x560cb55ae9f0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.812+0000 7f4e607e6740 1 --2- 172.21.15.40:0/1276560794 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560cb55a6020 0x560cb55a2290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.812+0000 7f4e607e6740 1 -- 172.21.15.40:0/1276560794 >> 172.21.15.40:0/1276560794 conn(0x560cb55a0750 msgr2=0x560cb55a1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:34.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.812+0000 7f4e607e6740 1 -- 172.21.15.40:0/1276560794 shutdown_connections 2024-04-21T23:19:34.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.812+0000 7f4e607e6740 1 -- 172.21.15.40:0/1276560794 wait complete. 2024-04-21T23:19:34.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.812+0000 7f4e607e6740 1 -- start start 2024-04-21T23:19:34.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.813+0000 7f4e607e6740 1 Processor -- start 2024-04-21T23:19:34.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.813+0000 7f4e607e6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560cb55a6020 0x560cb55c4b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:34.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.813+0000 7f4e607e6740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560cb55a6940 0x560cb55c1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:34.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.813+0000 7f4e607e6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560cb55af520 0x560cb55c2400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:34.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.813+0000 7f4e607e6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4e581af5d0 con 0x560cb55af520 2024-04-21T23:19:34.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.813+0000 7f4e607e6740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f4e581b0a40 con 0x560cb55a6940 2024-04-21T23:19:34.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.813+0000 7f4e607e6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4e581afca0 con 0x560cb55a6020 2024-04-21T23:19:34.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.813+0000 7f4e607e6740 10 client.0 _ll_register_callbacks cb 0x7f4e50009ae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:19:34.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.813+0000 7f4e57fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560cb55af520 0x560cb55c2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:34.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.813+0000 7f4e577fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560cb55a6020 0x560cb55c4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:34.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.813+0000 7f4e57fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560cb55af520 0x560cb55c2400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48214/0 (socket says 172.21.15.40:48214) 2024-04-21T23:19:34.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.813+0000 7f4e577fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560cb55a6020 0x560cb55c4b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.40:34754/0 (socket says 172.21.15.40:34754) 2024-04-21T23:19:34.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.813+0000 7f4e57fff640 1 -- 172.21.15.40:0/1637843230 learned_addr learned my addr 172.21.15.40:0/1637843230 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:34.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.813+0000 7f4e56ffd640 1 --2- 172.21.15.40:0/1637843230 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560cb55a6940 0x560cb55c1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:34.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.813+0000 7f4e57fff640 1 -- 172.21.15.40:0/1637843230 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560cb55a6020 msgr2=0x560cb55c4b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:34.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.813+0000 7f4e57fff640 1 --2- 172.21.15.40:0/1637843230 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560cb55a6020 0x560cb55c4b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.813+0000 7f4e57fff640 1 -- 172.21.15.40:0/1637843230 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560cb55a6940 msgr2=0x560cb55c1ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:34.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.813+0000 7f4e57fff640 1 --2- 172.21.15.40:0/1637843230 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560cb55a6940 0x560cb55c1ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.813+0000 7f4e57fff640 1 -- 172.21.15.40:0/1637843230 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560cb52e87d0 con 0x560cb55af520 2024-04-21T23:19:34.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.813+0000 7f4e577fe640 1 --2- 172.21.15.40:0/1637843230 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560cb55a6020 0x560cb55c4b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:19:34.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.814+0000 7f4e57fff640 1 --2- 172.21.15.40:0/1637843230 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560cb55af520 0x560cb55c2400 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f4e500141f0 tx=0x7f4e50004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88d11b960d92133e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:34.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.814+0000 7f4e36ffd640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-21T23:19:34.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.814+0000 7f4e36ffd640 1 -- 172.21.15.40:0/1637843230 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e50028070 con 0x560cb55af520 2024-04-21T23:19:34.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.814+0000 7f4e607e6740 1 -- 172.21.15.40:0/1637843230 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4e4c045bc0 con 0x560cb55af520 2024-04-21T23:19:34.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.814+0000 7f4e36ffd640 1 -- 172.21.15.40:0/1637843230 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f4e50011bc0 con 0x560cb55af520 2024-04-21T23:19:34.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.814+0000 7f4e36ffd640 1 -- 172.21.15.40:0/1637843230 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f4e5001a040 con 0x560cb55af520 2024-04-21T23:19:34.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.814+0000 7f4e607e6740 10 client.15957 fetch_fsmap learned FSMap version 25 2024-04-21T23:19:34.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.814+0000 7f4e607e6740 1 -- 172.21.15.40:0/1637843230 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x560cb5625390 con 0x560cb55af520 2024-04-21T23:19:34.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.814+0000 7f4e36ffd640 1 -- 172.21.15.40:0/1637843230 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e4c045bc0 con 0x560cb55af520 2024-04-21T23:19:34.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.815+0000 7f4e36ffd640 1 -- 172.21.15.40:0/1637843230 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f4e50024400 con 0x560cb55af520 2024-04-21T23:19:34.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.815+0000 7f4e607e6740 10 client.15957 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:19:34.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.815+0000 7f4e607e6740 10 client.15957 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-21T23:19:34.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.815+0000 7f4e607e6740 20 client.15957 populate_metadata read hostname 'smithi040' 2024-04-21T23:19:34.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.815+0000 7f4e607e6740 1 --2- 172.21.15.40:0/1637843230 >> [v2:172.21.15.111:6836/2703764182,v1:172.21.15.111:6837/2703764182] conn(0x560cb5625af0 0x560cb5627f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:34.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.815+0000 7f4e607e6740 4 client.15957 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T23:19:34.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.815+0000 7f4e607e6740 1 -- 172.21.15.40:0/1637843230 --> [v2:172.21.15.111:6836/2703764182,v1:172.21.15.111:6837/2703764182] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560cb5511dc0 con 0x560cb5625af0 2024-04-21T23:19:34.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.815+0000 7f4e577fe640 1 --2- 172.21.15.40:0/1637843230 >> [v2:172.21.15.111:6836/2703764182,v1:172.21.15.111:6837/2703764182] conn(0x560cb5625af0 0x560cb5627f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:34.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.815+0000 7f4e36ffd640 10 client.15957 ms_handle_connect on v2:172.21.15.111:6836/2703764182 2024-04-21T23:19:34.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.815+0000 7f4e577fe640 1 --2- 172.21.15.40:0/1637843230 >> [v2:172.21.15.111:6836/2703764182,v1:172.21.15.111:6837/2703764182] conn(0x560cb5625af0 0x560cb5627f10 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=325577ab48f21101 server_cookie=41c066db76411658 in_seq=0 out_seq=0 2024-04-21T23:19:34.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.816+0000 7f4e36ffd640 1 -- 172.21.15.40:0/1637843230 <== mds.2 v2:172.21.15.111:6836/2703764182 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560cb5511dc0 con 0x560cb5625af0 2024-04-21T23:19:34.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.816+0000 7f4e36ffd640 10 client.15957 handle_command_reply: tid=0 2024-04-21T23:19:34.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.816+0000 7f4e607e6740 1 client.15957 shutdown 2024-04-21T23:19:34.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.816+0000 7f4e607e6740 1 -- 172.21.15.40:0/1637843230 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560cb55af520 msgr2=0x560cb55c2400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:34.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.816+0000 7f4e607e6740 1 --2- 172.21.15.40:0/1637843230 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560cb55af520 0x560cb55c2400 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f4e500141f0 tx=0x7f4e50004a40 comp rx=0 tx=0).stop 2024-04-21T23:19:34.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.816+0000 7f4e607e6740 1 -- 172.21.15.40:0/1637843230 shutdown_connections 2024-04-21T23:19:34.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.816+0000 7f4e607e6740 1 --2- 172.21.15.40:0/1637843230 >> [v2:172.21.15.111:6836/2703764182,v1:172.21.15.111:6837/2703764182] conn(0x560cb5625af0 0x560cb5627f10 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.816+0000 7f4e607e6740 1 --2- 172.21.15.40:0/1637843230 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560cb55a6940 0x560cb55c1ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.816+0000 7f4e607e6740 1 --2- 172.21.15.40:0/1637843230 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560cb55af520 0x560cb55c2400 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.816+0000 7f4e607e6740 1 --2- 172.21.15.40:0/1637843230 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560cb55a6020 0x560cb55c4b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.816+0000 7f4e607e6740 1 -- 172.21.15.40:0/1637843230 >> 172.21.15.40:0/1637843230 conn(0x560cb55a0750 msgr2=0x560cb559efc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:34.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.816+0000 7f4e36ffd640 0 client.15957 ms_handle_reset on v2:172.21.15.111:6836/2703764182 2024-04-21T23:19:34.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.816+0000 7f4e607e6740 1 -- 172.21.15.40:0/1637843230 shutdown_connections 2024-04-21T23:19:34.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.816+0000 7f4e607e6740 1 -- 172.21.15.40:0/1637843230 wait complete. 2024-04-21T23:19:34.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.816+0000 7f4e607e6740 20 client.15957 trim_cache size 0 max 16384 2024-04-21T23:19:34.820 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-04-21T23:19:34.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.817+0000 7f4e5dbf7640 1 -- 172.21.15.40:0/4040574625 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f4e400a2eb0 msgr2=0x7f4e400a5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:34.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.817+0000 7f4e5dbf7640 1 --2- 172.21.15.40:0/4040574625 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f4e400a2eb0 0x7f4e400a5370 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4e50009fd0 tx=0x7f4e50004ab0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.817+0000 7f4e5dbf7640 1 -- 172.21.15.40:0/4040574625 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e58100610 msgr2=0x7f4e58196f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:34.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.817+0000 7f4e5dbf7640 1 --2- 172.21.15.40:0/4040574625 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e58100610 0x7f4e58196f50 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f4e4800e9c0 tx=0x7f4e4800ee80 comp rx=0 tx=0).stop 2024-04-21T23:19:34.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.817+0000 7f4e5dbf7640 1 -- 172.21.15.40:0/4040574625 shutdown_connections 2024-04-21T23:19:34.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.817+0000 7f4e5dbf7640 1 --2- 172.21.15.40:0/4040574625 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f4e400a2eb0 0x7f4e400a5370 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.817+0000 7f4e5dbf7640 1 --2- 172.21.15.40:0/4040574625 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e58100fd0 0x7f4e58197490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.817+0000 7f4e5dbf7640 1 --2- 172.21.15.40:0/4040574625 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4e581066b0 0x7f4e5819b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.817+0000 7f4e5dbf7640 1 --2- 172.21.15.40:0/4040574625 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e58100610 0x7f4e58196f50 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.817+0000 7f4e5dbf7640 1 -- 172.21.15.40:0/4040574625 >> 172.21.15.40:0/4040574625 conn(0x7f4e58077740 msgr2=0x7f4e580ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:34.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.817+0000 7f4e5dbf7640 1 -- 172.21.15.40:0/4040574625 shutdown_connections 2024-04-21T23:19:34.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.817+0000 7f4e5dbf7640 1 -- 172.21.15.40:0/4040574625 wait complete. 2024-04-21T23:19:34.830 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-21T23:19:34.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.937+0000 7fc4a3a84640 1 -- 172.21.15.40:0/948622470 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc49c1029c0 msgr2=0x7fc49c102da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:34.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.937+0000 7fc4a3a84640 1 --2- 172.21.15.40:0/948622470 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc49c1029c0 0x7fc49c102da0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fc498009900 tx=0x7fc49802bd60 comp rx=0 tx=0).stop 2024-04-21T23:19:34.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.937+0000 7fc4a3a84640 1 -- 172.21.15.40:0/948622470 shutdown_connections 2024-04-21T23:19:34.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.937+0000 7fc4a3a84640 1 --2- 172.21.15.40:0/948622470 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc49c10bd00 0x7fc49c10e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.937+0000 7fc4a3a84640 1 --2- 172.21.15.40:0/948622470 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc49c1032e0 0x7fc49c10b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.937+0000 7fc4a3a84640 1 --2- 172.21.15.40:0/948622470 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc49c1029c0 0x7fc49c102da0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.937+0000 7fc4a3a84640 1 -- 172.21.15.40:0/948622470 >> 172.21.15.40:0/948622470 conn(0x7fc49c077740 msgr2=0x7fc49c077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:34.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.937+0000 7fc4a3a84640 1 -- 172.21.15.40:0/948622470 shutdown_connections 2024-04-21T23:19:34.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.937+0000 7fc4a3a84640 1 -- 172.21.15.40:0/948622470 wait complete. 2024-04-21T23:19:34.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.938+0000 7fc4a3a84640 1 Processor -- start 2024-04-21T23:19:34.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.938+0000 7fc4a3a84640 1 -- start start 2024-04-21T23:19:34.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.938+0000 7fc4a3a84640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc49c1029c0 0x7fc49c197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:34.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.938+0000 7fc4a3a84640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc49c1032e0 0x7fc49c198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:34.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.938+0000 7fc4a3a84640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc49c10bd00 0x7fc49c19c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:34.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.938+0000 7fc4a3a84640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc49c111b50 con 0x7fc49c1029c0 2024-04-21T23:19:34.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.938+0000 7fc4a3a84640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc49c1119d0 con 0x7fc49c10bd00 2024-04-21T23:19:34.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.938+0000 7fc4a3a84640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc49c111cd0 con 0x7fc49c1032e0 2024-04-21T23:19:34.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.938+0000 7fc4a1820640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc49c1029c0 0x7fc49c197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:34.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.938+0000 7fc4a1820640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc49c1029c0 0x7fc49c197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48228/0 (socket says 172.21.15.40:48228) 2024-04-21T23:19:34.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.938+0000 7fc4a1820640 1 -- 172.21.15.40:0/2760268850 learned_addr learned my addr 172.21.15.40:0/2760268850 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:34.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.938+0000 7fc4a101f640 1 --2- 172.21.15.40:0/2760268850 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc49c1032e0 0x7fc49c198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:34.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.939+0000 7fc4a2021640 1 --2- 172.21.15.40:0/2760268850 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc49c10bd00 0x7fc49c19c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:34.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.939+0000 7fc4a1820640 1 -- 172.21.15.40:0/2760268850 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc49c1032e0 msgr2=0x7fc49c198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:34.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.939+0000 7fc4a1820640 1 --2- 172.21.15.40:0/2760268850 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc49c1032e0 0x7fc49c198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.939+0000 7fc4a1820640 1 -- 172.21.15.40:0/2760268850 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc49c10bd00 msgr2=0x7fc49c19c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:34.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.939+0000 7fc4a1820640 1 --2- 172.21.15.40:0/2760268850 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc49c10bd00 0x7fc49c19c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.939+0000 7fc4a1820640 1 -- 172.21.15.40:0/2760268850 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc49c19cfe0 con 0x7fc49c1029c0 2024-04-21T23:19:34.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.939+0000 7fc4a2021640 1 --2- 172.21.15.40:0/2760268850 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc49c10bd00 0x7fc49c19c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:19:34.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.939+0000 7fc4a1820640 1 --2- 172.21.15.40:0/2760268850 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc49c1029c0 0x7fc49c197ff0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fc498000c00 tx=0x7fc49802ddb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:34.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.939+0000 7fc492ffd640 1 -- 172.21.15.40:0/2760268850 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc498043020 con 0x7fc49c1029c0 2024-04-21T23:19:34.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.939+0000 7fc4a3a84640 1 -- 172.21.15.40:0/2760268850 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc49c19d270 con 0x7fc49c1029c0 2024-04-21T23:19:34.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.939+0000 7fc492ffd640 1 -- 172.21.15.40:0/2760268850 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4980054e0 con 0x7fc49c1029c0 2024-04-21T23:19:34.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.939+0000 7fc492ffd640 1 -- 172.21.15.40:0/2760268850 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc498003ce0 con 0x7fc49c1029c0 2024-04-21T23:19:34.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.939+0000 7fc4a3a84640 1 -- 172.21.15.40:0/2760268850 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc49c1a4b60 con 0x7fc49c1029c0 2024-04-21T23:19:34.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.942+0000 7fc492ffd640 1 -- 172.21.15.40:0/2760268850 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 25) ==== 138117+0+0 (secure 0 0 0) 0x7fc498046020 con 0x7fc49c1029c0 2024-04-21T23:19:34.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.942+0000 7fc4a666e740 1 Processor -- start 2024-04-21T23:19:34.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.942+0000 7fc4a666e740 1 -- start start 2024-04-21T23:19:34.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.943+0000 7fc4a666e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55edbdb2f020 0x55edbdb2f400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:34.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.943+0000 7fc4a666e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55edbdb2f940 0x55edbdb37fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:34.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.943+0000 7fc4a666e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55edbdb38520 0x55edbdb3a910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:34.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.943+0000 7fc4a666e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55edbda93530 con 0x55edbdb2f020 2024-04-21T23:19:34.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.943+0000 7fc4a666e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55edbda3b830 con 0x55edbdb2f940 2024-04-21T23:19:34.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.943+0000 7fc4a666e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55edbda8f4f0 con 0x55edbdb38520 2024-04-21T23:19:34.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.943+0000 7fc492ffd640 1 --2- 172.21.15.40:0/2760268850 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fc4880a2eb0 0x7fc4880a5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:34.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.943+0000 7fc4a2021640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55edbdb2f020 0x55edbdb2f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:34.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.943+0000 7fc4a2021640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55edbdb2f020 0x55edbdb2f400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48244/0 (socket says 172.21.15.40:48244) 2024-04-21T23:19:34.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.943+0000 7fc4a2021640 1 -- 172.21.15.40:0/4167262710 learned_addr learned my addr 172.21.15.40:0/4167262710 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:34.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.943+0000 7fc4a101f640 1 --2- 172.21.15.40:0/4167262710 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55edbdb38520 0x55edbdb3a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:34.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.943+0000 7fc492ffd640 1 -- 172.21.15.40:0/2760268850 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7fc4980f1cb0 con 0x7fc49c1029c0 2024-04-21T23:19:34.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.943+0000 7fc4a1820640 1 --2- 172.21.15.40:0/4167262710 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55edbdb2f940 0x55edbdb37fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:34.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.943+0000 7fc4a2021640 1 --2- 172.21.15.40:0/2760268850 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fc4880a2eb0 0x7fc4880a5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:34.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.944+0000 7fc4a101f640 1 -- 172.21.15.40:0/4167262710 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55edbdb2f940 msgr2=0x55edbdb37fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:34.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.944+0000 7fc4a101f640 1 --2- 172.21.15.40:0/4167262710 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55edbdb2f940 0x55edbdb37fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.944+0000 7fc4a101f640 1 -- 172.21.15.40:0/4167262710 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55edbdb2f020 msgr2=0x55edbdb2f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:34.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.944+0000 7fc4a101f640 1 --2- 172.21.15.40:0/4167262710 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55edbdb2f020 0x55edbdb2f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.944+0000 7fc4a101f640 1 -- 172.21.15.40:0/4167262710 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55edbda5f2c0 con 0x55edbdb38520 2024-04-21T23:19:34.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.944+0000 7fc4a1820640 1 --2- 172.21.15.40:0/4167262710 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55edbdb2f940 0x55edbdb37fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:19:34.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.944+0000 7fc4a2021640 1 --2- 172.21.15.40:0/2760268850 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fc4880a2eb0 0x7fc4880a5370 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc4940092f0 tx=0x7fc4940099c0 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:34.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.944+0000 7fc4a101f640 1 --2- 172.21.15.40:0/4167262710 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55edbdb38520 0x55edbdb3a910 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc48c00bbb0 tx=0x7fc48c00f360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e5ffa9a2cfb8ee87 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:34.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.944+0000 7fc487fff640 1 -- 172.21.15.40:0/4167262710 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc48c015070 con 0x55edbdb38520 2024-04-21T23:19:34.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.944+0000 7fc487fff640 1 -- 172.21.15.40:0/4167262710 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc48c00fd90 con 0x55edbdb38520 2024-04-21T23:19:34.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.944+0000 7fc487fff640 1 -- 172.21.15.40:0/4167262710 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc48c011440 con 0x55edbdb38520 2024-04-21T23:19:34.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.945+0000 7fc4a666e740 1 -- 172.21.15.40:0/4167262710 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55edbdb38520 msgr2=0x55edbdb3a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:34.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.945+0000 7fc4a666e740 1 --2- 172.21.15.40:0/4167262710 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55edbdb38520 0x55edbdb3a910 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc48c00bbb0 tx=0x7fc48c00f360 comp rx=0 tx=0).stop 2024-04-21T23:19:34.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.945+0000 7fc4a666e740 1 -- 172.21.15.40:0/4167262710 shutdown_connections 2024-04-21T23:19:34.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.945+0000 7fc4a666e740 1 --2- 172.21.15.40:0/4167262710 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55edbdb2f940 0x55edbdb37fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.945+0000 7fc4a666e740 1 --2- 172.21.15.40:0/4167262710 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55edbdb38520 0x55edbdb3a910 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.945+0000 7fc4a666e740 1 --2- 172.21.15.40:0/4167262710 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55edbdb2f020 0x55edbdb2f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.945+0000 7fc4a666e740 1 -- 172.21.15.40:0/4167262710 >> 172.21.15.40:0/4167262710 conn(0x55edbdb29750 msgr2=0x55edbdb29b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:34.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.945+0000 7fc4a666e740 1 -- 172.21.15.40:0/4167262710 shutdown_connections 2024-04-21T23:19:34.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.945+0000 7fc4a666e740 1 -- 172.21.15.40:0/4167262710 wait complete. 2024-04-21T23:19:34.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.945+0000 7fc4a666e740 1 -- start start 2024-04-21T23:19:34.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.945+0000 7fc4a666e740 1 Processor -- start 2024-04-21T23:19:34.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.946+0000 7fc4a666e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55edbdb2f020 0x55edbdb37ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:34.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.946+0000 7fc4a666e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55edbdb2f940 0x55edbdb35260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:34.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.946+0000 7fc4a666e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55edbdb38520 0x55edbdb35800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:34.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.946+0000 7fc4a666e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55edbdb3f720 con 0x55edbdb2f020 2024-04-21T23:19:34.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.946+0000 7fc4a666e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55edbda4ef10 con 0x55edbdb2f940 2024-04-21T23:19:34.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.946+0000 7fc4a666e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55edbdb3f8a0 con 0x55edbdb38520 2024-04-21T23:19:34.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.946+0000 7fc4a666e740 10 client.0 _ll_register_callbacks cb 0x55edbd7a6be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:19:34.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.946+0000 7fc4a1820640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55edbdb2f020 0x55edbdb37ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:34.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.946+0000 7fc4a1820640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55edbdb2f020 0x55edbdb37ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48258/0 (socket says 172.21.15.40:48258) 2024-04-21T23:19:34.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.946+0000 7fc4a1820640 1 -- 172.21.15.40:0/3374152422 learned_addr learned my addr 172.21.15.40:0/3374152422 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:34.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.946+0000 7fc4a101f640 1 --2- 172.21.15.40:0/3374152422 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55edbdb2f940 0x55edbdb35260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:34.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.946+0000 7fc4a2021640 1 --2- 172.21.15.40:0/3374152422 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55edbdb38520 0x55edbdb35800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:34.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.946+0000 7fc4a1820640 1 -- 172.21.15.40:0/3374152422 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55edbdb38520 msgr2=0x55edbdb35800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:34.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.946+0000 7fc4a1820640 1 --2- 172.21.15.40:0/3374152422 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55edbdb38520 0x55edbdb35800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.946+0000 7fc4a1820640 1 -- 172.21.15.40:0/3374152422 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55edbdb2f940 msgr2=0x55edbdb35260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:34.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.946+0000 7fc4a1820640 1 --2- 172.21.15.40:0/3374152422 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55edbdb2f940 0x55edbdb35260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.946+0000 7fc4a1820640 1 -- 172.21.15.40:0/3374152422 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55edbda20c80 con 0x55edbdb2f020 2024-04-21T23:19:34.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.946+0000 7fc4a2021640 1 --2- 172.21.15.40:0/3374152422 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55edbdb38520 0x55edbdb35800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:19:34.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.947+0000 7fc4a1820640 1 --2- 172.21.15.40:0/3374152422 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55edbdb2f020 0x55edbdb37ed0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fc4980ba150 tx=0x7fc4980bdb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=440ff8aa58de2b8d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:34.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.947+0000 7fc4857fa640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-21T23:19:34.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.947+0000 7fc4857fa640 1 -- 172.21.15.40:0/3374152422 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4980044a0 con 0x55edbdb2f020 2024-04-21T23:19:34.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.947+0000 7fc4857fa640 1 -- 172.21.15.40:0/3374152422 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7fc4980bdf50 con 0x55edbdb2f020 2024-04-21T23:19:34.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.947+0000 7fc4a666e740 1 -- 172.21.15.40:0/3374152422 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc48c00fd90 con 0x55edbdb2f020 2024-04-21T23:19:34.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.947+0000 7fc4857fa640 1 -- 172.21.15.40:0/3374152422 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fc498005690 con 0x55edbdb2f020 2024-04-21T23:19:34.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.947+0000 7fc4a666e740 10 client.15969 fetch_fsmap learned FSMap version 25 2024-04-21T23:19:34.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.947+0000 7fc4a666e740 1 -- 172.21.15.40:0/3374152422 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55edbd7a73f0 con 0x55edbdb2f020 2024-04-21T23:19:34.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.947+0000 7fc4857fa640 1 -- 172.21.15.40:0/3374152422 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc48c00fd90 con 0x55edbdb2f020 2024-04-21T23:19:34.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.947+0000 7fc4857fa640 1 -- 172.21.15.40:0/3374152422 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fc498086050 con 0x55edbdb2f020 2024-04-21T23:19:34.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.947+0000 7fc4a666e740 10 client.15969 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:19:34.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.947+0000 7fc4a666e740 10 client.15969 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T23:19:34.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.947+0000 7fc4a666e740 20 client.15969 populate_metadata read hostname 'smithi040' 2024-04-21T23:19:34.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.948+0000 7fc4a666e740 1 --2- 172.21.15.40:0/3374152422 >> [v2:172.21.15.174:6838/2467509274,v1:172.21.15.174:6839/2467509274] conn(0x55edbdb4cb90 0x55edbdb4f000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:34.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.948+0000 7fc4a666e740 4 client.15969 mds_command: new command op to 24502 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:19:34.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.948+0000 7fc4a666e740 1 -- 172.21.15.40:0/3374152422 --> [v2:172.21.15.174:6838/2467509274,v1:172.21.15.174:6839/2467509274] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55edbdaa0a70 con 0x55edbdb4cb90 2024-04-21T23:19:34.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.948+0000 7fc4a101f640 1 --2- 172.21.15.40:0/3374152422 >> [v2:172.21.15.174:6838/2467509274,v1:172.21.15.174:6839/2467509274] conn(0x55edbdb4cb90 0x55edbdb4f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:34.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.948+0000 7fc4857fa640 10 client.15969 ms_handle_connect on v2:172.21.15.174:6838/2467509274 2024-04-21T23:19:34.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.948+0000 7fc4a101f640 1 --2- 172.21.15.40:0/3374152422 >> [v2:172.21.15.174:6838/2467509274,v1:172.21.15.174:6839/2467509274] conn(0x55edbdb4cb90 0x55edbdb4f000 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b09578bc07ffdcaa server_cookie=eb8d6bb0f50077eb in_seq=0 out_seq=0 2024-04-21T23:19:34.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.949+0000 7fc4857fa640 1 -- 172.21.15.40:0/3374152422 <== mds.1 v2:172.21.15.174:6838/2467509274 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55edbdaa0a70 con 0x55edbdb4cb90 2024-04-21T23:19:34.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.949+0000 7fc4857fa640 10 client.15969 handle_command_reply: tid=0 2024-04-21T23:19:34.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.949+0000 7fc4a666e740 1 client.15969 shutdown 2024-04-21T23:19:34.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.949+0000 7fc4a666e740 1 -- 172.21.15.40:0/3374152422 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55edbdb2f020 msgr2=0x55edbdb37ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:34.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.949+0000 7fc4a666e740 1 --2- 172.21.15.40:0/3374152422 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55edbdb2f020 0x55edbdb37ed0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fc4980ba150 tx=0x7fc4980bdb80 comp rx=0 tx=0).stop 2024-04-21T23:19:34.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.949+0000 7fc4a666e740 1 -- 172.21.15.40:0/3374152422 shutdown_connections 2024-04-21T23:19:34.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.949+0000 7fc4a666e740 1 --2- 172.21.15.40:0/3374152422 >> [v2:172.21.15.174:6838/2467509274,v1:172.21.15.174:6839/2467509274] conn(0x55edbdb4cb90 0x55edbdb4f000 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.949+0000 7fc4a666e740 1 --2- 172.21.15.40:0/3374152422 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55edbdb2f940 0x55edbdb35260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.949+0000 7fc4a666e740 1 --2- 172.21.15.40:0/3374152422 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55edbdb38520 0x55edbdb35800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.949+0000 7fc4a666e740 1 --2- 172.21.15.40:0/3374152422 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55edbdb2f020 0x55edbdb37ed0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.949+0000 7fc4a666e740 1 -- 172.21.15.40:0/3374152422 >> 172.21.15.40:0/3374152422 conn(0x55edbdb29750 msgr2=0x55edbdb2b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:34.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.949+0000 7fc4857fa640 0 client.15969 ms_handle_reset on v2:172.21.15.174:6838/2467509274 2024-04-21T23:19:34.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.949+0000 7fc4a666e740 1 -- 172.21.15.40:0/3374152422 shutdown_connections 2024-04-21T23:19:34.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.949+0000 7fc4a666e740 1 -- 172.21.15.40:0/3374152422 wait complete. 2024-04-21T23:19:34.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.950+0000 7fc4a666e740 20 client.15969 trim_cache size 0 max 16384 2024-04-21T23:19:34.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.963+0000 7fc4a666e740 1 Processor -- start 2024-04-21T23:19:34.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.963+0000 7fc4a666e740 1 -- start start 2024-04-21T23:19:34.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.963+0000 7fc4a666e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55edbdb2f020 0x55edbdb485e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:34.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.963+0000 7fc4a666e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55edbdb2f940 0x55edbdb286d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:34.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.963+0000 7fc4a666e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55edbdb38520 0x55edbdb37990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:34.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.963+0000 7fc4a666e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc49c1af2b0 con 0x55edbdb2f020 2024-04-21T23:19:34.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.963+0000 7fc4a666e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc49c1acb80 con 0x55edbdb2f940 2024-04-21T23:19:34.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.963+0000 7fc4a666e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc49c1aa9d0 con 0x55edbdb38520 2024-04-21T23:19:34.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.963+0000 7fc4a1820640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55edbdb2f020 0x55edbdb485e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:34.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.963+0000 7fc4a1820640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55edbdb2f020 0x55edbdb485e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48274/0 (socket says 172.21.15.40:48274) 2024-04-21T23:19:34.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.963+0000 7fc4a1820640 1 -- 172.21.15.40:0/1489592926 learned_addr learned my addr 172.21.15.40:0/1489592926 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:34.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.964+0000 7fc4a101f640 1 --2- 172.21.15.40:0/1489592926 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55edbdb2f940 0x55edbdb286d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:34.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.964+0000 7fc4a2021640 1 --2- 172.21.15.40:0/1489592926 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55edbdb38520 0x55edbdb37990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:34.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.964+0000 7fc4a1820640 1 -- 172.21.15.40:0/1489592926 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55edbdb38520 msgr2=0x55edbdb37990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:34.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.964+0000 7fc4a1820640 1 --2- 172.21.15.40:0/1489592926 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55edbdb38520 0x55edbdb37990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.964+0000 7fc4a1820640 1 -- 172.21.15.40:0/1489592926 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55edbdb2f940 msgr2=0x55edbdb286d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:34.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.964+0000 7fc4a1820640 1 --2- 172.21.15.40:0/1489592926 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55edbdb2f940 0x55edbdb286d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.964+0000 7fc4a1820640 1 -- 172.21.15.40:0/1489592926 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55edbdb2f770 con 0x55edbdb2f020 2024-04-21T23:19:34.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.964+0000 7fc4a2021640 1 --2- 172.21.15.40:0/1489592926 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55edbdb38520 0x55edbdb37990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:19:34.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.964+0000 7fc4a1820640 1 --2- 172.21.15.40:0/1489592926 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55edbdb2f020 0x55edbdb485e0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fc4980ef060 tx=0x7fc498005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f92be1d37b42535c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:34.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.964+0000 7fc4877fe640 1 -- 172.21.15.40:0/1489592926 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4980044a0 con 0x55edbdb2f020 2024-04-21T23:19:34.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.964+0000 7fc4877fe640 1 -- 172.21.15.40:0/1489592926 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4980a0020 con 0x55edbdb2f020 2024-04-21T23:19:34.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.964+0000 7fc4877fe640 1 -- 172.21.15.40:0/1489592926 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4980be3f0 con 0x55edbdb2f020 2024-04-21T23:19:34.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.965+0000 7fc4a666e740 1 -- 172.21.15.40:0/1489592926 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55edbdb2f020 msgr2=0x55edbdb485e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:34.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.965+0000 7fc4a666e740 1 --2- 172.21.15.40:0/1489592926 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55edbdb2f020 0x55edbdb485e0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fc4980ef060 tx=0x7fc498005b50 comp rx=0 tx=0).stop 2024-04-21T23:19:34.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.965+0000 7fc4a666e740 1 -- 172.21.15.40:0/1489592926 shutdown_connections 2024-04-21T23:19:34.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.965+0000 7fc4a666e740 1 --2- 172.21.15.40:0/1489592926 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55edbdb2f940 0x55edbdb286d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.965+0000 7fc4a666e740 1 --2- 172.21.15.40:0/1489592926 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55edbdb38520 0x55edbdb37990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.965+0000 7fc4a666e740 1 --2- 172.21.15.40:0/1489592926 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55edbdb2f020 0x55edbdb485e0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.965+0000 7fc4a666e740 1 -- 172.21.15.40:0/1489592926 >> 172.21.15.40:0/1489592926 conn(0x55edbdb29750 msgr2=0x55edbdb27fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:34.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.965+0000 7fc4a666e740 1 -- 172.21.15.40:0/1489592926 shutdown_connections 2024-04-21T23:19:34.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.965+0000 7fc4a666e740 1 -- 172.21.15.40:0/1489592926 wait complete. 2024-04-21T23:19:34.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.965+0000 7fc4a666e740 1 -- start start 2024-04-21T23:19:34.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.966+0000 7fc4a666e740 1 Processor -- start 2024-04-21T23:19:34.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.966+0000 7fc4a666e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55edbdb2f940 0x55edbdb44a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:34.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.966+0000 7fc4a666e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55edbdb38520 0x55edbdb44fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:34.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.966+0000 7fc4a666e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55edbdb3ff30 0x55edbdb40390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:34.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.966+0000 7fc4a666e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc49c1ab040 con 0x55edbdb3ff30 2024-04-21T23:19:34.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.966+0000 7fc4a666e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc49c1ac4b0 con 0x55edbdb2f940 2024-04-21T23:19:34.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.966+0000 7fc4a666e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc49c1ab710 con 0x55edbdb38520 2024-04-21T23:19:34.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.966+0000 7fc4a666e740 10 client.0 _ll_register_callbacks cb 0x7fc498003ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:19:34.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.966+0000 7fc4a2021640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55edbdb3ff30 0x55edbdb40390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:34.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.966+0000 7fc4a2021640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55edbdb3ff30 0x55edbdb40390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48284/0 (socket says 172.21.15.40:48284) 2024-04-21T23:19:34.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.966+0000 7fc4a2021640 1 -- 172.21.15.40:0/453543783 learned_addr learned my addr 172.21.15.40:0/453543783 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:34.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.966+0000 7fc4a1820640 1 --2- 172.21.15.40:0/453543783 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55edbdb2f940 0x55edbdb44a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:34.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.966+0000 7fc4a101f640 1 --2- 172.21.15.40:0/453543783 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55edbdb38520 0x55edbdb44fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:34.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.967+0000 7fc4a2021640 1 -- 172.21.15.40:0/453543783 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55edbdb38520 msgr2=0x55edbdb44fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:34.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.967+0000 7fc4a2021640 1 --2- 172.21.15.40:0/453543783 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55edbdb38520 0x55edbdb44fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.967+0000 7fc4a2021640 1 -- 172.21.15.40:0/453543783 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55edbdb2f940 msgr2=0x55edbdb44a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:34.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.967+0000 7fc4a2021640 1 --2- 172.21.15.40:0/453543783 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55edbdb2f940 0x55edbdb44a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.967+0000 7fc4a2021640 1 -- 172.21.15.40:0/453543783 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55edbd8717d0 con 0x55edbdb3ff30 2024-04-21T23:19:34.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.967+0000 7fc4a2021640 1 --2- 172.21.15.40:0/453543783 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55edbdb3ff30 0x55edbdb40390 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fc494011db0 tx=0x7fc494009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a96464735675481 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:34.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.967+0000 7fc4857fa640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-21T23:19:34.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.967+0000 7fc4857fa640 1 -- 172.21.15.40:0/453543783 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc49402c060 con 0x55edbdb3ff30 2024-04-21T23:19:34.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.967+0000 7fc4857fa640 1 -- 172.21.15.40:0/453543783 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7fc4940273f0 con 0x55edbdb3ff30 2024-04-21T23:19:34.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.967+0000 7fc4a666e740 1 -- 172.21.15.40:0/453543783 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc4980a0020 con 0x55edbdb3ff30 2024-04-21T23:19:34.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.967+0000 7fc4857fa640 1 -- 172.21.15.40:0/453543783 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fc494033050 con 0x55edbdb3ff30 2024-04-21T23:19:34.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.967+0000 7fc4a666e740 10 client.15975 fetch_fsmap learned FSMap version 25 2024-04-21T23:19:34.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.967+0000 7fc4a666e740 1 -- 172.21.15.40:0/453543783 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55edbdbb2eb0 con 0x55edbdb3ff30 2024-04-21T23:19:34.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.968+0000 7fc4857fa640 1 -- 172.21.15.40:0/453543783 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4980a0020 con 0x55edbdb3ff30 2024-04-21T23:19:34.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.968+0000 7fc4857fa640 1 -- 172.21.15.40:0/453543783 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fc494026030 con 0x55edbdb3ff30 2024-04-21T23:19:34.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.968+0000 7fc4a666e740 10 client.15975 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:19:34.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.968+0000 7fc4a666e740 10 client.15975 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T23:19:34.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.968+0000 7fc4a666e740 20 client.15975 populate_metadata read hostname 'smithi040' 2024-04-21T23:19:34.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.968+0000 7fc4a666e740 1 --2- 172.21.15.40:0/453543783 >> [v2:172.21.15.174:6838/2467509274,v1:172.21.15.174:6839/2467509274] conn(0x55edbdbb3600 0x55edbdbb5a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:34.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.968+0000 7fc4a666e740 4 client.15975 mds_command: new command op to 24502 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T23:19:34.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.968+0000 7fc4a666e740 1 -- 172.21.15.40:0/453543783 --> [v2:172.21.15.174:6838/2467509274,v1:172.21.15.174:6839/2467509274] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55edbda9adc0 con 0x55edbdbb3600 2024-04-21T23:19:34.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.968+0000 7fc4a1820640 1 --2- 172.21.15.40:0/453543783 >> [v2:172.21.15.174:6838/2467509274,v1:172.21.15.174:6839/2467509274] conn(0x55edbdbb3600 0x55edbdbb5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:34.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.969+0000 7fc4a1820640 1 --2- 172.21.15.40:0/453543783 >> [v2:172.21.15.174:6838/2467509274,v1:172.21.15.174:6839/2467509274] conn(0x55edbdbb3600 0x55edbdbb5a20 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5d095173a9614d90 server_cookie=90b07ebda3e1fb3c in_seq=0 out_seq=0 2024-04-21T23:19:34.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.969+0000 7fc4857fa640 10 client.15975 ms_handle_connect on v2:172.21.15.174:6838/2467509274 2024-04-21T23:19:34.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.969+0000 7fc4857fa640 1 -- 172.21.15.40:0/453543783 <== mds.1 v2:172.21.15.174:6838/2467509274 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55edbda9adc0 con 0x55edbdbb3600 2024-04-21T23:19:34.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.969+0000 7fc4857fa640 10 client.15975 handle_command_reply: tid=0 2024-04-21T23:19:34.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.969+0000 7fc4a666e740 1 client.15975 shutdown 2024-04-21T23:19:34.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.969+0000 7fc4a666e740 1 -- 172.21.15.40:0/453543783 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55edbdb3ff30 msgr2=0x55edbdb40390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:34.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.969+0000 7fc4a666e740 1 --2- 172.21.15.40:0/453543783 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55edbdb3ff30 0x55edbdb40390 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fc494011db0 tx=0x7fc494009680 comp rx=0 tx=0).stop 2024-04-21T23:19:34.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.969+0000 7fc4a666e740 1 -- 172.21.15.40:0/453543783 shutdown_connections 2024-04-21T23:19:34.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.969+0000 7fc4a666e740 1 --2- 172.21.15.40:0/453543783 >> [v2:172.21.15.174:6838/2467509274,v1:172.21.15.174:6839/2467509274] conn(0x55edbdbb3600 0x55edbdbb5a20 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.969+0000 7fc4a666e740 1 --2- 172.21.15.40:0/453543783 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55edbdb3ff30 0x55edbdb40390 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.969+0000 7fc4a666e740 1 --2- 172.21.15.40:0/453543783 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55edbdb38520 0x55edbdb44fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.969+0000 7fc4857fa640 0 client.15975 ms_handle_reset on v2:172.21.15.174:6838/2467509274 2024-04-21T23:19:34.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.969+0000 7fc4a666e740 1 --2- 172.21.15.40:0/453543783 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55edbdb2f940 0x55edbdb44a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.969+0000 7fc4a666e740 1 -- 172.21.15.40:0/453543783 >> 172.21.15.40:0/453543783 conn(0x55edbdb29750 msgr2=0x55edbdb28850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:34.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.969+0000 7fc4a666e740 1 -- 172.21.15.40:0/453543783 shutdown_connections 2024-04-21T23:19:34.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.969+0000 7fc4a666e740 1 -- 172.21.15.40:0/453543783 wait complete. 2024-04-21T23:19:34.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.970+0000 7fc4a666e740 20 client.15975 trim_cache size 0 max 16384 2024-04-21T23:19:34.974 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-04-21T23:19:34.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.970+0000 7fc4a3a84640 1 -- 172.21.15.40:0/2760268850 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fc4880a2eb0 msgr2=0x7fc4880a5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:34.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.970+0000 7fc4a3a84640 1 --2- 172.21.15.40:0/2760268850 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fc4880a2eb0 0x7fc4880a5370 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc4940092f0 tx=0x7fc4940099c0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.970+0000 7fc4a3a84640 1 -- 172.21.15.40:0/2760268850 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc49c1029c0 msgr2=0x7fc49c197ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:34.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.970+0000 7fc4a3a84640 1 --2- 172.21.15.40:0/2760268850 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc49c1029c0 0x7fc49c197ff0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fc498000c00 tx=0x7fc49802ddb0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.970+0000 7fc4a3a84640 1 -- 172.21.15.40:0/2760268850 shutdown_connections 2024-04-21T23:19:34.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.970+0000 7fc4a3a84640 1 --2- 172.21.15.40:0/2760268850 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fc4880a2eb0 0x7fc4880a5370 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.970+0000 7fc4a3a84640 1 --2- 172.21.15.40:0/2760268850 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc49c10bd00 0x7fc49c19c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.970+0000 7fc4a3a84640 1 --2- 172.21.15.40:0/2760268850 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc49c1032e0 0x7fc49c198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.970+0000 7fc4a3a84640 1 --2- 172.21.15.40:0/2760268850 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc49c1029c0 0x7fc49c197ff0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:34.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.970+0000 7fc4a3a84640 1 -- 172.21.15.40:0/2760268850 >> 172.21.15.40:0/2760268850 conn(0x7fc49c077740 msgr2=0x7fc49c0fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:34.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.970+0000 7fc4a3a84640 1 -- 172.21.15.40:0/2760268850 shutdown_connections 2024-04-21T23:19:34.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:34.970+0000 7fc4a3a84640 1 -- 172.21.15.40:0/2760268850 wait complete. 2024-04-21T23:19:34.981 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T23:19:35.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.088+0000 7f7409235640 1 -- 172.21.15.40:0/1296568135 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7404100e60 msgr2=0x7f740410dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:35.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.088+0000 7f7409235640 1 --2- 172.21.15.40:0/1296568135 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7404100e60 0x7f740410dbb0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f73f8009900 tx=0x7f73f802bc30 comp rx=0 tx=0).stop 2024-04-21T23:19:35.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.088+0000 7f7409235640 1 -- 172.21.15.40:0/1296568135 shutdown_connections 2024-04-21T23:19:35.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.088+0000 7f7409235640 1 --2- 172.21.15.40:0/1296568135 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f740410e0f0 0x7f74041104e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:35.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.088+0000 7f7409235640 1 --2- 172.21.15.40:0/1296568135 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7404100e60 0x7f740410dbb0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:35.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.088+0000 7f7409235640 1 --2- 172.21.15.40:0/1296568135 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7404100540 0x7f7404100920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:35.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.088+0000 7f7409235640 1 -- 172.21.15.40:0/1296568135 >> 172.21.15.40:0/1296568135 conn(0x7f7404077740 msgr2=0x7f7404077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:35.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.088+0000 7f7409235640 1 -- 172.21.15.40:0/1296568135 shutdown_connections 2024-04-21T23:19:35.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.088+0000 7f7409235640 1 -- 172.21.15.40:0/1296568135 wait complete. 2024-04-21T23:19:35.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.089+0000 7f7409235640 1 Processor -- start 2024-04-21T23:19:35.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.089+0000 7f7409235640 1 -- start start 2024-04-21T23:19:35.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.089+0000 7f7409235640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7404100540 0x7f740419a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:35.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.089+0000 7f7409235640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7404100e60 0x7f740419a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:35.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.089+0000 7f7409235640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f740410e0f0 0x7f740419ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:35.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.089+0000 7f7409235640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7404113d40 con 0x7f740410e0f0 2024-04-21T23:19:35.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.089+0000 7f7409235640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f7404113bc0 con 0x7f7404100e60 2024-04-21T23:19:35.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.089+0000 7f7409235640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7404113ec0 con 0x7f7404100540 2024-04-21T23:19:35.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.089+0000 7f740359e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f740410e0f0 0x7f740419ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:35.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.089+0000 7f740359e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f740410e0f0 0x7f740419ead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48296/0 (socket says 172.21.15.40:48296) 2024-04-21T23:19:35.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.089+0000 7f740359e640 1 -- 172.21.15.40:0/1896824405 learned_addr learned my addr 172.21.15.40:0/1896824405 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:35.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.089+0000 7f740259c640 1 --2- 172.21.15.40:0/1896824405 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7404100e60 0x7f740419a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:35.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.089+0000 7f7402d9d640 1 --2- 172.21.15.40:0/1896824405 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7404100540 0x7f740419a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:35.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.090+0000 7f740359e640 1 -- 172.21.15.40:0/1896824405 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7404100540 msgr2=0x7f740419a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:35.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.090+0000 7f740359e640 1 --2- 172.21.15.40:0/1896824405 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7404100540 0x7f740419a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:35.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.090+0000 7f740359e640 1 -- 172.21.15.40:0/1896824405 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7404100e60 msgr2=0x7f740419a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:35.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.090+0000 7f740359e640 1 --2- 172.21.15.40:0/1896824405 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7404100e60 0x7f740419a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:35.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.090+0000 7f740359e640 1 -- 172.21.15.40:0/1896824405 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f740419f200 con 0x7f740410e0f0 2024-04-21T23:19:35.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.090+0000 7f740359e640 1 --2- 172.21.15.40:0/1896824405 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f740410e0f0 0x7f740419ead0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f73fc00eb40 tx=0x7f73fc00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:35.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.090+0000 7f73ebfff640 1 -- 172.21.15.40:0/1896824405 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73fc004b20 con 0x7f740410e0f0 2024-04-21T23:19:35.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.090+0000 7f7409235640 1 -- 172.21.15.40:0/1896824405 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f740419f4f0 con 0x7f740410e0f0 2024-04-21T23:19:35.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.090+0000 7f7409235640 1 -- 172.21.15.40:0/1896824405 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74041a6d80 con 0x7f740410e0f0 2024-04-21T23:19:35.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.091+0000 7f73ebfff640 1 -- 172.21.15.40:0/1896824405 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f73fc004cc0 con 0x7f740410e0f0 2024-04-21T23:19:35.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.091+0000 7f73ebfff640 1 -- 172.21.15.40:0/1896824405 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73fc010850 con 0x7f740410e0f0 2024-04-21T23:19:35.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.094+0000 7f73ebfff640 1 -- 172.21.15.40:0/1896824405 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 25) ==== 138117+0+0 (secure 0 0 0) 0x7f73fc0109f0 con 0x7f740410e0f0 2024-04-21T23:19:35.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.094+0000 7f740be1e740 1 Processor -- start 2024-04-21T23:19:35.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.094+0000 7f740be1e740 1 -- start start 2024-04-21T23:19:35.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.094+0000 7f740be1e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55eb62c1a020 0x55eb62c1a400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:35.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.094+0000 7f740be1e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eb62c1a940 0x55eb62c22fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:35.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.094+0000 7f740be1e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eb62c23520 0x55eb62c25910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:35.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.094+0000 7f740be1e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55eb62b7e530 con 0x55eb62c23520 2024-04-21T23:19:35.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.094+0000 7f740be1e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55eb62b26830 con 0x55eb62c1a940 2024-04-21T23:19:35.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.094+0000 7f740be1e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55eb62b7a4f0 con 0x55eb62c1a020 2024-04-21T23:19:35.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.094+0000 7f7402d9d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eb62c1a940 0x55eb62c22fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:35.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.094+0000 7f73ebfff640 1 --2- 172.21.15.40:0/1896824405 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f73ec0a2fb0 0x7f73ec0a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:35.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.094+0000 7f7402d9d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eb62c1a940 0x55eb62c22fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.40:53882/0 (socket says 172.21.15.40:53882) 2024-04-21T23:19:35.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.095+0000 7f7402d9d640 1 -- 172.21.15.40:0/2243318930 learned_addr learned my addr 172.21.15.40:0/2243318930 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:35.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.095+0000 7f740259c640 1 --2- 172.21.15.40:0/2243318930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eb62c23520 0x55eb62c25910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:35.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.095+0000 7f740259c640 1 --2- 172.21.15.40:0/2243318930 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55eb62c1a020 0x55eb62c1a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:35.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.095+0000 7f73ebfff640 1 -- 172.21.15.40:0/1896824405 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f73fc0d0050 con 0x7f740410e0f0 2024-04-21T23:19:35.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.095+0000 7f740359e640 1 --2- 172.21.15.40:0/1896824405 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f73ec0a2fb0 0x7f73ec0a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:35.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.095+0000 7f7402d9d640 1 -- 172.21.15.40:0/2243318930 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55eb62c1a020 msgr2=0x55eb62c1a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:35.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.095+0000 7f7402d9d640 1 --2- 172.21.15.40:0/2243318930 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55eb62c1a020 0x55eb62c1a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:35.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.095+0000 7f7402d9d640 1 -- 172.21.15.40:0/2243318930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eb62c23520 msgr2=0x55eb62c25910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:35.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.095+0000 7f7402d9d640 1 --2- 172.21.15.40:0/2243318930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eb62c23520 0x55eb62c25910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:35.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.095+0000 7f7402d9d640 1 -- 172.21.15.40:0/2243318930 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eb62b4a2c0 con 0x55eb62c1a940 2024-04-21T23:19:35.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.095+0000 7f7402d9d640 1 --2- 172.21.15.40:0/2243318930 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eb62c1a940 0x55eb62c22fe0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f73f4009cf0 tx=0x7f73f4011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=59d3de63e20cd7a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:35.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.095+0000 7f73e97fa640 1 -- 172.21.15.40:0/2243318930 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73f4004230 con 0x55eb62c1a940 2024-04-21T23:19:35.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.095+0000 7f73e97fa640 1 -- 172.21.15.40:0/2243318930 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f73f40043d0 con 0x55eb62c1a940 2024-04-21T23:19:35.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.095+0000 7f73e97fa640 1 -- 172.21.15.40:0/2243318930 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73f4019960 con 0x55eb62c1a940 2024-04-21T23:19:35.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.096+0000 7f740be1e740 1 -- 172.21.15.40:0/2243318930 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eb62c1a940 msgr2=0x55eb62c22fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:35.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.096+0000 7f740be1e740 1 --2- 172.21.15.40:0/2243318930 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eb62c1a940 0x55eb62c22fe0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f73f4009cf0 tx=0x7f73f4011810 comp rx=0 tx=0).stop 2024-04-21T23:19:35.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.096+0000 7f740359e640 1 --2- 172.21.15.40:0/1896824405 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f73ec0a2fb0 0x7f73ec0a5470 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f73fc0cd210 tx=0x7f73fc011000 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:35.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.096+0000 7f740be1e740 1 -- 172.21.15.40:0/2243318930 shutdown_connections 2024-04-21T23:19:35.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.096+0000 7f740be1e740 1 --2- 172.21.15.40:0/2243318930 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eb62c1a940 0x55eb62c22fe0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:35.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.096+0000 7f740be1e740 1 --2- 172.21.15.40:0/2243318930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eb62c23520 0x55eb62c25910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:35.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.096+0000 7f740be1e740 1 --2- 172.21.15.40:0/2243318930 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55eb62c1a020 0x55eb62c1a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:35.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.096+0000 7f740be1e740 1 -- 172.21.15.40:0/2243318930 >> 172.21.15.40:0/2243318930 conn(0x55eb62c14750 msgr2=0x55eb62c14b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:35.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.096+0000 7f740be1e740 1 -- 172.21.15.40:0/2243318930 shutdown_connections 2024-04-21T23:19:35.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.096+0000 7f740be1e740 1 -- 172.21.15.40:0/2243318930 wait complete. 2024-04-21T23:19:35.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.097+0000 7f740be1e740 1 -- start start 2024-04-21T23:19:35.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.097+0000 7f740be1e740 1 Processor -- start 2024-04-21T23:19:35.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.097+0000 7f740be1e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eb62c1a020 0x55eb62c22ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:35.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.097+0000 7f740be1e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eb62c1a940 0x55eb62c20280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:35.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.097+0000 7f740be1e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55eb62c23520 0x55eb62c20890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:35.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.097+0000 7f740be1e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55eb62c26310 con 0x55eb62c1a940 2024-04-21T23:19:35.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.097+0000 7f740be1e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55eb62b39f10 con 0x55eb62c1a020 2024-04-21T23:19:35.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.097+0000 7f740be1e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55eb62c26490 con 0x55eb62c23520 2024-04-21T23:19:35.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.097+0000 7f740be1e740 10 client.0 _ll_register_callbacks cb 0x55eb62891be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:19:35.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.097+0000 7f7402d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eb62c1a940 0x55eb62c20280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:35.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.097+0000 7f7402d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eb62c1a940 0x55eb62c20280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48314/0 (socket says 172.21.15.40:48314) 2024-04-21T23:19:35.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.098+0000 7f7402d9d640 1 -- 172.21.15.40:0/1287695495 learned_addr learned my addr 172.21.15.40:0/1287695495 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:35.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.098+0000 7f740259c640 1 --2- 172.21.15.40:0/1287695495 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eb62c1a020 0x55eb62c22ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:35.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.098+0000 7f740259c640 1 --2- 172.21.15.40:0/1287695495 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55eb62c23520 0x55eb62c20890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:35.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.098+0000 7f7402d9d640 1 -- 172.21.15.40:0/1287695495 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55eb62c23520 msgr2=0x55eb62c20890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:35.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.098+0000 7f7402d9d640 1 --2- 172.21.15.40:0/1287695495 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55eb62c23520 0x55eb62c20890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:35.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.098+0000 7f7402d9d640 1 -- 172.21.15.40:0/1287695495 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eb62c1a020 msgr2=0x55eb62c22ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:35.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.098+0000 7f7402d9d640 1 --2- 172.21.15.40:0/1287695495 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eb62c1a020 0x55eb62c22ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:35.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.098+0000 7f7402d9d640 1 -- 172.21.15.40:0/1287695495 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eb62b0bc80 con 0x55eb62c1a940 2024-04-21T23:19:35.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.098+0000 7f7402d9d640 1 --2- 172.21.15.40:0/1287695495 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eb62c1a940 0x55eb62c20280 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f73f4003e00 tx=0x7f73f4003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=caf7c3c774d10eda server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:35.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.098+0000 7f73e6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-21T23:19:35.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.098+0000 7f73e6ff5640 1 -- 172.21.15.40:0/1287695495 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73f4027070 con 0x55eb62c1a940 2024-04-21T23:19:35.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.098+0000 7f73e6ff5640 1 -- 172.21.15.40:0/1287695495 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f73f4005240 con 0x55eb62c1a940 2024-04-21T23:19:35.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.098+0000 7f740be1e740 1 -- 172.21.15.40:0/1287695495 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f73f40043d0 con 0x55eb62c1a940 2024-04-21T23:19:35.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.099+0000 7f73e6ff5640 1 -- 172.21.15.40:0/1287695495 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f73f4033050 con 0x55eb62c1a940 2024-04-21T23:19:35.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.099+0000 7f740be1e740 10 client.15987 fetch_fsmap learned FSMap version 25 2024-04-21T23:19:35.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.099+0000 7f740be1e740 1 -- 172.21.15.40:0/1287695495 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55eb628923f0 con 0x55eb62c1a940 2024-04-21T23:19:35.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.099+0000 7f73e6ff5640 1 -- 172.21.15.40:0/1287695495 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f73f40043d0 con 0x55eb62c1a940 2024-04-21T23:19:35.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.100+0000 7f73e6ff5640 1 -- 172.21.15.40:0/1287695495 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f73f4004ba0 con 0x55eb62c1a940 2024-04-21T23:19:35.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.100+0000 7f740be1e740 10 client.15987 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:19:35.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.100+0000 7f740be1e740 10 client.15987 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:19:35.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.100+0000 7f740be1e740 20 client.15987 populate_metadata read hostname 'smithi040' 2024-04-21T23:19:35.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.100+0000 7f740be1e740 1 --2- 172.21.15.40:0/1287695495 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55eb62c335b0 0x55eb62c359a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:35.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.100+0000 7f740be1e740 4 client.15987 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:19:35.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.100+0000 7f740be1e740 1 -- 172.21.15.40:0/1287695495 --> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eb62b8ba70 con 0x55eb62c335b0 2024-04-21T23:19:35.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.100+0000 7f740259c640 1 --2- 172.21.15.40:0/1287695495 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55eb62c335b0 0x55eb62c359a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:35.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.101+0000 7f73e6ff5640 10 client.15987 ms_handle_connect on v2:172.21.15.111:6838/3712235548 2024-04-21T23:19:35.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.101+0000 7f740259c640 1 --2- 172.21.15.40:0/1287695495 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55eb62c335b0 0x55eb62c359a0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=226d9add09ee817f server_cookie=128961f028d4740a in_seq=0 out_seq=0 2024-04-21T23:19:35.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.101+0000 7f73e6ff5640 1 -- 172.21.15.40:0/1287695495 <== mds.0 v2:172.21.15.111:6838/3712235548 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55eb62b8ba70 con 0x55eb62c335b0 2024-04-21T23:19:35.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.101+0000 7f73e6ff5640 10 client.15987 handle_command_reply: tid=0 2024-04-21T23:19:35.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.101+0000 7f740be1e740 1 client.15987 shutdown 2024-04-21T23:19:35.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.102+0000 7f740be1e740 1 -- 172.21.15.40:0/1287695495 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eb62c1a940 msgr2=0x55eb62c20280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:35.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.102+0000 7f740be1e740 1 --2- 172.21.15.40:0/1287695495 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eb62c1a940 0x55eb62c20280 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f73f4003e00 tx=0x7f73f4003820 comp rx=0 tx=0).stop 2024-04-21T23:19:35.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.102+0000 7f740be1e740 1 -- 172.21.15.40:0/1287695495 shutdown_connections 2024-04-21T23:19:35.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.102+0000 7f740be1e740 1 --2- 172.21.15.40:0/1287695495 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55eb62c335b0 0x55eb62c359a0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:35.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.102+0000 7f740be1e740 1 --2- 172.21.15.40:0/1287695495 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55eb62c23520 0x55eb62c20890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:35.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.102+0000 7f740be1e740 1 --2- 172.21.15.40:0/1287695495 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eb62c1a940 0x55eb62c20280 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:35.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.102+0000 7f73e6ff5640 0 client.15987 ms_handle_reset on v2:172.21.15.111:6838/3712235548 2024-04-21T23:19:35.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.102+0000 7f740be1e740 1 --2- 172.21.15.40:0/1287695495 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eb62c1a020 0x55eb62c22ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:35.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.102+0000 7f740be1e740 1 -- 172.21.15.40:0/1287695495 >> 172.21.15.40:0/1287695495 conn(0x55eb62c14750 msgr2=0x55eb62c164a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:35.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.102+0000 7f740be1e740 1 -- 172.21.15.40:0/1287695495 shutdown_connections 2024-04-21T23:19:35.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.102+0000 7f740be1e740 1 -- 172.21.15.40:0/1287695495 wait complete. 2024-04-21T23:19:35.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.102+0000 7f740be1e740 20 client.15987 trim_cache size 0 max 16384 2024-04-21T23:19:35.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.128+0000 7f740be1e740 1 Processor -- start 2024-04-21T23:19:35.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.128+0000 7f740be1e740 1 -- start start 2024-04-21T23:19:35.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.129+0000 7f740be1e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55eb62c1a020 0x55eb62c2f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:35.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.129+0000 7f740be1e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eb62c1a940 0x55eb62c16290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:35.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.129+0000 7f740be1e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eb62c23520 0x55eb62c229b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:35.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.129+0000 7f740be1e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f73f0012790 con 0x55eb62c1a940 2024-04-21T23:19:35.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.129+0000 7f740be1e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f73f0010060 con 0x55eb62c23520 2024-04-21T23:19:35.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.129+0000 7f740be1e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f73f000deb0 con 0x55eb62c1a020 2024-04-21T23:19:35.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.129+0000 7f7402d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eb62c1a940 0x55eb62c16290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:35.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.129+0000 7f7402d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eb62c1a940 0x55eb62c16290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48326/0 (socket says 172.21.15.40:48326) 2024-04-21T23:19:35.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.129+0000 7f7402d9d640 1 -- 172.21.15.40:0/117012015 learned_addr learned my addr 172.21.15.40:0/117012015 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:35.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.129+0000 7f740259c640 1 --2- 172.21.15.40:0/117012015 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55eb62c1a020 0x55eb62c2f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:35.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.129+0000 7f740259c640 1 --2- 172.21.15.40:0/117012015 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eb62c23520 0x55eb62c229b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:35.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.129+0000 7f7402d9d640 1 -- 172.21.15.40:0/117012015 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55eb62c1a020 msgr2=0x55eb62c2f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:35.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.129+0000 7f7402d9d640 1 --2- 172.21.15.40:0/117012015 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55eb62c1a020 0x55eb62c2f230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:35.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.129+0000 7f7402d9d640 1 -- 172.21.15.40:0/117012015 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eb62c23520 msgr2=0x55eb62c229b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:35.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.129+0000 7f7402d9d640 1 --2- 172.21.15.40:0/117012015 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eb62c23520 0x55eb62c229b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:35.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.129+0000 7f7402d9d640 1 -- 172.21.15.40:0/117012015 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eb62c20c00 con 0x55eb62c1a940 2024-04-21T23:19:35.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.129+0000 7f7402d9d640 1 --2- 172.21.15.40:0/117012015 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eb62c1a940 0x55eb62c16290 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f73f4011850 tx=0x7f73f4019a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef892e19f042695b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:35.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.130+0000 7f73e8ff9640 1 -- 172.21.15.40:0/117012015 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73f4030070 con 0x55eb62c1a940 2024-04-21T23:19:35.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.130+0000 7f73e8ff9640 1 -- 172.21.15.40:0/117012015 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f73f4019420 con 0x55eb62c1a940 2024-04-21T23:19:35.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.130+0000 7f73e8ff9640 1 -- 172.21.15.40:0/117012015 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73f4027070 con 0x55eb62c1a940 2024-04-21T23:19:35.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.131+0000 7f740be1e740 1 -- 172.21.15.40:0/117012015 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eb62c1a940 msgr2=0x55eb62c16290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:35.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.131+0000 7f740be1e740 1 --2- 172.21.15.40:0/117012015 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eb62c1a940 0x55eb62c16290 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f73f4011850 tx=0x7f73f4019a30 comp rx=0 tx=0).stop 2024-04-21T23:19:35.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.131+0000 7f740be1e740 1 -- 172.21.15.40:0/117012015 shutdown_connections 2024-04-21T23:19:35.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.131+0000 7f740be1e740 1 --2- 172.21.15.40:0/117012015 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eb62c23520 0x55eb62c229b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:35.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.131+0000 7f740be1e740 1 --2- 172.21.15.40:0/117012015 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eb62c1a940 0x55eb62c16290 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:35.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.131+0000 7f740be1e740 1 --2- 172.21.15.40:0/117012015 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55eb62c1a020 0x55eb62c2f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:35.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.131+0000 7f740be1e740 1 -- 172.21.15.40:0/117012015 >> 172.21.15.40:0/117012015 conn(0x55eb62c14750 msgr2=0x55eb62c15b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:35.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.131+0000 7f740be1e740 1 -- 172.21.15.40:0/117012015 shutdown_connections 2024-04-21T23:19:35.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.131+0000 7f740be1e740 1 -- 172.21.15.40:0/117012015 wait complete. 2024-04-21T23:19:35.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.131+0000 7f740be1e740 1 -- start start 2024-04-21T23:19:35.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.132+0000 7f740be1e740 1 Processor -- start 2024-04-21T23:19:35.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.132+0000 7f740be1e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eb62c1a020 0x55eb62c2ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:35.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.132+0000 7f740be1e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eb62c1a940 0x55eb62c26de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:35.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.132+0000 7f740be1e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55eb62c23520 0x55eb62c27320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:35.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.132+0000 7f740be1e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f73f000e520 con 0x55eb62c1a940 2024-04-21T23:19:35.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.132+0000 7f740be1e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f73f000f990 con 0x55eb62c1a020 2024-04-21T23:19:35.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.132+0000 7f740be1e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f73f000ebf0 con 0x55eb62c23520 2024-04-21T23:19:35.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.132+0000 7f7402d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eb62c1a940 0x55eb62c26de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:35.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.133+0000 7f7402d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eb62c1a940 0x55eb62c26de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48332/0 (socket says 172.21.15.40:48332) 2024-04-21T23:19:35.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.133+0000 7f7402d9d640 1 -- 172.21.15.40:0/3305320372 learned_addr learned my addr 172.21.15.40:0/3305320372 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:35.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.133+0000 7f740259c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eb62c1a020 0x55eb62c2ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:35.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.133+0000 7f740259c640 1 --2- 172.21.15.40:0/3305320372 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55eb62c23520 0x55eb62c27320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:35.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.133+0000 7f740259c640 1 -- 172.21.15.40:0/3305320372 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55eb62c23520 msgr2=0x55eb62c27320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:35.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.133+0000 7f740259c640 1 --2- 172.21.15.40:0/3305320372 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55eb62c23520 0x55eb62c27320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:35.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.133+0000 7f740259c640 1 -- 172.21.15.40:0/3305320372 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eb62c1a940 msgr2=0x55eb62c26de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:35.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.133+0000 7f740259c640 1 --2- 172.21.15.40:0/3305320372 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eb62c1a940 0x55eb62c26de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:35.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.133+0000 7f740259c640 1 -- 172.21.15.40:0/3305320372 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eb6295c7d0 con 0x55eb62c1a020 2024-04-21T23:19:35.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.133+0000 7f7402d9d640 1 --2- 172.21.15.40:0/3305320372 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eb62c1a940 0x55eb62c26de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:19:35.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.133+0000 7f740be1e740 10 client.0 _ll_register_callbacks cb 0x7f73f4009b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:19:35.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.133+0000 7f740259c640 1 --2- 172.21.15.40:0/3305320372 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eb62c1a020 0x55eb62c2ba20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f73f802c170 tx=0x7f73f4005d30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c3fdb4af7ad755e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:35.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.133+0000 7f73e6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-21T23:19:35.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.133+0000 7f740be1e740 1 -- 172.21.15.40:0/3305320372 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f73f4019420 con 0x55eb62c1a020 2024-04-21T23:19:35.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.133+0000 7f73e6ff5640 1 -- 172.21.15.40:0/3305320372 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73f80046c0 con 0x55eb62c1a020 2024-04-21T23:19:35.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.133+0000 7f73e6ff5640 1 -- 172.21.15.40:0/3305320372 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f73f803a040 con 0x55eb62c1a020 2024-04-21T23:19:35.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.134+0000 7f73e6ff5640 1 -- 172.21.15.40:0/3305320372 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f73f8004860 con 0x55eb62c1a020 2024-04-21T23:19:35.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.134+0000 7f740be1e740 10 client.25529 fetch_fsmap learned FSMap version 25 2024-04-21T23:19:35.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.134+0000 7f740be1e740 1 -- 172.21.15.40:0/3305320372 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({fsmap=25}) -- 0x55eb62c995e0 con 0x55eb62c1a020 2024-04-21T23:19:35.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.134+0000 7f73e6ff5640 1 -- 172.21.15.40:0/3305320372 <== mon.2 v2:172.21.15.111:3300/0 4 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f73f8004a00 con 0x55eb62c1a020 2024-04-21T23:19:35.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.134+0000 7f740be1e740 10 client.25529 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:19:35.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.134+0000 7f740be1e740 10 client.25529 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:19:35.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.134+0000 7f740be1e740 20 client.25529 populate_metadata read hostname 'smithi040' 2024-04-21T23:19:35.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.134+0000 7f740be1e740 1 --2- 172.21.15.40:0/3305320372 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55eb62c99c50 0x55eb62c9c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:35.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.134+0000 7f740be1e740 4 client.25529 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T23:19:35.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.134+0000 7f740be1e740 1 -- 172.21.15.40:0/3305320372 --> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f73f4019420 con 0x55eb62c99c50 2024-04-21T23:19:35.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.135+0000 7f7402d9d640 1 --2- 172.21.15.40:0/3305320372 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55eb62c99c50 0x55eb62c9c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:35.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.135+0000 7f73e6ff5640 10 client.25529 ms_handle_connect on v2:172.21.15.111:6838/3712235548 2024-04-21T23:19:35.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.135+0000 7f7402d9d640 1 --2- 172.21.15.40:0/3305320372 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55eb62c99c50 0x55eb62c9c0b0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b68527d965e499c8 server_cookie=65cfc88dc1b21214 in_seq=0 out_seq=0 2024-04-21T23:19:35.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.135+0000 7f73e6ff5640 1 -- 172.21.15.40:0/3305320372 <== mds.0 v2:172.21.15.111:6838/3712235548 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f73f4019420 con 0x55eb62c99c50 2024-04-21T23:19:35.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.135+0000 7f73e6ff5640 10 client.25529 handle_command_reply: tid=0 2024-04-21T23:19:35.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.135+0000 7f740be1e740 1 client.25529 shutdown 2024-04-21T23:19:35.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.135+0000 7f740be1e740 1 -- 172.21.15.40:0/3305320372 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eb62c1a020 msgr2=0x55eb62c2ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:35.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.135+0000 7f740be1e740 1 --2- 172.21.15.40:0/3305320372 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eb62c1a020 0x55eb62c2ba20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f73f802c170 tx=0x7f73f4005d30 comp rx=0 tx=0).stop 2024-04-21T23:19:35.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.135+0000 7f740be1e740 1 -- 172.21.15.40:0/3305320372 shutdown_connections 2024-04-21T23:19:35.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.135+0000 7f740be1e740 1 --2- 172.21.15.40:0/3305320372 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55eb62c99c50 0x55eb62c9c0b0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:35.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.135+0000 7f740be1e740 1 --2- 172.21.15.40:0/3305320372 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55eb62c23520 0x55eb62c27320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:35.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.135+0000 7f73e6ff5640 0 client.25529 ms_handle_reset on v2:172.21.15.111:6838/3712235548 2024-04-21T23:19:35.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.135+0000 7f740be1e740 1 --2- 172.21.15.40:0/3305320372 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eb62c1a940 0x55eb62c26de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:35.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.136+0000 7f740be1e740 1 --2- 172.21.15.40:0/3305320372 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eb62c1a020 0x55eb62c2ba20 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:35.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.136+0000 7f740be1e740 1 -- 172.21.15.40:0/3305320372 >> 172.21.15.40:0/3305320372 conn(0x55eb62c14750 msgr2=0x55eb62c162f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:35.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.136+0000 7f740be1e740 1 -- 172.21.15.40:0/3305320372 shutdown_connections 2024-04-21T23:19:35.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.136+0000 7f740be1e740 1 -- 172.21.15.40:0/3305320372 wait complete. 2024-04-21T23:19:35.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.136+0000 7f740be1e740 20 client.25529 trim_cache size 0 max 16384 2024-04-21T23:19:35.140 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-04-21T23:19:35.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.136+0000 7f7409235640 1 -- 172.21.15.40:0/1896824405 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f73ec0a2fb0 msgr2=0x7f73ec0a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:35.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.136+0000 7f7409235640 1 --2- 172.21.15.40:0/1896824405 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f73ec0a2fb0 0x7f73ec0a5470 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f73fc0cd210 tx=0x7f73fc011000 comp rx=0 tx=0).stop 2024-04-21T23:19:35.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.136+0000 7f7409235640 1 -- 172.21.15.40:0/1896824405 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f740410e0f0 msgr2=0x7f740419ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:35.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.136+0000 7f7409235640 1 --2- 172.21.15.40:0/1896824405 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f740410e0f0 0x7f740419ead0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f73fc00eb40 tx=0x7f73fc00c2d0 comp rx=0 tx=0).stop 2024-04-21T23:19:35.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.137+0000 7f7409235640 1 -- 172.21.15.40:0/1896824405 shutdown_connections 2024-04-21T23:19:35.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.137+0000 7f7409235640 1 --2- 172.21.15.40:0/1896824405 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f73ec0a2fb0 0x7f73ec0a5470 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:35.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.137+0000 7f7409235640 1 --2- 172.21.15.40:0/1896824405 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7404100e60 0x7f740419a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:35.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.137+0000 7f7409235640 1 --2- 172.21.15.40:0/1896824405 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f740410e0f0 0x7f740419ead0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:35.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.137+0000 7f7409235640 1 --2- 172.21.15.40:0/1896824405 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7404100540 0x7f740419a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:35.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.137+0000 7f7409235640 1 -- 172.21.15.40:0/1896824405 >> 172.21.15.40:0/1896824405 conn(0x7f7404077740 msgr2=0x7f740410f670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:35.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.137+0000 7f7409235640 1 -- 172.21.15.40:0/1896824405 shutdown_connections 2024-04-21T23:19:35.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:35.137+0000 7f7409235640 1 -- 172.21.15.40:0/1896824405 wait complete. 2024-04-21T23:19:35.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:19:35 smithi040 ceph-mon[27984]: pgmap v38: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-21T23:19:35.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:19:35 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/3448120976' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:19:35.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:19:35 smithi111 ceph-mon[31085]: pgmap v38: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-21T23:19:35.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:19:35 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/3448120976' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:19:35.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:19:35 smithi174 ceph-mon[31409]: pgmap v38: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-21T23:19:35.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:19:35 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/3448120976' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:19:36.151 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T23:19:36.152 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:19:36.152 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T23:19:36.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.268+0000 7efdc8197640 1 -- 172.21.15.40:0/1152383136 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efdc0069dc0 msgr2=0x7efdc006a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:36.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.268+0000 7efdc8197640 1 --2- 172.21.15.40:0/1152383136 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efdc0069dc0 0x7efdc006a240 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7efdb0009930 tx=0x7efdb002bc30 comp rx=0 tx=0).stop 2024-04-21T23:19:36.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.268+0000 7efdc8197640 1 -- 172.21.15.40:0/1152383136 shutdown_connections 2024-04-21T23:19:36.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.268+0000 7efdc8197640 1 --2- 172.21.15.40:0/1152383136 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efdc006a780 0x7efdc010e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:36.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.268+0000 7efdc8197640 1 --2- 172.21.15.40:0/1152383136 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efdc0069dc0 0x7efdc006a240 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:36.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.268+0000 7efdc8197640 1 --2- 172.21.15.40:0/1152383136 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efdc0107730 0x7efdc0107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:36.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.268+0000 7efdc8197640 1 -- 172.21.15.40:0/1152383136 >> 172.21.15.40:0/1152383136 conn(0x7efdc00776a0 msgr2=0x7efdc0077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:36.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.269+0000 7efdc8197640 1 -- 172.21.15.40:0/1152383136 shutdown_connections 2024-04-21T23:19:36.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.269+0000 7efdc8197640 1 -- 172.21.15.40:0/1152383136 wait complete. 2024-04-21T23:19:36.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.269+0000 7efdc8197640 1 Processor -- start 2024-04-21T23:19:36.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.269+0000 7efdc8197640 1 -- start start 2024-04-21T23:19:36.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.270+0000 7efdc8197640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efdc0069dc0 0x7efdc0198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:36.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.270+0000 7efdc8197640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efdc006a780 0x7efdc01985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:36.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.270+0000 7efdc8197640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efdc0107730 0x7efdc019c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:36.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.270+0000 7efdc8197640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7efdc0111b00 con 0x7efdc0107730 2024-04-21T23:19:36.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.270+0000 7efdc8197640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7efdc0111980 con 0x7efdc006a780 2024-04-21T23:19:36.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.270+0000 7efdc8197640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7efdc0111c80 con 0x7efdc0069dc0 2024-04-21T23:19:36.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.270+0000 7efdc6734640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efdc0107730 0x7efdc019c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:36.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.270+0000 7efdc6734640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efdc0107730 0x7efdc019c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48348/0 (socket says 172.21.15.40:48348) 2024-04-21T23:19:36.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.270+0000 7efdc6734640 1 -- 172.21.15.40:0/2605648367 learned_addr learned my addr 172.21.15.40:0/2605648367 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:36.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.270+0000 7efdc5f33640 1 --2- 172.21.15.40:0/2605648367 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efdc0069dc0 0x7efdc0198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:36.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.270+0000 7efdc5732640 1 --2- 172.21.15.40:0/2605648367 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efdc006a780 0x7efdc01985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:36.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.270+0000 7efdc6734640 1 -- 172.21.15.40:0/2605648367 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efdc0069dc0 msgr2=0x7efdc0198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:36.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.270+0000 7efdc6734640 1 --2- 172.21.15.40:0/2605648367 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efdc0069dc0 0x7efdc0198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:36.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.270+0000 7efdc6734640 1 -- 172.21.15.40:0/2605648367 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efdc006a780 msgr2=0x7efdc01985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:36.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.270+0000 7efdc6734640 1 --2- 172.21.15.40:0/2605648367 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efdc006a780 0x7efdc01985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:36.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.270+0000 7efdc6734640 1 -- 172.21.15.40:0/2605648367 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdc019d000 con 0x7efdc0107730 2024-04-21T23:19:36.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.270+0000 7efdc6734640 1 --2- 172.21.15.40:0/2605648367 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efdc0107730 0x7efdc019c8d0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7efdb800eaa0 tx=0x7efdb800eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:36.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.271+0000 7efdb6ffd640 1 -- 172.21.15.40:0/2605648367 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdb8004d20 con 0x7efdc0107730 2024-04-21T23:19:36.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.271+0000 7efdb6ffd640 1 -- 172.21.15.40:0/2605648367 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efdb8005020 con 0x7efdc0107730 2024-04-21T23:19:36.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.271+0000 7efdc8197640 1 -- 172.21.15.40:0/2605648367 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdc019d2f0 con 0x7efdc0107730 2024-04-21T23:19:36.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.271+0000 7efdb6ffd640 1 -- 172.21.15.40:0/2605648367 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdb80116d0 con 0x7efdc0107730 2024-04-21T23:19:36.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.271+0000 7efdc8197640 1 -- 172.21.15.40:0/2605648367 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efdc01a4b80 con 0x7efdc0107730 2024-04-21T23:19:36.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.273+0000 7efdcad80740 1 Processor -- start 2024-04-21T23:19:36.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.274+0000 7efdb6ffd640 1 -- 172.21.15.40:0/2605648367 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 25) ==== 138117+0+0 (secure 0 0 0) 0x7efdb80051d0 con 0x7efdc0107730 2024-04-21T23:19:36.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.274+0000 7efdcad80740 1 -- start start 2024-04-21T23:19:36.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.275+0000 7efdcad80740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a5d69b7720 0x55a5d69b6470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:36.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.275+0000 7efdcad80740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5d69b6a40 0x55a5d69b6ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:36.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.275+0000 7efdcad80740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a5d69bfdb0 0x55a5d69c21a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:36.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.275+0000 7efdcad80740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55a5d691a170 con 0x55a5d69bfdb0 2024-04-21T23:19:36.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.275+0000 7efdcad80740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55a5d68c27c0 con 0x55a5d69b6a40 2024-04-21T23:19:36.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.275+0000 7efdcad80740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55a5d6916130 con 0x55a5d69b7720 2024-04-21T23:19:36.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.275+0000 7efdc5732640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a5d69bfdb0 0x55a5d69c21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:36.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.275+0000 7efdc5732640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a5d69bfdb0 0x55a5d69c21a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48360/0 (socket says 172.21.15.40:48360) 2024-04-21T23:19:36.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.275+0000 7efdc5732640 1 -- 172.21.15.40:0/3501204305 learned_addr learned my addr 172.21.15.40:0/3501204305 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:36.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.275+0000 7efdc5732640 1 --2- 172.21.15.40:0/3501204305 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a5d69b7720 0x55a5d69b6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:36.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.275+0000 7efdc5f33640 1 --2- 172.21.15.40:0/3501204305 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5d69b6a40 0x55a5d69b6ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:36.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.275+0000 7efdb6ffd640 1 --2- 172.21.15.40:0/2605648367 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7efdac0a2f30 0x7efdac0a53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:36.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.275+0000 7efdc5732640 1 -- 172.21.15.40:0/3501204305 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a5d69b7720 msgr2=0x55a5d69b6470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:36.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.275+0000 7efdc5732640 1 --2- 172.21.15.40:0/3501204305 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a5d69b7720 0x55a5d69b6470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:36.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.275+0000 7efdc5732640 1 -- 172.21.15.40:0/3501204305 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5d69b6a40 msgr2=0x55a5d69b6ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:36.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.275+0000 7efdc5732640 1 --2- 172.21.15.40:0/3501204305 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5d69b6a40 0x55a5d69b6ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:36.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.275+0000 7efdc5732640 1 -- 172.21.15.40:0/3501204305 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a5d68d3e60 con 0x55a5d69bfdb0 2024-04-21T23:19:36.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.276+0000 7efdc6734640 1 --2- 172.21.15.40:0/2605648367 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7efdac0a2f30 0x7efdac0a53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:36.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.276+0000 7efdc5732640 1 --2- 172.21.15.40:0/3501204305 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a5d69bfdb0 0x55a5d69c21a0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7efdb00096c0 tx=0x7efdb0004e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe5eaa2378712fd1 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:36.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.276+0000 7efdabfff640 1 -- 172.21.15.40:0/3501204305 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdb0004230 con 0x55a5d69bfdb0 2024-04-21T23:19:36.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.276+0000 7efdb6ffd640 1 -- 172.21.15.40:0/2605648367 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7efdb80cfc60 con 0x7efdc0107730 2024-04-21T23:19:36.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.276+0000 7efdabfff640 1 -- 172.21.15.40:0/3501204305 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efdb002cce0 con 0x55a5d69bfdb0 2024-04-21T23:19:36.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.276+0000 7efdabfff640 1 -- 172.21.15.40:0/3501204305 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdb00053e0 con 0x55a5d69bfdb0 2024-04-21T23:19:36.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.277+0000 7efdcad80740 1 -- 172.21.15.40:0/3501204305 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a5d69bfdb0 msgr2=0x55a5d69c21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:36.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.277+0000 7efdcad80740 1 --2- 172.21.15.40:0/3501204305 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a5d69bfdb0 0x55a5d69c21a0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7efdb00096c0 tx=0x7efdb0004e70 comp rx=0 tx=0).stop 2024-04-21T23:19:36.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.277+0000 7efdc6734640 1 --2- 172.21.15.40:0/2605648367 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7efdac0a2f30 0x7efdac0a53f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7efdb80cce20 tx=0x7efdb8097800 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:36.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.277+0000 7efdcad80740 1 -- 172.21.15.40:0/3501204305 shutdown_connections 2024-04-21T23:19:36.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.277+0000 7efdcad80740 1 --2- 172.21.15.40:0/3501204305 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5d69b6a40 0x55a5d69b6ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:36.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.277+0000 7efdcad80740 1 --2- 172.21.15.40:0/3501204305 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a5d69bfdb0 0x55a5d69c21a0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:36.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.277+0000 7efdcad80740 1 --2- 172.21.15.40:0/3501204305 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a5d69b7720 0x55a5d69b6470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:36.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.277+0000 7efdcad80740 1 -- 172.21.15.40:0/3501204305 >> 172.21.15.40:0/3501204305 conn(0x55a5d697d4f0 msgr2=0x55a5d69b1490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:36.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.277+0000 7efdcad80740 1 -- 172.21.15.40:0/3501204305 shutdown_connections 2024-04-21T23:19:36.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.277+0000 7efdcad80740 1 -- 172.21.15.40:0/3501204305 wait complete. 2024-04-21T23:19:36.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.277+0000 7efdcad80740 1 -- start start 2024-04-21T23:19:36.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.278+0000 7efdcad80740 1 Processor -- start 2024-04-21T23:19:36.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.278+0000 7efdcad80740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a5d69b6a40 0x55a5d69bcd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:36.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.278+0000 7efdcad80740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5d69b7720 0x55a5d69bd2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:36.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.278+0000 7efdcad80740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a5d69bfdb0 0x55a5d69bd800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:36.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.278+0000 7efdcad80740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55a5d69c2bc0 con 0x55a5d69bfdb0 2024-04-21T23:19:36.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.278+0000 7efdcad80740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55a5d68d5d40 con 0x55a5d69b7720 2024-04-21T23:19:36.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.278+0000 7efdcad80740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55a5d69c2d40 con 0x55a5d69b6a40 2024-04-21T23:19:36.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.278+0000 7efdcad80740 10 client.0 _ll_register_callbacks cb 0x55a5d6911820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:19:36.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.278+0000 7efdc5732640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a5d69bfdb0 0x55a5d69bd800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:36.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.278+0000 7efdc5732640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a5d69bfdb0 0x55a5d69bd800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48366/0 (socket says 172.21.15.40:48366) 2024-04-21T23:19:36.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.278+0000 7efdc5732640 1 -- 172.21.15.40:0/1896010136 learned_addr learned my addr 172.21.15.40:0/1896010136 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:36.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.278+0000 7efdc5732640 1 --2- 172.21.15.40:0/1896010136 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a5d69b6a40 0x55a5d69bcd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:36.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.278+0000 7efdc5f33640 1 --2- 172.21.15.40:0/1896010136 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5d69b7720 0x55a5d69bd2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:36.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.279+0000 7efdc5732640 1 -- 172.21.15.40:0/1896010136 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a5d69b6a40 msgr2=0x55a5d69bcd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:36.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.279+0000 7efdc5732640 1 --2- 172.21.15.40:0/1896010136 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a5d69b6a40 0x55a5d69bcd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:36.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.279+0000 7efdc5732640 1 -- 172.21.15.40:0/1896010136 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5d69b7720 msgr2=0x55a5d69bd2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:36.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.279+0000 7efdc5732640 1 --2- 172.21.15.40:0/1896010136 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5d69b7720 0x55a5d69bd2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:36.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.279+0000 7efdc5732640 1 -- 172.21.15.40:0/1896010136 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a5d66f67e0 con 0x55a5d69bfdb0 2024-04-21T23:19:36.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.279+0000 7efdc5732640 1 --2- 172.21.15.40:0/1896010136 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a5d69bfdb0 0x55a5d69bd800 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7efdb0001eb0 tx=0x7efdb0043040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abb9efadc967cdfd server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:36.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.279+0000 7efda97fa640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-21T23:19:36.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.279+0000 7efda97fa640 1 -- 172.21.15.40:0/1896010136 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdb0054060 con 0x55a5d69bfdb0 2024-04-21T23:19:36.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.279+0000 7efdcad80740 1 -- 172.21.15.40:0/1896010136 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a5d68ee450 con 0x55a5d69bfdb0 2024-04-21T23:19:36.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.279+0000 7efda97fa640 1 -- 172.21.15.40:0/1896010136 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7efdb00046c0 con 0x55a5d69bfdb0 2024-04-21T23:19:36.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.279+0000 7efda97fa640 1 -- 172.21.15.40:0/1896010136 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7efdb005b050 con 0x55a5d69bfdb0 2024-04-21T23:19:36.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.279+0000 7efdcad80740 10 client.16005 fetch_fsmap learned FSMap version 25 2024-04-21T23:19:36.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.279+0000 7efdcad80740 1 -- 172.21.15.40:0/1896010136 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55a5d6756f70 con 0x55a5d69bfdb0 2024-04-21T23:19:36.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.280+0000 7efda97fa640 1 -- 172.21.15.40:0/1896010136 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55a5d68ee450 con 0x55a5d69bfdb0 2024-04-21T23:19:36.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.280+0000 7efda97fa640 1 -- 172.21.15.40:0/1896010136 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7efdb0003c50 con 0x55a5d69bfdb0 2024-04-21T23:19:36.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.280+0000 7efdcad80740 10 client.16005 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:19:36.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.280+0000 7efdcad80740 10 client.16005 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:19:36.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.280+0000 7efdcad80740 20 client.16005 populate_metadata read hostname 'smithi040' 2024-04-21T23:19:36.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.280+0000 7efdcad80740 1 --2- 172.21.15.40:0/1896010136 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55a5d69d8610 0x55a5d69daa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:36.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.280+0000 7efdcad80740 4 client.16005 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:19:36.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.280+0000 7efdcad80740 1 -- 172.21.15.40:0/1896010136 --> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a5d69276b0 con 0x55a5d69d8610 2024-04-21T23:19:36.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.280+0000 7efdc5f33640 1 --2- 172.21.15.40:0/1896010136 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55a5d69d8610 0x55a5d69daa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:36.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.281+0000 7efdc5f33640 1 --2- 172.21.15.40:0/1896010136 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55a5d69d8610 0x55a5d69daa20 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a8a8e1af2c73e75 server_cookie=93bfb1044d00dbe6 in_seq=0 out_seq=0 2024-04-21T23:19:36.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.281+0000 7efda97fa640 10 client.16005 ms_handle_connect on v2:172.21.15.111:6838/3712235548 2024-04-21T23:19:36.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.281+0000 7efda97fa640 1 -- 172.21.15.40:0/1896010136 <== mds.0 v2:172.21.15.111:6838/3712235548 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a5d69276b0 con 0x55a5d69d8610 2024-04-21T23:19:36.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.281+0000 7efda97fa640 10 client.16005 handle_command_reply: tid=0 2024-04-21T23:19:36.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.281+0000 7efdcad80740 1 client.16005 shutdown 2024-04-21T23:19:36.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.281+0000 7efdcad80740 1 -- 172.21.15.40:0/1896010136 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a5d69bfdb0 msgr2=0x55a5d69bd800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:36.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.281+0000 7efdcad80740 1 --2- 172.21.15.40:0/1896010136 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a5d69bfdb0 0x55a5d69bd800 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7efdb0001eb0 tx=0x7efdb0043040 comp rx=0 tx=0).stop 2024-04-21T23:19:36.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.282+0000 7efdcad80740 1 -- 172.21.15.40:0/1896010136 shutdown_connections 2024-04-21T23:19:36.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.282+0000 7efdcad80740 1 --2- 172.21.15.40:0/1896010136 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55a5d69d8610 0x55a5d69daa20 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:36.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.282+0000 7efdcad80740 1 --2- 172.21.15.40:0/1896010136 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5d69b7720 0x55a5d69bd2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:36.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.282+0000 7efdcad80740 1 --2- 172.21.15.40:0/1896010136 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a5d69bfdb0 0x55a5d69bd800 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:36.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.282+0000 7efdcad80740 1 --2- 172.21.15.40:0/1896010136 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a5d69b6a40 0x55a5d69bcd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:36.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.282+0000 7efdcad80740 1 -- 172.21.15.40:0/1896010136 >> 172.21.15.40:0/1896010136 conn(0x55a5d697d4f0 msgr2=0x55a5d69b31e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:36.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.282+0000 7efda97fa640 0 client.16005 ms_handle_reset on v2:172.21.15.111:6838/3712235548 2024-04-21T23:19:36.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.282+0000 7efdcad80740 1 -- 172.21.15.40:0/1896010136 shutdown_connections 2024-04-21T23:19:36.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.282+0000 7efdcad80740 1 -- 172.21.15.40:0/1896010136 wait complete. 2024-04-21T23:19:36.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.282+0000 7efdcad80740 20 client.16005 trim_cache size 0 max 16384 2024-04-21T23:19:36.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.300+0000 7efdcad80740 1 Processor -- start 2024-04-21T23:19:36.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.300+0000 7efdcad80740 1 -- start start 2024-04-21T23:19:36.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.300+0000 7efdcad80740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a5d69b6a40 0x55a5d69bf4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:36.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.300+0000 7efdcad80740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a5d69b7720 0x55a5d69cbaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:36.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.300+0000 7efdcad80740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5d69bfdb0 0x55a5d69b2ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:36.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.300+0000 7efdcad80740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7efdc01af330 con 0x55a5d69b7720 2024-04-21T23:19:36.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.300+0000 7efdcad80740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7efdc01acc00 con 0x55a5d69bfdb0 2024-04-21T23:19:36.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.300+0000 7efdcad80740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7efdc01aaa50 con 0x55a5d69b6a40 2024-04-21T23:19:36.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.300+0000 7efdc5f33640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a5d69b7720 0x55a5d69cbaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:36.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.300+0000 7efdc5f33640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a5d69b7720 0x55a5d69cbaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48378/0 (socket says 172.21.15.40:48378) 2024-04-21T23:19:36.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.300+0000 7efdc5f33640 1 -- 172.21.15.40:0/140848793 learned_addr learned my addr 172.21.15.40:0/140848793 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:36.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.300+0000 7efdc5732640 1 --2- 172.21.15.40:0/140848793 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5d69bfdb0 0x55a5d69b2ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:36.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.300+0000 7efdc5732640 1 --2- 172.21.15.40:0/140848793 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a5d69b6a40 0x55a5d69bf4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:36.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.300+0000 7efdc5f33640 1 -- 172.21.15.40:0/140848793 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a5d69b6a40 msgr2=0x55a5d69bf4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:36.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.300+0000 7efdc5f33640 1 --2- 172.21.15.40:0/140848793 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a5d69b6a40 0x55a5d69bf4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:36.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.300+0000 7efdc5f33640 1 -- 172.21.15.40:0/140848793 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5d69bfdb0 msgr2=0x55a5d69b2ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:36.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.300+0000 7efdc5f33640 1 --2- 172.21.15.40:0/140848793 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5d69bfdb0 0x55a5d69b2ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:36.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.300+0000 7efdc5f33640 1 -- 172.21.15.40:0/140848793 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a5d65ee170 con 0x55a5d69b7720 2024-04-21T23:19:36.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.300+0000 7efdc5f33640 1 --2- 172.21.15.40:0/140848793 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a5d69b7720 0x55a5d69cbaf0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7efdbc005e10 tx=0x7efdbc00ab40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12cd92f2a19ee689 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:36.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.301+0000 7efdab7fe640 1 -- 172.21.15.40:0/140848793 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdbc022070 con 0x55a5d69b7720 2024-04-21T23:19:36.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.301+0000 7efdab7fe640 1 -- 172.21.15.40:0/140848793 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efdbc00ae90 con 0x55a5d69b7720 2024-04-21T23:19:36.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.301+0000 7efdab7fe640 1 -- 172.21.15.40:0/140848793 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdbc004a00 con 0x55a5d69b7720 2024-04-21T23:19:36.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.301+0000 7efdcad80740 1 -- 172.21.15.40:0/140848793 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a5d69b7720 msgr2=0x55a5d69cbaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:36.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.301+0000 7efdcad80740 1 --2- 172.21.15.40:0/140848793 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a5d69b7720 0x55a5d69cbaf0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7efdbc005e10 tx=0x7efdbc00ab40 comp rx=0 tx=0).stop 2024-04-21T23:19:36.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.302+0000 7efdcad80740 1 -- 172.21.15.40:0/140848793 shutdown_connections 2024-04-21T23:19:36.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.302+0000 7efdcad80740 1 --2- 172.21.15.40:0/140848793 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5d69bfdb0 0x55a5d69b2ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:36.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.302+0000 7efdcad80740 1 --2- 172.21.15.40:0/140848793 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a5d69b7720 0x55a5d69cbaf0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:36.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.302+0000 7efdcad80740 1 --2- 172.21.15.40:0/140848793 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a5d69b6a40 0x55a5d69bf4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:36.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.302+0000 7efdcad80740 1 -- 172.21.15.40:0/140848793 >> 172.21.15.40:0/140848793 conn(0x55a5d697d4f0 msgr2=0x55a5d69b28e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:36.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.302+0000 7efdcad80740 1 -- 172.21.15.40:0/140848793 shutdown_connections 2024-04-21T23:19:36.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.302+0000 7efdcad80740 1 -- 172.21.15.40:0/140848793 wait complete. 2024-04-21T23:19:36.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.302+0000 7efdcad80740 1 -- start start 2024-04-21T23:19:36.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.302+0000 7efdcad80740 1 Processor -- start 2024-04-21T23:19:36.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.302+0000 7efdcad80740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a5d69b6a40 0x55a5d69c8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:36.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.302+0000 7efdcad80740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5d69b7720 0x55a5d69c8540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:36.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.302+0000 7efdcad80740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a5d69bfdb0 0x55a5d69ba1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:36.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.302+0000 7efdcad80740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7efdc01ab0c0 con 0x55a5d69bfdb0 2024-04-21T23:19:36.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.302+0000 7efdcad80740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7efdc01ac530 con 0x55a5d69b7720 2024-04-21T23:19:36.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.303+0000 7efdcad80740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7efdc01ab790 con 0x55a5d69b6a40 2024-04-21T23:19:36.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.303+0000 7efdcad80740 10 client.0 _ll_register_callbacks cb 0x7efdb0005160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:19:36.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.303+0000 7efdc5732640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a5d69bfdb0 0x55a5d69ba1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:36.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.303+0000 7efdc5f33640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5d69b7720 0x55a5d69c8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:36.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.303+0000 7efdc5f33640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5d69b7720 0x55a5d69c8540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.40:53964/0 (socket says 172.21.15.40:53964) 2024-04-21T23:19:36.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.303+0000 7efdc5f33640 1 -- 172.21.15.40:0/2323393937 learned_addr learned my addr 172.21.15.40:0/2323393937 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:36.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.303+0000 7efdc5732640 1 --2- 172.21.15.40:0/2323393937 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a5d69b6a40 0x55a5d69c8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:36.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.303+0000 7efdc5f33640 1 -- 172.21.15.40:0/2323393937 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a5d69b6a40 msgr2=0x55a5d69c8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:36.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.303+0000 7efdc5f33640 1 --2- 172.21.15.40:0/2323393937 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a5d69b6a40 0x55a5d69c8000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:36.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.303+0000 7efdc5f33640 1 -- 172.21.15.40:0/2323393937 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a5d69bfdb0 msgr2=0x55a5d69ba1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:36.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.303+0000 7efdc5f33640 1 --2- 172.21.15.40:0/2323393937 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a5d69bfdb0 0x55a5d69ba1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:36.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.303+0000 7efdc5f33640 1 -- 172.21.15.40:0/2323393937 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a5d69ba9b0 con 0x55a5d69b7720 2024-04-21T23:19:36.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.303+0000 7efdc5732640 1 --2- 172.21.15.40:0/2323393937 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a5d69bfdb0 0x55a5d69ba1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:19:36.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.303+0000 7efdc5f33640 1 --2- 172.21.15.40:0/2323393937 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5d69b7720 0x55a5d69c8540 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7efdbc005660 tx=0x7efdbc005300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=486035b8a74e41ab server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:36.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.303+0000 7efda97fa640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-21T23:19:36.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.304+0000 7efda97fa640 1 -- 172.21.15.40:0/2323393937 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdbc02c020 con 0x55a5d69b7720 2024-04-21T23:19:36.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.304+0000 7efda97fa640 1 -- 172.21.15.40:0/2323393937 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7efdbc01d640 con 0x55a5d69b7720 2024-04-21T23:19:36.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.304+0000 7efdcad80740 1 -- 172.21.15.40:0/2323393937 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efdbc00ae90 con 0x55a5d69b7720 2024-04-21T23:19:36.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.304+0000 7efda97fa640 1 -- 172.21.15.40:0/2323393937 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7efdbc037050 con 0x55a5d69b7720 2024-04-21T23:19:36.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.304+0000 7efdcad80740 10 client.25544 fetch_fsmap learned FSMap version 25 2024-04-21T23:19:36.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.304+0000 7efdcad80740 1 -- 172.21.15.40:0/2323393937 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55a5d69c1240 con 0x55a5d69b7720 2024-04-21T23:19:36.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.304+0000 7efda97fa640 1 -- 172.21.15.40:0/2323393937 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efdbc023040 con 0x55a5d69b7720 2024-04-21T23:19:36.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.304+0000 7efda97fa640 1 -- 172.21.15.40:0/2323393937 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7efdbc0284b0 con 0x55a5d69b7720 2024-04-21T23:19:36.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.304+0000 7efdcad80740 10 client.25544 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:19:36.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.304+0000 7efdcad80740 10 client.25544 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:19:36.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.304+0000 7efdcad80740 20 client.25544 populate_metadata read hostname 'smithi040' 2024-04-21T23:19:36.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.304+0000 7efdcad80740 1 --2- 172.21.15.40:0/2323393937 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55a5d6a3eea0 0x55a5d6a412c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:36.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.304+0000 7efdcad80740 4 client.25544 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T23:19:36.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.304+0000 7efdcad80740 1 -- 172.21.15.40:0/2323393937 --> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x7efdbc00ae90 con 0x55a5d6a3eea0 2024-04-21T23:19:36.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.305+0000 7efdc5732640 1 --2- 172.21.15.40:0/2323393937 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55a5d6a3eea0 0x55a5d6a412c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:36.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.305+0000 7efdc5732640 1 --2- 172.21.15.40:0/2323393937 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55a5d6a3eea0 0x55a5d6a412c0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cf16bc927f14dec2 server_cookie=44ca381fdd2da4c3 in_seq=0 out_seq=0 2024-04-21T23:19:36.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.305+0000 7efda97fa640 10 client.25544 ms_handle_connect on v2:172.21.15.111:6838/3712235548 2024-04-21T23:19:36.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.306+0000 7efda97fa640 1 -- 172.21.15.40:0/2323393937 <== mds.0 v2:172.21.15.111:6838/3712235548 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x7efdbc00ae90 con 0x55a5d6a3eea0 2024-04-21T23:19:36.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.306+0000 7efda97fa640 10 client.25544 handle_command_reply: tid=0 2024-04-21T23:19:36.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.306+0000 7efdcad80740 1 client.25544 shutdown 2024-04-21T23:19:36.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.306+0000 7efdcad80740 1 -- 172.21.15.40:0/2323393937 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5d69b7720 msgr2=0x55a5d69c8540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:36.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.306+0000 7efdcad80740 1 --2- 172.21.15.40:0/2323393937 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5d69b7720 0x55a5d69c8540 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7efdbc005660 tx=0x7efdbc005300 comp rx=0 tx=0).stop 2024-04-21T23:19:36.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.307+0000 7efdcad80740 1 -- 172.21.15.40:0/2323393937 shutdown_connections 2024-04-21T23:19:36.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.307+0000 7efdcad80740 1 --2- 172.21.15.40:0/2323393937 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55a5d6a3eea0 0x55a5d6a412c0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:36.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.307+0000 7efda97fa640 0 client.25544 ms_handle_reset on v2:172.21.15.111:6838/3712235548 2024-04-21T23:19:36.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.307+0000 7efdcad80740 1 --2- 172.21.15.40:0/2323393937 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5d69b7720 0x55a5d69c8540 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:36.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.307+0000 7efdcad80740 1 --2- 172.21.15.40:0/2323393937 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a5d69bfdb0 0x55a5d69ba1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:36.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.307+0000 7efdcad80740 1 --2- 172.21.15.40:0/2323393937 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a5d69b6a40 0x55a5d69c8000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:36.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.307+0000 7efdcad80740 1 -- 172.21.15.40:0/2323393937 >> 172.21.15.40:0/2323393937 conn(0x55a5d697d4f0 msgr2=0x55a5d69b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:36.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.307+0000 7efdcad80740 1 -- 172.21.15.40:0/2323393937 shutdown_connections 2024-04-21T23:19:36.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.307+0000 7efdcad80740 1 -- 172.21.15.40:0/2323393937 wait complete. 2024-04-21T23:19:36.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.307+0000 7efdcad80740 20 client.25544 trim_cache size 0 max 16384 2024-04-21T23:19:36.313 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-21T23:19:36.313 INFO:teuthology.orchestra.run.smithi040.stdout: "return_code": 0, 2024-04-21T23:19:36.314 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_tag": "9d181165-d6e4-436d-ad23-285885355ae4", 2024-04-21T23:19:36.314 INFO:teuthology.orchestra.run.smithi040.stdout: "mode": "asynchronous" 2024-04-21T23:19:36.314 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-21T23:19:36.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.308+0000 7efdc8197640 1 -- 172.21.15.40:0/2605648367 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7efdac0a2f30 msgr2=0x7efdac0a53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:36.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.308+0000 7efdc8197640 1 --2- 172.21.15.40:0/2605648367 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7efdac0a2f30 0x7efdac0a53f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7efdb80cce20 tx=0x7efdb8097800 comp rx=0 tx=0).stop 2024-04-21T23:19:36.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.308+0000 7efdc8197640 1 -- 172.21.15.40:0/2605648367 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efdc0107730 msgr2=0x7efdc019c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:36.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.308+0000 7efdc8197640 1 --2- 172.21.15.40:0/2605648367 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efdc0107730 0x7efdc019c8d0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7efdb800eaa0 tx=0x7efdb800eea0 comp rx=0 tx=0).stop 2024-04-21T23:19:36.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.308+0000 7efdc8197640 1 -- 172.21.15.40:0/2605648367 shutdown_connections 2024-04-21T23:19:36.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.308+0000 7efdc8197640 1 --2- 172.21.15.40:0/2605648367 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7efdac0a2f30 0x7efdac0a53f0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:36.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.308+0000 7efdc8197640 1 --2- 172.21.15.40:0/2605648367 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efdc006a780 0x7efdc01985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:36.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.308+0000 7efdc8197640 1 --2- 172.21.15.40:0/2605648367 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efdc0107730 0x7efdc019c8d0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:36.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.308+0000 7efdc8197640 1 --2- 172.21.15.40:0/2605648367 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efdc0069dc0 0x7efdc0198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:36.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.308+0000 7efdc8197640 1 -- 172.21.15.40:0/2605648367 >> 172.21.15.40:0/2605648367 conn(0x7efdc00776a0 msgr2=0x7efdc00fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:36.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.308+0000 7efdc8197640 1 -- 172.21.15.40:0/2605648367 shutdown_connections 2024-04-21T23:19:36.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.308+0000 7efdc8197640 1 -- 172.21.15.40:0/2605648367 wait complete. 2024-04-21T23:19:36.319 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T23:19:36.319 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T23:19:36.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.435+0000 7f048b86d640 1 -- 172.21.15.40:0/3777400428 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f048410bd00 msgr2=0x7f048410e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:36.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.435+0000 7f048b86d640 1 --2- 172.21.15.40:0/3777400428 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f048410bd00 0x7f048410e1a0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f047c009ca0 tx=0x7f047c02c210 comp rx=0 tx=0).stop 2024-04-21T23:19:36.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.435+0000 7f048b86d640 1 -- 172.21.15.40:0/3777400428 shutdown_connections 2024-04-21T23:19:36.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.435+0000 7f048b86d640 1 --2- 172.21.15.40:0/3777400428 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f048410bd00 0x7f048410e1a0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:36.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.435+0000 7f048b86d640 1 --2- 172.21.15.40:0/3777400428 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f04841032e0 0x7f048410b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:36.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.435+0000 7f048b86d640 1 --2- 172.21.15.40:0/3777400428 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04841029c0 0x7f0484102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:36.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.435+0000 7f048b86d640 1 -- 172.21.15.40:0/3777400428 >> 172.21.15.40:0/3777400428 conn(0x7f0484077740 msgr2=0x7f0484077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:36.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.435+0000 7f048b86d640 1 -- 172.21.15.40:0/3777400428 shutdown_connections 2024-04-21T23:19:36.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.435+0000 7f048b86d640 1 -- 172.21.15.40:0/3777400428 wait complete. 2024-04-21T23:19:36.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.436+0000 7f048b86d640 1 Processor -- start 2024-04-21T23:19:36.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.436+0000 7f048b86d640 1 -- start start 2024-04-21T23:19:36.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.436+0000 7f048b86d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f04841029c0 0x7f0484198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:36.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.436+0000 7f048b86d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04841032e0 0x7f0484198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:36.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.436+0000 7f048b86d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f048410bd00 0x7f048419c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:36.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.436+0000 7f048b86d640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0484111b00 con 0x7f048410bd00 2024-04-21T23:19:36.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.436+0000 7f048b86d640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f0484111980 con 0x7f04841032e0 2024-04-21T23:19:36.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.436+0000 7f048b86d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0484111c80 con 0x7f04841029c0 2024-04-21T23:19:36.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.436+0000 7f0488e08640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04841032e0 0x7f0484198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:36.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.436+0000 7f0489e0a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f048410bd00 0x7f048419c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:36.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.436+0000 7f0489e0a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f048410bd00 0x7f048419c870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48410/0 (socket says 172.21.15.40:48410) 2024-04-21T23:19:36.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.436+0000 7f0489e0a640 1 -- 172.21.15.40:0/4181979093 learned_addr learned my addr 172.21.15.40:0/4181979093 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:36.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.437+0000 7f0489609640 1 --2- 172.21.15.40:0/4181979093 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f04841029c0 0x7f0484198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:36.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.437+0000 7f0488e08640 1 -- 172.21.15.40:0/4181979093 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f04841029c0 msgr2=0x7f0484198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:36.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.437+0000 7f0488e08640 1 --2- 172.21.15.40:0/4181979093 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f04841029c0 0x7f0484198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:36.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.437+0000 7f0488e08640 1 -- 172.21.15.40:0/4181979093 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f048410bd00 msgr2=0x7f048419c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:36.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.437+0000 7f0488e08640 1 --2- 172.21.15.40:0/4181979093 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f048410bd00 0x7f048419c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:36.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.437+0000 7f0488e08640 1 -- 172.21.15.40:0/4181979093 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f048419cfa0 con 0x7f04841032e0 2024-04-21T23:19:36.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.437+0000 7f0489e0a640 1 --2- 172.21.15.40:0/4181979093 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f048410bd00 0x7f048419c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:19:36.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.437+0000 7f0488e08640 1 --2- 172.21.15.40:0/4181979093 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04841032e0 0x7f0484198540 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0474005fd0 tx=0x7f047400dde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:36.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.438+0000 7f047a7fc640 1 -- 172.21.15.40:0/4181979093 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0474009c30 con 0x7f04841032e0 2024-04-21T23:19:36.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.438+0000 7f048b86d640 1 -- 172.21.15.40:0/4181979093 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f048419d290 con 0x7f04841032e0 2024-04-21T23:19:36.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.438+0000 7f048b86d640 1 -- 172.21.15.40:0/4181979093 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0484100540 con 0x7f04841032e0 2024-04-21T23:19:36.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.438+0000 7f047a7fc640 1 -- 172.21.15.40:0/4181979093 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0474009dd0 con 0x7f04841032e0 2024-04-21T23:19:36.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.438+0000 7f047a7fc640 1 -- 172.21.15.40:0/4181979093 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0474010460 con 0x7f04841032e0 2024-04-21T23:19:36.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.441+0000 7f048e44d740 1 Processor -- start 2024-04-21T23:19:36.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.442+0000 7f047a7fc640 1 -- 172.21.15.40:0/4181979093 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) ==== 138117+0+0 (secure 0 0 0) 0x7f04740035f0 con 0x7f04841032e0 2024-04-21T23:19:36.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.442+0000 7f048e44d740 1 -- start start 2024-04-21T23:19:36.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.442+0000 7f048e44d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b048562f30 0x55b048563310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:36.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.442+0000 7f048e44d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b048563850 0x55b04856bef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:36.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.442+0000 7f048e44d740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b04856c430 0x55b04856e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:36.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.442+0000 7f048e44d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55b0484c7560 con 0x55b048563850 2024-04-21T23:19:36.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.442+0000 7f048e44d740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55b04846f730 con 0x55b04856c430 2024-04-21T23:19:36.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.442+0000 7f048e44d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55b0484c3520 con 0x55b048562f30 2024-04-21T23:19:36.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.442+0000 7f0489e0a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b048562f30 0x55b048563310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:36.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.442+0000 7f0488e08640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b04856c430 0x55b04856e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:36.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.442+0000 7f0489609640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b048563850 0x55b04856bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:36.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.442+0000 7f0489e0a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b048562f30 0x55b048563310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.40:34908/0 (socket says 172.21.15.40:34908) 2024-04-21T23:19:36.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.442+0000 7f0489e0a640 1 -- 172.21.15.40:0/12884648 learned_addr learned my addr 172.21.15.40:0/12884648 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:36.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.442+0000 7f047a7fc640 1 --2- 172.21.15.40:0/4181979093 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f04700a2f30 0x7f04700a53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:36.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.442+0000 7f0489e0a640 1 -- 172.21.15.40:0/12884648 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b04856c430 msgr2=0x55b04856e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:36.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.442+0000 7f0489e0a640 1 --2- 172.21.15.40:0/12884648 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b04856c430 0x55b04856e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:36.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.442+0000 7f0489e0a640 1 -- 172.21.15.40:0/12884648 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b048563850 msgr2=0x55b04856bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:36.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.443+0000 7f0489e0a640 1 --2- 172.21.15.40:0/12884648 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b048563850 0x55b04856bef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:36.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.443+0000 7f0489e0a640 1 -- 172.21.15.40:0/12884648 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b0482a57d0 con 0x55b048562f30 2024-04-21T23:19:36.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.443+0000 7f0489609640 1 --2- 172.21.15.40:0/12884648 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b048563850 0x55b04856bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:19:36.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.443+0000 7f047a7fc640 1 -- 172.21.15.40:0/4181979093 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f04740cf360 con 0x7f04841032e0 2024-04-21T23:19:36.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.443+0000 7f0489e0a640 1 --2- 172.21.15.40:0/4181979093 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f04700a2f30 0x7f04700a53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:36.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.443+0000 7f0489e0a640 1 --2- 172.21.15.40:0/12884648 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b048562f30 0x55b048563310 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f047c005f80 tx=0x7f047c0098c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f3a312605f8e7d71 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:36.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.444+0000 7f046f7fe640 1 -- 172.21.15.40:0/12884648 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f047c050070 con 0x55b048562f30 2024-04-21T23:19:36.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.444+0000 7f0489e0a640 1 --2- 172.21.15.40:0/4181979093 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f04700a2f30 0x7f04700a53f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f047c009a60 tx=0x7f047c04d000 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:36.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.444+0000 7f046f7fe640 1 -- 172.21.15.40:0/12884648 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f047c03a040 con 0x55b048562f30 2024-04-21T23:19:36.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.444+0000 7f046f7fe640 1 -- 172.21.15.40:0/12884648 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f047c039040 con 0x55b048562f30 2024-04-21T23:19:36.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.445+0000 7f048e44d740 1 -- 172.21.15.40:0/12884648 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b048562f30 msgr2=0x55b048563310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:36.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.445+0000 7f048e44d740 1 --2- 172.21.15.40:0/12884648 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b048562f30 0x55b048563310 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f047c005f80 tx=0x7f047c0098c0 comp rx=0 tx=0).stop 2024-04-21T23:19:36.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.445+0000 7f048e44d740 1 -- 172.21.15.40:0/12884648 shutdown_connections 2024-04-21T23:19:36.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.445+0000 7f048e44d740 1 --2- 172.21.15.40:0/12884648 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b04856c430 0x55b04856e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:36.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.445+0000 7f048e44d740 1 --2- 172.21.15.40:0/12884648 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b048563850 0x55b04856bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:36.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.445+0000 7f048e44d740 1 --2- 172.21.15.40:0/12884648 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b048562f30 0x55b048563310 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:36.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.445+0000 7f048e44d740 1 -- 172.21.15.40:0/12884648 >> 172.21.15.40:0/12884648 conn(0x55b04855d660 msgr2=0x55b04855da40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:36.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.445+0000 7f048e44d740 1 -- 172.21.15.40:0/12884648 shutdown_connections 2024-04-21T23:19:36.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.445+0000 7f048e44d740 1 -- 172.21.15.40:0/12884648 wait complete. 2024-04-21T23:19:36.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.445+0000 7f048e44d740 1 -- start start 2024-04-21T23:19:36.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.445+0000 7f048e44d740 1 Processor -- start 2024-04-21T23:19:36.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.446+0000 7f048e44d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b048562f30 0x55b048569480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:36.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.446+0000 7f048e44d740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b048563850 0x55b0485699c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:36.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.446+0000 7f048e44d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b04856c430 0x55b048569f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:36.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.446+0000 7f048e44d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55b04856f220 con 0x55b048562f30 2024-04-21T23:19:36.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.446+0000 7f048e44d740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55b048482e00 con 0x55b048563850 2024-04-21T23:19:36.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.446+0000 7f048e44d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55b04856f3a0 con 0x55b04856c430 2024-04-21T23:19:36.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.446+0000 7f048e44d740 10 client.0 _ll_register_callbacks cb 0x55b0484bdeb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:19:36.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.446+0000 7f0489609640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b048562f30 0x55b048569480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:36.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.446+0000 7f0489609640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b048562f30 0x55b048569480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48420/0 (socket says 172.21.15.40:48420) 2024-04-21T23:19:36.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.446+0000 7f0489e0a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b04856c430 0x55b048569f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:36.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.446+0000 7f0489609640 1 -- 172.21.15.40:0/2345479379 learned_addr learned my addr 172.21.15.40:0/2345479379 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:36.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.446+0000 7f0488e08640 1 --2- 172.21.15.40:0/2345479379 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b048563850 0x55b0485699c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:36.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.447+0000 7f0489e0a640 1 -- 172.21.15.40:0/2345479379 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b048563850 msgr2=0x55b0485699c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:36.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.447+0000 7f0489e0a640 1 --2- 172.21.15.40:0/2345479379 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b048563850 0x55b0485699c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:36.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.447+0000 7f0489e0a640 1 -- 172.21.15.40:0/2345479379 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b048562f30 msgr2=0x55b048569480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:36.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.447+0000 7f0489e0a640 1 --2- 172.21.15.40:0/2345479379 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b048562f30 0x55b048569480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:36.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.447+0000 7f0489e0a640 1 -- 172.21.15.40:0/2345479379 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b0481db3f0 con 0x55b04856c430 2024-04-21T23:19:36.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.447+0000 7f0489609640 1 --2- 172.21.15.40:0/2345479379 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b048562f30 0x55b048569480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:19:36.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.447+0000 7f0489e0a640 1 --2- 172.21.15.40:0/2345479379 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b04856c430 0x55b048569f00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f047c005f80 tx=0x7f047c009800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c0cdf3c39f48d745 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:36.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.447+0000 7f046cff9640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-04-21T23:19:36.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.447+0000 7f046cff9640 1 -- 172.21.15.40:0/2345479379 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f047c05a020 con 0x55b04856c430 2024-04-21T23:19:36.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.447+0000 7f046cff9640 1 -- 172.21.15.40:0/2345479379 <== mon.1 v2:172.21.15.174:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f047c032ac0 con 0x55b04856c430 2024-04-21T23:19:36.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.447+0000 7f048e44d740 1 -- 172.21.15.40:0/2345479379 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f047c03a040 con 0x55b04856c430 2024-04-21T23:19:36.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.448+0000 7f046cff9640 1 -- 172.21.15.40:0/2345479379 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f047c065050 con 0x55b04856c430 2024-04-21T23:19:36.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.448+0000 7f048e44d740 10 client.25546 fetch_fsmap learned FSMap version 25 2024-04-21T23:19:36.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.448+0000 7f048e44d740 1 -- 172.21.15.40:0/2345479379 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55b04819b0f0 con 0x55b04856c430 2024-04-21T23:19:36.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.448+0000 7f046cff9640 1 -- 172.21.15.40:0/2345479379 <== mon.1 v2:172.21.15.174:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f047c03a040 con 0x55b04856c430 2024-04-21T23:19:36.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.448+0000 7f046cff9640 1 -- 172.21.15.40:0/2345479379 <== mon.1 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f047c056820 con 0x55b04856c430 2024-04-21T23:19:36.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.448+0000 7f048e44d740 10 client.25546 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:19:36.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.448+0000 7f048e44d740 10 client.25546 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:19:36.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.448+0000 7f048e44d740 20 client.25546 populate_metadata read hostname 'smithi040' 2024-04-21T23:19:36.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.448+0000 7f048e44d740 1 --2- 172.21.15.40:0/2345479379 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55b04857c6c0 0x55b04857eab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:36.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.449+0000 7f048e44d740 4 client.25546 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:19:36.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.449+0000 7f048e44d740 1 -- 172.21.15.40:0/2345479379 --> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b0484d4aa0 con 0x55b04857c6c0 2024-04-21T23:19:36.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.449+0000 7f0489609640 1 --2- 172.21.15.40:0/2345479379 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55b04857c6c0 0x55b04857eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:36.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.449+0000 7f046cff9640 10 client.25546 ms_handle_connect on v2:172.21.15.111:6838/3712235548 2024-04-21T23:19:36.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.449+0000 7f0489609640 1 --2- 172.21.15.40:0/2345479379 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55b04857c6c0 0x55b04857eab0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3513e72107076c48 server_cookie=910cf67b0fe290e2 in_seq=0 out_seq=0 2024-04-21T23:19:36.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.450+0000 7f046cff9640 1 -- 172.21.15.40:0/2345479379 <== mds.0 v2:172.21.15.111:6838/3712235548 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b0484d4aa0 con 0x55b04857c6c0 2024-04-21T23:19:36.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.450+0000 7f046cff9640 10 client.25546 handle_command_reply: tid=0 2024-04-21T23:19:36.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.450+0000 7f048e44d740 1 client.25546 shutdown 2024-04-21T23:19:36.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.450+0000 7f048e44d740 1 -- 172.21.15.40:0/2345479379 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b04856c430 msgr2=0x55b048569f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:36.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.450+0000 7f048e44d740 1 --2- 172.21.15.40:0/2345479379 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b04856c430 0x55b048569f00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f047c005f80 tx=0x7f047c009800 comp rx=0 tx=0).stop 2024-04-21T23:19:36.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.450+0000 7f048e44d740 1 -- 172.21.15.40:0/2345479379 shutdown_connections 2024-04-21T23:19:36.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.450+0000 7f048e44d740 1 --2- 172.21.15.40:0/2345479379 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55b04857c6c0 0x55b04857eab0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:36.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.450+0000 7f046cff9640 0 client.25546 ms_handle_reset on v2:172.21.15.111:6838/3712235548 2024-04-21T23:19:36.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.450+0000 7f048e44d740 1 --2- 172.21.15.40:0/2345479379 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b048563850 0x55b0485699c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:36.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.450+0000 7f048e44d740 1 --2- 172.21.15.40:0/2345479379 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b04856c430 0x55b048569f00 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:36.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.450+0000 7f048e44d740 1 --2- 172.21.15.40:0/2345479379 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b048562f30 0x55b048569480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:36.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.450+0000 7f048e44d740 1 -- 172.21.15.40:0/2345479379 >> 172.21.15.40:0/2345479379 conn(0x55b04855d660 msgr2=0x55b04855f3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:36.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.451+0000 7f048e44d740 1 -- 172.21.15.40:0/2345479379 shutdown_connections 2024-04-21T23:19:36.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.451+0000 7f048e44d740 1 -- 172.21.15.40:0/2345479379 wait complete. 2024-04-21T23:19:36.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.451+0000 7f048e44d740 20 client.25546 trim_cache size 0 max 16384 2024-04-21T23:19:36.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.469+0000 7f048e44d740 1 Processor -- start 2024-04-21T23:19:36.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.469+0000 7f048e44d740 1 -- start start 2024-04-21T23:19:36.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.469+0000 7f048e44d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b048562f30 0x55b04856bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:36.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.469+0000 7f048e44d740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b048563850 0x55b048578140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:36.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.469+0000 7f048e44d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b04856c430 0x55b048569b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:36.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.469+0000 7f048e44d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f04680127e0 con 0x55b048562f30 2024-04-21T23:19:36.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.469+0000 7f048e44d740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f04680100b0 con 0x55b048563850 2024-04-21T23:19:36.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.469+0000 7f048e44d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f046800df00 con 0x55b04856c430 2024-04-21T23:19:36.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.469+0000 7f0489609640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b048562f30 0x55b04856bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:36.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.469+0000 7f0489609640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b048562f30 0x55b04856bbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48428/0 (socket says 172.21.15.40:48428) 2024-04-21T23:19:36.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.469+0000 7f0489609640 1 -- 172.21.15.40:0/1297693202 learned_addr learned my addr 172.21.15.40:0/1297693202 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:36.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.469+0000 7f0488e08640 1 --2- 172.21.15.40:0/1297693202 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b048563850 0x55b048578140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:36.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.469+0000 7f0489e0a640 1 --2- 172.21.15.40:0/1297693202 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b04856c430 0x55b048569b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:36.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.469+0000 7f0489609640 1 -- 172.21.15.40:0/1297693202 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b04856c430 msgr2=0x55b048569b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:36.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.469+0000 7f0489609640 1 --2- 172.21.15.40:0/1297693202 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b04856c430 0x55b048569b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:36.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.469+0000 7f0489609640 1 -- 172.21.15.40:0/1297693202 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b048563850 msgr2=0x55b048578140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:36.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.469+0000 7f0489609640 1 --2- 172.21.15.40:0/1297693202 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b048563850 0x55b048578140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:36.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.469+0000 7f0489609640 1 -- 172.21.15.40:0/1297693202 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b04855e230 con 0x55b048562f30 2024-04-21T23:19:36.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.469+0000 7f0489e0a640 1 --2- 172.21.15.40:0/1297693202 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b04856c430 0x55b048569b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:19:36.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.470+0000 7f0489609640 1 --2- 172.21.15.40:0/1297693202 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b048562f30 0x55b04856bbb0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f0480009690 tx=0x7f0480004410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58cab8c4672757fa server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:36.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.470+0000 7f046effd640 1 -- 172.21.15.40:0/1297693202 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f048002d020 con 0x55b048562f30 2024-04-21T23:19:36.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.470+0000 7f046effd640 1 -- 172.21.15.40:0/1297693202 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0480009cd0 con 0x55b048562f30 2024-04-21T23:19:36.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.470+0000 7f046effd640 1 -- 172.21.15.40:0/1297693202 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f048000e6f0 con 0x55b048562f30 2024-04-21T23:19:36.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.471+0000 7f048e44d740 1 -- 172.21.15.40:0/1297693202 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b048562f30 msgr2=0x55b04856bbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:36.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.471+0000 7f048e44d740 1 --2- 172.21.15.40:0/1297693202 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b048562f30 0x55b04856bbb0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f0480009690 tx=0x7f0480004410 comp rx=0 tx=0).stop 2024-04-21T23:19:36.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.471+0000 7f048e44d740 1 -- 172.21.15.40:0/1297693202 shutdown_connections 2024-04-21T23:19:36.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.471+0000 7f048e44d740 1 --2- 172.21.15.40:0/1297693202 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b048563850 0x55b048578140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:36.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.471+0000 7f048e44d740 1 --2- 172.21.15.40:0/1297693202 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b04856c430 0x55b048569b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:36.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.471+0000 7f048e44d740 1 --2- 172.21.15.40:0/1297693202 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b048562f30 0x55b04856bbb0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:36.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.471+0000 7f048e44d740 1 -- 172.21.15.40:0/1297693202 >> 172.21.15.40:0/1297693202 conn(0x55b04855d660 msgr2=0x55b048569440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:36.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.471+0000 7f048e44d740 1 -- 172.21.15.40:0/1297693202 shutdown_connections 2024-04-21T23:19:36.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.471+0000 7f048e44d740 1 -- 172.21.15.40:0/1297693202 wait complete. 2024-04-21T23:19:36.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.471+0000 7f048e44d740 1 -- start start 2024-04-21T23:19:36.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.471+0000 7f048e44d740 1 Processor -- start 2024-04-21T23:19:36.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.472+0000 7f048e44d740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b048562f30 0x55b048574bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:36.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.472+0000 7f048e44d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b048563850 0x55b048566040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:36.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.472+0000 7f048e44d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b04856c430 0x55b048566780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:36.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.472+0000 7f048e44d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f046800e570 con 0x55b048563850 2024-04-21T23:19:36.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.472+0000 7f048e44d740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f046800f9e0 con 0x55b048562f30 2024-04-21T23:19:36.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.472+0000 7f0488e08640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b048563850 0x55b048566040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:36.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.472+0000 7f048e44d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f046800ec40 con 0x55b04856c430 2024-04-21T23:19:36.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.472+0000 7f0488e08640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b048563850 0x55b048566040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48434/0 (socket says 172.21.15.40:48434) 2024-04-21T23:19:36.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.472+0000 7f0488e08640 1 -- 172.21.15.40:0/3689971937 learned_addr learned my addr 172.21.15.40:0/3689971937 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:36.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.472+0000 7f048e44d740 10 client.0 _ll_register_callbacks cb 0x7f0480017bd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:19:36.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.472+0000 7f0489e0a640 1 --2- 172.21.15.40:0/3689971937 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b04856c430 0x55b048566780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:36.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.473+0000 7f0488e08640 1 -- 172.21.15.40:0/3689971937 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b04856c430 msgr2=0x55b048566780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:36.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.473+0000 7f0488e08640 1 --2- 172.21.15.40:0/3689971937 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b04856c430 0x55b048566780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:36.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.473+0000 7f0488e08640 1 -- 172.21.15.40:0/3689971937 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b048562f30 msgr2=0x55b048574bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:36.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.473+0000 7f0488e08640 1 --2- 172.21.15.40:0/3689971937 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b048562f30 0x55b048574bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:36.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.473+0000 7f0488e08640 1 -- 172.21.15.40:0/3689971937 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b0485e19b0 con 0x55b048563850 2024-04-21T23:19:36.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.473+0000 7f0488e08640 1 --2- 172.21.15.40:0/3689971937 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b048563850 0x55b048566040 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f0474009610 tx=0x7f04740976a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70df95830a24fce4 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:36.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.473+0000 7f046cff9640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-21T23:19:36.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.473+0000 7f046cff9640 1 -- 172.21.15.40:0/3689971937 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0474020040 con 0x55b048563850 2024-04-21T23:19:36.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.473+0000 7f046cff9640 1 -- 172.21.15.40:0/3689971937 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f04740bb2b0 con 0x55b048563850 2024-04-21T23:19:36.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.473+0000 7f048e44d740 1 -- 172.21.15.40:0/3689971937 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b0484cedf0 con 0x55b048563850 2024-04-21T23:19:36.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.473+0000 7f046cff9640 1 -- 172.21.15.40:0/3689971937 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f047409b340 con 0x55b048563850 2024-04-21T23:19:36.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.473+0000 7f048e44d740 10 client.16029 fetch_fsmap learned FSMap version 25 2024-04-21T23:19:36.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.473+0000 7f048e44d740 1 -- 172.21.15.40:0/3689971937 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55b0485e2ad0 con 0x55b048563850 2024-04-21T23:19:36.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.473+0000 7f046cff9640 1 -- 172.21.15.40:0/3689971937 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04740cce50 con 0x55b048563850 2024-04-21T23:19:36.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.474+0000 7f046cff9640 1 -- 172.21.15.40:0/3689971937 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f047409a790 con 0x55b048563850 2024-04-21T23:19:36.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.474+0000 7f048e44d740 10 client.16029 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:19:36.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.474+0000 7f048e44d740 10 client.16029 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:19:36.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.474+0000 7f048e44d740 20 client.16029 populate_metadata read hostname 'smithi040' 2024-04-21T23:19:36.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.474+0000 7f048e44d740 1 --2- 172.21.15.40:0/3689971937 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55b04857c6c0 0x55b0485e52e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:36.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.474+0000 7f048e44d740 4 client.16029 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T23:19:36.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.474+0000 7f048e44d740 1 -- 172.21.15.40:0/3689971937 --> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b0484cedf0 con 0x55b04857c6c0 2024-04-21T23:19:36.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.474+0000 7f0489609640 1 --2- 172.21.15.40:0/3689971937 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55b04857c6c0 0x55b0485e52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:36.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.475+0000 7f046cff9640 10 client.16029 ms_handle_connect on v2:172.21.15.111:6838/3712235548 2024-04-21T23:19:36.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.475+0000 7f0489609640 1 --2- 172.21.15.40:0/3689971937 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55b04857c6c0 0x55b0485e52e0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c0f8f28cac17588b server_cookie=3ba14c8046c69f1c in_seq=0 out_seq=0 2024-04-21T23:19:36.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.475+0000 7f046cff9640 1 -- 172.21.15.40:0/3689971937 <== mds.0 v2:172.21.15.111:6838/3712235548 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55b0484cedf0 con 0x55b04857c6c0 2024-04-21T23:19:36.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.475+0000 7f046cff9640 10 client.16029 handle_command_reply: tid=0 2024-04-21T23:19:36.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.475+0000 7f048e44d740 1 client.16029 shutdown 2024-04-21T23:19:36.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.475+0000 7f048e44d740 1 -- 172.21.15.40:0/3689971937 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b048563850 msgr2=0x55b048566040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:36.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.475+0000 7f048e44d740 1 --2- 172.21.15.40:0/3689971937 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b048563850 0x55b048566040 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f0474009610 tx=0x7f04740976a0 comp rx=0 tx=0).stop 2024-04-21T23:19:36.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.475+0000 7f048e44d740 1 -- 172.21.15.40:0/3689971937 shutdown_connections 2024-04-21T23:19:36.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.475+0000 7f048e44d740 1 --2- 172.21.15.40:0/3689971937 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55b04857c6c0 0x55b0485e52e0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:36.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.475+0000 7f048e44d740 1 --2- 172.21.15.40:0/3689971937 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b04856c430 0x55b048566780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:36.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.475+0000 7f048e44d740 1 --2- 172.21.15.40:0/3689971937 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b048563850 0x55b048566040 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:36.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.475+0000 7f048e44d740 1 --2- 172.21.15.40:0/3689971937 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b048562f30 0x55b048574bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:36.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.475+0000 7f048e44d740 1 -- 172.21.15.40:0/3689971937 >> 172.21.15.40:0/3689971937 conn(0x55b04855d660 msgr2=0x55b04855c6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:36.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.475+0000 7f046cff9640 0 client.16029 ms_handle_reset on v2:172.21.15.111:6838/3712235548 2024-04-21T23:19:36.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.475+0000 7f048e44d740 1 -- 172.21.15.40:0/3689971937 shutdown_connections 2024-04-21T23:19:36.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.475+0000 7f048e44d740 1 -- 172.21.15.40:0/3689971937 wait complete. 2024-04-21T23:19:36.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.475+0000 7f048e44d740 20 client.16029 trim_cache size 0 max 16384 2024-04-21T23:19:36.481 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-21T23:19:36.481 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-21T23:19:36.481 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": { 2024-04-21T23:19:36.481 INFO:teuthology.orchestra.run.smithi040.stdout: "9d181165-d6e4-436d-ad23-285885355ae4": { 2024-04-21T23:19:36.481 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/", 2024-04-21T23:19:36.481 INFO:teuthology.orchestra.run.smithi040.stdout: "tag": "9d181165-d6e4-436d-ad23-285885355ae4", 2024-04-21T23:19:36.481 INFO:teuthology.orchestra.run.smithi040.stdout: "options": "recursive,force" 2024-04-21T23:19:36.481 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:19:36.481 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:19:36.482 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-21T23:19:36.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.476+0000 7f048b86d640 1 -- 172.21.15.40:0/4181979093 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f04700a2f30 msgr2=0x7f04700a53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:36.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.476+0000 7f048b86d640 1 --2- 172.21.15.40:0/4181979093 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f04700a2f30 0x7f04700a53f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f047c009a60 tx=0x7f047c04d000 comp rx=0 tx=0).stop 2024-04-21T23:19:36.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.476+0000 7f048b86d640 1 -- 172.21.15.40:0/4181979093 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04841032e0 msgr2=0x7f0484198540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:36.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.476+0000 7f048b86d640 1 --2- 172.21.15.40:0/4181979093 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04841032e0 0x7f0484198540 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0474005fd0 tx=0x7f047400dde0 comp rx=0 tx=0).stop 2024-04-21T23:19:36.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.476+0000 7f048b86d640 1 -- 172.21.15.40:0/4181979093 shutdown_connections 2024-04-21T23:19:36.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.476+0000 7f048b86d640 1 --2- 172.21.15.40:0/4181979093 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f04700a2f30 0x7f04700a53f0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:36.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.476+0000 7f048b86d640 1 --2- 172.21.15.40:0/4181979093 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04841032e0 0x7f0484198540 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:36.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.476+0000 7f048b86d640 1 --2- 172.21.15.40:0/4181979093 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f048410bd00 0x7f048419c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:36.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.476+0000 7f048b86d640 1 --2- 172.21.15.40:0/4181979093 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f04841029c0 0x7f0484198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:36.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.476+0000 7f048b86d640 1 -- 172.21.15.40:0/4181979093 >> 172.21.15.40:0/4181979093 conn(0x7f0484077740 msgr2=0x7f04840fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:36.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.476+0000 7f048b86d640 1 -- 172.21.15.40:0/4181979093 shutdown_connections 2024-04-21T23:19:36.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:19:36.476+0000 7f048b86d640 1 -- 172.21.15.40:0/4181979093 wait complete. 2024-04-21T23:19:36.487 INFO:tasks.cephfs.filesystem:scrub status for tag:9d181165-d6e4-436d-ad23-285885355ae4 - {'path': '/', 'tag': '9d181165-d6e4-436d-ad23-285885355ae4', 'options': 'recursive,force'} 2024-04-21T23:19:37.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:19:37 smithi040 ceph-mon[27984]: pgmap v39: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-21T23:19:37.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:19:37 smithi040 ceph-mon[27984]: scrub queued for path: / 2024-04-21T23:19:37.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:19:37 smithi040 ceph-mon[27984]: scrub summary: idle+waiting paths [/] 2024-04-21T23:19:37.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:19:37 smithi040 ceph-mon[27984]: scrub summary: active paths [/] 2024-04-21T23:19:37.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:19:37 smithi040 ceph-mon[27984]: scrub summary: idle+waiting paths [/] 2024-04-21T23:19:37.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:19:37 smithi174 ceph-mon[31409]: pgmap v39: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-21T23:19:37.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:19:37 smithi174 ceph-mon[31409]: scrub queued for path: / 2024-04-21T23:19:37.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:19:37 smithi174 ceph-mon[31409]: scrub summary: idle+waiting paths [/] 2024-04-21T23:19:37.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:19:37 smithi174 ceph-mon[31409]: scrub summary: active paths [/] 2024-04-21T23:19:37.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:19:37 smithi174 ceph-mon[31409]: scrub summary: idle+waiting paths [/] 2024-04-21T23:19:37.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:19:37 smithi111 ceph-mon[31085]: pgmap v39: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-21T23:19:37.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:19:37 smithi111 ceph-mon[31085]: scrub queued for path: / 2024-04-21T23:19:37.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:19:37 smithi111 ceph-mon[31085]: scrub summary: idle+waiting paths [/] 2024-04-21T23:19:37.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:19:37 smithi111 ceph-mon[31085]: scrub summary: active paths [/] 2024-04-21T23:19:37.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:19:37 smithi111 ceph-mon[31085]: scrub summary: idle+waiting paths [/] 2024-04-21T23:19:38.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:19:38 smithi040 ceph-mon[27984]: pgmap v40: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-21T23:19:38.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:19:38 smithi174 ceph-mon[31409]: pgmap v40: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-21T23:19:38.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:19:38 smithi111 ceph-mon[31085]: pgmap v40: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-21T23:19:38.844 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.845+0000 7f8d0393a640 1 -- 172.21.15.40:0/1691906218 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8cfc106630 msgr2=0x7f8cfc106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:38.845 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.845+0000 7f8d0393a640 1 --2- 172.21.15.40:0/1691906218 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8cfc106630 0x7f8cfc106a10 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f8cf8009930 tx=0x7f8cf802bc30 comp rx=0 tx=0).stop 2024-04-21T23:19:38.845 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.845+0000 7f8d0393a640 1 -- 172.21.15.40:0/1691906218 shutdown_connections 2024-04-21T23:19:38.845 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.845+0000 7f8d0393a640 1 --2- 172.21.15.40:0/1691906218 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8cfc101030 0x7f8cfc10d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:38.845 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.845+0000 7f8d0393a640 1 --2- 172.21.15.40:0/1691906218 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8cfc100670 0x7f8cfc100af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:38.845 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.845+0000 7f8d0393a640 1 --2- 172.21.15.40:0/1691906218 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8cfc106630 0x7f8cfc106a10 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:38.845 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.845+0000 7f8d0393a640 1 -- 172.21.15.40:0/1691906218 >> 172.21.15.40:0/1691906218 conn(0x7f8cfc0777c0 msgr2=0x7f8cfc077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:38.845 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.845+0000 7f8d0393a640 1 -- 172.21.15.40:0/1691906218 shutdown_connections 2024-04-21T23:19:38.845 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.845+0000 7f8d0393a640 1 -- 172.21.15.40:0/1691906218 wait complete. 2024-04-21T23:19:38.847 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.846+0000 7f8d0393a640 1 Processor -- start 2024-04-21T23:19:38.847 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.846+0000 7f8d0393a640 1 -- start start 2024-04-21T23:19:38.847 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.846+0000 7f8d0393a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8cfc100670 0x7f8cfc198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:38.847 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.846+0000 7f8d0393a640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8cfc101030 0x7f8cfc198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:38.847 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.846+0000 7f8d0393a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8cfc106630 0x7f8cfc19c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:38.847 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.847+0000 7f8d0393a640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8cfc110a40 con 0x7f8cfc106630 2024-04-21T23:19:38.847 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.847+0000 7f8d01ed7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8cfc106630 0x7f8cfc19c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:38.847 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.847+0000 7f8d00ed5640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8cfc101030 0x7f8cfc198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:38.848 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.847+0000 7f8d00ed5640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8cfc101030 0x7f8cfc198560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.40:54046/0 (socket says 172.21.15.40:54046) 2024-04-21T23:19:38.848 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.847+0000 7f8d00ed5640 1 -- 172.21.15.40:0/2620069225 learned_addr learned my addr 172.21.15.40:0/2620069225 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:38.848 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.847+0000 7f8d0393a640 1 -- 172.21.15.40:0/2620069225 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f8cfc1108c0 con 0x7f8cfc101030 2024-04-21T23:19:38.848 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.847+0000 7f8d0393a640 1 -- 172.21.15.40:0/2620069225 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8cfc110bc0 con 0x7f8cfc100670 2024-04-21T23:19:38.848 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.847+0000 7f8d016d6640 1 --2- 172.21.15.40:0/2620069225 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8cfc100670 0x7f8cfc198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:38.848 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.847+0000 7f8d01ed7640 1 -- 172.21.15.40:0/2620069225 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8cfc100670 msgr2=0x7f8cfc198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:38.848 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.847+0000 7f8d01ed7640 1 --2- 172.21.15.40:0/2620069225 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8cfc100670 0x7f8cfc198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:38.848 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.847+0000 7f8d01ed7640 1 -- 172.21.15.40:0/2620069225 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8cfc101030 msgr2=0x7f8cfc198560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:38.848 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.847+0000 7f8d01ed7640 1 --2- 172.21.15.40:0/2620069225 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8cfc101030 0x7f8cfc198560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:38.848 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.847+0000 7f8d01ed7640 1 -- 172.21.15.40:0/2620069225 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cfc19d010 con 0x7f8cfc106630 2024-04-21T23:19:38.848 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.847+0000 7f8d01ed7640 1 --2- 172.21.15.40:0/2620069225 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8cfc106630 0x7f8cfc19c8e0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f8cf400e940 tx=0x7f8cf400ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:38.848 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.848+0000 7f8cf27fc640 1 -- 172.21.15.40:0/2620069225 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cf4004d20 con 0x7f8cfc106630 2024-04-21T23:19:38.848 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.848+0000 7f8cf27fc640 1 -- 172.21.15.40:0/2620069225 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8cf4005020 con 0x7f8cfc106630 2024-04-21T23:19:38.849 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.848+0000 7f8cf27fc640 1 -- 172.21.15.40:0/2620069225 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cf40116b0 con 0x7f8cfc106630 2024-04-21T23:19:38.849 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.848+0000 7f8d0393a640 1 -- 172.21.15.40:0/2620069225 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8cfc19d300 con 0x7f8cfc106630 2024-04-21T23:19:38.849 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.849+0000 7f8d0393a640 1 -- 172.21.15.40:0/2620069225 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8cfc1a4b90 con 0x7f8cfc106630 2024-04-21T23:19:38.850 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.851+0000 7f8cf27fc640 1 -- 172.21.15.40:0/2620069225 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 25) ==== 138117+0+0 (secure 0 0 0) 0x7f8cf40054d0 con 0x7f8cfc106630 2024-04-21T23:19:38.852 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.852+0000 7f8d065fe740 1 Processor -- start 2024-04-21T23:19:38.852 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.852+0000 7f8d065fe740 1 -- start start 2024-04-21T23:19:38.852 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.852+0000 7f8cf27fc640 1 --2- 172.21.15.40:0/2620069225 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f8ce80a2fd0 0x7f8ce80a5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:38.852 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.852+0000 7f8d065fe740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55568a0bfb20 0x55568a0bff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:38.852 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.852+0000 7f8d065fe740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55568a0c0440 0x55568a0c8cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:38.852 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.852+0000 7f8d065fe740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55568a0c9210 0x55568a0cb600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:38.852 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.852+0000 7f8d065fe740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55568a023b70 con 0x55568a0c9210 2024-04-21T23:19:38.852 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.852+0000 7f8d065fe740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x555689fcc470 con 0x55568a0c0440 2024-04-21T23:19:38.853 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.852+0000 7f8d065fe740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55568a01fb30 con 0x55568a0bfb20 2024-04-21T23:19:38.853 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.852+0000 7f8cf27fc640 1 -- 172.21.15.40:0/2620069225 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f8cf40cf720 con 0x7f8cfc106630 2024-04-21T23:19:38.853 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.853+0000 7f8d00ed5640 1 --2- 172.21.15.40:0/2620069225 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f8ce80a2fd0 0x7f8ce80a5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:38.853 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.853+0000 7f8d01ed7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55568a0c9210 0x55568a0cb600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:38.853 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.853+0000 7f8d01ed7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55568a0c9210 0x55568a0cb600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48456/0 (socket says 172.21.15.40:48456) 2024-04-21T23:19:38.853 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.853+0000 7f8d01ed7640 1 -- 172.21.15.40:0/1042503080 learned_addr learned my addr 172.21.15.40:0/1042503080 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:38.853 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.853+0000 7f8d016d6640 1 --2- 172.21.15.40:0/1042503080 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55568a0bfb20 0x55568a0bff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:38.857 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.853+0000 7f8d01ed7640 1 -- 172.21.15.40:0/1042503080 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55568a0bfb20 msgr2=0x55568a0bff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:38.857 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.853+0000 7f8d01ed7640 1 --2- 172.21.15.40:0/1042503080 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55568a0bfb20 0x55568a0bff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:38.858 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.853+0000 7f8d01ed7640 1 -- 172.21.15.40:0/1042503080 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55568a0c0440 msgr2=0x55568a0c8cd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T23:19:38.858 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.853+0000 7f8d01ed7640 1 --2- 172.21.15.40:0/1042503080 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55568a0c0440 0x55568a0c8cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:38.858 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.853+0000 7f8d01ed7640 1 -- 172.21.15.40:0/1042503080 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555689d38210 con 0x55568a0c9210 2024-04-21T23:19:38.858 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.853+0000 7f8d016d6640 1 --2- 172.21.15.40:0/1042503080 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55568a0bfb20 0x55568a0bff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:19:38.858 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.853+0000 7f8d01ed7640 1 --2- 172.21.15.40:0/1042503080 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55568a0c9210 0x55568a0cb600 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f8cf4005c40 tx=0x7f8cf409b120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=843c381e0a21e5db server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:38.858 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.853+0000 7f8d00ed5640 1 --2- 172.21.15.40:0/2620069225 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f8ce80a2fd0 0x7f8ce80a5490 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8cfc078180 tx=0x7f8cec00b410 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:38.858 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.853+0000 7f8ce77fe640 1 -- 172.21.15.40:0/1042503080 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cf401f040 con 0x55568a0c9210 2024-04-21T23:19:38.858 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.854+0000 7f8ce77fe640 1 -- 172.21.15.40:0/1042503080 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8cf40b9070 con 0x55568a0c9210 2024-04-21T23:19:38.858 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.854+0000 7f8ce77fe640 1 -- 172.21.15.40:0/1042503080 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cf409ab50 con 0x55568a0c9210 2024-04-21T23:19:38.858 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.854+0000 7f8d065fe740 1 -- 172.21.15.40:0/1042503080 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55568a0c9210 msgr2=0x55568a0cb600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:38.858 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.854+0000 7f8d065fe740 1 --2- 172.21.15.40:0/1042503080 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55568a0c9210 0x55568a0cb600 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f8cf4005c40 tx=0x7f8cf409b120 comp rx=0 tx=0).stop 2024-04-21T23:19:38.858 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.854+0000 7f8d065fe740 1 -- 172.21.15.40:0/1042503080 shutdown_connections 2024-04-21T23:19:38.859 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.855+0000 7f8d065fe740 1 --2- 172.21.15.40:0/1042503080 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55568a0c0440 0x55568a0c8cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:38.859 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.855+0000 7f8d065fe740 1 --2- 172.21.15.40:0/1042503080 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55568a0c9210 0x55568a0cb600 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:38.859 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.855+0000 7f8d065fe740 1 --2- 172.21.15.40:0/1042503080 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55568a0bfb20 0x55568a0bff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:38.859 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.855+0000 7f8d065fe740 1 -- 172.21.15.40:0/1042503080 >> 172.21.15.40:0/1042503080 conn(0x55568a0ba440 msgr2=0x55568a0ba820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:38.859 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.855+0000 7f8d065fe740 1 -- 172.21.15.40:0/1042503080 shutdown_connections 2024-04-21T23:19:38.859 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.855+0000 7f8d065fe740 1 -- 172.21.15.40:0/1042503080 wait complete. 2024-04-21T23:19:38.859 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.855+0000 7f8d065fe740 1 -- start start 2024-04-21T23:19:38.859 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.855+0000 7f8d065fe740 1 Processor -- start 2024-04-21T23:19:38.859 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.856+0000 7f8d065fe740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55568a0bfb20 0x55568a0c6330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:38.859 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.856+0000 7f8d065fe740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55568a0c0440 0x55568a0c6870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:38.859 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.856+0000 7f8d065fe740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55568a0c9210 0x55568a0c6db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:38.859 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.856+0000 7f8d065fe740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55568a0cbde0 con 0x55568a0c0440 2024-04-21T23:19:38.859 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.856+0000 7f8d00ed5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55568a0c0440 0x55568a0c6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:38.859 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.856+0000 7f8d00ed5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55568a0c0440 0x55568a0c6870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48472/0 (socket says 172.21.15.40:48472) 2024-04-21T23:19:38.859 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.856+0000 7f8d00ed5640 1 -- 172.21.15.40:0/159429177 learned_addr learned my addr 172.21.15.40:0/159429177 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:38.860 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.856+0000 7f8d065fe740 1 -- 172.21.15.40:0/159429177 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x555689fdfa10 con 0x55568a0c9210 2024-04-21T23:19:38.860 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.856+0000 7f8d065fe740 1 -- 172.21.15.40:0/159429177 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55568a0cbf60 con 0x55568a0bfb20 2024-04-21T23:19:38.860 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.856+0000 7f8d016d6640 1 --2- 172.21.15.40:0/159429177 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55568a0bfb20 0x55568a0c6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:38.860 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.856+0000 7f8d01ed7640 1 --2- 172.21.15.40:0/159429177 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55568a0c9210 0x55568a0c6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:38.860 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.856+0000 7f8d065fe740 10 client.0 _ll_register_callbacks cb 0x55568a01ab60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:19:38.860 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.856+0000 7f8d016d6640 1 -- 172.21.15.40:0/159429177 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55568a0c9210 msgr2=0x55568a0c6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:38.860 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.856+0000 7f8d016d6640 1 --2- 172.21.15.40:0/159429177 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55568a0c9210 0x55568a0c6db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:38.860 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.856+0000 7f8d016d6640 1 -- 172.21.15.40:0/159429177 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55568a0c0440 msgr2=0x55568a0c6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:38.860 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.856+0000 7f8d016d6640 1 --2- 172.21.15.40:0/159429177 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55568a0c0440 0x55568a0c6870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:38.860 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.856+0000 7f8d016d6640 1 -- 172.21.15.40:0/159429177 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555689e04620 con 0x55568a0bfb20 2024-04-21T23:19:38.860 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.856+0000 7f8d00ed5640 1 --2- 172.21.15.40:0/159429177 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55568a0c0440 0x55568a0c6870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:19:38.860 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.857+0000 7f8d065fe740 1 -- 172.21.15.40:0/159429177 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8cf40b9070 con 0x55568a0bfb20 2024-04-21T23:19:38.860 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.857+0000 7f8d016d6640 1 --2- 172.21.15.40:0/159429177 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55568a0bfb20 0x55568a0c6330 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f8cf8009230 tx=0x7f8cf80095c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7314ea61f7fda49e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:38.860 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.857+0000 7f8ce4ff9640 10 client.25561 ms_handle_connect on v2:172.21.15.174:3300/0 2024-04-21T23:19:38.861 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.857+0000 7f8ce4ff9640 1 -- 172.21.15.40:0/159429177 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cf8004300 con 0x55568a0bfb20 2024-04-21T23:19:38.861 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.857+0000 7f8ce4ff9640 1 -- 172.21.15.40:0/159429177 <== mon.1 v2:172.21.15.174:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f8cf80046c0 con 0x55568a0bfb20 2024-04-21T23:19:38.861 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.857+0000 7f8ce4ff9640 1 -- 172.21.15.40:0/159429177 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f8cf80048e0 con 0x55568a0bfb20 2024-04-21T23:19:38.861 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.857+0000 7f8d065fe740 10 client.25561 fetch_fsmap learned FSMap version 25 2024-04-21T23:19:38.861 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.857+0000 7f8d065fe740 1 -- 172.21.15.40:0/159429177 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({fsmap=25}) -- 0x55568a0d7fa0 con 0x55568a0bfb20 2024-04-21T23:19:38.861 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.858+0000 7f8ce4ff9640 1 -- 172.21.15.40:0/159429177 <== mon.1 v2:172.21.15.174:3300/0 4 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f8cf8004a80 con 0x55568a0bfb20 2024-04-21T23:19:38.861 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.858+0000 7f8d065fe740 10 client.25561 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:19:38.861 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.858+0000 7f8d065fe740 10 client.25561 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:19:38.861 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.858+0000 7f8d065fe740 20 client.25561 populate_metadata read hostname 'smithi040' 2024-04-21T23:19:38.861 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.858+0000 7f8d065fe740 1 --2- 172.21.15.40:0/159429177 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55568a0d93f0 0x55568a0db880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:38.861 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.858+0000 7f8d065fe740 4 client.25561 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:19:38.861 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.858+0000 7f8d065fe740 1 -- 172.21.15.40:0/159429177 --> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55568a0310b0 con 0x55568a0d93f0 2024-04-21T23:19:38.861 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.858+0000 7f8d00ed5640 1 --2- 172.21.15.40:0/159429177 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55568a0d93f0 0x55568a0db880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:38.863 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.859+0000 7f8ce4ff9640 10 client.25561 ms_handle_connect on v2:172.21.15.111:6838/3712235548 2024-04-21T23:19:38.863 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.859+0000 7f8d00ed5640 1 --2- 172.21.15.40:0/159429177 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55568a0d93f0 0x55568a0db880 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7a80a10ebb9b013d server_cookie=18c1e64760e38237 in_seq=0 out_seq=0 2024-04-21T23:19:38.863 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.859+0000 7f8ce4ff9640 1 -- 172.21.15.40:0/159429177 <== mds.0 v2:172.21.15.111:6838/3712235548 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55568a0310b0 con 0x55568a0d93f0 2024-04-21T23:19:38.863 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.859+0000 7f8ce4ff9640 10 client.25561 handle_command_reply: tid=0 2024-04-21T23:19:38.864 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.859+0000 7f8d065fe740 1 client.25561 shutdown 2024-04-21T23:19:38.864 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.860+0000 7f8d065fe740 1 -- 172.21.15.40:0/159429177 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55568a0bfb20 msgr2=0x55568a0c6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:38.864 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.860+0000 7f8d065fe740 1 --2- 172.21.15.40:0/159429177 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55568a0bfb20 0x55568a0c6330 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f8cf8009230 tx=0x7f8cf80095c0 comp rx=0 tx=0).stop 2024-04-21T23:19:38.864 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.860+0000 7f8d065fe740 1 -- 172.21.15.40:0/159429177 shutdown_connections 2024-04-21T23:19:38.864 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.860+0000 7f8d065fe740 1 --2- 172.21.15.40:0/159429177 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55568a0d93f0 0x55568a0db880 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:38.864 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.860+0000 7f8d065fe740 1 --2- 172.21.15.40:0/159429177 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55568a0c9210 0x55568a0c6db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:38.864 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.860+0000 7f8ce4ff9640 0 client.25561 ms_handle_reset on v2:172.21.15.111:6838/3712235548 2024-04-21T23:19:38.864 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.860+0000 7f8d065fe740 1 --2- 172.21.15.40:0/159429177 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55568a0c0440 0x55568a0c6870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:38.864 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.860+0000 7f8d065fe740 1 --2- 172.21.15.40:0/159429177 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55568a0bfb20 0x55568a0c6330 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:38.864 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.860+0000 7f8d065fe740 1 -- 172.21.15.40:0/159429177 >> 172.21.15.40:0/159429177 conn(0x55568a0ba440 msgr2=0x55568a0bc190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:38.864 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.860+0000 7f8d065fe740 1 -- 172.21.15.40:0/159429177 shutdown_connections 2024-04-21T23:19:38.864 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.860+0000 7f8d065fe740 1 -- 172.21.15.40:0/159429177 wait complete. 2024-04-21T23:19:38.864 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.860+0000 7f8d065fe740 20 client.25561 trim_cache size 0 max 16384 2024-04-21T23:19:38.891 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.890+0000 7f8d065fe740 1 Processor -- start 2024-04-21T23:19:38.891 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.890+0000 7f8d065fe740 1 -- start start 2024-04-21T23:19:38.891 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.891+0000 7f8d065fe740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55568a0bfb20 0x55568a0c8a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:38.891 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.891+0000 7f8d065fe740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55568a0c0440 0x55568a0d4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:38.891 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.891+0000 7f8d065fe740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55568a0c9210 0x55568a0c0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:38.891 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.891+0000 7f8d065fe740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8cfc1af350 con 0x55568a0c0440 2024-04-21T23:19:38.891 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.891+0000 7f8d065fe740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f8cfc1acc20 con 0x55568a0bfb20 2024-04-21T23:19:38.891 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.891+0000 7f8d065fe740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8cfc1aaa70 con 0x55568a0c9210 2024-04-21T23:19:38.891 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.891+0000 7f8d00ed5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55568a0c0440 0x55568a0d4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:38.891 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.891+0000 7f8d016d6640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55568a0bfb20 0x55568a0c8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:38.892 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.891+0000 7f8d00ed5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55568a0c0440 0x55568a0d4ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48486/0 (socket says 172.21.15.40:48486) 2024-04-21T23:19:38.892 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.891+0000 7f8d00ed5640 1 -- 172.21.15.40:0/162992873 learned_addr learned my addr 172.21.15.40:0/162992873 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:38.892 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.891+0000 7f8d01ed7640 1 --2- 172.21.15.40:0/162992873 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55568a0c9210 0x55568a0c0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:38.892 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.891+0000 7f8d016d6640 1 -- 172.21.15.40:0/162992873 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55568a0c9210 msgr2=0x55568a0c0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:38.892 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.891+0000 7f8d016d6640 1 --2- 172.21.15.40:0/162992873 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55568a0c9210 0x55568a0c0ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:38.892 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.891+0000 7f8d016d6640 1 -- 172.21.15.40:0/162992873 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55568a0c0440 msgr2=0x55568a0d4ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:38.892 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.891+0000 7f8d016d6640 1 --2- 172.21.15.40:0/162992873 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55568a0c0440 0x55568a0d4ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:38.892 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.891+0000 7f8d016d6640 1 -- 172.21.15.40:0/162992873 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55568a0d5be0 con 0x55568a0bfb20 2024-04-21T23:19:38.892 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.891+0000 7f8d016d6640 1 --2- 172.21.15.40:0/162992873 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55568a0bfb20 0x55568a0c8a60 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f8cf802c170 tx=0x7f8cf802e070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd7e70721a1a3a3a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:38.892 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.892+0000 7f8ce6ffd640 1 -- 172.21.15.40:0/162992873 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cf80046c0 con 0x55568a0bfb20 2024-04-21T23:19:38.892 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.892+0000 7f8ce6ffd640 1 -- 172.21.15.40:0/162992873 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8cf40b9070 con 0x55568a0bfb20 2024-04-21T23:19:38.892 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.892+0000 7f8ce6ffd640 1 -- 172.21.15.40:0/162992873 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cf80035f0 con 0x55568a0bfb20 2024-04-21T23:19:38.896 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.892+0000 7f8d065fe740 1 -- 172.21.15.40:0/162992873 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55568a0bfb20 msgr2=0x55568a0c8a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:38.896 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.892+0000 7f8d065fe740 1 --2- 172.21.15.40:0/162992873 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55568a0bfb20 0x55568a0c8a60 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f8cf802c170 tx=0x7f8cf802e070 comp rx=0 tx=0).stop 2024-04-21T23:19:38.896 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.892+0000 7f8d065fe740 1 -- 172.21.15.40:0/162992873 shutdown_connections 2024-04-21T23:19:38.897 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.892+0000 7f8d065fe740 1 --2- 172.21.15.40:0/162992873 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55568a0c9210 0x55568a0c0ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:38.897 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.892+0000 7f8d065fe740 1 --2- 172.21.15.40:0/162992873 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55568a0c0440 0x55568a0d4ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:38.897 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.892+0000 7f8d065fe740 1 --2- 172.21.15.40:0/162992873 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55568a0bfb20 0x55568a0c8a60 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:38.897 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.892+0000 7f8d065fe740 1 -- 172.21.15.40:0/162992873 >> 172.21.15.40:0/162992873 conn(0x55568a0ba440 msgr2=0x55568a0bb860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:38.897 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.892+0000 7f8d065fe740 1 -- 172.21.15.40:0/162992873 shutdown_connections 2024-04-21T23:19:38.897 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.892+0000 7f8d065fe740 1 -- 172.21.15.40:0/162992873 wait complete. 2024-04-21T23:19:38.897 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.893+0000 7f8d065fe740 1 -- start start 2024-04-21T23:19:38.897 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.893+0000 7f8d065fe740 1 Processor -- start 2024-04-21T23:19:38.897 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.893+0000 7f8d065fe740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55568a0bfb20 0x55568a0c3400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:38.897 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.893+0000 7f8d065fe740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55568a0c0440 0x55568a0c3940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:38.897 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.894+0000 7f8d016d6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55568a0bfb20 0x55568a0c3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:38.897 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.894+0000 7f8d016d6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55568a0bfb20 0x55568a0c3400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48500/0 (socket says 172.21.15.40:48500) 2024-04-21T23:19:38.898 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.894+0000 7f8d00ed5640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55568a0c0440 0x55568a0c3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:38.898 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.894+0000 7f8d00ed5640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55568a0c0440 0x55568a0c3940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.40:34992/0 (socket says 172.21.15.40:34992) 2024-04-21T23:19:38.898 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.894+0000 7f8d065fe740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55568a0c9210 0x55568a13e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:38.898 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.894+0000 7f8d016d6640 1 -- 172.21.15.40:0/1733982434 learned_addr learned my addr 172.21.15.40:0/1733982434 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:38.898 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.894+0000 7f8d065fe740 1 -- 172.21.15.40:0/1733982434 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8cfc1ab0e0 con 0x55568a0bfb20 2024-04-21T23:19:38.898 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.894+0000 7f8d065fe740 1 -- 172.21.15.40:0/1733982434 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f8cfc1ac550 con 0x55568a0c9210 2024-04-21T23:19:38.898 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.894+0000 7f8d065fe740 1 -- 172.21.15.40:0/1733982434 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8cfc1ab7b0 con 0x55568a0c0440 2024-04-21T23:19:38.898 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.894+0000 7f8d065fe740 10 client.0 _ll_register_callbacks cb 0x7f8cf802c330 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:19:38.898 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.894+0000 7f8d01ed7640 1 --2- 172.21.15.40:0/1733982434 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55568a0c9210 0x55568a13e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:38.898 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.894+0000 7f8d016d6640 1 -- 172.21.15.40:0/1733982434 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55568a0c0440 msgr2=0x55568a0c3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:38.899 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.894+0000 7f8d016d6640 1 --2- 172.21.15.40:0/1733982434 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55568a0c0440 0x55568a0c3940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:38.899 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.894+0000 7f8d016d6640 1 -- 172.21.15.40:0/1733982434 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55568a0c9210 msgr2=0x55568a13e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:38.899 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.894+0000 7f8d016d6640 1 --2- 172.21.15.40:0/1733982434 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55568a0c9210 0x55568a13e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:38.899 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.894+0000 7f8d016d6640 1 -- 172.21.15.40:0/1733982434 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55568a13ef10 con 0x55568a0bfb20 2024-04-21T23:19:38.899 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.894+0000 7f8d01ed7640 1 --2- 172.21.15.40:0/1733982434 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55568a0c9210 0x55568a13e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:19:38.899 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.895+0000 7f8d016d6640 1 --2- 172.21.15.40:0/1733982434 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55568a0bfb20 0x55568a0c3400 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f8cf8004e60 tx=0x7f8cf8005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a55dcb96cc60cf8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:38.899 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.895+0000 7f8ce4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-21T23:19:38.899 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.895+0000 7f8ce4ff9640 1 -- 172.21.15.40:0/1733982434 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cf80054e0 con 0x55568a0bfb20 2024-04-21T23:19:38.899 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.895+0000 7f8ce4ff9640 1 -- 172.21.15.40:0/1733982434 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f8cf8005680 con 0x55568a0bfb20 2024-04-21T23:19:38.899 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.895+0000 7f8d065fe740 1 -- 172.21.15.40:0/1733982434 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8cf40b9070 con 0x55568a0bfb20 2024-04-21T23:19:38.899 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.895+0000 7f8ce4ff9640 1 -- 172.21.15.40:0/1733982434 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f8cf8004100 con 0x55568a0bfb20 2024-04-21T23:19:38.899 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.895+0000 7f8d065fe740 10 client.16047 fetch_fsmap learned FSMap version 25 2024-04-21T23:19:38.900 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.895+0000 7f8d065fe740 1 -- 172.21.15.40:0/1733982434 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55568a140080 con 0x55568a0bfb20 2024-04-21T23:19:38.900 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.895+0000 7f8ce4ff9640 1 -- 172.21.15.40:0/1733982434 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8cf40b9070 con 0x55568a0bfb20 2024-04-21T23:19:38.900 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.895+0000 7f8ce4ff9640 1 -- 172.21.15.40:0/1733982434 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f8cf8051500 con 0x55568a0bfb20 2024-04-21T23:19:38.900 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.895+0000 7f8d065fe740 10 client.16047 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:19:38.900 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.895+0000 7f8d065fe740 10 client.16047 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:19:38.900 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.895+0000 7f8d065fe740 20 client.16047 populate_metadata read hostname 'smithi040' 2024-04-21T23:19:38.900 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.896+0000 7f8d065fe740 1 --2- 172.21.15.40:0/1733982434 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55568a0d93f0 0x55568a142770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:38.900 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.896+0000 7f8d065fe740 4 client.16047 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T23:19:38.900 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.896+0000 7f8d065fe740 1 -- 172.21.15.40:0/1733982434 --> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55568a0b5900 con 0x55568a0d93f0 2024-04-21T23:19:38.900 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.896+0000 7f8d00ed5640 1 --2- 172.21.15.40:0/1733982434 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55568a0d93f0 0x55568a142770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:38.901 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.896+0000 7f8ce4ff9640 10 client.16047 ms_handle_connect on v2:172.21.15.111:6838/3712235548 2024-04-21T23:19:38.901 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.896+0000 7f8d00ed5640 1 --2- 172.21.15.40:0/1733982434 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55568a0d93f0 0x55568a142770 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=851007b5aa966eff server_cookie=3341866258b5f7d9 in_seq=0 out_seq=0 2024-04-21T23:19:38.902 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.900+0000 7f8ce4ff9640 1 -- 172.21.15.40:0/1733982434 <== mds.0 v2:172.21.15.111:6838/3712235548 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55568a0b5900 con 0x55568a0d93f0 2024-04-21T23:19:38.903 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.900+0000 7f8ce4ff9640 10 client.16047 handle_command_reply: tid=0 2024-04-21T23:19:38.903 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.900+0000 7f8d065fe740 1 client.16047 shutdown 2024-04-21T23:19:38.903 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.900+0000 7f8d065fe740 1 -- 172.21.15.40:0/1733982434 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55568a0bfb20 msgr2=0x55568a0c3400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:38.903 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.900+0000 7f8d065fe740 1 --2- 172.21.15.40:0/1733982434 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55568a0bfb20 0x55568a0c3400 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f8cf8004e60 tx=0x7f8cf8005fd0 comp rx=0 tx=0).stop 2024-04-21T23:19:38.903 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.900+0000 7f8d065fe740 1 -- 172.21.15.40:0/1733982434 shutdown_connections 2024-04-21T23:19:38.903 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.900+0000 7f8d065fe740 1 --2- 172.21.15.40:0/1733982434 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55568a0d93f0 0x55568a142770 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:38.903 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.900+0000 7f8d065fe740 1 --2- 172.21.15.40:0/1733982434 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55568a0c9210 0x55568a13e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:38.903 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.900+0000 7f8ce4ff9640 0 client.16047 ms_handle_reset on v2:172.21.15.111:6838/3712235548 2024-04-21T23:19:38.903 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.900+0000 7f8d065fe740 1 --2- 172.21.15.40:0/1733982434 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55568a0c0440 0x55568a0c3940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:38.903 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.900+0000 7f8d065fe740 1 --2- 172.21.15.40:0/1733982434 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55568a0bfb20 0x55568a0c3400 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:38.903 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.900+0000 7f8d065fe740 1 -- 172.21.15.40:0/1733982434 >> 172.21.15.40:0/1733982434 conn(0x55568a0ba440 msgr2=0x55568a0bc0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:38.903 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.901+0000 7f8d065fe740 1 -- 172.21.15.40:0/1733982434 shutdown_connections 2024-04-21T23:19:38.904 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.901+0000 7f8d065fe740 1 -- 172.21.15.40:0/1733982434 wait complete. 2024-04-21T23:19:38.904 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.901+0000 7f8d065fe740 20 client.16047 trim_cache size 0 max 16384 2024-04-21T23:19:38.904 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:{ 2024-04-21T23:19:38.904 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "message": "", 2024-04-21T23:19:38.904 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "return_code": 0 2024-04-21T23:19:38.904 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:} 2024-04-21T23:19:38.905 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.902+0000 7f8d0393a640 1 -- 172.21.15.40:0/2620069225 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f8ce80a2fd0 msgr2=0x7f8ce80a5490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:38.905 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.902+0000 7f8d0393a640 1 --2- 172.21.15.40:0/2620069225 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f8ce80a2fd0 0x7f8ce80a5490 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8cfc078180 tx=0x7f8cec00b410 comp rx=0 tx=0).stop 2024-04-21T23:19:38.905 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.902+0000 7f8d0393a640 1 -- 172.21.15.40:0/2620069225 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8cfc106630 msgr2=0x7f8cfc19c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:38.906 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.902+0000 7f8d0393a640 1 --2- 172.21.15.40:0/2620069225 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8cfc106630 0x7f8cfc19c8e0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f8cf400e940 tx=0x7f8cf400ee00 comp rx=0 tx=0).stop 2024-04-21T23:19:38.906 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.902+0000 7f8d0393a640 1 -- 172.21.15.40:0/2620069225 shutdown_connections 2024-04-21T23:19:38.906 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.902+0000 7f8d0393a640 1 --2- 172.21.15.40:0/2620069225 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f8ce80a2fd0 0x7f8ce80a5490 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:38.906 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.902+0000 7f8d0393a640 1 --2- 172.21.15.40:0/2620069225 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8cfc101030 0x7f8cfc198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:38.906 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.902+0000 7f8d0393a640 1 --2- 172.21.15.40:0/2620069225 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8cfc106630 0x7f8cfc19c8e0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:38.906 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.902+0000 7f8d0393a640 1 --2- 172.21.15.40:0/2620069225 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8cfc100670 0x7f8cfc198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:38.906 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.902+0000 7f8d0393a640 1 -- 172.21.15.40:0/2620069225 >> 172.21.15.40:0/2620069225 conn(0x7f8cfc0777c0 msgr2=0x7f8cfc0ff540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:38.906 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.902+0000 7f8d0393a640 1 -- 172.21.15.40:0/2620069225 shutdown_connections 2024-04-21T23:19:38.906 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:38.902+0000 7f8d0393a640 1 -- 172.21.15.40:0/2620069225 wait complete. 2024-04-21T23:19:40.931 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:19:40 smithi040 ceph-mon[27984]: pgmap v41: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 511 B/s wr, 9 op/s 2024-04-21T23:19:40.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:19:40 smithi040 ceph-mon[27984]: scrub summary: idle 2024-04-21T23:19:40.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:19:40 smithi174 ceph-mon[31409]: pgmap v41: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 511 B/s wr, 9 op/s 2024-04-21T23:19:40.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:19:40 smithi174 ceph-mon[31409]: scrub summary: idle 2024-04-21T23:19:40.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:19:40 smithi111 ceph-mon[31085]: pgmap v41: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 511 B/s wr, 9 op/s 2024-04-21T23:19:40.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:19:40 smithi111 ceph-mon[31085]: scrub summary: idle 2024-04-21T23:19:42.734 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:19:42 smithi040 ceph-mon[27984]: pgmap v42: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-21T23:19:42.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:19:42 smithi174 ceph-mon[31409]: pgmap v42: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-21T23:19:42.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:19:42 smithi111 ceph-mon[31085]: pgmap v42: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-21T23:19:43.750 INFO:tasks.workunit.client.0.smithi040.stderr:Updating files: 93% (11644/12485) Updating files: 94% (11736/12485) Updating files: 95% (11861/12485) Updating files: 96% (11986/12485) Updating files: 97% (12111/12485) Updating files: 98% (12236/12485) Updating files: 99% (12361/12485) Updating files: 100% (12485/12485) Updating files: 100% (12485/12485), done. 2024-04-21T23:19:44.465 INFO:tasks.workunit.client.0.smithi040.stderr:Note: switching to '430e09df97c8fc7dc2b2ae424f68ed11366c540f'. 2024-04-21T23:19:44.465 INFO:tasks.workunit.client.0.smithi040.stderr: 2024-04-21T23:19:44.465 INFO:tasks.workunit.client.0.smithi040.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-21T23:19:44.465 INFO:tasks.workunit.client.0.smithi040.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-21T23:19:44.465 INFO:tasks.workunit.client.0.smithi040.stderr:state without impacting any branches by switching back to a branch. 2024-04-21T23:19:44.465 INFO:tasks.workunit.client.0.smithi040.stderr: 2024-04-21T23:19:44.466 INFO:tasks.workunit.client.0.smithi040.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-21T23:19:44.466 INFO:tasks.workunit.client.0.smithi040.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-21T23:19:44.466 INFO:tasks.workunit.client.0.smithi040.stderr: 2024-04-21T23:19:44.466 INFO:tasks.workunit.client.0.smithi040.stderr: git switch -c 2024-04-21T23:19:44.466 INFO:tasks.workunit.client.0.smithi040.stderr: 2024-04-21T23:19:44.466 INFO:tasks.workunit.client.0.smithi040.stderr:Or undo this operation with: 2024-04-21T23:19:44.466 INFO:tasks.workunit.client.0.smithi040.stderr: 2024-04-21T23:19:44.466 INFO:tasks.workunit.client.0.smithi040.stderr: git switch - 2024-04-21T23:19:44.466 INFO:tasks.workunit.client.0.smithi040.stderr: 2024-04-21T23:19:44.466 INFO:tasks.workunit.client.0.smithi040.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-21T23:19:44.466 INFO:tasks.workunit.client.0.smithi040.stderr: 2024-04-21T23:19:44.466 INFO:tasks.workunit.client.0.smithi040.stderr:HEAD is now at 430e09df97c Merge pull request #57017 from zdover23/wip-doc-2024-04-21-backport-56904-to-squid 2024-04-21T23:19:44.469 DEBUG:teuthology.orchestra.run.smithi040:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-21T23:19:44.538 INFO:tasks.workunit.client.0.smithi040.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-21T23:19:44.540 INFO:tasks.workunit.client.0.smithi040.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-21T23:19:44.540 INFO:tasks.workunit.client.0.smithi040.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-21T23:19:44.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:19:44 smithi040 ceph-mon[27984]: pgmap v43: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-21T23:19:44.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:19:44 smithi111 ceph-mon[31085]: pgmap v43: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-21T23:19:44.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:19:44 smithi174 ceph-mon[31409]: pgmap v43: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-21T23:19:45.164 INFO:tasks.workunit.client.0.smithi040.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-21T23:19:45.225 INFO:tasks.workunit.client.0.smithi040.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-21T23:19:45.265 INFO:tasks.workunit.client.0.smithi040.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-21T23:19:45.267 INFO:tasks.workunit.client.0.smithi040.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-21T23:19:45.267 INFO:tasks.workunit.client.0.smithi040.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-21T23:19:45.307 INFO:tasks.workunit.client.0.smithi040.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-21T23:19:45.310 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-21T23:19:45.310 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-21T23:19:45.366 INFO:tasks.workunit:Running workunits matching suites/fsx.sh on client.0... 2024-04-21T23:19:45.367 INFO:tasks.workunit:Running workunit suites/fsx.sh... 2024-04-21T23:19:45.368 DEBUG:teuthology.orchestra.run.smithi040:workunit test suites/fsx.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=430e09df97c8fc7dc2b2ae424f68ed11366c540f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh 2024-04-21T23:19:45.426 INFO:tasks.workunit.client.0.smithi040.stderr:+ set -e 2024-04-21T23:19:45.426 INFO:tasks.workunit.client.0.smithi040.stderr:+ git clone https://git.ceph.com/xfstests-dev.git 2024-04-21T23:19:45.429 INFO:tasks.workunit.client.0.smithi040.stderr:Cloning into 'xfstests-dev'... 2024-04-21T23:19:46.931 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:19:46 smithi040 ceph-mon[27984]: pgmap v44: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-21T23:19:46.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:19:46 smithi174 ceph-mon[31409]: pgmap v44: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-21T23:19:46.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:19:46 smithi111 ceph-mon[31085]: pgmap v44: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-21T23:19:48.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:19:48 smithi040 ceph-mon[27984]: pgmap v45: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-21T23:19:48.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:19:48 smithi174 ceph-mon[31409]: pgmap v45: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-21T23:19:48.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:19:48 smithi111 ceph-mon[31085]: pgmap v45: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-21T23:19:50.931 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:19:50 smithi040 ceph-mon[27984]: pgmap v46: 129 pgs: 129 active+clean; 14 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 1001 KiB/s rd, 1.1 MiB/s wr, 12 op/s 2024-04-21T23:19:50.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:19:50 smithi111 ceph-mon[31085]: pgmap v46: 129 pgs: 129 active+clean; 14 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 1001 KiB/s rd, 1.1 MiB/s wr, 12 op/s 2024-04-21T23:19:50.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:19:50 smithi174 ceph-mon[31409]: pgmap v46: 129 pgs: 129 active+clean; 14 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 1001 KiB/s rd, 1.1 MiB/s wr, 12 op/s 2024-04-21T23:19:51.931 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:19:51 smithi040 ceph-mon[27984]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-21T23:19:51.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:19:51 smithi174 ceph-mon[31409]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-21T23:19:51.943 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:19:51 smithi111 ceph-mon[31085]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-21T23:19:52.000 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.001+0000 7fb5fae40640 1 -- 172.21.15.40:0/988054401 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5f4103310 msgr2=0x7fb5f410b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:52.001 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.001+0000 7fb5fae40640 1 --2- 172.21.15.40:0/988054401 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5f4103310 0x7fb5f410b7f0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fb5e8009900 tx=0x7fb5e802bcd0 comp rx=0 tx=0).stop 2024-04-21T23:19:52.001 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.001+0000 7fb5fae40640 1 -- 172.21.15.40:0/988054401 shutdown_connections 2024-04-21T23:19:52.001 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.001+0000 7fb5fae40640 1 --2- 172.21.15.40:0/988054401 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5f410bdc0 0x7fb5f410e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:52.002 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.001+0000 7fb5fae40640 1 --2- 172.21.15.40:0/988054401 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5f4103310 0x7fb5f410b7f0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:52.003 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.001+0000 7fb5fae40640 1 --2- 172.21.15.40:0/988054401 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb5f41029f0 0x7fb5f4102dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:52.003 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.001+0000 7fb5fae40640 1 -- 172.21.15.40:0/988054401 >> 172.21.15.40:0/988054401 conn(0x7fb5f40769c0 msgr2=0x7fb5f4076dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:52.003 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.001+0000 7fb5fae40640 1 -- 172.21.15.40:0/988054401 shutdown_connections 2024-04-21T23:19:52.003 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.001+0000 7fb5fae40640 1 -- 172.21.15.40:0/988054401 wait complete. 2024-04-21T23:19:52.003 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.002+0000 7fb5fae40640 1 Processor -- start 2024-04-21T23:19:52.003 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.002+0000 7fb5fae40640 1 -- start start 2024-04-21T23:19:52.003 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.002+0000 7fb5fae40640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5f41029f0 0x7fb5f4100640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:52.003 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.002+0000 7fb5fae40640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb5f4103310 0x7fb5f4100b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:52.003 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.002+0000 7fb5fae40640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5f410bdc0 0x7fb5f41010c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:52.003 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.002+0000 7fb5fae40640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb5f4111bd0 con 0x7fb5f41029f0 2024-04-21T23:19:52.003 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.002+0000 7fb5fae40640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb5f4111a50 con 0x7fb5f410bdc0 2024-04-21T23:19:52.003 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.002+0000 7fb5fae40640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb5f4111d50 con 0x7fb5f4103310 2024-04-21T23:19:52.003 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.002+0000 7fb5f8bdc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5f41029f0 0x7fb5f4100640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:52.003 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.002+0000 7fb5f8bdc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5f41029f0 0x7fb5f4100640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:53982/0 (socket says 172.21.15.40:53982) 2024-04-21T23:19:52.004 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.002+0000 7fb5f8bdc640 1 -- 172.21.15.40:0/2364247243 learned_addr learned my addr 172.21.15.40:0/2364247243 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:52.004 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.003+0000 7fb5f93dd640 1 --2- 172.21.15.40:0/2364247243 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5f410bdc0 0x7fb5f41010c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:52.004 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.003+0000 7fb5f3fff640 1 --2- 172.21.15.40:0/2364247243 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb5f4103310 0x7fb5f4100b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:52.004 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.003+0000 7fb5f8bdc640 1 -- 172.21.15.40:0/2364247243 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb5f4103310 msgr2=0x7fb5f4100b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:52.004 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.003+0000 7fb5f8bdc640 1 --2- 172.21.15.40:0/2364247243 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb5f4103310 0x7fb5f4100b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:52.004 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.003+0000 7fb5f8bdc640 1 -- 172.21.15.40:0/2364247243 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5f410bdc0 msgr2=0x7fb5f41010c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:52.004 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.003+0000 7fb5f8bdc640 1 --2- 172.21.15.40:0/2364247243 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5f410bdc0 0x7fb5f41010c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:52.004 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.003+0000 7fb5f8bdc640 1 -- 172.21.15.40:0/2364247243 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5f41a64f0 con 0x7fb5f41029f0 2024-04-21T23:19:52.004 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.003+0000 7fb5f3fff640 1 --2- 172.21.15.40:0/2364247243 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb5f4103310 0x7fb5f4100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:19:52.007 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.003+0000 7fb5f8bdc640 1 --2- 172.21.15.40:0/2364247243 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5f41029f0 0x7fb5f4100640 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fb5e400b4a0 tx=0x7fb5e400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:52.007 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.003+0000 7fb5f1ffb640 1 -- 172.21.15.40:0/2364247243 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5e4014070 con 0x7fb5f41029f0 2024-04-21T23:19:52.007 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.003+0000 7fb5f1ffb640 1 -- 172.21.15.40:0/2364247243 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5e4010af0 con 0x7fb5f41029f0 2024-04-21T23:19:52.007 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.003+0000 7fb5fae40640 1 -- 172.21.15.40:0/2364247243 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5f41a67e0 con 0x7fb5f41029f0 2024-04-21T23:19:52.007 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.003+0000 7fb5f1ffb640 1 -- 172.21.15.40:0/2364247243 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5e4017d10 con 0x7fb5f41029f0 2024-04-21T23:19:52.008 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.003+0000 7fb5fae40640 1 -- 172.21.15.40:0/2364247243 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5f41a6ce0 con 0x7fb5f41029f0 2024-04-21T23:19:52.008 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.005+0000 7fb5f1ffb640 1 -- 172.21.15.40:0/2364247243 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 25) ==== 138117+0+0 (secure 0 0 0) 0x7fb5e4017070 con 0x7fb5f41029f0 2024-04-21T23:19:52.008 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.006+0000 7fb5fda2f740 1 Processor -- start 2024-04-21T23:19:52.008 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.006+0000 7fb5fda2f740 1 -- start start 2024-04-21T23:19:52.008 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.006+0000 7fb5fda2f740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5649cf097b20 0x5649cf097f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:52.008 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.006+0000 7fb5fda2f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5649cf098440 0x5649cf0a0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:52.008 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.006+0000 7fb5fda2f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5649cf0a1210 0x5649cf0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:52.008 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.006+0000 7fb5fda2f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5649ceffbb70 con 0x5649cf0a1210 2024-04-21T23:19:52.008 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.006+0000 7fb5fda2f740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5649cefa4470 con 0x5649cf097b20 2024-04-21T23:19:52.008 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.006+0000 7fb5fda2f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5649ceff7b30 con 0x5649cf098440 2024-04-21T23:19:52.008 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.006+0000 7fb5f1ffb640 1 --2- 172.21.15.40:0/2364247243 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fb5dc0a2f50 0x7fb5dc0a5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:52.008 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.006+0000 7fb5f93dd640 1 --2- 172.21.15.40:0/2364247243 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fb5dc0a2f50 0x7fb5dc0a5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:52.008 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.006+0000 7fb5f3fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5649cf0a1210 0x5649cf0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:52.008 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.006+0000 7fb5f3fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5649cf0a1210 0x5649cf0a3600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:53996/0 (socket says 172.21.15.40:53996) 2024-04-21T23:19:52.009 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.006+0000 7fb5f3fff640 1 -- 172.21.15.40:0/2987353519 learned_addr learned my addr 172.21.15.40:0/2987353519 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:52.009 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.006+0000 7fb5f1ffb640 1 -- 172.21.15.40:0/2364247243 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7fb5e40ceec0 con 0x7fb5f41029f0 2024-04-21T23:19:52.009 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.006+0000 7fb5f93dd640 1 --2- 172.21.15.40:0/2987353519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5649cf097b20 0x5649cf097f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:52.009 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.006+0000 7fb5f8bdc640 1 --2- 172.21.15.40:0/2987353519 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5649cf098440 0x5649cf0a0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:52.009 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.007+0000 7fb5f3fff640 1 -- 172.21.15.40:0/2987353519 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5649cf098440 msgr2=0x5649cf0a0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:52.009 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.007+0000 7fb5f3fff640 1 --2- 172.21.15.40:0/2987353519 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5649cf098440 0x5649cf0a0cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:52.009 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.007+0000 7fb5f3fff640 1 -- 172.21.15.40:0/2987353519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5649cf097b20 msgr2=0x5649cf097f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:52.009 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.007+0000 7fb5f3fff640 1 --2- 172.21.15.40:0/2987353519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5649cf097b20 0x5649cf097f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:52.009 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.007+0000 7fb5f3fff640 1 -- 172.21.15.40:0/2987353519 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5649ced10210 con 0x5649cf0a1210 2024-04-21T23:19:52.009 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.007+0000 7fb5f93dd640 1 --2- 172.21.15.40:0/2364247243 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fb5dc0a2f50 0x7fb5dc0a5410 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec005c40 tx=0x7fb5ec00ac90 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:52.009 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.007+0000 7fb5f3fff640 1 --2- 172.21.15.40:0/2987353519 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5649cf0a1210 0x5649cf0a3600 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fb5e8036040 tx=0x7fb5e802c6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e451e287e69a8e00 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:52.009 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.007+0000 7fb5daffd640 1 -- 172.21.15.40:0/2987353519 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5e8041070 con 0x5649cf0a1210 2024-04-21T23:19:52.009 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.007+0000 7fb5daffd640 1 -- 172.21.15.40:0/2987353519 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5e802cc60 con 0x5649cf0a1210 2024-04-21T23:19:52.009 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.007+0000 7fb5daffd640 1 -- 172.21.15.40:0/2987353519 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5e8004900 con 0x5649cf0a1210 2024-04-21T23:19:52.010 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.008+0000 7fb5fda2f740 1 -- 172.21.15.40:0/2987353519 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5649cf0a1210 msgr2=0x5649cf0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:52.010 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.008+0000 7fb5fda2f740 1 --2- 172.21.15.40:0/2987353519 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5649cf0a1210 0x5649cf0a3600 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fb5e8036040 tx=0x7fb5e802c6f0 comp rx=0 tx=0).stop 2024-04-21T23:19:52.010 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.008+0000 7fb5fda2f740 1 -- 172.21.15.40:0/2987353519 shutdown_connections 2024-04-21T23:19:52.010 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.008+0000 7fb5fda2f740 1 --2- 172.21.15.40:0/2987353519 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5649cf0a1210 0x5649cf0a3600 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:52.010 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.008+0000 7fb5fda2f740 1 --2- 172.21.15.40:0/2987353519 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5649cf098440 0x5649cf0a0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:52.010 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.008+0000 7fb5fda2f740 1 --2- 172.21.15.40:0/2987353519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5649cf097b20 0x5649cf097f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:52.010 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.008+0000 7fb5fda2f740 1 -- 172.21.15.40:0/2987353519 >> 172.21.15.40:0/2987353519 conn(0x5649cf092440 msgr2=0x5649cf092820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:52.010 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.008+0000 7fb5fda2f740 1 -- 172.21.15.40:0/2987353519 shutdown_connections 2024-04-21T23:19:52.010 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.008+0000 7fb5fda2f740 1 -- 172.21.15.40:0/2987353519 wait complete. 2024-04-21T23:19:52.010 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.008+0000 7fb5fda2f740 1 -- start start 2024-04-21T23:19:52.010 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.008+0000 7fb5fda2f740 1 Processor -- start 2024-04-21T23:19:52.015 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.009+0000 7fb5fda2f740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5649cf097b20 0x5649cf09e200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:52.015 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.009+0000 7fb5fda2f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5649cf098440 0x5649cf09e740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:52.015 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.009+0000 7fb5fda2f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5649cf0a1210 0x5649cf09ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:52.015 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.009+0000 7fb5fda2f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5649cf0a3de0 con 0x5649cf098440 2024-04-21T23:19:52.015 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.009+0000 7fb5fda2f740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5649cefb7a10 con 0x5649cf097b20 2024-04-21T23:19:52.015 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.009+0000 7fb5fda2f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5649cf0a3f60 con 0x5649cf0a1210 2024-04-21T23:19:52.015 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.009+0000 7fb5fda2f740 10 client.0 _ll_register_callbacks cb 0x5649ceff2b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:19:52.016 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.009+0000 7fb5f3fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5649cf098440 0x5649cf09e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:52.016 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.009+0000 7fb5f3fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5649cf098440 0x5649cf09e740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54002/0 (socket says 172.21.15.40:54002) 2024-04-21T23:19:52.016 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.009+0000 7fb5f3fff640 1 -- 172.21.15.40:0/3556204586 learned_addr learned my addr 172.21.15.40:0/3556204586 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:52.016 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.009+0000 7fb5f8bdc640 1 --2- 172.21.15.40:0/3556204586 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5649cf097b20 0x5649cf09e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:52.016 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.009+0000 7fb5f93dd640 1 --2- 172.21.15.40:0/3556204586 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5649cf0a1210 0x5649cf09ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:52.016 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.009+0000 7fb5f3fff640 1 -- 172.21.15.40:0/3556204586 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5649cf0a1210 msgr2=0x5649cf09ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:52.016 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.009+0000 7fb5f3fff640 1 --2- 172.21.15.40:0/3556204586 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5649cf0a1210 0x5649cf09ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:52.016 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.009+0000 7fb5f3fff640 1 -- 172.21.15.40:0/3556204586 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5649cf097b20 msgr2=0x5649cf09e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:52.016 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.009+0000 7fb5f3fff640 1 --2- 172.21.15.40:0/3556204586 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5649cf097b20 0x5649cf09e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:52.016 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.009+0000 7fb5f3fff640 1 -- 172.21.15.40:0/3556204586 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5649ceddc620 con 0x5649cf098440 2024-04-21T23:19:52.016 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.009+0000 7fb5f8bdc640 1 --2- 172.21.15.40:0/3556204586 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5649cf097b20 0x5649cf09e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:19:52.016 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.009+0000 7fb5f93dd640 1 --2- 172.21.15.40:0/3556204586 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5649cf0a1210 0x5649cf09ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:19:52.016 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.009+0000 7fb5f3fff640 1 --2- 172.21.15.40:0/3556204586 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5649cf098440 0x5649cf09e740 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fb5e802bd10 tx=0x7fb5e8005500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83133cd8bbb6bf5 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:52.016 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.009+0000 7fb5d3fff640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-21T23:19:52.017 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.009+0000 7fb5d3fff640 1 -- 172.21.15.40:0/3556204586 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5e804a070 con 0x5649cf098440 2024-04-21T23:19:52.017 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.010+0000 7fb5d3fff640 1 -- 172.21.15.40:0/3556204586 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7fb5e804f460 con 0x5649cf098440 2024-04-21T23:19:52.017 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.010+0000 7fb5fda2f740 1 -- 172.21.15.40:0/3556204586 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb5e802cc60 con 0x5649cf098440 2024-04-21T23:19:52.017 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.010+0000 7fb5d3fff640 1 -- 172.21.15.40:0/3556204586 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fb5e8058050 con 0x5649cf098440 2024-04-21T23:19:52.017 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.010+0000 7fb5fda2f740 10 client.16059 fetch_fsmap learned FSMap version 26 2024-04-21T23:19:52.017 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.010+0000 7fb5fda2f740 1 -- 172.21.15.40:0/3556204586 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5649cf0afff0 con 0x5649cf098440 2024-04-21T23:19:52.017 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.010+0000 7fb5d3fff640 1 -- 172.21.15.40:0/3556204586 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5e802cc60 con 0x5649cf098440 2024-04-21T23:19:52.017 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.010+0000 7fb5d3fff640 1 -- 172.21.15.40:0/3556204586 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fb5e8046570 con 0x5649cf098440 2024-04-21T23:19:52.017 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.010+0000 7fb5fda2f740 10 client.16059 fetch_fsmap finished waiting for FSMap version 26 2024-04-21T23:19:52.017 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.010+0000 7fb5fda2f740 10 client.16059 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:19:52.018 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.010+0000 7fb5fda2f740 20 client.16059 populate_metadata read hostname 'smithi040' 2024-04-21T23:19:52.018 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.010+0000 7fb5fda2f740 1 --2- 172.21.15.40:0/3556204586 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x5649cf0b1340 0x5649cf0b3730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:52.018 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.010+0000 7fb5fda2f740 4 client.16059 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:19:52.018 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.010+0000 7fb5fda2f740 1 -- 172.21.15.40:0/3556204586 --> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5649cf0090b0 con 0x5649cf0b1340 2024-04-21T23:19:52.019 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.010+0000 7fb5f8bdc640 1 --2- 172.21.15.40:0/3556204586 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x5649cf0b1340 0x5649cf0b3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:52.019 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.011+0000 7fb5f8bdc640 1 --2- 172.21.15.40:0/3556204586 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x5649cf0b1340 0x5649cf0b3730 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed18b0ce2ba04ae2 server_cookie=c0b0f601410cf56e in_seq=0 out_seq=0 2024-04-21T23:19:52.019 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.011+0000 7fb5d3fff640 10 client.16059 ms_handle_connect on v2:172.21.15.111:6838/3712235548 2024-04-21T23:19:52.019 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.011+0000 7fb5d3fff640 1 -- 172.21.15.40:0/3556204586 <== mds.0 v2:172.21.15.111:6838/3712235548 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5649cf0090b0 con 0x5649cf0b1340 2024-04-21T23:19:52.019 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.011+0000 7fb5d3fff640 10 client.16059 handle_command_reply: tid=0 2024-04-21T23:19:52.019 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.011+0000 7fb5fda2f740 1 client.16059 shutdown 2024-04-21T23:19:52.020 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.012+0000 7fb5fda2f740 1 -- 172.21.15.40:0/3556204586 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5649cf098440 msgr2=0x5649cf09e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:52.020 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.012+0000 7fb5fda2f740 1 --2- 172.21.15.40:0/3556204586 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5649cf098440 0x5649cf09e740 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fb5e802bd10 tx=0x7fb5e8005500 comp rx=0 tx=0).stop 2024-04-21T23:19:52.020 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.012+0000 7fb5fda2f740 1 -- 172.21.15.40:0/3556204586 shutdown_connections 2024-04-21T23:19:52.020 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.012+0000 7fb5fda2f740 1 --2- 172.21.15.40:0/3556204586 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x5649cf0b1340 0x5649cf0b3730 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:52.020 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.012+0000 7fb5fda2f740 1 --2- 172.21.15.40:0/3556204586 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5649cf0a1210 0x5649cf09ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:52.020 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.012+0000 7fb5fda2f740 1 --2- 172.21.15.40:0/3556204586 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5649cf098440 0x5649cf09e740 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:52.021 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.012+0000 7fb5fda2f740 1 --2- 172.21.15.40:0/3556204586 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5649cf097b20 0x5649cf09e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:52.021 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.012+0000 7fb5d3fff640 0 client.16059 ms_handle_reset on v2:172.21.15.111:6838/3712235548 2024-04-21T23:19:52.021 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.012+0000 7fb5fda2f740 1 -- 172.21.15.40:0/3556204586 >> 172.21.15.40:0/3556204586 conn(0x5649cf092440 msgr2=0x5649cf094190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:52.021 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.012+0000 7fb5fda2f740 1 -- 172.21.15.40:0/3556204586 shutdown_connections 2024-04-21T23:19:52.021 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.012+0000 7fb5fda2f740 1 -- 172.21.15.40:0/3556204586 wait complete. 2024-04-21T23:19:52.021 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.012+0000 7fb5fda2f740 20 client.16059 trim_cache size 0 max 16384 2024-04-21T23:19:52.035 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.027+0000 7fb5fda2f740 1 Processor -- start 2024-04-21T23:19:52.035 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.027+0000 7fb5fda2f740 1 -- start start 2024-04-21T23:19:52.036 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.027+0000 7fb5fda2f740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5649cf097b20 0x5649cf0a0930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:52.036 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.027+0000 7fb5fda2f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5649cf098440 0x5649cf0acca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:52.036 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.027+0000 7fb5fda2f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5649cf0a1210 0x5649cf0b8d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:52.036 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.027+0000 7fb5fda2f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb5f41b36f0 con 0x5649cf0a1210 2024-04-21T23:19:52.036 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.027+0000 7fb5fda2f740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb5f41b0fc0 con 0x5649cf097b20 2024-04-21T23:19:52.036 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.027+0000 7fb5fda2f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb5f41aee10 con 0x5649cf098440 2024-04-21T23:19:52.037 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.027+0000 7fb5f93dd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5649cf0a1210 0x5649cf0b8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:52.037 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.027+0000 7fb5f93dd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5649cf0a1210 0x5649cf0b8d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54004/0 (socket says 172.21.15.40:54004) 2024-04-21T23:19:52.037 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.027+0000 7fb5f93dd640 1 -- 172.21.15.40:0/3713361691 learned_addr learned my addr 172.21.15.40:0/3713361691 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:52.037 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.027+0000 7fb5f3fff640 1 --2- 172.21.15.40:0/3713361691 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5649cf098440 0x5649cf0acca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:52.037 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.027+0000 7fb5f8bdc640 1 --2- 172.21.15.40:0/3713361691 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5649cf097b20 0x5649cf0a0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:52.037 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.027+0000 7fb5f93dd640 1 -- 172.21.15.40:0/3713361691 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5649cf098440 msgr2=0x5649cf0acca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:52.038 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.027+0000 7fb5f93dd640 1 --2- 172.21.15.40:0/3713361691 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5649cf098440 0x5649cf0acca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:52.038 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.027+0000 7fb5f93dd640 1 -- 172.21.15.40:0/3713361691 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5649cf097b20 msgr2=0x5649cf0a0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:52.038 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.027+0000 7fb5f93dd640 1 --2- 172.21.15.40:0/3713361691 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5649cf097b20 0x5649cf0a0930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:52.038 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.027+0000 7fb5f93dd640 1 -- 172.21.15.40:0/3713361691 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5649cf0a3600 con 0x5649cf0a1210 2024-04-21T23:19:52.038 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.027+0000 7fb5f8bdc640 1 --2- 172.21.15.40:0/3713361691 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5649cf097b20 0x5649cf0a0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:19:52.038 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.027+0000 7fb5f93dd640 1 --2- 172.21.15.40:0/3713361691 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5649cf0a1210 0x5649cf0b8d00 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec00a7b0 tx=0x7fb5ec01d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f87c994e167b0b91 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:52.038 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.028+0000 7fb5da7fc640 1 -- 172.21.15.40:0/3713361691 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5ec003b70 con 0x5649cf0a1210 2024-04-21T23:19:52.038 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.028+0000 7fb5da7fc640 1 -- 172.21.15.40:0/3713361691 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5ec003d10 con 0x5649cf0a1210 2024-04-21T23:19:52.038 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.028+0000 7fb5da7fc640 1 -- 172.21.15.40:0/3713361691 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5ec011c30 con 0x5649cf0a1210 2024-04-21T23:19:52.038 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.028+0000 7fb5fda2f740 1 -- 172.21.15.40:0/3713361691 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5649cf0a1210 msgr2=0x5649cf0b8d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:52.039 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.028+0000 7fb5fda2f740 1 --2- 172.21.15.40:0/3713361691 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5649cf0a1210 0x5649cf0b8d00 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec00a7b0 tx=0x7fb5ec01d000 comp rx=0 tx=0).stop 2024-04-21T23:19:52.039 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.028+0000 7fb5fda2f740 1 -- 172.21.15.40:0/3713361691 shutdown_connections 2024-04-21T23:19:52.039 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.028+0000 7fb5fda2f740 1 --2- 172.21.15.40:0/3713361691 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5649cf0a1210 0x5649cf0b8d00 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:52.039 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.028+0000 7fb5fda2f740 1 --2- 172.21.15.40:0/3713361691 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5649cf098440 0x5649cf0acca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:52.039 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.028+0000 7fb5fda2f740 1 --2- 172.21.15.40:0/3713361691 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5649cf097b20 0x5649cf0a0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:52.039 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.028+0000 7fb5fda2f740 1 -- 172.21.15.40:0/3713361691 >> 172.21.15.40:0/3713361691 conn(0x5649cf092440 msgr2=0x5649cf096e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:52.039 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.028+0000 7fb5fda2f740 1 -- 172.21.15.40:0/3713361691 shutdown_connections 2024-04-21T23:19:52.039 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.028+0000 7fb5fda2f740 1 -- 172.21.15.40:0/3713361691 wait complete. 2024-04-21T23:19:52.039 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.029+0000 7fb5fda2f740 1 -- start start 2024-04-21T23:19:52.039 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.029+0000 7fb5fda2f740 1 Processor -- start 2024-04-21T23:19:52.039 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.029+0000 7fb5fda2f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5649cf097b20 0x5649cf09af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:52.039 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.029+0000 7fb5fda2f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5649cf098440 0x5649cf09b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:52.039 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.029+0000 7fb5fda2f740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5649cf0a1210 0x5649cf09b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:52.039 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.029+0000 7fb5fda2f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb5f41af480 con 0x5649cf097b20 2024-04-21T23:19:52.040 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.029+0000 7fb5fda2f740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb5f41b08f0 con 0x5649cf0a1210 2024-04-21T23:19:52.040 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.029+0000 7fb5fda2f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb5f41afb50 con 0x5649cf098440 2024-04-21T23:19:52.040 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.029+0000 7fb5fda2f740 10 client.0 _ll_register_callbacks cb 0x7fb5ec0135c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:19:52.040 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.029+0000 7fb5f8bdc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5649cf097b20 0x5649cf09af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:52.040 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.029+0000 7fb5f8bdc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5649cf097b20 0x5649cf09af10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54006/0 (socket says 172.21.15.40:54006) 2024-04-21T23:19:52.040 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.029+0000 7fb5f8bdc640 1 -- 172.21.15.40:0/1954007919 learned_addr learned my addr 172.21.15.40:0/1954007919 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:19:52.040 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.029+0000 7fb5f93dd640 1 --2- 172.21.15.40:0/1954007919 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5649cf0a1210 0x5649cf09b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:52.040 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.029+0000 7fb5f3fff640 1 --2- 172.21.15.40:0/1954007919 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5649cf098440 0x5649cf09b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:52.040 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.030+0000 7fb5f8bdc640 1 -- 172.21.15.40:0/1954007919 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5649cf098440 msgr2=0x5649cf09b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:52.040 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.030+0000 7fb5f8bdc640 1 --2- 172.21.15.40:0/1954007919 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5649cf098440 0x5649cf09b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:52.040 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.030+0000 7fb5f8bdc640 1 -- 172.21.15.40:0/1954007919 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5649cf0a1210 msgr2=0x5649cf09b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:52.040 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.030+0000 7fb5f8bdc640 1 --2- 172.21.15.40:0/1954007919 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5649cf0a1210 0x5649cf09b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:52.040 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.030+0000 7fb5f8bdc640 1 -- 172.21.15.40:0/1954007919 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5649cf116570 con 0x5649cf097b20 2024-04-21T23:19:52.041 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.030+0000 7fb5f3fff640 1 --2- 172.21.15.40:0/1954007919 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5649cf098440 0x5649cf09b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:19:52.041 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.030+0000 7fb5f8bdc640 1 --2- 172.21.15.40:0/1954007919 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5649cf097b20 0x5649cf09af10 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fb5e4097360 tx=0x7fb5e4017730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8cb4198cd54b9e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:52.041 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.030+0000 7fb5d3fff640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-21T23:19:52.041 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.030+0000 7fb5d3fff640 1 -- 172.21.15.40:0/1954007919 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5e401e040 con 0x5649cf097b20 2024-04-21T23:19:52.041 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.030+0000 7fb5d3fff640 1 -- 172.21.15.40:0/1954007919 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7fb5e4097570 con 0x5649cf097b20 2024-04-21T23:19:52.041 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.030+0000 7fb5fda2f740 1 -- 172.21.15.40:0/1954007919 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb5ec003d10 con 0x5649cf097b20 2024-04-21T23:19:52.041 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.030+0000 7fb5d3fff640 1 -- 172.21.15.40:0/1954007919 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fb5e409f2b0 con 0x5649cf097b20 2024-04-21T23:19:52.041 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.030+0000 7fb5fda2f740 10 client.16065 fetch_fsmap learned FSMap version 26 2024-04-21T23:19:52.041 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.030+0000 7fb5fda2f740 1 -- 172.21.15.40:0/1954007919 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5649cf117830 con 0x5649cf097b20 2024-04-21T23:19:52.041 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.030+0000 7fb5d3fff640 1 -- 172.21.15.40:0/1954007919 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5ec003d10 con 0x5649cf097b20 2024-04-21T23:19:52.041 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.030+0000 7fb5d3fff640 1 -- 172.21.15.40:0/1954007919 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fb5e40a36e0 con 0x5649cf097b20 2024-04-21T23:19:52.041 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.030+0000 7fb5fda2f740 10 client.16065 fetch_fsmap finished waiting for FSMap version 26 2024-04-21T23:19:52.041 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.030+0000 7fb5fda2f740 10 client.16065 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:19:52.041 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.030+0000 7fb5fda2f740 20 client.16065 populate_metadata read hostname 'smithi040' 2024-04-21T23:19:52.041 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.030+0000 7fb5fda2f740 1 --2- 172.21.15.40:0/1954007919 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x5649cf118010 0x5649cf11a430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:52.042 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.030+0000 7fb5fda2f740 4 client.16065 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T23:19:52.042 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.030+0000 7fb5fda2f740 1 -- 172.21.15.40:0/1954007919 --> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5649cf003400 con 0x5649cf118010 2024-04-21T23:19:52.042 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.031+0000 7fb5f3fff640 1 --2- 172.21.15.40:0/1954007919 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x5649cf118010 0x5649cf11a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:52.042 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.031+0000 7fb5d3fff640 10 client.16065 ms_handle_connect on v2:172.21.15.111:6838/3712235548 2024-04-21T23:19:52.042 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.031+0000 7fb5f3fff640 1 --2- 172.21.15.40:0/1954007919 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x5649cf118010 0x5649cf11a430 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ef5f9e031f82bbf server_cookie=ab2fccca719f8ed2 in_seq=0 out_seq=0 2024-04-21T23:19:52.750 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.750+0000 7fb5d3fff640 1 -- 172.21.15.40:0/1954007919 <== mds.0 v2:172.21.15.111:6838/3712235548 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5649cf003400 con 0x5649cf118010 2024-04-21T23:19:52.750 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.750+0000 7fb5d3fff640 10 client.16065 handle_command_reply: tid=0 2024-04-21T23:19:52.750 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.750+0000 7fb5fda2f740 1 client.16065 shutdown 2024-04-21T23:19:52.750 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.750+0000 7fb5fda2f740 1 -- 172.21.15.40:0/1954007919 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5649cf097b20 msgr2=0x5649cf09af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:52.751 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.750+0000 7fb5fda2f740 1 --2- 172.21.15.40:0/1954007919 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5649cf097b20 0x5649cf09af10 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fb5e4097360 tx=0x7fb5e4017730 comp rx=0 tx=0).stop 2024-04-21T23:19:52.751 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.750+0000 7fb5fda2f740 1 -- 172.21.15.40:0/1954007919 shutdown_connections 2024-04-21T23:19:52.751 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.750+0000 7fb5fda2f740 1 --2- 172.21.15.40:0/1954007919 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x5649cf118010 0x5649cf11a430 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:52.751 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.750+0000 7fb5fda2f740 1 --2- 172.21.15.40:0/1954007919 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5649cf0a1210 0x5649cf09b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:52.751 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.750+0000 7fb5fda2f740 1 --2- 172.21.15.40:0/1954007919 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5649cf098440 0x5649cf09b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:52.751 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.750+0000 7fb5d3fff640 0 client.16065 ms_handle_reset on v2:172.21.15.111:6838/3712235548 2024-04-21T23:19:52.751 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.750+0000 7fb5fda2f740 1 --2- 172.21.15.40:0/1954007919 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5649cf097b20 0x5649cf09af10 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:52.751 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.750+0000 7fb5fda2f740 1 -- 172.21.15.40:0/1954007919 >> 172.21.15.40:0/1954007919 conn(0x5649cf092440 msgr2=0x5649cf09e120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:52.751 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.750+0000 7fb5fda2f740 1 -- 172.21.15.40:0/1954007919 shutdown_connections 2024-04-21T23:19:52.751 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.750+0000 7fb5fda2f740 1 -- 172.21.15.40:0/1954007919 wait complete. 2024-04-21T23:19:52.751 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.750+0000 7fb5fda2f740 20 client.16065 trim_cache size 0 max 16384 2024-04-21T23:19:52.751 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:{ 2024-04-21T23:19:52.752 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "message": "", 2024-04-21T23:19:52.752 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "return_code": 0 2024-04-21T23:19:52.752 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:} 2024-04-21T23:19:52.752 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.751+0000 7fb5fae40640 1 -- 172.21.15.40:0/2364247243 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fb5dc0a2f50 msgr2=0x7fb5dc0a5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:52.752 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.751+0000 7fb5fae40640 1 --2- 172.21.15.40:0/2364247243 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fb5dc0a2f50 0x7fb5dc0a5410 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec005c40 tx=0x7fb5ec00ac90 comp rx=0 tx=0).stop 2024-04-21T23:19:52.752 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.751+0000 7fb5fae40640 1 -- 172.21.15.40:0/2364247243 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5f41029f0 msgr2=0x7fb5f4100640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:52.753 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.751+0000 7fb5fae40640 1 --2- 172.21.15.40:0/2364247243 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5f41029f0 0x7fb5f4100640 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fb5e400b4a0 tx=0x7fb5e400b960 comp rx=0 tx=0).stop 2024-04-21T23:19:52.753 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.751+0000 7fb5fae40640 1 -- 172.21.15.40:0/2364247243 shutdown_connections 2024-04-21T23:19:52.753 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.751+0000 7fb5fae40640 1 --2- 172.21.15.40:0/2364247243 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fb5dc0a2f50 0x7fb5dc0a5410 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:52.753 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.751+0000 7fb5fae40640 1 --2- 172.21.15.40:0/2364247243 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5f410bdc0 0x7fb5f41010c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:52.753 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.751+0000 7fb5fae40640 1 --2- 172.21.15.40:0/2364247243 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb5f4103310 0x7fb5f4100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:52.753 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.751+0000 7fb5fae40640 1 --2- 172.21.15.40:0/2364247243 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5f41029f0 0x7fb5f4100640 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:52.753 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.751+0000 7fb5fae40640 1 -- 172.21.15.40:0/2364247243 >> 172.21.15.40:0/2364247243 conn(0x7fb5f40769c0 msgr2=0x7fb5f40fd4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:52.753 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.751+0000 7fb5fae40640 1 -- 172.21.15.40:0/2364247243 shutdown_connections 2024-04-21T23:19:52.753 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:19:52.751+0000 7fb5fae40640 1 -- 172.21.15.40:0/2364247243 wait complete. 2024-04-21T23:19:52.931 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:19:52 smithi040 ceph-mon[27984]: pgmap v47: 129 pgs: 129 active+clean; 14 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 999 KiB/s rd, 1.1 MiB/s wr, 8 op/s 2024-04-21T23:19:52.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:19:52 smithi174 ceph-mon[31409]: pgmap v47: 129 pgs: 129 active+clean; 14 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 999 KiB/s rd, 1.1 MiB/s wr, 8 op/s 2024-04-21T23:19:52.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:19:52 smithi111 ceph-mon[31085]: pgmap v47: 129 pgs: 129 active+clean; 14 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 999 KiB/s rd, 1.1 MiB/s wr, 8 op/s 2024-04-21T23:19:55.181 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:19:54 smithi040 ceph-mon[27984]: pgmap v48: 129 pgs: 129 active+clean; 14 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 999 KiB/s rd, 1.1 MiB/s wr, 8 op/s 2024-04-21T23:19:55.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:19:54 smithi040 ceph-mon[27984]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-21T23:19:55.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:19:54 smithi174 ceph-mon[31409]: pgmap v48: 129 pgs: 129 active+clean; 14 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 999 KiB/s rd, 1.1 MiB/s wr, 8 op/s 2024-04-21T23:19:55.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:19:54 smithi174 ceph-mon[31409]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-21T23:19:55.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:19:54 smithi111 ceph-mon[31085]: pgmap v48: 129 pgs: 129 active+clean; 14 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 999 KiB/s rd, 1.1 MiB/s wr, 8 op/s 2024-04-21T23:19:55.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:19:54 smithi111 ceph-mon[31085]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-21T23:19:57.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:19:56 smithi040 ceph-mon[27984]: pgmap v49: 129 pgs: 129 active+clean; 20 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 281 op/s 2024-04-21T23:19:57.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:19:56 smithi040 ceph-mon[27984]: mgrmap e26: x(active, since 92s), standbys: y, z 2024-04-21T23:19:57.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:19:56 smithi174 ceph-mon[31409]: pgmap v49: 129 pgs: 129 active+clean; 20 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 281 op/s 2024-04-21T23:19:57.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:19:56 smithi174 ceph-mon[31409]: mgrmap e26: x(active, since 92s), standbys: y, z 2024-04-21T23:19:57.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:19:56 smithi111 ceph-mon[31085]: pgmap v49: 129 pgs: 129 active+clean; 20 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 281 op/s 2024-04-21T23:19:57.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:19:56 smithi111 ceph-mon[31085]: mgrmap e26: x(active, since 92s), standbys: y, z 2024-04-21T23:19:59.181 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:19:58 smithi040 ceph-mon[27984]: pgmap v50: 129 pgs: 129 active+clean; 20 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 279 op/s 2024-04-21T23:19:59.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:19:58 smithi174 ceph-mon[31409]: pgmap v50: 129 pgs: 129 active+clean; 20 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 279 op/s 2024-04-21T23:19:59.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:19:58 smithi111 ceph-mon[31085]: pgmap v50: 129 pgs: 129 active+clean; 20 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 279 op/s 2024-04-21T23:20:00.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:19:59 smithi174 ceph-mon[31409]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-21T23:20:00.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:19:59 smithi111 ceph-mon[31085]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-21T23:20:00.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:19:59 smithi040 ceph-mon[27984]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-21T23:20:00.881 INFO:tasks.workunit.client.0.smithi040.stderr:Updating files: 7% (321/4290) Updating files: 8% (344/4290) Updating files: 9% (387/4290) Updating files: 10% (429/4290) Updating files: 11% (472/4290) Updating files: 11% (490/4290) Updating files: 12% (515/4290) Updating files: 13% (558/4290) Updating files: 14% (601/4290) Updating files: 15% (644/4290) Updating files: 16% (687/4290) Updating files: 17% (730/4290) Updating files: 18% (773/4290) Updating files: 19% (816/4290) Updating files: 20% (858/4290) Updating files: 21% (901/4290) Updating files: 22% (944/4290) Updating files: 23% (987/4290) Updating files: 24% (1030/4290) Updating files: 25% (1073/4290) Updating files: 26% (1116/4290) Updating files: 27% (1159/4290) Updating files: 28% (1202/4290) Updating files: 29% (1245/4290) Updating files: 30% (1287/4290) Updating files: 30% (1288/4290) Updating files: 31% (1330/4290) Updating files: 32% (1373/4290) Updating files: 33% (1416/4290) Updating files: 34% (1459/4290) Updating files: 35% (1502/4290) Updating files: 36% (1545/4290) Updating files: 37% (1588/4290) Updating files: 38% (1631/4290) Updating files: 39% (1674/4290) Updating files: 40% (1716/4290) Updating files: 41% (1759/4290) Updating files: 42% (1802/4290) Updating files: 43% (1845/4290) Updating files: 44% (1888/4290) Updating files: 45% (1931/4290) Updating files: 46% (1974/4290) Updating files: 47% (2017/4290) Updating files: 47% (2029/4290) Updating files: 48% (2060/4290) Updating files: 49% (2103/4290) Updating files: 50% (2145/4290) Updating files: 51% (2188/4290) Updating files: 52% (2231/4290) Updating files: 53% (2274/4290) Updating files: 54% (2317/4290) Updating files: 55% (2360/4290) Updating files: 56% (2403/4290) Updating files: 57% (2446/4290) Updating files: 58% (2489/4290) Updating files: 59% (2532/4290) Updating files: 59% (2560/4290) Updating files: 60% (2574/4290) Updating files: 61% (2617/4290) Updating files: 62% (2660/4290) Updating files: 63% (2703/4290) Updating files: 64% (2746/4290) Updating files: 65% (2789/4290) Updating files: 66% (2832/4290) Updating files: 66% (2860/4290) Updating files: 67% (2875/4290) Updating files: 68% (2918/4290) Updating files: 69% (2961/4290) Updating files: 70% (3003/4290) Updating files: 70% (3022/4290) Updating files: 71% (3046/4290) Updating files: 72% (3089/4290) Updating files: 73% (3132/4290) Updating files: 74% (3175/4290) Updating files: 75% (3218/4290) Updating files: 76% (3261/4290) Updating files: 77% (3304/4290) Updating files: 78% (3347/4290) Updating files: 79% (3390/4290) Updating files: 80% (3432/4290) Updating files: 81% (3475/4290) Updating files: 82% (3518/4290) Updating files: 83% (3561/4290) Updating files: 84% (3604/4290) Updating files: 84% (3608/4290) Updating files: 85% (3647/4290) Updating files: 86% (3690/4290) Updating files: 87% (3733/4290) Updating files: 88% (3776/4290) Updating files: 89% (3819/4290) Updating files: 90% (3861/4290) Updating files: 91% (3904/4290) Updating files: 92% (3947/4290) Updating files: 93% (3990/4290) Updating files: 94% (4033/4290) Updating files: 94% (4069/4290) Updating files: 95% (4076/4290) Updating files: 96% (4119/4290) Updating files: 97% (4162/4290) Updating files: 98% (4205/4290) Updating files: 99% (4248/4290) Updating files: 100% (4290/4290) Updating files: 100% (4290/4290), done. 2024-04-21T23:20:00.889 INFO:tasks.workunit.client.0.smithi040.stderr:+ cd xfstests-dev 2024-04-21T23:20:00.919 INFO:tasks.workunit.client.0.smithi040.stderr:+ git checkout 12973fc04fd10d4af086901e10ffa8e48866b735 2024-04-21T23:20:01.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:00 smithi174 ceph-mon[31409]: pgmap v51: 129 pgs: 129 active+clean; 22 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.9 MiB/s wr, 290 op/s 2024-04-21T23:20:01.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:00 smithi174 ceph-mon[31409]: overall HEALTH_OK 2024-04-21T23:20:01.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:00 smithi111 ceph-mon[31085]: pgmap v51: 129 pgs: 129 active+clean; 22 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.9 MiB/s wr, 290 op/s 2024-04-21T23:20:01.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:00 smithi111 ceph-mon[31085]: overall HEALTH_OK 2024-04-21T23:20:01.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:00 smithi040 ceph-mon[27984]: pgmap v51: 129 pgs: 129 active+clean; 22 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.9 MiB/s wr, 290 op/s 2024-04-21T23:20:01.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:00 smithi040 ceph-mon[27984]: overall HEALTH_OK 2024-04-21T23:20:03.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:02 smithi111 ceph-mon[31085]: pgmap v52: 129 pgs: 129 active+clean; 22 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 680 KiB/s rd, 857 KiB/s wr, 284 op/s 2024-04-21T23:20:03.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:02 smithi174 ceph-mon[31409]: pgmap v52: 129 pgs: 129 active+clean; 22 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 680 KiB/s rd, 857 KiB/s wr, 284 op/s 2024-04-21T23:20:03.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:02 smithi040 ceph-mon[27984]: pgmap v52: 129 pgs: 129 active+clean; 22 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 680 KiB/s rd, 857 KiB/s wr, 284 op/s 2024-04-21T23:20:05.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:04 smithi174 ceph-mon[31409]: pgmap v53: 129 pgs: 129 active+clean; 22 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 680 KiB/s rd, 857 KiB/s wr, 284 op/s 2024-04-21T23:20:05.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:04 smithi111 ceph-mon[31085]: pgmap v53: 129 pgs: 129 active+clean; 22 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 680 KiB/s rd, 857 KiB/s wr, 284 op/s 2024-04-21T23:20:05.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:04 smithi040 ceph-mon[27984]: pgmap v53: 129 pgs: 129 active+clean; 22 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 680 KiB/s rd, 857 KiB/s wr, 284 op/s 2024-04-21T23:20:05.843 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.841+0000 7f7ad1f37640 1 -- 172.21.15.40:0/3558852297 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7acc106630 msgr2=0x7f7acc106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:05.843 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.841+0000 7f7ad1f37640 1 --2- 172.21.15.40:0/3558852297 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7acc106630 0x7f7acc106a10 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7abc009900 tx=0x7f7abc02bc30 comp rx=0 tx=0).stop 2024-04-21T23:20:05.843 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.841+0000 7f7ad1f37640 1 -- 172.21.15.40:0/3558852297 shutdown_connections 2024-04-21T23:20:05.843 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.841+0000 7f7ad1f37640 1 --2- 172.21.15.40:0/3558852297 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7acc101030 0x7f7acc10d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:05.843 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.841+0000 7f7ad1f37640 1 --2- 172.21.15.40:0/3558852297 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7acc100670 0x7f7acc100af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.841+0000 7f7ad1f37640 1 --2- 172.21.15.40:0/3558852297 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7acc106630 0x7f7acc106a10 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.841+0000 7f7ad1f37640 1 -- 172.21.15.40:0/3558852297 >> 172.21.15.40:0/3558852297 conn(0x7f7acc0777c0 msgr2=0x7f7acc077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.842+0000 7f7ad1f37640 1 -- 172.21.15.40:0/3558852297 shutdown_connections 2024-04-21T23:20:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.842+0000 7f7ad1f37640 1 -- 172.21.15.40:0/3558852297 wait complete. 2024-04-21T23:20:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.842+0000 7f7ad1f37640 1 Processor -- start 2024-04-21T23:20:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.842+0000 7f7ad1f37640 1 -- start start 2024-04-21T23:20:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.843+0000 7f7ad1f37640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7acc100670 0x7f7acc197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.843+0000 7f7ad1f37640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7acc101030 0x7f7acc1984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.843+0000 7f7ad1f37640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7acc106630 0x7f7acc19c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.843+0000 7f7ad1f37640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7acc110a40 con 0x7f7acc106630 2024-04-21T23:20:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.843+0000 7f7ad1f37640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f7acc1108c0 con 0x7f7acc100670 2024-04-21T23:20:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.843+0000 7f7ad1f37640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7acc110bc0 con 0x7f7acc101030 2024-04-21T23:20:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.843+0000 7f7acbfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7acc106630 0x7f7acc19c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.843+0000 7f7acb7fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7acc100670 0x7f7acc197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.843+0000 7f7acb7fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7acc100670 0x7f7acc197f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.40:53568/0 (socket says 172.21.15.40:53568) 2024-04-21T23:20:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.843+0000 7f7acb7fe640 1 -- 172.21.15.40:0/970566437 learned_addr learned my addr 172.21.15.40:0/970566437 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:20:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.843+0000 7f7acbfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7acc106630 0x7f7acc19c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36994/0 (socket says 172.21.15.40:36994) 2024-04-21T23:20:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.843+0000 7f7acaffd640 1 --2- 172.21.15.40:0/970566437 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7acc101030 0x7f7acc1984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.843+0000 7f7acb7fe640 1 -- 172.21.15.40:0/970566437 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7acc101030 msgr2=0x7f7acc1984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.843+0000 7f7acb7fe640 1 --2- 172.21.15.40:0/970566437 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7acc101030 0x7f7acc1984d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.843+0000 7f7acb7fe640 1 -- 172.21.15.40:0/970566437 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7acc106630 msgr2=0x7f7acc19c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.843+0000 7f7acb7fe640 1 --2- 172.21.15.40:0/970566437 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7acc106630 0x7f7acc19c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.843+0000 7f7acb7fe640 1 -- 172.21.15.40:0/970566437 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7acc19cf80 con 0x7f7acc100670 2024-04-21T23:20:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.843+0000 7f7acaffd640 1 --2- 172.21.15.40:0/970566437 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7acc101030 0x7f7acc1984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:20:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.843+0000 7f7acb7fe640 1 --2- 172.21.15.40:0/970566437 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7acc100670 0x7f7acc197f90 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f7abc005940 tx=0x7f7abc0042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.843+0000 7f7ac8ff9640 1 -- 172.21.15.40:0/970566437 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7abc041070 con 0x7f7acc100670 2024-04-21T23:20:05.846 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.843+0000 7f7ac8ff9640 1 -- 172.21.15.40:0/970566437 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7abc02ccc0 con 0x7f7acc100670 2024-04-21T23:20:05.846 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.843+0000 7f7ac8ff9640 1 -- 172.21.15.40:0/970566437 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7abc003d80 con 0x7f7acc100670 2024-04-21T23:20:05.846 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.843+0000 7f7ad1f37640 1 -- 172.21.15.40:0/970566437 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7acc19d210 con 0x7f7acc100670 2024-04-21T23:20:05.846 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.844+0000 7f7ad1f37640 1 -- 172.21.15.40:0/970566437 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7acc1a4b00 con 0x7f7acc100670 2024-04-21T23:20:05.853 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.846+0000 7f7ac8ff9640 1 -- 172.21.15.40:0/970566437 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f7abc044070 con 0x7f7acc100670 2024-04-21T23:20:05.853 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.846+0000 7f7ad4bfb740 1 Processor -- start 2024-04-21T23:20:05.853 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.846+0000 7f7ad4bfb740 1 -- start start 2024-04-21T23:20:05.853 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.846+0000 7f7ad4bfb740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564602dcab20 0x564602dcaf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:05.853 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.846+0000 7f7ad4bfb740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564602dcb440 0x564602dd3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:05.853 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.846+0000 7f7ad4bfb740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564602dd4210 0x564602dd6600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:05.853 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.846+0000 7f7ad4bfb740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x564602d2eb70 con 0x564602dd4210 2024-04-21T23:20:05.853 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.846+0000 7f7ad4bfb740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x564602cd7470 con 0x564602dcb440 2024-04-21T23:20:05.853 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.846+0000 7f7ad4bfb740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x564602d2ab30 con 0x564602dcab20 2024-04-21T23:20:05.853 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.846+0000 7f7acaffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564602dd4210 0x564602dd6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:05.854 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.846+0000 7f7acaffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564602dd4210 0x564602dd6600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37006/0 (socket says 172.21.15.40:37006) 2024-04-21T23:20:05.854 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.846+0000 7f7acaffd640 1 -- 172.21.15.40:0/2515714222 learned_addr learned my addr 172.21.15.40:0/2515714222 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:20:05.854 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.846+0000 7f7ac8ff9640 1 --2- 172.21.15.40:0/970566437 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f7ab80a2da0 0x7f7ab80a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:05.854 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.846+0000 7f7acb7fe640 1 --2- 172.21.15.40:0/2515714222 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564602dcb440 0x564602dd3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:05.854 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.846+0000 7f7acbfff640 1 --2- 172.21.15.40:0/2515714222 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564602dcab20 0x564602dcaf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:05.854 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.846+0000 7f7acbfff640 1 --2- 172.21.15.40:0/970566437 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f7ab80a2da0 0x7f7ab80a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:05.854 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.846+0000 7f7acaffd640 1 -- 172.21.15.40:0/2515714222 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564602dcab20 msgr2=0x564602dcaf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:05.854 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.846+0000 7f7acaffd640 1 --2- 172.21.15.40:0/2515714222 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564602dcab20 0x564602dcaf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:05.854 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.846+0000 7f7acaffd640 1 -- 172.21.15.40:0/2515714222 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564602dcb440 msgr2=0x564602dd3cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:05.854 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.846+0000 7f7acaffd640 1 --2- 172.21.15.40:0/2515714222 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564602dcb440 0x564602dd3cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:05.854 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.846+0000 7f7acaffd640 1 -- 172.21.15.40:0/2515714222 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564602a43210 con 0x564602dd4210 2024-04-21T23:20:05.854 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.847+0000 7f7ac8ff9640 1 -- 172.21.15.40:0/970566437 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f7abc0f1610 con 0x7f7acc100670 2024-04-21T23:20:05.854 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.847+0000 7f7acaffd640 1 --2- 172.21.15.40:0/2515714222 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564602dd4210 0x564602dd6600 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f7ab400bd30 tx=0x7f7ab400f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8277843cdf70e47 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:05.855 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.847+0000 7f7acbfff640 1 --2- 172.21.15.40:0/970566437 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f7ab80a2da0 0x7f7ab80a5260 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7ac4005c40 tx=0x7f7ac4009500 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:05.855 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.847+0000 7f7ac1ffb640 1 -- 172.21.15.40:0/2515714222 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ab4015070 con 0x564602dd4210 2024-04-21T23:20:05.855 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.847+0000 7f7ac1ffb640 1 -- 172.21.15.40:0/2515714222 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ab400be20 con 0x564602dd4210 2024-04-21T23:20:05.855 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.847+0000 7f7ac1ffb640 1 -- 172.21.15.40:0/2515714222 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ab4011a10 con 0x564602dd4210 2024-04-21T23:20:05.855 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.847+0000 7f7ad4bfb740 1 -- 172.21.15.40:0/2515714222 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564602dd4210 msgr2=0x564602dd6600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:05.855 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.848+0000 7f7ad4bfb740 1 --2- 172.21.15.40:0/2515714222 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564602dd4210 0x564602dd6600 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f7ab400bd30 tx=0x7f7ab400f730 comp rx=0 tx=0).stop 2024-04-21T23:20:05.855 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.848+0000 7f7ad4bfb740 1 -- 172.21.15.40:0/2515714222 shutdown_connections 2024-04-21T23:20:05.855 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.848+0000 7f7ad4bfb740 1 --2- 172.21.15.40:0/2515714222 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564602dcb440 0x564602dd3cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:05.855 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.848+0000 7f7ad4bfb740 1 --2- 172.21.15.40:0/2515714222 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564602dd4210 0x564602dd6600 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:05.855 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.848+0000 7f7ad4bfb740 1 --2- 172.21.15.40:0/2515714222 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564602dcab20 0x564602dcaf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:05.855 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.848+0000 7f7ad4bfb740 1 -- 172.21.15.40:0/2515714222 >> 172.21.15.40:0/2515714222 conn(0x564602dc5440 msgr2=0x564602dc5820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:05.855 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.848+0000 7f7ad4bfb740 1 -- 172.21.15.40:0/2515714222 shutdown_connections 2024-04-21T23:20:05.855 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.848+0000 7f7ad4bfb740 1 -- 172.21.15.40:0/2515714222 wait complete. 2024-04-21T23:20:05.856 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.848+0000 7f7ad4bfb740 1 -- start start 2024-04-21T23:20:05.856 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.848+0000 7f7ad4bfb740 1 Processor -- start 2024-04-21T23:20:05.856 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.848+0000 7f7ad4bfb740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564602dcab20 0x564602dd1260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:05.856 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.848+0000 7f7ad4bfb740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564602dcb440 0x564602dd17a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:05.856 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.848+0000 7f7ad4bfb740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564602dd4210 0x564602dd1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:05.856 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.848+0000 7f7ad4bfb740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x564602dd6e40 con 0x564602dcab20 2024-04-21T23:20:05.856 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.848+0000 7f7ad4bfb740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x564602ceaa10 con 0x564602dd4210 2024-04-21T23:20:05.856 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.848+0000 7f7ad4bfb740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x564602dd6fc0 con 0x564602dcb440 2024-04-21T23:20:05.856 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.848+0000 7f7ad4bfb740 10 client.0 _ll_register_callbacks cb 0x564602d25b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:20:05.856 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.848+0000 7f7acb7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564602dcab20 0x564602dd1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:05.856 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.849+0000 7f7acb7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564602dcab20 0x564602dd1260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37010/0 (socket says 172.21.15.40:37010) 2024-04-21T23:20:05.856 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.849+0000 7f7acb7fe640 1 -- 172.21.15.40:0/2538188292 learned_addr learned my addr 172.21.15.40:0/2538188292 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:20:05.856 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.849+0000 7f7acbfff640 1 --2- 172.21.15.40:0/2538188292 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564602dd4210 0x564602dd1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:05.857 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.849+0000 7f7acaffd640 1 --2- 172.21.15.40:0/2538188292 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564602dcb440 0x564602dd17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:05.857 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.849+0000 7f7acb7fe640 1 -- 172.21.15.40:0/2538188292 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564602dcb440 msgr2=0x564602dd17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:05.857 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.849+0000 7f7acb7fe640 1 --2- 172.21.15.40:0/2538188292 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564602dcb440 0x564602dd17a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:05.857 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.849+0000 7f7acb7fe640 1 -- 172.21.15.40:0/2538188292 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564602dd4210 msgr2=0x564602dd1ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:05.857 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.849+0000 7f7acb7fe640 1 --2- 172.21.15.40:0/2538188292 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564602dd4210 0x564602dd1ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:05.857 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.849+0000 7f7acb7fe640 1 -- 172.21.15.40:0/2538188292 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564602b0f620 con 0x564602dcab20 2024-04-21T23:20:05.857 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.849+0000 7f7acaffd640 1 --2- 172.21.15.40:0/2538188292 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564602dcb440 0x564602dd17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:20:05.857 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.849+0000 7f7acb7fe640 1 --2- 172.21.15.40:0/2538188292 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564602dcab20 0x564602dd1260 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f7abc0bd590 tx=0x7f7abc005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8e3e948124411a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:05.857 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.849+0000 7f7aaaffd640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-21T23:20:05.857 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.849+0000 7f7aaaffd640 1 -- 172.21.15.40:0/2538188292 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7abc04b070 con 0x564602dcab20 2024-04-21T23:20:05.857 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.849+0000 7f7aaaffd640 1 -- 172.21.15.40:0/2538188292 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f7abc0a80c0 con 0x564602dcab20 2024-04-21T23:20:05.857 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.849+0000 7f7ad4bfb740 1 -- 172.21.15.40:0/2538188292 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ab400be20 con 0x564602dcab20 2024-04-21T23:20:05.857 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.849+0000 7f7aaaffd640 1 -- 172.21.15.40:0/2538188292 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7abc0bd890 con 0x564602dcab20 2024-04-21T23:20:05.857 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.849+0000 7f7ad4bfb740 10 client.16077 fetch_fsmap learned FSMap version 28 2024-04-21T23:20:05.858 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.849+0000 7f7ad4bfb740 1 -- 172.21.15.40:0/2538188292 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564602de30c0 con 0x564602dcab20 2024-04-21T23:20:05.858 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.850+0000 7f7aaaffd640 1 -- 172.21.15.40:0/2538188292 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ab400be20 con 0x564602dcab20 2024-04-21T23:20:05.858 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.850+0000 7f7aaaffd640 1 -- 172.21.15.40:0/2538188292 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f7abc0eec30 con 0x564602dcab20 2024-04-21T23:20:05.858 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.850+0000 7f7ad4bfb740 10 client.16077 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T23:20:05.858 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.850+0000 7f7ad4bfb740 10 client.16077 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:20:05.858 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.850+0000 7f7ad4bfb740 20 client.16077 populate_metadata read hostname 'smithi040' 2024-04-21T23:20:05.858 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.850+0000 7f7ad4bfb740 1 --2- 172.21.15.40:0/2538188292 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x564602de45a0 0x564602de6a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:05.858 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.850+0000 7f7ad4bfb740 4 client.16077 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:20:05.858 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.850+0000 7f7ad4bfb740 1 -- 172.21.15.40:0/2538188292 --> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564602d3c0b0 con 0x564602de45a0 2024-04-21T23:20:05.858 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.850+0000 7f7acaffd640 1 --2- 172.21.15.40:0/2538188292 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x564602de45a0 0x564602de6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:05.858 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.850+0000 7f7acaffd640 1 --2- 172.21.15.40:0/2538188292 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x564602de45a0 0x564602de6a40 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c0af67e3f4e0c865 server_cookie=d21658212991c486 in_seq=0 out_seq=0 2024-04-21T23:20:05.858 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.850+0000 7f7aaaffd640 10 client.16077 ms_handle_connect on v2:172.21.15.111:6838/3712235548 2024-04-21T23:20:05.858 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.851+0000 7f7aaaffd640 1 -- 172.21.15.40:0/2538188292 <== mds.0 v2:172.21.15.111:6838/3712235548 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564602d3c0b0 con 0x564602de45a0 2024-04-21T23:20:05.859 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.851+0000 7f7aaaffd640 10 client.16077 handle_command_reply: tid=0 2024-04-21T23:20:05.859 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.851+0000 7f7ad4bfb740 1 client.16077 shutdown 2024-04-21T23:20:05.859 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.851+0000 7f7ad4bfb740 1 -- 172.21.15.40:0/2538188292 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564602dcab20 msgr2=0x564602dd1260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:05.859 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.851+0000 7f7ad4bfb740 1 --2- 172.21.15.40:0/2538188292 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564602dcab20 0x564602dd1260 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f7abc0bd590 tx=0x7f7abc005bd0 comp rx=0 tx=0).stop 2024-04-21T23:20:05.859 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.851+0000 7f7ad4bfb740 1 -- 172.21.15.40:0/2538188292 shutdown_connections 2024-04-21T23:20:05.859 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.851+0000 7f7ad4bfb740 1 --2- 172.21.15.40:0/2538188292 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x564602de45a0 0x564602de6a40 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:05.859 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.851+0000 7f7ad4bfb740 1 --2- 172.21.15.40:0/2538188292 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564602dd4210 0x564602dd1ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:05.859 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.851+0000 7f7ad4bfb740 1 --2- 172.21.15.40:0/2538188292 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564602dcb440 0x564602dd17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:05.859 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.851+0000 7f7aaaffd640 0 client.16077 ms_handle_reset on v2:172.21.15.111:6838/3712235548 2024-04-21T23:20:05.859 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.851+0000 7f7ad4bfb740 1 --2- 172.21.15.40:0/2538188292 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564602dcab20 0x564602dd1260 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:05.859 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.851+0000 7f7ad4bfb740 1 -- 172.21.15.40:0/2538188292 >> 172.21.15.40:0/2538188292 conn(0x564602dc5440 msgr2=0x564602dc7190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:05.859 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.851+0000 7f7ad4bfb740 1 -- 172.21.15.40:0/2538188292 shutdown_connections 2024-04-21T23:20:05.859 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.851+0000 7f7ad4bfb740 1 -- 172.21.15.40:0/2538188292 wait complete. 2024-04-21T23:20:05.859 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.851+0000 7f7ad4bfb740 20 client.16077 trim_cache size 0 max 16384 2024-04-21T23:20:05.870 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.866+0000 7f7ad4bfb740 1 Processor -- start 2024-04-21T23:20:05.870 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.866+0000 7f7ad4bfb740 1 -- start start 2024-04-21T23:20:05.870 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.866+0000 7f7ad4bfb740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564602dcab20 0x564602dd3990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:05.870 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.866+0000 7f7ad4bfb740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564602dcb440 0x564602ddfd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:05.870 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.866+0000 7f7ad4bfb740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564602dd4210 0x564602dec350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:05.870 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.866+0000 7f7ad4bfb740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7acc1b14a0 con 0x564602dd4210 2024-04-21T23:20:05.870 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.866+0000 7f7ad4bfb740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f7acc1aed10 con 0x564602dcb440 2024-04-21T23:20:05.870 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.866+0000 7f7ad4bfb740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7acc1acb60 con 0x564602dcab20 2024-04-21T23:20:05.870 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.866+0000 7f7acbfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564602dd4210 0x564602dec350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:05.870 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.866+0000 7f7acaffd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564602dcb440 0x564602ddfd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:05.870 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.866+0000 7f7acaffd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564602dcb440 0x564602ddfd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.40:53588/0 (socket says 172.21.15.40:53588) 2024-04-21T23:20:05.871 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.866+0000 7f7acaffd640 1 -- 172.21.15.40:0/1839668792 learned_addr learned my addr 172.21.15.40:0/1839668792 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:20:05.871 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.866+0000 7f7acbfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564602dd4210 0x564602dec350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37018/0 (socket says 172.21.15.40:37018) 2024-04-21T23:20:05.871 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.867+0000 7f7acaffd640 1 -- 172.21.15.40:0/1839668792 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564602dcab20 msgr2=0x564602dd3990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:05.871 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.867+0000 7f7acaffd640 1 --2- 172.21.15.40:0/1839668792 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564602dcab20 0x564602dd3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:05.871 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.867+0000 7f7acaffd640 1 -- 172.21.15.40:0/1839668792 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564602dd4210 msgr2=0x564602dec350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:05.871 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.867+0000 7f7acaffd640 1 --2- 172.21.15.40:0/1839668792 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564602dd4210 0x564602dec350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:05.871 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.867+0000 7f7acaffd640 1 -- 172.21.15.40:0/1839668792 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564602dc6600 con 0x564602dcb440 2024-04-21T23:20:05.871 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.867+0000 7f7acaffd640 1 --2- 172.21.15.40:0/1839668792 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564602dcb440 0x564602ddfd00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7ab40115c0 tx=0x7f7ab40115f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f28daee89d8081d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:05.871 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.867+0000 7f7ac17fa640 1 -- 172.21.15.40:0/1839668792 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ab402b070 con 0x564602dcb440 2024-04-21T23:20:05.871 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.867+0000 7f7ac17fa640 1 -- 172.21.15.40:0/1839668792 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ab400b3a0 con 0x564602dcb440 2024-04-21T23:20:05.871 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.867+0000 7f7ac17fa640 1 -- 172.21.15.40:0/1839668792 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ab4015050 con 0x564602dcb440 2024-04-21T23:20:05.871 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.867+0000 7f7ad4bfb740 1 -- 172.21.15.40:0/1839668792 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564602dcb440 msgr2=0x564602ddfd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:05.871 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.867+0000 7f7ad4bfb740 1 --2- 172.21.15.40:0/1839668792 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564602dcb440 0x564602ddfd00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7ab40115c0 tx=0x7f7ab40115f0 comp rx=0 tx=0).stop 2024-04-21T23:20:05.872 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.867+0000 7f7ad4bfb740 1 -- 172.21.15.40:0/1839668792 shutdown_connections 2024-04-21T23:20:05.872 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.867+0000 7f7ad4bfb740 1 --2- 172.21.15.40:0/1839668792 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564602dcb440 0x564602ddfd00 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:05.872 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.867+0000 7f7ad4bfb740 1 --2- 172.21.15.40:0/1839668792 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564602dd4210 0x564602dec350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:05.872 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.867+0000 7f7ad4bfb740 1 --2- 172.21.15.40:0/1839668792 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564602dcab20 0x564602dd3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:05.872 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.868+0000 7f7ad4bfb740 1 -- 172.21.15.40:0/1839668792 >> 172.21.15.40:0/1839668792 conn(0x564602dc5440 msgr2=0x564602dcc590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:05.872 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.868+0000 7f7ad4bfb740 1 -- 172.21.15.40:0/1839668792 shutdown_connections 2024-04-21T23:20:05.872 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.868+0000 7f7ad4bfb740 1 -- 172.21.15.40:0/1839668792 wait complete. 2024-04-21T23:20:05.872 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.868+0000 7f7ad4bfb740 1 -- start start 2024-04-21T23:20:05.872 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.868+0000 7f7ad4bfb740 1 Processor -- start 2024-04-21T23:20:05.872 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.868+0000 7f7ad4bfb740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564602dcab20 0x564602dce590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:05.872 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.868+0000 7f7ad4bfb740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564602dcb440 0x564602dcead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:05.872 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.868+0000 7f7ad4bfb740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564602dd4210 0x564602e49bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:05.872 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.868+0000 7f7ad4bfb740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7acc1ad1d0 con 0x564602dcab20 2024-04-21T23:20:05.873 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.868+0000 7f7ad4bfb740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f7acc1ae640 con 0x564602dcb440 2024-04-21T23:20:05.873 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.868+0000 7f7ad4bfb740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7acc1ad8a0 con 0x564602dd4210 2024-04-21T23:20:05.873 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.868+0000 7f7ad4bfb740 10 client.0 _ll_register_callbacks cb 0x7f7abc0044e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:20:05.873 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.868+0000 7f7acb7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564602dcab20 0x564602dce590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:05.873 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.868+0000 7f7acb7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564602dcab20 0x564602dce590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37024/0 (socket says 172.21.15.40:37024) 2024-04-21T23:20:05.873 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.868+0000 7f7acb7fe640 1 -- 172.21.15.40:0/3769380006 learned_addr learned my addr 172.21.15.40:0/3769380006 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:20:05.873 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.868+0000 7f7acaffd640 1 --2- 172.21.15.40:0/3769380006 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564602dcb440 0x564602dcead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:05.873 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.869+0000 7f7acb7fe640 1 -- 172.21.15.40:0/3769380006 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564602dd4210 msgr2=0x564602e49bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:05.873 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.869+0000 7f7acb7fe640 1 --2- 172.21.15.40:0/3769380006 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564602dd4210 0x564602e49bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:05.873 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.869+0000 7f7acb7fe640 1 -- 172.21.15.40:0/3769380006 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564602dcb440 msgr2=0x564602dcead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:05.873 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.869+0000 7f7acb7fe640 1 --2- 172.21.15.40:0/3769380006 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564602dcb440 0x564602dcead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:05.873 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.869+0000 7f7acb7fe640 1 -- 172.21.15.40:0/3769380006 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564602e4a100 con 0x564602dcab20 2024-04-21T23:20:05.873 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.869+0000 7f7acb7fe640 1 --2- 172.21.15.40:0/3769380006 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564602dcab20 0x564602dce590 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f7abc0bbb40 tx=0x7f7abc0bd460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36fcbd1b39d47827 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:05.873 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.869+0000 7f7aaaffd640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-21T23:20:05.874 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.869+0000 7f7aaaffd640 1 -- 172.21.15.40:0/3769380006 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7abc04b070 con 0x564602dcab20 2024-04-21T23:20:05.874 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.869+0000 7f7aaaffd640 1 -- 172.21.15.40:0/3769380006 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f7abc0eeb70 con 0x564602dcab20 2024-04-21T23:20:05.874 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.869+0000 7f7ad4bfb740 1 -- 172.21.15.40:0/3769380006 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ab400b3a0 con 0x564602dcab20 2024-04-21T23:20:05.874 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.869+0000 7f7aaaffd640 1 -- 172.21.15.40:0/3769380006 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7abc0b9b30 con 0x564602dcab20 2024-04-21T23:20:05.874 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.869+0000 7f7ad4bfb740 10 client.16083 fetch_fsmap learned FSMap version 28 2024-04-21T23:20:05.874 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.869+0000 7f7ad4bfb740 1 -- 172.21.15.40:0/3769380006 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564602e4b370 con 0x564602dcab20 2024-04-21T23:20:05.874 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.869+0000 7f7aaaffd640 1 -- 172.21.15.40:0/3769380006 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ab400b3a0 con 0x564602dcab20 2024-04-21T23:20:05.874 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.870+0000 7f7aaaffd640 1 -- 172.21.15.40:0/3769380006 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f7abc0f12e0 con 0x564602dcab20 2024-04-21T23:20:05.874 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.870+0000 7f7ad4bfb740 10 client.16083 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T23:20:05.874 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.870+0000 7f7ad4bfb740 10 client.16083 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:20:05.874 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.870+0000 7f7ad4bfb740 20 client.16083 populate_metadata read hostname 'smithi040' 2024-04-21T23:20:05.874 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.870+0000 7f7ad4bfb740 1 --2- 172.21.15.40:0/3769380006 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x564602e4b900 0x564602e4dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:05.874 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.870+0000 7f7ad4bfb740 4 client.16083 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T23:20:05.875 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.870+0000 7f7ad4bfb740 1 -- 172.21.15.40:0/3769380006 --> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564602d36400 con 0x564602e4b900 2024-04-21T23:20:05.875 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.870+0000 7f7acaffd640 1 --2- 172.21.15.40:0/3769380006 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x564602e4b900 0x564602e4dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:05.875 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.870+0000 7f7aaaffd640 10 client.16083 ms_handle_connect on v2:172.21.15.111:6838/3712235548 2024-04-21T23:20:05.875 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.870+0000 7f7acaffd640 1 --2- 172.21.15.40:0/3769380006 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x564602e4b900 0x564602e4dd60 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4b7485b270410b server_cookie=3a1279f63dc1239f in_seq=0 out_seq=0 2024-04-21T23:20:05.879 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.879+0000 7f7aaaffd640 1 -- 172.21.15.40:0/3769380006 <== mds.0 v2:172.21.15.111:6838/3712235548 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564602d36400 con 0x564602e4b900 2024-04-21T23:20:05.879 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.879+0000 7f7aaaffd640 10 client.16083 handle_command_reply: tid=0 2024-04-21T23:20:05.879 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.879+0000 7f7ad4bfb740 1 client.16083 shutdown 2024-04-21T23:20:05.879 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.879+0000 7f7ad4bfb740 1 -- 172.21.15.40:0/3769380006 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564602dcab20 msgr2=0x564602dce590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:05.879 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.879+0000 7f7ad4bfb740 1 --2- 172.21.15.40:0/3769380006 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564602dcab20 0x564602dce590 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f7abc0bbb40 tx=0x7f7abc0bd460 comp rx=0 tx=0).stop 2024-04-21T23:20:05.879 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.879+0000 7f7ad4bfb740 1 -- 172.21.15.40:0/3769380006 shutdown_connections 2024-04-21T23:20:05.879 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.879+0000 7f7ad4bfb740 1 --2- 172.21.15.40:0/3769380006 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x564602e4b900 0x564602e4dd60 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:05.879 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.879+0000 7f7ad4bfb740 1 --2- 172.21.15.40:0/3769380006 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564602dcb440 0x564602dcead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:05.879 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.879+0000 7f7ad4bfb740 1 --2- 172.21.15.40:0/3769380006 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564602dd4210 0x564602e49bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:05.879 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.879+0000 7f7aaaffd640 0 client.16083 ms_handle_reset on v2:172.21.15.111:6838/3712235548 2024-04-21T23:20:05.879 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.879+0000 7f7ad4bfb740 1 --2- 172.21.15.40:0/3769380006 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564602dcab20 0x564602dce590 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:05.879 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.879+0000 7f7ad4bfb740 1 -- 172.21.15.40:0/3769380006 >> 172.21.15.40:0/3769380006 conn(0x564602dc5440 msgr2=0x564602dd1220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:05.879 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.879+0000 7f7ad4bfb740 1 -- 172.21.15.40:0/3769380006 shutdown_connections 2024-04-21T23:20:05.879 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.879+0000 7f7ad4bfb740 1 -- 172.21.15.40:0/3769380006 wait complete. 2024-04-21T23:20:05.880 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.879+0000 7f7ad4bfb740 20 client.16083 trim_cache size 0 max 16384 2024-04-21T23:20:05.880 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:{ 2024-04-21T23:20:05.880 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "message": "", 2024-04-21T23:20:05.880 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "return_code": 0 2024-04-21T23:20:05.880 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:} 2024-04-21T23:20:05.881 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.880+0000 7f7ad1f37640 1 -- 172.21.15.40:0/970566437 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f7ab80a2da0 msgr2=0x7f7ab80a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:05.881 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.880+0000 7f7ad1f37640 1 --2- 172.21.15.40:0/970566437 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f7ab80a2da0 0x7f7ab80a5260 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7ac4005c40 tx=0x7f7ac4009500 comp rx=0 tx=0).stop 2024-04-21T23:20:05.881 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.880+0000 7f7ad1f37640 1 -- 172.21.15.40:0/970566437 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7acc100670 msgr2=0x7f7acc197f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:05.881 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.880+0000 7f7ad1f37640 1 --2- 172.21.15.40:0/970566437 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7acc100670 0x7f7acc197f90 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f7abc005940 tx=0x7f7abc0042f0 comp rx=0 tx=0).stop 2024-04-21T23:20:05.881 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.880+0000 7f7ad1f37640 1 -- 172.21.15.40:0/970566437 shutdown_connections 2024-04-21T23:20:05.881 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.880+0000 7f7ad1f37640 1 --2- 172.21.15.40:0/970566437 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f7ab80a2da0 0x7f7ab80a5260 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:05.881 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.880+0000 7f7ad1f37640 1 --2- 172.21.15.40:0/970566437 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7acc106630 0x7f7acc19c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:05.881 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.880+0000 7f7ad1f37640 1 --2- 172.21.15.40:0/970566437 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7acc101030 0x7f7acc1984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:05.881 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.880+0000 7f7ad1f37640 1 --2- 172.21.15.40:0/970566437 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7acc100670 0x7f7acc197f90 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:05.881 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.880+0000 7f7ad1f37640 1 -- 172.21.15.40:0/970566437 >> 172.21.15.40:0/970566437 conn(0x7f7acc0777c0 msgr2=0x7f7acc069090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:05.882 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.880+0000 7f7ad1f37640 1 -- 172.21.15.40:0/970566437 shutdown_connections 2024-04-21T23:20:05.882 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:05.880+0000 7f7ad1f37640 1 -- 172.21.15.40:0/970566437 wait complete. 2024-04-21T23:20:06.488 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T23:20:06.488 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T23:20:06.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.598+0000 7ff5782c6640 1 -- 172.21.15.40:0/71601790 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5701020d0 msgr2=0x7ff57010e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:06.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.598+0000 7ff5782c6640 1 --2- 172.21.15.40:0/71601790 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5701020d0 0x7ff57010e5e0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7ff568009cd0 tx=0x7ff56802c0f0 comp rx=0 tx=0).stop 2024-04-21T23:20:06.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.598+0000 7ff5782c6640 1 -- 172.21.15.40:0/71601790 shutdown_connections 2024-04-21T23:20:06.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.598+0000 7ff5782c6640 1 --2- 172.21.15.40:0/71601790 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5701020d0 0x7ff57010e5e0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:06.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.598+0000 7ff5782c6640 1 --2- 172.21.15.40:0/71601790 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff570101710 0x7ff570101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:06.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.598+0000 7ff5782c6640 1 --2- 172.21.15.40:0/71601790 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff570107720 0x7ff570107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:06.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.598+0000 7ff5782c6640 1 -- 172.21.15.40:0/71601790 >> 172.21.15.40:0/71601790 conn(0x7ff570078b90 msgr2=0x7ff570078fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:06.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.599+0000 7ff5782c6640 1 -- 172.21.15.40:0/71601790 shutdown_connections 2024-04-21T23:20:06.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.599+0000 7ff5782c6640 1 -- 172.21.15.40:0/71601790 wait complete. 2024-04-21T23:20:06.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.599+0000 7ff5782c6640 1 Processor -- start 2024-04-21T23:20:06.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.599+0000 7ff5782c6640 1 -- start start 2024-04-21T23:20:06.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.599+0000 7ff5782c6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff570101710 0x7ff570073a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:06.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.599+0000 7ff5782c6640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff5701020d0 0x7ff570073f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:06.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.599+0000 7ff5782c6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff570107720 0x7ff5700744c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:06.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.599+0000 7ff5782c6640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff570111b30 con 0x7ff570107720 2024-04-21T23:20:06.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.599+0000 7ff5782c6640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff5701119b0 con 0x7ff5701020d0 2024-04-21T23:20:06.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.599+0000 7ff5782c6640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff570111cb0 con 0x7ff570101710 2024-04-21T23:20:06.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.599+0000 7ff576863640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff570107720 0x7ff5700744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:06.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.600+0000 7ff576863640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff570107720 0x7ff5700744c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37048/0 (socket says 172.21.15.40:37048) 2024-04-21T23:20:06.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.600+0000 7ff576863640 1 -- 172.21.15.40:0/52004761 learned_addr learned my addr 172.21.15.40:0/52004761 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:20:06.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.600+0000 7ff575861640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff5701020d0 0x7ff570073f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:06.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.600+0000 7ff576062640 1 --2- 172.21.15.40:0/52004761 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff570101710 0x7ff570073a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:06.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.600+0000 7ff576863640 1 -- 172.21.15.40:0/52004761 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff570101710 msgr2=0x7ff570073a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:06.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.600+0000 7ff576863640 1 --2- 172.21.15.40:0/52004761 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff570101710 0x7ff570073a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:06.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.600+0000 7ff576863640 1 -- 172.21.15.40:0/52004761 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff5701020d0 msgr2=0x7ff570073f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:06.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.600+0000 7ff576863640 1 --2- 172.21.15.40:0/52004761 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff5701020d0 0x7ff570073f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:06.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.600+0000 7ff576863640 1 -- 172.21.15.40:0/52004761 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5701a6450 con 0x7ff570107720 2024-04-21T23:20:06.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.600+0000 7ff575861640 1 --2- 172.21.15.40:0/52004761 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff5701020d0 0x7ff570073f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:20:06.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.600+0000 7ff576863640 1 --2- 172.21.15.40:0/52004761 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff570107720 0x7ff5700744c0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7ff568004af0 tx=0x7ff568032ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:06.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.600+0000 7ff5677fe640 1 -- 172.21.15.40:0/52004761 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff568041020 con 0x7ff570107720 2024-04-21T23:20:06.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.600+0000 7ff5677fe640 1 -- 172.21.15.40:0/52004761 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff568034cd0 con 0x7ff570107720 2024-04-21T23:20:06.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.600+0000 7ff5782c6640 1 -- 172.21.15.40:0/52004761 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5701a66e0 con 0x7ff570107720 2024-04-21T23:20:06.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.600+0000 7ff5677fe640 1 -- 172.21.15.40:0/52004761 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff568039040 con 0x7ff570107720 2024-04-21T23:20:06.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.600+0000 7ff5782c6640 1 -- 172.21.15.40:0/52004761 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5701a6c20 con 0x7ff570107720 2024-04-21T23:20:06.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.602+0000 7ff5677fe640 1 -- 172.21.15.40:0/52004761 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7ff568044020 con 0x7ff570107720 2024-04-21T23:20:06.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.603+0000 7ff57af92740 1 Processor -- start 2024-04-21T23:20:06.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.603+0000 7ff57af92740 1 -- start start 2024-04-21T23:20:06.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.604+0000 7ff57af92740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f6cafdf30 0x557f6cafe310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:06.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.604+0000 7ff5677fe640 1 --2- 172.21.15.40:0/52004761 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7ff55c0a2d80 0x7ff55c0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:06.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.604+0000 7ff57af92740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557f6cafe850 0x557f6cb06ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:06.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.604+0000 7ff57af92740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557f6cb07430 0x557f6cb09820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:06.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.604+0000 7ff57af92740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x557f6ca62560 con 0x557f6cafe850 2024-04-21T23:20:06.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.604+0000 7ff57af92740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x557f6ca0a730 con 0x557f6cb07430 2024-04-21T23:20:06.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.604+0000 7ff57af92740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x557f6ca5e520 con 0x557f6cafdf30 2024-04-21T23:20:06.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.604+0000 7ff576863640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557f6cb07430 0x557f6cb09820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:06.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.604+0000 7ff576062640 1 --2- 172.21.15.40:0/52004761 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7ff55c0a2d80 0x7ff55c0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:06.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.604+0000 7ff575861640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f6cafdf30 0x557f6cafe310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:06.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.604+0000 7ff576863640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557f6cb07430 0x557f6cb09820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.40:53618/0 (socket says 172.21.15.40:53618) 2024-04-21T23:20:06.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.604+0000 7ff576863640 1 -- 172.21.15.40:0/280914355 learned_addr learned my addr 172.21.15.40:0/280914355 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:20:06.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.604+0000 7ff5677fe640 1 -- 172.21.15.40:0/52004761 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7ff5680f1290 con 0x7ff570107720 2024-04-21T23:20:06.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.604+0000 7ff575861640 1 --2- 172.21.15.40:0/280914355 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557f6cafe850 0x557f6cb06ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:06.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.604+0000 7ff576863640 1 -- 172.21.15.40:0/280914355 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f6cafdf30 msgr2=0x557f6cafe310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:06.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.604+0000 7ff576863640 1 --2- 172.21.15.40:0/280914355 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f6cafdf30 0x557f6cafe310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:06.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.604+0000 7ff576863640 1 -- 172.21.15.40:0/280914355 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557f6cafe850 msgr2=0x557f6cb06ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:06.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.604+0000 7ff576863640 1 --2- 172.21.15.40:0/280914355 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557f6cafe850 0x557f6cb06ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:06.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.604+0000 7ff576863640 1 -- 172.21.15.40:0/280914355 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f6c8407d0 con 0x557f6cb07430 2024-04-21T23:20:06.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.604+0000 7ff575861640 1 --2- 172.21.15.40:0/280914355 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557f6cafe850 0x557f6cb06ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:20:06.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.604+0000 7ff576062640 1 --2- 172.21.15.40:0/52004761 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7ff55c0a2d80 0x7ff55c0a5240 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff56c009b50 tx=0x7ff56c005f90 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:06.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.604+0000 7ff576863640 1 --2- 172.21.15.40:0/280914355 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557f6cb07430 0x557f6cb09820 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff568009ca0 tx=0x7ff568048000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=25b36cf1b69dbc47 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:06.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.605+0000 7ff564bfe640 1 -- 172.21.15.40:0/280914355 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5680bd050 con 0x557f6cb07430 2024-04-21T23:20:06.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.605+0000 7ff564bfe640 1 -- 172.21.15.40:0/280914355 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff568030200 con 0x557f6cb07430 2024-04-21T23:20:06.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.605+0000 7ff564bfe640 1 -- 172.21.15.40:0/280914355 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff568003e60 con 0x557f6cb07430 2024-04-21T23:20:06.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.605+0000 7ff57af92740 1 -- 172.21.15.40:0/280914355 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557f6cb07430 msgr2=0x557f6cb09820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:06.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.605+0000 7ff57af92740 1 --2- 172.21.15.40:0/280914355 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557f6cb07430 0x557f6cb09820 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff568009ca0 tx=0x7ff568048000 comp rx=0 tx=0).stop 2024-04-21T23:20:06.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.605+0000 7ff57af92740 1 -- 172.21.15.40:0/280914355 shutdown_connections 2024-04-21T23:20:06.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.605+0000 7ff57af92740 1 --2- 172.21.15.40:0/280914355 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557f6cb07430 0x557f6cb09820 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:06.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.605+0000 7ff57af92740 1 --2- 172.21.15.40:0/280914355 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557f6cafe850 0x557f6cb06ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:06.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.605+0000 7ff57af92740 1 --2- 172.21.15.40:0/280914355 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f6cafdf30 0x557f6cafe310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:06.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.605+0000 7ff57af92740 1 -- 172.21.15.40:0/280914355 >> 172.21.15.40:0/280914355 conn(0x557f6caf8660 msgr2=0x557f6caf8a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:06.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.606+0000 7ff57af92740 1 -- 172.21.15.40:0/280914355 shutdown_connections 2024-04-21T23:20:06.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.606+0000 7ff57af92740 1 -- 172.21.15.40:0/280914355 wait complete. 2024-04-21T23:20:06.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.606+0000 7ff57af92740 1 -- start start 2024-04-21T23:20:06.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.606+0000 7ff57af92740 1 Processor -- start 2024-04-21T23:20:06.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.606+0000 7ff57af92740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557f6cafdf30 0x557f6cb04410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:06.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.606+0000 7ff57af92740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f6cafe850 0x557f6cb04950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:06.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.606+0000 7ff57af92740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557f6cb07430 0x557f6cb04e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:06.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.606+0000 7ff57af92740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x557f6cb09fe0 con 0x557f6cafdf30 2024-04-21T23:20:06.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.606+0000 7ff57af92740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x557f6ca1de00 con 0x557f6cb07430 2024-04-21T23:20:06.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.606+0000 7ff57af92740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x557f6cb0a160 con 0x557f6cafe850 2024-04-21T23:20:06.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.606+0000 7ff57af92740 10 client.0 _ll_register_callbacks cb 0x557f6ca58eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:20:06.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.606+0000 7ff576062640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557f6cafdf30 0x557f6cb04410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:06.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.606+0000 7ff576062640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557f6cafdf30 0x557f6cb04410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37068/0 (socket says 172.21.15.40:37068) 2024-04-21T23:20:06.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.606+0000 7ff576062640 1 -- 172.21.15.40:0/3859536883 learned_addr learned my addr 172.21.15.40:0/3859536883 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:20:06.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.606+0000 7ff576863640 1 --2- 172.21.15.40:0/3859536883 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557f6cb07430 0x557f6cb04e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:06.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.606+0000 7ff575861640 1 --2- 172.21.15.40:0/3859536883 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f6cafe850 0x557f6cb04950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:06.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.607+0000 7ff576863640 1 -- 172.21.15.40:0/3859536883 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f6cafe850 msgr2=0x557f6cb04950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:06.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.607+0000 7ff576863640 1 --2- 172.21.15.40:0/3859536883 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f6cafe850 0x557f6cb04950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:06.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.607+0000 7ff576863640 1 -- 172.21.15.40:0/3859536883 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557f6cafdf30 msgr2=0x557f6cb04410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:06.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.607+0000 7ff576863640 1 --2- 172.21.15.40:0/3859536883 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557f6cafdf30 0x557f6cb04410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:06.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.607+0000 7ff576863640 1 -- 172.21.15.40:0/3859536883 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f6c7763f0 con 0x557f6cb07430 2024-04-21T23:20:06.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.607+0000 7ff576062640 1 --2- 172.21.15.40:0/3859536883 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557f6cafdf30 0x557f6cb04410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:20:06.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.607+0000 7ff576863640 1 --2- 172.21.15.40:0/3859536883 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557f6cb07430 0x557f6cb04e90 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x557f6caf97f0 tx=0x7ff5680095c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ae3e321e31a6b10 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:06.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.607+0000 7ff559ffb640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-21T23:20:06.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.607+0000 7ff559ffb640 1 -- 172.21.15.40:0/3859536883 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5680bce70 con 0x557f6cb07430 2024-04-21T23:20:06.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.607+0000 7ff559ffb640 1 -- 172.21.15.40:0/3859536883 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7ff5680dd410 con 0x557f6cb07430 2024-04-21T23:20:06.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.607+0000 7ff57af92740 1 -- 172.21.15.40:0/3859536883 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x557f6ca6faa0 con 0x557f6cb07430 2024-04-21T23:20:06.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.607+0000 7ff559ffb640 1 -- 172.21.15.40:0/3859536883 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff5680bd050 con 0x557f6cb07430 2024-04-21T23:20:06.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.607+0000 7ff57af92740 10 client.25613 fetch_fsmap learned FSMap version 28 2024-04-21T23:20:06.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.607+0000 7ff57af92740 1 -- 172.21.15.40:0/3859536883 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x557f6c7360f0 con 0x557f6cb07430 2024-04-21T23:20:06.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.607+0000 7ff559ffb640 1 -- 172.21.15.40:0/3859536883 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x557f6ca6faa0 con 0x557f6cb07430 2024-04-21T23:20:06.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.607+0000 7ff559ffb640 1 -- 172.21.15.40:0/3859536883 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7ff56803c030 con 0x557f6cb07430 2024-04-21T23:20:06.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.608+0000 7ff57af92740 10 client.25613 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T23:20:06.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.608+0000 7ff57af92740 10 client.25613 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:20:06.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.608+0000 7ff57af92740 20 client.25613 populate_metadata read hostname 'smithi040' 2024-04-21T23:20:06.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.608+0000 7ff57af92740 1 --2- 172.21.15.40:0/3859536883 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x557f6cb17320 0x557f6cb19710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:06.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.608+0000 7ff57af92740 4 client.25613 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:20:06.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.608+0000 7ff57af92740 1 -- 172.21.15.40:0/3859536883 --> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557f6ca69df0 con 0x557f6cb17320 2024-04-21T23:20:06.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.608+0000 7ff576062640 1 --2- 172.21.15.40:0/3859536883 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x557f6cb17320 0x557f6cb19710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:06.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.608+0000 7ff576062640 1 --2- 172.21.15.40:0/3859536883 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x557f6cb17320 0x557f6cb19710 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b1784d3edc505471 server_cookie=4f695d185f19caa3 in_seq=0 out_seq=0 2024-04-21T23:20:06.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.608+0000 7ff559ffb640 10 client.25613 ms_handle_connect on v2:172.21.15.111:6838/3712235548 2024-04-21T23:20:06.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.609+0000 7ff559ffb640 1 -- 172.21.15.40:0/3859536883 <== mds.0 v2:172.21.15.111:6838/3712235548 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557f6ca69df0 con 0x557f6cb17320 2024-04-21T23:20:06.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.609+0000 7ff559ffb640 10 client.25613 handle_command_reply: tid=0 2024-04-21T23:20:06.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.609+0000 7ff57af92740 1 client.25613 shutdown 2024-04-21T23:20:06.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.610+0000 7ff57af92740 1 -- 172.21.15.40:0/3859536883 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557f6cb07430 msgr2=0x557f6cb04e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:06.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.610+0000 7ff57af92740 1 --2- 172.21.15.40:0/3859536883 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557f6cb07430 0x557f6cb04e90 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x557f6caf97f0 tx=0x7ff5680095c0 comp rx=0 tx=0).stop 2024-04-21T23:20:06.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.610+0000 7ff57af92740 1 -- 172.21.15.40:0/3859536883 shutdown_connections 2024-04-21T23:20:06.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.610+0000 7ff57af92740 1 --2- 172.21.15.40:0/3859536883 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x557f6cb17320 0x557f6cb19710 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:06.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.610+0000 7ff57af92740 1 --2- 172.21.15.40:0/3859536883 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557f6cb07430 0x557f6cb04e90 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:06.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.610+0000 7ff57af92740 1 --2- 172.21.15.40:0/3859536883 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f6cafe850 0x557f6cb04950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:06.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.610+0000 7ff559ffb640 0 client.25613 ms_handle_reset on v2:172.21.15.111:6838/3712235548 2024-04-21T23:20:06.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.610+0000 7ff57af92740 1 --2- 172.21.15.40:0/3859536883 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557f6cafdf30 0x557f6cb04410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:06.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.610+0000 7ff57af92740 1 -- 172.21.15.40:0/3859536883 >> 172.21.15.40:0/3859536883 conn(0x557f6caf8660 msgr2=0x557f6cafa3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:06.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.610+0000 7ff57af92740 1 -- 172.21.15.40:0/3859536883 shutdown_connections 2024-04-21T23:20:06.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.610+0000 7ff57af92740 1 -- 172.21.15.40:0/3859536883 wait complete. 2024-04-21T23:20:06.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.610+0000 7ff57af92740 20 client.25613 trim_cache size 0 max 16384 2024-04-21T23:20:06.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.629+0000 7ff57af92740 1 Processor -- start 2024-04-21T23:20:06.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.629+0000 7ff57af92740 1 -- start start 2024-04-21T23:20:06.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.629+0000 7ff57af92740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557f6caf8660 0x557f6cb12dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:06.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.629+0000 7ff57af92740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f6cafdf30 0x557f6cb04cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:06.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.629+0000 7ff57af92740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557f6cafe850 0x557f6cb1ef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:06.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.629+0000 7ff57af92740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff5701b3690 con 0x557f6cafe850 2024-04-21T23:20:06.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.629+0000 7ff57af92740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff5701b0f00 con 0x557f6caf8660 2024-04-21T23:20:06.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.629+0000 7ff57af92740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff5701aed50 con 0x557f6cafdf30 2024-04-21T23:20:06.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.629+0000 7ff576863640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557f6cafe850 0x557f6cb1ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:06.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.629+0000 7ff576863640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557f6cafe850 0x557f6cb1ef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37082/0 (socket says 172.21.15.40:37082) 2024-04-21T23:20:06.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.629+0000 7ff576863640 1 -- 172.21.15.40:0/2355954336 learned_addr learned my addr 172.21.15.40:0/2355954336 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:20:06.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.629+0000 7ff576062640 1 --2- 172.21.15.40:0/2355954336 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557f6caf8660 0x557f6cb12dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:06.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.630+0000 7ff576863640 1 -- 172.21.15.40:0/2355954336 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f6cafdf30 msgr2=0x557f6cb04cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:06.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.630+0000 7ff576863640 1 --2- 172.21.15.40:0/2355954336 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f6cafdf30 0x557f6cb04cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:06.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.630+0000 7ff576863640 1 -- 172.21.15.40:0/2355954336 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557f6caf8660 msgr2=0x557f6cb12dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:06.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.630+0000 7ff576863640 1 --2- 172.21.15.40:0/2355954336 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557f6caf8660 0x557f6cb12dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:06.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.630+0000 7ff576863640 1 -- 172.21.15.40:0/2355954336 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f6cb02960 con 0x557f6cafe850 2024-04-21T23:20:06.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.630+0000 7ff576863640 1 --2- 172.21.15.40:0/2355954336 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557f6cafe850 0x557f6cb1ef70 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7ff5680bd210 tx=0x7ff5680bba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48354d21a5e1c0d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:06.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.630+0000 7ff55bfff640 1 -- 172.21.15.40:0/2355954336 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff56809c0d0 con 0x557f6cafe850 2024-04-21T23:20:06.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.630+0000 7ff55bfff640 1 -- 172.21.15.40:0/2355954336 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff568003e40 con 0x557f6cafe850 2024-04-21T23:20:06.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.630+0000 7ff55bfff640 1 -- 172.21.15.40:0/2355954336 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5680c6180 con 0x557f6cafe850 2024-04-21T23:20:06.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.631+0000 7ff57af92740 1 -- 172.21.15.40:0/2355954336 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557f6cafe850 msgr2=0x557f6cb1ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:06.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.631+0000 7ff57af92740 1 --2- 172.21.15.40:0/2355954336 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557f6cafe850 0x557f6cb1ef70 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7ff5680bd210 tx=0x7ff5680bba20 comp rx=0 tx=0).stop 2024-04-21T23:20:06.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.631+0000 7ff57af92740 1 -- 172.21.15.40:0/2355954336 shutdown_connections 2024-04-21T23:20:06.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.631+0000 7ff57af92740 1 --2- 172.21.15.40:0/2355954336 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557f6cafe850 0x557f6cb1ef70 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:06.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.631+0000 7ff57af92740 1 --2- 172.21.15.40:0/2355954336 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f6cafdf30 0x557f6cb04cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:06.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.631+0000 7ff57af92740 1 --2- 172.21.15.40:0/2355954336 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557f6caf8660 0x557f6cb12dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:06.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.631+0000 7ff57af92740 1 -- 172.21.15.40:0/2355954336 >> 172.21.15.40:0/2355954336 conn(0x557f6cb03050 msgr2=0x557f6caf76e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:06.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.631+0000 7ff57af92740 1 -- 172.21.15.40:0/2355954336 shutdown_connections 2024-04-21T23:20:06.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.631+0000 7ff57af92740 1 -- 172.21.15.40:0/2355954336 wait complete. 2024-04-21T23:20:06.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.631+0000 7ff57af92740 1 -- start start 2024-04-21T23:20:06.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.631+0000 7ff57af92740 1 Processor -- start 2024-04-21T23:20:06.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.631+0000 7ff57af92740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557f6caf8660 0x557f6cb0aca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:06.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.631+0000 7ff57af92740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557f6cafdf30 0x557f6cb0b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:06.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.631+0000 7ff57af92740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f6cafe850 0x557f6cb0b720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:06.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.631+0000 7ff57af92740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff5701af3c0 con 0x557f6caf8660 2024-04-21T23:20:06.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.631+0000 7ff57af92740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff5701b0830 con 0x557f6cafdf30 2024-04-21T23:20:06.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.631+0000 7ff57af92740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff5701afa90 con 0x557f6cafe850 2024-04-21T23:20:06.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.631+0000 7ff57af92740 10 client.0 _ll_register_callbacks cb 0x7ff56802cde0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:20:06.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.631+0000 7ff576062640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557f6caf8660 0x557f6cb0aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:06.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.631+0000 7ff576062640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557f6caf8660 0x557f6cb0aca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37096/0 (socket says 172.21.15.40:37096) 2024-04-21T23:20:06.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.631+0000 7ff576062640 1 -- 172.21.15.40:0/3476464493 learned_addr learned my addr 172.21.15.40:0/3476464493 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:20:06.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.631+0000 7ff575861640 1 --2- 172.21.15.40:0/3476464493 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557f6cafdf30 0x557f6cb0b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:06.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.632+0000 7ff576062640 1 -- 172.21.15.40:0/3476464493 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f6cafe850 msgr2=0x557f6cb0b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:06.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.632+0000 7ff576062640 1 --2- 172.21.15.40:0/3476464493 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f6cafe850 0x557f6cb0b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:06.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.632+0000 7ff576062640 1 -- 172.21.15.40:0/3476464493 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557f6cafdf30 msgr2=0x557f6cb0b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:06.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.632+0000 7ff576062640 1 --2- 172.21.15.40:0/3476464493 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557f6cafdf30 0x557f6cb0b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:06.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.632+0000 7ff576062640 1 -- 172.21.15.40:0/3476464493 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f6cb7c7e0 con 0x557f6caf8660 2024-04-21T23:20:06.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.632+0000 7ff576062640 1 --2- 172.21.15.40:0/3476464493 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557f6caf8660 0x557f6cb0aca0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x557f6cb19eb0 tx=0x7ff56c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9d28555b118fcc8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:06.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.632+0000 7ff559ffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-21T23:20:06.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.632+0000 7ff559ffb640 1 -- 172.21.15.40:0/3476464493 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff56c01ea70 con 0x557f6caf8660 2024-04-21T23:20:06.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.632+0000 7ff559ffb640 1 -- 172.21.15.40:0/3476464493 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7ff56c01ec10 con 0x557f6caf8660 2024-04-21T23:20:06.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.632+0000 7ff57af92740 1 -- 172.21.15.40:0/3476464493 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x557f6c9e75a0 con 0x557f6caf8660 2024-04-21T23:20:06.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.632+0000 7ff559ffb640 1 -- 172.21.15.40:0/3476464493 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff56c037050 con 0x557f6caf8660 2024-04-21T23:20:06.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.632+0000 7ff57af92740 10 client.16101 fetch_fsmap learned FSMap version 28 2024-04-21T23:20:06.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.632+0000 7ff57af92740 1 -- 172.21.15.40:0/3476464493 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x557f6cb7daf0 con 0x557f6caf8660 2024-04-21T23:20:06.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.632+0000 7ff559ffb640 1 -- 172.21.15.40:0/3476464493 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x557f6c9e75a0 con 0x557f6caf8660 2024-04-21T23:20:06.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.632+0000 7ff559ffb640 1 -- 172.21.15.40:0/3476464493 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7ff56c00e4d0 con 0x557f6caf8660 2024-04-21T23:20:06.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.633+0000 7ff57af92740 10 client.16101 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T23:20:06.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.633+0000 7ff57af92740 10 client.16101 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:20:06.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.633+0000 7ff57af92740 20 client.16101 populate_metadata read hostname 'smithi040' 2024-04-21T23:20:06.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.633+0000 7ff57af92740 1 --2- 172.21.15.40:0/3476464493 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x557f6cb07430 0x557f6cb802c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:06.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.633+0000 7ff57af92740 4 client.16101 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T23:20:06.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.633+0000 7ff57af92740 1 -- 172.21.15.40:0/3476464493 --> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] -- command(tid 0: {"prefix": "scrub status"}) -- 0x557f6ca33c00 con 0x557f6cb07430 2024-04-21T23:20:06.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.633+0000 7ff575861640 1 --2- 172.21.15.40:0/3476464493 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x557f6cb07430 0x557f6cb802c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:06.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.633+0000 7ff575861640 1 --2- 172.21.15.40:0/3476464493 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x557f6cb07430 0x557f6cb802c0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=929f039ec0d3cae7 server_cookie=d4d033238e5d5d8b in_seq=0 out_seq=0 2024-04-21T23:20:06.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.633+0000 7ff559ffb640 10 client.16101 ms_handle_connect on v2:172.21.15.111:6838/3712235548 2024-04-21T23:20:06.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.633+0000 7ff559ffb640 1 -- 172.21.15.40:0/3476464493 <== mds.0 v2:172.21.15.111:6838/3712235548 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x557f6ca33c00 con 0x557f6cb07430 2024-04-21T23:20:06.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.633+0000 7ff559ffb640 10 client.16101 handle_command_reply: tid=0 2024-04-21T23:20:06.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.633+0000 7ff57af92740 1 client.16101 shutdown 2024-04-21T23:20:06.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.633+0000 7ff57af92740 1 -- 172.21.15.40:0/3476464493 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557f6caf8660 msgr2=0x557f6cb0aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:06.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.633+0000 7ff57af92740 1 --2- 172.21.15.40:0/3476464493 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557f6caf8660 0x557f6cb0aca0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x557f6cb19eb0 tx=0x7ff56c004730 comp rx=0 tx=0).stop 2024-04-21T23:20:06.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.633+0000 7ff57af92740 1 -- 172.21.15.40:0/3476464493 shutdown_connections 2024-04-21T23:20:06.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.633+0000 7ff57af92740 1 --2- 172.21.15.40:0/3476464493 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x557f6cb07430 0x557f6cb802c0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:06.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.633+0000 7ff57af92740 1 --2- 172.21.15.40:0/3476464493 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557f6cafdf30 0x557f6cb0b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:06.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.633+0000 7ff57af92740 1 --2- 172.21.15.40:0/3476464493 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f6cafe850 0x557f6cb0b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:06.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.633+0000 7ff57af92740 1 --2- 172.21.15.40:0/3476464493 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557f6caf8660 0x557f6cb0aca0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:06.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.633+0000 7ff559ffb640 0 client.16101 ms_handle_reset on v2:172.21.15.111:6838/3712235548 2024-04-21T23:20:06.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.633+0000 7ff57af92740 1 -- 172.21.15.40:0/3476464493 >> 172.21.15.40:0/3476464493 conn(0x557f6cb03050 msgr2=0x557f6caf76e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:06.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.634+0000 7ff57af92740 1 -- 172.21.15.40:0/3476464493 shutdown_connections 2024-04-21T23:20:06.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.634+0000 7ff57af92740 1 -- 172.21.15.40:0/3476464493 wait complete. 2024-04-21T23:20:06.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.634+0000 7ff57af92740 20 client.16101 trim_cache size 0 max 16384 2024-04-21T23:20:06.639 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-21T23:20:06.639 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "no active scrubs running", 2024-04-21T23:20:06.639 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": {} 2024-04-21T23:20:06.639 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-21T23:20:06.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.634+0000 7ff5782c6640 1 -- 172.21.15.40:0/52004761 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7ff55c0a2d80 msgr2=0x7ff55c0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:06.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.634+0000 7ff5782c6640 1 --2- 172.21.15.40:0/52004761 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7ff55c0a2d80 0x7ff55c0a5240 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff56c009b50 tx=0x7ff56c005f90 comp rx=0 tx=0).stop 2024-04-21T23:20:06.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.634+0000 7ff5782c6640 1 -- 172.21.15.40:0/52004761 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff570107720 msgr2=0x7ff5700744c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:06.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.634+0000 7ff5782c6640 1 --2- 172.21.15.40:0/52004761 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff570107720 0x7ff5700744c0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7ff568004af0 tx=0x7ff568032ec0 comp rx=0 tx=0).stop 2024-04-21T23:20:06.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.634+0000 7ff5782c6640 1 -- 172.21.15.40:0/52004761 shutdown_connections 2024-04-21T23:20:06.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.634+0000 7ff5782c6640 1 --2- 172.21.15.40:0/52004761 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7ff55c0a2d80 0x7ff55c0a5240 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:06.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.634+0000 7ff5782c6640 1 --2- 172.21.15.40:0/52004761 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff5701020d0 0x7ff570073f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:06.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.634+0000 7ff5782c6640 1 --2- 172.21.15.40:0/52004761 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff570107720 0x7ff5700744c0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:06.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.634+0000 7ff5782c6640 1 --2- 172.21.15.40:0/52004761 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff570101710 0x7ff570073a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:06.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.634+0000 7ff5782c6640 1 -- 172.21.15.40:0/52004761 >> 172.21.15.40:0/52004761 conn(0x7ff570078b90 msgr2=0x7ff5700fefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:06.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.634+0000 7ff5782c6640 1 -- 172.21.15.40:0/52004761 shutdown_connections 2024-04-21T23:20:06.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.634+0000 7ff5782c6640 1 -- 172.21.15.40:0/52004761 wait complete. 2024-04-21T23:20:06.644 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T23:20:06.644 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:20:06.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.754+0000 7f32b7095640 1 -- 172.21.15.40:0/1580741819 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32b010bd70 msgr2=0x7f32b010e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:06.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.754+0000 7f32b7095640 1 --2- 172.21.15.40:0/1580741819 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32b010bd70 0x7f32b010e1a0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f32a8009ca0 tx=0x7f32a802c180 comp rx=0 tx=0).stop 2024-04-21T23:20:06.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.754+0000 7f32b7095640 1 -- 172.21.15.40:0/1580741819 shutdown_connections 2024-04-21T23:20:06.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.754+0000 7f32b7095640 1 --2- 172.21.15.40:0/1580741819 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32b010bd70 0x7f32b010e1a0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:06.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.754+0000 7f32b7095640 1 --2- 172.21.15.40:0/1580741819 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f32b01032c0 0x7f32b010b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:06.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.754+0000 7f32b7095640 1 --2- 172.21.15.40:0/1580741819 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32b01029a0 0x7f32b0102d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:06.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.754+0000 7f32b7095640 1 -- 172.21.15.40:0/1580741819 >> 172.21.15.40:0/1580741819 conn(0x7f32b00769c0 msgr2=0x7f32b0076dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:06.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.754+0000 7f32b7095640 1 -- 172.21.15.40:0/1580741819 shutdown_connections 2024-04-21T23:20:06.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.754+0000 7f32b7095640 1 -- 172.21.15.40:0/1580741819 wait complete. 2024-04-21T23:20:06.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.755+0000 7f32b7095640 1 Processor -- start 2024-04-21T23:20:06.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.755+0000 7f32b7095640 1 -- start start 2024-04-21T23:20:06.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.755+0000 7f32b7095640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f32b01029a0 0x7f32b01005f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:06.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.755+0000 7f32b7095640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32b01032c0 0x7f32b0100b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:06.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.755+0000 7f32b7095640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32b010bd70 0x7f32b0101070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:06.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.755+0000 7f32b7095640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f32b0111b00 con 0x7f32b01032c0 2024-04-21T23:20:06.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.755+0000 7f32b7095640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f32b0111980 con 0x7f32b010bd70 2024-04-21T23:20:06.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.755+0000 7f32b7095640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f32b0111c80 con 0x7f32b01029a0 2024-04-21T23:20:06.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.755+0000 7f32affff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32b01032c0 0x7f32b0100b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:06.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.755+0000 7f32affff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32b01032c0 0x7f32b0100b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37116/0 (socket says 172.21.15.40:37116) 2024-04-21T23:20:06.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.755+0000 7f32affff640 1 -- 172.21.15.40:0/350943167 learned_addr learned my addr 172.21.15.40:0/350943167 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:20:06.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.755+0000 7f32b5632640 1 --2- 172.21.15.40:0/350943167 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32b010bd70 0x7f32b0101070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:06.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.756+0000 7f32b4e31640 1 --2- 172.21.15.40:0/350943167 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f32b01029a0 0x7f32b01005f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:06.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.756+0000 7f32affff640 1 -- 172.21.15.40:0/350943167 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f32b01029a0 msgr2=0x7f32b01005f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:06.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.756+0000 7f32affff640 1 --2- 172.21.15.40:0/350943167 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f32b01029a0 0x7f32b01005f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:06.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.756+0000 7f32affff640 1 -- 172.21.15.40:0/350943167 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32b010bd70 msgr2=0x7f32b0101070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:06.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.756+0000 7f32affff640 1 --2- 172.21.15.40:0/350943167 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32b010bd70 0x7f32b0101070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:06.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.756+0000 7f32affff640 1 -- 172.21.15.40:0/350943167 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32b01a63c0 con 0x7f32b01032c0 2024-04-21T23:20:06.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.756+0000 7f32b5632640 1 --2- 172.21.15.40:0/350943167 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32b010bd70 0x7f32b0101070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:20:06.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.756+0000 7f32affff640 1 --2- 172.21.15.40:0/350943167 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32b01032c0 0x7f32b0100b30 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f329800c920 tx=0x7f329800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:06.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.756+0000 7f32adffb640 1 -- 172.21.15.40:0/350943167 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3298007c70 con 0x7f32b01032c0 2024-04-21T23:20:06.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.756+0000 7f32b7095640 1 -- 172.21.15.40:0/350943167 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32b01a66b0 con 0x7f32b01032c0 2024-04-21T23:20:06.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.756+0000 7f32adffb640 1 -- 172.21.15.40:0/350943167 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3298007e10 con 0x7f32b01032c0 2024-04-21T23:20:06.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.756+0000 7f32adffb640 1 -- 172.21.15.40:0/350943167 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3298018890 con 0x7f32b01032c0 2024-04-21T23:20:06.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.756+0000 7f32b7095640 1 -- 172.21.15.40:0/350943167 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32b01a6c50 con 0x7f32b01032c0 2024-04-21T23:20:06.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.757+0000 7f32b7095640 1 -- 172.21.15.40:0/350943167 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32b006a5c0 con 0x7f32b01032c0 2024-04-21T23:20:06.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.758+0000 7f32adffb640 1 -- 172.21.15.40:0/350943167 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f3298018a30 con 0x7f32b01032c0 2024-04-21T23:20:06.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.759+0000 7f32adffb640 1 --2- 172.21.15.40:0/350943167 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f329c0a2d30 0x7f329c0a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:06.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.759+0000 7f32adffb640 1 -- 172.21.15.40:0/350943167 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f32980cf8b0 con 0x7f32b01032c0 2024-04-21T23:20:06.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.759+0000 7f32b4e31640 1 --2- 172.21.15.40:0/350943167 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f329c0a2d30 0x7f329c0a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:06.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.759+0000 7f32b4e31640 1 --2- 172.21.15.40:0/350943167 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f329c0a2d30 0x7f329c0a51f0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f32a00095f0 tx=0x7f32a0005f90 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:06.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.763+0000 7f32adffb640 1 -- 172.21.15.40:0/350943167 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f3298014030 con 0x7f32b01032c0 2024-04-21T23:20:06.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.990+0000 7f32b7095640 1 -- 172.21.15.40:0/350943167 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f32b01a6f10 con 0x7f32b01032c0 2024-04-21T23:20:06.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.991+0000 7f32adffb640 1 -- 172.21.15.40:0/350943167 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11888 (secure 0 0 0) 0x7f329809c120 con 0x7f32b01032c0 2024-04-21T23:20:06.990 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:20:06.990 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/1571948830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1571948830},{"type":"v1","addr":"172.21.15.40:6835","nonce":1571948830}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/969356642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":969356642},{"type":"v1","addr":"172.21.15.40:6837","nonce":969356642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6839/3025057845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":3025057845},{"type":"v1","addr":"172.21.15.40:6839","nonce":3025057845}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6841/3350488804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":3350488804},{"type":"v1","addr":"172.21.15.40:6841","nonce":3350488804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/3438493203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":3438493203},{"type":"v1","addr":"172.21.15.174:6833","nonce":3438493203}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/3105752678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":3105752678},{"type":"v1","addr":"172.21.15.111:6833","nonce":3105752678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T23:15:55.742233+0000","modified":"2024-04-21T23:19:58.773643+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24521,"mds_1":24502,"mds_2":24494},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24461":{"gid":24461,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6835/2704462092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2704462092},{"type":"v1","addr":"172.21.15.111:6835","nonce":2704462092}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/3905733045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3905733045},{"type":"v1","addr":"172.21.15.174:6835","nonce":3905733045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6837/465579746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":465579746},{"type":"v1","addr":"172.21.15.174:6837","nonce":465579746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/2703764182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":2703764182},{"type":"v1","addr":"172.21.15.111:6837","nonce":2703764182}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24502":{"gid":24502,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.174:6839/2467509274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":2467509274},{"type":"v1","addr":"172.21.15.174:6839","nonce":2467509274}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3712235548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3712235548},{"type":"v1","addr":"172.21.15.111:6839","nonce":3712235548}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24521,24494,24502]},"id":1}]} 2024-04-21T23:20:06.990 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 28 2024-04-21T23:20:06.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.993+0000 7f32b7095640 1 -- 172.21.15.40:0/350943167 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f329c0a2d30 msgr2=0x7f329c0a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:06.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.993+0000 7f32b7095640 1 --2- 172.21.15.40:0/350943167 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f329c0a2d30 0x7f329c0a51f0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f32a00095f0 tx=0x7f32a0005f90 comp rx=0 tx=0).stop 2024-04-21T23:20:06.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.993+0000 7f32b7095640 1 -- 172.21.15.40:0/350943167 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32b01032c0 msgr2=0x7f32b0100b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:06.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.993+0000 7f32b7095640 1 --2- 172.21.15.40:0/350943167 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32b01032c0 0x7f32b0100b30 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f329800c920 tx=0x7f329800cde0 comp rx=0 tx=0).stop 2024-04-21T23:20:06.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.993+0000 7f32b7095640 1 -- 172.21.15.40:0/350943167 shutdown_connections 2024-04-21T23:20:06.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.993+0000 7f32b7095640 1 --2- 172.21.15.40:0/350943167 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f329c0a2d30 0x7f329c0a51f0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:06.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.993+0000 7f32b7095640 1 --2- 172.21.15.40:0/350943167 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32b010bd70 0x7f32b0101070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:06.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.993+0000 7f32b7095640 1 --2- 172.21.15.40:0/350943167 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32b01032c0 0x7f32b0100b30 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:06.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.993+0000 7f32b7095640 1 --2- 172.21.15.40:0/350943167 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f32b01029a0 0x7f32b01005f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:06.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.993+0000 7f32b7095640 1 -- 172.21.15.40:0/350943167 >> 172.21.15.40:0/350943167 conn(0x7f32b00769c0 msgr2=0x7f32b00fd460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:06.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.993+0000 7f32b7095640 1 -- 172.21.15.40:0/350943167 shutdown_connections 2024-04-21T23:20:06.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:06.993+0000 7f32b7095640 1 -- 172.21.15.40:0/350943167 wait complete. 2024-04-21T23:20:07.003 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-21T23:20:07.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.110+0000 7fec62f02640 1 -- 172.21.15.40:0/3309044872 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec5c1020d0 msgr2=0x7fec5c10e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:07.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.110+0000 7fec62f02640 1 --2- 172.21.15.40:0/3309044872 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec5c1020d0 0x7fec5c10e5e0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fec54009cd0 tx=0x7fec5402c0f0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.110+0000 7fec62f02640 1 -- 172.21.15.40:0/3309044872 shutdown_connections 2024-04-21T23:20:07.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.110+0000 7fec62f02640 1 --2- 172.21.15.40:0/3309044872 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec5c101710 0x7fec5c101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.110+0000 7fec62f02640 1 --2- 172.21.15.40:0/3309044872 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec5c1020d0 0x7fec5c10e5e0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.110+0000 7fec62f02640 1 --2- 172.21.15.40:0/3309044872 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fec5c107720 0x7fec5c107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.110+0000 7fec62f02640 1 -- 172.21.15.40:0/3309044872 >> 172.21.15.40:0/3309044872 conn(0x7fec5c078b90 msgr2=0x7fec5c078fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:07.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.110+0000 7fec62f02640 1 -- 172.21.15.40:0/3309044872 shutdown_connections 2024-04-21T23:20:07.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.110+0000 7fec62f02640 1 -- 172.21.15.40:0/3309044872 wait complete. 2024-04-21T23:20:07.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.111+0000 7fec62f02640 1 Processor -- start 2024-04-21T23:20:07.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.111+0000 7fec62f02640 1 -- start start 2024-04-21T23:20:07.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.111+0000 7fec62f02640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec5c101710 0x7fec5c198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:07.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.111+0000 7fec62f02640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec5c1020d0 0x7fec5c198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:07.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.111+0000 7fec62f02640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fec5c107720 0x7fec5c19c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:07.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.111+0000 7fec62f02640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fec5c111b30 con 0x7fec5c1020d0 2024-04-21T23:20:07.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.111+0000 7fec62f02640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fec5c1119b0 con 0x7fec5c101710 2024-04-21T23:20:07.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.111+0000 7fec62f02640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fec5c111cb0 con 0x7fec5c107720 2024-04-21T23:20:07.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.111+0000 7fec5bfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec5c1020d0 0x7fec5c198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:07.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.111+0000 7fec5bfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec5c1020d0 0x7fec5c198570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37136/0 (socket says 172.21.15.40:37136) 2024-04-21T23:20:07.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.111+0000 7fec5bfff640 1 -- 172.21.15.40:0/2955249391 learned_addr learned my addr 172.21.15.40:0/2955249391 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:20:07.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.111+0000 7fec60c9e640 1 --2- 172.21.15.40:0/2955249391 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec5c101710 0x7fec5c198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:07.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.111+0000 7fec5bfff640 1 -- 172.21.15.40:0/2955249391 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fec5c107720 msgr2=0x7fec5c19c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:07.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.111+0000 7fec5bfff640 1 --2- 172.21.15.40:0/2955249391 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fec5c107720 0x7fec5c19c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.111+0000 7fec5bfff640 1 -- 172.21.15.40:0/2955249391 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec5c101710 msgr2=0x7fec5c198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:07.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.111+0000 7fec5bfff640 1 --2- 172.21.15.40:0/2955249391 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec5c101710 0x7fec5c198030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.111+0000 7fec5bfff640 1 -- 172.21.15.40:0/2955249391 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec5c19d020 con 0x7fec5c1020d0 2024-04-21T23:20:07.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.112+0000 7fec5bfff640 1 --2- 172.21.15.40:0/2955249391 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec5c1020d0 0x7fec5c198570 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fec4800e9f0 tx=0x7fec4800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:07.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.112+0000 7fec59ffb640 1 -- 172.21.15.40:0/2955249391 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec4800cd80 con 0x7fec5c1020d0 2024-04-21T23:20:07.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.112+0000 7fec59ffb640 1 -- 172.21.15.40:0/2955249391 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec48010910 con 0x7fec5c1020d0 2024-04-21T23:20:07.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.112+0000 7fec62f02640 1 -- 172.21.15.40:0/2955249391 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec5c19d310 con 0x7fec5c1020d0 2024-04-21T23:20:07.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.112+0000 7fec59ffb640 1 -- 172.21.15.40:0/2955249391 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec48018990 con 0x7fec5c1020d0 2024-04-21T23:20:07.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.112+0000 7fec62f02640 1 -- 172.21.15.40:0/2955249391 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec5c1a4ba0 con 0x7fec5c1020d0 2024-04-21T23:20:07.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.114+0000 7fec59ffb640 1 -- 172.21.15.40:0/2955249391 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fec48010ac0 con 0x7fec5c1020d0 2024-04-21T23:20:07.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.115+0000 7fec65bc8740 1 Processor -- start 2024-04-21T23:20:07.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.115+0000 7fec65bc8740 1 -- start start 2024-04-21T23:20:07.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.115+0000 7fec65bc8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf228eb020 0x55cf228eb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:07.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.115+0000 7fec65bc8740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cf228eb940 0x55cf228f3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:07.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.115+0000 7fec59ffb640 1 --2- 172.21.15.40:0/2955249391 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fec4c0a2d80 0x7fec4c0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:07.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.115+0000 7fec65bc8740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cf228f4520 0x55cf228f6910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:07.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.115+0000 7fec65bc8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55cf2284f530 con 0x55cf228eb020 2024-04-21T23:20:07.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.115+0000 7fec65bc8740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55cf227f7830 con 0x55cf228eb940 2024-04-21T23:20:07.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.115+0000 7fec65bc8740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55cf2284b4f0 con 0x55cf228f4520 2024-04-21T23:20:07.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.115+0000 7fec6149f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf228eb020 0x55cf228eb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:07.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.115+0000 7fec60c9e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cf228eb940 0x55cf228f3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:07.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.115+0000 7fec5bfff640 1 --2- 172.21.15.40:0/2955249391 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fec4c0a2d80 0x7fec4c0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:07.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.115+0000 7fec6149f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf228eb020 0x55cf228eb400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37140/0 (socket says 172.21.15.40:37140) 2024-04-21T23:20:07.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.115+0000 7fec60c9e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cf228eb940 0x55cf228f3fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.40:53684/0 (socket says 172.21.15.40:53684) 2024-04-21T23:20:07.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.115+0000 7fec6149f640 1 -- 172.21.15.40:0/3049756472 learned_addr learned my addr 172.21.15.40:0/3049756472 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:20:07.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.115+0000 7fec59ffb640 1 -- 172.21.15.40:0/2955249391 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7fec480cfb80 con 0x7fec5c1020d0 2024-04-21T23:20:07.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.115+0000 7fec6149f640 1 --2- 172.21.15.40:0/3049756472 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cf228f4520 0x55cf228f6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:07.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.115+0000 7fec60c9e640 1 -- 172.21.15.40:0/3049756472 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cf228f4520 msgr2=0x55cf228f6910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:07.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.115+0000 7fec60c9e640 1 --2- 172.21.15.40:0/3049756472 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cf228f4520 0x55cf228f6910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.115+0000 7fec60c9e640 1 -- 172.21.15.40:0/3049756472 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf228eb020 msgr2=0x55cf228eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:07.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.115+0000 7fec60c9e640 1 --2- 172.21.15.40:0/3049756472 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf228eb020 0x55cf228eb400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.115+0000 7fec60c9e640 1 -- 172.21.15.40:0/3049756472 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf2281b2c0 con 0x55cf228eb940 2024-04-21T23:20:07.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.116+0000 7fec5bfff640 1 --2- 172.21.15.40:0/2955249391 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fec4c0a2d80 0x7fec4c0a5240 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fec48098660 tx=0x7fec48018420 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:07.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.116+0000 7fec60c9e640 1 --2- 172.21.15.40:0/3049756472 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cf228eb940 0x55cf228f3fe0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fec500045b0 tx=0x7fec5000bb20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c9533d15ca5507b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:07.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.116+0000 7fec46ffd640 1 -- 172.21.15.40:0/3049756472 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec50003b50 con 0x55cf228eb940 2024-04-21T23:20:07.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.116+0000 7fec46ffd640 1 -- 172.21.15.40:0/3049756472 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec50015070 con 0x55cf228eb940 2024-04-21T23:20:07.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.116+0000 7fec46ffd640 1 -- 172.21.15.40:0/3049756472 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec50004d00 con 0x55cf228eb940 2024-04-21T23:20:07.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.116+0000 7fec65bc8740 1 -- 172.21.15.40:0/3049756472 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cf228eb940 msgr2=0x55cf228f3fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:07.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.116+0000 7fec65bc8740 1 --2- 172.21.15.40:0/3049756472 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cf228eb940 0x55cf228f3fe0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fec500045b0 tx=0x7fec5000bb20 comp rx=0 tx=0).stop 2024-04-21T23:20:07.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.116+0000 7fec65bc8740 1 -- 172.21.15.40:0/3049756472 shutdown_connections 2024-04-21T23:20:07.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.116+0000 7fec65bc8740 1 --2- 172.21.15.40:0/3049756472 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cf228eb940 0x55cf228f3fe0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.116+0000 7fec65bc8740 1 --2- 172.21.15.40:0/3049756472 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cf228f4520 0x55cf228f6910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.117+0000 7fec65bc8740 1 --2- 172.21.15.40:0/3049756472 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf228eb020 0x55cf228eb400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.117+0000 7fec65bc8740 1 -- 172.21.15.40:0/3049756472 >> 172.21.15.40:0/3049756472 conn(0x55cf228e5750 msgr2=0x55cf228e5b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:07.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.117+0000 7fec65bc8740 1 -- 172.21.15.40:0/3049756472 shutdown_connections 2024-04-21T23:20:07.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.117+0000 7fec65bc8740 1 -- 172.21.15.40:0/3049756472 wait complete. 2024-04-21T23:20:07.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.117+0000 7fec65bc8740 1 -- start start 2024-04-21T23:20:07.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.117+0000 7fec65bc8740 1 Processor -- start 2024-04-21T23:20:07.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.117+0000 7fec65bc8740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cf228eb020 0x55cf228f3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:07.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.117+0000 7fec65bc8740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cf228eb940 0x55cf228f1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:07.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.117+0000 7fec65bc8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf228f4520 0x55cf228f17b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:07.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.117+0000 7fec65bc8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55cf228f7310 con 0x55cf228f4520 2024-04-21T23:20:07.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.117+0000 7fec65bc8740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55cf2280af10 con 0x55cf228eb020 2024-04-21T23:20:07.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.117+0000 7fec65bc8740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55cf228f7490 con 0x55cf228eb940 2024-04-21T23:20:07.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.117+0000 7fec65bc8740 10 client.0 _ll_register_callbacks cb 0x55cf22562be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:20:07.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.117+0000 7fec6149f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf228f4520 0x55cf228f17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:07.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.117+0000 7fec6149f640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cf228eb020 0x55cf228f3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:07.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.117+0000 7fec6149f640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cf228eb020 0x55cf228f3ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.40:53688/0 (socket says 172.21.15.40:53688) 2024-04-21T23:20:07.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.117+0000 7fec6149f640 1 -- 172.21.15.40:0/3333458716 learned_addr learned my addr 172.21.15.40:0/3333458716 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:20:07.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.118+0000 7fec60c9e640 1 --2- 172.21.15.40:0/3333458716 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cf228eb940 0x55cf228f1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:07.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.118+0000 7fec6149f640 1 -- 172.21.15.40:0/3333458716 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cf228eb940 msgr2=0x55cf228f1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:07.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.118+0000 7fec6149f640 1 --2- 172.21.15.40:0/3333458716 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cf228eb940 0x55cf228f1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.118+0000 7fec6149f640 1 -- 172.21.15.40:0/3333458716 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf228f4520 msgr2=0x55cf228f17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:07.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.118+0000 7fec6149f640 1 --2- 172.21.15.40:0/3333458716 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf228f4520 0x55cf228f17b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.118+0000 7fec6149f640 1 -- 172.21.15.40:0/3333458716 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf227dcc80 con 0x55cf228eb020 2024-04-21T23:20:07.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.118+0000 7fec60c9e640 1 --2- 172.21.15.40:0/3333458716 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cf228eb940 0x55cf228f1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:20:07.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.118+0000 7fec6149f640 1 --2- 172.21.15.40:0/3333458716 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cf228eb020 0x55cf228f3ea0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fec54032530 tx=0x7fec54005dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b0df9d1d8e0f9b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:07.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.118+0000 7fec3bfff640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-21T23:20:07.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.118+0000 7fec3bfff640 1 -- 172.21.15.40:0/3333458716 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec5404b060 con 0x55cf228eb020 2024-04-21T23:20:07.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.118+0000 7fec65bc8740 1 -- 172.21.15.40:0/3333458716 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fec50015070 con 0x55cf228eb020 2024-04-21T23:20:07.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.118+0000 7fec3bfff640 1 -- 172.21.15.40:0/3333458716 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7fec54034640 con 0x55cf228eb020 2024-04-21T23:20:07.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.118+0000 7fec3bfff640 1 -- 172.21.15.40:0/3333458716 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fec54052050 con 0x55cf228eb020 2024-04-21T23:20:07.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.118+0000 7fec65bc8740 10 client.25637 fetch_fsmap learned FSMap version 28 2024-04-21T23:20:07.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.118+0000 7fec65bc8740 1 -- 172.21.15.40:0/3333458716 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55cf225633f0 con 0x55cf228eb020 2024-04-21T23:20:07.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.118+0000 7fec3bfff640 1 -- 172.21.15.40:0/3333458716 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec50015070 con 0x55cf228eb020 2024-04-21T23:20:07.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.119+0000 7fec3bfff640 1 -- 172.21.15.40:0/3333458716 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fec54044860 con 0x55cf228eb020 2024-04-21T23:20:07.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.119+0000 7fec65bc8740 10 client.25637 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T23:20:07.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.119+0000 7fec65bc8740 10 client.25637 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-21T23:20:07.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.119+0000 7fec65bc8740 20 client.25637 populate_metadata read hostname 'smithi040' 2024-04-21T23:20:07.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.119+0000 7fec65bc8740 1 --2- 172.21.15.40:0/3333458716 >> [v2:172.21.15.111:6836/2703764182,v1:172.21.15.111:6837/2703764182] conn(0x55cf22904610 0x55cf22906a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:07.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.119+0000 7fec65bc8740 4 client.25637 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:20:07.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.119+0000 7fec65bc8740 1 -- 172.21.15.40:0/3333458716 --> [v2:172.21.15.111:6836/2703764182,v1:172.21.15.111:6837/2703764182] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cf2285ca70 con 0x55cf22904610 2024-04-21T23:20:07.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.119+0000 7fec60c9e640 1 --2- 172.21.15.40:0/3333458716 >> [v2:172.21.15.111:6836/2703764182,v1:172.21.15.111:6837/2703764182] conn(0x55cf22904610 0x55cf22906a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:07.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.119+0000 7fec60c9e640 1 --2- 172.21.15.40:0/3333458716 >> [v2:172.21.15.111:6836/2703764182,v1:172.21.15.111:6837/2703764182] conn(0x55cf22904610 0x55cf22906a20 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e41d661e7bc4dc71 server_cookie=5add545586238504 in_seq=0 out_seq=0 2024-04-21T23:20:07.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.119+0000 7fec3bfff640 10 client.25637 ms_handle_connect on v2:172.21.15.111:6836/2703764182 2024-04-21T23:20:07.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.123+0000 7fec3bfff640 1 -- 172.21.15.40:0/3333458716 <== mds.2 v2:172.21.15.111:6836/2703764182 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cf2285ca70 con 0x55cf22904610 2024-04-21T23:20:07.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.123+0000 7fec3bfff640 10 client.25637 handle_command_reply: tid=0 2024-04-21T23:20:07.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.123+0000 7fec65bc8740 1 client.25637 shutdown 2024-04-21T23:20:07.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.123+0000 7fec65bc8740 1 -- 172.21.15.40:0/3333458716 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cf228eb020 msgr2=0x55cf228f3ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:07.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.123+0000 7fec65bc8740 1 --2- 172.21.15.40:0/3333458716 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cf228eb020 0x55cf228f3ea0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fec54032530 tx=0x7fec54005dc0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.123+0000 7fec65bc8740 1 -- 172.21.15.40:0/3333458716 shutdown_connections 2024-04-21T23:20:07.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.123+0000 7fec65bc8740 1 --2- 172.21.15.40:0/3333458716 >> [v2:172.21.15.111:6836/2703764182,v1:172.21.15.111:6837/2703764182] conn(0x55cf22904610 0x55cf22906a20 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.123+0000 7fec65bc8740 1 --2- 172.21.15.40:0/3333458716 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf228f4520 0x55cf228f17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.123+0000 7fec65bc8740 1 --2- 172.21.15.40:0/3333458716 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cf228eb940 0x55cf228f1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.123+0000 7fec65bc8740 1 --2- 172.21.15.40:0/3333458716 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cf228eb020 0x55cf228f3ea0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.123+0000 7fec3bfff640 0 client.25637 ms_handle_reset on v2:172.21.15.111:6836/2703764182 2024-04-21T23:20:07.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.123+0000 7fec65bc8740 1 -- 172.21.15.40:0/3333458716 >> 172.21.15.40:0/3333458716 conn(0x55cf228e5750 msgr2=0x55cf228e74a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:07.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.123+0000 7fec65bc8740 1 -- 172.21.15.40:0/3333458716 shutdown_connections 2024-04-21T23:20:07.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.123+0000 7fec65bc8740 1 -- 172.21.15.40:0/3333458716 wait complete. 2024-04-21T23:20:07.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.123+0000 7fec65bc8740 20 client.25637 trim_cache size 0 max 16384 2024-04-21T23:20:07.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.136+0000 7fec65bc8740 1 Processor -- start 2024-04-21T23:20:07.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.136+0000 7fec65bc8740 1 -- start start 2024-04-21T23:20:07.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.137+0000 7fec65bc8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf228eb020 0x55cf228e7410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:07.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.137+0000 7fec65bc8740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cf228eb940 0x55cf228e46d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:07.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.137+0000 7fec65bc8740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cf228f4520 0x55cf228f3960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:07.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.137+0000 7fec65bc8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fec5c1b1640 con 0x55cf228eb020 2024-04-21T23:20:07.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.137+0000 7fec65bc8740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fec5c1aeeb0 con 0x55cf228f4520 2024-04-21T23:20:07.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.137+0000 7fec65bc8740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fec5c1acd00 con 0x55cf228eb940 2024-04-21T23:20:07.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.137+0000 7fec6149f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf228eb020 0x55cf228e7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:07.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.137+0000 7fec6149f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf228eb020 0x55cf228e7410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37162/0 (socket says 172.21.15.40:37162) 2024-04-21T23:20:07.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.137+0000 7fec6149f640 1 -- 172.21.15.40:0/224627821 learned_addr learned my addr 172.21.15.40:0/224627821 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:20:07.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.137+0000 7fec6149f640 1 --2- 172.21.15.40:0/224627821 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cf228f4520 0x55cf228f3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:07.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.137+0000 7fec6149f640 1 -- 172.21.15.40:0/224627821 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cf228eb940 msgr2=0x55cf228e46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:07.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.137+0000 7fec6149f640 1 --2- 172.21.15.40:0/224627821 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cf228eb940 0x55cf228e46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.137+0000 7fec6149f640 1 -- 172.21.15.40:0/224627821 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cf228f4520 msgr2=0x55cf228f3960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:07.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.137+0000 7fec6149f640 1 --2- 172.21.15.40:0/224627821 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cf228f4520 0x55cf228f3960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.137+0000 7fec6149f640 1 -- 172.21.15.40:0/224627821 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf228eb770 con 0x55cf228eb020 2024-04-21T23:20:07.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.137+0000 7fec6149f640 1 --2- 172.21.15.40:0/224627821 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf228eb020 0x55cf228e7410 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fec54004aa0 tx=0x7fec54009800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad0313a195f20fd7 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:07.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.137+0000 7fec467fc640 1 -- 172.21.15.40:0/224627821 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec5405c070 con 0x55cf228eb020 2024-04-21T23:20:07.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.137+0000 7fec467fc640 1 -- 172.21.15.40:0/224627821 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec5402c770 con 0x55cf228eb020 2024-04-21T23:20:07.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.137+0000 7fec467fc640 1 -- 172.21.15.40:0/224627821 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec5404b030 con 0x55cf228eb020 2024-04-21T23:20:07.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.138+0000 7fec65bc8740 1 -- 172.21.15.40:0/224627821 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf228eb020 msgr2=0x55cf228e7410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:07.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.138+0000 7fec65bc8740 1 --2- 172.21.15.40:0/224627821 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf228eb020 0x55cf228e7410 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fec54004aa0 tx=0x7fec54009800 comp rx=0 tx=0).stop 2024-04-21T23:20:07.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.138+0000 7fec65bc8740 1 -- 172.21.15.40:0/224627821 shutdown_connections 2024-04-21T23:20:07.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.138+0000 7fec65bc8740 1 --2- 172.21.15.40:0/224627821 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cf228f4520 0x55cf228f3960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.138+0000 7fec65bc8740 1 --2- 172.21.15.40:0/224627821 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cf228eb940 0x55cf228e46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.138+0000 7fec65bc8740 1 --2- 172.21.15.40:0/224627821 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf228eb020 0x55cf228e7410 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.138+0000 7fec65bc8740 1 -- 172.21.15.40:0/224627821 >> 172.21.15.40:0/224627821 conn(0x55cf228e5750 msgr2=0x55cf228e3fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:07.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.138+0000 7fec65bc8740 1 -- 172.21.15.40:0/224627821 shutdown_connections 2024-04-21T23:20:07.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.138+0000 7fec65bc8740 1 -- 172.21.15.40:0/224627821 wait complete. 2024-04-21T23:20:07.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.138+0000 7fec65bc8740 1 -- start start 2024-04-21T23:20:07.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.138+0000 7fec65bc8740 1 Processor -- start 2024-04-21T23:20:07.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.139+0000 7fec65bc8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf228eb020 0x55cf228ee7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:07.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.139+0000 7fec65bc8740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cf228eb940 0x55cf228eece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:07.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.139+0000 7fec65bc8740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cf228f4520 0x55cf228ed4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:07.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.139+0000 7fec65bc8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fec5c1ad370 con 0x55cf228eb020 2024-04-21T23:20:07.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.139+0000 7fec65bc8740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fec5c1ae7e0 con 0x55cf228eb940 2024-04-21T23:20:07.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.139+0000 7fec65bc8740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fec5c1ada40 con 0x55cf228f4520 2024-04-21T23:20:07.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.139+0000 7fec65bc8740 10 client.0 _ll_register_callbacks cb 0x7fec48005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:20:07.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.139+0000 7fec6149f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf228eb020 0x55cf228ee7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:07.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.139+0000 7fec6149f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf228eb020 0x55cf228ee7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37172/0 (socket says 172.21.15.40:37172) 2024-04-21T23:20:07.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.139+0000 7fec6149f640 1 -- 172.21.15.40:0/3490426733 learned_addr learned my addr 172.21.15.40:0/3490426733 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:20:07.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.139+0000 7fec60c9e640 1 --2- 172.21.15.40:0/3490426733 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cf228eb940 0x55cf228eece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:07.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.139+0000 7fec6149f640 1 --2- 172.21.15.40:0/3490426733 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cf228f4520 0x55cf228ed4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:07.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.139+0000 7fec60c9e640 1 -- 172.21.15.40:0/3490426733 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cf228f4520 msgr2=0x55cf228ed4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:07.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.139+0000 7fec60c9e640 1 --2- 172.21.15.40:0/3490426733 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cf228f4520 0x55cf228ed4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.139+0000 7fec60c9e640 1 -- 172.21.15.40:0/3490426733 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf228eb020 msgr2=0x55cf228ee7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:07.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.139+0000 7fec60c9e640 1 --2- 172.21.15.40:0/3490426733 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf228eb020 0x55cf228ee7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.139+0000 7fec60c9e640 1 -- 172.21.15.40:0/3490426733 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf2262d7d0 con 0x55cf228eb940 2024-04-21T23:20:07.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.139+0000 7fec6149f640 1 --2- 172.21.15.40:0/3490426733 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf228eb020 0x55cf228ee7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:20:07.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.139+0000 7fec60c9e640 1 --2- 172.21.15.40:0/3490426733 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cf228eb940 0x55cf228eece0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fec50006010 tx=0x7fec5000bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6bd19b936d676d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:07.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.139+0000 7fec3bfff640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-21T23:20:07.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.140+0000 7fec3bfff640 1 -- 172.21.15.40:0/3490426733 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec50003ea0 con 0x55cf228eb940 2024-04-21T23:20:07.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.140+0000 7fec65bc8740 1 -- 172.21.15.40:0/3490426733 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fec5402c770 con 0x55cf228eb940 2024-04-21T23:20:07.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.140+0000 7fec3bfff640 1 -- 172.21.15.40:0/3490426733 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7fec50005020 con 0x55cf228eb940 2024-04-21T23:20:07.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.140+0000 7fec3bfff640 1 -- 172.21.15.40:0/3490426733 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fec50005220 con 0x55cf228eb940 2024-04-21T23:20:07.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.140+0000 7fec65bc8740 10 client.25643 fetch_fsmap learned FSMap version 28 2024-04-21T23:20:07.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.140+0000 7fec65bc8740 1 -- 172.21.15.40:0/3490426733 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55cf2296f3c0 con 0x55cf228eb940 2024-04-21T23:20:07.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.140+0000 7fec3bfff640 1 -- 172.21.15.40:0/3490426733 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec5402c770 con 0x55cf228eb940 2024-04-21T23:20:07.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.140+0000 7fec3bfff640 1 -- 172.21.15.40:0/3490426733 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fec50029170 con 0x55cf228eb940 2024-04-21T23:20:07.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.140+0000 7fec65bc8740 10 client.25643 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T23:20:07.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.140+0000 7fec65bc8740 10 client.25643 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-21T23:20:07.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.140+0000 7fec65bc8740 20 client.25643 populate_metadata read hostname 'smithi040' 2024-04-21T23:20:07.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.140+0000 7fec65bc8740 1 --2- 172.21.15.40:0/3490426733 >> [v2:172.21.15.111:6836/2703764182,v1:172.21.15.111:6837/2703764182] conn(0x55cf2296fa00 0x55cf22971e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:07.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.140+0000 7fec65bc8740 4 client.25643 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T23:20:07.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.140+0000 7fec65bc8740 1 -- 172.21.15.40:0/3490426733 --> [v2:172.21.15.111:6836/2703764182,v1:172.21.15.111:6837/2703764182] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55cf22856dc0 con 0x55cf2296fa00 2024-04-21T23:20:07.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.140+0000 7fec6149f640 1 --2- 172.21.15.40:0/3490426733 >> [v2:172.21.15.111:6836/2703764182,v1:172.21.15.111:6837/2703764182] conn(0x55cf2296fa00 0x55cf22971e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:07.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.141+0000 7fec3bfff640 10 client.25643 ms_handle_connect on v2:172.21.15.111:6836/2703764182 2024-04-21T23:20:07.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.141+0000 7fec6149f640 1 --2- 172.21.15.40:0/3490426733 >> [v2:172.21.15.111:6836/2703764182,v1:172.21.15.111:6837/2703764182] conn(0x55cf2296fa00 0x55cf22971e20 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=bee5386f7a262fbb server_cookie=eed7c0b571fe5c18 in_seq=0 out_seq=0 2024-04-21T23:20:07.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.145+0000 7fec3bfff640 1 -- 172.21.15.40:0/3490426733 <== mds.2 v2:172.21.15.111:6836/2703764182 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55cf22856dc0 con 0x55cf2296fa00 2024-04-21T23:20:07.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.145+0000 7fec3bfff640 10 client.25643 handle_command_reply: tid=0 2024-04-21T23:20:07.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.145+0000 7fec65bc8740 1 client.25643 shutdown 2024-04-21T23:20:07.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.145+0000 7fec65bc8740 1 -- 172.21.15.40:0/3490426733 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cf228eb940 msgr2=0x55cf228eece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:07.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.145+0000 7fec65bc8740 1 --2- 172.21.15.40:0/3490426733 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cf228eb940 0x55cf228eece0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fec50006010 tx=0x7fec5000bed0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.145+0000 7fec65bc8740 1 -- 172.21.15.40:0/3490426733 shutdown_connections 2024-04-21T23:20:07.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.145+0000 7fec65bc8740 1 --2- 172.21.15.40:0/3490426733 >> [v2:172.21.15.111:6836/2703764182,v1:172.21.15.111:6837/2703764182] conn(0x55cf2296fa00 0x55cf22971e20 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.145+0000 7fec65bc8740 1 --2- 172.21.15.40:0/3490426733 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cf228eb940 0x55cf228eece0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.145+0000 7fec65bc8740 1 --2- 172.21.15.40:0/3490426733 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cf228f4520 0x55cf228ed4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.145+0000 7fec65bc8740 1 --2- 172.21.15.40:0/3490426733 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf228eb020 0x55cf228ee7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.145+0000 7fec3bfff640 0 client.25643 ms_handle_reset on v2:172.21.15.111:6836/2703764182 2024-04-21T23:20:07.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.145+0000 7fec65bc8740 1 -- 172.21.15.40:0/3490426733 >> 172.21.15.40:0/3490426733 conn(0x55cf228e5750 msgr2=0x55cf228e4850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:07.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.146+0000 7fec65bc8740 1 -- 172.21.15.40:0/3490426733 shutdown_connections 2024-04-21T23:20:07.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.146+0000 7fec65bc8740 1 -- 172.21.15.40:0/3490426733 wait complete. 2024-04-21T23:20:07.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.146+0000 7fec65bc8740 20 client.25643 trim_cache size 0 max 16384 2024-04-21T23:20:07.149 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-04-21T23:20:07.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.146+0000 7fec62f02640 1 -- 172.21.15.40:0/2955249391 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fec4c0a2d80 msgr2=0x7fec4c0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:07.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.146+0000 7fec62f02640 1 --2- 172.21.15.40:0/2955249391 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fec4c0a2d80 0x7fec4c0a5240 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fec48098660 tx=0x7fec48018420 comp rx=0 tx=0).stop 2024-04-21T23:20:07.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.146+0000 7fec62f02640 1 -- 172.21.15.40:0/2955249391 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec5c1020d0 msgr2=0x7fec5c198570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:07.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.146+0000 7fec62f02640 1 --2- 172.21.15.40:0/2955249391 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec5c1020d0 0x7fec5c198570 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fec4800e9f0 tx=0x7fec4800eeb0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.146+0000 7fec62f02640 1 -- 172.21.15.40:0/2955249391 shutdown_connections 2024-04-21T23:20:07.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.146+0000 7fec62f02640 1 --2- 172.21.15.40:0/2955249391 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fec4c0a2d80 0x7fec4c0a5240 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.146+0000 7fec62f02640 1 --2- 172.21.15.40:0/2955249391 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fec5c107720 0x7fec5c19c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.146+0000 7fec62f02640 1 --2- 172.21.15.40:0/2955249391 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec5c1020d0 0x7fec5c198570 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.146+0000 7fec62f02640 1 --2- 172.21.15.40:0/2955249391 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec5c101710 0x7fec5c198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.146+0000 7fec62f02640 1 -- 172.21.15.40:0/2955249391 >> 172.21.15.40:0/2955249391 conn(0x7fec5c078b90 msgr2=0x7fec5c0ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:07.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.146+0000 7fec62f02640 1 -- 172.21.15.40:0/2955249391 shutdown_connections 2024-04-21T23:20:07.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.146+0000 7fec62f02640 1 -- 172.21.15.40:0/2955249391 wait complete. 2024-04-21T23:20:07.156 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-21T23:20:07.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:06 smithi174 ceph-mon[31409]: pgmap v54: 129 pgs: 129 active+clean; 34 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.9 MiB/s wr, 322 op/s 2024-04-21T23:20:07.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:06 smithi111 ceph-mon[31085]: pgmap v54: 129 pgs: 129 active+clean; 34 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.9 MiB/s wr, 322 op/s 2024-04-21T23:20:07.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.262+0000 7f3d96575640 1 -- 172.21.15.40:0/2836557649 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d901029c0 msgr2=0x7f3d90102da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:07.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.262+0000 7f3d96575640 1 --2- 172.21.15.40:0/2836557649 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d901029c0 0x7f3d90102da0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f3d78009900 tx=0x7f3d7802bd60 comp rx=0 tx=0).stop 2024-04-21T23:20:07.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.262+0000 7f3d96575640 1 -- 172.21.15.40:0/2836557649 shutdown_connections 2024-04-21T23:20:07.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.262+0000 7f3d96575640 1 --2- 172.21.15.40:0/2836557649 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d901032e0 0x7f3d9010b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.262+0000 7f3d96575640 1 --2- 172.21.15.40:0/2836557649 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3d9010bd00 0x7f3d9010e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.262+0000 7f3d96575640 1 --2- 172.21.15.40:0/2836557649 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d901029c0 0x7f3d90102da0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.262+0000 7f3d96575640 1 -- 172.21.15.40:0/2836557649 >> 172.21.15.40:0/2836557649 conn(0x7f3d90077740 msgr2=0x7f3d90077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:07.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.262+0000 7f3d96575640 1 -- 172.21.15.40:0/2836557649 shutdown_connections 2024-04-21T23:20:07.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.262+0000 7f3d96575640 1 -- 172.21.15.40:0/2836557649 wait complete. 2024-04-21T23:20:07.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.262+0000 7f3d96575640 1 Processor -- start 2024-04-21T23:20:07.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.262+0000 7f3d96575640 1 -- start start 2024-04-21T23:20:07.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.263+0000 7f3d96575640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d901029c0 0x7f3d90198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:07.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.263+0000 7f3d96575640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d901032e0 0x7f3d901985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:07.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.263+0000 7f3d96575640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3d9010bd00 0x7f3d9019c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:07.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.263+0000 7f3d96575640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3d90111b50 con 0x7f3d901029c0 2024-04-21T23:20:07.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.263+0000 7f3d96575640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3d901119d0 con 0x7f3d901032e0 2024-04-21T23:20:07.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.263+0000 7f3d96575640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3d90111cd0 con 0x7f3d9010bd00 2024-04-21T23:20:07.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.263+0000 7f3d8ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d901029c0 0x7f3d90198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:07.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.263+0000 7f3d8ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d901029c0 0x7f3d90198070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37188/0 (socket says 172.21.15.40:37188) 2024-04-21T23:20:07.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.263+0000 7f3d8ffff640 1 -- 172.21.15.40:0/4264413824 learned_addr learned my addr 172.21.15.40:0/4264413824 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:20:07.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.263+0000 7f3d8f7fe640 1 --2- 172.21.15.40:0/4264413824 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d901032e0 0x7f3d901985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:07.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.263+0000 7f3d94b12640 1 --2- 172.21.15.40:0/4264413824 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3d9010bd00 0x7f3d9019c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:07.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.263+0000 7f3d8ffff640 1 -- 172.21.15.40:0/4264413824 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3d9010bd00 msgr2=0x7f3d9019c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:07.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.263+0000 7f3d8ffff640 1 --2- 172.21.15.40:0/4264413824 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3d9010bd00 0x7f3d9019c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.263+0000 7f3d8ffff640 1 -- 172.21.15.40:0/4264413824 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d901032e0 msgr2=0x7f3d901985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:07.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.263+0000 7f3d8ffff640 1 --2- 172.21.15.40:0/4264413824 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d901032e0 0x7f3d901985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.263+0000 7f3d8ffff640 1 -- 172.21.15.40:0/4264413824 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d9019d010 con 0x7f3d901029c0 2024-04-21T23:20:07.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.263+0000 7f3d8f7fe640 1 --2- 172.21.15.40:0/4264413824 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d901032e0 0x7f3d901985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:20:07.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.263+0000 7f3d8ffff640 1 --2- 172.21.15.40:0/4264413824 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d901029c0 0x7f3d90198070 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f3d78009a20 tx=0x7f3d7802dd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:07.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.264+0000 7f3d8d7fa640 1 -- 172.21.15.40:0/4264413824 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d78043020 con 0x7f3d901029c0 2024-04-21T23:20:07.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.264+0000 7f3d8d7fa640 1 -- 172.21.15.40:0/4264413824 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d780054e0 con 0x7f3d901029c0 2024-04-21T23:20:07.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.264+0000 7f3d96575640 1 -- 172.21.15.40:0/4264413824 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d9019d2a0 con 0x7f3d901029c0 2024-04-21T23:20:07.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.264+0000 7f3d8d7fa640 1 -- 172.21.15.40:0/4264413824 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d78003bd0 con 0x7f3d901029c0 2024-04-21T23:20:07.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.264+0000 7f3d96575640 1 -- 172.21.15.40:0/4264413824 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d901a4b90 con 0x7f3d901029c0 2024-04-21T23:20:07.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.266+0000 7f3d8d7fa640 1 -- 172.21.15.40:0/4264413824 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f3d78046020 con 0x7f3d901029c0 2024-04-21T23:20:07.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.266+0000 7f3d99233740 1 Processor -- start 2024-04-21T23:20:07.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.266+0000 7f3d99233740 1 -- start start 2024-04-21T23:20:07.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.266+0000 7f3d8d7fa640 1 --2- 172.21.15.40:0/4264413824 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f3d7c0a2d00 0x7f3d7c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:07.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.267+0000 7f3d99233740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c384a22020 0x55c384a22400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:07.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.267+0000 7f3d99233740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c384a22940 0x55c384a2afe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:07.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.267+0000 7f3d99233740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c384a2b520 0x55c384a2d910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:07.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.267+0000 7f3d99233740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55c384986530 con 0x55c384a22020 2024-04-21T23:20:07.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.267+0000 7f3d99233740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55c38492e830 con 0x55c384a2b520 2024-04-21T23:20:07.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.267+0000 7f3d94b12640 1 --2- 172.21.15.40:0/4264413824 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f3d7c0a2d00 0x7f3d7c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:07.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.267+0000 7f3d99233740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55c3849824f0 con 0x55c384a22940 2024-04-21T23:20:07.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.267+0000 7f3d8d7fa640 1 -- 172.21.15.40:0/4264413824 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f3d780f17c0 con 0x7f3d901029c0 2024-04-21T23:20:07.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.267+0000 7f3d8ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c384a22020 0x55c384a22400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:07.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.267+0000 7f3d8ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c384a22020 0x55c384a22400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37200/0 (socket says 172.21.15.40:37200) 2024-04-21T23:20:07.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.267+0000 7f3d8ffff640 1 -- 172.21.15.40:0/3725483863 learned_addr learned my addr 172.21.15.40:0/3725483863 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:20:07.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.267+0000 7f3d94b12640 1 --2- 172.21.15.40:0/3725483863 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c384a2b520 0x55c384a2d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:07.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.267+0000 7f3d8f7fe640 1 --2- 172.21.15.40:0/3725483863 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c384a22940 0x55c384a2afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:07.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.267+0000 7f3d8ffff640 1 -- 172.21.15.40:0/3725483863 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c384a22940 msgr2=0x55c384a2afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:07.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.267+0000 7f3d8ffff640 1 --2- 172.21.15.40:0/3725483863 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c384a22940 0x55c384a2afe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.267+0000 7f3d8ffff640 1 -- 172.21.15.40:0/3725483863 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c384a2b520 msgr2=0x55c384a2d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:07.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.267+0000 7f3d8ffff640 1 --2- 172.21.15.40:0/3725483863 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c384a2b520 0x55c384a2d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.267+0000 7f3d8ffff640 1 -- 172.21.15.40:0/3725483863 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c3849522c0 con 0x55c384a22020 2024-04-21T23:20:07.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.267+0000 7f3d94b12640 1 --2- 172.21.15.40:0/4264413824 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f3d7c0a2d00 0x7f3d7c0a51c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3d88005fd0 tx=0x7f3d88014b70 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:07.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.267+0000 7f3d8ffff640 1 --2- 172.21.15.40:0/3725483863 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c384a22020 0x55c384a22400 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f3d780bd740 tx=0x7f3d780bd8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bbcb1bff381f5c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:07.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.267+0000 7f3d867fc640 1 -- 172.21.15.40:0/3725483863 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d78005690 con 0x55c384a22020 2024-04-21T23:20:07.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.267+0000 7f3d867fc640 1 -- 172.21.15.40:0/3725483863 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d78004c80 con 0x55c384a22020 2024-04-21T23:20:07.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.267+0000 7f3d867fc640 1 -- 172.21.15.40:0/3725483863 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d780043c0 con 0x55c384a22020 2024-04-21T23:20:07.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.268+0000 7f3d99233740 1 -- 172.21.15.40:0/3725483863 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c384a22020 msgr2=0x55c384a22400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:07.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.268+0000 7f3d99233740 1 --2- 172.21.15.40:0/3725483863 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c384a22020 0x55c384a22400 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f3d780bd740 tx=0x7f3d780bd8d0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.268+0000 7f3d99233740 1 -- 172.21.15.40:0/3725483863 shutdown_connections 2024-04-21T23:20:07.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.268+0000 7f3d99233740 1 --2- 172.21.15.40:0/3725483863 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c384a2b520 0x55c384a2d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.268+0000 7f3d99233740 1 --2- 172.21.15.40:0/3725483863 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c384a22940 0x55c384a2afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.268+0000 7f3d99233740 1 --2- 172.21.15.40:0/3725483863 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c384a22020 0x55c384a22400 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.268+0000 7f3d99233740 1 -- 172.21.15.40:0/3725483863 >> 172.21.15.40:0/3725483863 conn(0x55c384a1c750 msgr2=0x55c384a1cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:07.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.268+0000 7f3d99233740 1 -- 172.21.15.40:0/3725483863 shutdown_connections 2024-04-21T23:20:07.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.268+0000 7f3d99233740 1 -- 172.21.15.40:0/3725483863 wait complete. 2024-04-21T23:20:07.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.268+0000 7f3d99233740 1 -- start start 2024-04-21T23:20:07.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.269+0000 7f3d99233740 1 Processor -- start 2024-04-21T23:20:07.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.269+0000 7f3d99233740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c384a22020 0x55c384a2ade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:07.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.269+0000 7f3d99233740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c384a22940 0x55c384a28170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:07.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.269+0000 7f3d99233740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c384a2b520 0x55c384a286b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:07.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.269+0000 7f3d99233740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55c384a2e0d0 con 0x55c384a22940 2024-04-21T23:20:07.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.269+0000 7f3d99233740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55c384941f10 con 0x55c384a22020 2024-04-21T23:20:07.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.269+0000 7f3d99233740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55c384a2e250 con 0x55c384a2b520 2024-04-21T23:20:07.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.269+0000 7f3d99233740 10 client.0 _ll_register_callbacks cb 0x55c384699be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:20:07.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.269+0000 7f3d8f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c384a22940 0x55c384a28170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:07.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.269+0000 7f3d8ffff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c384a22020 0x55c384a2ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:07.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.269+0000 7f3d8f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c384a22940 0x55c384a28170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37214/0 (socket says 172.21.15.40:37214) 2024-04-21T23:20:07.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.269+0000 7f3d8f7fe640 1 -- 172.21.15.40:0/4262230312 learned_addr learned my addr 172.21.15.40:0/4262230312 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:20:07.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.269+0000 7f3d94b12640 1 --2- 172.21.15.40:0/4262230312 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c384a2b520 0x55c384a286b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:07.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.269+0000 7f3d8f7fe640 1 -- 172.21.15.40:0/4262230312 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c384a2b520 msgr2=0x55c384a286b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:07.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.269+0000 7f3d8f7fe640 1 --2- 172.21.15.40:0/4262230312 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c384a2b520 0x55c384a286b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.269+0000 7f3d8f7fe640 1 -- 172.21.15.40:0/4262230312 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c384a22020 msgr2=0x55c384a2ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:07.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.269+0000 7f3d8f7fe640 1 --2- 172.21.15.40:0/4262230312 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c384a22020 0x55c384a2ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.269+0000 7f3d8f7fe640 1 -- 172.21.15.40:0/4262230312 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c384913c80 con 0x55c384a22940 2024-04-21T23:20:07.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.269+0000 7f3d8ffff640 1 --2- 172.21.15.40:0/4262230312 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c384a22020 0x55c384a2ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:20:07.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.270+0000 7f3d8f7fe640 1 --2- 172.21.15.40:0/4262230312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c384a22940 0x55c384a28170 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f3d80013990 tx=0x7f3d80013e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=749b9e06d6db02e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:07.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.270+0000 7f3d6f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-21T23:20:07.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.270+0000 7f3d6f7fe640 1 -- 172.21.15.40:0/4262230312 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d80011d30 con 0x55c384a22940 2024-04-21T23:20:07.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.270+0000 7f3d6f7fe640 1 -- 172.21.15.40:0/4262230312 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f3d80005020 con 0x55c384a22940 2024-04-21T23:20:07.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.270+0000 7f3d99233740 1 -- 172.21.15.40:0/4262230312 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3d78004c80 con 0x55c384a22940 2024-04-21T23:20:07.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.270+0000 7f3d6f7fe640 1 -- 172.21.15.40:0/4262230312 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f3d80026050 con 0x55c384a22940 2024-04-21T23:20:07.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.270+0000 7f3d99233740 10 client.16137 fetch_fsmap learned FSMap version 28 2024-04-21T23:20:07.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.270+0000 7f3d99233740 1 -- 172.21.15.40:0/4262230312 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c38469a3f0 con 0x55c384a22940 2024-04-21T23:20:07.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.270+0000 7f3d6f7fe640 1 -- 172.21.15.40:0/4262230312 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d78004c80 con 0x55c384a22940 2024-04-21T23:20:07.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.270+0000 7f3d6f7fe640 1 -- 172.21.15.40:0/4262230312 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f3d8001b410 con 0x55c384a22940 2024-04-21T23:20:07.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.270+0000 7f3d99233740 10 client.16137 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T23:20:07.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.270+0000 7f3d99233740 10 client.16137 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T23:20:07.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.270+0000 7f3d99233740 20 client.16137 populate_metadata read hostname 'smithi040' 2024-04-21T23:20:07.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.270+0000 7f3d99233740 1 --2- 172.21.15.40:0/4262230312 >> [v2:172.21.15.174:6838/2467509274,v1:172.21.15.174:6839/2467509274] conn(0x55c384a3b2b0 0x55c384a3d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:07.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.270+0000 7f3d99233740 4 client.16137 mds_command: new command op to 24502 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:20:07.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.270+0000 7f3d99233740 1 -- 172.21.15.40:0/4262230312 --> [v2:172.21.15.174:6838/2467509274,v1:172.21.15.174:6839/2467509274] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c384993a70 con 0x55c384a3b2b0 2024-04-21T23:20:07.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.271+0000 7f3d8ffff640 1 --2- 172.21.15.40:0/4262230312 >> [v2:172.21.15.174:6838/2467509274,v1:172.21.15.174:6839/2467509274] conn(0x55c384a3b2b0 0x55c384a3d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:07.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.271+0000 7f3d6f7fe640 10 client.16137 ms_handle_connect on v2:172.21.15.174:6838/2467509274 2024-04-21T23:20:07.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.271+0000 7f3d8ffff640 1 --2- 172.21.15.40:0/4262230312 >> [v2:172.21.15.174:6838/2467509274,v1:172.21.15.174:6839/2467509274] conn(0x55c384a3b2b0 0x55c384a3d710 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4deb68bbfca5d0de server_cookie=1e7108b5a422077d in_seq=0 out_seq=0 2024-04-21T23:20:07.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.272+0000 7f3d6f7fe640 1 -- 172.21.15.40:0/4262230312 <== mds.1 v2:172.21.15.174:6838/2467509274 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c384993a70 con 0x55c384a3b2b0 2024-04-21T23:20:07.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.272+0000 7f3d6f7fe640 10 client.16137 handle_command_reply: tid=0 2024-04-21T23:20:07.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.272+0000 7f3d99233740 1 client.16137 shutdown 2024-04-21T23:20:07.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.272+0000 7f3d99233740 1 -- 172.21.15.40:0/4262230312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c384a22940 msgr2=0x55c384a28170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:07.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.272+0000 7f3d99233740 1 --2- 172.21.15.40:0/4262230312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c384a22940 0x55c384a28170 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f3d80013990 tx=0x7f3d80013e50 comp rx=0 tx=0).stop 2024-04-21T23:20:07.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.272+0000 7f3d99233740 1 -- 172.21.15.40:0/4262230312 shutdown_connections 2024-04-21T23:20:07.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.272+0000 7f3d99233740 1 --2- 172.21.15.40:0/4262230312 >> [v2:172.21.15.174:6838/2467509274,v1:172.21.15.174:6839/2467509274] conn(0x55c384a3b2b0 0x55c384a3d710 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.272+0000 7f3d99233740 1 --2- 172.21.15.40:0/4262230312 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c384a2b520 0x55c384a286b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.272+0000 7f3d99233740 1 --2- 172.21.15.40:0/4262230312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c384a22940 0x55c384a28170 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.272+0000 7f3d6f7fe640 0 client.16137 ms_handle_reset on v2:172.21.15.174:6838/2467509274 2024-04-21T23:20:07.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.272+0000 7f3d99233740 1 --2- 172.21.15.40:0/4262230312 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c384a22020 0x55c384a2ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.272+0000 7f3d99233740 1 -- 172.21.15.40:0/4262230312 >> 172.21.15.40:0/4262230312 conn(0x55c384a1c750 msgr2=0x55c384a1e4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:07.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.272+0000 7f3d99233740 1 -- 172.21.15.40:0/4262230312 shutdown_connections 2024-04-21T23:20:07.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.272+0000 7f3d99233740 1 -- 172.21.15.40:0/4262230312 wait complete. 2024-04-21T23:20:07.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.272+0000 7f3d99233740 20 client.16137 trim_cache size 0 max 16384 2024-04-21T23:20:07.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.286+0000 7f3d99233740 1 Processor -- start 2024-04-21T23:20:07.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.286+0000 7f3d99233740 1 -- start start 2024-04-21T23:20:07.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.286+0000 7f3d99233740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c384a1c750 0x55c384a36dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:07.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.286+0000 7f3d99233740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c384a22020 0x55c384a2a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:07.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.286+0000 7f3d99233740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c384a22940 0x55c384a2ade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:07.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.286+0000 7f3d99233740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3d901b1660 con 0x55c384a22940 2024-04-21T23:20:07.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.286+0000 7f3d99233740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3d901aeed0 con 0x55c384a1c750 2024-04-21T23:20:07.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.286+0000 7f3d99233740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3d901acd20 con 0x55c384a22020 2024-04-21T23:20:07.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.286+0000 7f3d94b12640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c384a22940 0x55c384a2ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:07.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.286+0000 7f3d94b12640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c384a22940 0x55c384a2ade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37224/0 (socket says 172.21.15.40:37224) 2024-04-21T23:20:07.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.286+0000 7f3d94b12640 1 -- 172.21.15.40:0/216752256 learned_addr learned my addr 172.21.15.40:0/216752256 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:20:07.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.286+0000 7f3d8ffff640 1 --2- 172.21.15.40:0/216752256 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c384a1c750 0x55c384a36dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:07.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.286+0000 7f3d8f7fe640 1 --2- 172.21.15.40:0/216752256 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c384a22020 0x55c384a2a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:07.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.286+0000 7f3d94b12640 1 -- 172.21.15.40:0/216752256 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c384a22020 msgr2=0x55c384a2a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:07.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.286+0000 7f3d94b12640 1 --2- 172.21.15.40:0/216752256 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c384a22020 0x55c384a2a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.286+0000 7f3d94b12640 1 -- 172.21.15.40:0/216752256 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c384a1c750 msgr2=0x55c384a36dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:07.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.286+0000 7f3d94b12640 1 --2- 172.21.15.40:0/216752256 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c384a1c750 0x55c384a36dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.286+0000 7f3d94b12640 1 -- 172.21.15.40:0/216752256 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c3847647d0 con 0x55c384a22940 2024-04-21T23:20:07.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.287+0000 7f3d94b12640 1 --2- 172.21.15.40:0/216752256 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c384a22940 0x55c384a2ade0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f3d88012c80 tx=0x7f3d88009600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b061e4b1146d8b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:07.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.287+0000 7f3d85ffb640 1 -- 172.21.15.40:0/216752256 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d88026070 con 0x55c384a22940 2024-04-21T23:20:07.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.287+0000 7f3d85ffb640 1 -- 172.21.15.40:0/216752256 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d8800fc30 con 0x55c384a22940 2024-04-21T23:20:07.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.287+0000 7f3d85ffb640 1 -- 172.21.15.40:0/216752256 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d880215f0 con 0x55c384a22940 2024-04-21T23:20:07.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.287+0000 7f3d99233740 1 -- 172.21.15.40:0/216752256 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c384a22940 msgr2=0x55c384a2ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:07.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.287+0000 7f3d99233740 1 --2- 172.21.15.40:0/216752256 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c384a22940 0x55c384a2ade0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f3d88012c80 tx=0x7f3d88009600 comp rx=0 tx=0).stop 2024-04-21T23:20:07.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.287+0000 7f3d99233740 1 -- 172.21.15.40:0/216752256 shutdown_connections 2024-04-21T23:20:07.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.287+0000 7f3d99233740 1 --2- 172.21.15.40:0/216752256 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c384a22940 0x55c384a2ade0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.287+0000 7f3d99233740 1 --2- 172.21.15.40:0/216752256 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c384a22020 0x55c384a2a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.287+0000 7f3d99233740 1 --2- 172.21.15.40:0/216752256 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c384a1c750 0x55c384a36dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.287+0000 7f3d99233740 1 -- 172.21.15.40:0/216752256 >> 172.21.15.40:0/216752256 conn(0x55c384a28810 msgr2=0x55c384a236f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:07.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.287+0000 7f3d99233740 1 -- 172.21.15.40:0/216752256 shutdown_connections 2024-04-21T23:20:07.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.288+0000 7f3d99233740 1 -- 172.21.15.40:0/216752256 wait complete. 2024-04-21T23:20:07.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.288+0000 7f3d99233740 1 -- start start 2024-04-21T23:20:07.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.288+0000 7f3d99233740 1 Processor -- start 2024-04-21T23:20:07.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.288+0000 7f3d99233740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c384a1c750 0x55c384a335e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:07.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.288+0000 7f3d99233740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c384a22020 0x55c384a2e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:07.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.288+0000 7f3d99233740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c384a22940 0x55c384a2eee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:07.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.288+0000 7f3d99233740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3d901ad390 con 0x55c384a22020 2024-04-21T23:20:07.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.288+0000 7f3d99233740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3d901ae800 con 0x55c384a1c750 2024-04-21T23:20:07.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.288+0000 7f3d99233740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3d901ada60 con 0x55c384a22940 2024-04-21T23:20:07.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.288+0000 7f3d99233740 10 client.0 _ll_register_callbacks cb 0x7f3d800135a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:20:07.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.288+0000 7f3d8f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c384a22020 0x55c384a2e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:07.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.288+0000 7f3d8f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c384a22020 0x55c384a2e9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37230/0 (socket says 172.21.15.40:37230) 2024-04-21T23:20:07.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.288+0000 7f3d8f7fe640 1 -- 172.21.15.40:0/925420343 learned_addr learned my addr 172.21.15.40:0/925420343 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:20:07.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.288+0000 7f3d8ffff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c384a1c750 0x55c384a335e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:07.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.289+0000 7f3d94b12640 1 --2- 172.21.15.40:0/925420343 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c384a22940 0x55c384a2eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:07.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.289+0000 7f3d8f7fe640 1 -- 172.21.15.40:0/925420343 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c384a22940 msgr2=0x55c384a2eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:07.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.289+0000 7f3d8f7fe640 1 --2- 172.21.15.40:0/925420343 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c384a22940 0x55c384a2eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.289+0000 7f3d8f7fe640 1 -- 172.21.15.40:0/925420343 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c384a1c750 msgr2=0x55c384a335e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:07.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.289+0000 7f3d8f7fe640 1 --2- 172.21.15.40:0/925420343 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c384a1c750 0x55c384a335e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.289+0000 7f3d8f7fe640 1 -- 172.21.15.40:0/925420343 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c384a2f6e0 con 0x55c384a22020 2024-04-21T23:20:07.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.289+0000 7f3d94b12640 1 --2- 172.21.15.40:0/925420343 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c384a22940 0x55c384a2eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:20:07.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.289+0000 7f3d8f7fe640 1 --2- 172.21.15.40:0/925420343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c384a22020 0x55c384a2e9a0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f3d8001ba20 tx=0x7f3d80011f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7544a1e304afb526 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:07.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.289+0000 7f3d6f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-21T23:20:07.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.289+0000 7f3d6f7fe640 1 -- 172.21.15.40:0/925420343 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d8002e070 con 0x55c384a22020 2024-04-21T23:20:07.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.289+0000 7f3d6f7fe640 1 -- 172.21.15.40:0/925420343 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f3d80013220 con 0x55c384a22020 2024-04-21T23:20:07.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.289+0000 7f3d99233740 1 -- 172.21.15.40:0/925420343 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3d8800fc30 con 0x55c384a22020 2024-04-21T23:20:07.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.289+0000 7f3d6f7fe640 1 -- 172.21.15.40:0/925420343 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f3d80037050 con 0x55c384a22020 2024-04-21T23:20:07.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.289+0000 7f3d99233740 10 client.16143 fetch_fsmap learned FSMap version 28 2024-04-21T23:20:07.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.289+0000 7f3d99233740 1 -- 172.21.15.40:0/925420343 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c384aa5bd0 con 0x55c384a22020 2024-04-21T23:20:07.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.289+0000 7f3d6f7fe640 1 -- 172.21.15.40:0/925420343 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d8800fc30 con 0x55c384a22020 2024-04-21T23:20:07.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.290+0000 7f3d6f7fe640 1 -- 172.21.15.40:0/925420343 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f3d80031020 con 0x55c384a22020 2024-04-21T23:20:07.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.290+0000 7f3d99233740 10 client.16143 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T23:20:07.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.290+0000 7f3d99233740 10 client.16143 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T23:20:07.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.290+0000 7f3d99233740 20 client.16143 populate_metadata read hostname 'smithi040' 2024-04-21T23:20:07.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.290+0000 7f3d99233740 1 --2- 172.21.15.40:0/925420343 >> [v2:172.21.15.174:6838/2467509274,v1:172.21.15.174:6839/2467509274] conn(0x55c384a2b520 0x55c384aa8410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:07.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.290+0000 7f3d99233740 4 client.16143 mds_command: new command op to 24502 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T23:20:07.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.290+0000 7f3d99233740 1 -- 172.21.15.40:0/925420343 --> [v2:172.21.15.174:6838/2467509274,v1:172.21.15.174:6839/2467509274] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c38498ddc0 con 0x55c384a2b520 2024-04-21T23:20:07.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.290+0000 7f3d8ffff640 1 --2- 172.21.15.40:0/925420343 >> [v2:172.21.15.174:6838/2467509274,v1:172.21.15.174:6839/2467509274] conn(0x55c384a2b520 0x55c384aa8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:07.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.290+0000 7f3d8ffff640 1 --2- 172.21.15.40:0/925420343 >> [v2:172.21.15.174:6838/2467509274,v1:172.21.15.174:6839/2467509274] conn(0x55c384a2b520 0x55c384aa8410 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=521f7fc55b7e5a6d server_cookie=b7c741b2963a1ecc in_seq=0 out_seq=0 2024-04-21T23:20:07.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.290+0000 7f3d6f7fe640 10 client.16143 ms_handle_connect on v2:172.21.15.174:6838/2467509274 2024-04-21T23:20:07.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.291+0000 7f3d6f7fe640 1 -- 172.21.15.40:0/925420343 <== mds.1 v2:172.21.15.174:6838/2467509274 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c38498ddc0 con 0x55c384a2b520 2024-04-21T23:20:07.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.291+0000 7f3d6f7fe640 10 client.16143 handle_command_reply: tid=0 2024-04-21T23:20:07.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.291+0000 7f3d99233740 1 client.16143 shutdown 2024-04-21T23:20:07.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.291+0000 7f3d99233740 1 -- 172.21.15.40:0/925420343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c384a22020 msgr2=0x55c384a2e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:07.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.291+0000 7f3d99233740 1 --2- 172.21.15.40:0/925420343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c384a22020 0x55c384a2e9a0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f3d8001ba20 tx=0x7f3d80011f60 comp rx=0 tx=0).stop 2024-04-21T23:20:07.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.291+0000 7f3d99233740 1 -- 172.21.15.40:0/925420343 shutdown_connections 2024-04-21T23:20:07.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.291+0000 7f3d99233740 1 --2- 172.21.15.40:0/925420343 >> [v2:172.21.15.174:6838/2467509274,v1:172.21.15.174:6839/2467509274] conn(0x55c384a2b520 0x55c384aa8410 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.291+0000 7f3d99233740 1 --2- 172.21.15.40:0/925420343 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c384a22940 0x55c384a2eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.291+0000 7f3d99233740 1 --2- 172.21.15.40:0/925420343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c384a22020 0x55c384a2e9a0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.291+0000 7f3d6f7fe640 0 client.16143 ms_handle_reset on v2:172.21.15.174:6838/2467509274 2024-04-21T23:20:07.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.291+0000 7f3d99233740 1 --2- 172.21.15.40:0/925420343 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c384a1c750 0x55c384a335e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.291+0000 7f3d99233740 1 -- 172.21.15.40:0/925420343 >> 172.21.15.40:0/925420343 conn(0x55c384a28810 msgr2=0x55c384a1afc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:07.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.291+0000 7f3d99233740 1 -- 172.21.15.40:0/925420343 shutdown_connections 2024-04-21T23:20:07.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.291+0000 7f3d99233740 1 -- 172.21.15.40:0/925420343 wait complete. 2024-04-21T23:20:07.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.291+0000 7f3d99233740 20 client.16143 trim_cache size 0 max 16384 2024-04-21T23:20:07.298 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-04-21T23:20:07.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.292+0000 7f3d96575640 1 -- 172.21.15.40:0/4264413824 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f3d7c0a2d00 msgr2=0x7f3d7c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:07.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.292+0000 7f3d96575640 1 --2- 172.21.15.40:0/4264413824 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f3d7c0a2d00 0x7f3d7c0a51c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3d88005fd0 tx=0x7f3d88014b70 comp rx=0 tx=0).stop 2024-04-21T23:20:07.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.292+0000 7f3d96575640 1 -- 172.21.15.40:0/4264413824 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d901029c0 msgr2=0x7f3d90198070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:07.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.292+0000 7f3d96575640 1 --2- 172.21.15.40:0/4264413824 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d901029c0 0x7f3d90198070 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f3d78009a20 tx=0x7f3d7802dd30 comp rx=0 tx=0).stop 2024-04-21T23:20:07.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.292+0000 7f3d96575640 1 -- 172.21.15.40:0/4264413824 shutdown_connections 2024-04-21T23:20:07.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.292+0000 7f3d96575640 1 --2- 172.21.15.40:0/4264413824 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f3d7c0a2d00 0x7f3d7c0a51c0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.292+0000 7f3d96575640 1 --2- 172.21.15.40:0/4264413824 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d901032e0 0x7f3d901985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.292+0000 7f3d96575640 1 --2- 172.21.15.40:0/4264413824 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3d9010bd00 0x7f3d9019c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.292+0000 7f3d96575640 1 --2- 172.21.15.40:0/4264413824 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d901029c0 0x7f3d90198070 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.292+0000 7f3d96575640 1 -- 172.21.15.40:0/4264413824 >> 172.21.15.40:0/4264413824 conn(0x7f3d90077740 msgr2=0x7f3d900fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:07.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.292+0000 7f3d96575640 1 -- 172.21.15.40:0/4264413824 shutdown_connections 2024-04-21T23:20:07.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.292+0000 7f3d96575640 1 -- 172.21.15.40:0/4264413824 wait complete. 2024-04-21T23:20:07.300 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:06 smithi040 ceph-mon[27984]: pgmap v54: 129 pgs: 129 active+clean; 34 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.9 MiB/s wr, 322 op/s 2024-04-21T23:20:07.301 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T23:20:07.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.409+0000 7fd772ec6640 1 -- 172.21.15.40:0/1957107607 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd76c100fd0 msgr2=0x7fd76c10d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:07.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.409+0000 7fd772ec6640 1 --2- 172.21.15.40:0/1957107607 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd76c100fd0 0x7fd76c10d4e0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fd764009cd0 tx=0x7fd76402c0f0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.409+0000 7fd772ec6640 1 -- 172.21.15.40:0/1957107607 shutdown_connections 2024-04-21T23:20:07.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.409+0000 7fd772ec6640 1 --2- 172.21.15.40:0/1957107607 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd76c100fd0 0x7fd76c10d4e0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.409+0000 7fd772ec6640 1 --2- 172.21.15.40:0/1957107607 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd76c100610 0x7fd76c100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.409+0000 7fd772ec6640 1 --2- 172.21.15.40:0/1957107607 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd76c1066b0 0x7fd76c106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.409+0000 7fd772ec6640 1 -- 172.21.15.40:0/1957107607 >> 172.21.15.40:0/1957107607 conn(0x7fd76c077740 msgr2=0x7fd76c077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:07.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.409+0000 7fd772ec6640 1 -- 172.21.15.40:0/1957107607 shutdown_connections 2024-04-21T23:20:07.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.409+0000 7fd772ec6640 1 -- 172.21.15.40:0/1957107607 wait complete. 2024-04-21T23:20:07.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.410+0000 7fd772ec6640 1 Processor -- start 2024-04-21T23:20:07.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.410+0000 7fd772ec6640 1 -- start start 2024-04-21T23:20:07.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.410+0000 7fd772ec6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd76c100610 0x7fd76c197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:07.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.410+0000 7fd772ec6640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd76c100fd0 0x7fd76c1984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:07.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.410+0000 7fd772ec6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd76c1066b0 0x7fd76c19c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:07.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.410+0000 7fd772ec6640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd76c111b30 con 0x7fd76c100610 2024-04-21T23:20:07.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.410+0000 7fd772ec6640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd76c1119b0 con 0x7fd76c100fd0 2024-04-21T23:20:07.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.410+0000 7fd772ec6640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd76c111cb0 con 0x7fd76c1066b0 2024-04-21T23:20:07.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.410+0000 7fd770c62640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd76c100610 0x7fd76c197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:07.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.410+0000 7fd770c62640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd76c100610 0x7fd76c197f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37244/0 (socket says 172.21.15.40:37244) 2024-04-21T23:20:07.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.410+0000 7fd770c62640 1 -- 172.21.15.40:0/1483416215 learned_addr learned my addr 172.21.15.40:0/1483416215 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:20:07.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.410+0000 7fd76bfff640 1 --2- 172.21.15.40:0/1483416215 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd76c100fd0 0x7fd76c1984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:07.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.410+0000 7fd771463640 1 --2- 172.21.15.40:0/1483416215 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd76c1066b0 0x7fd76c19c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:07.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.410+0000 7fd770c62640 1 -- 172.21.15.40:0/1483416215 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd76c1066b0 msgr2=0x7fd76c19c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:07.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.410+0000 7fd770c62640 1 --2- 172.21.15.40:0/1483416215 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd76c1066b0 0x7fd76c19c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.410+0000 7fd770c62640 1 -- 172.21.15.40:0/1483416215 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd76c100fd0 msgr2=0x7fd76c1984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:07.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.410+0000 7fd770c62640 1 --2- 172.21.15.40:0/1483416215 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd76c100fd0 0x7fd76c1984d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.410+0000 7fd770c62640 1 -- 172.21.15.40:0/1483416215 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd76c19cf80 con 0x7fd76c100610 2024-04-21T23:20:07.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.410+0000 7fd770c62640 1 --2- 172.21.15.40:0/1483416215 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd76c100610 0x7fd76c197f90 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fd75c00e9c0 tx=0x7fd75c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:07.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.411+0000 7fd769ffb640 1 -- 172.21.15.40:0/1483416215 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd75c00cd80 con 0x7fd76c100610 2024-04-21T23:20:07.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.411+0000 7fd769ffb640 1 -- 172.21.15.40:0/1483416215 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd75c0035f0 con 0x7fd76c100610 2024-04-21T23:20:07.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.411+0000 7fd772ec6640 1 -- 172.21.15.40:0/1483416215 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd76c19d270 con 0x7fd76c100610 2024-04-21T23:20:07.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.411+0000 7fd769ffb640 1 -- 172.21.15.40:0/1483416215 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd75c010690 con 0x7fd76c100610 2024-04-21T23:20:07.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.411+0000 7fd772ec6640 1 -- 172.21.15.40:0/1483416215 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd76c1a4b00 con 0x7fd76c100610 2024-04-21T23:20:07.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.413+0000 7fd769ffb640 1 -- 172.21.15.40:0/1483416215 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fd75c0037a0 con 0x7fd76c100610 2024-04-21T23:20:07.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.413+0000 7fd775b93740 1 Processor -- start 2024-04-21T23:20:07.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.413+0000 7fd775b93740 1 -- start start 2024-04-21T23:20:07.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.413+0000 7fd769ffb640 1 --2- 172.21.15.40:0/1483416215 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fd7540a2d00 0x7fd7540a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:07.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.413+0000 7fd775b93740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55742d79c020 0x55742d79c400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:07.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.414+0000 7fd771463640 1 --2- 172.21.15.40:0/1483416215 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fd7540a2d00 0x7fd7540a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:07.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.414+0000 7fd769ffb640 1 -- 172.21.15.40:0/1483416215 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7fd75c0cfbc0 con 0x7fd76c100610 2024-04-21T23:20:07.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.414+0000 7fd775b93740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55742d79c940 0x55742d7a4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:07.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.414+0000 7fd775b93740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55742d7a5520 0x55742d7a7910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:07.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.414+0000 7fd775b93740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55742d700530 con 0x55742d79c940 2024-04-21T23:20:07.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.414+0000 7fd775b93740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55742d6a8830 con 0x55742d7a5520 2024-04-21T23:20:07.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.414+0000 7fd775b93740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55742d6fc4f0 con 0x55742d79c020 2024-04-21T23:20:07.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.414+0000 7fd76bfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55742d79c940 0x55742d7a4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:07.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.414+0000 7fd76bfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55742d79c940 0x55742d7a4fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37250/0 (socket says 172.21.15.40:37250) 2024-04-21T23:20:07.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.414+0000 7fd76bfff640 1 -- 172.21.15.40:0/1449096626 learned_addr learned my addr 172.21.15.40:0/1449096626 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:20:07.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.414+0000 7fd771463640 1 --2- 172.21.15.40:0/1449096626 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55742d7a5520 0x55742d7a7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:07.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.414+0000 7fd770c62640 1 --2- 172.21.15.40:0/1449096626 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55742d79c020 0x55742d79c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:07.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.414+0000 7fd76bfff640 1 -- 172.21.15.40:0/1449096626 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55742d79c020 msgr2=0x55742d79c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:07.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.414+0000 7fd76bfff640 1 --2- 172.21.15.40:0/1449096626 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55742d79c020 0x55742d79c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.414+0000 7fd76bfff640 1 -- 172.21.15.40:0/1449096626 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55742d7a5520 msgr2=0x55742d7a7910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:07.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.414+0000 7fd76bfff640 1 --2- 172.21.15.40:0/1449096626 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55742d7a5520 0x55742d7a7910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.414+0000 7fd76bfff640 1 -- 172.21.15.40:0/1449096626 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55742d6cc2c0 con 0x55742d79c940 2024-04-21T23:20:07.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.414+0000 7fd771463640 1 --2- 172.21.15.40:0/1449096626 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55742d7a5520 0x55742d7a7910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:20:07.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.414+0000 7fd771463640 1 --2- 172.21.15.40:0/1483416215 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fd7540a2d00 0x7fd7540a51c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd764005cc0 tx=0x7fd76402ce60 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:07.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.414+0000 7fd76bfff640 1 --2- 172.21.15.40:0/1449096626 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55742d79c940 0x55742d7a4fe0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fd76000b740 tx=0x7fd76000bc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1a7504a5875c665 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:07.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.414+0000 7fd752ffd640 1 -- 172.21.15.40:0/1449096626 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd760003dd0 con 0x55742d79c940 2024-04-21T23:20:07.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.415+0000 7fd752ffd640 1 -- 172.21.15.40:0/1449096626 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd760015070 con 0x55742d79c940 2024-04-21T23:20:07.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.415+0000 7fd752ffd640 1 -- 172.21.15.40:0/1449096626 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd760004e50 con 0x55742d79c940 2024-04-21T23:20:07.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.415+0000 7fd775b93740 1 -- 172.21.15.40:0/1449096626 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55742d79c940 msgr2=0x55742d7a4fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:07.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.415+0000 7fd775b93740 1 --2- 172.21.15.40:0/1449096626 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55742d79c940 0x55742d7a4fe0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fd76000b740 tx=0x7fd76000bc00 comp rx=0 tx=0).stop 2024-04-21T23:20:07.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.415+0000 7fd775b93740 1 -- 172.21.15.40:0/1449096626 shutdown_connections 2024-04-21T23:20:07.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.415+0000 7fd775b93740 1 --2- 172.21.15.40:0/1449096626 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55742d7a5520 0x55742d7a7910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.415+0000 7fd775b93740 1 --2- 172.21.15.40:0/1449096626 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55742d79c940 0x55742d7a4fe0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.415+0000 7fd775b93740 1 --2- 172.21.15.40:0/1449096626 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55742d79c020 0x55742d79c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.415+0000 7fd775b93740 1 -- 172.21.15.40:0/1449096626 >> 172.21.15.40:0/1449096626 conn(0x55742d796750 msgr2=0x55742d796b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:07.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.415+0000 7fd775b93740 1 -- 172.21.15.40:0/1449096626 shutdown_connections 2024-04-21T23:20:07.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.415+0000 7fd775b93740 1 -- 172.21.15.40:0/1449096626 wait complete. 2024-04-21T23:20:07.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.416+0000 7fd775b93740 1 -- start start 2024-04-21T23:20:07.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.416+0000 7fd775b93740 1 Processor -- start 2024-04-21T23:20:07.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.416+0000 7fd775b93740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55742d79c020 0x55742d7a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:07.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.416+0000 7fd775b93740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55742d79c940 0x55742d7a2260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:07.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.416+0000 7fd775b93740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55742d7a5520 0x55742d7a2800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:07.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.416+0000 7fd775b93740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55742d7a8310 con 0x55742d79c020 2024-04-21T23:20:07.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.416+0000 7fd775b93740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55742d6bbf10 con 0x55742d7a5520 2024-04-21T23:20:07.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.416+0000 7fd775b93740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55742d7a8490 con 0x55742d79c940 2024-04-21T23:20:07.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.416+0000 7fd775b93740 10 client.0 _ll_register_callbacks cb 0x55742d413be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:20:07.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.416+0000 7fd770c62640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55742d79c020 0x55742d7a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:07.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.416+0000 7fd770c62640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55742d79c020 0x55742d7a4e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37266/0 (socket says 172.21.15.40:37266) 2024-04-21T23:20:07.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.416+0000 7fd770c62640 1 -- 172.21.15.40:0/3058902213 learned_addr learned my addr 172.21.15.40:0/3058902213 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:20:07.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.416+0000 7fd771463640 1 --2- 172.21.15.40:0/3058902213 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55742d7a5520 0x55742d7a2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:07.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.416+0000 7fd76bfff640 1 --2- 172.21.15.40:0/3058902213 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55742d79c940 0x55742d7a2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:07.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.416+0000 7fd770c62640 1 -- 172.21.15.40:0/3058902213 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55742d79c940 msgr2=0x55742d7a2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:07.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.416+0000 7fd770c62640 1 --2- 172.21.15.40:0/3058902213 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55742d79c940 0x55742d7a2260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.416+0000 7fd770c62640 1 -- 172.21.15.40:0/3058902213 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55742d7a5520 msgr2=0x55742d7a2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:07.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.416+0000 7fd770c62640 1 --2- 172.21.15.40:0/3058902213 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55742d7a5520 0x55742d7a2800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.416+0000 7fd770c62640 1 -- 172.21.15.40:0/3058902213 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55742d68dc80 con 0x55742d79c020 2024-04-21T23:20:07.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.417+0000 7fd770c62640 1 --2- 172.21.15.40:0/3058902213 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55742d79c020 0x55742d7a4e60 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fd75c0ccfc0 tx=0x7fd75c09b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45b6fdf94add2315 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:07.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.417+0000 7fd74bfff640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-21T23:20:07.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.417+0000 7fd74bfff640 1 -- 172.21.15.40:0/3058902213 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd75c0a49a0 con 0x55742d79c020 2024-04-21T23:20:07.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.417+0000 7fd74bfff640 1 -- 172.21.15.40:0/3058902213 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7fd75c0983a0 con 0x55742d79c020 2024-04-21T23:20:07.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.417+0000 7fd775b93740 1 -- 172.21.15.40:0/3058902213 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd760015070 con 0x55742d79c020 2024-04-21T23:20:07.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.417+0000 7fd74bfff640 1 -- 172.21.15.40:0/3058902213 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd75c09b180 con 0x55742d79c020 2024-04-21T23:20:07.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.417+0000 7fd775b93740 10 client.16155 fetch_fsmap learned FSMap version 28 2024-04-21T23:20:07.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.417+0000 7fd775b93740 1 -- 172.21.15.40:0/3058902213 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55742d4143f0 con 0x55742d79c020 2024-04-21T23:20:07.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.417+0000 7fd74bfff640 1 -- 172.21.15.40:0/3058902213 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd760015070 con 0x55742d79c020 2024-04-21T23:20:07.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.417+0000 7fd74bfff640 1 -- 172.21.15.40:0/3058902213 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fd75c0b2030 con 0x55742d79c020 2024-04-21T23:20:07.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.417+0000 7fd775b93740 10 client.16155 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T23:20:07.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.417+0000 7fd775b93740 10 client.16155 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:20:07.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.417+0000 7fd775b93740 20 client.16155 populate_metadata read hostname 'smithi040' 2024-04-21T23:20:07.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.417+0000 7fd775b93740 1 --2- 172.21.15.40:0/3058902213 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55742d7b5500 0x55742d7b7950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:07.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.417+0000 7fd775b93740 4 client.16155 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:20:07.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.417+0000 7fd775b93740 1 -- 172.21.15.40:0/3058902213 --> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55742d70da70 con 0x55742d7b5500 2024-04-21T23:20:07.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.418+0000 7fd76bfff640 1 --2- 172.21.15.40:0/3058902213 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55742d7b5500 0x55742d7b7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:07.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.418+0000 7fd76bfff640 1 --2- 172.21.15.40:0/3058902213 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55742d7b5500 0x55742d7b7950 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d2aeedd26a46def2 server_cookie=e99b9ab6a729de03 in_seq=0 out_seq=0 2024-04-21T23:20:07.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.418+0000 7fd74bfff640 10 client.16155 ms_handle_connect on v2:172.21.15.111:6838/3712235548 2024-04-21T23:20:07.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.419+0000 7fd74bfff640 1 -- 172.21.15.40:0/3058902213 <== mds.0 v2:172.21.15.111:6838/3712235548 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55742d70da70 con 0x55742d7b5500 2024-04-21T23:20:07.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.419+0000 7fd74bfff640 10 client.16155 handle_command_reply: tid=0 2024-04-21T23:20:07.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.419+0000 7fd775b93740 1 client.16155 shutdown 2024-04-21T23:20:07.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.419+0000 7fd775b93740 1 -- 172.21.15.40:0/3058902213 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55742d79c020 msgr2=0x55742d7a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:07.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.419+0000 7fd775b93740 1 --2- 172.21.15.40:0/3058902213 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55742d79c020 0x55742d7a4e60 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fd75c0ccfc0 tx=0x7fd75c09b980 comp rx=0 tx=0).stop 2024-04-21T23:20:07.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.419+0000 7fd775b93740 1 -- 172.21.15.40:0/3058902213 shutdown_connections 2024-04-21T23:20:07.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.419+0000 7fd775b93740 1 --2- 172.21.15.40:0/3058902213 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55742d7b5500 0x55742d7b7950 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.419+0000 7fd775b93740 1 --2- 172.21.15.40:0/3058902213 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55742d7a5520 0x55742d7a2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.419+0000 7fd775b93740 1 --2- 172.21.15.40:0/3058902213 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55742d79c940 0x55742d7a2260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.419+0000 7fd74bfff640 0 client.16155 ms_handle_reset on v2:172.21.15.111:6838/3712235548 2024-04-21T23:20:07.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.419+0000 7fd775b93740 1 --2- 172.21.15.40:0/3058902213 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55742d79c020 0x55742d7a4e60 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.419+0000 7fd775b93740 1 -- 172.21.15.40:0/3058902213 >> 172.21.15.40:0/3058902213 conn(0x55742d796750 msgr2=0x55742d7984a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:07.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.419+0000 7fd775b93740 1 -- 172.21.15.40:0/3058902213 shutdown_connections 2024-04-21T23:20:07.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.419+0000 7fd775b93740 1 -- 172.21.15.40:0/3058902213 wait complete. 2024-04-21T23:20:07.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.419+0000 7fd775b93740 20 client.16155 trim_cache size 0 max 16384 2024-04-21T23:20:07.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.433+0000 7fd775b93740 1 Processor -- start 2024-04-21T23:20:07.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.433+0000 7fd775b93740 1 -- start start 2024-04-21T23:20:07.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.433+0000 7fd775b93740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55742d79c020 0x55742d7a1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:07.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.433+0000 7fd775b93740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55742d79c940 0x55742d7a4990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:07.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.433+0000 7fd775b93740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55742d7a5520 0x55742d7bd220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:07.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.433+0000 7fd775b93740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd76c1b1620 con 0x55742d7a5520 2024-04-21T23:20:07.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.433+0000 7fd775b93740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd76c1aee90 con 0x55742d79c940 2024-04-21T23:20:07.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.433+0000 7fd775b93740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd76c1acce0 con 0x55742d79c020 2024-04-21T23:20:07.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.433+0000 7fd771463640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55742d7a5520 0x55742d7bd220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:07.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.433+0000 7fd771463640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55742d7a5520 0x55742d7bd220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37270/0 (socket says 172.21.15.40:37270) 2024-04-21T23:20:07.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.433+0000 7fd771463640 1 -- 172.21.15.40:0/2040429861 learned_addr learned my addr 172.21.15.40:0/2040429861 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:20:07.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.433+0000 7fd76bfff640 1 --2- 172.21.15.40:0/2040429861 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55742d79c940 0x55742d7a4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:07.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.433+0000 7fd770c62640 1 --2- 172.21.15.40:0/2040429861 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55742d79c020 0x55742d7a1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:07.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.433+0000 7fd771463640 1 -- 172.21.15.40:0/2040429861 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55742d79c020 msgr2=0x55742d7a1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:07.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.433+0000 7fd771463640 1 --2- 172.21.15.40:0/2040429861 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55742d79c020 0x55742d7a1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.433+0000 7fd771463640 1 -- 172.21.15.40:0/2040429861 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55742d79c940 msgr2=0x55742d7a4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:07.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.433+0000 7fd771463640 1 --2- 172.21.15.40:0/2040429861 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55742d79c940 0x55742d7a4990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.433+0000 7fd771463640 1 -- 172.21.15.40:0/2040429861 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55742d7a2b70 con 0x55742d7a5520 2024-04-21T23:20:07.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.434+0000 7fd771463640 1 --2- 172.21.15.40:0/2040429861 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55742d7a5520 0x55742d7bd220 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fd764009cd0 tx=0x7fd76402e050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c6c62532da58572 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:07.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.434+0000 7fd7527fc640 1 -- 172.21.15.40:0/2040429861 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd764039040 con 0x55742d7a5520 2024-04-21T23:20:07.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.434+0000 7fd7527fc640 1 -- 172.21.15.40:0/2040429861 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd764052070 con 0x55742d7a5520 2024-04-21T23:20:07.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.434+0000 7fd7527fc640 1 -- 172.21.15.40:0/2040429861 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd76404d450 con 0x55742d7a5520 2024-04-21T23:20:07.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.434+0000 7fd775b93740 1 -- 172.21.15.40:0/2040429861 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55742d7a5520 msgr2=0x55742d7bd220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:07.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.434+0000 7fd775b93740 1 --2- 172.21.15.40:0/2040429861 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55742d7a5520 0x55742d7bd220 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fd764009cd0 tx=0x7fd76402e050 comp rx=0 tx=0).stop 2024-04-21T23:20:07.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.434+0000 7fd775b93740 1 -- 172.21.15.40:0/2040429861 shutdown_connections 2024-04-21T23:20:07.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.434+0000 7fd775b93740 1 --2- 172.21.15.40:0/2040429861 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55742d79c940 0x55742d7a4990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.434+0000 7fd775b93740 1 --2- 172.21.15.40:0/2040429861 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55742d7a5520 0x55742d7bd220 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.434+0000 7fd775b93740 1 --2- 172.21.15.40:0/2040429861 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55742d79c020 0x55742d7a1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.434+0000 7fd775b93740 1 -- 172.21.15.40:0/2040429861 >> 172.21.15.40:0/2040429861 conn(0x55742d796750 msgr2=0x55742d795850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:07.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.434+0000 7fd775b93740 1 -- 172.21.15.40:0/2040429861 shutdown_connections 2024-04-21T23:20:07.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.434+0000 7fd775b93740 1 -- 172.21.15.40:0/2040429861 wait complete. 2024-04-21T23:20:07.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.435+0000 7fd775b93740 1 -- start start 2024-04-21T23:20:07.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.435+0000 7fd775b93740 1 Processor -- start 2024-04-21T23:20:07.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.435+0000 7fd775b93740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55742d79c020 0x55742d7ada80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:07.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.435+0000 7fd775b93740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55742d79c940 0x55742d79ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:07.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.435+0000 7fd775b93740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55742d7a5520 0x55742d79f100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:07.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.435+0000 7fd775b93740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd76c1ad350 con 0x55742d79c020 2024-04-21T23:20:07.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.435+0000 7fd775b93740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd76c1ae7c0 con 0x55742d79c940 2024-04-21T23:20:07.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.435+0000 7fd775b93740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd76c1ada20 con 0x55742d7a5520 2024-04-21T23:20:07.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.435+0000 7fd775b93740 10 client.0 _ll_register_callbacks cb 0x7fd75c003e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:20:07.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.435+0000 7fd770c62640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55742d79c020 0x55742d7ada80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:07.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.435+0000 7fd770c62640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55742d79c020 0x55742d7ada80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37280/0 (socket says 172.21.15.40:37280) 2024-04-21T23:20:07.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.435+0000 7fd770c62640 1 -- 172.21.15.40:0/2059001636 learned_addr learned my addr 172.21.15.40:0/2059001636 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:20:07.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.435+0000 7fd76bfff640 1 --2- 172.21.15.40:0/2059001636 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55742d79c940 0x55742d79ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:07.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.435+0000 7fd771463640 1 --2- 172.21.15.40:0/2059001636 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55742d7a5520 0x55742d79f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:07.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.436+0000 7fd770c62640 1 -- 172.21.15.40:0/2059001636 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55742d7a5520 msgr2=0x55742d79f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:07.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.436+0000 7fd770c62640 1 --2- 172.21.15.40:0/2059001636 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55742d7a5520 0x55742d79f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.436+0000 7fd770c62640 1 -- 172.21.15.40:0/2059001636 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55742d79c940 msgr2=0x55742d79ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:07.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.436+0000 7fd770c62640 1 --2- 172.21.15.40:0/2059001636 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55742d79c940 0x55742d79ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.436+0000 7fd770c62640 1 -- 172.21.15.40:0/2059001636 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55742d4de7d0 con 0x55742d79c020 2024-04-21T23:20:07.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.436+0000 7fd770c62640 1 --2- 172.21.15.40:0/2059001636 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55742d79c020 0x55742d7ada80 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fd75c09ba30 tx=0x7fd75c0985d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8263ed5f535c8bc0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:07.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.436+0000 7fd74bfff640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-21T23:20:07.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.436+0000 7fd74bfff640 1 -- 172.21.15.40:0/2059001636 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd75c0a1620 con 0x55742d79c020 2024-04-21T23:20:07.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.436+0000 7fd74bfff640 1 -- 172.21.15.40:0/2059001636 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7fd75c0a1270 con 0x55742d79c020 2024-04-21T23:20:07.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.436+0000 7fd775b93740 1 -- 172.21.15.40:0/2059001636 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd764052070 con 0x55742d79c020 2024-04-21T23:20:07.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.436+0000 7fd74bfff640 1 -- 172.21.15.40:0/2059001636 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd75c0ae050 con 0x55742d79c020 2024-04-21T23:20:07.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.436+0000 7fd775b93740 10 client.16161 fetch_fsmap learned FSMap version 28 2024-04-21T23:20:07.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.436+0000 7fd775b93740 1 -- 172.21.15.40:0/2059001636 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55742d81b880 con 0x55742d79c020 2024-04-21T23:20:07.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.436+0000 7fd74bfff640 1 -- 172.21.15.40:0/2059001636 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd764052070 con 0x55742d79c020 2024-04-21T23:20:07.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.436+0000 7fd74bfff640 1 -- 172.21.15.40:0/2059001636 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fd75c0b2040 con 0x55742d79c020 2024-04-21T23:20:07.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.436+0000 7fd775b93740 10 client.16161 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T23:20:07.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.436+0000 7fd775b93740 10 client.16161 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:20:07.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.436+0000 7fd775b93740 20 client.16161 populate_metadata read hostname 'smithi040' 2024-04-21T23:20:07.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.436+0000 7fd775b93740 1 --2- 172.21.15.40:0/2059001636 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55742d81c030 0x55742d81e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:07.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.436+0000 7fd775b93740 4 client.16161 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T23:20:07.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.436+0000 7fd775b93740 1 -- 172.21.15.40:0/2059001636 --> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55742d707dc0 con 0x55742d81c030 2024-04-21T23:20:07.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.437+0000 7fd76bfff640 1 --2- 172.21.15.40:0/2059001636 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55742d81c030 0x55742d81e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:07.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.437+0000 7fd76bfff640 1 --2- 172.21.15.40:0/2059001636 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55742d81c030 0x55742d81e450 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f5e6543471af030b server_cookie=9dc465519aa0ad14 in_seq=0 out_seq=0 2024-04-21T23:20:07.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.437+0000 7fd74bfff640 10 client.16161 ms_handle_connect on v2:172.21.15.111:6838/3712235548 2024-04-21T23:20:07.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.437+0000 7fd74bfff640 1 -- 172.21.15.40:0/2059001636 <== mds.0 v2:172.21.15.111:6838/3712235548 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55742d707dc0 con 0x55742d81c030 2024-04-21T23:20:07.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.437+0000 7fd74bfff640 10 client.16161 handle_command_reply: tid=0 2024-04-21T23:20:07.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.437+0000 7fd775b93740 1 client.16161 shutdown 2024-04-21T23:20:07.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.437+0000 7fd775b93740 1 -- 172.21.15.40:0/2059001636 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55742d79c020 msgr2=0x55742d7ada80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:07.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.437+0000 7fd775b93740 1 --2- 172.21.15.40:0/2059001636 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55742d79c020 0x55742d7ada80 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fd75c09ba30 tx=0x7fd75c0985d0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.437+0000 7fd775b93740 1 -- 172.21.15.40:0/2059001636 shutdown_connections 2024-04-21T23:20:07.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.437+0000 7fd775b93740 1 --2- 172.21.15.40:0/2059001636 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55742d81c030 0x55742d81e450 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.437+0000 7fd775b93740 1 --2- 172.21.15.40:0/2059001636 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55742d79c940 0x55742d79ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.437+0000 7fd775b93740 1 --2- 172.21.15.40:0/2059001636 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55742d7a5520 0x55742d79f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.437+0000 7fd775b93740 1 --2- 172.21.15.40:0/2059001636 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55742d79c020 0x55742d7ada80 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.437+0000 7fd775b93740 1 -- 172.21.15.40:0/2059001636 >> 172.21.15.40:0/2059001636 conn(0x55742d796750 msgr2=0x55742d797750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:07.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.437+0000 7fd74bfff640 0 client.16161 ms_handle_reset on v2:172.21.15.111:6838/3712235548 2024-04-21T23:20:07.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.438+0000 7fd775b93740 1 -- 172.21.15.40:0/2059001636 shutdown_connections 2024-04-21T23:20:07.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.438+0000 7fd775b93740 1 -- 172.21.15.40:0/2059001636 wait complete. 2024-04-21T23:20:07.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.438+0000 7fd775b93740 20 client.16161 trim_cache size 0 max 16384 2024-04-21T23:20:07.443 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-04-21T23:20:07.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.438+0000 7fd772ec6640 1 -- 172.21.15.40:0/1483416215 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fd7540a2d00 msgr2=0x7fd7540a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:07.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.438+0000 7fd772ec6640 1 --2- 172.21.15.40:0/1483416215 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fd7540a2d00 0x7fd7540a51c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd764005cc0 tx=0x7fd76402ce60 comp rx=0 tx=0).stop 2024-04-21T23:20:07.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.438+0000 7fd772ec6640 1 -- 172.21.15.40:0/1483416215 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd76c100610 msgr2=0x7fd76c197f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:07.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.438+0000 7fd772ec6640 1 --2- 172.21.15.40:0/1483416215 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd76c100610 0x7fd76c197f90 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fd75c00e9c0 tx=0x7fd75c00ee80 comp rx=0 tx=0).stop 2024-04-21T23:20:07.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.438+0000 7fd772ec6640 1 -- 172.21.15.40:0/1483416215 shutdown_connections 2024-04-21T23:20:07.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.438+0000 7fd772ec6640 1 --2- 172.21.15.40:0/1483416215 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fd7540a2d00 0x7fd7540a51c0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.438+0000 7fd772ec6640 1 --2- 172.21.15.40:0/1483416215 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd76c100fd0 0x7fd76c1984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.438+0000 7fd772ec6640 1 --2- 172.21.15.40:0/1483416215 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd76c1066b0 0x7fd76c19c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.438+0000 7fd772ec6640 1 --2- 172.21.15.40:0/1483416215 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd76c100610 0x7fd76c197f90 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:07.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.438+0000 7fd772ec6640 1 -- 172.21.15.40:0/1483416215 >> 172.21.15.40:0/1483416215 conn(0x7fd76c077740 msgr2=0x7fd76c0ff0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:07.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.438+0000 7fd772ec6640 1 -- 172.21.15.40:0/1483416215 shutdown_connections 2024-04-21T23:20:07.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:07.438+0000 7fd772ec6640 1 -- 172.21.15.40:0/1483416215 wait complete. 2024-04-21T23:20:08.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:07 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/350943167' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:20:08.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:07 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/350943167' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:20:08.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:07 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/350943167' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:20:08.448 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T23:20:08.448 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:20:08.448 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T23:20:08.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.558+0000 7fe1924ed640 1 -- 172.21.15.40:0/2522769199 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe18c101710 msgr2=0x7fe18c101b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:08.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.558+0000 7fe1924ed640 1 --2- 172.21.15.40:0/2522769199 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe18c101710 0x7fe18c101b90 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fe180009930 tx=0x7fe18002bc30 comp rx=0 tx=0).stop 2024-04-21T23:20:08.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.558+0000 7fe1924ed640 1 -- 172.21.15.40:0/2522769199 shutdown_connections 2024-04-21T23:20:08.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.558+0000 7fe1924ed640 1 --2- 172.21.15.40:0/2522769199 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe18c1020d0 0x7fe18c10e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:08.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.558+0000 7fe1924ed640 1 --2- 172.21.15.40:0/2522769199 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe18c101710 0x7fe18c101b90 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:08.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.558+0000 7fe1924ed640 1 --2- 172.21.15.40:0/2522769199 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe18c107720 0x7fe18c107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:08.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.558+0000 7fe1924ed640 1 -- 172.21.15.40:0/2522769199 >> 172.21.15.40:0/2522769199 conn(0x7fe18c078b90 msgr2=0x7fe18c078fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:08.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.559+0000 7fe1924ed640 1 -- 172.21.15.40:0/2522769199 shutdown_connections 2024-04-21T23:20:08.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.559+0000 7fe1924ed640 1 -- 172.21.15.40:0/2522769199 wait complete. 2024-04-21T23:20:08.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.559+0000 7fe1924ed640 1 Processor -- start 2024-04-21T23:20:08.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.559+0000 7fe1924ed640 1 -- start start 2024-04-21T23:20:08.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.559+0000 7fe1924ed640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe18c101710 0x7fe18c073a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:08.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.560+0000 7fe1924ed640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe18c1020d0 0x7fe18c073f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:08.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.560+0000 7fe1924ed640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe18c107720 0x7fe18c0744c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:08.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.560+0000 7fe1924ed640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe18c111b30 con 0x7fe18c107720 2024-04-21T23:20:08.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.560+0000 7fe1924ed640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe18c1119b0 con 0x7fe18c101710 2024-04-21T23:20:08.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.560+0000 7fe1924ed640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe18c111cb0 con 0x7fe18c1020d0 2024-04-21T23:20:08.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.560+0000 7fe190a8a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe18c107720 0x7fe18c0744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:08.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.560+0000 7fe18bfff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe18c101710 0x7fe18c073a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:08.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.560+0000 7fe18bfff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe18c101710 0x7fe18c073a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.40:53820/0 (socket says 172.21.15.40:53820) 2024-04-21T23:20:08.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.560+0000 7fe18bfff640 1 -- 172.21.15.40:0/2362277256 learned_addr learned my addr 172.21.15.40:0/2362277256 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:20:08.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.560+0000 7fe18b7fe640 1 --2- 172.21.15.40:0/2362277256 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe18c1020d0 0x7fe18c073f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:08.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.560+0000 7fe18bfff640 1 -- 172.21.15.40:0/2362277256 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe18c1020d0 msgr2=0x7fe18c073f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:08.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.560+0000 7fe18bfff640 1 --2- 172.21.15.40:0/2362277256 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe18c1020d0 0x7fe18c073f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:08.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.560+0000 7fe18bfff640 1 -- 172.21.15.40:0/2362277256 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe18c107720 msgr2=0x7fe18c0744c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:08.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.560+0000 7fe18bfff640 1 --2- 172.21.15.40:0/2362277256 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe18c107720 0x7fe18c0744c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:08.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.560+0000 7fe18bfff640 1 -- 172.21.15.40:0/2362277256 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe18c1a65b0 con 0x7fe18c101710 2024-04-21T23:20:08.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.560+0000 7fe190a8a640 1 --2- 172.21.15.40:0/2362277256 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe18c107720 0x7fe18c0744c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:20:08.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.560+0000 7fe18b7fe640 1 --2- 172.21.15.40:0/2362277256 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe18c1020d0 0x7fe18c073f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:20:08.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.560+0000 7fe18bfff640 1 --2- 172.21.15.40:0/2362277256 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe18c101710 0x7fe18c073a40 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe17c00d770 tx=0x7fe17c00dc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:08.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.560+0000 7fe1897fa640 1 -- 172.21.15.40:0/2362277256 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe17c017070 con 0x7fe18c101710 2024-04-21T23:20:08.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.560+0000 7fe1897fa640 1 -- 172.21.15.40:0/2362277256 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe17c005500 con 0x7fe18c101710 2024-04-21T23:20:08.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.560+0000 7fe1924ed640 1 -- 172.21.15.40:0/2362277256 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe18c1a68a0 con 0x7fe18c101710 2024-04-21T23:20:08.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.560+0000 7fe1897fa640 1 -- 172.21.15.40:0/2362277256 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe17c004e00 con 0x7fe18c101710 2024-04-21T23:20:08.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.560+0000 7fe1924ed640 1 -- 172.21.15.40:0/2362277256 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe18c1a6da0 con 0x7fe18c101710 2024-04-21T23:20:08.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.563+0000 7fe1897fa640 1 -- 172.21.15.40:0/2362277256 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fe17c004fa0 con 0x7fe18c101710 2024-04-21T23:20:08.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.563+0000 7fe1951b5740 1 Processor -- start 2024-04-21T23:20:08.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.563+0000 7fe1951b5740 1 -- start start 2024-04-21T23:20:08.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.563+0000 7fe1951b5740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d6715fe720 0x55d6715fd470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:08.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.563+0000 7fe1951b5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d6715fda40 0x55d6715fdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:08.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.563+0000 7fe1951b5740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d671606db0 0x55d6716091a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:08.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.563+0000 7fe1951b5740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55d671561170 con 0x55d6715fe720 2024-04-21T23:20:08.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.563+0000 7fe1951b5740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55d6715097c0 con 0x55d671606db0 2024-04-21T23:20:08.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.563+0000 7fe1951b5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55d67155d130 con 0x55d6715fda40 2024-04-21T23:20:08.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.563+0000 7fe190a8a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d6715fe720 0x55d6715fd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:08.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.563+0000 7fe190a8a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d6715fe720 0x55d6715fd470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37320/0 (socket says 172.21.15.40:37320) 2024-04-21T23:20:08.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.563+0000 7fe190a8a640 1 -- 172.21.15.40:0/2893282844 learned_addr learned my addr 172.21.15.40:0/2893282844 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:20:08.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.563+0000 7fe18b7fe640 1 --2- 172.21.15.40:0/2893282844 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d671606db0 0x55d6716091a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:08.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.563+0000 7fe1897fa640 1 --2- 172.21.15.40:0/2362277256 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fe1740a2d00 0x7fe1740a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:08.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.564+0000 7fe190a8a640 1 --2- 172.21.15.40:0/2362277256 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fe1740a2d00 0x7fe1740a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:08.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.564+0000 7fe1897fa640 1 -- 172.21.15.40:0/2362277256 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7fe17c0cf220 con 0x7fe18c101710 2024-04-21T23:20:08.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.564+0000 7fe18bfff640 1 --2- 172.21.15.40:0/2893282844 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d6715fda40 0x55d6715fdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:08.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.564+0000 7fe190a8a640 1 -- 172.21.15.40:0/2893282844 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d6715fda40 msgr2=0x55d6715fdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:08.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.564+0000 7fe190a8a640 1 --2- 172.21.15.40:0/2893282844 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d6715fda40 0x55d6715fdec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:08.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.564+0000 7fe190a8a640 1 -- 172.21.15.40:0/2893282844 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d671606db0 msgr2=0x55d6716091a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:08.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.564+0000 7fe190a8a640 1 --2- 172.21.15.40:0/2893282844 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d671606db0 0x55d6716091a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:08.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.564+0000 7fe190a8a640 1 -- 172.21.15.40:0/2893282844 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d67151ae60 con 0x55d6715fe720 2024-04-21T23:20:08.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.564+0000 7fe18b7fe640 1 --2- 172.21.15.40:0/2893282844 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d671606db0 0x55d6716091a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:20:08.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.564+0000 7fe18bfff640 1 --2- 172.21.15.40:0/2893282844 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d6715fda40 0x55d6715fdec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:20:08.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.564+0000 7fe190a8a640 1 --2- 172.21.15.40:0/2893282844 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d6715fe720 0x55d6715fd470 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fe184009fd0 tx=0x7fe184013830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94bb030b30a027ee server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:08.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.564+0000 7fe190a8a640 1 --2- 172.21.15.40:0/2362277256 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fe1740a2d00 0x7fe1740a51c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fe18401d000 tx=0x7fe18401f000 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:08.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.564+0000 7fe1727fc640 1 -- 172.21.15.40:0/2893282844 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe18400f8b0 con 0x55d6715fe720 2024-04-21T23:20:08.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.564+0000 7fe1727fc640 1 -- 172.21.15.40:0/2893282844 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe184022020 con 0x55d6715fe720 2024-04-21T23:20:08.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.564+0000 7fe1727fc640 1 -- 172.21.15.40:0/2893282844 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe184010a00 con 0x55d6715fe720 2024-04-21T23:20:08.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.565+0000 7fe1951b5740 1 -- 172.21.15.40:0/2893282844 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d6715fe720 msgr2=0x55d6715fd470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:08.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.565+0000 7fe1951b5740 1 --2- 172.21.15.40:0/2893282844 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d6715fe720 0x55d6715fd470 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fe184009fd0 tx=0x7fe184013830 comp rx=0 tx=0).stop 2024-04-21T23:20:08.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.565+0000 7fe1951b5740 1 -- 172.21.15.40:0/2893282844 shutdown_connections 2024-04-21T23:20:08.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.565+0000 7fe1951b5740 1 --2- 172.21.15.40:0/2893282844 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d671606db0 0x55d6716091a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:08.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.565+0000 7fe1951b5740 1 --2- 172.21.15.40:0/2893282844 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d6715fda40 0x55d6715fdec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:08.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.565+0000 7fe1951b5740 1 --2- 172.21.15.40:0/2893282844 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d6715fe720 0x55d6715fd470 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:08.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.565+0000 7fe1951b5740 1 -- 172.21.15.40:0/2893282844 >> 172.21.15.40:0/2893282844 conn(0x55d6715c44f0 msgr2=0x55d6715f8490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:08.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.565+0000 7fe1951b5740 1 -- 172.21.15.40:0/2893282844 shutdown_connections 2024-04-21T23:20:08.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.565+0000 7fe1951b5740 1 -- 172.21.15.40:0/2893282844 wait complete. 2024-04-21T23:20:08.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.565+0000 7fe1951b5740 1 -- start start 2024-04-21T23:20:08.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.565+0000 7fe1951b5740 1 Processor -- start 2024-04-21T23:20:08.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.566+0000 7fe1951b5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d6715fda40 0x55d671603e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:08.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.566+0000 7fe1951b5740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d6715fe720 0x55d671604340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:08.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.566+0000 7fe1951b5740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d671606db0 0x55d671604880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:08.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.566+0000 7fe1951b5740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55d671609bc0 con 0x55d6715fe720 2024-04-21T23:20:08.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.566+0000 7fe1951b5740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55d67151cd40 con 0x55d671606db0 2024-04-21T23:20:08.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.566+0000 7fe1951b5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55d671609d40 con 0x55d6715fda40 2024-04-21T23:20:08.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.566+0000 7fe1951b5740 10 client.0 _ll_register_callbacks cb 0x55d671558820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:20:08.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.566+0000 7fe18b7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d6715fe720 0x55d671604340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:08.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.566+0000 7fe190a8a640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d671606db0 0x55d671604880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:08.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.566+0000 7fe18b7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d6715fe720 0x55d671604340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37330/0 (socket says 172.21.15.40:37330) 2024-04-21T23:20:08.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.566+0000 7fe18b7fe640 1 -- 172.21.15.40:0/2593950283 learned_addr learned my addr 172.21.15.40:0/2593950283 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:20:08.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.566+0000 7fe18bfff640 1 --2- 172.21.15.40:0/2593950283 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d6715fda40 0x55d671603e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:08.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.566+0000 7fe18b7fe640 1 -- 172.21.15.40:0/2593950283 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d6715fda40 msgr2=0x55d671603e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:08.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.566+0000 7fe18b7fe640 1 --2- 172.21.15.40:0/2593950283 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d6715fda40 0x55d671603e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:08.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.566+0000 7fe18b7fe640 1 -- 172.21.15.40:0/2593950283 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d671606db0 msgr2=0x55d671604880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:08.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.566+0000 7fe18b7fe640 1 --2- 172.21.15.40:0/2593950283 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d671606db0 0x55d671604880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:08.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.566+0000 7fe18b7fe640 1 -- 172.21.15.40:0/2593950283 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d67133d7e0 con 0x55d6715fe720 2024-04-21T23:20:08.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.566+0000 7fe190a8a640 1 --2- 172.21.15.40:0/2593950283 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d671606db0 0x55d671604880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:20:08.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.566+0000 7fe18b7fe640 1 --2- 172.21.15.40:0/2593950283 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d6715fe720 0x55d671604340 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fe18002cfd0 tx=0x7fe180005d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1062c3624688346f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:08.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.566+0000 7fe16b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-21T23:20:08.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.566+0000 7fe16b7fe640 1 -- 172.21.15.40:0/2593950283 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe180005030 con 0x55d6715fe720 2024-04-21T23:20:08.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.566+0000 7fe16b7fe640 1 -- 172.21.15.40:0/2593950283 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7fe1800051d0 con 0x55d6715fe720 2024-04-21T23:20:08.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.567+0000 7fe1951b5740 1 -- 172.21.15.40:0/2593950283 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe184022020 con 0x55d6715fe720 2024-04-21T23:20:08.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.567+0000 7fe16b7fe640 1 -- 172.21.15.40:0/2593950283 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe18004c050 con 0x55d6715fe720 2024-04-21T23:20:08.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.567+0000 7fe1951b5740 10 client.16173 fetch_fsmap learned FSMap version 28 2024-04-21T23:20:08.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.567+0000 7fe1951b5740 1 -- 172.21.15.40:0/2593950283 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d67139df70 con 0x55d6715fe720 2024-04-21T23:20:08.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.567+0000 7fe16b7fe640 1 -- 172.21.15.40:0/2593950283 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe184022020 con 0x55d6715fe720 2024-04-21T23:20:08.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.567+0000 7fe16b7fe640 1 -- 172.21.15.40:0/2593950283 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fe1800035f0 con 0x55d6715fe720 2024-04-21T23:20:08.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.567+0000 7fe1951b5740 10 client.16173 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T23:20:08.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.567+0000 7fe1951b5740 10 client.16173 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:20:08.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.567+0000 7fe1951b5740 20 client.16173 populate_metadata read hostname 'smithi040' 2024-04-21T23:20:08.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.567+0000 7fe1951b5740 1 --2- 172.21.15.40:0/2593950283 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55d671616f00 0x55d671619320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:08.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.567+0000 7fe1951b5740 4 client.16173 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:20:08.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.567+0000 7fe1951b5740 1 -- 172.21.15.40:0/2593950283 --> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d671535450 con 0x55d671616f00 2024-04-21T23:20:08.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.567+0000 7fe18bfff640 1 --2- 172.21.15.40:0/2593950283 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55d671616f00 0x55d671619320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:08.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.568+0000 7fe18bfff640 1 --2- 172.21.15.40:0/2593950283 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55d671616f00 0x55d671619320 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d3d43a10d512e5b0 server_cookie=e6e38747ac027023 in_seq=0 out_seq=0 2024-04-21T23:20:08.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.568+0000 7fe16b7fe640 10 client.16173 ms_handle_connect on v2:172.21.15.111:6838/3712235548 2024-04-21T23:20:08.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.569+0000 7fe16b7fe640 1 -- 172.21.15.40:0/2593950283 <== mds.0 v2:172.21.15.111:6838/3712235548 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d671535450 con 0x55d671616f00 2024-04-21T23:20:08.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.569+0000 7fe16b7fe640 10 client.16173 handle_command_reply: tid=0 2024-04-21T23:20:08.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.569+0000 7fe1951b5740 1 client.16173 shutdown 2024-04-21T23:20:08.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.569+0000 7fe1951b5740 1 -- 172.21.15.40:0/2593950283 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d6715fe720 msgr2=0x55d671604340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:08.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.569+0000 7fe1951b5740 1 --2- 172.21.15.40:0/2593950283 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d6715fe720 0x55d671604340 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fe18002cfd0 tx=0x7fe180005d70 comp rx=0 tx=0).stop 2024-04-21T23:20:08.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.569+0000 7fe1951b5740 1 -- 172.21.15.40:0/2593950283 shutdown_connections 2024-04-21T23:20:08.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.569+0000 7fe1951b5740 1 --2- 172.21.15.40:0/2593950283 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55d671616f00 0x55d671619320 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:08.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.569+0000 7fe1951b5740 1 --2- 172.21.15.40:0/2593950283 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d671606db0 0x55d671604880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:08.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.569+0000 7fe1951b5740 1 --2- 172.21.15.40:0/2593950283 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d6715fe720 0x55d671604340 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:08.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.569+0000 7fe16b7fe640 0 client.16173 ms_handle_reset on v2:172.21.15.111:6838/3712235548 2024-04-21T23:20:08.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.569+0000 7fe1951b5740 1 --2- 172.21.15.40:0/2593950283 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d6715fda40 0x55d671603e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:08.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.569+0000 7fe1951b5740 1 -- 172.21.15.40:0/2593950283 >> 172.21.15.40:0/2593950283 conn(0x55d6715c44f0 msgr2=0x55d6715fa1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:08.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.569+0000 7fe1951b5740 1 -- 172.21.15.40:0/2593950283 shutdown_connections 2024-04-21T23:20:08.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.569+0000 7fe1951b5740 1 -- 172.21.15.40:0/2593950283 wait complete. 2024-04-21T23:20:08.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.569+0000 7fe1951b5740 20 client.16173 trim_cache size 0 max 16384 2024-04-21T23:20:08.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.588+0000 7fe1951b5740 1 Processor -- start 2024-04-21T23:20:08.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.588+0000 7fe1951b5740 1 -- start start 2024-04-21T23:20:08.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.588+0000 7fe1951b5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d6715fda40 0x55d671612af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:08.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.588+0000 7fe1951b5740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d6715fe720 0x55d6716046a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:08.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.588+0000 7fe1951b5740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d671606db0 0x55d6715f7370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:08.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.588+0000 7fe1951b5740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe18c1b37f0 con 0x55d671606db0 2024-04-21T23:20:08.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.588+0000 7fe1951b5740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe18c1b1060 con 0x55d6715fe720 2024-04-21T23:20:08.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.588+0000 7fe1951b5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe18c1aeeb0 con 0x55d6715fda40 2024-04-21T23:20:08.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.588+0000 7fe190a8a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d671606db0 0x55d6715f7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:08.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.588+0000 7fe190a8a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d671606db0 0x55d6715f7370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37346/0 (socket says 172.21.15.40:37346) 2024-04-21T23:20:08.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.588+0000 7fe190a8a640 1 -- 172.21.15.40:0/117843963 learned_addr learned my addr 172.21.15.40:0/117843963 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:20:08.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.588+0000 7fe18b7fe640 1 --2- 172.21.15.40:0/117843963 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d6715fe720 0x55d6716046a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:08.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.588+0000 7fe18bfff640 1 --2- 172.21.15.40:0/117843963 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d6715fda40 0x55d671612af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:08.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.589+0000 7fe190a8a640 1 -- 172.21.15.40:0/117843963 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d6715fda40 msgr2=0x55d671612af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:08.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.589+0000 7fe190a8a640 1 --2- 172.21.15.40:0/117843963 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d6715fda40 0x55d671612af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:08.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.589+0000 7fe190a8a640 1 -- 172.21.15.40:0/117843963 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d6715fe720 msgr2=0x55d6716046a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:08.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.589+0000 7fe190a8a640 1 --2- 172.21.15.40:0/117843963 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d6715fe720 0x55d6716046a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:08.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.589+0000 7fe190a8a640 1 -- 172.21.15.40:0/117843963 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d671602db0 con 0x55d671606db0 2024-04-21T23:20:08.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.589+0000 7fe18bfff640 1 --2- 172.21.15.40:0/117843963 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d6715fda40 0x55d671612af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:20:08.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.589+0000 7fe190a8a640 1 --2- 172.21.15.40:0/117843963 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d671606db0 0x55d6715f7370 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fe184009560 tx=0x7fe18400fdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a86be25fe42c29f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:08.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.589+0000 7fe171ffb640 1 -- 172.21.15.40:0/117843963 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe184010670 con 0x55d671606db0 2024-04-21T23:20:08.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.589+0000 7fe171ffb640 1 -- 172.21.15.40:0/117843963 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe184022070 con 0x55d671606db0 2024-04-21T23:20:08.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.589+0000 7fe171ffb640 1 -- 172.21.15.40:0/117843963 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1840055b0 con 0x55d671606db0 2024-04-21T23:20:08.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.590+0000 7fe1951b5740 1 -- 172.21.15.40:0/117843963 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d671606db0 msgr2=0x55d6715f7370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:08.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.590+0000 7fe1951b5740 1 --2- 172.21.15.40:0/117843963 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d671606db0 0x55d6715f7370 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fe184009560 tx=0x7fe18400fdc0 comp rx=0 tx=0).stop 2024-04-21T23:20:08.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.590+0000 7fe1951b5740 1 -- 172.21.15.40:0/117843963 shutdown_connections 2024-04-21T23:20:08.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.590+0000 7fe1951b5740 1 --2- 172.21.15.40:0/117843963 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d6715fe720 0x55d6716046a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:08.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.590+0000 7fe1951b5740 1 --2- 172.21.15.40:0/117843963 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d671606db0 0x55d6715f7370 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:08.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.590+0000 7fe1951b5740 1 --2- 172.21.15.40:0/117843963 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d6715fda40 0x55d671612af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:08.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.590+0000 7fe1951b5740 1 -- 172.21.15.40:0/117843963 >> 172.21.15.40:0/117843963 conn(0x55d6715c44f0 msgr2=0x55d6715f6c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:08.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.590+0000 7fe1951b5740 1 -- 172.21.15.40:0/117843963 shutdown_connections 2024-04-21T23:20:08.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.590+0000 7fe1951b5740 1 -- 172.21.15.40:0/117843963 wait complete. 2024-04-21T23:20:08.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.590+0000 7fe1951b5740 1 -- start start 2024-04-21T23:20:08.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.590+0000 7fe1951b5740 1 Processor -- start 2024-04-21T23:20:08.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.590+0000 7fe1951b5740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d6715fda40 0x55d671607c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:08.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.590+0000 7fe1951b5740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d6715fe720 0x55d671608160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:08.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.591+0000 7fe1951b5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d671606db0 0x55d6716086a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:08.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.591+0000 7fe1951b5740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe18c1af520 con 0x55d6715fda40 2024-04-21T23:20:08.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.591+0000 7fe1951b5740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe18c1b0990 con 0x55d6715fe720 2024-04-21T23:20:08.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.591+0000 7fe1951b5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe18c1afbf0 con 0x55d671606db0 2024-04-21T23:20:08.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.591+0000 7fe1951b5740 10 client.0 _ll_register_callbacks cb 0x7fe180009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:20:08.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.591+0000 7fe18bfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d6715fda40 0x55d671607c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:08.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.591+0000 7fe18bfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d6715fda40 0x55d671607c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37358/0 (socket says 172.21.15.40:37358) 2024-04-21T23:20:08.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.591+0000 7fe18bfff640 1 -- 172.21.15.40:0/3883692796 learned_addr learned my addr 172.21.15.40:0/3883692796 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:20:08.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.591+0000 7fe18b7fe640 1 --2- 172.21.15.40:0/3883692796 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d6715fe720 0x55d671608160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:08.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.591+0000 7fe18bfff640 1 -- 172.21.15.40:0/3883692796 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d671606db0 msgr2=0x55d6716086a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:08.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.591+0000 7fe18bfff640 1 --2- 172.21.15.40:0/3883692796 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d671606db0 0x55d6716086a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:08.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.591+0000 7fe18bfff640 1 -- 172.21.15.40:0/3883692796 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d6715fe720 msgr2=0x55d671608160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:08.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.591+0000 7fe18bfff640 1 --2- 172.21.15.40:0/3883692796 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d6715fe720 0x55d671608160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:08.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.591+0000 7fe18bfff640 1 -- 172.21.15.40:0/3883692796 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d671235170 con 0x55d6715fda40 2024-04-21T23:20:08.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.591+0000 7fe18bfff640 1 --2- 172.21.15.40:0/3883692796 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d6715fda40 0x55d671607c20 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fe17c009610 tx=0x7fe17c0bb040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea29e1e11eb5305e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:08.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.591+0000 7fe16b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-21T23:20:08.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.591+0000 7fe16b7fe640 1 -- 172.21.15.40:0/3883692796 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe17c020040 con 0x55d6715fda40 2024-04-21T23:20:08.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.591+0000 7fe16b7fe640 1 -- 172.21.15.40:0/3883692796 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7fe17c009cc0 con 0x55d6715fda40 2024-04-21T23:20:08.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.591+0000 7fe1951b5740 1 -- 172.21.15.40:0/3883692796 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe184022070 con 0x55d6715fda40 2024-04-21T23:20:08.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.591+0000 7fe16b7fe640 1 -- 172.21.15.40:0/3883692796 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe17c09f740 con 0x55d6715fda40 2024-04-21T23:20:08.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.591+0000 7fe1951b5740 10 client.16179 fetch_fsmap learned FSMap version 28 2024-04-21T23:20:08.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.591+0000 7fe1951b5740 1 -- 172.21.15.40:0/3883692796 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d67167d830 con 0x55d6715fda40 2024-04-21T23:20:08.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.592+0000 7fe16b7fe640 1 -- 172.21.15.40:0/3883692796 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe184022070 con 0x55d6715fda40 2024-04-21T23:20:08.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.592+0000 7fe16b7fe640 1 -- 172.21.15.40:0/3883692796 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fe17c097a00 con 0x55d6715fda40 2024-04-21T23:20:08.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.592+0000 7fe1951b5740 10 client.16179 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T23:20:08.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.592+0000 7fe1951b5740 10 client.16179 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:20:08.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.592+0000 7fe1951b5740 20 client.16179 populate_metadata read hostname 'smithi040' 2024-04-21T23:20:08.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.592+0000 7fe1951b5740 1 --2- 172.21.15.40:0/3883692796 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55d67167dfc0 0x55d671680400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:08.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.592+0000 7fe1951b5740 4 client.16179 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T23:20:08.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.592+0000 7fe1951b5740 1 -- 172.21.15.40:0/3883692796 --> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55d67156e6b0 con 0x55d67167dfc0 2024-04-21T23:20:08.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.592+0000 7fe18b7fe640 1 --2- 172.21.15.40:0/3883692796 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55d67167dfc0 0x55d671680400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:08.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.592+0000 7fe16b7fe640 10 client.16179 ms_handle_connect on v2:172.21.15.111:6838/3712235548 2024-04-21T23:20:08.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.592+0000 7fe18b7fe640 1 --2- 172.21.15.40:0/3883692796 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55d67167dfc0 0x55d671680400 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6fd2ccb831b31022 server_cookie=dc4662cdb0eb6bfc in_seq=0 out_seq=0 2024-04-21T23:20:08.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.593+0000 7fe16b7fe640 1 -- 172.21.15.40:0/3883692796 <== mds.0 v2:172.21.15.111:6838/3712235548 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55d67156e6b0 con 0x55d67167dfc0 2024-04-21T23:20:08.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.593+0000 7fe16b7fe640 10 client.16179 handle_command_reply: tid=0 2024-04-21T23:20:08.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.593+0000 7fe1951b5740 1 client.16179 shutdown 2024-04-21T23:20:08.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.593+0000 7fe1951b5740 1 -- 172.21.15.40:0/3883692796 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d6715fda40 msgr2=0x55d671607c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:08.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.593+0000 7fe1951b5740 1 --2- 172.21.15.40:0/3883692796 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d6715fda40 0x55d671607c20 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fe17c009610 tx=0x7fe17c0bb040 comp rx=0 tx=0).stop 2024-04-21T23:20:08.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.593+0000 7fe1951b5740 1 -- 172.21.15.40:0/3883692796 shutdown_connections 2024-04-21T23:20:08.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.593+0000 7fe1951b5740 1 --2- 172.21.15.40:0/3883692796 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55d67167dfc0 0x55d671680400 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:08.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.593+0000 7fe1951b5740 1 --2- 172.21.15.40:0/3883692796 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d6715fe720 0x55d671608160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:08.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.593+0000 7fe1951b5740 1 --2- 172.21.15.40:0/3883692796 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d671606db0 0x55d6716086a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:08.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.593+0000 7fe1951b5740 1 --2- 172.21.15.40:0/3883692796 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d6715fda40 0x55d671607c20 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:08.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.593+0000 7fe16b7fe640 0 client.16179 ms_handle_reset on v2:172.21.15.111:6838/3712235548 2024-04-21T23:20:08.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.593+0000 7fe1951b5740 1 -- 172.21.15.40:0/3883692796 >> 172.21.15.40:0/3883692796 conn(0x55d6715c44f0 msgr2=0x55d6715f74f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:08.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.593+0000 7fe1951b5740 1 -- 172.21.15.40:0/3883692796 shutdown_connections 2024-04-21T23:20:08.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.593+0000 7fe1951b5740 1 -- 172.21.15.40:0/3883692796 wait complete. 2024-04-21T23:20:08.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.593+0000 7fe1951b5740 20 client.16179 trim_cache size 0 max 16384 2024-04-21T23:20:08.601 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-21T23:20:08.601 INFO:teuthology.orchestra.run.smithi040.stdout: "return_code": 0, 2024-04-21T23:20:08.601 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_tag": "f1345386-ae92-4f68-9149-480c1be2784f", 2024-04-21T23:20:08.601 INFO:teuthology.orchestra.run.smithi040.stdout: "mode": "asynchronous" 2024-04-21T23:20:08.601 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-21T23:20:08.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.594+0000 7fe1924ed640 1 -- 172.21.15.40:0/2362277256 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fe1740a2d00 msgr2=0x7fe1740a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:08.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.594+0000 7fe1924ed640 1 --2- 172.21.15.40:0/2362277256 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fe1740a2d00 0x7fe1740a51c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fe18401d000 tx=0x7fe18401f000 comp rx=0 tx=0).stop 2024-04-21T23:20:08.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.594+0000 7fe1924ed640 1 -- 172.21.15.40:0/2362277256 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe18c101710 msgr2=0x7fe18c073a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:08.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.594+0000 7fe1924ed640 1 --2- 172.21.15.40:0/2362277256 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe18c101710 0x7fe18c073a40 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe17c00d770 tx=0x7fe17c00dc30 comp rx=0 tx=0).stop 2024-04-21T23:20:08.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.594+0000 7fe1924ed640 1 -- 172.21.15.40:0/2362277256 shutdown_connections 2024-04-21T23:20:08.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.594+0000 7fe1924ed640 1 --2- 172.21.15.40:0/2362277256 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fe1740a2d00 0x7fe1740a51c0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:08.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.594+0000 7fe1924ed640 1 --2- 172.21.15.40:0/2362277256 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe18c107720 0x7fe18c0744c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:08.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.594+0000 7fe1924ed640 1 --2- 172.21.15.40:0/2362277256 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe18c1020d0 0x7fe18c073f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:08.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.594+0000 7fe1924ed640 1 --2- 172.21.15.40:0/2362277256 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe18c101710 0x7fe18c073a40 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:08.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.594+0000 7fe1924ed640 1 -- 172.21.15.40:0/2362277256 >> 172.21.15.40:0/2362277256 conn(0x7fe18c078b90 msgr2=0x7fe18c0ff1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:08.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.594+0000 7fe1924ed640 1 -- 172.21.15.40:0/2362277256 shutdown_connections 2024-04-21T23:20:08.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.594+0000 7fe1924ed640 1 -- 172.21.15.40:0/2362277256 wait complete. 2024-04-21T23:20:08.605 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T23:20:08.605 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T23:20:08.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.712+0000 7f32be22e640 1 -- 172.21.15.40:0/878863031 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32b8107720 msgr2=0x7f32b8107b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:08.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.712+0000 7f32be22e640 1 --2- 172.21.15.40:0/878863031 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32b8107720 0x7f32b8107b00 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f32a8009930 tx=0x7f32a802bc30 comp rx=0 tx=0).stop 2024-04-21T23:20:08.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.712+0000 7f32be22e640 1 -- 172.21.15.40:0/878863031 shutdown_connections 2024-04-21T23:20:08.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.712+0000 7f32be22e640 1 --2- 172.21.15.40:0/878863031 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32b8101710 0x7f32b8101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:08.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.712+0000 7f32be22e640 1 --2- 172.21.15.40:0/878863031 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f32b81020d0 0x7f32b810e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:08.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.712+0000 7f32be22e640 1 --2- 172.21.15.40:0/878863031 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32b8107720 0x7f32b8107b00 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:08.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.712+0000 7f32be22e640 1 -- 172.21.15.40:0/878863031 >> 172.21.15.40:0/878863031 conn(0x7f32b80fd180 msgr2=0x7f32b80ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:08.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.712+0000 7f32be22e640 1 -- 172.21.15.40:0/878863031 shutdown_connections 2024-04-21T23:20:08.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.712+0000 7f32be22e640 1 -- 172.21.15.40:0/878863031 wait complete. 2024-04-21T23:20:08.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.713+0000 7f32be22e640 1 Processor -- start 2024-04-21T23:20:08.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.713+0000 7f32be22e640 1 -- start start 2024-04-21T23:20:08.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.713+0000 7f32be22e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32b8101710 0x7f32b8073a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:08.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.713+0000 7f32be22e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32b81020d0 0x7f32b8073f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:08.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.713+0000 7f32be22e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f32b8107720 0x7f32b80744c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:08.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.713+0000 7f32be22e640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f32b8111af0 con 0x7f32b81020d0 2024-04-21T23:20:08.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.713+0000 7f32be22e640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f32b8111970 con 0x7f32b8101710 2024-04-21T23:20:08.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.713+0000 7f32be22e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f32b8111c70 con 0x7f32b8107720 2024-04-21T23:20:08.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.713+0000 7f32b6ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32b81020d0 0x7f32b8073f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:08.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.713+0000 7f32b6ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32b81020d0 0x7f32b8073f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37366/0 (socket says 172.21.15.40:37366) 2024-04-21T23:20:08.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.713+0000 7f32b6ffd640 1 -- 172.21.15.40:0/317838875 learned_addr learned my addr 172.21.15.40:0/317838875 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:20:08.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.713+0000 7f32b77fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32b8101710 0x7f32b8073a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:08.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.713+0000 7f32b7fff640 1 --2- 172.21.15.40:0/317838875 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f32b8107720 0x7f32b80744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:08.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.713+0000 7f32b6ffd640 1 -- 172.21.15.40:0/317838875 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f32b8107720 msgr2=0x7f32b80744c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:08.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.713+0000 7f32b6ffd640 1 --2- 172.21.15.40:0/317838875 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f32b8107720 0x7f32b80744c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:08.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.714+0000 7f32b6ffd640 1 -- 172.21.15.40:0/317838875 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32b8101710 msgr2=0x7f32b8073a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:08.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.714+0000 7f32b6ffd640 1 --2- 172.21.15.40:0/317838875 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32b8101710 0x7f32b8073a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:08.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.714+0000 7f32b6ffd640 1 -- 172.21.15.40:0/317838875 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32b81a6530 con 0x7f32b81020d0 2024-04-21T23:20:08.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.714+0000 7f32b77fe640 1 --2- 172.21.15.40:0/317838875 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32b8101710 0x7f32b8073a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:20:08.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.714+0000 7f32b6ffd640 1 --2- 172.21.15.40:0/317838875 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32b81020d0 0x7f32b8073f80 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f32ac00e9f0 tx=0x7f32ac00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:08.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.714+0000 7f32b4ff9640 1 -- 172.21.15.40:0/317838875 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32ac00cd80 con 0x7f32b81020d0 2024-04-21T23:20:08.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.714+0000 7f32be22e640 1 -- 172.21.15.40:0/317838875 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32b81a6820 con 0x7f32b81020d0 2024-04-21T23:20:08.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.714+0000 7f32b4ff9640 1 -- 172.21.15.40:0/317838875 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32ac010910 con 0x7f32b81020d0 2024-04-21T23:20:08.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.714+0000 7f32b4ff9640 1 -- 172.21.15.40:0/317838875 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32ac0188f0 con 0x7f32b81020d0 2024-04-21T23:20:08.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.714+0000 7f32be22e640 1 -- 172.21.15.40:0/317838875 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32b81a6d70 con 0x7f32b81020d0 2024-04-21T23:20:08.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.716+0000 7f32b4ff9640 1 -- 172.21.15.40:0/317838875 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f32ac010ac0 con 0x7f32b81020d0 2024-04-21T23:20:08.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.716+0000 7f32c0e14740 1 Processor -- start 2024-04-21T23:20:08.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.716+0000 7f32c0e14740 1 -- start start 2024-04-21T23:20:08.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.716+0000 7f32c0e14740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560a49453f30 0x560a49454310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:08.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.716+0000 7f32c0e14740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a49454850 0x560a4945cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:08.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.716+0000 7f32c0e14740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a4945d430 0x560a4945f820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:08.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.716+0000 7f32c0e14740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x560a493b8560 con 0x560a49454850 2024-04-21T23:20:08.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.716+0000 7f32c0e14740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x560a49360730 con 0x560a4945d430 2024-04-21T23:20:08.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.716+0000 7f32c0e14740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x560a493b4520 con 0x560a49453f30 2024-04-21T23:20:08.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.717+0000 7f32b77fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a49454850 0x560a4945cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:08.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.717+0000 7f32b77fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a49454850 0x560a4945cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37378/0 (socket says 172.21.15.40:37378) 2024-04-21T23:20:08.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.717+0000 7f32b4ff9640 1 --2- 172.21.15.40:0/317838875 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f32a00a2d00 0x7f32a00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:08.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.717+0000 7f32b77fe640 1 -- 172.21.15.40:0/4028897317 learned_addr learned my addr 172.21.15.40:0/4028897317 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:20:08.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.717+0000 7f32b6ffd640 1 --2- 172.21.15.40:0/4028897317 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a4945d430 0x560a4945f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:08.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.717+0000 7f32b7fff640 1 --2- 172.21.15.40:0/4028897317 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560a49453f30 0x560a49454310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:08.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.717+0000 7f32b7fff640 1 --2- 172.21.15.40:0/317838875 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f32a00a2d00 0x7f32a00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:08.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.717+0000 7f32b77fe640 1 -- 172.21.15.40:0/4028897317 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560a49453f30 msgr2=0x560a49454310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:08.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.717+0000 7f32b77fe640 1 --2- 172.21.15.40:0/4028897317 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560a49453f30 0x560a49454310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:08.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.717+0000 7f32b4ff9640 1 -- 172.21.15.40:0/317838875 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f32ac0cfc40 con 0x7f32b81020d0 2024-04-21T23:20:08.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.717+0000 7f32b77fe640 1 -- 172.21.15.40:0/4028897317 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a4945d430 msgr2=0x560a4945f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:08.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.717+0000 7f32b77fe640 1 --2- 172.21.15.40:0/4028897317 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a4945d430 0x560a4945f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:08.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.717+0000 7f32b77fe640 1 -- 172.21.15.40:0/4028897317 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560a491967d0 con 0x560a49454850 2024-04-21T23:20:08.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.717+0000 7f32b77fe640 1 --2- 172.21.15.40:0/4028897317 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a49454850 0x560a4945cef0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f32a8006010 tx=0x7f32a8005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22eca298af9bb0b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:08.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.717+0000 7f32b7fff640 1 --2- 172.21.15.40:0/317838875 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f32a00a2d00 0x7f32a00a51c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f32b0009370 tx=0x7f32b0009240 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:08.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.717+0000 7f329dffb640 1 -- 172.21.15.40:0/4028897317 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32a8042070 con 0x560a49454850 2024-04-21T23:20:08.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.718+0000 7f329dffb640 1 -- 172.21.15.40:0/4028897317 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32a8004a70 con 0x560a49454850 2024-04-21T23:20:08.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.718+0000 7f329dffb640 1 -- 172.21.15.40:0/4028897317 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32a8005130 con 0x560a49454850 2024-04-21T23:20:08.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.718+0000 7f32c0e14740 1 -- 172.21.15.40:0/4028897317 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a49454850 msgr2=0x560a4945cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:08.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.718+0000 7f32c0e14740 1 --2- 172.21.15.40:0/4028897317 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a49454850 0x560a4945cef0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f32a8006010 tx=0x7f32a8005d50 comp rx=0 tx=0).stop 2024-04-21T23:20:08.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.718+0000 7f32c0e14740 1 -- 172.21.15.40:0/4028897317 shutdown_connections 2024-04-21T23:20:08.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.718+0000 7f32c0e14740 1 --2- 172.21.15.40:0/4028897317 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a4945d430 0x560a4945f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:08.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.718+0000 7f32c0e14740 1 --2- 172.21.15.40:0/4028897317 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a49454850 0x560a4945cef0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:08.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.718+0000 7f32c0e14740 1 --2- 172.21.15.40:0/4028897317 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560a49453f30 0x560a49454310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:08.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.718+0000 7f32c0e14740 1 -- 172.21.15.40:0/4028897317 >> 172.21.15.40:0/4028897317 conn(0x560a4944e660 msgr2=0x560a4944ea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:08.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.718+0000 7f32c0e14740 1 -- 172.21.15.40:0/4028897317 shutdown_connections 2024-04-21T23:20:08.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.718+0000 7f32c0e14740 1 -- 172.21.15.40:0/4028897317 wait complete. 2024-04-21T23:20:08.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.719+0000 7f32c0e14740 1 -- start start 2024-04-21T23:20:08.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.719+0000 7f32c0e14740 1 Processor -- start 2024-04-21T23:20:08.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.719+0000 7f32c0e14740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560a49453f30 0x560a4945a470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:08.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.719+0000 7f32c0e14740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a49454850 0x560a4945a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:08.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.719+0000 7f32c0e14740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a4945d430 0x560a4945aef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:08.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.719+0000 7f32c0e14740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x560a494601c0 con 0x560a49454850 2024-04-21T23:20:08.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.719+0000 7f32c0e14740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x560a49373e00 con 0x560a4945d430 2024-04-21T23:20:08.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.719+0000 7f32c0e14740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x560a49460340 con 0x560a49453f30 2024-04-21T23:20:08.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.719+0000 7f32c0e14740 10 client.0 _ll_register_callbacks cb 0x560a493aeeb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:20:08.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.719+0000 7f32b6ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a49454850 0x560a4945a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:08.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.719+0000 7f32b6ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a49454850 0x560a4945a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37388/0 (socket says 172.21.15.40:37388) 2024-04-21T23:20:08.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.719+0000 7f32b6ffd640 1 -- 172.21.15.40:0/2604478410 learned_addr learned my addr 172.21.15.40:0/2604478410 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:20:08.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.719+0000 7f32b7fff640 1 --2- 172.21.15.40:0/2604478410 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a4945d430 0x560a4945aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:08.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.720+0000 7f32b77fe640 1 --2- 172.21.15.40:0/2604478410 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560a49453f30 0x560a4945a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:08.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.720+0000 7f32b6ffd640 1 -- 172.21.15.40:0/2604478410 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560a49453f30 msgr2=0x560a4945a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:08.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.720+0000 7f32b6ffd640 1 --2- 172.21.15.40:0/2604478410 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560a49453f30 0x560a4945a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:08.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.720+0000 7f32b6ffd640 1 -- 172.21.15.40:0/2604478410 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a4945d430 msgr2=0x560a4945aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:08.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.720+0000 7f32b6ffd640 1 --2- 172.21.15.40:0/2604478410 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a4945d430 0x560a4945aef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:08.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.720+0000 7f32b6ffd640 1 -- 172.21.15.40:0/2604478410 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560a490cc3f0 con 0x560a49454850 2024-04-21T23:20:08.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.720+0000 7f32b6ffd640 1 --2- 172.21.15.40:0/2604478410 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a49454850 0x560a4945a9b0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f32ac0cd090 tx=0x7f32ac09b7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ef6d4aca22f16f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:08.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.720+0000 7f3296ffd640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-21T23:20:08.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.720+0000 7f3296ffd640 1 -- 172.21.15.40:0/2604478410 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32ac0bd040 con 0x560a49454850 2024-04-21T23:20:08.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.720+0000 7f3296ffd640 1 -- 172.21.15.40:0/2604478410 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f32ac09b400 con 0x560a49454850 2024-04-21T23:20:08.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.720+0000 7f32c0e14740 1 -- 172.21.15.40:0/2604478410 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f32a8004a70 con 0x560a49454850 2024-04-21T23:20:08.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.720+0000 7f3296ffd640 1 -- 172.21.15.40:0/2604478410 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f32ac0a4a20 con 0x560a49454850 2024-04-21T23:20:08.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.720+0000 7f32c0e14740 10 client.16191 fetch_fsmap learned FSMap version 28 2024-04-21T23:20:08.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.720+0000 7f32c0e14740 1 -- 172.21.15.40:0/2604478410 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x560a4908c0f0 con 0x560a49454850 2024-04-21T23:20:08.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.720+0000 7f3296ffd640 1 -- 172.21.15.40:0/2604478410 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32a8004a70 con 0x560a49454850 2024-04-21T23:20:08.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.720+0000 7f3296ffd640 1 -- 172.21.15.40:0/2604478410 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f32ac086040 con 0x560a49454850 2024-04-21T23:20:08.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.720+0000 7f32c0e14740 10 client.16191 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T23:20:08.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.721+0000 7f32c0e14740 10 client.16191 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:20:08.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.721+0000 7f32c0e14740 20 client.16191 populate_metadata read hostname 'smithi040' 2024-04-21T23:20:08.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.721+0000 7f32c0e14740 1 --2- 172.21.15.40:0/2604478410 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x560a4946d800 0x560a4946fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:08.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.721+0000 7f32c0e14740 4 client.16191 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:20:08.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.721+0000 7f32c0e14740 1 -- 172.21.15.40:0/2604478410 --> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560a493c5aa0 con 0x560a4946d800 2024-04-21T23:20:08.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.721+0000 7f32b77fe640 1 --2- 172.21.15.40:0/2604478410 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x560a4946d800 0x560a4946fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:08.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.721+0000 7f32b77fe640 1 --2- 172.21.15.40:0/2604478410 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x560a4946d800 0x560a4946fbf0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3df30d88c82fc772 server_cookie=1820f76458204af4 in_seq=0 out_seq=0 2024-04-21T23:20:08.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.721+0000 7f3296ffd640 10 client.16191 ms_handle_connect on v2:172.21.15.111:6838/3712235548 2024-04-21T23:20:08.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.722+0000 7f3296ffd640 1 -- 172.21.15.40:0/2604478410 <== mds.0 v2:172.21.15.111:6838/3712235548 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560a493c5aa0 con 0x560a4946d800 2024-04-21T23:20:08.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.722+0000 7f3296ffd640 10 client.16191 handle_command_reply: tid=0 2024-04-21T23:20:08.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.722+0000 7f32c0e14740 1 client.16191 shutdown 2024-04-21T23:20:08.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.722+0000 7f32c0e14740 1 -- 172.21.15.40:0/2604478410 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a49454850 msgr2=0x560a4945a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:08.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.722+0000 7f32c0e14740 1 --2- 172.21.15.40:0/2604478410 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a49454850 0x560a4945a9b0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f32ac0cd090 tx=0x7f32ac09b7d0 comp rx=0 tx=0).stop 2024-04-21T23:20:08.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.722+0000 7f32c0e14740 1 -- 172.21.15.40:0/2604478410 shutdown_connections 2024-04-21T23:20:08.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.722+0000 7f32c0e14740 1 --2- 172.21.15.40:0/2604478410 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x560a4946d800 0x560a4946fbf0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:08.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.722+0000 7f32c0e14740 1 --2- 172.21.15.40:0/2604478410 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a4945d430 0x560a4945aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:08.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.722+0000 7f32c0e14740 1 --2- 172.21.15.40:0/2604478410 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a49454850 0x560a4945a9b0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:08.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.722+0000 7f3296ffd640 0 client.16191 ms_handle_reset on v2:172.21.15.111:6838/3712235548 2024-04-21T23:20:08.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.722+0000 7f32c0e14740 1 --2- 172.21.15.40:0/2604478410 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560a49453f30 0x560a4945a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:08.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.722+0000 7f32c0e14740 1 -- 172.21.15.40:0/2604478410 >> 172.21.15.40:0/2604478410 conn(0x560a4944e660 msgr2=0x560a494503b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:08.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.722+0000 7f32c0e14740 1 -- 172.21.15.40:0/2604478410 shutdown_connections 2024-04-21T23:20:08.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.722+0000 7f32c0e14740 1 -- 172.21.15.40:0/2604478410 wait complete. 2024-04-21T23:20:08.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.722+0000 7f32c0e14740 20 client.16191 trim_cache size 0 max 16384 2024-04-21T23:20:08.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.741+0000 7f32c0e14740 1 Processor -- start 2024-04-21T23:20:08.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.741+0000 7f32c0e14740 1 -- start start 2024-04-21T23:20:08.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.741+0000 7f32c0e14740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a49453f30 0x560a4945cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:08.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.741+0000 7f32c0e14740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a49454850 0x560a49469080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:08.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.741+0000 7f32c0e14740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560a4945d430 0x560a4945ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:08.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.741+0000 7f32c0e14740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f32b81b3680 con 0x560a49454850 2024-04-21T23:20:08.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.741+0000 7f32c0e14740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f32b81b0ef0 con 0x560a49453f30 2024-04-21T23:20:08.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.741+0000 7f32c0e14740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f32b81aed40 con 0x560a4945d430 2024-04-21T23:20:08.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.741+0000 7f32b6ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a49454850 0x560a49469080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:08.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.741+0000 7f32b6ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a49454850 0x560a49469080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37390/0 (socket says 172.21.15.40:37390) 2024-04-21T23:20:08.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.741+0000 7f32b6ffd640 1 -- 172.21.15.40:0/1774678995 learned_addr learned my addr 172.21.15.40:0/1774678995 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:20:08.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.741+0000 7f32b77fe640 1 --2- 172.21.15.40:0/1774678995 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a49453f30 0x560a4945cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:08.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.741+0000 7f32b6ffd640 1 -- 172.21.15.40:0/1774678995 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560a4945d430 msgr2=0x560a4945ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:08.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.741+0000 7f32b6ffd640 1 --2- 172.21.15.40:0/1774678995 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560a4945d430 0x560a4945ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:08.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.741+0000 7f32b6ffd640 1 -- 172.21.15.40:0/1774678995 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a49453f30 msgr2=0x560a4945cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:08.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.741+0000 7f32b6ffd640 1 --2- 172.21.15.40:0/1774678995 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a49453f30 0x560a4945cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:08.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.741+0000 7f32b6ffd640 1 -- 172.21.15.40:0/1774678995 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560a494551c0 con 0x560a49454850 2024-04-21T23:20:08.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.741+0000 7f32b6ffd640 1 --2- 172.21.15.40:0/1774678995 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a49454850 0x560a49469080 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f32ac098780 tx=0x7f32ac09ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b14ca4a7786f677 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:08.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.741+0000 7f329d7fa640 1 -- 172.21.15.40:0/1774678995 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32ac018c90 con 0x560a49454850 2024-04-21T23:20:08.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.742+0000 7f329d7fa640 1 -- 172.21.15.40:0/1774678995 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32ac09b200 con 0x560a49454850 2024-04-21T23:20:08.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.742+0000 7f329d7fa640 1 -- 172.21.15.40:0/1774678995 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32ac09afe0 con 0x560a49454850 2024-04-21T23:20:08.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.742+0000 7f32c0e14740 1 -- 172.21.15.40:0/1774678995 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a49454850 msgr2=0x560a49469080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:08.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.742+0000 7f32c0e14740 1 --2- 172.21.15.40:0/1774678995 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a49454850 0x560a49469080 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f32ac098780 tx=0x7f32ac09ba20 comp rx=0 tx=0).stop 2024-04-21T23:20:08.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.742+0000 7f32c0e14740 1 -- 172.21.15.40:0/1774678995 shutdown_connections 2024-04-21T23:20:08.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.742+0000 7f32c0e14740 1 --2- 172.21.15.40:0/1774678995 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560a4945d430 0x560a4945ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:08.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.742+0000 7f32c0e14740 1 --2- 172.21.15.40:0/1774678995 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a49454850 0x560a49469080 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:08.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.742+0000 7f32c0e14740 1 --2- 172.21.15.40:0/1774678995 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a49453f30 0x560a4945cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:08.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.742+0000 7f32c0e14740 1 -- 172.21.15.40:0/1774678995 >> 172.21.15.40:0/1774678995 conn(0x560a4944e660 msgr2=0x560a4945a430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:08.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.742+0000 7f32c0e14740 1 -- 172.21.15.40:0/1774678995 shutdown_connections 2024-04-21T23:20:08.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.742+0000 7f32c0e14740 1 -- 172.21.15.40:0/1774678995 wait complete. 2024-04-21T23:20:08.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.742+0000 7f32c0e14740 1 -- start start 2024-04-21T23:20:08.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.743+0000 7f32c0e14740 1 Processor -- start 2024-04-21T23:20:08.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.743+0000 7f32c0e14740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a49453f30 0x560a49457230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:08.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.743+0000 7f32c0e14740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560a49454850 0x560a49457770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:08.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.743+0000 7f32c0e14740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a4945d430 0x560a49457cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:08.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.743+0000 7f32c0e14740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f32b81af3b0 con 0x560a4945d430 2024-04-21T23:20:08.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.743+0000 7f32c0e14740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f32b81b0820 con 0x560a49453f30 2024-04-21T23:20:08.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.743+0000 7f32c0e14740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f32b81afa80 con 0x560a49454850 2024-04-21T23:20:08.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.743+0000 7f32c0e14740 10 client.0 _ll_register_callbacks cb 0x7f32ac00e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:20:08.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.743+0000 7f32b7fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a4945d430 0x560a49457cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:08.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.743+0000 7f32b7fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a4945d430 0x560a49457cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37394/0 (socket says 172.21.15.40:37394) 2024-04-21T23:20:08.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.743+0000 7f32b77fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a49453f30 0x560a49457230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:08.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.743+0000 7f32b7fff640 1 -- 172.21.15.40:0/2263721444 learned_addr learned my addr 172.21.15.40:0/2263721444 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:20:08.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.743+0000 7f32b6ffd640 1 --2- 172.21.15.40:0/2263721444 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560a49454850 0x560a49457770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:08.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.743+0000 7f32b7fff640 1 -- 172.21.15.40:0/2263721444 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560a49454850 msgr2=0x560a49457770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:08.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.743+0000 7f32b7fff640 1 --2- 172.21.15.40:0/2263721444 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560a49454850 0x560a49457770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:08.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.743+0000 7f32b7fff640 1 -- 172.21.15.40:0/2263721444 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a49453f30 msgr2=0x560a49457230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:08.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.743+0000 7f32b7fff640 1 --2- 172.21.15.40:0/2263721444 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a49453f30 0x560a49457230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:08.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.743+0000 7f32b7fff640 1 -- 172.21.15.40:0/2263721444 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560a494d2dd0 con 0x560a4945d430 2024-04-21T23:20:08.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.744+0000 7f32b7fff640 1 --2- 172.21.15.40:0/2263721444 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a4945d430 0x560a49457cb0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f32b00092e0 tx=0x7f32b0009970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd968107518bf8ec server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:08.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.744+0000 7f3296ffd640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-21T23:20:08.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.744+0000 7f3296ffd640 1 -- 172.21.15.40:0/2263721444 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32b0004d20 con 0x560a4945d430 2024-04-21T23:20:08.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.744+0000 7f3296ffd640 1 -- 172.21.15.40:0/2263721444 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f32b000f650 con 0x560a4945d430 2024-04-21T23:20:08.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.744+0000 7f32c0e14740 1 -- 172.21.15.40:0/2263721444 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f32ac09b200 con 0x560a4945d430 2024-04-21T23:20:08.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.744+0000 7f3296ffd640 1 -- 172.21.15.40:0/2263721444 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f32b0033050 con 0x560a4945d430 2024-04-21T23:20:08.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.744+0000 7f32c0e14740 10 client.16197 fetch_fsmap learned FSMap version 28 2024-04-21T23:20:08.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.744+0000 7f32c0e14740 1 -- 172.21.15.40:0/2263721444 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x560a494d4120 con 0x560a4945d430 2024-04-21T23:20:08.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.744+0000 7f3296ffd640 1 -- 172.21.15.40:0/2263721444 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32ac09b200 con 0x560a4945d430 2024-04-21T23:20:08.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.744+0000 7f3296ffd640 1 -- 172.21.15.40:0/2263721444 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f32b002b850 con 0x560a4945d430 2024-04-21T23:20:08.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.744+0000 7f32c0e14740 10 client.16197 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T23:20:08.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.744+0000 7f32c0e14740 10 client.16197 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:20:08.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.744+0000 7f32c0e14740 20 client.16197 populate_metadata read hostname 'smithi040' 2024-04-21T23:20:08.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.744+0000 7f32c0e14740 1 --2- 172.21.15.40:0/2263721444 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x560a494d4960 0x560a494d6d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:08.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.744+0000 7f32c0e14740 4 client.16197 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T23:20:08.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.744+0000 7f32c0e14740 1 -- 172.21.15.40:0/2263721444 --> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] -- command(tid 0: {"prefix": "scrub status"}) -- 0x560a493bfdf0 con 0x560a494d4960 2024-04-21T23:20:08.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.745+0000 7f32b77fe640 1 --2- 172.21.15.40:0/2263721444 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x560a494d4960 0x560a494d6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:08.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.745+0000 7f32b77fe640 1 --2- 172.21.15.40:0/2263721444 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x560a494d4960 0x560a494d6d80 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b62be07d1c91a32f server_cookie=6597e9dc9a971fb1 in_seq=0 out_seq=0 2024-04-21T23:20:08.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.745+0000 7f3296ffd640 10 client.16197 ms_handle_connect on v2:172.21.15.111:6838/3712235548 2024-04-21T23:20:08.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.745+0000 7f3296ffd640 1 -- 172.21.15.40:0/2263721444 <== mds.0 v2:172.21.15.111:6838/3712235548 1 ==== command_reply(tid 0: 0 ) ==== 8+0+265 (crc 0 0 0) 0x560a493bfdf0 con 0x560a494d4960 2024-04-21T23:20:08.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.745+0000 7f3296ffd640 10 client.16197 handle_command_reply: tid=0 2024-04-21T23:20:08.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.745+0000 7f32c0e14740 1 client.16197 shutdown 2024-04-21T23:20:08.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.745+0000 7f32c0e14740 1 -- 172.21.15.40:0/2263721444 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a4945d430 msgr2=0x560a49457cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:08.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.745+0000 7f32c0e14740 1 --2- 172.21.15.40:0/2263721444 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a4945d430 0x560a49457cb0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f32b00092e0 tx=0x7f32b0009970 comp rx=0 tx=0).stop 2024-04-21T23:20:08.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.745+0000 7f32c0e14740 1 -- 172.21.15.40:0/2263721444 shutdown_connections 2024-04-21T23:20:08.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.745+0000 7f32c0e14740 1 --2- 172.21.15.40:0/2263721444 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x560a494d4960 0x560a494d6d80 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:08.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.745+0000 7f32c0e14740 1 --2- 172.21.15.40:0/2263721444 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a4945d430 0x560a49457cb0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:08.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.745+0000 7f32c0e14740 1 --2- 172.21.15.40:0/2263721444 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560a49454850 0x560a49457770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:08.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.745+0000 7f3296ffd640 0 client.16197 ms_handle_reset on v2:172.21.15.111:6838/3712235548 2024-04-21T23:20:08.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.745+0000 7f32c0e14740 1 --2- 172.21.15.40:0/2263721444 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a49453f30 0x560a49457230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:08.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.745+0000 7f32c0e14740 1 -- 172.21.15.40:0/2263721444 >> 172.21.15.40:0/2263721444 conn(0x560a4944e660 msgr2=0x560a4945acc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:08.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.745+0000 7f32c0e14740 1 -- 172.21.15.40:0/2263721444 shutdown_connections 2024-04-21T23:20:08.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.745+0000 7f32c0e14740 1 -- 172.21.15.40:0/2263721444 wait complete. 2024-04-21T23:20:08.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.746+0000 7f32c0e14740 20 client.16197 trim_cache size 0 max 16384 2024-04-21T23:20:08.751 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-21T23:20:08.751 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "scrub active (0 inodes in the stack)", 2024-04-21T23:20:08.751 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": { 2024-04-21T23:20:08.751 INFO:teuthology.orchestra.run.smithi040.stdout: "f1345386-ae92-4f68-9149-480c1be2784f": { 2024-04-21T23:20:08.752 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/", 2024-04-21T23:20:08.752 INFO:teuthology.orchestra.run.smithi040.stdout: "tag": "f1345386-ae92-4f68-9149-480c1be2784f", 2024-04-21T23:20:08.752 INFO:teuthology.orchestra.run.smithi040.stdout: "options": "recursive,force" 2024-04-21T23:20:08.752 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:20:08.752 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:20:08.752 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-21T23:20:08.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.746+0000 7f32be22e640 1 -- 172.21.15.40:0/317838875 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f32a00a2d00 msgr2=0x7f32a00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:08.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.746+0000 7f32be22e640 1 --2- 172.21.15.40:0/317838875 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f32a00a2d00 0x7f32a00a51c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f32b0009370 tx=0x7f32b0009240 comp rx=0 tx=0).stop 2024-04-21T23:20:08.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.746+0000 7f32be22e640 1 -- 172.21.15.40:0/317838875 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32b81020d0 msgr2=0x7f32b8073f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:08.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.746+0000 7f32be22e640 1 --2- 172.21.15.40:0/317838875 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32b81020d0 0x7f32b8073f80 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f32ac00e9f0 tx=0x7f32ac00eeb0 comp rx=0 tx=0).stop 2024-04-21T23:20:08.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.746+0000 7f32be22e640 1 -- 172.21.15.40:0/317838875 shutdown_connections 2024-04-21T23:20:08.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.746+0000 7f32be22e640 1 --2- 172.21.15.40:0/317838875 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f32a00a2d00 0x7f32a00a51c0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:08.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.746+0000 7f32be22e640 1 --2- 172.21.15.40:0/317838875 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f32b8107720 0x7f32b80744c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:08.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.746+0000 7f32be22e640 1 --2- 172.21.15.40:0/317838875 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32b81020d0 0x7f32b8073f80 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:08.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.746+0000 7f32be22e640 1 --2- 172.21.15.40:0/317838875 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32b8101710 0x7f32b8073a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:08.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.746+0000 7f32be22e640 1 -- 172.21.15.40:0/317838875 >> 172.21.15.40:0/317838875 conn(0x7f32b80fd180 msgr2=0x7f32b80fd950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:08.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.746+0000 7f32be22e640 1 -- 172.21.15.40:0/317838875 shutdown_connections 2024-04-21T23:20:08.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:08.746+0000 7f32be22e640 1 -- 172.21.15.40:0/317838875 wait complete. 2024-04-21T23:20:08.756 INFO:tasks.cephfs.filesystem:scrub status for tag:f1345386-ae92-4f68-9149-480c1be2784f - {'path': '/', 'tag': 'f1345386-ae92-4f68-9149-480c1be2784f', 'options': 'recursive,force'} 2024-04-21T23:20:09.181 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:08 smithi040 ceph-mon[27984]: pgmap v55: 129 pgs: 129 active+clean; 34 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 49 op/s 2024-04-21T23:20:09.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:08 smithi040 ceph-mon[27984]: scrub summary: active paths [/] 2024-04-21T23:20:09.442 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:08 smithi111 ceph-mon[31085]: pgmap v55: 129 pgs: 129 active+clean; 34 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 49 op/s 2024-04-21T23:20:09.443 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:08 smithi111 ceph-mon[31085]: scrub summary: active paths [/] 2024-04-21T23:20:09.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:08 smithi174 ceph-mon[31409]: pgmap v55: 129 pgs: 129 active+clean; 34 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 49 op/s 2024-04-21T23:20:09.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:08 smithi174 ceph-mon[31409]: scrub summary: active paths [/] 2024-04-21T23:20:09.694 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:09 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:09.442+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000006c7 [...4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/ auth{0=1} fragtree_t(*^6) v15748 ap=1 f(v1 m2024-04-21T23:20:08.647664+0000 1191=1191+0) n(v1 rc2024-04-21T23:20:08.647664+0000 1192=1191+1) old_inodes=1 (inest mix) (ifile excl) caps={25132=pAsLsXsFsx/pAsLsXsFsxcral@2196},l=25132 | dirtyscattered=0 request=0 lock=0 importingcaps=0 dirfrag=64 caps=1 dirtyrstat=0 replicated=1 dirty=1 authpin=1 scrubqueue=0 0x5572f3139b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(2)0x100000006c7:[<0x100000003a2/generic v1914>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[]","memoryvalue":"(2)0x100000006c7:[<0x100000003a2/generic v15748>,<0x10000000007/tests v1534>,<0x10000000006/xfstests-dev v162>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-04-21T23:20:08.647664+0000 1191=1191+0)","ondisk_value.rstat":"n(v0 rc2024-04-21T23:20:08.647664+0000 1192=1191+1)","memory_value.dirstat":"f(v1 m2024-04-21T23:20:08.647664+0000 1191=1191+0)","memory_value.rstat":"n(v1 rc2024-04-21T23:20:08.647664+0000 1192=1191+1)","error_str":""},"return_code":0} 2024-04-21T23:20:09.694 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:09 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:09.453+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003a4 [...4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/ auth fragtree_t(*^3) v11658 ap=1 f(v0 m2024-04-21T23:20:05.462708+0000 404=404+0) n(v0 rc2024-04-21T23:20:05.462708+0000 405=404+1) old_inodes=1 (inest lock) (ifile excl) caps={25132=pAsLsXsFsx/pAsLsXsFsxcral@672},l=25132 | request=0 lock=0 importingcaps=0 dirfrag=8 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f313a100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(2)0x100000003a4:[<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[]","memoryvalue":"(2)0x100000003a4:[<0x100000003a2/btrfs v11658>,<0x10000000007/tests v1534>,<0x10000000006/xfstests-dev v162>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-04-21T23:20:05.462708+0000 404=404+0)","ondisk_value.rstat":"n(v0 rc2024-04-21T23:20:05.462708+0000 405=404+1)","memory_value.dirstat":"f(v0 m2024-04-21T23:20:05.462708+0000 404=404+0)","memory_value.rstat":"n(v0 rc2024-04-21T23:20:05.462708+0000 405=404+1)","error_str":""},"return_code":0} 2024-04-21T23:20:09.694 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:09 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:09.456+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000630 [...4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/cifs/ auth{0=1} v11672 ap=1 f(v0 m2024-04-21T23:20:05.471708+0000 4=4+0) n(v0 rc2024-04-21T23:20:05.471708+0000 5=4+1) old_inodes=1 (inest mix) (ifile excl) caps={25132=pAsLsXsFsx/pAsLsXsFsxcral@6},l=25132 | lock=0 importingcaps=0 dirfrag=1 caps=1 replicated=1 dirty=1 authpin=1 scrubqueue=0 0x5572f33f6000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(2)0x10000000630:[<0x100000003a2/cifs v1316>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[]","memoryvalue":"(2)0x10000000630:[<0x100000003a2/cifs v11672>,<0x10000000007/tests v1534>,<0x10000000006/xfstests-dev v162>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-04-21T23:20:05.471708+0000 4=4+0)","ondisk_value.rstat":"n(v0 rc2024-04-21T23:20:05.471708+0000 5=4+1)","memory_value.dirstat":"f(v0 m2024-04-21T23:20:05.471708+0000 4=4+0)","memory_value.rstat":"n(v0 rc2024-04-21T23:20:05.471708+0000 5=4+1)","error_str":""},"return_code":0} 2024-04-21T23:20:09.694 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:09 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:09.457+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000006c1 [...4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/f2fs/ auth{0=1} v12012 ap=1 f(v0 m2024-04-21T23:20:05.704705+0000 4=4+0) n(v0 rc2024-04-21T23:20:05.704705+0000 5=4+1) old_inodes=1 (inest mix) (ifile excl) caps={25132=pAsLsXsFsx/pAsLsXsFsxcral@13},l=25132 | request=0 lock=0 importingcaps=0 dirfrag=1 caps=1 replicated=1 dirty=1 authpin=1 scrubqueue=0 0x5572f33f6580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(2)0x100000006c1:[<0x100000003a2/f2fs v1610>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[]","memoryvalue":"(2)0x100000006c1:[<0x100000003a2/f2fs v12012>,<0x10000000007/tests v1534>,<0x10000000006/xfstests-dev v162>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-04-21T23:20:05.704705+0000 4=4+0)","ondisk_value.rstat":"n(v0 rc2024-04-21T23:20:05.704705+0000 5=4+1)","memory_value.dirstat":"f(v0 m2024-04-21T23:20:05.704705+0000 4=4+0)","memory_value.rstat":"n(v0 rc2024-04-21T23:20:05.704705+0000 5=4+1)","error_str":""},"return_code":0} 2024-04-21T23:20:10.017 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:09 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:09.732+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000713 [3,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/037.out auth v152 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25132=0-4194304@2} caps={25132=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25132 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5572f3406580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000713:[<0x100000006c7/037.out v152>,<0x100000003a2/generic v1916>,<0x10000000007/tests v780>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v10>,<0x100000001fb/client.0 v14>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v30>,<0x10000000001/sv_1 v53>,<0x10000000000/qa v43>,<0x1/volumes v38>]//[3]","memoryvalue":"(4)0x10000000713:[<0x100000006c7/037.out v152>,<0x100000003a2/generic v15748>,<0x10000000007/tests v1539>,<0x10000000006/xfstests-dev v164>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:10.017 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:09 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:09.737+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000752 [3,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/067.out auth v278 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25132=0-4194304@2} caps={25132=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25132 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5572f3409700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000752:[<0x100000006c7/067.out v278>,<0x100000003a2/generic v1916>,<0x10000000007/tests v780>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v10>,<0x100000001fb/client.0 v14>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v30>,<0x10000000001/sv_1 v53>,<0x10000000000/qa v43>,<0x1/volumes v38>]//[3]","memoryvalue":"(4)0x10000000752:[<0x100000006c7/067.out v278>,<0x100000003a2/generic v15748>,<0x10000000007/tests v1539>,<0x10000000006/xfstests-dev v164>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:10.017 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:09 smithi111 ceph-mon[31085]: scrub queued for path: / 2024-04-21T23:20:10.017 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:09 smithi111 ceph-mon[31085]: scrub summary: idle+waiting paths [/] 2024-04-21T23:20:10.017 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:09 smithi111 ceph-mon[31085]: scrub summary: active paths [/] 2024-04-21T23:20:10.017 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:09 smithi111 ceph-mon[31085]: scrub summary: idle+waiting paths [/] 2024-04-21T23:20:10.017 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:09 smithi111 ceph-mon[31085]: scrub summary: active paths [/] 2024-04-21T23:20:10.017 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:09 smithi111 ceph-mon[31085]: scrub summary: idle+waiting paths [/] 2024-04-21T23:20:10.018 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:09 smithi111 ceph-mon[31085]: scrub summary: active paths [/] 2024-04-21T23:20:10.018 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:09 smithi111 ceph-mon[31085]: scrub summary: idle+waiting paths [/] 2024-04-21T23:20:10.018 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:09 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000006c7 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic) see mds.h log and `damage ls` output for details 2024-04-21T23:20:10.018 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:09 smithi111 ceph-mon[31085]: scrub summary: active paths [/] 2024-04-21T23:20:10.018 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:09 smithi111 ceph-mon[31085]: scrub summary: idle+waiting paths [/] 2024-04-21T23:20:10.018 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:09 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000003a4 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs) see mds.h log and `damage ls` output for details 2024-04-21T23:20:10.018 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:09 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000630 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/cifs) see mds.h log and `damage ls` output for details 2024-04-21T23:20:10.018 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:09 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000006c1 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/f2fs) see mds.h log and `damage ls` output for details 2024-04-21T23:20:10.018 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:09 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000713 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/037.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:10.018 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:09 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000752 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/067.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:10.019 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:09 smithi111 ceph-mon[31085]: Scrub error on inode 0x1000000071b (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/041.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:10.019 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:09 smithi111 ceph-mon[31085]: Scrub error on inode 0x1000000070b (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/034.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:10.019 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:09 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000725 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/046.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:10.019 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:09 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000006eb (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/018.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:10.019 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:09 smithi111 ceph-mon[31085]: Scrub error on inode 0x1000000071f (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/043.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:10.019 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:09 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000719 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/040.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:10.020 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:09 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:09.745+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000071b [3,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/041.out auth v168 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25132=0-4194304@2} caps={25132=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25132 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5572f341d180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000071b:[<0x100000006c7/041.out v168>,<0x100000003a2/generic v1916>,<0x10000000007/tests v780>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v10>,<0x100000001fb/client.0 v14>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v30>,<0x10000000001/sv_1 v53>,<0x10000000000/qa v43>,<0x1/volumes v38>]//[3]","memoryvalue":"(4)0x1000000071b:[<0x100000006c7/041.out v168>,<0x100000003a2/generic v15748>,<0x10000000007/tests v1539>,<0x10000000006/xfstests-dev v164>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:10.020 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:09 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:09.747+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000070b [3,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/034.out auth v136 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25132=0-4194304@2} caps={25132=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25132 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5572f3428000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000070b:[<0x100000006c7/034.out v136>,<0x100000003a2/generic v1916>,<0x10000000007/tests v780>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v10>,<0x100000001fb/client.0 v14>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v30>,<0x10000000001/sv_1 v53>,<0x10000000000/qa v43>,<0x1/volumes v38>]//[3]","memoryvalue":"(4)0x1000000070b:[<0x100000006c7/034.out v136>,<0x100000003a2/generic v15748>,<0x10000000007/tests v1539>,<0x10000000006/xfstests-dev v164>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:10.020 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:09 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:09.748+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000725 [3,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/046.out auth v188 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25132=0-4194304@2} caps={25132=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25132 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5572f3429080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000725:[<0x100000006c7/046.out v188>,<0x100000003a2/generic v1916>,<0x10000000007/tests v780>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v10>,<0x100000001fb/client.0 v14>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v30>,<0x10000000001/sv_1 v53>,<0x10000000000/qa v43>,<0x1/volumes v38>]//[3]","memoryvalue":"(4)0x10000000725:[<0x100000006c7/046.out v188>,<0x100000003a2/generic v15748>,<0x10000000007/tests v1539>,<0x10000000006/xfstests-dev v164>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:10.020 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:09 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:09.750+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000006eb [3,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/018.out auth v72 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25132=0-4194304@2} caps={25132=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25132 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5572f342a100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000006eb:[<0x100000006c7/018.out v72>,<0x100000003a2/generic v1916>,<0x10000000007/tests v780>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v10>,<0x100000001fb/client.0 v14>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v30>,<0x10000000001/sv_1 v53>,<0x10000000000/qa v43>,<0x1/volumes v38>]//[3]","memoryvalue":"(4)0x100000006eb:[<0x100000006c7/018.out v72>,<0x100000003a2/generic v15748>,<0x10000000007/tests v1539>,<0x10000000006/xfstests-dev v164>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:10.020 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:09 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:09.751+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000071f [3,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/043.out auth v176 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25132=0-4194304@2} caps={25132=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25132 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5572f342a680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000071f:[<0x100000006c7/043.out v176>,<0x100000003a2/generic v1916>,<0x10000000007/tests v780>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v10>,<0x100000001fb/client.0 v14>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v30>,<0x10000000001/sv_1 v53>,<0x10000000000/qa v43>,<0x1/volumes v38>]//[3]","memoryvalue":"(4)0x1000000071f:[<0x100000006c7/043.out v176>,<0x100000003a2/generic v15748>,<0x10000000007/tests v1539>,<0x10000000006/xfstests-dev v164>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:10.020 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:09 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:09.793+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000719 [3,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/040.out auth v164 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25132=0-4194304@2} caps={25132=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25132 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5572f33df180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000719:[<0x100000006c7/040.out v164>,<0x100000003a2/generic v1916>,<0x10000000007/tests v780>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v10>,<0x100000001fb/client.0 v14>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v30>,<0x10000000001/sv_1 v53>,<0x10000000000/qa v43>,<0x1/volumes v38>]//[3]","memoryvalue":"(4)0x10000000719:[<0x100000006c7/040.out v164>,<0x100000003a2/generic v15748>,<0x10000000007/tests v1539>,<0x10000000006/xfstests-dev v164>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:10.020 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:09 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:09.798+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000006cf [3,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/004.out auth v16 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25132=0-4194304@2} caps={25132=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25132 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5572f36a3b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000006cf:[<0x100000006c7/004.out v16>,<0x100000003a2/generic v1916>,<0x10000000007/tests v780>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v10>,<0x100000001fb/client.0 v14>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v30>,<0x10000000001/sv_1 v53>,<0x10000000000/qa v43>,<0x1/volumes v38>]//[3]","memoryvalue":"(4)0x100000006cf:[<0x100000006c7/004.out v16>,<0x100000003a2/generic v15748>,<0x10000000007/tests v1539>,<0x10000000006/xfstests-dev v164>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:10.021 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:09 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:09.801+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000006d7 [3,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/008.out auth v32 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25132=0-4194304@2} caps={25132=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25132 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5572f36a4c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000006d7:[<0x100000006c7/008.out v32>,<0x100000003a2/generic v1916>,<0x10000000007/tests v780>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v10>,<0x100000001fb/client.0 v14>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v30>,<0x10000000001/sv_1 v53>,<0x10000000000/qa v43>,<0x1/volumes v38>]//[3]","memoryvalue":"(4)0x100000006d7:[<0x100000006c7/008.out v32>,<0x100000003a2/generic v15748>,<0x10000000007/tests v1539>,<0x10000000006/xfstests-dev v164>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:10.021 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:09 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:09.804+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000006dd [3,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/011.out auth v44 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25132=0-4194304@2} caps={25132=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25132 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5572f36aeb00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000006dd:[<0x100000006c7/011.out v44>,<0x100000003a2/generic v1916>,<0x10000000007/tests v780>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v10>,<0x100000001fb/client.0 v14>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v30>,<0x10000000001/sv_1 v53>,<0x10000000000/qa v43>,<0x1/volumes v38>]//[3]","memoryvalue":"(4)0x100000006dd:[<0x100000006c7/011.out v44>,<0x100000003a2/generic v15748>,<0x10000000007/tests v1539>,<0x10000000006/xfstests-dev v164>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:10.021 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:10.017+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000006d1 [3,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/005.out auth v20 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25132=0-4194304@2} caps={25132=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25132 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5572f36b7080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000006d1:[<0x100000006c7/005.out v20>,<0x100000003a2/generic v1916>,<0x10000000007/tests v780>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v10>,<0x100000001fb/client.0 v14>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v30>,<0x10000000001/sv_1 v53>,<0x10000000000/qa v43>,<0x1/volumes v38>]//[3]","memoryvalue":"(4)0x100000006d1:[<0x100000006c7/005.out v20>,<0x100000003a2/generic v15748>,<0x10000000007/tests v1539>,<0x10000000006/xfstests-dev v164>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:10.282 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:10.154+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000006ed [3,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/019.out auth v76 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25132=0-4194304@2} caps={25132=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25132 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5572f36b9180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000006ed:[<0x100000006c7/019.out v76>,<0x100000003a2/generic v1916>,<0x10000000007/tests v780>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v10>,<0x100000001fb/client.0 v14>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v30>,<0x10000000001/sv_1 v53>,<0x10000000000/qa v43>,<0x1/volumes v38>]//[3]","memoryvalue":"(4)0x100000006ed:[<0x100000006c7/019.out v76>,<0x100000003a2/generic v15748>,<0x10000000007/tests v1539>,<0x10000000006/xfstests-dev v164>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:10.282 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:10.160+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000075e [3,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/073.out auth v302 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25132=0-4194304@2} caps={25132=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25132 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5572f36ca580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000075e:[<0x100000006c7/073.out v302>,<0x100000003a2/generic v1916>,<0x10000000007/tests v780>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v10>,<0x100000001fb/client.0 v14>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v30>,<0x10000000001/sv_1 v53>,<0x10000000000/qa v43>,<0x1/volumes v38>]//[3]","memoryvalue":"(4)0x1000000075e:[<0x100000006c7/073.out v302>,<0x100000003a2/generic v15748>,<0x10000000007/tests v1539>,<0x10000000006/xfstests-dev v164>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:10.282 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:10.167+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000721 [3,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/044.out auth v180 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25132=0-4194304@2} caps={25132=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25132 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5572f36d7180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000721:[<0x100000006c7/044.out v180>,<0x100000003a2/generic v1916>,<0x10000000007/tests v780>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v10>,<0x100000001fb/client.0 v14>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v30>,<0x10000000001/sv_1 v53>,<0x10000000000/qa v43>,<0x1/volumes v38>]//[3]","memoryvalue":"(4)0x10000000721:[<0x100000006c7/044.out v180>,<0x100000003a2/generic v15748>,<0x10000000007/tests v1539>,<0x10000000006/xfstests-dev v164>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:10.282 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:10.183+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000006df [3,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/012.out auth v48 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25132=0-4194304@2} caps={25132=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25132 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5572f3704c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000006df:[<0x100000006c7/012.out v48>,<0x100000003a2/generic v1916>,<0x10000000007/tests v780>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v10>,<0x100000001fb/client.0 v14>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v30>,<0x10000000001/sv_1 v53>,<0x10000000000/qa v43>,<0x1/volumes v38>]//[3]","memoryvalue":"(4)0x100000006df:[<0x100000006c7/012.out v48>,<0x100000003a2/generic v15748>,<0x10000000007/tests v1539>,<0x10000000006/xfstests-dev v164>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:10.282 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:10.184+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000006cd [3,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/003.out auth v12 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25132=0-4194304@2} caps={25132=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25132 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5572f3705180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000006cd:[<0x100000006c7/003.out v12>,<0x100000003a2/generic v1916>,<0x10000000007/tests v780>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v10>,<0x100000001fb/client.0 v14>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v30>,<0x10000000001/sv_1 v53>,<0x10000000000/qa v43>,<0x1/volumes v38>]//[3]","memoryvalue":"(4)0x100000006cd:[<0x100000006c7/003.out v12>,<0x100000003a2/generic v15748>,<0x10000000007/tests v1539>,<0x10000000006/xfstests-dev v164>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:10.282 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:10.197+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000074c [3,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/064.out auth v266 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25132=0-4194304@2} caps={25132=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25132 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5572f3725600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000074c:[<0x100000006c7/064.out v266>,<0x100000003a2/generic v1916>,<0x10000000007/tests v780>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v10>,<0x100000001fb/client.0 v14>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v30>,<0x10000000001/sv_1 v53>,<0x10000000000/qa v43>,<0x1/volumes v38>]//[3]","memoryvalue":"(4)0x1000000074c:[<0x100000006c7/064.out v266>,<0x100000003a2/generic v15748>,<0x10000000007/tests v1539>,<0x10000000006/xfstests-dev v164>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:10.283 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:10.200+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000006e7 [3,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/016.out auth v64 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25132=0-4194304@2} caps={25132=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25132 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5572f372eb00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000006e7:[<0x100000006c7/016.out v64>,<0x100000003a2/generic v1916>,<0x10000000007/tests v780>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v10>,<0x100000001fb/client.0 v14>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v30>,<0x10000000001/sv_1 v53>,<0x10000000000/qa v43>,<0x1/volumes v38>]//[3]","memoryvalue":"(4)0x100000006e7:[<0x100000006c7/016.out v64>,<0x100000003a2/generic v15748>,<0x10000000007/tests v1539>,<0x10000000006/xfstests-dev v164>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:10.283 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:10.227+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000075a [3,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/071.out auth v294 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25132=0-4194304@2} caps={25132=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25132 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5572f3777180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000075a:[<0x100000006c7/071.out v294>,<0x100000003a2/generic v1916>,<0x10000000007/tests v780>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v10>,<0x100000001fb/client.0 v14>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v30>,<0x10000000001/sv_1 v53>,<0x10000000000/qa v43>,<0x1/volumes v38>]//[3]","memoryvalue":"(4)0x1000000075a:[<0x100000006c7/071.out v294>,<0x100000003a2/generic v15748>,<0x10000000007/tests v1539>,<0x10000000006/xfstests-dev v164>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:10.283 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:10.231+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000723 [3,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/045.out auth v184 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25132=0-4194304@2} caps={25132=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25132 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5572f3780b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000723:[<0x100000006c7/045.out v184>,<0x100000003a2/generic v1916>,<0x10000000007/tests v780>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v10>,<0x100000001fb/client.0 v14>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v30>,<0x10000000001/sv_1 v53>,<0x10000000000/qa v43>,<0x1/volumes v38>]//[3]","memoryvalue":"(4)0x10000000723:[<0x100000006c7/045.out v184>,<0x100000003a2/generic v15748>,<0x10000000007/tests v1539>,<0x10000000006/xfstests-dev v164>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:10.283 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:10.231+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000729 [3,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/048.out auth v196 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25132=0-4194304@2} caps={25132=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25132 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5572f3781600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000729:[<0x100000006c7/048.out v196>,<0x100000003a2/generic v1916>,<0x10000000007/tests v780>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v10>,<0x100000001fb/client.0 v14>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v30>,<0x10000000001/sv_1 v53>,<0x10000000000/qa v43>,<0x1/volumes v38>]//[3]","memoryvalue":"(4)0x10000000729:[<0x100000006c7/048.out v196>,<0x100000003a2/generic v15748>,<0x10000000007/tests v1539>,<0x10000000006/xfstests-dev v164>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:10.283 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:10.234+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000006db [3,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/010.out auth v40 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25132=0-4194304@2} caps={25132=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25132 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5572f378e580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000006db:[<0x100000006c7/010.out v40>,<0x100000003a2/generic v1916>,<0x10000000007/tests v780>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v10>,<0x100000001fb/client.0 v14>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v30>,<0x10000000001/sv_1 v53>,<0x10000000000/qa v43>,<0x1/volumes v38>]//[3]","memoryvalue":"(4)0x100000006db:[<0x100000006c7/010.out v40>,<0x100000003a2/generic v15748>,<0x10000000007/tests v1539>,<0x10000000006/xfstests-dev v164>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:10.283 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:10.238+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000006fb [3,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/026.out auth v104 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25132=0-4194304@2} caps={25132=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25132 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5572f379b600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000006fb:[<0x100000006c7/026.out v104>,<0x100000003a2/generic v1916>,<0x10000000007/tests v780>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v10>,<0x100000001fb/client.0 v14>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v30>,<0x10000000001/sv_1 v53>,<0x10000000000/qa v43>,<0x1/volumes v38>]//[3]","memoryvalue":"(4)0x100000006fb:[<0x100000006c7/026.out v104>,<0x100000003a2/generic v15748>,<0x10000000007/tests v1539>,<0x10000000006/xfstests-dev v164>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:10.283 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:10.244+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000006f9 [3,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/025.out auth v100 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25132=0-4194304@2} caps={25132=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25132 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5572f37a9600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000006f9:[<0x100000006c7/025.out v100>,<0x100000003a2/generic v1916>,<0x10000000007/tests v780>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v10>,<0x100000001fb/client.0 v14>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v30>,<0x10000000001/sv_1 v53>,<0x10000000000/qa v43>,<0x1/volumes v38>]//[3]","memoryvalue":"(4)0x100000006f9:[<0x100000006c7/025.out v100>,<0x100000003a2/generic v15748>,<0x10000000007/tests v1539>,<0x10000000006/xfstests-dev v164>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:10.283 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:10.255+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000744 [3,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/060.out auth v250 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25132=0-4194304@2} caps={25132=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25132 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5572f37c5180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000744:[<0x100000006c7/060.out v250>,<0x100000003a2/generic v1916>,<0x10000000007/tests v780>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v10>,<0x100000001fb/client.0 v14>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v30>,<0x10000000001/sv_1 v53>,<0x10000000000/qa v43>,<0x1/volumes v38>]//[3]","memoryvalue":"(4)0x10000000744:[<0x100000006c7/060.out v250>,<0x100000003a2/generic v15748>,<0x10000000007/tests v1539>,<0x10000000006/xfstests-dev v164>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:10.283 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:10.282+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000734 [3,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/052.out auth v218 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25132=0-4194304@2} caps={25132=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25132 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5572f3809180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000734:[<0x100000006c7/052.out v218>,<0x100000003a2/generic v1916>,<0x10000000007/tests v780>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v10>,<0x100000001fb/client.0 v14>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v30>,<0x10000000001/sv_1 v53>,<0x10000000000/qa v43>,<0x1/volumes v38>]//[3]","memoryvalue":"(4)0x10000000734:[<0x100000006c7/052.out v218>,<0x100000003a2/generic v15748>,<0x10000000007/tests v1539>,<0x10000000006/xfstests-dev v164>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:10.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:09 smithi040 ceph-mon[27984]: scrub queued for path: / 2024-04-21T23:20:10.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:09 smithi040 ceph-mon[27984]: scrub summary: idle+waiting paths [/] 2024-04-21T23:20:10.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:09 smithi040 ceph-mon[27984]: scrub summary: active paths [/] 2024-04-21T23:20:10.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:09 smithi040 ceph-mon[27984]: scrub summary: idle+waiting paths [/] 2024-04-21T23:20:10.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:09 smithi040 ceph-mon[27984]: scrub summary: active paths [/] 2024-04-21T23:20:10.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:09 smithi040 ceph-mon[27984]: scrub summary: idle+waiting paths [/] 2024-04-21T23:20:10.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:09 smithi040 ceph-mon[27984]: scrub summary: active paths [/] 2024-04-21T23:20:10.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:09 smithi040 ceph-mon[27984]: scrub summary: idle+waiting paths [/] 2024-04-21T23:20:10.433 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:09 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000006c7 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic) see mds.h log and `damage ls` output for details 2024-04-21T23:20:10.433 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:09 smithi040 ceph-mon[27984]: scrub summary: active paths [/] 2024-04-21T23:20:10.433 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:09 smithi040 ceph-mon[27984]: scrub summary: idle+waiting paths [/] 2024-04-21T23:20:10.433 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:09 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000003a4 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs) see mds.h log and `damage ls` output for details 2024-04-21T23:20:10.433 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:09 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000630 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/cifs) see mds.h log and `damage ls` output for details 2024-04-21T23:20:10.433 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:09 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000006c1 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/f2fs) see mds.h log and `damage ls` output for details 2024-04-21T23:20:10.433 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:09 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000713 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/037.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:10.433 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:09 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000752 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/067.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:10.433 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:09 smithi040 ceph-mon[27984]: Scrub error on inode 0x1000000071b (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/041.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:10.433 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:09 smithi040 ceph-mon[27984]: Scrub error on inode 0x1000000070b (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/034.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:10.433 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:09 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000725 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/046.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:10.433 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:09 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000006eb (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/018.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:10.433 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:09 smithi040 ceph-mon[27984]: Scrub error on inode 0x1000000071f (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/043.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:10.433 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:09 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000719 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/040.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:10.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:09 smithi174 ceph-mon[31409]: scrub queued for path: / 2024-04-21T23:20:10.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:09 smithi174 ceph-mon[31409]: scrub summary: idle+waiting paths [/] 2024-04-21T23:20:10.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:09 smithi174 ceph-mon[31409]: scrub summary: active paths [/] 2024-04-21T23:20:10.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:09 smithi174 ceph-mon[31409]: scrub summary: idle+waiting paths [/] 2024-04-21T23:20:10.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:09 smithi174 ceph-mon[31409]: scrub summary: active paths [/] 2024-04-21T23:20:10.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:09 smithi174 ceph-mon[31409]: scrub summary: idle+waiting paths [/] 2024-04-21T23:20:10.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:09 smithi174 ceph-mon[31409]: scrub summary: active paths [/] 2024-04-21T23:20:10.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:09 smithi174 ceph-mon[31409]: scrub summary: idle+waiting paths [/] 2024-04-21T23:20:10.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:09 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000006c7 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic) see mds.h log and `damage ls` output for details 2024-04-21T23:20:10.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:09 smithi174 ceph-mon[31409]: scrub summary: active paths [/] 2024-04-21T23:20:10.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:09 smithi174 ceph-mon[31409]: scrub summary: idle+waiting paths [/] 2024-04-21T23:20:10.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:09 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000003a4 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs) see mds.h log and `damage ls` output for details 2024-04-21T23:20:10.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:09 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000630 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/cifs) see mds.h log and `damage ls` output for details 2024-04-21T23:20:10.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:09 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000006c1 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/f2fs) see mds.h log and `damage ls` output for details 2024-04-21T23:20:10.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:09 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000713 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/037.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:10.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:09 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000752 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/067.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:10.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:09 smithi174 ceph-mon[31409]: Scrub error on inode 0x1000000071b (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/041.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:10.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:09 smithi174 ceph-mon[31409]: Scrub error on inode 0x1000000070b (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/034.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:10.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:09 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000725 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/046.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:10.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:09 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000006eb (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/018.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:10.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:09 smithi174 ceph-mon[31409]: Scrub error on inode 0x1000000071f (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/043.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:10.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:09 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000719 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/040.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:10.561 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:10.287+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000071d [3,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/042.out auth v172 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25132=0-4194304@2} caps={25132=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25132 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5572f3813b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000071d:[<0x100000006c7/042.out v172>,<0x100000003a2/generic v1916>,<0x10000000007/tests v780>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v10>,<0x100000001fb/client.0 v14>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v30>,<0x10000000001/sv_1 v53>,<0x10000000000/qa v43>,<0x1/volumes v38>]//[3]","memoryvalue":"(4)0x1000000071d:[<0x100000006c7/042.out v172>,<0x100000003a2/generic v15748>,<0x10000000007/tests v1539>,<0x10000000006/xfstests-dev v164>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:10.562 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:10.298+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000006e3 [3,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/014.out auth v56 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25132=0-4194304@2} caps={25132=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25132 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5572f382bb80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000006e3:[<0x100000006c7/014.out v56>,<0x100000003a2/generic v1916>,<0x10000000007/tests v780>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v10>,<0x100000001fb/client.0 v14>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v30>,<0x10000000001/sv_1 v53>,<0x10000000000/qa v43>,<0x1/volumes v38>]//[3]","memoryvalue":"(4)0x100000006e3:[<0x100000006c7/014.out v56>,<0x100000003a2/generic v15748>,<0x10000000007/tests v1539>,<0x10000000006/xfstests-dev v164>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:10.562 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:10.299+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000006f7 [3,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/024.out auth v96 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25132=0-4194304@2} caps={25132=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25132 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5572f382c680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000006f7:[<0x100000006c7/024.out v96>,<0x100000003a2/generic v1916>,<0x10000000007/tests v780>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v10>,<0x100000001fb/client.0 v14>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v30>,<0x10000000001/sv_1 v53>,<0x10000000000/qa v43>,<0x1/volumes v38>]//[3]","memoryvalue":"(4)0x100000006f7:[<0x100000006c7/024.out v96>,<0x100000003a2/generic v15748>,<0x10000000007/tests v1539>,<0x10000000006/xfstests-dev v164>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:10.562 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:10.301+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000006f5 [3,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/023.out auth v92 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25132=0-4194304@2} caps={25132=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25132 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5572f383c000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000006f5:[<0x100000006c7/023.out v92>,<0x100000003a2/generic v1916>,<0x10000000007/tests v780>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v10>,<0x100000001fb/client.0 v14>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v30>,<0x10000000001/sv_1 v53>,<0x10000000000/qa v43>,<0x1/volumes v38>]//[3]","memoryvalue":"(4)0x100000006f5:[<0x100000006c7/023.out v92>,<0x100000003a2/generic v15748>,<0x10000000007/tests v1539>,<0x10000000006/xfstests-dev v164>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:10.562 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:10.307+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000758 [3,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/070.out auth v290 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25132=0-4194304@2} caps={25132=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25132 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5572f383f700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000758:[<0x100000006c7/070.out v290>,<0x100000003a2/generic v1916>,<0x10000000007/tests v780>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v10>,<0x100000001fb/client.0 v14>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v30>,<0x10000000001/sv_1 v53>,<0x10000000000/qa v43>,<0x1/volumes v38>]//[3]","memoryvalue":"(4)0x10000000758:[<0x100000006c7/070.out v290>,<0x100000003a2/generic v15748>,<0x10000000007/tests v1539>,<0x10000000006/xfstests-dev v164>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:10.562 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:10.312+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000006d5 [3,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/007.out auth v28 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25132=0-4194304@2} caps={25132=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25132 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5572f3849700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000006d5:[<0x100000006c7/007.out v28>,<0x100000003a2/generic v1916>,<0x10000000007/tests v780>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v10>,<0x100000001fb/client.0 v14>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v30>,<0x10000000001/sv_1 v53>,<0x10000000000/qa v43>,<0x1/volumes v38>]//[3]","memoryvalue":"(4)0x100000006d5:[<0x100000006c7/007.out v28>,<0x100000003a2/generic v15748>,<0x10000000007/tests v1539>,<0x10000000006/xfstests-dev v164>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:10.562 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:10.317+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000754 [3,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/068.out auth v282 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25132=0-4194304@2} caps={25132=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25132 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5572f3855b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000754:[<0x100000006c7/068.out v282>,<0x100000003a2/generic v1916>,<0x10000000007/tests v780>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v10>,<0x100000001fb/client.0 v14>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v30>,<0x10000000001/sv_1 v53>,<0x10000000000/qa v43>,<0x1/volumes v38>]//[3]","memoryvalue":"(4)0x10000000754:[<0x100000006c7/068.out v282>,<0x100000003a2/generic v15748>,<0x10000000007/tests v1539>,<0x10000000006/xfstests-dev v164>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:10.562 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:10.317+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000748 [3,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/062.out auth v258 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25132=0-4194304@2} caps={25132=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25132 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5572f3856c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000748:[<0x100000006c7/062.out v258>,<0x100000003a2/generic v1916>,<0x10000000007/tests v780>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v10>,<0x100000001fb/client.0 v14>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v30>,<0x10000000001/sv_1 v53>,<0x10000000000/qa v43>,<0x1/volumes v38>]//[3]","memoryvalue":"(4)0x10000000748:[<0x100000006c7/062.out v258>,<0x100000003a2/generic v15748>,<0x10000000007/tests v1539>,<0x10000000006/xfstests-dev v164>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:10.563 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:10.333+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000750 [3,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/066.out auth v274 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25132=0-4194304@2} caps={25132=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25132 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5572f387bb80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000750:[<0x100000006c7/066.out v274>,<0x100000003a2/generic v1916>,<0x10000000007/tests v780>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v10>,<0x100000001fb/client.0 v14>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v30>,<0x10000000001/sv_1 v53>,<0x10000000000/qa v43>,<0x1/volumes v38>]//[3]","memoryvalue":"(4)0x10000000750:[<0x100000006c7/066.out v274>,<0x100000003a2/generic v15748>,<0x10000000007/tests v1539>,<0x10000000006/xfstests-dev v164>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:10.563 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:10.337+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000072b [3,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/049.out auth v200 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25132=0-4194304@2} caps={25132=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25132 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5572f3889600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000072b:[<0x100000006c7/049.out v200>,<0x100000003a2/generic v1916>,<0x10000000007/tests v780>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v10>,<0x100000001fb/client.0 v14>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v30>,<0x10000000001/sv_1 v53>,<0x10000000000/qa v43>,<0x1/volumes v38>]//[3]","memoryvalue":"(4)0x1000000072b:[<0x100000006c7/049.out v200>,<0x100000003a2/generic v15748>,<0x10000000007/tests v1539>,<0x10000000006/xfstests-dev v164>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:10.563 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:10.348+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000738 [3,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/054.out auth v226 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25132=0-4194304@2} caps={25132=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25132 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5572f38a0580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000738:[<0x100000006c7/054.out v226>,<0x100000003a2/generic v1916>,<0x10000000007/tests v780>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v10>,<0x100000001fb/client.0 v14>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v30>,<0x10000000001/sv_1 v53>,<0x10000000000/qa v43>,<0x1/volumes v38>]//[3]","memoryvalue":"(4)0x10000000738:[<0x100000006c7/054.out v226>,<0x100000003a2/generic v15748>,<0x10000000007/tests v1539>,<0x10000000006/xfstests-dev v164>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:10.563 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:10.349+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000070e [3,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/035.out.default auth v142 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25132=0-4194304@2} caps={25132=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25132 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5572f38a0b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000070e:[<0x100000006c7/035.out.default v142>,<0x100000003a2/generic v1916>,<0x10000000007/tests v780>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v10>,<0x100000001fb/client.0 v14>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v30>,<0x10000000001/sv_1 v53>,<0x10000000000/qa v43>,<0x1/volumes v38>]//[3]","memoryvalue":"(4)0x1000000070e:[<0x100000006c7/035.out.default v142>,<0x100000003a2/generic v15748>,<0x10000000007/tests v1539>,<0x10000000006/xfstests-dev v164>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:10.563 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:10.352+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000736 [3,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/053.out auth v222 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25132=0-4194304@2} caps={25132=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25132 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5572f38ae580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000736:[<0x100000006c7/053.out v222>,<0x100000003a2/generic v1916>,<0x10000000007/tests v780>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v10>,<0x100000001fb/client.0 v14>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v30>,<0x10000000001/sv_1 v53>,<0x10000000000/qa v43>,<0x1/volumes v38>]//[3]","memoryvalue":"(4)0x10000000736:[<0x100000006c7/053.out v222>,<0x100000003a2/generic v15748>,<0x10000000007/tests v1539>,<0x10000000006/xfstests-dev v164>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:10.563 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:10.362+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000746 [3,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/061.out auth v254 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25132=0-4194304@2} caps={25132=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25132 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5572f38c6b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000746:[<0x100000006c7/061.out v254>,<0x100000003a2/generic v1916>,<0x10000000007/tests v780>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v10>,<0x100000001fb/client.0 v14>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v30>,<0x10000000001/sv_1 v53>,<0x10000000000/qa v43>,<0x1/volumes v38>]//[3]","memoryvalue":"(4)0x10000000746:[<0x100000006c7/061.out v254>,<0x100000003a2/generic v15748>,<0x10000000007/tests v1539>,<0x10000000006/xfstests-dev v164>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:10.563 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:10.387+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000730 [3,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/050.out.xfsquota auth v210 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25132=0-4194304@2} caps={25132=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25132 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5572f3905700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000730:[<0x100000006c7/050.out.xfsquota v210>,<0x100000003a2/generic v1916>,<0x10000000007/tests v780>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v10>,<0x100000001fb/client.0 v14>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v30>,<0x10000000001/sv_1 v53>,<0x10000000000/qa v43>,<0x1/volumes v38>]//[3]","memoryvalue":"(4)0x10000000730:[<0x100000006c7/050.out.xfsquota v210>,<0x100000003a2/generic v15748>,<0x10000000007/tests v1539>,<0x10000000006/xfstests-dev v164>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:10.563 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:10.388+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000006cb [3,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/002.out auth v8 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25132=0-4194304@2} caps={25132=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25132 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5572f390c000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000006cb:[<0x100000006c7/002.out v8>,<0x100000003a2/generic v1916>,<0x10000000007/tests v780>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v10>,<0x100000001fb/client.0 v14>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v30>,<0x10000000001/sv_1 v53>,<0x10000000000/qa v43>,<0x1/volumes v38>]//[3]","memoryvalue":"(4)0x100000006cb:[<0x100000006c7/002.out v8>,<0x100000003a2/generic v15748>,<0x10000000007/tests v1539>,<0x10000000006/xfstests-dev v164>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:10.563 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:10.402+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000711 [3,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/036.out auth v148 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25132=0-4194304@2} caps={25132=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25132 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5572f392a680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000711:[<0x100000006c7/036.out v148>,<0x100000003a2/generic v1916>,<0x10000000007/tests v780>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v10>,<0x100000001fb/client.0 v14>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v30>,<0x10000000001/sv_1 v53>,<0x10000000000/qa v43>,<0x1/volumes v38>]//[3]","memoryvalue":"(4)0x10000000711:[<0x100000006c7/036.out v148>,<0x100000003a2/generic v15748>,<0x10000000007/tests v1539>,<0x10000000006/xfstests-dev v164>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:10.563 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:10.408+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000006f3 [3,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/022.out auth v88 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25132=0-4194304@2} caps={25132=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25132 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5572f3942b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000006f3:[<0x100000006c7/022.out v88>,<0x100000003a2/generic v1916>,<0x10000000007/tests v780>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v10>,<0x100000001fb/client.0 v14>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v30>,<0x10000000001/sv_1 v53>,<0x10000000000/qa v43>,<0x1/volumes v38>]//[3]","memoryvalue":"(4)0x100000006f3:[<0x100000006c7/022.out v88>,<0x100000003a2/generic v15748>,<0x10000000007/tests v1539>,<0x10000000006/xfstests-dev v164>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:10.563 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:10.413+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000006d9 [3,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/009.out auth v36 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25132=0-4194304@2} caps={25132=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25132 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5572f3951180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000006d9:[<0x100000006c7/009.out v36>,<0x100000003a2/generic v1916>,<0x10000000007/tests v780>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v10>,<0x100000001fb/client.0 v14>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v30>,<0x10000000001/sv_1 v53>,<0x10000000000/qa v43>,<0x1/volumes v38>]//[3]","memoryvalue":"(4)0x100000006d9:[<0x100000006c7/009.out v36>,<0x100000003a2/generic v15748>,<0x10000000007/tests v1539>,<0x10000000006/xfstests-dev v164>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:10.563 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:10.415+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000070d [3,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/035.cfg auth v140 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25132=0-4194304@2} caps={25132=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25132 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5572f395a000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000070d:[<0x100000006c7/035.cfg v140>,<0x100000003a2/generic v1916>,<0x10000000007/tests v780>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v10>,<0x100000001fb/client.0 v14>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v30>,<0x10000000001/sv_1 v53>,<0x10000000000/qa v43>,<0x1/volumes v38>]//[3]","memoryvalue":"(4)0x1000000070d:[<0x100000006c7/035.cfg v140>,<0x100000003a2/generic v15748>,<0x10000000007/tests v1539>,<0x10000000006/xfstests-dev v164>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:10.564 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:10.418+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000006d3 [3,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/006.out auth v24 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25132=0-4194304@2} caps={25132=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25132 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5572f395c100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000006d3:[<0x100000006c7/006.out v24>,<0x100000003a2/generic v1916>,<0x10000000007/tests v780>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v10>,<0x100000001fb/client.0 v14>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v30>,<0x10000000001/sv_1 v53>,<0x10000000000/qa v43>,<0x1/volumes v38>]//[3]","memoryvalue":"(4)0x100000006d3:[<0x100000006c7/006.out v24>,<0x100000003a2/generic v15748>,<0x10000000007/tests v1539>,<0x10000000006/xfstests-dev v164>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:10.564 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:10.440+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000072d [3,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/050.cfg auth v204 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25132=0-4194304@2} caps={25132=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25132 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5572f3995180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000072d:[<0x100000006c7/050.cfg v204>,<0x100000003a2/generic v1916>,<0x10000000007/tests v780>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v10>,<0x100000001fb/client.0 v14>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v30>,<0x10000000001/sv_1 v53>,<0x10000000000/qa v43>,<0x1/volumes v38>]//[3]","memoryvalue":"(4)0x1000000072d:[<0x100000006c7/050.cfg v204>,<0x100000003a2/generic v15748>,<0x10000000007/tests v1539>,<0x10000000006/xfstests-dev v164>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:10.564 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:10.465+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000006e9 [3,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/017.out auth v68 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25132=0-4194304@2} caps={25132=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25132 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5572f39cac00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000006e9:[<0x100000006c7/017.out v68>,<0x100000003a2/generic v1916>,<0x10000000007/tests v780>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v10>,<0x100000001fb/client.0 v14>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v30>,<0x10000000001/sv_1 v53>,<0x10000000000/qa v43>,<0x1/volumes v38>]//[3]","memoryvalue":"(4)0x100000006e9:[<0x100000006c7/017.out v68>,<0x100000003a2/generic v15748>,<0x10000000007/tests v1539>,<0x10000000006/xfstests-dev v164>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:10.564 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:10.475+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000727 [3,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/047.out auth v192 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25132=0-4194304@2} caps={25132=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25132 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5572f39e5600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000727:[<0x100000006c7/047.out v192>,<0x100000003a2/generic v1916>,<0x10000000007/tests v780>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v10>,<0x100000001fb/client.0 v14>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v30>,<0x10000000001/sv_1 v53>,<0x10000000000/qa v43>,<0x1/volumes v38>]//[3]","memoryvalue":"(4)0x10000000727:[<0x100000006c7/047.out v192>,<0x100000003a2/generic v15748>,<0x10000000007/tests v1539>,<0x10000000006/xfstests-dev v164>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:10.564 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:10.476+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000740 [3,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/058.out auth v242 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25132=0-4194304@2} caps={25132=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25132 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5572f39e6680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000740:[<0x100000006c7/058.out v242>,<0x100000003a2/generic v1916>,<0x10000000007/tests v780>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v10>,<0x100000001fb/client.0 v14>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v30>,<0x10000000001/sv_1 v53>,<0x10000000000/qa v43>,<0x1/volumes v38>]//[3]","memoryvalue":"(4)0x10000000740:[<0x100000006c7/058.out v242>,<0x100000003a2/generic v15748>,<0x10000000007/tests v1539>,<0x10000000006/xfstests-dev v164>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:10.564 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:10.499+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000006f1 [3,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/021.out auth v84 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25132=0-4194304@2} caps={25132=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25132 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5572f3a1d700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000006f1:[<0x100000006c7/021.out v84>,<0x100000003a2/generic v1916>,<0x10000000007/tests v780>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v10>,<0x100000001fb/client.0 v14>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v30>,<0x10000000001/sv_1 v53>,<0x10000000000/qa v43>,<0x1/volumes v38>]//[3]","memoryvalue":"(4)0x100000006f1:[<0x100000006c7/021.out v84>,<0x100000003a2/generic v15748>,<0x10000000007/tests v1539>,<0x10000000006/xfstests-dev v164>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:10.564 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:10.500+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000006e5 [3,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/015.out auth v60 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25132=0-4194304@2} caps={25132=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25132 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5572f3a2a580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000006e5:[<0x100000006c7/015.out v60>,<0x100000003a2/generic v1916>,<0x10000000007/tests v780>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v10>,<0x100000001fb/client.0 v14>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v30>,<0x10000000001/sv_1 v53>,<0x10000000000/qa v43>,<0x1/volumes v38>]//[3]","memoryvalue":"(4)0x100000006e5:[<0x100000006c7/015.out v60>,<0x100000003a2/generic v15748>,<0x10000000007/tests v1539>,<0x10000000006/xfstests-dev v164>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:10.564 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:10.502+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000074e [3,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/065.out auth v270 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25132=0-4194304@2} caps={25132=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25132 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5572f3a2b080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000074e:[<0x100000006c7/065.out v270>,<0x100000003a2/generic v1916>,<0x10000000007/tests v780>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v10>,<0x100000001fb/client.0 v14>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v30>,<0x10000000001/sv_1 v53>,<0x10000000000/qa v43>,<0x1/volumes v38>]//[3]","memoryvalue":"(4)0x1000000074e:[<0x100000006c7/065.out v270>,<0x100000003a2/generic v15748>,<0x10000000007/tests v1539>,<0x10000000006/xfstests-dev v164>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:10.564 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:10.509+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000717 [3,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/039.out auth v160 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25132=0-4194304@2} caps={25132=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25132 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5572f3a38580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000717:[<0x100000006c7/039.out v160>,<0x100000003a2/generic v1916>,<0x10000000007/tests v780>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v10>,<0x100000001fb/client.0 v14>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v30>,<0x10000000001/sv_1 v53>,<0x10000000000/qa v43>,<0x1/volumes v38>]//[3]","memoryvalue":"(4)0x10000000717:[<0x100000006c7/039.out v160>,<0x100000003a2/generic v15748>,<0x10000000007/tests v1539>,<0x10000000006/xfstests-dev v164>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:10.564 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:10.509+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000715 [3,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/038.out auth v156 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25132=0-4194304@2} caps={25132=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25132 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5572f3a39080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000715:[<0x100000006c7/038.out v156>,<0x100000003a2/generic v1916>,<0x10000000007/tests v780>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v10>,<0x100000001fb/client.0 v14>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v30>,<0x10000000001/sv_1 v53>,<0x10000000000/qa v43>,<0x1/volumes v38>]//[3]","memoryvalue":"(4)0x10000000715:[<0x100000006c7/038.out v156>,<0x100000003a2/generic v15748>,<0x10000000007/tests v1539>,<0x10000000006/xfstests-dev v164>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:10.564 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:10.531+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000072e [3,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/050.out.default auth v206 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25132=0-4194304@2} caps={25132=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25132 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5572f3a62680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000072e:[<0x100000006c7/050.out.default v206>,<0x100000003a2/generic v1916>,<0x10000000007/tests v780>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v10>,<0x100000001fb/client.0 v14>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v30>,<0x10000000001/sv_1 v53>,<0x10000000000/qa v43>,<0x1/volumes v38>]//[3]","memoryvalue":"(4)0x1000000072e:[<0x100000006c7/050.out.default v206>,<0x100000003a2/generic v15748>,<0x10000000007/tests v1539>,<0x10000000006/xfstests-dev v164>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:10.565 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:10.560+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000756 [3,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/069.out auth v286 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25132=0-4194304@2} caps={25132=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25132 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5572f3ab2680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000756:[<0x100000006c7/069.out v286>,<0x100000003a2/generic v1916>,<0x10000000007/tests v780>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v10>,<0x100000001fb/client.0 v14>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v30>,<0x10000000001/sv_1 v53>,<0x10000000000/qa v43>,<0x1/volumes v38>]//[3]","memoryvalue":"(4)0x10000000756:[<0x100000006c7/069.out v286>,<0x100000003a2/generic v15748>,<0x10000000007/tests v1539>,<0x10000000006/xfstests-dev v164>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:10.944 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:10.586+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000732 [3,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/051.out auth v214 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25132=0-4194304@2} caps={25132=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25132 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5572f3b05080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000732:[<0x100000006c7/051.out v214>,<0x100000003a2/generic v1916>,<0x10000000007/tests v780>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v10>,<0x100000001fb/client.0 v14>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v30>,<0x10000000001/sv_1 v53>,<0x10000000000/qa v43>,<0x1/volumes v38>]//[3]","memoryvalue":"(4)0x10000000732:[<0x100000006c7/051.out v214>,<0x100000003a2/generic v15748>,<0x10000000007/tests v1539>,<0x10000000006/xfstests-dev v164>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:10.945 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:10.587+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000006fd [3,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/027.out auth v108 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25132=0-4194304@2} caps={25132=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25132 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5572f3b06680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000006fd:[<0x100000006c7/027.out v108>,<0x100000003a2/generic v1916>,<0x10000000007/tests v780>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v10>,<0x100000001fb/client.0 v14>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v30>,<0x10000000001/sv_1 v53>,<0x10000000000/qa v43>,<0x1/volumes v38>]//[3]","memoryvalue":"(4)0x100000006fd:[<0x100000006c7/027.out v108>,<0x100000003a2/generic v15748>,<0x10000000007/tests v1539>,<0x10000000006/xfstests-dev v164>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:10.945 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:10.591+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000073c [3,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/056.out auth v234 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25132=0-4194304@2} caps={25132=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25132 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5572f3b0f600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000073c:[<0x100000006c7/056.out v234>,<0x100000003a2/generic v1916>,<0x10000000007/tests v780>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v10>,<0x100000001fb/client.0 v14>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v30>,<0x10000000001/sv_1 v53>,<0x10000000000/qa v43>,<0x1/volumes v38>]//[3]","memoryvalue":"(4)0x1000000073c:[<0x100000006c7/056.out v234>,<0x100000003a2/generic v15748>,<0x10000000007/tests v1539>,<0x10000000006/xfstests-dev v164>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:10.945 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:10.591+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000074a [3,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/063.out auth v262 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25132=0-4194304@2} caps={25132=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25132 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5572f3b10100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000074a:[<0x100000006c7/063.out v262>,<0x100000003a2/generic v1916>,<0x10000000007/tests v780>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v10>,<0x100000001fb/client.0 v14>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v30>,<0x10000000001/sv_1 v53>,<0x10000000000/qa v43>,<0x1/volumes v38>]//[3]","memoryvalue":"(4)0x1000000074a:[<0x100000006c7/063.out v262>,<0x100000003a2/generic v15748>,<0x10000000007/tests v1539>,<0x10000000006/xfstests-dev v164>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:10.945 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:10.619+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000006e1 [3,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/013.out auth v52 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25132=0-4194304@2} caps={25132=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25132 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5572f3b4eb00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000006e1:[<0x100000006c7/013.out v52>,<0x100000003a2/generic v1916>,<0x10000000007/tests v780>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v10>,<0x100000001fb/client.0 v14>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v30>,<0x10000000001/sv_1 v53>,<0x10000000000/qa v43>,<0x1/volumes v38>]//[3]","memoryvalue":"(4)0x100000006e1:[<0x100000006c7/013.out v52>,<0x100000003a2/generic v15748>,<0x10000000007/tests v1539>,<0x10000000006/xfstests-dev v164>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:10.945 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:10.624+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000075c [3,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/072.out auth v298 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25132=0-4194304@2} caps={25132=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25132 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5572f3b61080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000075c:[<0x100000006c7/072.out v298>,<0x100000003a2/generic v1916>,<0x10000000007/tests v780>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v10>,<0x100000001fb/client.0 v14>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v30>,<0x10000000001/sv_1 v53>,<0x10000000000/qa v43>,<0x1/volumes v38>]//[3]","memoryvalue":"(4)0x1000000075c:[<0x100000006c7/072.out v298>,<0x100000003a2/generic v15748>,<0x10000000007/tests v1539>,<0x10000000006/xfstests-dev v164>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:10.945 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:10.633+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000073a [3,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/055.out auth v230 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25132=0-4194304@2} caps={25132=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25132 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5572f3b7a100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000073a:[<0x100000006c7/055.out v230>,<0x100000003a2/generic v1916>,<0x10000000007/tests v780>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v10>,<0x100000001fb/client.0 v14>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v30>,<0x10000000001/sv_1 v53>,<0x10000000000/qa v43>,<0x1/volumes v38>]//[3]","memoryvalue":"(4)0x1000000073a:[<0x100000006c7/055.out v230>,<0x100000003a2/generic v15748>,<0x10000000007/tests v1539>,<0x10000000006/xfstests-dev v164>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:10.945 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:10.658+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000070f [3,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/035.out.nfs auth v144 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25132=0-4194304@2} caps={25132=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25132 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5572f3bb7700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000070f:[<0x100000006c7/035.out.nfs v144>,<0x100000003a2/generic v1916>,<0x10000000007/tests v780>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v10>,<0x100000001fb/client.0 v14>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v30>,<0x10000000001/sv_1 v53>,<0x10000000000/qa v43>,<0x1/volumes v38>]//[3]","memoryvalue":"(4)0x1000000070f:[<0x100000006c7/035.out.nfs v144>,<0x100000003a2/generic v15748>,<0x10000000007/tests v1539>,<0x10000000006/xfstests-dev v164>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:10.945 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:10.665+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000073e [3,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/057.out auth v238 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25132=0-4194304@2} caps={25132=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25132 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5572f3bc0c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000073e:[<0x100000006c7/057.out v238>,<0x100000003a2/generic v1916>,<0x10000000007/tests v780>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v10>,<0x100000001fb/client.0 v14>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v30>,<0x10000000001/sv_1 v53>,<0x10000000000/qa v43>,<0x1/volumes v38>]//[3]","memoryvalue":"(4)0x1000000073e:[<0x100000006c7/057.out v238>,<0x100000003a2/generic v15748>,<0x10000000007/tests v1539>,<0x10000000006/xfstests-dev v164>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:10.945 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:10.698+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000006c9 [3,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/001.out auth v4 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25132=0-4194304@2} caps={25132=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25132 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5572f3c09700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000006c9:[<0x100000006c7/001.out v4>,<0x100000003a2/generic v1916>,<0x10000000007/tests v780>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v10>,<0x100000001fb/client.0 v14>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v30>,<0x10000000001/sv_1 v53>,<0x10000000000/qa v43>,<0x1/volumes v38>]//[3]","memoryvalue":"(4)0x100000006c9:[<0x100000006c7/001.out v4>,<0x100000003a2/generic v15748>,<0x10000000007/tests v1539>,<0x10000000006/xfstests-dev v164>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:10.945 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:10.721+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000072f [3,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/050.out.nojournal auth v208 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25132=0-4194304@2} caps={25132=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25132 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5572f3c3cc00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000072f:[<0x100000006c7/050.out.nojournal v208>,<0x100000003a2/generic v1916>,<0x10000000007/tests v780>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v10>,<0x100000001fb/client.0 v14>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v30>,<0x10000000001/sv_1 v53>,<0x10000000000/qa v43>,<0x1/volumes v38>]//[3]","memoryvalue":"(4)0x1000000072f:[<0x100000006c7/050.out.nojournal v208>,<0x100000003a2/generic v15748>,<0x10000000007/tests v1539>,<0x10000000006/xfstests-dev v164>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:10.945 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:10.744+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000006ff [3,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/028.out auth v112 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25132=0-4194304@2} caps={25132=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25132 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5572f3c72c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000006ff:[<0x100000006c7/028.out v112>,<0x100000003a2/generic v1916>,<0x10000000007/tests v780>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v10>,<0x100000001fb/client.0 v14>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v30>,<0x10000000001/sv_1 v53>,<0x10000000000/qa v43>,<0x1/volumes v38>]//[3]","memoryvalue":"(4)0x100000006ff:[<0x100000006c7/028.out v112>,<0x100000003a2/generic v15748>,<0x10000000007/tests v1539>,<0x10000000006/xfstests-dev v164>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:10.946 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:10.760+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000742 [3,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/059.out auth v246 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25132=0-4194304@2} caps={25132=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25132 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5572f3c98000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000742:[<0x100000006c7/059.out v246>,<0x100000003a2/generic v1916>,<0x10000000007/tests v780>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v10>,<0x100000001fb/client.0 v14>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v30>,<0x10000000001/sv_1 v53>,<0x10000000000/qa v43>,<0x1/volumes v38>]//[3]","memoryvalue":"(4)0x10000000742:[<0x100000006c7/059.out v246>,<0x100000003a2/generic v15748>,<0x10000000007/tests v1539>,<0x10000000006/xfstests-dev v164>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:11.434 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:10 smithi040 ceph-mon[27984]: pgmap v56: 129 pgs: 129 active+clean; 41 MiB data, 486 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.2 MiB/s wr, 63 op/s 2024-04-21T23:20:11.434 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:10 smithi040 ceph-mon[27984]: scrub summary: active paths [/] 2024-04-21T23:20:11.434 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:10 smithi040 ceph-mon[27984]: scrub summary: idle+waiting paths [/] 2024-04-21T23:20:11.434 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:10 smithi040 ceph-mon[27984]: scrub summary: active paths [/] 2024-04-21T23:20:11.434 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:10 smithi040 ceph-mon[27984]: scrub summary: active paths [/] 2024-04-21T23:20:11.434 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:10 smithi040 ceph-mon[27984]: scrub summary: idle+waiting paths [/] 2024-04-21T23:20:11.434 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:10 smithi040 ceph-mon[27984]: scrub summary: idle+waiting paths [/] 2024-04-21T23:20:11.434 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:10 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000006cf (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/004.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.434 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:10 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000006d7 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/008.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.434 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:10 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000006dd (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/011.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.435 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:10 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000006d1 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/005.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.435 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:10 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000006ed (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/019.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.435 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:10 smithi040 ceph-mon[27984]: Scrub error on inode 0x1000000075e (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/073.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.435 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:10 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000721 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/044.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.435 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:10 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000006df (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/012.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.435 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:10 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000006cd (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/003.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.435 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:10 smithi040 ceph-mon[27984]: Scrub error on inode 0x1000000074c (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/064.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.435 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:10 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000006e7 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/016.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.435 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:10 smithi040 ceph-mon[27984]: Scrub error on inode 0x1000000075a (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/071.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.436 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:10 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000723 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/045.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.436 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:10 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000729 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/048.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.436 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:10 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000006db (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/010.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.436 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:10 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000006fb (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/026.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.436 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:10 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000006f9 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/025.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.436 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:10 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000744 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/060.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.436 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:10 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000734 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/052.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.436 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:10 smithi040 ceph-mon[27984]: Scrub error on inode 0x1000000071d (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/042.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.436 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:10 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000006e3 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/014.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.436 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:10 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000006f7 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/024.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.436 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:10 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000006f5 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/023.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.436 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:10 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000758 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/070.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.436 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:10 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000006d5 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/007.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.436 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:10 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000754 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/068.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.436 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:10 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000748 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/062.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.436 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:10 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000750 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/066.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.436 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:10 smithi040 ceph-mon[27984]: Scrub error on inode 0x1000000072b (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/049.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.437 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:10 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000738 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/054.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.437 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:10 smithi040 ceph-mon[27984]: Scrub error on inode 0x1000000070e (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/035.out.default) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.437 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:10 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000736 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/053.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.437 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:10 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000746 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/061.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.437 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:10 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000730 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/050.out.xfsquota) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.437 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:10 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000006cb (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/002.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.437 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:10 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000711 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/036.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.437 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:10 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000006f3 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/022.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.437 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:10 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000006d9 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/009.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.437 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:10 smithi040 ceph-mon[27984]: Scrub error on inode 0x1000000070d (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/035.cfg) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.437 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:10 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000006d3 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/006.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.437 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:10 smithi040 ceph-mon[27984]: Scrub error on inode 0x1000000072d (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/050.cfg) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.437 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:10 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000006e9 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/017.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.437 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:10 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000727 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/047.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.437 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:10 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000740 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/058.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.438 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:10 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000006f1 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/021.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.438 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:10 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000006e5 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/015.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.438 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:10 smithi040 ceph-mon[27984]: Scrub error on inode 0x1000000074e (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/065.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.438 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:10 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000717 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/039.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.438 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:10 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000715 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/038.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.438 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:10 smithi040 ceph-mon[27984]: Scrub error on inode 0x1000000072e (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/050.out.default) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.438 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:10 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000756 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/069.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.438 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:10 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000732 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/051.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.438 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:10 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000006fd (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/027.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.438 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:10 smithi040 ceph-mon[27984]: Scrub error on inode 0x1000000073c (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/056.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.438 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:10 smithi040 ceph-mon[27984]: Scrub error on inode 0x1000000074a (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/063.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.438 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:10 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000006e1 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/013.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.439 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:10 smithi040 ceph-mon[27984]: Scrub error on inode 0x1000000075c (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/072.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.439 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:10 smithi040 ceph-mon[27984]: Scrub error on inode 0x1000000073a (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/055.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.439 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:10 smithi040 ceph-mon[27984]: Scrub error on inode 0x1000000070f (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/035.out.nfs) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.439 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:10 smithi040 ceph-mon[27984]: Scrub error on inode 0x1000000073e (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/057.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.439 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:10 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000006c9 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/001.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.439 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:10 smithi040 ceph-mon[27984]: Scrub error on inode 0x1000000072f (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/050.out.nojournal) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.439 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:10 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000006ff (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/028.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.439 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:10 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000742 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/059.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:10 smithi174 ceph-mon[31409]: pgmap v56: 129 pgs: 129 active+clean; 41 MiB data, 486 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.2 MiB/s wr, 63 op/s 2024-04-21T23:20:11.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:10 smithi174 ceph-mon[31409]: scrub summary: active paths [/] 2024-04-21T23:20:11.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:10 smithi174 ceph-mon[31409]: scrub summary: idle+waiting paths [/] 2024-04-21T23:20:11.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:10 smithi174 ceph-mon[31409]: scrub summary: active paths [/] 2024-04-21T23:20:11.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:10 smithi174 ceph-mon[31409]: scrub summary: active paths [/] 2024-04-21T23:20:11.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:10 smithi174 ceph-mon[31409]: scrub summary: idle+waiting paths [/] 2024-04-21T23:20:11.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:10 smithi174 ceph-mon[31409]: scrub summary: idle+waiting paths [/] 2024-04-21T23:20:11.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:10 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000006cf (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/004.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:10 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000006d7 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/008.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:10 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000006dd (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/011.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:10 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000006d1 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/005.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:10 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000006ed (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/019.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:10 smithi174 ceph-mon[31409]: Scrub error on inode 0x1000000075e (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/073.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:10 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000721 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/044.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:10 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000006df (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/012.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:10 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000006cd (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/003.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:10 smithi174 ceph-mon[31409]: Scrub error on inode 0x1000000074c (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/064.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:10 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000006e7 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/016.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:10 smithi174 ceph-mon[31409]: Scrub error on inode 0x1000000075a (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/071.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:10 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000723 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/045.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:10 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000729 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/048.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.446 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:10 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000006db (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/010.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.447 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:10 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000006fb (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/026.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.447 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:10 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000006f9 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/025.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.447 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:10 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000744 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/060.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.447 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:10 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000734 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/052.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.447 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:10 smithi174 ceph-mon[31409]: Scrub error on inode 0x1000000071d (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/042.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.447 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:10 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000006e3 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/014.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.447 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:10 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000006f7 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/024.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.447 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:10 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000006f5 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/023.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.447 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:10 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000758 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/070.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.447 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:10 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000006d5 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/007.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.447 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:10 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000754 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/068.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.447 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:10 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000748 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/062.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.447 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:10 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000750 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/066.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.448 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:10 smithi174 ceph-mon[31409]: Scrub error on inode 0x1000000072b (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/049.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.448 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:10 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000738 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/054.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.448 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:10 smithi174 ceph-mon[31409]: Scrub error on inode 0x1000000070e (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/035.out.default) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.448 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:10 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000736 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/053.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.448 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:10 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000746 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/061.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.448 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:10 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000730 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/050.out.xfsquota) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.448 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:10 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000006cb (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/002.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.448 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:10 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000711 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/036.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.448 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:10 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000006f3 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/022.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.448 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:10 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000006d9 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/009.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.448 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:10 smithi174 ceph-mon[31409]: Scrub error on inode 0x1000000070d (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/035.cfg) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.448 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:10 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000006d3 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/006.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.448 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:10 smithi174 ceph-mon[31409]: Scrub error on inode 0x1000000072d (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/050.cfg) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.448 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:10 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000006e9 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/017.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.448 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:10 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000727 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/047.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.448 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:10 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000740 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/058.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.449 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:10 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000006f1 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/021.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.449 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:10 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000006e5 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/015.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.449 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:10 smithi174 ceph-mon[31409]: Scrub error on inode 0x1000000074e (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/065.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.449 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:10 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000717 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/039.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.449 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:10 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000715 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/038.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.449 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:10 smithi174 ceph-mon[31409]: Scrub error on inode 0x1000000072e (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/050.out.default) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.449 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:10 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000756 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/069.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.449 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:10 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000732 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/051.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.449 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:10 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000006fd (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/027.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.449 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:10 smithi174 ceph-mon[31409]: Scrub error on inode 0x1000000073c (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/056.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.449 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:10 smithi174 ceph-mon[31409]: Scrub error on inode 0x1000000074a (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/063.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.449 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:10 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000006e1 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/013.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.449 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:10 smithi174 ceph-mon[31409]: Scrub error on inode 0x1000000075c (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/072.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.449 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:10 smithi174 ceph-mon[31409]: Scrub error on inode 0x1000000073a (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/055.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.450 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:10 smithi174 ceph-mon[31409]: Scrub error on inode 0x1000000070f (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/035.out.nfs) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.450 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:10 smithi174 ceph-mon[31409]: Scrub error on inode 0x1000000073e (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/057.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.450 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:10 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000006c9 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/001.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.450 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:10 smithi174 ceph-mon[31409]: Scrub error on inode 0x1000000072f (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/050.out.nojournal) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.450 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:10 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000006ff (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/028.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.450 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:10 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000742 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/059.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.450 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-mon[31085]: pgmap v56: 129 pgs: 129 active+clean; 41 MiB data, 486 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.2 MiB/s wr, 63 op/s 2024-04-21T23:20:11.450 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-mon[31085]: scrub summary: active paths [/] 2024-04-21T23:20:11.450 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-mon[31085]: scrub summary: idle+waiting paths [/] 2024-04-21T23:20:11.450 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-mon[31085]: scrub summary: active paths [/] 2024-04-21T23:20:11.450 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-mon[31085]: scrub summary: active paths [/] 2024-04-21T23:20:11.450 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-mon[31085]: scrub summary: idle+waiting paths [/] 2024-04-21T23:20:11.450 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-mon[31085]: scrub summary: idle+waiting paths [/] 2024-04-21T23:20:11.451 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000006cf (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/004.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.451 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000006d7 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/008.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.451 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000006dd (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/011.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.451 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000006d1 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/005.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.451 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000006ed (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/019.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.451 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-mon[31085]: Scrub error on inode 0x1000000075e (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/073.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.451 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000721 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/044.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.451 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000006df (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/012.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.451 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000006cd (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/003.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.451 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-mon[31085]: Scrub error on inode 0x1000000074c (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/064.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.451 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000006e7 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/016.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.451 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-mon[31085]: Scrub error on inode 0x1000000075a (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/071.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.452 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000723 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/045.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.452 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000729 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/048.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.452 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000006db (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/010.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.452 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000006fb (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/026.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.452 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000006f9 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/025.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.452 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000744 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/060.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.452 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000734 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/052.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.452 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-mon[31085]: Scrub error on inode 0x1000000071d (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/042.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.452 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000006e3 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/014.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.452 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000006f7 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/024.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.452 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000006f5 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/023.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.452 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000758 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/070.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.452 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000006d5 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/007.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.452 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000754 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/068.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.452 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000748 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/062.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.453 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000750 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/066.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.453 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-mon[31085]: Scrub error on inode 0x1000000072b (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/049.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.453 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000738 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/054.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.453 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-mon[31085]: Scrub error on inode 0x1000000070e (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/035.out.default) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.453 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000736 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/053.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.453 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000746 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/061.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.453 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000730 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/050.out.xfsquota) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.453 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000006cb (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/002.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.453 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000711 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/036.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.453 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000006f3 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/022.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.453 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000006d9 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/009.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.453 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-mon[31085]: Scrub error on inode 0x1000000070d (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/035.cfg) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.453 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000006d3 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/006.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.453 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-mon[31085]: Scrub error on inode 0x1000000072d (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/050.cfg) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.453 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000006e9 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/017.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.453 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000727 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/047.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.454 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000740 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/058.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.454 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000006f1 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/021.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.454 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000006e5 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/015.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.454 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-mon[31085]: Scrub error on inode 0x1000000074e (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/065.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.454 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000717 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/039.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.454 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000715 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/038.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.454 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-mon[31085]: Scrub error on inode 0x1000000072e (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/050.out.default) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.454 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000756 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/069.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.454 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000732 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/051.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.454 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000006fd (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/027.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.454 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-mon[31085]: Scrub error on inode 0x1000000073c (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/056.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.454 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-mon[31085]: Scrub error on inode 0x1000000074a (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/063.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.454 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000006e1 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/013.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.454 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-mon[31085]: Scrub error on inode 0x1000000075c (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/072.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.455 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-mon[31085]: Scrub error on inode 0x1000000073a (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/055.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.455 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-mon[31085]: Scrub error on inode 0x1000000070f (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/035.out.nfs) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.455 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-mon[31085]: Scrub error on inode 0x1000000073e (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/057.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.455 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000006c9 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/001.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.455 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-mon[31085]: Scrub error on inode 0x1000000072f (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/050.out.nojournal) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.455 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000006ff (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/028.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:11.455 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:10 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000742 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/059.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:13.233 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:12 smithi040 ceph-mon[27984]: pgmap v57: 129 pgs: 129 active+clean; 41 MiB data, 486 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.0 MiB/s wr, 51 op/s 2024-04-21T23:20:13.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:12 smithi174 ceph-mon[31409]: pgmap v57: 129 pgs: 129 active+clean; 41 MiB data, 486 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.0 MiB/s wr, 51 op/s 2024-04-21T23:20:13.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:12 smithi111 ceph-mon[31085]: pgmap v57: 129 pgs: 129 active+clean; 41 MiB data, 486 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.0 MiB/s wr, 51 op/s 2024-04-21T23:20:13.945 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:13.533+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004b4 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/136.out auth v644 pv752 ap=2 s=43 n(v0 b43 1=1+0)->n(v0 b43 1=1+0) (ifile excl w=1) cr={25132=0-4194304@2} caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=1 lock=1 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3d60680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004b4:[<0x100000003a4/136.out v324>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000004b4:[<0x100000003a4/136.out v644>,<0x100000003a2/btrfs v19438>,<0x10000000007/tests v1939>,<0x10000000006/xfstests-dev v232>,<0x10000000005/tmp v45>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:13.945 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:13.534+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003ee [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/037.out auth v662 ap=2 s=125 n(v0 rc2024-04-21T23:19:50.253922+0000 b125 1=1+0) need_snapflush={3=25132} (ifile excl) cr={25132=0-4194304@2} caps={25132=pAsLsXsFscb/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3d61700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003ee:[<0x100000003a4/037.out v148>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000003ee:[<0x100000003a4/037.out v662>,<0x100000003a2/btrfs v19438>,<0x10000000007/tests v1939>,<0x10000000006/xfstests-dev v232>,<0x10000000005/tmp v45>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:13.945 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:13.535+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000486 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/113.out auth v648 ap=2 s=443 n(v0 rc2024-04-21T23:19:50.749915+0000 b443 1=1+0) need_snapflush={3=25132} (ifile excl) cr={25132=0-4194304@2} caps={25132=pAsLsXsFscb/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3d72b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000486:[<0x100000003a4/113.out v320>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000486:[<0x100000003a4/113.out v648>,<0x100000003a2/btrfs v19438>,<0x10000000007/tests v1939>,<0x10000000006/xfstests-dev v232>,<0x10000000005/tmp v45>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:13.945 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:13.535+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000042a [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/067.out auth v652 ap=2 s=43 n(v0 rc2024-04-21T23:19:50.314921+0000 b43 1=1+0) need_snapflush={3=25132} (ifile excl) cr={25132=0-4194304@2} caps={25132=pAsLsXsFscb/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3d73600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000042a:[<0x100000003a4/067.out v268>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x1000000042a:[<0x100000003a4/067.out v652>,<0x100000003a2/btrfs v19438>,<0x10000000007/tests v1939>,<0x10000000006/xfstests-dev v232>,<0x10000000005/tmp v45>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:13.945 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:13.536+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004d0 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/150.out auth v642 ap=2 s=118 n(v0 rc2024-04-21T23:19:50.850914+0000 b118 1=1+0) need_snapflush={3=25132} (ifile excl) cr={25132=0-4194304@2} caps={25132=pAsLsXsFscb/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3d74100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004d0:[<0x100000003a4/150.out v336>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000004d0:[<0x100000003a4/150.out v642>,<0x100000003a2/btrfs v19438>,<0x10000000007/tests v1939>,<0x10000000006/xfstests-dev v232>,<0x10000000005/tmp v45>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:13.945 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:13.538+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003f6 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/041.out auth v658 pv704 ap=2 s=1845 n(v0 rc2024-04-21T23:19:50.260922+0000 b1845 1=1+0)->n(v0 rc2024-04-21T23:19:50.260922+0000 b1845 1=1+0) (ifile excl w=1) cr={25132=0-4194304@2} caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=1 lock=1 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3d7f080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003f6:[<0x100000003a4/041.out v164>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000003f6:[<0x100000003a4/041.out v658>,<0x100000003a2/btrfs v19438>,<0x10000000007/tests v1939>,<0x10000000006/xfstests-dev v232>,<0x10000000005/tmp v45>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:13.946 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:13.539+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003e8 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/034.out auth v666 ap=2 s=93239 n(v0 rc2024-04-21T23:19:50.247922+0000 b93239 1=1+0) need_snapflush={3=25132} (ifile excl) cr={25132=0-4194304@2} caps={25132=pAsLsXsFscb/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3d7f600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003e8:[<0x100000003a4/034.out v136>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000003e8:[<0x100000003a4/034.out v666>,<0x100000003a2/btrfs v19438>,<0x10000000007/tests v1939>,<0x10000000006/xfstests-dev v232>,<0x10000000005/tmp v45>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:13.946 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:13.540+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000400 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/046.out auth v654 pv728 ap=2 s=10836 n(v0 rc2024-04-21T23:19:50.271922+0000 b10836 1=1+0)->n(v0 rc2024-04-21T23:19:50.271922+0000 b10836 1=1+0) (ifile excl w=1) cr={25132=0-4194304@2} caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=1 lock=1 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3d80680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000400:[<0x100000003a4/046.out v184>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000400:[<0x100000003a4/046.out v654>,<0x100000003a2/btrfs v19438>,<0x10000000007/tests v1939>,<0x10000000006/xfstests-dev v232>,<0x10000000005/tmp v45>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:13.946 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:13.540+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003c8 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/018.out auth v668 pv698 ap=2 s=43 n(v0 rc2024-04-21T23:19:50.215923+0000 b43 1=1+0)->n(v0 rc2024-04-21T23:19:50.215923+0000 b43 1=1+0) (ifile excl w=1) cr={25132=0-4194304@2} caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=1 lock=1 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3d80c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003c8:[<0x100000003a4/018.out v72>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000003c8:[<0x100000003a4/018.out v668>,<0x100000003a2/btrfs v19438>,<0x10000000007/tests v1939>,<0x10000000006/xfstests-dev v232>,<0x10000000005/tmp v45>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:13.946 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:13.540+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003fa [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/043.out auth v680 ap=1 s=25 n(v0 rc2024-04-21T23:19:50.264922+0000 b25 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3d81180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003fa:[<0x100000003a4/043.out v172>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000003fa:[<0x100000003a4/043.out v680>,<0x100000003a2/btrfs v19438>,<0x10000000007/tests v1939>,<0x10000000006/xfstests-dev v232>,<0x10000000005/tmp v45>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:13.946 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:13.543+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003ea [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/035.out auth v684 ap=1 s=418 n(v0 rc2024-04-21T23:19:50.249922+0000 b418 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3d8cb00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003ea:[<0x100000003a4/035.out v140>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000003ea:[<0x100000003a4/035.out v684>,<0x100000003a2/btrfs v19438>,<0x10000000007/tests v1939>,<0x10000000006/xfstests-dev v232>,<0x10000000005/tmp v45>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:13.946 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:13.546+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000510 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/182.out auth v634 pv714 ap=2 s=43 n(v0 rc2024-04-21T23:19:50.919913+0000 b43 1=1+0)->n(v0 rc2024-04-21T23:19:50.919913+0000 b43 1=1+0) (ifile excl w=1) cr={25132=0-4194304@2} caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=1 lock=1 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3d9b080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000510:[<0x100000003a4/182.out v354>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000510:[<0x100000003a4/182.out v634>,<0x100000003a2/btrfs v19438>,<0x10000000007/tests v1939>,<0x10000000006/xfstests-dev v232>,<0x10000000005/tmp v45>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:13.946 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:13.548+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000051c [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/188.out auth v632 pv718 ap=2 s=531 n(v0 rc2024-04-21T23:19:50.931913+0000 b531 1=1+0)->n(v0 rc2024-04-21T23:19:50.931913+0000 b531 1=1+0) (ifile excl w=1) cr={25132=0-4194304@2} caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=1 lock=1 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3d9d180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000051c:[<0x100000003a4/188.out v356>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x1000000051c:[<0x100000003a4/188.out v632>,<0x100000003a2/btrfs v19438>,<0x10000000007/tests v1939>,<0x10000000006/xfstests-dev v232>,<0x10000000005/tmp v45>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:13.946 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:13.548+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000480 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/110.out auth v650 pv778 ap=2 s=696 n(v0 rc2024-04-21T23:19:50.739915+0000 b696 1=1+0)->n(v0 rc2024-04-21T23:19:50.739915+0000 b696 1=1+0) (ifile excl w=1) cr={25132=0-4194304@2} caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=1 lock=1 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3da4b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000480:[<0x100000003a4/110.out v318>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000480:[<0x100000003a4/110.out v650>,<0x100000003a2/btrfs v19438>,<0x10000000007/tests v1939>,<0x10000000006/xfstests-dev v232>,<0x10000000005/tmp v45>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:13.946 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:13.548+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004f8 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/170.out auth v638 ap=2 s=368 n(v0 rc2024-04-21T23:19:50.893913+0000 b368 1=1+0) need_snapflush={3=25132} (ifile excl) cr={25132=0-4194304@2} caps={25132=pAsLsXsFscb/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3da5600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004f8:[<0x100000003a4/170.out v350>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000004f8:[<0x100000003a4/170.out v638>,<0x100000003a2/btrfs v19438>,<0x10000000007/tests v1939>,<0x10000000006/xfstests-dev v232>,<0x10000000005/tmp v45>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:13.946 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:13.550+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000050a [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/179.out auth v636 pv762 ap=2 s=43 n(v0 rc2024-04-21T23:19:50.912913+0000 b43 1=1+0)->n(v0 rc2024-04-21T23:19:50.912913+0000 b43 1=1+0) (ifile excl w=1) cr={25132=0-4194304@2} caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=1 lock=1 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3da6c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000050a:[<0x100000003a4/179.out v352>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x1000000050a:[<0x100000003a4/179.out v636>,<0x100000003a2/btrfs v19438>,<0x10000000007/tests v1939>,<0x10000000006/xfstests-dev v232>,<0x10000000005/tmp v45>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:13.946 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:13.551+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004e8 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/162.out auth v640 ap=2 s=167 n(v0 rc2024-04-21T23:19:50.876914+0000 b167 1=1+0) need_snapflush={3=25132} (ifile excl) cr={25132=0-4194304@2} caps={25132=pAsLsXsFscb/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3da7700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004e8:[<0x100000003a4/162.out v342>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000004e8:[<0x100000003a4/162.out v640>,<0x100000003a2/btrfs v19438>,<0x10000000007/tests v1939>,<0x10000000006/xfstests-dev v232>,<0x10000000005/tmp v45>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:13.946 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:13.552+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003f4 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/040.out auth v660 ap=2 s=25 n(v0 b25 1=1+0) need_snapflush={3=25132} (ifile excl) cr={25132=0-4194304@2} caps={25132=pAsLsXsFscb/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3db1600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003f4:[<0x100000003a4/040.out v160>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000003f4:[<0x100000003a4/040.out v660>,<0x100000003a2/btrfs v19438>,<0x10000000007/tests v1939>,<0x10000000006/xfstests-dev v232>,<0x10000000005/tmp v45>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:13.947 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:13.552+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000048a [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/115.out auth v646 pv724 ap=2 s=43 n(v0 rc2024-04-21T23:19:50.755915+0000 b43 1=1+0)->n(v0 rc2024-04-21T23:19:50.755915+0000 b43 1=1+0) (ifile excl w=1) cr={25132=0-4194304@2} caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=1 lock=1 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3db1b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000048a:[<0x100000003a4/115.out v322>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x1000000048a:[<0x100000003a4/115.out v646>,<0x100000003a2/btrfs v19438>,<0x10000000007/tests v1939>,<0x10000000006/xfstests-dev v232>,<0x10000000005/tmp v45>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:13.948 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:13.554+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003ac [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/004.out auth v658 pv696 ap=2 s=59 n(v0 rc2024-04-21T23:19:50.187923+0000 b59 1=1+0)->n(v0 rc2024-04-21T23:19:50.187923+0000 b59 1=1+0) (ifile excl w=1) cr={25132=0-4194304@2} caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=1 lock=1 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3dc2000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003ac:[<0x100000003a4/004.out v16>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000003ac:[<0x100000003a4/004.out v658>,<0x100000003a2/btrfs v19438>,<0x10000000007/tests v1939>,<0x10000000006/xfstests-dev v232>,<0x10000000005/tmp v45>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:13.949 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:13.554+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003b4 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/008.out auth v674 ap=1 s=43 n(v0 rc2024-04-21T23:19:50.195923+0000 b43 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3dc2580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003b4:[<0x100000003a4/008.out v32>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000003b4:[<0x100000003a4/008.out v674>,<0x100000003a2/btrfs v19438>,<0x10000000007/tests v1939>,<0x10000000006/xfstests-dev v232>,<0x10000000005/tmp v45>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:13.949 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:13.555+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003ba [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/011.out auth v652 ap=2 s=57 n(v0 rc2024-04-21T23:19:50.201923+0000 b57 1=1+0) need_snapflush={3=25132} (ifile excl) cr={25132=0-4194304@2} caps={25132=pAsLsXsFscb/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3dc3600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003ba:[<0x100000003a4/011.out v44>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000003ba:[<0x100000003a4/011.out v652>,<0x100000003a2/btrfs v19438>,<0x10000000007/tests v1939>,<0x10000000006/xfstests-dev v232>,<0x10000000005/tmp v45>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:13.949 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:13.556+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000498 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/122.out auth v628 pv686 ap=2 s=25 n(v0 rc2024-04-21T23:19:50.777915+0000 b25 1=1+0)->n(v0 rc2024-04-21T23:19:50.777915+0000 b25 1=1+0) (ifile excl w=1) cr={25132=0-4194304@2} caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=1 lock=1 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3dc4680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000498:[<0x100000003a4/122.out v318>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000498:[<0x100000003a4/122.out v628>,<0x100000003a2/btrfs v19438>,<0x10000000007/tests v1939>,<0x10000000006/xfstests-dev v232>,<0x10000000005/tmp v45>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:13.949 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:13.557+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003ae [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/005.out auth v664 ap=1 s=564 n(v0 rc2024-04-21T23:19:50.189923+0000 b564 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3dc4c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003ae:[<0x100000003a4/005.out v20>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000003ae:[<0x100000003a4/005.out v664>,<0x100000003a2/btrfs v19438>,<0x10000000007/tests v1939>,<0x10000000006/xfstests-dev v232>,<0x10000000005/tmp v45>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:13.949 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:13.558+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000466 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/097.out auth v634 ap=2 s=453 n(v0 rc2024-04-21T23:19:50.697916+0000 b453 1=1+0) need_snapflush={3=25132} (ifile excl) cr={25132=0-4194304@2} caps={25132=pAsLsXsFscb/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3dc5180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000466:[<0x100000003a4/097.out v312>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000466:[<0x100000003a4/097.out v634>,<0x100000003a2/btrfs v19438>,<0x10000000007/tests v1939>,<0x10000000006/xfstests-dev v232>,<0x10000000005/tmp v45>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:13.949 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:13.558+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003ca [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/019.out auth v648 pv710 ap=2 s=43 n(v0 rc2024-04-21T23:19:50.217923+0000 b43 1=1+0)->n(v0 rc2024-04-21T23:19:50.217923+0000 b43 1=1+0) (ifile excl w=1) cr={25132=0-4194304@2} caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=1 lock=1 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3dc5700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003ca:[<0x100000003a4/019.out v76>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000003ca:[<0x100000003a4/019.out v648>,<0x100000003a2/btrfs v19438>,<0x10000000007/tests v1939>,<0x10000000006/xfstests-dev v232>,<0x10000000005/tmp v45>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:13.949 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:13.559+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000043e [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/077.out auth v640 pv714 ap=2 s=43 n(v0 rc2024-04-21T23:19:50.604917+0000 b43 1=1+0)->n(v0 rc2024-04-21T23:19:50.604917+0000 b43 1=1+0) (ifile excl w=1) cr={25132=0-4194304@2} caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=1 lock=1 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3dd2000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000043e:[<0x100000003a4/077.out v304>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x1000000043e:[<0x100000003a4/077.out v640>,<0x100000003a2/btrfs v19438>,<0x10000000007/tests v1939>,<0x10000000006/xfstests-dev v232>,<0x10000000005/tmp v45>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:13.949 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:13.560+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000048e [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/117.out auth v632 pv794 ap=2 s=658 n(v0 rc2024-04-21T23:19:50.762915+0000 b658 1=1+0)->n(v0 rc2024-04-21T23:19:50.762915+0000 b658 1=1+0) (ifile excl w=1) cr={25132=0-4194304@2} caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=1 lock=1 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3dd2580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000048e:[<0x100000003a4/117.out v314>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x1000000048e:[<0x100000003a4/117.out v632>,<0x100000003a2/btrfs v19438>,<0x10000000007/tests v1939>,<0x10000000006/xfstests-dev v232>,<0x10000000005/tmp v45>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:13.949 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:13.560+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000436 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/073.out auth v642 ap=2 s=43 n(v0 rc2024-04-21T23:19:50.326921+0000 b43 1=1+0) need_snapflush={3=25132} (ifile excl) cr={25132=0-4194304@2} caps={25132=pAsLsXsFscb/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3dd3b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000436:[<0x100000003a4/073.out v292>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000436:[<0x100000003a4/073.out v642>,<0x100000003a2/btrfs v19438>,<0x10000000007/tests v1939>,<0x10000000006/xfstests-dev v232>,<0x10000000005/tmp v45>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:13.949 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:13.561+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000492 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/119.out auth v630 pv682 ap=2 s=388 n(v0 rc2024-04-21T23:19:50.768915+0000 b388 1=1+0)->n(v0 rc2024-04-21T23:19:50.768915+0000 b388 1=1+0) (ifile excl w=1) cr={25132=0-4194304@2} caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=1 lock=1 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3dd4680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000492:[<0x100000003a4/119.out v316>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000492:[<0x100000003a4/119.out v630>,<0x100000003a2/btrfs v19438>,<0x10000000007/tests v1939>,<0x10000000006/xfstests-dev v232>,<0x10000000005/tmp v45>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:13.949 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:13.561+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000512 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/183.out auth v608 ap=2 s=968 n(v0 rc2024-04-21T23:19:50.921913+0000 b968 1=1+0) need_snapflush={3=25132} (ifile excl) cr={25132=0-4194304@2} caps={25132=pAsLsXsFscb/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3dd4c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000512:[<0x100000003a4/183.out v344>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000512:[<0x100000003a4/183.out v608>,<0x100000003a2/btrfs v19438>,<0x10000000007/tests v1939>,<0x10000000006/xfstests-dev v232>,<0x10000000005/tmp v45>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:13.950 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:13.562+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000049c [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/124.out auth v626 pv744 ap=2 s=118 n(v0 rc2024-04-21T23:19:50.784915+0000 b118 1=1+0)->n(v0 rc2024-04-21T23:19:50.784915+0000 b118 1=1+0) (ifile excl w=1) cr={25132=0-4194304@2} caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=1 lock=1 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3ddc000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000049c:[<0x100000003a4/124.out v320>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x1000000049c:[<0x100000003a4/124.out v626>,<0x100000003a2/btrfs v19438>,<0x10000000007/tests v1939>,<0x10000000006/xfstests-dev v232>,<0x10000000005/tmp v45>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:13.950 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:13.563+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003fc [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/044.out auth v646 pv702 ap=2 s=25 n(v0 rc2024-04-21T23:19:50.267922+0000 b25 1=1+0)->n(v0 rc2024-04-21T23:19:50.267922+0000 b25 1=1+0) (ifile excl w=1) cr={25132=0-4194304@2} caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=1 lock=1 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3ddc580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003fc:[<0x100000003a4/044.out v176>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000003fc:[<0x100000003a4/044.out v646>,<0x100000003a2/btrfs v19438>,<0x10000000007/tests v1939>,<0x10000000006/xfstests-dev v232>,<0x10000000005/tmp v45>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:13.950 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:13.565+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000049e [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/125.out auth v624 pv722 ap=2 s=118 n(v0 rc2024-04-21T23:19:50.787915+0000 b118 1=1+0)->n(v0 rc2024-04-21T23:19:50.787915+0000 b118 1=1+0) (ifile excl w=1) cr={25132=0-4194304@2} caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=1 lock=1 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3dddb80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000049e:[<0x100000003a4/125.out v322>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x1000000049e:[<0x100000003a4/125.out v624>,<0x100000003a2/btrfs v19438>,<0x10000000007/tests v1939>,<0x10000000006/xfstests-dev v232>,<0x10000000005/tmp v45>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:13.950 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:13.565+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004f6 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/169.out auth v614 pv752 ap=2 s=541 n(v0 rc2024-04-21T23:19:50.890913+0000 b541 1=1+0)->n(v0 rc2024-04-21T23:19:50.890913+0000 b541 1=1+0) (ifile excl w=1) cr={25132=0-4194304@2} caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=1 lock=1 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3ddf700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004f6:[<0x100000003a4/169.out v334>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000004f6:[<0x100000003a4/169.out v614>,<0x100000003a2/btrfs v19438>,<0x10000000007/tests v1939>,<0x10000000006/xfstests-dev v232>,<0x10000000005/tmp v45>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:13.950 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:13.566+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000051e [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/189.out auth v606 ap=2 s=846 n(v0 rc2024-04-21T23:19:50.933913+0000 b846 1=1+0) need_snapflush={3=25132} (ifile excl) cr={25132=0-4194304@2} caps={25132=pAsLsXsFscb/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3ded600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000051e:[<0x100000003a4/189.out v348>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x1000000051e:[<0x100000003a4/189.out v606>,<0x100000003a2/btrfs v19438>,<0x10000000007/tests v1939>,<0x10000000006/xfstests-dev v232>,<0x10000000005/tmp v45>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:13.950 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:13.567+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000045a [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/091.out auth v638 pv756 ap=2 s=173 n(v0 rc2024-04-21T23:19:50.678916+0000 b173 1=1+0)->n(v0 rc2024-04-21T23:19:50.678916+0000 b173 1=1+0) (ifile excl w=1) cr={25132=0-4194304@2} caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=1 lock=1 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3dee100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000045a:[<0x100000003a4/091.out v306>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x1000000045a:[<0x100000003a4/091.out v638>,<0x100000003a2/btrfs v19438>,<0x10000000007/tests v1939>,<0x10000000006/xfstests-dev v232>,<0x10000000005/tmp v45>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:13.950 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:13.568+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004fa [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/171.out auth v612 ap=2 s=43 n(v0 rc2024-04-21T23:19:50.895913+0000 b43 1=1+0) need_snapflush={3=25132} (ifile excl) cr={25132=0-4194304@2} caps={25132=pAsLsXsFscb/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3dee680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004fa:[<0x100000003a4/171.out v336>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000004fa:[<0x100000003a4/171.out v612>,<0x100000003a2/btrfs v19438>,<0x10000000007/tests v1939>,<0x10000000006/xfstests-dev v232>,<0x10000000005/tmp v45>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:13.950 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:13.568+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003aa [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/003.out auth v668 ap=1 s=43 n(v0 rc2024-04-21T23:19:50.184923+0000 b43 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3df4580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003aa:[<0x100000003a4/003.out v12>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000003aa:[<0x100000003a4/003.out v668>,<0x100000003a2/btrfs v19438>,<0x10000000007/tests v1939>,<0x10000000006/xfstests-dev v232>,<0x10000000005/tmp v45>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:13.950 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:13.569+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004de [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/157.out auth v616 pv760 ap=2 s=180 n(v0 rc2024-04-21T23:19:50.865914+0000 b180 1=1+0)->n(v0 rc2024-04-21T23:19:50.865914+0000 b180 1=1+0) (ifile excl w=1) cr={25132=0-4194304@2} caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=1 lock=1 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3df5600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004de:[<0x100000003a4/157.out v332>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000004de:[<0x100000003a4/157.out v616>,<0x100000003a2/btrfs v19438>,<0x10000000007/tests v1939>,<0x10000000006/xfstests-dev v232>,<0x10000000005/tmp v45>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:13.950 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:13.569+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004a0 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/126.out auth v622 pv726 ap=2 s=53 n(v0 rc2024-04-21T23:19:50.790915+0000 b53 1=1+0)->n(v0 rc2024-04-21T23:19:50.790915+0000 b53 1=1+0) (ifile excl w=1) cr={25132=0-4194304@2} caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=1 lock=1 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3df5b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004a0:[<0x100000003a4/126.out v324>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000004a0:[<0x100000003a4/126.out v622>,<0x100000003a2/btrfs v19438>,<0x10000000007/tests v1939>,<0x10000000006/xfstests-dev v232>,<0x10000000005/tmp v45>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:13.950 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:13.570+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000050e [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/181.out auth v610 ap=2 s=43 n(v0 rc2024-04-21T23:19:50.916913+0000 b43 1=1+0) need_snapflush={3=25132} (ifile excl) cr={25132=0-4194304@2} caps={25132=pAsLsXsFscb/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3df6c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000050e:[<0x100000003a4/181.out v342>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x1000000050e:[<0x100000003a4/181.out v610>,<0x100000003a2/btrfs v19438>,<0x10000000007/tests v1939>,<0x10000000006/xfstests-dev v232>,<0x10000000005/tmp v45>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:13.950 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:13.571+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000424 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/064.out auth v644 pv690 ap=2 s=43 n(v0 rc2024-04-21T23:19:50.308922+0000 b43 1=1+0)->n(v0 rc2024-04-21T23:19:50.308922+0000 b43 1=1+0) (ifile excl w=1) cr={25132=0-4194304@2} caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=1 lock=1 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3df7700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000424:[<0x100000003a4/064.out v256>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000424:[<0x100000003a4/064.out v644>,<0x100000003a2/btrfs v19438>,<0x10000000007/tests v1939>,<0x10000000006/xfstests-dev v232>,<0x10000000005/tmp v45>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:13.951 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:13.572+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003c4 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/016.out auth v650 pv784 ap=2 s=43 n(v0 rc2024-04-21T23:19:50.210923+0000 b43 1=1+0)->n(v0 rc2024-04-21T23:19:50.210923+0000 b43 1=1+0) (ifile excl w=1) cr={25132=0-4194304@2} caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=1 lock=1 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3e04b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003c4:[<0x100000003a4/016.out v64>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000003c4:[<0x100000003a4/016.out v650>,<0x100000003a2/btrfs v19438>,<0x10000000007/tests v1939>,<0x10000000006/xfstests-dev v232>,<0x10000000005/tmp v45>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:13.951 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:13.574+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004d4 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/152.out auth v618 ap=2 s=487 n(v0 rc2024-04-21T23:19:50.855914+0000 b487 1=1+0) need_snapflush={3=25132} (ifile excl) cr={25132=0-4194304@2} caps={25132=pAsLsXsFscb/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3e06680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004d4:[<0x100000003a4/152.out v330>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000004d4:[<0x100000003a4/152.out v618>,<0x100000003a2/btrfs v19438>,<0x10000000007/tests v1939>,<0x10000000006/xfstests-dev v232>,<0x10000000005/tmp v45>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:13.951 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:13.574+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004a4 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/128.out auth v620 pv748 ap=2 s=43 n(v0 rc2024-04-21T23:19:50.797915+0000 b43 1=1+0)->n(v0 rc2024-04-21T23:19:50.797915+0000 b43 1=1+0) (ifile excl w=1) cr={25132=0-4194304@2} caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=1 lock=1 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3e07180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004a4:[<0x100000003a4/128.out v326>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000004a4:[<0x100000003a4/128.out v620>,<0x100000003a2/btrfs v19438>,<0x10000000007/tests v1939>,<0x10000000006/xfstests-dev v232>,<0x10000000005/tmp v45>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:13.951 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:13.575+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000464 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/096.out auth v636 pv706 ap=2 s=181 n(v0 rc2024-04-21T23:19:50.694916+0000 b181 1=1+0)->n(v0 rc2024-04-21T23:19:50.694916+0000 b181 1=1+0) (ifile excl w=1) cr={25132=0-4194304@2} caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=1 lock=1 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3e18c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000464:[<0x100000003a4/096.out v310>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000464:[<0x100000003a4/096.out v636>,<0x100000003a2/btrfs v19438>,<0x10000000007/tests v1939>,<0x10000000006/xfstests-dev v232>,<0x10000000005/tmp v45>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:14.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:13 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000004b4 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/136.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.433 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:13 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000003ee (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/037.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.433 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:13 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000486 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/113.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.433 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:13 smithi040 ceph-mon[27984]: Scrub error on inode 0x1000000042a (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/067.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.433 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:13 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000004d0 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/150.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.433 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:13 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000003f6 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/041.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.433 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:13 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000003e8 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/034.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.433 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:13 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000400 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/046.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.433 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:13 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000003c8 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/018.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.433 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:13 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000003fa (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/043.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.433 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:13 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000003ea (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/035.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.434 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:13 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000510 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/182.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.434 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:13 smithi040 ceph-mon[27984]: Scrub error on inode 0x1000000051c (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/188.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.434 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:13 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000480 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/110.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.434 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:13 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000004f8 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/170.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.434 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:13 smithi040 ceph-mon[27984]: Scrub error on inode 0x1000000050a (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/179.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.434 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:13 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000004e8 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/162.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.434 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:13 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000003f4 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/040.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.434 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:13 smithi040 ceph-mon[27984]: Scrub error on inode 0x1000000048a (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/115.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.434 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:13 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000003ac (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/004.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.434 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:13 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000003b4 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/008.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.434 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:13 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000003ba (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/011.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.434 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:13 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000498 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/122.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.434 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:13 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000003ae (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/005.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.435 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:13 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000466 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/097.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.435 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:13 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000003ca (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/019.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.435 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:13 smithi040 ceph-mon[27984]: Scrub error on inode 0x1000000043e (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/077.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.435 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:13 smithi040 ceph-mon[27984]: Scrub error on inode 0x1000000048e (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/117.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.435 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:13 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000436 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/073.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.435 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:13 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000492 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/119.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.435 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:13 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000512 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/183.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.435 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:13 smithi040 ceph-mon[27984]: Scrub error on inode 0x1000000049c (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/124.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.435 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:13 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000003fc (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/044.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.435 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:13 smithi040 ceph-mon[27984]: Scrub error on inode 0x1000000049e (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/125.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.435 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:13 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000004f6 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/169.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.435 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:13 smithi040 ceph-mon[27984]: Scrub error on inode 0x1000000051e (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/189.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.436 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:13 smithi040 ceph-mon[27984]: Scrub error on inode 0x1000000045a (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/091.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.436 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:13 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000004fa (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/171.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.436 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:13 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000003aa (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/003.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.436 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:13 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000004de (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/157.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.436 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:13 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000004a0 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/126.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.436 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:13 smithi040 ceph-mon[27984]: Scrub error on inode 0x1000000050e (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/181.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.436 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:13 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000424 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/064.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.436 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:13 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000003c4 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/016.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.436 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:13 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000004d4 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/152.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.436 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:13 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000004a4 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/128.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.436 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:13 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000464 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/096.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:13 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000004b4 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/136.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:13 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000003ee (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/037.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:13 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000486 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/113.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:13 smithi174 ceph-mon[31409]: Scrub error on inode 0x1000000042a (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/067.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:13 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000004d0 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/150.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:13 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000003f6 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/041.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:13 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000003e8 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/034.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:13 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000400 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/046.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:13 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000003c8 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/018.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:13 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000003fa (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/043.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:13 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000003ea (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/035.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:13 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000510 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/182.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:13 smithi174 ceph-mon[31409]: Scrub error on inode 0x1000000051c (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/188.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:13 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000480 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/110.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:13 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000004f8 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/170.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:13 smithi174 ceph-mon[31409]: Scrub error on inode 0x1000000050a (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/179.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:13 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000004e8 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/162.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:13 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000003f4 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/040.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:13 smithi174 ceph-mon[31409]: Scrub error on inode 0x1000000048a (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/115.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:13 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000003ac (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/004.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:13 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000003b4 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/008.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.446 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:13 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000003ba (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/011.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.446 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:13 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000498 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/122.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.446 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:13 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000003ae (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/005.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.446 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:13 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000466 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/097.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.446 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:13 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000003ca (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/019.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.446 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:13 smithi174 ceph-mon[31409]: Scrub error on inode 0x1000000043e (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/077.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.446 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:13 smithi174 ceph-mon[31409]: Scrub error on inode 0x1000000048e (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/117.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.446 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:13 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000436 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/073.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.446 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:13 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000492 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/119.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.446 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:13 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000512 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/183.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.446 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:13 smithi174 ceph-mon[31409]: Scrub error on inode 0x1000000049c (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/124.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.446 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:13 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000003fc (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/044.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.446 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:13 smithi174 ceph-mon[31409]: Scrub error on inode 0x1000000049e (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/125.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.447 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:13 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000004f6 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/169.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.447 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:13 smithi174 ceph-mon[31409]: Scrub error on inode 0x1000000051e (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/189.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.447 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:13 smithi174 ceph-mon[31409]: Scrub error on inode 0x1000000045a (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/091.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.447 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:13 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000004fa (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/171.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.447 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:13 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000003aa (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/003.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.447 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:13 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000004de (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/157.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.447 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:13 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000004a0 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/126.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.447 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:13 smithi174 ceph-mon[31409]: Scrub error on inode 0x1000000050e (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/181.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.447 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:13 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000424 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/064.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.447 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:13 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000003c4 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/016.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.447 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:13 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000004d4 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/152.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.447 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:13 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000004a4 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/128.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.447 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:13 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000464 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/096.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.448 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000004b4 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/136.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.448 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000003ee (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/037.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.448 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000486 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/113.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.448 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-mon[31085]: Scrub error on inode 0x1000000042a (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/067.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.448 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000004d0 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/150.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.448 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000003f6 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/041.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.448 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000003e8 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/034.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.448 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000400 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/046.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.448 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000003c8 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/018.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.449 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000003fa (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/043.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.449 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000003ea (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/035.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.449 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000510 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/182.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.449 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-mon[31085]: Scrub error on inode 0x1000000051c (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/188.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.449 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000480 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/110.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.449 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000004f8 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/170.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.449 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-mon[31085]: Scrub error on inode 0x1000000050a (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/179.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.449 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000004e8 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/162.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.449 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000003f4 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/040.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.449 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-mon[31085]: Scrub error on inode 0x1000000048a (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/115.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.449 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000003ac (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/004.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.449 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000003b4 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/008.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.449 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000003ba (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/011.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.449 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000498 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/122.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.450 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000003ae (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/005.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.450 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000466 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/097.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.450 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000003ca (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/019.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.450 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-mon[31085]: Scrub error on inode 0x1000000043e (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/077.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.450 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-mon[31085]: Scrub error on inode 0x1000000048e (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/117.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.450 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000436 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/073.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.450 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000492 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/119.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.450 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000512 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/183.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.450 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-mon[31085]: Scrub error on inode 0x1000000049c (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/124.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.450 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000003fc (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/044.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.450 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-mon[31085]: Scrub error on inode 0x1000000049e (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/125.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.450 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000004f6 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/169.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.450 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-mon[31085]: Scrub error on inode 0x1000000051e (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/189.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.450 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-mon[31085]: Scrub error on inode 0x1000000045a (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/091.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.451 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000004fa (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/171.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.451 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000003aa (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/003.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.451 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000004de (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/157.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.451 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000004a0 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/126.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.451 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-mon[31085]: Scrub error on inode 0x1000000050e (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/181.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.451 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000424 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/064.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.451 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000003c4 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/016.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.451 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000004d4 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/152.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.451 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000004a4 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/128.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:14.451 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:13 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000464 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/096.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:15.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:14 smithi040 ceph-mon[27984]: pgmap v58: 129 pgs: 129 active+clean; 41 MiB data, 486 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.0 MiB/s wr, 51 op/s 2024-04-21T23:20:15.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:14 smithi174 ceph-mon[31409]: pgmap v58: 129 pgs: 129 active+clean; 41 MiB data, 486 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.0 MiB/s wr, 51 op/s 2024-04-21T23:20:15.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:14 smithi111 ceph-mon[31085]: pgmap v58: 129 pgs: 129 active+clean; 41 MiB data, 486 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.0 MiB/s wr, 51 op/s 2024-04-21T23:20:17.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:16 smithi040 ceph-mon[27984]: pgmap v59: 129 pgs: 129 active+clean; 63 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.1 MiB/s wr, 443 op/s 2024-04-21T23:20:17.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:16 smithi174 ceph-mon[31409]: pgmap v59: 129 pgs: 129 active+clean; 63 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.1 MiB/s wr, 443 op/s 2024-04-21T23:20:17.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:16 smithi111 ceph-mon[31085]: pgmap v59: 129 pgs: 129 active+clean; 63 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.1 MiB/s wr, 443 op/s 2024-04-21T23:20:18.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:17 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mon-a[27980]: 2024-04-21T23:20:17.968+0000 7fd81e2c2700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-04-21T23:20:18.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:18 smithi040 ceph-mon[27984]: Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-04-21T23:20:18.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:18 smithi040 ceph-mon[27984]: mds.? [v2:172.21.15.111:6836/2703764182,v1:172.21.15.111:6837/2703764182] up:active 2024-04-21T23:20:18.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:18 smithi040 ceph-mon[27984]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-21T23:20:18.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:18 smithi174 ceph-mon[31409]: Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-04-21T23:20:18.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:18 smithi174 ceph-mon[31409]: mds.? [v2:172.21.15.111:6836/2703764182,v1:172.21.15.111:6837/2703764182] up:active 2024-04-21T23:20:18.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:18 smithi174 ceph-mon[31409]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-21T23:20:18.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:18 smithi111 ceph-mon[31085]: Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-04-21T23:20:18.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:18 smithi111 ceph-mon[31085]: mds.? [v2:172.21.15.111:6836/2703764182,v1:172.21.15.111:6837/2703764182] up:active 2024-04-21T23:20:18.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:18 smithi111 ceph-mon[31085]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-21T23:20:18.974 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.973+0000 7f3f235e5640 1 -- 172.21.15.40:0/2878537498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f1c103310 msgr2=0x7f3f1c10b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:18.975 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.973+0000 7f3f235e5640 1 --2- 172.21.15.40:0/2878537498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f1c103310 0x7f3f1c10b7f0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f3f0c009900 tx=0x7f3f0c02bcd0 comp rx=0 tx=0).stop 2024-04-21T23:20:18.975 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.973+0000 7f3f235e5640 1 -- 172.21.15.40:0/2878537498 shutdown_connections 2024-04-21T23:20:18.975 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.973+0000 7f3f235e5640 1 --2- 172.21.15.40:0/2878537498 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f1c10bdc0 0x7f3f1c10e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:18.975 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.973+0000 7f3f235e5640 1 --2- 172.21.15.40:0/2878537498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f1c103310 0x7f3f1c10b7f0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:18.975 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.973+0000 7f3f235e5640 1 --2- 172.21.15.40:0/2878537498 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f1c1029f0 0x7f3f1c102dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:18.975 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.973+0000 7f3f235e5640 1 -- 172.21.15.40:0/2878537498 >> 172.21.15.40:0/2878537498 conn(0x7f3f1c0769c0 msgr2=0x7f3f1c076dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:18.975 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.973+0000 7f3f235e5640 1 -- 172.21.15.40:0/2878537498 shutdown_connections 2024-04-21T23:20:18.975 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.973+0000 7f3f235e5640 1 -- 172.21.15.40:0/2878537498 wait complete. 2024-04-21T23:20:18.975 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.974+0000 7f3f235e5640 1 Processor -- start 2024-04-21T23:20:18.975 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.974+0000 7f3f235e5640 1 -- start start 2024-04-21T23:20:18.975 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.974+0000 7f3f235e5640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f1c1029f0 0x7f3f1c197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:18.976 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.974+0000 7f3f235e5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f1c103310 0x7f3f1c198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:18.976 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.974+0000 7f3f235e5640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f1c10bdc0 0x7f3f1c19c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:18.976 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.974+0000 7f3f235e5640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3f1c111b50 con 0x7f3f1c103310 2024-04-21T23:20:18.976 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.974+0000 7f3f235e5640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3f1c1119d0 con 0x7f3f1c1029f0 2024-04-21T23:20:18.976 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.974+0000 7f3f235e5640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3f1c111cd0 con 0x7f3f1c10bdc0 2024-04-21T23:20:18.976 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.974+0000 7f3f20b80640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f1c103310 0x7f3f1c198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:18.976 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.974+0000 7f3f20b80640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f1c103310 0x7f3f1c198510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:53186/0 (socket says 172.21.15.40:53186) 2024-04-21T23:20:18.976 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.974+0000 7f3f20b80640 1 -- 172.21.15.40:0/1862785130 learned_addr learned my addr 172.21.15.40:0/1862785130 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:20:18.976 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.974+0000 7f3f21381640 1 --2- 172.21.15.40:0/1862785130 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f1c1029f0 0x7f3f1c197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:18.976 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.974+0000 7f3f20b80640 1 -- 172.21.15.40:0/1862785130 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f1c10bdc0 msgr2=0x7f3f1c19c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:18.976 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.974+0000 7f3f21b82640 1 --2- 172.21.15.40:0/1862785130 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f1c10bdc0 0x7f3f1c19c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:18.976 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.974+0000 7f3f20b80640 1 --2- 172.21.15.40:0/1862785130 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f1c10bdc0 0x7f3f1c19c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:18.976 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.974+0000 7f3f20b80640 1 -- 172.21.15.40:0/1862785130 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f1c1029f0 msgr2=0x7f3f1c197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:18.976 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.974+0000 7f3f20b80640 1 --2- 172.21.15.40:0/1862785130 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f1c1029f0 0x7f3f1c197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:18.977 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.974+0000 7f3f20b80640 1 -- 172.21.15.40:0/1862785130 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f1c19cfc0 con 0x7f3f1c103310 2024-04-21T23:20:18.977 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.974+0000 7f3f21381640 1 --2- 172.21.15.40:0/1862785130 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f1c1029f0 0x7f3f1c197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:20:18.977 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.975+0000 7f3f20b80640 1 --2- 172.21.15.40:0/1862785130 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f1c103310 0x7f3f1c198510 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f3f0c005940 tx=0x7f3f0c02cd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:18.977 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.975+0000 7f3f127fc640 1 -- 172.21.15.40:0/1862785130 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f0c041070 con 0x7f3f1c103310 2024-04-21T23:20:18.977 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.975+0000 7f3f127fc640 1 -- 172.21.15.40:0/1862785130 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f0c004450 con 0x7f3f1c103310 2024-04-21T23:20:18.977 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.975+0000 7f3f235e5640 1 -- 172.21.15.40:0/1862785130 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f1c19d250 con 0x7f3f1c103310 2024-04-21T23:20:18.977 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.975+0000 7f3f127fc640 1 -- 172.21.15.40:0/1862785130 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f0c034440 con 0x7f3f1c103310 2024-04-21T23:20:18.977 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.975+0000 7f3f235e5640 1 -- 172.21.15.40:0/1862785130 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f1c1a4b40 con 0x7f3f1c103310 2024-04-21T23:20:18.984 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.977+0000 7f3f127fc640 1 -- 172.21.15.40:0/1862785130 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f3f0c044070 con 0x7f3f1c103310 2024-04-21T23:20:18.985 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.977+0000 7f3f261d0740 1 Processor -- start 2024-04-21T23:20:18.985 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.977+0000 7f3f261d0740 1 -- start start 2024-04-21T23:20:18.985 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.978+0000 7f3f261d0740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55897f9944e0 0x55897f9948c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:18.985 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.978+0000 7f3f261d0740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55897f994e00 0x55897f9a20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:18.985 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.978+0000 7f3f261d0740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55897f9a2620 0x55897f9a4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:18.985 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.978+0000 7f3f261d0740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55897f8f8b70 con 0x55897f994e00 2024-04-21T23:20:18.985 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.978+0000 7f3f261d0740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55897f8a1470 con 0x55897f9a2620 2024-04-21T23:20:18.985 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.978+0000 7f3f261d0740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55897f8f4b30 con 0x55897f9944e0 2024-04-21T23:20:18.985 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.978+0000 7f3f127fc640 1 --2- 172.21.15.40:0/1862785130 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f3f080a2e20 0x7f3f080a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:18.985 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.978+0000 7f3f21381640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55897f994e00 0x55897f9a20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:18.986 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.978+0000 7f3f21381640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55897f994e00 0x55897f9a20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:53202/0 (socket says 172.21.15.40:53202) 2024-04-21T23:20:18.986 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.978+0000 7f3f21381640 1 -- 172.21.15.40:0/711781374 learned_addr learned my addr 172.21.15.40:0/711781374 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:20:18.986 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.978+0000 7f3f20b80640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55897f9a2620 0x55897f9a4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:18.986 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.978+0000 7f3f21b82640 1 --2- 172.21.15.40:0/711781374 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55897f9944e0 0x55897f9948c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:18.986 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.978+0000 7f3f127fc640 1 -- 172.21.15.40:0/1862785130 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f3f0c0f1290 con 0x7f3f1c103310 2024-04-21T23:20:18.986 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.978+0000 7f3f21b82640 1 --2- 172.21.15.40:0/1862785130 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f3f080a2e20 0x7f3f080a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:18.986 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.978+0000 7f3f21381640 1 -- 172.21.15.40:0/711781374 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55897f9944e0 msgr2=0x55897f9948c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:18.986 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.978+0000 7f3f21381640 1 --2- 172.21.15.40:0/711781374 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55897f9944e0 0x55897f9948c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:18.986 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.978+0000 7f3f21381640 1 -- 172.21.15.40:0/711781374 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55897f9a2620 msgr2=0x55897f9a4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:18.986 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.978+0000 7f3f21381640 1 --2- 172.21.15.40:0/711781374 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55897f9a2620 0x55897f9a4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:18.987 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.978+0000 7f3f21381640 1 -- 172.21.15.40:0/711781374 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55897f60d210 con 0x55897f994e00 2024-04-21T23:20:18.987 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.978+0000 7f3f21b82640 1 --2- 172.21.15.40:0/711781374 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55897f9944e0 0x55897f9948c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:20:18.987 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.978+0000 7f3f21381640 1 --2- 172.21.15.40:0/711781374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55897f994e00 0x55897f9a20e0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f3f18010480 tx=0x7f3f18010940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64436d4d7c6deffe server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:18.987 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.978+0000 7f3f21b82640 1 --2- 172.21.15.40:0/1862785130 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f3f080a2e20 0x7f3f080a52e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f3f1c06b050 tx=0x7f3f14005c60 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:18.987 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.979+0000 7f3f077fe640 1 -- 172.21.15.40:0/711781374 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f18011760 con 0x55897f994e00 2024-04-21T23:20:18.987 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.979+0000 7f3f077fe640 1 -- 172.21.15.40:0/711781374 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f18011de0 con 0x55897f994e00 2024-04-21T23:20:18.987 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.979+0000 7f3f077fe640 1 -- 172.21.15.40:0/711781374 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f1800e430 con 0x55897f994e00 2024-04-21T23:20:18.987 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.979+0000 7f3f261d0740 1 -- 172.21.15.40:0/711781374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55897f994e00 msgr2=0x55897f9a20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:18.987 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.979+0000 7f3f261d0740 1 --2- 172.21.15.40:0/711781374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55897f994e00 0x55897f9a20e0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f3f18010480 tx=0x7f3f18010940 comp rx=0 tx=0).stop 2024-04-21T23:20:18.987 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.979+0000 7f3f261d0740 1 -- 172.21.15.40:0/711781374 shutdown_connections 2024-04-21T23:20:18.987 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.979+0000 7f3f261d0740 1 --2- 172.21.15.40:0/711781374 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55897f9a2620 0x55897f9a4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:18.987 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.979+0000 7f3f261d0740 1 --2- 172.21.15.40:0/711781374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55897f994e00 0x55897f9a20e0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:18.988 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.979+0000 7f3f261d0740 1 --2- 172.21.15.40:0/711781374 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55897f9944e0 0x55897f9948c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:18.988 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.979+0000 7f3f261d0740 1 -- 172.21.15.40:0/711781374 >> 172.21.15.40:0/711781374 conn(0x55897f98f440 msgr2=0x55897f98f820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:18.988 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.979+0000 7f3f261d0740 1 -- 172.21.15.40:0/711781374 shutdown_connections 2024-04-21T23:20:18.988 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.979+0000 7f3f261d0740 1 -- 172.21.15.40:0/711781374 wait complete. 2024-04-21T23:20:18.988 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.980+0000 7f3f261d0740 1 -- start start 2024-04-21T23:20:18.988 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.980+0000 7f3f261d0740 1 Processor -- start 2024-04-21T23:20:18.988 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.980+0000 7f3f261d0740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55897f9944e0 0x55897f99f640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:18.988 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.980+0000 7f3f261d0740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55897f994e00 0x55897f99fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:18.988 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.980+0000 7f3f261d0740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55897f9a2620 0x55897f9a00c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:18.988 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.980+0000 7f3f261d0740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55897f9a5250 con 0x55897f9a2620 2024-04-21T23:20:18.988 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.980+0000 7f3f261d0740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55897f8b4a10 con 0x55897f994e00 2024-04-21T23:20:18.989 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.980+0000 7f3f261d0740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55897f9a53d0 con 0x55897f9944e0 2024-04-21T23:20:18.989 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.980+0000 7f3f261d0740 10 client.0 _ll_register_callbacks cb 0x55897f8efb60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:20:18.989 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.980+0000 7f3f21b82640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55897f9a2620 0x55897f9a00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:18.989 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.980+0000 7f3f21b82640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55897f9a2620 0x55897f9a00c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:53206/0 (socket says 172.21.15.40:53206) 2024-04-21T23:20:18.989 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.980+0000 7f3f21b82640 1 -- 172.21.15.40:0/1981016735 learned_addr learned my addr 172.21.15.40:0/1981016735 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:20:18.989 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.980+0000 7f3f20b80640 1 --2- 172.21.15.40:0/1981016735 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55897f994e00 0x55897f99fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:18.989 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.981+0000 7f3f21381640 1 --2- 172.21.15.40:0/1981016735 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55897f9944e0 0x55897f99f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:18.989 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.981+0000 7f3f21b82640 1 -- 172.21.15.40:0/1981016735 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55897f9944e0 msgr2=0x55897f99f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:18.989 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.981+0000 7f3f21b82640 1 --2- 172.21.15.40:0/1981016735 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55897f9944e0 0x55897f99f640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:18.989 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.981+0000 7f3f21b82640 1 -- 172.21.15.40:0/1981016735 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55897f994e00 msgr2=0x55897f99fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:18.990 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.981+0000 7f3f21b82640 1 --2- 172.21.15.40:0/1981016735 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55897f994e00 0x55897f99fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:18.990 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.981+0000 7f3f21b82640 1 -- 172.21.15.40:0/1981016735 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55897f6d9620 con 0x55897f9a2620 2024-04-21T23:20:18.990 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.981+0000 7f3f21b82640 1 --2- 172.21.15.40:0/1981016735 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55897f9a2620 0x55897f9a00c0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f3f1400dbd0 tx=0x7f3f14004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87f6ac7987925ea1 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:18.990 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.981+0000 7f3f04ff9640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-21T23:20:18.990 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.981+0000 7f3f04ff9640 1 -- 172.21.15.40:0/1981016735 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f1401d410 con 0x55897f9a2620 2024-04-21T23:20:18.990 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.981+0000 7f3f04ff9640 1 -- 172.21.15.40:0/1981016735 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f3f1401d5b0 con 0x55897f9a2620 2024-04-21T23:20:18.990 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.981+0000 7f3f261d0740 1 -- 172.21.15.40:0/1981016735 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f18011de0 con 0x55897f9a2620 2024-04-21T23:20:18.990 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.981+0000 7f3f04ff9640 1 -- 172.21.15.40:0/1981016735 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3f1401d7b0 con 0x55897f9a2620 2024-04-21T23:20:18.990 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.981+0000 7f3f261d0740 10 client.16209 fetch_fsmap learned FSMap version 29 2024-04-21T23:20:18.990 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.981+0000 7f3f261d0740 1 -- 172.21.15.40:0/1981016735 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55897f9b14f0 con 0x55897f9a2620 2024-04-21T23:20:18.991 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.981+0000 7f3f04ff9640 1 -- 172.21.15.40:0/1981016735 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f18011de0 con 0x55897f9a2620 2024-04-21T23:20:18.991 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.981+0000 7f3f04ff9640 1 -- 172.21.15.40:0/1981016735 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f3f14019070 con 0x55897f9a2620 2024-04-21T23:20:18.991 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.981+0000 7f3f261d0740 10 client.16209 fetch_fsmap finished waiting for FSMap version 29 2024-04-21T23:20:18.991 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.981+0000 7f3f261d0740 10 client.16209 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:20:18.991 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.981+0000 7f3f261d0740 20 client.16209 populate_metadata read hostname 'smithi040' 2024-04-21T23:20:18.991 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.981+0000 7f3f261d0740 1 --2- 172.21.15.40:0/1981016735 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55897f9b29b0 0x55897f9b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:18.991 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.982+0000 7f3f261d0740 4 client.16209 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:20:18.991 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.982+0000 7f3f261d0740 1 -- 172.21.15.40:0/1981016735 --> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55897f9060b0 con 0x55897f9b29b0 2024-04-21T23:20:18.991 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.982+0000 7f3f21381640 1 --2- 172.21.15.40:0/1981016735 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55897f9b29b0 0x55897f9b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:18.991 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.982+0000 7f3f21381640 1 --2- 172.21.15.40:0/1981016735 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55897f9b29b0 0x55897f9b4e20 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7a8bda93fd48b0ba server_cookie=a13c9cf09fd8ed3 in_seq=0 out_seq=0 2024-04-21T23:20:18.992 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.982+0000 7f3f04ff9640 10 client.16209 ms_handle_connect on v2:172.21.15.111:6838/3712235548 2024-04-21T23:20:18.992 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.983+0000 7f3f04ff9640 1 -- 172.21.15.40:0/1981016735 <== mds.0 v2:172.21.15.111:6838/3712235548 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55897f9060b0 con 0x55897f9b29b0 2024-04-21T23:20:18.992 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.983+0000 7f3f04ff9640 10 client.16209 handle_command_reply: tid=0 2024-04-21T23:20:18.992 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.983+0000 7f3f261d0740 1 client.16209 shutdown 2024-04-21T23:20:18.992 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.983+0000 7f3f261d0740 1 -- 172.21.15.40:0/1981016735 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55897f9a2620 msgr2=0x55897f9a00c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:18.992 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.983+0000 7f3f261d0740 1 --2- 172.21.15.40:0/1981016735 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55897f9a2620 0x55897f9a00c0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f3f1400dbd0 tx=0x7f3f14004a40 comp rx=0 tx=0).stop 2024-04-21T23:20:18.992 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.983+0000 7f3f261d0740 1 -- 172.21.15.40:0/1981016735 shutdown_connections 2024-04-21T23:20:18.992 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.983+0000 7f3f261d0740 1 --2- 172.21.15.40:0/1981016735 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55897f9b29b0 0x55897f9b4e20 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:18.992 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.983+0000 7f3f261d0740 1 --2- 172.21.15.40:0/1981016735 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55897f994e00 0x55897f99fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:18.992 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.983+0000 7f3f261d0740 1 --2- 172.21.15.40:0/1981016735 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55897f9a2620 0x55897f9a00c0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:18.992 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.983+0000 7f3f04ff9640 0 client.16209 ms_handle_reset on v2:172.21.15.111:6838/3712235548 2024-04-21T23:20:18.993 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.983+0000 7f3f261d0740 1 --2- 172.21.15.40:0/1981016735 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55897f9944e0 0x55897f99f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:18.993 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.983+0000 7f3f261d0740 1 -- 172.21.15.40:0/1981016735 >> 172.21.15.40:0/1981016735 conn(0x55897f98f440 msgr2=0x55897f991190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:18.993 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.983+0000 7f3f261d0740 1 -- 172.21.15.40:0/1981016735 shutdown_connections 2024-04-21T23:20:18.993 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.983+0000 7f3f261d0740 1 -- 172.21.15.40:0/1981016735 wait complete. 2024-04-21T23:20:18.993 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.983+0000 7f3f261d0740 20 client.16209 trim_cache size 0 max 16384 2024-04-21T23:20:19.002 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.998+0000 7f3f261d0740 1 Processor -- start 2024-04-21T23:20:19.002 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.998+0000 7f3f261d0740 1 -- start start 2024-04-21T23:20:19.002 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.998+0000 7f3f261d0740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55897f9944e0 0x55897f9a1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.998+0000 7f3f261d0740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55897f994e00 0x55897f9ae110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.998+0000 7f3f261d0740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55897f9a2620 0x55897f98a220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.998+0000 7f3f261d0740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3f1c1af2e0 con 0x55897f9a2620 2024-04-21T23:20:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.998+0000 7f3f261d0740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3f1c1acb50 con 0x55897f9944e0 2024-04-21T23:20:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.998+0000 7f3f261d0740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3f1c1aa9a0 con 0x55897f994e00 2024-04-21T23:20:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.998+0000 7f3f21b82640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55897f9a2620 0x55897f98a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.998+0000 7f3f21b82640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55897f9a2620 0x55897f98a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:53214/0 (socket says 172.21.15.40:53214) 2024-04-21T23:20:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.998+0000 7f3f21b82640 1 -- 172.21.15.40:0/121572908 learned_addr learned my addr 172.21.15.40:0/121572908 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:20:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.998+0000 7f3f21381640 1 --2- 172.21.15.40:0/121572908 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55897f9944e0 0x55897f9a1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.999+0000 7f3f20b80640 1 --2- 172.21.15.40:0/121572908 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55897f994e00 0x55897f9ae110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.999+0000 7f3f21b82640 1 -- 172.21.15.40:0/121572908 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55897f994e00 msgr2=0x55897f9ae110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.999+0000 7f3f21b82640 1 --2- 172.21.15.40:0/121572908 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55897f994e00 0x55897f9ae110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.999+0000 7f3f21b82640 1 -- 172.21.15.40:0/121572908 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55897f9944e0 msgr2=0x55897f9a1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:19.004 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.999+0000 7f3f21b82640 1 --2- 172.21.15.40:0/121572908 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55897f9944e0 0x55897f9a1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:19.004 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.999+0000 7f3f21b82640 1 -- 172.21.15.40:0/121572908 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55897f99e650 con 0x55897f9a2620 2024-04-21T23:20:19.004 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.999+0000 7f3f20b80640 1 --2- 172.21.15.40:0/121572908 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55897f994e00 0x55897f9ae110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:20:19.004 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.999+0000 7f3f21b82640 1 --2- 172.21.15.40:0/121572908 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55897f9a2620 0x55897f98a220 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f3f14012db0 tx=0x7f3f1400d8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8ed8ed304c40cdf server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:19.004 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.999+0000 7f3f06ffd640 1 -- 172.21.15.40:0/121572908 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f14037070 con 0x55897f9a2620 2024-04-21T23:20:19.004 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.999+0000 7f3f06ffd640 1 -- 172.21.15.40:0/121572908 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f14016070 con 0x55897f9a2620 2024-04-21T23:20:19.004 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.999+0000 7f3f06ffd640 1 -- 172.21.15.40:0/121572908 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f14025660 con 0x55897f9a2620 2024-04-21T23:20:19.004 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.999+0000 7f3f261d0740 1 -- 172.21.15.40:0/121572908 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55897f9a2620 msgr2=0x55897f98a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:19.004 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:18.999+0000 7f3f261d0740 1 --2- 172.21.15.40:0/121572908 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55897f9a2620 0x55897f98a220 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f3f14012db0 tx=0x7f3f1400d8b0 comp rx=0 tx=0).stop 2024-04-21T23:20:19.004 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:19.000+0000 7f3f261d0740 1 -- 172.21.15.40:0/121572908 shutdown_connections 2024-04-21T23:20:19.004 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:19.000+0000 7f3f261d0740 1 --2- 172.21.15.40:0/121572908 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55897f9a2620 0x55897f98a220 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:19.004 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:19.000+0000 7f3f261d0740 1 --2- 172.21.15.40:0/121572908 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55897f994e00 0x55897f9ae110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:19.000+0000 7f3f261d0740 1 --2- 172.21.15.40:0/121572908 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55897f9944e0 0x55897f9a1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:19.000+0000 7f3f261d0740 1 -- 172.21.15.40:0/121572908 >> 172.21.15.40:0/121572908 conn(0x55897f98f440 msgr2=0x55897f991120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:19.000+0000 7f3f261d0740 1 -- 172.21.15.40:0/121572908 shutdown_connections 2024-04-21T23:20:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:19.000+0000 7f3f261d0740 1 -- 172.21.15.40:0/121572908 wait complete. 2024-04-21T23:20:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:19.000+0000 7f3f261d0740 1 -- start start 2024-04-21T23:20:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:19.000+0000 7f3f261d0740 1 Processor -- start 2024-04-21T23:20:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:19.000+0000 7f3f261d0740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55897f9944e0 0x55897f9aa810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:19.000+0000 7f3f261d0740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55897f994e00 0x55897f9a75f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:19.000+0000 7f3f261d0740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55897f9a2620 0x55897f9a7b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:19.000+0000 7f3f261d0740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3f1c1ab010 con 0x55897f9a2620 2024-04-21T23:20:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:19.000+0000 7f3f261d0740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3f1c1ac480 con 0x55897f9944e0 2024-04-21T23:20:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:19.000+0000 7f3f261d0740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3f1c1ab6e0 con 0x55897f994e00 2024-04-21T23:20:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:19.000+0000 7f3f261d0740 10 client.0 _ll_register_callbacks cb 0x7f3f1400aac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:20:19.006 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:19.000+0000 7f3f21b82640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55897f9a2620 0x55897f9a7b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:19.006 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:19.000+0000 7f3f21b82640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55897f9a2620 0x55897f9a7b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:53226/0 (socket says 172.21.15.40:53226) 2024-04-21T23:20:19.006 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:19.000+0000 7f3f21b82640 1 -- 172.21.15.40:0/3738882050 learned_addr learned my addr 172.21.15.40:0/3738882050 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:20:19.006 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:19.000+0000 7f3f21381640 1 --2- 172.21.15.40:0/3738882050 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55897f9944e0 0x55897f9aa810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:19.006 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:19.001+0000 7f3f20b80640 1 --2- 172.21.15.40:0/3738882050 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55897f994e00 0x55897f9a75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:19.006 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:19.001+0000 7f3f21b82640 1 -- 172.21.15.40:0/3738882050 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55897f994e00 msgr2=0x55897f9a75f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:19.006 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:19.001+0000 7f3f21b82640 1 --2- 172.21.15.40:0/3738882050 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55897f994e00 0x55897f9a75f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:19.006 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:19.001+0000 7f3f21b82640 1 -- 172.21.15.40:0/3738882050 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55897f9944e0 msgr2=0x55897f9aa810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:19.006 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:19.001+0000 7f3f21b82640 1 --2- 172.21.15.40:0/3738882050 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55897f9944e0 0x55897f9aa810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:19.006 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:19.001+0000 7f3f21b82640 1 -- 172.21.15.40:0/3738882050 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55897f9a80f0 con 0x55897f9a2620 2024-04-21T23:20:19.006 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:19.001+0000 7f3f21b82640 1 --2- 172.21.15.40:0/3738882050 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55897f9a2620 0x55897f9a7b30 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f3f14025ab0 tx=0x7f3f14004c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ac28c6d2a19b1ca server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:19.007 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:19.001+0000 7f3f04ff9640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-21T23:20:19.007 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:19.001+0000 7f3f04ff9640 1 -- 172.21.15.40:0/3738882050 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f14037040 con 0x55897f9a2620 2024-04-21T23:20:19.007 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:19.001+0000 7f3f04ff9640 1 -- 172.21.15.40:0/3738882050 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f3f14025610 con 0x55897f9a2620 2024-04-21T23:20:19.007 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:19.001+0000 7f3f261d0740 1 -- 172.21.15.40:0/3738882050 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f14016070 con 0x55897f9a2620 2024-04-21T23:20:19.007 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:19.001+0000 7f3f04ff9640 1 -- 172.21.15.40:0/3738882050 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3f1404a050 con 0x55897f9a2620 2024-04-21T23:20:19.007 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:19.001+0000 7f3f261d0740 10 client.16215 fetch_fsmap learned FSMap version 29 2024-04-21T23:20:19.007 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:19.001+0000 7f3f261d0740 1 -- 172.21.15.40:0/3738882050 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55897fa19d00 con 0x55897f9a2620 2024-04-21T23:20:19.007 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:19.001+0000 7f3f04ff9640 1 -- 172.21.15.40:0/3738882050 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f14016070 con 0x55897f9a2620 2024-04-21T23:20:19.007 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:19.002+0000 7f3f04ff9640 1 -- 172.21.15.40:0/3738882050 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f3f14019040 con 0x55897f9a2620 2024-04-21T23:20:19.007 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:19.002+0000 7f3f261d0740 10 client.16215 fetch_fsmap finished waiting for FSMap version 29 2024-04-21T23:20:19.007 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:19.002+0000 7f3f261d0740 10 client.16215 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:20:19.007 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:19.002+0000 7f3f261d0740 20 client.16215 populate_metadata read hostname 'smithi040' 2024-04-21T23:20:19.008 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:19.002+0000 7f3f261d0740 1 --2- 172.21.15.40:0/3738882050 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55897fa1a350 0x55897fa1c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:19.008 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:19.002+0000 7f3f261d0740 4 client.16215 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T23:20:19.008 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:19.002+0000 7f3f261d0740 1 -- 172.21.15.40:0/3738882050 --> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55897f98a900 con 0x55897fa1a350 2024-04-21T23:20:19.008 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:19.002+0000 7f3f21381640 1 --2- 172.21.15.40:0/3738882050 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55897fa1a350 0x55897fa1c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:19.008 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:19.002+0000 7f3f21381640 1 --2- 172.21.15.40:0/3738882050 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55897fa1a350 0x55897fa1c740 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b95b8bac6cf270af server_cookie=b14f481fd2d5fc51 in_seq=0 out_seq=0 2024-04-21T23:20:19.008 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:19.002+0000 7f3f04ff9640 10 client.16215 ms_handle_connect on v2:172.21.15.111:6838/3712235548 2024-04-21T23:20:19.018 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:19.018+0000 7f3f04ff9640 1 -- 172.21.15.40:0/3738882050 <== mds.0 v2:172.21.15.111:6838/3712235548 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55897f98a900 con 0x55897fa1a350 2024-04-21T23:20:19.018 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:19.018+0000 7f3f04ff9640 10 client.16215 handle_command_reply: tid=0 2024-04-21T23:20:19.018 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:19.018+0000 7f3f261d0740 1 client.16215 shutdown 2024-04-21T23:20:19.018 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:19.018+0000 7f3f261d0740 1 -- 172.21.15.40:0/3738882050 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55897f9a2620 msgr2=0x55897f9a7b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:19.018 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:19.018+0000 7f3f261d0740 1 --2- 172.21.15.40:0/3738882050 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55897f9a2620 0x55897f9a7b30 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f3f14025ab0 tx=0x7f3f14004c50 comp rx=0 tx=0).stop 2024-04-21T23:20:19.018 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:19.018+0000 7f3f261d0740 1 -- 172.21.15.40:0/3738882050 shutdown_connections 2024-04-21T23:20:19.018 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:19.018+0000 7f3f261d0740 1 --2- 172.21.15.40:0/3738882050 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55897fa1a350 0x55897fa1c740 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:19.018 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:19.018+0000 7f3f261d0740 1 --2- 172.21.15.40:0/3738882050 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55897f9a2620 0x55897f9a7b30 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:19.018 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:19.018+0000 7f3f261d0740 1 --2- 172.21.15.40:0/3738882050 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55897f994e00 0x55897f9a75f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:19.018 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:19.018+0000 7f3f04ff9640 0 client.16215 ms_handle_reset on v2:172.21.15.111:6838/3712235548 2024-04-21T23:20:19.019 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:19.018+0000 7f3f261d0740 1 --2- 172.21.15.40:0/3738882050 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55897f9944e0 0x55897f9aa810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:19.019 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:19.018+0000 7f3f261d0740 1 -- 172.21.15.40:0/3738882050 >> 172.21.15.40:0/3738882050 conn(0x55897f98f440 msgr2=0x55897f99f600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:19.019 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:19.018+0000 7f3f261d0740 1 -- 172.21.15.40:0/3738882050 shutdown_connections 2024-04-21T23:20:19.019 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:19.018+0000 7f3f261d0740 1 -- 172.21.15.40:0/3738882050 wait complete. 2024-04-21T23:20:19.019 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:19.018+0000 7f3f261d0740 20 client.16215 trim_cache size 0 max 16384 2024-04-21T23:20:19.019 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:{ 2024-04-21T23:20:19.019 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "message": "", 2024-04-21T23:20:19.019 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "return_code": 0 2024-04-21T23:20:19.019 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:} 2024-04-21T23:20:19.020 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:19.019+0000 7f3f235e5640 1 -- 172.21.15.40:0/1862785130 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f3f080a2e20 msgr2=0x7f3f080a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:19.020 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:19.019+0000 7f3f235e5640 1 --2- 172.21.15.40:0/1862785130 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f3f080a2e20 0x7f3f080a52e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f3f1c06b050 tx=0x7f3f14005c60 comp rx=0 tx=0).stop 2024-04-21T23:20:19.021 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:19.019+0000 7f3f235e5640 1 -- 172.21.15.40:0/1862785130 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f1c103310 msgr2=0x7f3f1c198510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:19.021 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:19.019+0000 7f3f235e5640 1 --2- 172.21.15.40:0/1862785130 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f1c103310 0x7f3f1c198510 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f3f0c005940 tx=0x7f3f0c02cd10 comp rx=0 tx=0).stop 2024-04-21T23:20:19.021 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:19.019+0000 7f3f235e5640 1 -- 172.21.15.40:0/1862785130 shutdown_connections 2024-04-21T23:20:19.021 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:19.019+0000 7f3f235e5640 1 --2- 172.21.15.40:0/1862785130 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f3f080a2e20 0x7f3f080a52e0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:19.021 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:19.019+0000 7f3f235e5640 1 --2- 172.21.15.40:0/1862785130 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f1c10bdc0 0x7f3f1c19c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:19.021 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:19.019+0000 7f3f235e5640 1 --2- 172.21.15.40:0/1862785130 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f1c103310 0x7f3f1c198510 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:19.021 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:19.019+0000 7f3f235e5640 1 --2- 172.21.15.40:0/1862785130 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f1c1029f0 0x7f3f1c197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:19.021 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:19.019+0000 7f3f235e5640 1 -- 172.21.15.40:0/1862785130 >> 172.21.15.40:0/1862785130 conn(0x7f3f1c0769c0 msgr2=0x7f3f1c0fd4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:19.021 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:19.019+0000 7f3f235e5640 1 -- 172.21.15.40:0/1862785130 shutdown_connections 2024-04-21T23:20:19.021 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:19.019+0000 7f3f235e5640 1 -- 172.21.15.40:0/1862785130 wait complete. 2024-04-21T23:20:19.022 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:18 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:18.733+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000468 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/098.out auth v316 pv880 ap=3 s=0 n(v0 1=1+0)->n(v0 rc2024-04-21T23:19:50.701916+0000 b558 1=1+0) (ifile excl w=2) cr={25132=0-4194304@2} caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=2 lock=1 importingcaps=0 caps=1 needsnapflush=0 replicated=0 authpin=1 scrubqueue=0 0x5572f3e21600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000468:[<0x100000003a4/098.out v316>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000468:[<0x100000003a4/098.out v316>,<0x100000003a2/btrfs v29072>,<0x10000000007/tests v2937>,<0x10000000006/xfstests-dev v260>,<0x10000000005/tmp v53>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:19.022 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:18 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:18.737+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004c8 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/146.out auth v352 pv888 ap=3 s=0 n(v0 1=1+0)->n(v0 rc2024-04-21T23:19:50.841914+0000 b55 1=1+0) (ifile excl w=2) cr={25132=0-4194304@2} caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=2 lock=1 importingcaps=0 caps=1 needsnapflush=0 replicated=0 authpin=1 scrubqueue=0 0x5572f3e22c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004c8:[<0x100000003a4/146.out v352>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000004c8:[<0x100000003a4/146.out v352>,<0x100000003a2/btrfs v29072>,<0x10000000007/tests v2937>,<0x10000000006/xfstests-dev v260>,<0x10000000005/tmp v53>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:19.022 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:18 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:18.737+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000634 [...4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/ auth{0=1} fragtree_t(*^3) v29074 pv29884 ap=160 f(v0 m2024-04-21T23:20:05.690705+0000 107=107+0)->f(v0 m2024-04-21T23:20:05.690705+0000 107=107+0) n(v0 rc2024-04-21T23:20:05.690705+0000 108=107+1)->n(v0 rc2024-04-21T23:20:05.690705+0000 b75765 108=107+1) old_inodes=1 (inest mix w=159) (ifile excl) (iversion lock w=159) caps={25132=pAsLsXsFsx/pAsLsXsFsxcral@145},l=25132 | request=0 lock=2 importingcaps=0 dirfrag=8 caps=1 dirtyrstat=0 replicated=1 dirty=1 waiter=0 authpin=1 scrubqueue=0 0x5572f3139600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(2)0x10000000634:[<0x100000003a2/ext4 v1598>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[]","memoryvalue":"(2)0x10000000634:[<0x100000003a2/ext4 v29074>,<0x10000000007/tests v2937>,<0x10000000006/xfstests-dev v260>,<0x10000000005/tmp v53>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-04-21T23:20:05.690705+0000 107=107+0)","ondisk_value.rstat":"n(v0 rc2024-04-21T23:20:05.690705+0000 108=107+1)","memory_value.dirstat":"f(v0 m2024-04-21T23:20:05.690705+0000 107=107+0)","memory_value.rstat":"n(v0 rc2024-04-21T23:20:05.690705+0000 108=107+1)","error_str":""},"return_code":0} 2024-04-21T23:20:19.022 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:18 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:18.738+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000534 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/200.out auth v374 pv884 ap=3 s=0 n(v0 1=1+0)->n(v0 rc2024-04-21T23:19:50.956912+0000 b639 1=1+0) (ifile excl w=2) cr={25132=0-4194304@2} caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=2 lock=1 importingcaps=0 caps=1 needsnapflush=0 replicated=0 authpin=1 scrubqueue=0 0x5572f3e2c000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000534:[<0x100000003a4/200.out v374>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000534:[<0x100000003a4/200.out v374>,<0x100000003a2/btrfs v29072>,<0x10000000007/tests v2937>,<0x10000000006/xfstests-dev v260>,<0x10000000005/tmp v53>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:19.022 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:18 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:18.738+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003fe [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/045.out auth v180 pv890 ap=3 s=0 n(v0 1=1+0)->n(v0 rc2024-04-21T23:19:50.269922+0000 b25 1=1+0) (ifile excl w=2) cr={25132=0-4194304@2} caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=2 lock=1 importingcaps=0 caps=1 needsnapflush=0 replicated=0 authpin=1 scrubqueue=0 0x5572f3e2e100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003fe:[<0x100000003a4/045.out v180>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000003fe:[<0x100000003a4/045.out v180>,<0x100000003a2/btrfs v29072>,<0x10000000007/tests v2937>,<0x10000000006/xfstests-dev v260>,<0x10000000005/tmp v53>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:19.279 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:19 smithi111 ceph-mon[31085]: pgmap v60: 129 pgs: 129 active+clean; 63 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.0 MiB/s wr, 406 op/s 2024-04-21T23:20:19.279 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:19 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000468 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/098.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:19.280 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:19 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000004c8 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/146.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:19.280 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:19 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000634 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4) see mds.h log and `damage ls` output for details 2024-04-21T23:20:19.280 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:19 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000534 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/200.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:19.280 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:19 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000003fe (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/045.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:19.280 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:19 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:19.031+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000432 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/071.out auth v894 ap=1 s=43 n(v0 b43 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3e2d600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000432:[<0x100000003a4/071.out v284>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000432:[<0x100000003a4/071.out v894>,<0x100000003a2/btrfs v29566>,<0x10000000007/tests v2937>,<0x10000000006/xfstests-dev v260>,<0x10000000005/tmp v53>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:19.281 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:19 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:19.087+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000404 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/048.out auth v886 ap=1 s=1835 n(v0 rc2024-04-21T23:19:50.275922+0000 b1835 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3e2e680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000404:[<0x100000003a4/048.out v192>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000404:[<0x100000003a4/048.out v886>,<0x100000003a2/btrfs v29566>,<0x10000000007/tests v2937>,<0x10000000006/xfstests-dev v260>,<0x10000000005/tmp v53>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:19.281 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:19 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:19.088+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004c0 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/142.out auth v898 ap=1 s=2652 n(v0 rc2024-04-21T23:19:50.832914+0000 b2652 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3e2ec00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004c0:[<0x100000003a4/142.out v348>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000004c0:[<0x100000003a4/142.out v898>,<0x100000003a2/btrfs v29566>,<0x10000000007/tests v2937>,<0x10000000006/xfstests-dev v260>,<0x10000000005/tmp v53>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:19.281 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:19 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:19.088+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003b8 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/010.out auth v902 ap=1 s=43 n(v0 rc2024-04-21T23:19:50.199923+0000 b43 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3e3c000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003b8:[<0x100000003a4/010.out v40>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000003b8:[<0x100000003a4/010.out v902>,<0x100000003a2/btrfs v29566>,<0x10000000007/tests v2937>,<0x10000000006/xfstests-dev v260>,<0x10000000005/tmp v53>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:19.281 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:19 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:19.089+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000044a [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/083.out auth v906 ap=1 s=43 n(v0 rc2024-04-21T23:19:50.617917+0000 b43 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3e3c580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000044a:[<0x100000003a4/083.out v306>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x1000000044a:[<0x100000003a4/083.out v906>,<0x100000003a2/btrfs v29566>,<0x10000000007/tests v2937>,<0x10000000006/xfstests-dev v260>,<0x10000000005/tmp v53>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:19.281 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:19 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:19.097+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003d8 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/026.out auth v882 ap=1 s=742 n(v0 rc2024-04-21T23:19:50.230923+0000 b742 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3e3e680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003d8:[<0x100000003a4/026.out v104>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000003d8:[<0x100000003a4/026.out v882>,<0x100000003a2/btrfs v29566>,<0x10000000007/tests v2937>,<0x10000000006/xfstests-dev v260>,<0x10000000005/tmp v53>,<0x100000001fb/client.0 v18>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:19.281 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:19 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:19.280+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003d6 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/025.out auth v878 ap=1 s=307 n(v0 rc2024-04-21T23:19:50.228923+0000 b307 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3e48580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003d6:[<0x100000003a4/025.out v100>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000003d6:[<0x100000003a4/025.out v878>,<0x100000003a2/btrfs v29902>,<0x10000000007/tests v2937>,<0x10000000006/xfstests-dev v265>,<0x10000000005/tmp v57>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:19.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:19 smithi040 ceph-mon[27984]: pgmap v60: 129 pgs: 129 active+clean; 63 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.0 MiB/s wr, 406 op/s 2024-04-21T23:20:19.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:19 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000468 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/098.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:19.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:19 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000004c8 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/146.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:19.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:19 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000634 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4) see mds.h log and `damage ls` output for details 2024-04-21T23:20:19.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:19 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000534 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/200.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:19.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:19 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000003fe (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/045.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:19.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:19 smithi174 ceph-mon[31409]: pgmap v60: 129 pgs: 129 active+clean; 63 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.0 MiB/s wr, 406 op/s 2024-04-21T23:20:19.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:19 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000468 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/098.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:19.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:19 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000004c8 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/146.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:19.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:19 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000634 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4) see mds.h log and `damage ls` output for details 2024-04-21T23:20:19.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:19 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000534 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/200.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:19.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:19 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000003fe (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/045.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:19.694 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:19 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:19.281+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004b6 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/137.out auth v894 ap=1 s=1934 n(v0 rc2024-04-21T23:19:50.819914+0000 b1934 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3e49080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004b6:[<0x100000003a4/137.out v344>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000004b6:[<0x100000003a4/137.out v894>,<0x100000003a2/btrfs v29902>,<0x10000000007/tests v2937>,<0x10000000006/xfstests-dev v265>,<0x10000000005/tmp v57>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:19.694 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:19 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:19.282+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004a6 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/129.out auth v886 ap=1 s=43 n(v0 rc2024-04-21T23:19:50.800915+0000 b43 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3e49b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004a6:[<0x100000003a4/129.out v334>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000004a6:[<0x100000003a4/129.out v886>,<0x100000003a2/btrfs v29902>,<0x10000000007/tests v2937>,<0x10000000006/xfstests-dev v265>,<0x10000000005/tmp v57>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:19.695 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:19 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:19.283+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004b2 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/135.out auth v890 ap=1 s=157 n(v0 rc2024-04-21T23:19:50.814914+0000 b157 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3e4b180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004b2:[<0x100000003a4/135.out v338>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000004b2:[<0x100000003a4/135.out v890>,<0x100000003a2/btrfs v29902>,<0x10000000007/tests v2937>,<0x10000000006/xfstests-dev v265>,<0x10000000005/tmp v57>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:19.695 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:19 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:19.283+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000041c [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/060.out auth v882 ap=1 s=43 n(v0 rc2024-04-21T23:19:50.300922+0000 b43 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3e4b700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000041c:[<0x100000003a4/060.out v240>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x1000000041c:[<0x100000003a4/060.out v882>,<0x100000003a2/btrfs v29902>,<0x10000000007/tests v2937>,<0x10000000006/xfstests-dev v265>,<0x10000000005/tmp v57>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:20.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:20 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000432 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/071.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:20.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:20 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000404 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/048.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:20.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:20 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000004c0 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/142.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:20.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:20 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000003b8 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/010.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:20.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:20 smithi040 ceph-mon[27984]: Scrub error on inode 0x1000000044a (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/083.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:20.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:20 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000003d8 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/026.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:20.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:20 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000003d6 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/025.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:20.433 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:20 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000004b6 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/137.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:20.433 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:20 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000004a6 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/129.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:20.433 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:20 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000004b2 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/135.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:20.433 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:20 smithi040 ceph-mon[27984]: Scrub error on inode 0x1000000041c (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/060.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:20.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:20 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000432 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/071.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:20.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:20 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000404 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/048.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:20.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:20 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000004c0 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/142.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:20.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:20 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000003b8 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/010.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:20.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:20 smithi174 ceph-mon[31409]: Scrub error on inode 0x1000000044a (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/083.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:20.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:20 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000003d8 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/026.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:20.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:20 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000003d6 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/025.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:20.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:20 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000004b6 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/137.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:20.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:20 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000004a6 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/129.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:20.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:20 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000004b2 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/135.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:20.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:20 smithi174 ceph-mon[31409]: Scrub error on inode 0x1000000041c (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/060.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:20.445 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:20 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:20.181+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004ba [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/139.out auth v888 ap=1 s=774 n(v0 rc2024-04-21T23:19:50.825914+0000 b774 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3e59600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004ba:[<0x100000003a4/139.out v346>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000004ba:[<0x100000003a4/139.out v888>,<0x100000003a2/btrfs v30962>,<0x10000000007/tests v2985>,<0x10000000006/xfstests-dev v265>,<0x10000000005/tmp v57>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:20.445 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:20 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:20.184+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000046a [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/099.out auth v884 ap=1 s=633 n(v0 rc2024-04-21T23:19:50.704916+0000 b633 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3e5b180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000046a:[<0x100000003a4/099.out v318>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x1000000046a:[<0x100000003a4/099.out v884>,<0x100000003a2/btrfs v30962>,<0x10000000007/tests v2985>,<0x10000000006/xfstests-dev v265>,<0x10000000005/tmp v57>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:20.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:20 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000432 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/071.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:20.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:20 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000404 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/048.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:20.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:20 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000004c0 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/142.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:20.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:20 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000003b8 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/010.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:20.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:20 smithi111 ceph-mon[31085]: Scrub error on inode 0x1000000044a (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/083.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:20.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:20 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000003d8 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/026.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:20.446 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:20 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000003d6 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/025.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:20.446 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:20 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000004b6 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/137.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:20.446 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:20 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000004a6 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/129.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:20.446 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:20 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000004b2 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/135.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:20.446 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:20 smithi111 ceph-mon[31085]: Scrub error on inode 0x1000000041c (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/060.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:21.423 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-mon[31085]: pgmap v61: 129 pgs: 129 active+clean; 86 MiB data, 711 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 5.1 MiB/s wr, 574 op/s 2024-04-21T23:20:21.423 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000004ba (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/139.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:21.423 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-mon[31085]: Scrub error on inode 0x1000000046a (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/099.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:21.423 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:20 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:20.973+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004c6 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/145.out auth v880 ap=1 s=109 n(v0 rc2024-04-21T23:19:50.839914+0000 b109 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3e61b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004c6:[<0x100000003a4/145.out v350>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000004c6:[<0x100000003a4/145.out v880>,<0x100000003a2/btrfs v32130>,<0x10000000007/tests v2985>,<0x10000000006/xfstests-dev v265>,<0x10000000005/tmp v57>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.424 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:20 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:20.974+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000050c [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/180.out auth v878 ap=1 s=128 n(v0 rc2024-04-21T23:19:50.914913+0000 b128 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3e63180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000050c:[<0x100000003a4/180.out v364>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x1000000050c:[<0x100000003a4/180.out v878>,<0x100000003a2/btrfs v32130>,<0x10000000007/tests v2985>,<0x10000000006/xfstests-dev v265>,<0x10000000005/tmp v57>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.430 INFO:tasks.workunit.client.0.smithi040.stderr:Note: switching to '12973fc04fd10d4af086901e10ffa8e48866b735'. 2024-04-21T23:20:21.430 INFO:tasks.workunit.client.0.smithi040.stderr: 2024-04-21T23:20:21.431 INFO:tasks.workunit.client.0.smithi040.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-21T23:20:21.431 INFO:tasks.workunit.client.0.smithi040.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-21T23:20:21.431 INFO:tasks.workunit.client.0.smithi040.stderr:state without impacting any branches by switching back to a branch. 2024-04-21T23:20:21.431 INFO:tasks.workunit.client.0.smithi040.stderr: 2024-04-21T23:20:21.431 INFO:tasks.workunit.client.0.smithi040.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-21T23:20:21.431 INFO:tasks.workunit.client.0.smithi040.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-21T23:20:21.431 INFO:tasks.workunit.client.0.smithi040.stderr: 2024-04-21T23:20:21.431 INFO:tasks.workunit.client.0.smithi040.stderr: git switch -c 2024-04-21T23:20:21.431 INFO:tasks.workunit.client.0.smithi040.stderr: 2024-04-21T23:20:21.431 INFO:tasks.workunit.client.0.smithi040.stderr:Or undo this operation with: 2024-04-21T23:20:21.431 INFO:tasks.workunit.client.0.smithi040.stderr: 2024-04-21T23:20:21.431 INFO:tasks.workunit.client.0.smithi040.stderr: git switch - 2024-04-21T23:20:21.431 INFO:tasks.workunit.client.0.smithi040.stderr: 2024-04-21T23:20:21.431 INFO:tasks.workunit.client.0.smithi040.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-21T23:20:21.432 INFO:tasks.workunit.client.0.smithi040.stderr: 2024-04-21T23:20:21.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:21 smithi040 ceph-mon[27984]: pgmap v61: 129 pgs: 129 active+clean; 86 MiB data, 711 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 5.1 MiB/s wr, 574 op/s 2024-04-21T23:20:21.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:21 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000004ba (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/139.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:21.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:21 smithi040 ceph-mon[27984]: Scrub error on inode 0x1000000046a (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/099.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:21.432 INFO:tasks.workunit.client.0.smithi040.stderr:HEAD is now at 12973fc0 btrfs/022: Correct seq number in golden output 2024-04-21T23:20:21.436 INFO:tasks.workunit.client.0.smithi040.stderr:+ make -j4 2024-04-21T23:20:21.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:21 smithi174 ceph-mon[31409]: pgmap v61: 129 pgs: 129 active+clean; 86 MiB data, 711 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 5.1 MiB/s wr, 574 op/s 2024-04-21T23:20:21.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:21 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000004ba (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/139.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:21.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:21 smithi174 ceph-mon[31409]: Scrub error on inode 0x1000000046a (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/099.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:21.451 INFO:tasks.workunit.client.0.smithi040.stdout:libtoolize -cfi 2024-04-21T23:20:21.695 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.423+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000040c [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/052.out auth v882 ap=1 s=15846 n(v0 rc2024-04-21T23:19:50.283922+0000 b15846 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3e70580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000040c:[<0x100000003a4/052.out v208>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x1000000040c:[<0x100000003a4/052.out v882>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.695 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.424+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003f8 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/042.out auth v880 ap=1 s=43 n(v0 rc2024-04-21T23:19:50.262922+0000 b43 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3e71b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003f8:[<0x100000003a4/042.out v168>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000003f8:[<0x100000003a4/042.out v880>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.695 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.424+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000472 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/103.out auth v888 ap=1 s=2294 n(v0 rc2024-04-21T23:19:50.716916+0000 b2294 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3e73180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000472:[<0x100000003a4/103.out v320>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000472:[<0x100000003a4/103.out v888>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.695 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.425+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004d2 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/151.out auth v884 ap=1 s=37 n(v0 rc2024-04-21T23:19:50.852914+0000 b37 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3e73700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004d2:[<0x100000003a4/151.out v356>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000004d2:[<0x100000003a4/151.out v884>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.695 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.426+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000454 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/088.out auth v912 ap=1 s=346 n(v0 rc2024-04-21T23:19:50.629917+0000 b346 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3e7bb80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000454:[<0x100000003a4/088.out v310>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000454:[<0x100000003a4/088.out v912>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.695 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.426+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004ac [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/132.out auth v908 ap=1 s=43 n(v0 rc2024-04-21T23:19:50.807915+0000 b43 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3e7c100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004ac:[<0x100000003a4/132.out v336>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000004ac:[<0x100000003a4/132.out v908>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.696 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.426+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003c0 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/014.out auth v892 ap=1 s=68 n(v0 rc2024-04-21T23:19:50.206923+0000 b68 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3e7cc00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003c0:[<0x100000003a4/014.out v56>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000003c0:[<0x100000003a4/014.out v892>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.696 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.426+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004cc [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/148.out auth v904 ap=1 s=709 n(v0 rc2024-04-21T23:19:50.846914+0000 b709 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3e88000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004cc:[<0x100000003a4/148.out v354>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000004cc:[<0x100000003a4/148.out v904>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.696 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.427+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003d2 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/023.out auth v896 ap=1 s=43 n(v0 rc2024-04-21T23:19:50.224923+0000 b43 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3e88580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003d2:[<0x100000003a4/023.out v92>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000003d2:[<0x100000003a4/023.out v896>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.696 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.432+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000047c [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/108.out auth v900 ap=1 s=617 n(v0 rc2024-04-21T23:19:50.732916+0000 b617 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3e88b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000047c:[<0x100000003a4/108.out v322>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x1000000047c:[<0x100000003a4/108.out v900>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.696 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.432+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000430 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/070.out auth v888 ap=1 s=43 n(v0 rc2024-04-21T23:19:50.320921+0000 b43 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3e8ac00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000430:[<0x100000003a4/070.out v280>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000430:[<0x100000003a4/070.out v888>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.696 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.432+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004ee [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/165.out auth v854 ap=1 s=43 n(v0 rc2024-04-21T23:19:50.882913+0000 b43 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3e96580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004ee:[<0x100000003a4/165.out v368>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000004ee:[<0x100000003a4/165.out v854>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.698 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.435+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000448 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/082.out auth v862 ap=1 s=43 n(v0 rc2024-04-21T23:19:50.614917+0000 b43 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3e98100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000448:[<0x100000003a4/082.out v310>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000448:[<0x100000003a4/082.out v862>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.698 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.435+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000496 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/121.out auth v874 ap=1 s=43 n(v0 rc2024-04-21T23:19:50.774915+0000 b43 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3e99180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000496:[<0x100000003a4/121.out v342>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000496:[<0x100000003a4/121.out v874>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.698 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.435+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000042c [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/068.out auth v858 ap=1 s=43 n(v0 rc2024-04-21T23:19:50.316921+0000 b43 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3e99700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000042c:[<0x100000003a4/068.out v272>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x1000000042c:[<0x100000003a4/068.out v858>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.699 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.436+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003b2 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/007.out auth v866 ap=1 s=58 n(v0 rc2024-04-21T23:19:50.193923+0000 b58 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3e97b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003b2:[<0x100000003a4/007.out v28>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000003b2:[<0x100000003a4/007.out v866>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.699 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.437+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000420 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/062.out auth v870 ap=1 s=43 n(v0 rc2024-04-21T23:19:50.304922+0000 b43 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3ea4b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000420:[<0x100000003a4/062.out v248>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000420:[<0x100000003a4/062.out v870>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.699 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.437+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004ec [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/164.out auth v850 ap=1 s=167 n(v0 rc2024-04-21T23:19:50.880913+0000 b167 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3ea6680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004ec:[<0x100000003a4/164.out v366>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000004ec:[<0x100000003a4/164.out v850>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.699 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.438+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000494 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/120.out auth v858 ap=1 s=215 n(v0 rc2024-04-21T23:19:50.771915+0000 b215 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3eb0000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000494:[<0x100000003a4/120.out v340>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000494:[<0x100000003a4/120.out v858>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.699 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.439+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004a8 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/130.out auth v862 ap=1 s=122 n(v0 rc2024-04-21T23:19:50.803915+0000 b122 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3ea6c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004a8:[<0x100000003a4/130.out v344>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000004a8:[<0x100000003a4/130.out v862>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.699 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.439+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000428 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/066.out auth v854 ap=1 s=43 n(v0 rc2024-04-21T23:19:50.312921+0000 b43 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3eb0b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000428:[<0x100000003a4/066.out v264>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000428:[<0x100000003a4/066.out v854>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.699 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.443+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000044e [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/085.out auth v862 ap=1 s=152 n(v0 rc2024-04-21T23:19:50.622917+0000 b152 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3eb3700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000044e:[<0x100000003a4/085.out v314>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x1000000044e:[<0x100000003a4/085.out v862>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.699 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.444+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000410 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/054.out auth v850 ap=1 s=25 n(v0 rc2024-04-21T23:19:50.287922+0000 b25 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3ebf080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000410:[<0x100000003a4/054.out v216>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000410:[<0x100000003a4/054.out v850>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.699 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.444+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004ae [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/133.out auth v854 ap=1 s=157 n(v0 rc2024-04-21T23:19:50.810914+0000 b157 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3ebf600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004ae:[<0x100000003a4/133.out v348>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000004ae:[<0x100000003a4/133.out v854>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.699 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.445+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000040e [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/053.out auth v844 ap=1 s=25 n(v0 rc2024-04-21T23:19:50.285922+0000 b25 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3ec0c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000040e:[<0x100000003a4/053.out v212>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x1000000040e:[<0x100000003a4/053.out v844>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.700 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.446+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004d6 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/153.out auth v858 ap=1 s=43 n(v0 b43 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3ec0100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004d6:[<0x100000003a4/153.out v358>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000004d6:[<0x100000003a4/153.out v858>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.700 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.446+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000041e [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/061.out auth v852 ap=1 s=43 n(v0 rc2024-04-21T23:19:50.302922+0000 b43 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3ed0c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000041e:[<0x100000003a4/061.out v244>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x1000000041e:[<0x100000003a4/061.out v852>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.700 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.446+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000442 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/079.out auth v848 ap=1 s=43 n(v0 rc2024-04-21T23:19:50.608917+0000 b43 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3ed0100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000442:[<0x100000003a4/079.out v304>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000442:[<0x100000003a4/079.out v848>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.700 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.447+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000044c [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/084.out auth v856 ap=1 s=43 n(v0 rc2024-04-21T23:19:50.619917+0000 b43 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3eda000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000044c:[<0x100000003a4/084.out v312>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x1000000044c:[<0x100000003a4/084.out v856>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.700 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.448+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000444 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/080.out auth v846 ap=1 s=43 n(v0 rc2024-04-21T23:19:50.610917+0000 b43 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3edcc00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000444:[<0x100000003a4/080.out v308>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000444:[<0x100000003a4/080.out v846>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.700 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.449+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004aa [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/131.out auth v850 ap=1 s=558 n(v0 rc2024-04-21T23:19:50.805915+0000 b558 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3ee8b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004aa:[<0x100000003a4/131.out v346>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000004aa:[<0x100000003a4/131.out v850>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.700 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.451+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004b0 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/134.out auth v852 ap=1 s=109 n(v0 b109 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3eea100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004b0:[<0x100000003a4/134.out v352>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000004b0:[<0x100000003a4/134.out v852>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.700 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.451+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003a8 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/002.out auth v848 ap=1 s=43 n(v0 rc2024-04-21T23:19:50.182923+0000 b43 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3eeb180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003a8:[<0x100000003a4/002.out v8>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000003a8:[<0x100000003a4/002.out v848>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.700 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.451+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000452 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/087.out auth v856 ap=1 s=43 n(v0 rc2024-04-21T23:19:50.627917+0000 b43 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3eea680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000452:[<0x100000003a4/087.out v316>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000452:[<0x100000003a4/087.out v856>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.700 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.452+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000046e [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/101.out auth v854 ap=1 s=370 n(v0 rc2024-04-21T23:19:50.710916+0000 b370 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3ef7600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000046e:[<0x100000003a4/101.out v330>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x1000000046e:[<0x100000003a4/101.out v854>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.700 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.452+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004c2 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/143.out auth v870 ap=1 s=2652 n(v0 rc2024-04-21T23:19:50.835914+0000 b2652 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3ef8c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004c2:[<0x100000003a4/143.out v356>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000004c2:[<0x100000003a4/143.out v870>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.700 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.453+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000520 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/190.out auth v862 ap=1 s=43 n(v0 rc2024-04-21T23:19:50.935913+0000 b43 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3f00000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000520:[<0x100000003a4/190.out v382>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000520:[<0x100000003a4/190.out v862>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.701 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.453+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003ec [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/036.out auth v858 ap=1 s=54 n(v0 rc2024-04-21T23:19:50.251922+0000 b54 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3ef9700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003ec:[<0x100000003a4/036.out v144>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000003ec:[<0x100000003a4/036.out v858>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.701 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.453+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004dc [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/156.out auth v866 ap=1 s=43 n(v0 rc2024-04-21T23:19:50.863914+0000 b43 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3ef6000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004dc:[<0x100000003a4/156.out v360>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000004dc:[<0x100000003a4/156.out v866>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.701 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.453+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000504 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/176.out auth v874 ap=1 s=84 n(v0 rc2024-04-21T23:19:50.906913+0000 b84 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3f00580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000504:[<0x100000003a4/176.out v376>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000504:[<0x100000003a4/176.out v874>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.701 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.454+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003d0 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/022.out auth v866 ap=1 s=43 n(v0 rc2024-04-21T23:19:50.222923+0000 b43 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3f02680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003d0:[<0x100000003a4/022.out v88>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000003d0:[<0x100000003a4/022.out v866>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.701 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.454+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004da [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/155.out auth v874 ap=1 s=109 n(v0 rc2024-04-21T23:19:50.861914+0000 b109 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3f03180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004da:[<0x100000003a4/155.out v348>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000004da:[<0x100000003a4/155.out v874>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.701 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.456+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000476 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/105.out auth v864 ap=1 s=538 n(v0 rc2024-04-21T23:19:50.723916+0000 b538 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3f11600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000476:[<0x100000003a4/105.out v318>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000476:[<0x100000003a4/105.out v864>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.701 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.456+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004e2 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/159.out auth v870 ap=1 s=401 n(v0 rc2024-04-21T23:19:50.870914+0000 b401 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3f03700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004e2:[<0x100000003a4/159.out v350>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000004e2:[<0x100000003a4/159.out v870>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.701 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.457+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003b6 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/009.out auth v868 ap=1 s=43 n(v0 rc2024-04-21T23:19:50.197923+0000 b43 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3f11b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003b6:[<0x100000003a4/009.out v36>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000003b6:[<0x100000003a4/009.out v868>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.701 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.458+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000048c [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/116.out auth v874 ap=1 s=415 n(v0 rc2024-04-21T23:19:50.758915+0000 b415 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3f1f700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000048c:[<0x100000003a4/116.out v326>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x1000000048c:[<0x100000003a4/116.out v874>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.702 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.459+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000522 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/191.out auth v870 ap=1 s=768 n(v0 rc2024-04-21T23:19:50.937913+0000 b768 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3f1f180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000522:[<0x100000003a4/191.out v370>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000522:[<0x100000003a4/191.out v870>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.702 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.459+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004e6 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/161.out auth v864 ap=1 s=167 n(v0 rc2024-04-21T23:19:50.874914+0000 b167 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3f24000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004e6:[<0x100000003a4/161.out v354>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000004e6:[<0x100000003a4/161.out v864>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.707 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.459+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000458 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/090.out auth v868 ap=1 s=43 n(v0 rc2024-04-21T23:19:50.675916+0000 b43 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3f24580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000458:[<0x100000003a4/090.out v312>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000458:[<0x100000003a4/090.out v868>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.707 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.462+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004f0 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/166.out auth v870 ap=1 s=43 n(v0 rc2024-04-21T23:19:50.884913+0000 b43 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3f38c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004f0:[<0x100000003a4/166.out v356>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000004f0:[<0x100000003a4/166.out v870>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.707 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.463+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000482 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/111.out auth v866 ap=1 s=667 n(v0 rc2024-04-21T23:19:50.742915+0000 b667 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3f1e680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000482:[<0x100000003a4/111.out v324>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000482:[<0x100000003a4/111.out v866>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.707 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.463+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003c6 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/017.out auth v874 ap=1 s=74 n(v0 rc2024-04-21T23:19:50.213923+0000 b74 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3f46b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003c6:[<0x100000003a4/017.out v68>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000003c6:[<0x100000003a4/017.out v874>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.707 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.464+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000440 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/078.out auth v866 ap=1 s=43 n(v0 rc2024-04-21T23:19:50.606917+0000 b43 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3f46580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000440:[<0x100000003a4/078.out v306>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000440:[<0x100000003a4/078.out v866>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.707 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.464+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000418 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/058.out auth v868 ap=1 s=25 n(v0 rc2024-04-21T23:19:50.295922+0000 b25 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3f52580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000418:[<0x100000003a4/058.out v232>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000418:[<0x100000003a4/058.out v868>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.707 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.465+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000402 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/047.out auth v864 ap=1 s=87 n(v0 rc2024-04-21T23:19:50.273922+0000 b87 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3f52000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000402:[<0x100000003a4/047.out v188>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000402:[<0x100000003a4/047.out v864>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.707 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.467+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000460 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/094.out auth v866 ap=1 s=1329 n(v0 rc2024-04-21T23:19:50.688916+0000 b1329 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3f62000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000460:[<0x100000003a4/094.out v314>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000460:[<0x100000003a4/094.out v866>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.708 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.467+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003e2 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/031.out auth v874 ap=1 s=1304 n(v0 rc2024-04-21T23:19:50.240922+0000 b1304 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3f55700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003e2:[<0x100000003a4/031.out v124>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000003e2:[<0x100000003a4/031.out v874>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.708 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.468+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004c4 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/144.out auth v870 ap=1 s=109 n(v0 rc2024-04-21T23:19:50.837914+0000 b109 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3f62580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004c4:[<0x100000003a4/144.out v342>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000004c4:[<0x100000003a4/144.out v870>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.708 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.470+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000528 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/194.out auth v882 ap=1 s=43 n(v0 rc2024-04-21T23:19:50.943913+0000 b43 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3f6e580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000528:[<0x100000003a4/194.out v374>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000528:[<0x100000003a4/194.out v882>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.708 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.470+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004f4 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/168.out auth v886 ap=1 s=341 n(v0 rc2024-04-21T23:19:50.888913+0000 b341 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3f6eb00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004f4:[<0x100000003a4/168.out v358>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000004f4:[<0x100000003a4/168.out v886>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.708 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.470+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003ce [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/021.out auth v874 ap=1 s=43 n(v0 rc2024-04-21T23:19:50.220923+0000 b43 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3f6f080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003ce:[<0x100000003a4/021.out v84>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000003ce:[<0x100000003a4/021.out v874>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.708 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.470+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003c2 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/015.out auth v870 ap=1 s=43 n(v0 rc2024-04-21T23:19:50.208923+0000 b43 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3f6f600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003c2:[<0x100000003a4/015.out v60>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000003c2:[<0x100000003a4/015.out v870>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.708 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.471+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000052a [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/195.out auth v716 ap=1 s=43 n(v0 rc2024-04-21T23:19:50.945913+0000 b43 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=0 authpin=1 scrubqueue=0 0x5572f3f70680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000052a:[<0x100000003a4/195.out v362>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x1000000052a:[<0x100000003a4/195.out v716>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.708 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.472+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000426 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/065.out auth v878 ap=1 s=43 n(v0 rc2024-04-21T23:19:50.310922+0000 b43 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3f6fb80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000426:[<0x100000003a4/065.out v260>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000426:[<0x100000003a4/065.out v878>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.708 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.472+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000046c [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/100.out auth v668 ap=1 s=371 n(v0 rc2024-04-21T23:19:50.707916+0000 b371 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=0 authpin=1 scrubqueue=0 0x5572f3f70c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000046c:[<0x100000003a4/100.out v312>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x1000000046c:[<0x100000003a4/100.out v668>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.708 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.473+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003f2 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/039.out auth v704 ap=1 s=25 n(v0 rc2024-04-21T23:19:50.257922+0000 b25 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=0 authpin=1 scrubqueue=0 0x5572f3f78580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003f2:[<0x100000003a4/039.out v156>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000003f2:[<0x100000003a4/039.out v704>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.708 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.473+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003f0 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/038.out auth v790 ap=1 s=424 n(v0 rc2024-04-21T23:19:50.255922+0000 b424 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=0 authpin=1 scrubqueue=0 0x5572f3f79080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003f0:[<0x100000003a4/038.out v152>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000003f0:[<0x100000003a4/038.out v790>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.709 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.474+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003e0 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/030.out auth v676 ap=1 s=25 n(v0 rc2024-04-21T23:19:50.238923+0000 b25 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=0 authpin=1 scrubqueue=0 0x5572f3f8a000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003e0:[<0x100000003a4/030.out v120>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000003e0:[<0x100000003a4/030.out v676>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.709 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.474+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004a2 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/127.out auth v712 ap=1 s=43 n(v0 rc2024-04-21T23:19:50.793915+0000 b43 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=0 authpin=1 scrubqueue=0 0x5572f3f8b080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004a2:[<0x100000003a4/127.out v326>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000004a2:[<0x100000003a4/127.out v712>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.709 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.474+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000524 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/192.out auth v736 ap=1 s=43 n(v0 rc2024-04-21T23:19:50.939913+0000 b43 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=0 authpin=1 scrubqueue=0 0x5572f3f8c100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000524:[<0x100000003a4/192.out v360>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000524:[<0x100000003a4/192.out v736>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.709 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.475+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000508 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/178.out auth v818 ap=1 s=109 n(v0 rc2024-04-21T23:19:50.910913+0000 b109 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=0 authpin=1 scrubqueue=0 0x5572f3f8d180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000508:[<0x100000003a4/178.out v352>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000508:[<0x100000003a4/178.out v818>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.709 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.476+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000516 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/185.out auth v728 ap=1 s=43 n(v0 rc2024-04-21T23:19:50.925913+0000 b43 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=0 authpin=1 scrubqueue=0 0x5572f3f97600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000516:[<0x100000003a4/185.out v356>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000516:[<0x100000003a4/185.out v728>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.709 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.477+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000470 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/102.out auth v694 ap=1 s=43 n(v0 rc2024-04-21T23:19:50.713916+0000 b43 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=0 authpin=1 scrubqueue=0 0x5572f3f96580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000470:[<0x100000003a4/102.out v316>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000470:[<0x100000003a4/102.out v694>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.709 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.477+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000052c [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/196.out auth v748 ap=1 s=43 n(v0 rc2024-04-21T23:19:50.947913+0000 b43 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=0 authpin=1 scrubqueue=0 0x5572f3f98100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000052c:[<0x100000003a4/196.out v364>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x1000000052c:[<0x100000003a4/196.out v748>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.709 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.478+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000047e [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/109.out auth v760 ap=1 s=616 n(v0 rc2024-04-21T23:19:50.735916+0000 b616 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=0 authpin=1 scrubqueue=0 0x5572f3f98680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000047e:[<0x100000003a4/109.out v320>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x1000000047e:[<0x100000003a4/109.out v760>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.709 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.478+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000047a [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/107.out auth v698 ap=1 s=120 n(v0 rc2024-04-21T23:19:50.729916+0000 b120 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=0 authpin=1 scrubqueue=0 0x5572f3f99700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000047a:[<0x100000003a4/107.out v318>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x1000000047a:[<0x100000003a4/107.out v698>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.709 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.478+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000456 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/089.out auth v776 ap=1 s=43 n(v0 rc2024-04-21T23:19:50.632917+0000 b43 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=0 authpin=1 scrubqueue=0 0x5572f3f99180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000456:[<0x100000003a4/089.out v308>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000456:[<0x100000003a4/089.out v776>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.709 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.480+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003cc [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/020.out auth v786 ap=1 s=43 n(v0 b43 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=0 authpin=1 scrubqueue=0 0x5572f3fae580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003cc:[<0x100000003a4/020.out v80>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000003cc:[<0x100000003a4/020.out v786>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.709 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.480+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000042e [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/069.out auth v708 ap=1 s=43 n(v0 rc2024-04-21T23:19:50.318921+0000 b43 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=0 authpin=1 scrubqueue=0 0x5572f3fa3700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000042e:[<0x100000003a4/069.out v276>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x1000000042e:[<0x100000003a4/069.out v708>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.710 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.480+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000502 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/175.out auth v814 ap=1 s=258 n(v0 rc2024-04-21T23:19:50.904913+0000 b258 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=0 authpin=1 scrubqueue=0 0x5572f3faf080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000502:[<0x100000003a4/175.out v348>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000502:[<0x100000003a4/175.out v814>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.710 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.481+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000051a [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/187.out auth v742 ap=1 s=159 n(v0 rc2024-04-21T23:19:50.929913+0000 b159 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=0 authpin=1 scrubqueue=0 0x5572f3fb0680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000051a:[<0x100000003a4/187.out v358>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x1000000051a:[<0x100000003a4/187.out v742>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.710 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.481+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000446 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/081.out auth v810 ap=1 s=10189 n(v0 rc2024-04-21T23:19:50.612917+0000 b10189 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=0 authpin=1 scrubqueue=0 0x5572f3fbe580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000446:[<0x100000003a4/081.out v304>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000446:[<0x100000003a4/081.out v810>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.710 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.482+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000438 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/074.out auth v732 ap=1 s=43 n(v0 rc2024-04-21T23:19:50.328921+0000 b43 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=0 authpin=1 scrubqueue=0 0x5572f3fbeb00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000438:[<0x100000003a4/074.out v296>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000438:[<0x100000003a4/074.out v732>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.710 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.482+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000040a [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/051.out auth v822 ap=1 s=25 n(v0 rc2024-04-21T23:19:50.280922+0000 b25 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=0 authpin=1 scrubqueue=0 0x5572f3fc0100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000040a:[<0x100000003a4/051.out v204>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x1000000040a:[<0x100000003a4/051.out v822>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.710 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.483+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003da [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/027.out auth v666 ap=1 s=43 n(v0 rc2024-04-21T23:19:50.232923+0000 b43 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=0 authpin=1 scrubqueue=0 0x5572f3fc0c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003da:[<0x100000003a4/027.out v108>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000003da:[<0x100000003a4/027.out v666>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.710 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.483+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000518 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/186.out auth v708 ap=1 s=403 n(v0 rc2024-04-21T23:19:50.927913+0000 b403 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=0 authpin=1 scrubqueue=0 0x5572f3fc1180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000518:[<0x100000003a4/186.out v344>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000518:[<0x100000003a4/186.out v708>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.710 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.483+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000422 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/063.out auth v794 ap=1 s=43 n(v0 rc2024-04-21T23:19:50.306922+0000 b43 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=0 authpin=1 scrubqueue=0 0x5572f3fc6580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000422:[<0x100000003a4/063.out v252>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000422:[<0x100000003a4/063.out v794>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.710 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.484+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000414 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/056.out auth v650 ap=1 s=2419 n(v0 rc2024-04-21T23:19:50.291922+0000 b2419 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=0 authpin=1 scrubqueue=0 0x5572f3fc6000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000414:[<0x100000003a4/056.out v224>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000414:[<0x100000003a4/056.out v650>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.710 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.484+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004f2 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/167.out auth v798 ap=1 s=43 n(v0 rc2024-04-21T23:19:50.886913+0000 b43 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=0 authpin=1 scrubqueue=0 0x5572f3fc7b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004f2:[<0x100000003a4/167.out v340>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000004f2:[<0x100000003a4/167.out v798>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.711 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.484+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000538 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/202.out auth v704 ap=1 s=149 n(v0 rc2024-04-21T23:19:50.960912+0000 b149 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=0 authpin=1 scrubqueue=0 0x5572f3fc8100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000538:[<0x100000003a4/202.out v352>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000538:[<0x100000003a4/202.out v704>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.711 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.485+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004bc [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/140.out auth v744 ap=1 s=2652 n(v0 rc2024-04-21T23:19:50.828914+0000 b2652 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=0 authpin=1 scrubqueue=0 0x5572f3fc9180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004bc:[<0x100000003a4/140.out v324>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000004bc:[<0x100000003a4/140.out v744>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.711 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.487+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004b8 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/138.out auth v740 ap=1 s=43 n(v0 rc2024-04-21T23:19:50.822914+0000 b43 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=0 authpin=1 scrubqueue=0 0x5572f3fdd700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004b8:[<0x100000003a4/138.out v322>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000004b8:[<0x100000003a4/138.out v740>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.711 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.487+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003be [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/013.out auth v646 ap=1 s=43 n(v0 rc2024-04-21T23:19:50.204923+0000 b43 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=0 authpin=1 scrubqueue=0 0x5572f3fe6b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003be:[<0x100000003a4/013.out v52>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000003be:[<0x100000003a4/013.out v646>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.711 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.488+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000045e [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/093.out auth v756 ap=1 s=1027 n(v0 rc2024-04-21T23:19:50.685916+0000 b1027 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=0 authpin=1 scrubqueue=0 0x5572f3fda580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000045e:[<0x100000003a4/093.out v306>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x1000000045e:[<0x100000003a4/093.out v756>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.711 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.489+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000514 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/184.out auth v802 ap=1 s=43 n(v0 rc2024-04-21T23:19:50.923913+0000 b43 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=0 authpin=1 scrubqueue=0 0x5572f3fe7080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000514:[<0x100000003a4/184.out v342>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000514:[<0x100000003a4/184.out v802>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.711 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.490+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000434 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/072.out auth v692 ap=1 s=43 n(v0 rc2024-04-21T23:19:50.323921+0000 b43 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=0 authpin=1 scrubqueue=0 0x5572f3fe8680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000434:[<0x100000003a4/072.out v288>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000434:[<0x100000003a4/072.out v692>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.711 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.491+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004ce [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/149.out auth v768 ap=1 s=412 n(v0 rc2024-04-21T23:19:50.848914+0000 b412 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=0 authpin=1 scrubqueue=0 0x5572f3ff0000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004ce:[<0x100000003a4/149.out v328>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000004ce:[<0x100000003a4/149.out v768>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.711 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.492+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004e4 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/160.out auth v748 ap=1 s=115 n(v0 rc2024-04-21T23:19:50.872914+0000 b115 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=0 authpin=1 scrubqueue=0 0x5572f3ff2680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004e4:[<0x100000003a4/160.out v336>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000004e4:[<0x100000003a4/160.out v748>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.711 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.493+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000412 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/055.out auth v676 ap=1 s=9130 n(v0 rc2024-04-21T23:19:50.289922+0000 b9130 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=0 authpin=1 scrubqueue=0 0x5572f3ff2100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000412:[<0x100000003a4/055.out v220>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000412:[<0x100000003a4/055.out v676>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.711 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.494+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000490 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/118.out auth v686 ap=1 s=43 n(v0 rc2024-04-21T23:19:50.765915+0000 b43 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=0 authpin=1 scrubqueue=0 0x5572f4004c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000490:[<0x100000003a4/118.out v314>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000490:[<0x100000003a4/118.out v686>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.711 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.494+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004e0 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/158.out auth v718 ap=1 s=180 n(v0 rc2024-04-21T23:19:50.868914+0000 b180 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=0 authpin=1 scrubqueue=0 0x5572f400c580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004e0:[<0x100000003a4/158.out v334>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000004e0:[<0x100000003a4/158.out v718>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.712 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.495+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000416 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/057.out auth v672 ap=1 s=43 n(v0 rc2024-04-21T23:19:50.293922+0000 b43 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=0 authpin=1 scrubqueue=0 0x5572f400d600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000416:[<0x100000003a4/057.out v228>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000416:[<0x100000003a4/057.out v672>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.712 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.497+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000408 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/050.out auth v738 ap=1 s=25 n(v0 rc2024-04-21T23:19:50.278922+0000 b25 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=0 authpin=1 scrubqueue=0 0x5572f4018100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000408:[<0x100000003a4/050.out v200>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000408:[<0x100000003a4/050.out v738>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.712 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.497+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003a6 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/001.out auth v804 ap=1 s=695 n(v0 rc2024-04-21T23:19:50.180923+0000 b695 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=0 authpin=1 scrubqueue=0 0x5572f4019700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003a6:[<0x100000003a4/001.out v4>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000003a6:[<0x100000003a4/001.out v804>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.712 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.497+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003e4 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/032.out auth v810 ap=1 s=126 n(v0 rc2024-04-21T23:19:50.242922+0000 b126 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=0 authpin=1 scrubqueue=0 0x5572f4027080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003e4:[<0x100000003a4/032.out v128>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000003e4:[<0x100000003a4/032.out v810>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.712 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.498+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000488 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/114.out auth v860 ap=1 s=43 n(v0 rc2024-04-21T23:19:50.752915+0000 b43 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=0 authpin=1 scrubqueue=0 0x5572f4027b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000488:[<0x100000003a4/114.out v322>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000488:[<0x100000003a4/114.out v860>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.712 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.498+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000474 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/104.out auth v848 ap=1 s=25 n(v0 rc2024-04-21T23:19:50.720916+0000 b25 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=0 authpin=1 scrubqueue=0 0x5572f4029180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000474:[<0x100000003a4/104.out v318>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000474:[<0x100000003a4/104.out v848>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.712 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.499+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000450 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/086.out auth v780 ap=1 s=64 n(v0 rc2024-04-21T23:19:50.624917+0000 b64 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=0 authpin=1 scrubqueue=0 0x5572f4028c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000450:[<0x100000003a4/086.out v308>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000450:[<0x100000003a4/086.out v780>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.712 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.499+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000532 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/199.out auth v828 ap=1 s=43 n(v0 rc2024-04-21T23:19:50.953912+0000 b43 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=0 authpin=1 scrubqueue=0 0x5572f4035600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000532:[<0x100000003a4/199.out v360>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000532:[<0x100000003a4/199.out v828>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.712 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.500+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003e6 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/033.out auth v814 ap=1 s=43 n(v0 rc2024-04-21T23:19:50.244922+0000 b43 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=0 authpin=1 scrubqueue=0 0x5572f4036680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003e6:[<0x100000003a4/033.out v132>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000003e6:[<0x100000003a4/033.out v814>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.712 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.501+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000526 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/193.out auth v792 ap=1 s=361 n(v0 rc2024-04-21T23:19:50.941913+0000 b361 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=0 authpin=1 scrubqueue=0 0x5572f4040000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000526:[<0x100000003a4/193.out v354>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000526:[<0x100000003a4/193.out v792>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.712 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.501+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004ca [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/147.out auth v742 ap=1 s=109 n(v0 b109 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=0 authpin=1 scrubqueue=0 0x5572f4042680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004ca:[<0x100000003a4/147.out v336>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000004ca:[<0x100000003a4/147.out v742>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.712 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.501+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000530 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/198.out auth v864 ap=1 s=654 n(v0 rc2024-04-21T23:19:50.951912+0000 b654 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=0 authpin=1 scrubqueue=0 0x5572f404e000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000530:[<0x100000003a4/198.out v358>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000530:[<0x100000003a4/198.out v864>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.712 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.501+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000052e [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/197.out auth v768 ap=1 s=654 n(v0 rc2024-04-21T23:19:50.949913+0000 b654 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=0 authpin=1 scrubqueue=0 0x5572f404e580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000052e:[<0x100000003a4/197.out v356>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x1000000052e:[<0x100000003a4/197.out v768>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.713 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.502+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000043a [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/075.out auth v832 ap=1 s=43 n(v0 rc2024-04-21T23:19:50.330921+0000 b43 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=0 authpin=1 scrubqueue=0 0x5572f404f600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000043a:[<0x100000003a4/075.out v300>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x1000000043a:[<0x100000003a4/075.out v832>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.713 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.503+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004be [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/141.out auth v756 ap=1 s=2652 n(v0 rc2024-04-21T23:19:50.830914+0000 b2652 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=0 authpin=1 scrubqueue=0 0x5572f4041b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004be:[<0x100000003a4/141.out v334>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000004be:[<0x100000003a4/141.out v756>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.713 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.503+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003dc [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/028.out auth v796 ap=1 s=43 n(v0 rc2024-04-21T23:19:50.234923+0000 b43 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=0 authpin=1 scrubqueue=0 0x5572f4050100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003dc:[<0x100000003a4/028.out v112>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000003dc:[<0x100000003a4/028.out v796>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.713 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.503+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000536 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/201.out auth v722 ap=1 s=960 n(v0 rc2024-04-21T23:19:50.958912+0000 b960 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=0 authpin=1 scrubqueue=0 0x5572f4050680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000536:[<0x100000003a4/201.out v362>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000536:[<0x100000003a4/201.out v722>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.713 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.504+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000462 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/095.out auth v710 ap=1 s=919 n(v0 rc2024-04-21T23:19:50.691916+0000 b919 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=0 authpin=1 scrubqueue=0 0x5572f405b080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000462:[<0x100000003a4/095.out v314>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000462:[<0x100000003a4/095.out v710>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.713 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.505+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000041a [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/059.out auth v730 ap=1 s=340 n(v0 rc2024-04-21T23:19:50.297922+0000 b340 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=0 authpin=1 scrubqueue=0 0x5572f405c100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000041a:[<0x100000003a4/059.out v236>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x1000000041a:[<0x100000003a4/059.out v730>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.713 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.506+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000049a [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/123.out auth v750 ap=1 s=3102 n(v0 rc2024-04-21T23:19:50.781915+0000 b3102 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=0 authpin=1 scrubqueue=0 0x5572f405a000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000049a:[<0x100000003a4/123.out v324>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x1000000049a:[<0x100000003a4/123.out v750>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.713 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.506+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000632 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/cifs/001.out auth v28 ap=1 s=957 n(v0 rc2024-04-21T23:19:51.231909+0000 b957 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f4068680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000632:[<0x10000000630/001.out v4>,<0x100000003a2/cifs v1316>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000632:[<0x10000000630/001.out v28>,<0x100000003a2/cifs v31124>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.713 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.507+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000045c [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/092.out auth v772 ap=1 s=43 n(v0 rc2024-04-21T23:19:50.681916+0000 b43 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=0 authpin=1 scrubqueue=0 0x5572f4067600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000045c:[<0x100000003a4/092.out v310>,<0x100000003a2/btrfs v1284>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x1000000045c:[<0x100000003a4/092.out v772>,<0x100000003a2/btrfs v32276>,<0x10000000007/tests v3005>,<0x10000000006/xfstests-dev v267>,<0x10000000005/tmp v60>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.714 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.622+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000681 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/037.out auth v872 ap=1 s=43 n(v0 rc2024-04-21T23:19:51.310907+0000 b43 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3ca8c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000681:[<0x10000000634/037.out v154>,<0x100000003a2/ext4 v1598>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000681:[<0x10000000634/037.out v872>,<0x100000003a2/ext4 v32278>,<0x10000000007/tests v3037>,<0x10000000006/xfstests-dev v271>,<0x10000000005/tmp v62>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.714 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.623+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000689 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/041.out auth v864 ap=1 s=112 n(v0 rc2024-04-21T23:19:51.317907+0000 b112 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3cb2000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000689:[<0x10000000634/041.out v170>,<0x100000003a2/ext4 v1598>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000689:[<0x10000000634/041.out v864>,<0x100000003a2/ext4 v32278>,<0x10000000007/tests v3037>,<0x10000000006/xfstests-dev v271>,<0x10000000005/tmp v62>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.714 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.623+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000067b [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/034.out auth v868 ap=1 s=306 n(v0 rc2024-04-21T23:19:51.304908+0000 b306 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3cb2580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000067b:[<0x10000000634/034.out v142>,<0x100000003a2/ext4 v1598>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x1000000067b:[<0x10000000634/034.out v868>,<0x100000003a2/ext4 v32278>,<0x10000000007/tests v3037>,<0x10000000006/xfstests-dev v271>,<0x10000000005/tmp v62>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.714 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.624+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000065b [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/018.out auth v880 ap=1 s=200 n(v0 rc2024-04-21T23:19:51.273908+0000 b200 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3cb3080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000065b:[<0x10000000634/018.out v78>,<0x100000003a2/ext4 v1598>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x1000000065b:[<0x10000000634/018.out v880>,<0x100000003a2/ext4 v32278>,<0x10000000007/tests v3037>,<0x10000000006/xfstests-dev v271>,<0x10000000005/tmp v62>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.714 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.624+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000067d [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/035.out auth v860 ap=1 s=77 n(v0 rc2024-04-21T23:19:51.306908+0000 b77 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3cb4680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000067d:[<0x10000000634/035.out v146>,<0x100000003a2/ext4 v1598>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x1000000067d:[<0x10000000634/035.out v860>,<0x100000003a2/ext4 v32278>,<0x10000000007/tests v3037>,<0x10000000006/xfstests-dev v271>,<0x10000000005/tmp v62>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.714 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.625+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000063f [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/004.out auth v890 ap=1 s=43 n(v0 rc2024-04-21T23:19:51.244908+0000 b43 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3cbc580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000063f:[<0x10000000634/004.out v22>,<0x100000003a2/ext4 v1598>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x1000000063f:[<0x10000000634/004.out v890>,<0x100000003a2/ext4 v32278>,<0x10000000007/tests v3037>,<0x10000000006/xfstests-dev v271>,<0x10000000005/tmp v62>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.714 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.625+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000647 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/008.out auth v870 ap=1 s=166 n(v0 rc2024-04-21T23:19:51.253908+0000 b166 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3cbcb00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000647:[<0x10000000634/008.out v38>,<0x100000003a2/ext4 v1598>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000647:[<0x10000000634/008.out v870>,<0x100000003a2/ext4 v32278>,<0x10000000007/tests v3037>,<0x10000000006/xfstests-dev v271>,<0x10000000005/tmp v62>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.714 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.626+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000064d [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/011.out auth v874 ap=1 s=166 n(v0 rc2024-04-21T23:19:51.259908+0000 b166 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3cbd080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000064d:[<0x10000000634/011.out v50>,<0x100000003a2/ext4 v1598>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x1000000064d:[<0x10000000634/011.out v874>,<0x100000003a2/ext4 v32278>,<0x10000000007/tests v3037>,<0x10000000006/xfstests-dev v271>,<0x10000000005/tmp v62>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.717 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.626+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000641 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/005.out auth v894 ap=1 s=43 n(v0 rc2024-04-21T23:19:51.247908+0000 b43 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3cbdb80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000641:[<0x10000000634/005.out v26>,<0x100000003a2/ext4 v1598>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000641:[<0x10000000634/005.out v894>,<0x100000003a2/ext4 v32278>,<0x10000000007/tests v3037>,<0x10000000006/xfstests-dev v271>,<0x10000000005/tmp v62>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.717 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.627+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000065d [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/019.out auth v886 ap=1 s=185 n(v0 rc2024-04-21T23:19:51.275908+0000 b185 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3cbe100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000065d:[<0x10000000634/019.out v82>,<0x100000003a2/ext4 v1598>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x1000000065d:[<0x10000000634/019.out v886>,<0x100000003a2/ext4 v32278>,<0x10000000007/tests v3037>,<0x10000000006/xfstests-dev v271>,<0x10000000005/tmp v62>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.717 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.627+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000064f [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/012.out auth v878 ap=1 s=166 n(v0 rc2024-04-21T23:19:51.261908+0000 b166 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3cce000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000064f:[<0x10000000634/012.out v54>,<0x100000003a2/ext4 v1598>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x1000000064f:[<0x10000000634/012.out v878>,<0x100000003a2/ext4 v32278>,<0x10000000007/tests v3037>,<0x10000000006/xfstests-dev v271>,<0x10000000005/tmp v62>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.717 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.628+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000657 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/016.out auth v866 ap=1 s=198 n(v0 rc2024-04-21T23:19:51.269908+0000 b198 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3cceb00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000657:[<0x10000000634/016.out v70>,<0x100000003a2/ext4 v1598>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000657:[<0x10000000634/016.out v866>,<0x100000003a2/ext4 v32278>,<0x10000000007/tests v3037>,<0x10000000006/xfstests-dev v271>,<0x10000000005/tmp v62>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.717 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.628+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000063d [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/003.out auth v898 ap=1 s=128 n(v0 rc2024-04-21T23:19:51.242908+0000 b128 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3cce580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000063d:[<0x10000000634/003.out v18>,<0x100000003a2/ext4 v1598>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x1000000063d:[<0x10000000634/003.out v898>,<0x100000003a2/ext4 v32278>,<0x10000000007/tests v3037>,<0x10000000006/xfstests-dev v271>,<0x10000000005/tmp v62>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.717 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.629+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000006b7 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/304.out auth v882 ap=1 s=54 n(v0 rc2024-04-21T23:19:51.362907+0000 b54 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3cd0100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000006b7:[<0x10000000634/304.out v262>,<0x100000003a2/ext4 v1598>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000006b7:[<0x10000000634/304.out v882>,<0x100000003a2/ext4 v32278>,<0x10000000007/tests v3037>,<0x10000000006/xfstests-dev v271>,<0x10000000005/tmp v62>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.717 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.629+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000687 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/040.out auth v876 ap=1 s=112 n(v0 b112 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3cb5700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000687:[<0x10000000634/040.out v166>,<0x100000003a2/ext4 v1598>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000687:[<0x10000000634/040.out v876>,<0x100000003a2/ext4 v32278>,<0x10000000007/tests v3037>,<0x10000000006/xfstests-dev v271>,<0x10000000005/tmp v62>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.717 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.630+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000006bd [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/307.out auth v876 ap=1 s=89 n(v0 rc2024-04-21T23:19:51.368907+0000 b89 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3cd1700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000006bd:[<0x10000000634/307.out v274>,<0x100000003a2/ext4 v1598>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000006bd:[<0x10000000634/307.out v876>,<0x100000003a2/ext4 v32278>,<0x10000000007/tests v3037>,<0x10000000006/xfstests-dev v271>,<0x10000000005/tmp v62>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.717 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.630+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000064b [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/010.out auth v868 ap=1 s=200 n(v0 rc2024-04-21T23:19:51.257908+0000 b200 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3cdf080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000064b:[<0x10000000634/010.out v46>,<0x100000003a2/ext4 v1598>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x1000000064b:[<0x10000000634/010.out v868>,<0x100000003a2/ext4 v32278>,<0x10000000007/tests v3037>,<0x10000000006/xfstests-dev v271>,<0x10000000005/tmp v62>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.718 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.631+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000669 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/025.out auth v872 ap=1 s=138 n(v0 rc2024-04-21T23:19:51.286908+0000 b138 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3cdfb80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000669:[<0x10000000634/025.out v106>,<0x100000003a2/ext4 v1598>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000669:[<0x10000000634/025.out v872>,<0x100000003a2/ext4 v32278>,<0x10000000007/tests v3037>,<0x10000000006/xfstests-dev v271>,<0x10000000005/tmp v62>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.718 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.631+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000066b [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/026.out auth v900 ap=1 s=1271 n(v0 b1271 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3cdf600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000066b:[<0x10000000634/026.out v110>,<0x100000003a2/ext4 v1598>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x1000000066b:[<0x10000000634/026.out v900>,<0x100000003a2/ext4 v32278>,<0x10000000007/tests v3037>,<0x10000000006/xfstests-dev v271>,<0x10000000005/tmp v62>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.718 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.631+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000006b5 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/303.out auth v904 ap=1 s=54 n(v0 rc2024-04-21T23:19:51.360907+0000 b54 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3ce0680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000006b5:[<0x10000000634/303.out v258>,<0x100000003a2/ext4 v1598>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000006b5:[<0x10000000634/303.out v904>,<0x100000003a2/ext4 v32278>,<0x10000000007/tests v3037>,<0x10000000006/xfstests-dev v271>,<0x10000000005/tmp v62>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.718 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.632+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000653 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/014.out auth v892 ap=1 s=235 n(v0 rc2024-04-21T23:19:51.265908+0000 b235 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3ce9080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000653:[<0x10000000634/014.out v62>,<0x100000003a2/ext4 v1598>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000653:[<0x10000000634/014.out v892>,<0x100000003a2/ext4 v32278>,<0x10000000007/tests v3037>,<0x10000000006/xfstests-dev v271>,<0x10000000005/tmp v62>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.718 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.633+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000006b9 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/305.out auth v888 ap=1 s=43 n(v0 rc2024-04-21T23:19:51.364907+0000 b43 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3ce1700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000006b9:[<0x10000000634/305.out v266>,<0x100000003a2/ext4 v1598>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000006b9:[<0x10000000634/305.out v888>,<0x100000003a2/ext4 v32278>,<0x10000000007/tests v3037>,<0x10000000006/xfstests-dev v271>,<0x10000000005/tmp v62>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.718 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.633+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000068b [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/042.out auth v896 ap=1 s=77 n(v0 b77 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3ce8b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000068b:[<0x10000000634/042.out v174>,<0x100000003a2/ext4 v1598>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x1000000068b:[<0x10000000634/042.out v896>,<0x100000003a2/ext4 v32278>,<0x10000000007/tests v3037>,<0x10000000006/xfstests-dev v271>,<0x10000000005/tmp v62>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.718 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.633+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000667 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/024.out auth v880 ap=1 s=39 n(v0 rc2024-04-21T23:19:51.284908+0000 b39 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3ce9600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000667:[<0x10000000634/024.out v102>,<0x100000003a2/ext4 v1598>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000667:[<0x10000000634/024.out v880>,<0x100000003a2/ext4 v32278>,<0x10000000007/tests v3037>,<0x10000000006/xfstests-dev v271>,<0x10000000005/tmp v62>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.718 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.633+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000665 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/023.out auth v884 ap=1 s=51 n(v0 rc2024-04-21T23:19:51.283908+0000 b51 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3ce9b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000665:[<0x10000000634/023.out v98>,<0x100000003a2/ext4 v1598>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000665:[<0x10000000634/023.out v884>,<0x100000003a2/ext4 v32278>,<0x10000000007/tests v3037>,<0x10000000006/xfstests-dev v271>,<0x10000000005/tmp v62>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.718 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.634+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000645 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/007.out auth v854 ap=1 s=210 n(v0 rc2024-04-21T23:19:51.251908+0000 b210 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3ceb700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000645:[<0x10000000634/007.out v34>,<0x100000003a2/ext4 v1598>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000645:[<0x10000000634/007.out v854>,<0x100000003a2/ext4 v32278>,<0x10000000007/tests v3037>,<0x10000000006/xfstests-dev v271>,<0x10000000005/tmp v62>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.718 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.635+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000006b1 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/301.out auth v858 ap=1 s=54 n(v0 rc2024-04-21T23:19:51.356907+0000 b54 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3cfac00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000006b1:[<0x10000000634/301.out v250>,<0x100000003a2/ext4 v1598>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000006b1:[<0x10000000634/301.out v858>,<0x100000003a2/ext4 v32278>,<0x10000000007/tests v3037>,<0x10000000006/xfstests-dev v271>,<0x10000000005/tmp v62>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.719 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.636+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000067f [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/036.out auth v862 ap=1 s=43 n(v0 rc2024-04-21T23:19:51.308907+0000 b43 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3d05080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000067f:[<0x10000000634/036.out v150>,<0x100000003a2/ext4 v1598>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x1000000067f:[<0x10000000634/036.out v862>,<0x100000003a2/ext4 v32278>,<0x10000000007/tests v3037>,<0x10000000006/xfstests-dev v271>,<0x10000000005/tmp v62>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.719 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.636+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000663 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/022.out auth v882 ap=1 s=6043 n(v0 rc2024-04-21T23:19:51.281908+0000 b6043 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3d06680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000663:[<0x10000000634/022.out v94>,<0x100000003a2/ext4 v1598>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000663:[<0x10000000634/022.out v882>,<0x100000003a2/ext4 v32278>,<0x10000000007/tests v3037>,<0x10000000006/xfstests-dev v271>,<0x10000000005/tmp v62>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.719 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.637+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000006bf [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/308.out auth v906 ap=1 s=33582 n(v0 rc2024-04-21T23:19:51.370907+0000 b33582 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3d0e000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000006bf:[<0x10000000634/308.out v278>,<0x100000003a2/ext4 v1598>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000006bf:[<0x10000000634/308.out v906>,<0x100000003a2/ext4 v32278>,<0x10000000007/tests v3037>,<0x10000000006/xfstests-dev v271>,<0x10000000005/tmp v62>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.719 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.637+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000643 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/006.out auth v890 ap=1 s=89 n(v0 rc2024-04-21T23:19:51.249908+0000 b89 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3d07700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000643:[<0x10000000634/006.out v30>,<0x100000003a2/ext4 v1598>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000643:[<0x10000000634/006.out v890>,<0x100000003a2/ext4 v32278>,<0x10000000007/tests v3037>,<0x10000000006/xfstests-dev v271>,<0x10000000005/tmp v62>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.719 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.637+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000649 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/009.out auth v878 ap=1 s=200 n(v0 rc2024-04-21T23:19:51.255908+0000 b200 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3d07180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000649:[<0x10000000634/009.out v42>,<0x100000003a2/ext4 v1598>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000649:[<0x10000000634/009.out v878>,<0x100000003a2/ext4 v32278>,<0x10000000007/tests v3037>,<0x10000000006/xfstests-dev v271>,<0x10000000005/tmp v62>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.719 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.638+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000671 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/029.out auth v886 ap=1 s=106 n(v0 rc2024-04-21T23:19:51.294908+0000 b106 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3d0fb80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000671:[<0x10000000634/029.out v122>,<0x100000003a2/ext4 v1598>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000671:[<0x10000000634/029.out v886>,<0x100000003a2/ext4 v32278>,<0x10000000007/tests v3037>,<0x10000000006/xfstests-dev v271>,<0x10000000005/tmp v62>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.719 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.639+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000006af [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/271.out auth v898 ap=1 s=693 n(v0 b693 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3d0f080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000006af:[<0x10000000634/271.out v246>,<0x100000003a2/ext4 v1598>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000006af:[<0x10000000634/271.out v898>,<0x100000003a2/ext4 v32278>,<0x10000000007/tests v3037>,<0x10000000006/xfstests-dev v271>,<0x10000000005/tmp v62>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.719 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.639+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000659 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/017.out auth v870 ap=1 s=198 n(v0 rc2024-04-21T23:19:51.271908+0000 b198 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3d10100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000659:[<0x10000000634/017.out v74>,<0x100000003a2/ext4 v1598>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000659:[<0x10000000634/017.out v870>,<0x100000003a2/ext4 v32278>,<0x10000000007/tests v3037>,<0x10000000006/xfstests-dev v271>,<0x10000000005/tmp v62>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.719 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.639+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000636 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/001.cfg auth v902 ap=1 s=87 n(v0 rc2024-04-21T23:19:51.236908+0000 b87 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3d21080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000636:[<0x10000000634/001.cfg v4>,<0x100000003a2/ext4 v1598>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000636:[<0x10000000634/001.cfg v902>,<0x100000003a2/ext4 v32278>,<0x10000000007/tests v3037>,<0x10000000006/xfstests-dev v271>,<0x10000000005/tmp v62>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.720 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.640+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000675 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/031.out auth v894 ap=1 s=43 n(v0 rc2024-04-21T23:19:51.298908+0000 b43 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3d21600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000675:[<0x10000000634/031.out v130>,<0x100000003a2/ext4 v1598>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000675:[<0x10000000634/031.out v894>,<0x100000003a2/ext4 v32278>,<0x10000000007/tests v3037>,<0x10000000006/xfstests-dev v271>,<0x10000000005/tmp v62>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.720 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.640+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000661 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/021.out auth v874 ap=1 s=62 n(v0 rc2024-04-21T23:19:51.279908+0000 b62 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3d21b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000661:[<0x10000000634/021.out v90>,<0x100000003a2/ext4 v1598>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000661:[<0x10000000634/021.out v874>,<0x100000003a2/ext4 v32278>,<0x10000000007/tests v3037>,<0x10000000006/xfstests-dev v271>,<0x10000000005/tmp v62>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.720 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.640+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000655 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/015.out auth v910 ap=1 s=200 n(v0 rc2024-04-21T23:19:51.267908+0000 b200 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3d22100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000655:[<0x10000000634/015.out v66>,<0x100000003a2/ext4 v1598>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000655:[<0x10000000634/015.out v910>,<0x100000003a2/ext4 v32278>,<0x10000000007/tests v3037>,<0x10000000006/xfstests-dev v271>,<0x10000000005/tmp v62>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.720 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.640+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000683 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/038.out auth v870 ap=1 s=43 n(v0 rc2024-04-21T23:19:51.312907+0000 b43 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3d23180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000683:[<0x10000000634/038.out v158>,<0x100000003a2/ext4 v1598>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000683:[<0x10000000634/038.out v870>,<0x100000003a2/ext4 v32278>,<0x10000000007/tests v3037>,<0x10000000006/xfstests-dev v271>,<0x10000000005/tmp v62>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.720 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.641+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000685 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/039.out auth v858 ap=1 s=91 n(v0 rc2024-04-21T23:19:51.314907+0000 b91 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3d22c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000685:[<0x10000000634/039.out v162>,<0x100000003a2/ext4 v1598>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000685:[<0x10000000634/039.out v858>,<0x100000003a2/ext4 v32278>,<0x10000000007/tests v3037>,<0x10000000006/xfstests-dev v271>,<0x10000000005/tmp v62>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.720 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.641+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000639 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/001.out.nozero auth v866 ap=1 s=8125 n(v0 rc2024-04-21T23:19:51.239908+0000 b8125 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3d22680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000639:[<0x10000000634/001.out.nozero v10>,<0x100000003a2/ext4 v1598>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000639:[<0x10000000634/001.out.nozero v866>,<0x100000003a2/ext4 v32278>,<0x10000000007/tests v3037>,<0x10000000006/xfstests-dev v271>,<0x10000000005/tmp v62>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.720 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.641+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000673 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/030.out auth v862 ap=1 s=43 n(v0 rc2024-04-21T23:19:51.296908+0000 b43 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3d2a580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000673:[<0x10000000634/030.out v126>,<0x100000003a2/ext4 v1598>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000673:[<0x10000000634/030.out v862>,<0x100000003a2/ext4 v32278>,<0x10000000007/tests v3037>,<0x10000000006/xfstests-dev v271>,<0x10000000005/tmp v62>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.720 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.642+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000065f [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/020.out auth v874 ap=1 s=242 n(v0 rc2024-04-21T23:19:51.277908+0000 b242 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3d2bb80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000065f:[<0x10000000634/020.out v86>,<0x100000003a2/ext4 v1598>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x1000000065f:[<0x10000000634/020.out v874>,<0x100000003a2/ext4 v32278>,<0x10000000007/tests v3037>,<0x10000000006/xfstests-dev v271>,<0x10000000005/tmp v62>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.720 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.642+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000066d [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/027.out auth v862 ap=1 s=121 n(v0 rc2024-04-21T23:19:51.290908+0000 b121 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3d2cc00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000066d:[<0x10000000634/027.out v114>,<0x100000003a2/ext4 v1598>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x1000000066d:[<0x10000000634/027.out v862>,<0x100000003a2/ext4 v32278>,<0x10000000007/tests v3037>,<0x10000000006/xfstests-dev v271>,<0x10000000005/tmp v62>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.720 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.643+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000651 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/013.out auth v874 ap=1 s=235 n(v0 rc2024-04-21T23:19:51.263908+0000 b235 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3d37600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000651:[<0x10000000634/013.out v58>,<0x100000003a2/ext4 v1598>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000651:[<0x10000000634/013.out v874>,<0x100000003a2/ext4 v32278>,<0x10000000007/tests v3037>,<0x10000000006/xfstests-dev v271>,<0x10000000005/tmp v62>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.720 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.643+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000637 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/001.out.default auth v866 ap=1 s=8125 n(v0 rc2024-04-21T23:19:51.237908+0000 b8125 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3d38100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000637:[<0x10000000634/001.out.default v6>,<0x100000003a2/ext4 v1598>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000637:[<0x10000000634/001.out.default v866>,<0x100000003a2/ext4 v32278>,<0x10000000007/tests v3037>,<0x10000000006/xfstests-dev v271>,<0x10000000005/tmp v62>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.720 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.643+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000006b3 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/302.out auth v870 ap=1 s=54 n(v0 rc2024-04-21T23:19:51.358907+0000 b54 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3d38680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000006b3:[<0x10000000634/302.out v254>,<0x100000003a2/ext4 v1598>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000006b3:[<0x10000000634/302.out v870>,<0x100000003a2/ext4 v32278>,<0x10000000007/tests v3037>,<0x10000000006/xfstests-dev v271>,<0x10000000005/tmp v62>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.720 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.644+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000677 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/032.out auth v860 ap=1 s=4065 n(v0 rc2024-04-21T23:19:51.300908+0000 b4065 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3d47b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000677:[<0x10000000634/032.out v134>,<0x100000003a2/ext4 v1598>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000677:[<0x10000000634/032.out v860>,<0x100000003a2/ext4 v32278>,<0x10000000007/tests v3037>,<0x10000000006/xfstests-dev v271>,<0x10000000005/tmp v62>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.720 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.644+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000638 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/001.out.nodelalloc auth v858 ap=1 s=8125 n(v0 rc2024-04-21T23:19:51.238908+0000 b8125 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3d46580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000638:[<0x10000000634/001.out.nodelalloc v8>,<0x100000003a2/ext4 v1598>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000638:[<0x10000000634/001.out.nodelalloc v858>,<0x100000003a2/ext4 v32278>,<0x10000000007/tests v3037>,<0x10000000006/xfstests-dev v271>,<0x10000000005/tmp v62>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.721 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.644+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000679 [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/033.out auth v856 ap=1 s=155 n(v0 rc2024-04-21T23:19:51.302908+0000 b155 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3d48c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000679:[<0x10000000634/033.out v138>,<0x100000003a2/ext4 v1598>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x10000000679:[<0x10000000634/033.out v856>,<0x100000003a2/ext4 v32278>,<0x10000000007/tests v3037>,<0x10000000006/xfstests-dev v271>,<0x10000000005/tmp v62>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.721 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.644+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000066f [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/028.out auth v868 ap=1 s=56 n(v0 rc2024-04-21T23:19:51.292908+0000 b56 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3d49700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000066f:[<0x10000000634/028.out v118>,<0x100000003a2/ext4 v1598>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x1000000066f:[<0x10000000634/028.out v868>,<0x100000003a2/ext4 v32278>,<0x10000000007/tests v3037>,<0x10000000006/xfstests-dev v271>,<0x10000000005/tmp v62>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.721 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:20:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:20:21.645+0000 7f567f7a1700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000006bb [4,head] /volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/306.out auth v864 ap=1 s=479 n(v0 rc2024-04-21T23:19:51.366907+0000 b479 1=1+0) caps={25132=pAsLsXsFsc/-@1},l=25132(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 needsnapflush=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x5572f3d54580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000006bb:[<0x10000000634/306.out v270>,<0x100000003a2/ext4 v1598>,<0x10000000007/tests v776>,<0x10000000006/xfstests-dev v74>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v12>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v28>,<0x10000000001/sv_1 v51>,<0x10000000000/qa v41>,<0x1/volumes v36>]//[3]","memoryvalue":"(4)0x100000006bb:[<0x10000000634/306.out v864>,<0x100000003a2/ext4 v32278>,<0x10000000007/tests v3037>,<0x10000000006/xfstests-dev v271>,<0x10000000005/tmp v62>,<0x100000001fb/client.0 v20>,<0x100000001fa/aa392c15-a144-41c4-854f-c3936c335a66 v18>,<0x10000000001/sv_1 v39>,<0x10000000000/qa v29>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-21T23:20:21.897 INFO:tasks.workunit.client.0.smithi040.stdout:libtoolize: putting auxiliary files in '.'. 2024-04-21T23:20:21.898 INFO:tasks.workunit.client.0.smithi040.stdout:libtoolize: copying file './config.guess' 2024-04-21T23:20:21.914 INFO:tasks.workunit.client.0.smithi040.stdout:libtoolize: copying file './config.sub' 2024-04-21T23:20:21.926 INFO:tasks.workunit.client.0.smithi040.stdout:libtoolize: copying file './install-sh' 2024-04-21T23:20:21.938 INFO:tasks.workunit.client.0.smithi040.stdout:libtoolize: copying file './ltmain.sh' 2024-04-21T23:20:21.982 INFO:tasks.workunit.client.0.smithi040.stdout:libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'm4'. 2024-04-21T23:20:21.982 INFO:tasks.workunit.client.0.smithi040.stdout:libtoolize: copying file 'm4/libtool.m4' 2024-04-21T23:20:21.991 INFO:tasks.workunit.client.0.smithi040.stdout:libtoolize: copying file 'm4/ltoptions.m4' 2024-04-21T23:20:22.000 INFO:tasks.workunit.client.0.smithi040.stdout:libtoolize: copying file 'm4/ltsugar.m4' 2024-04-21T23:20:22.009 INFO:tasks.workunit.client.0.smithi040.stdout:libtoolize: copying file 'm4/ltversion.m4' 2024-04-21T23:20:22.018 INFO:tasks.workunit.client.0.smithi040.stdout:libtoolize: copying file 'm4/lt~obsolete.m4' 2024-04-21T23:20:22.028 INFO:tasks.workunit.client.0.smithi040.stdout:libtoolize: Consider adding '-I m4' to ACLOCAL_AMFLAGS in Makefile.am. 2024-04-21T23:20:22.030 INFO:tasks.workunit.client.0.smithi040.stdout:cp include/install-sh . 2024-04-21T23:20:22.045 INFO:tasks.workunit.client.0.smithi040.stdout:aclocal -I m4 2024-04-21T23:20:22.433 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000004c6 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/145.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.433 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x1000000050c (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/180.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.433 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x1000000040c (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/052.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.433 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000003f8 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/042.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.433 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000472 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/103.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.433 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000004d2 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/151.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.433 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000454 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/088.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.434 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000004ac (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/132.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.434 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000003c0 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/014.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.434 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000004cc (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/148.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.434 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000003d2 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/023.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.434 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x1000000047c (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/108.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.434 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000430 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/070.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.434 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000004ee (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/165.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.434 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000448 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/082.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.434 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000496 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/121.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.434 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x1000000042c (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/068.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.434 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000003b2 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/007.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.434 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000420 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/062.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.435 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000004ec (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/164.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.435 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000494 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/120.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.435 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000004a8 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/130.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.435 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000428 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/066.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.435 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x1000000044e (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/085.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.435 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000410 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/054.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.435 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000004ae (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/133.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.435 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x1000000040e (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/053.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.435 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000004d6 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/153.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.435 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x1000000041e (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/061.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.435 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000442 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/079.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.435 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x1000000044c (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/084.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.435 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000444 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/080.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.435 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000004aa (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/131.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.435 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000004b0 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/134.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.436 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000003a8 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/002.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.436 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000452 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/087.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.436 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x1000000046e (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/101.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.436 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000004c2 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/143.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.436 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000520 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/190.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.436 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000003ec (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/036.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.436 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000004dc (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/156.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.436 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000504 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/176.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.436 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000003d0 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/022.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.436 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000004da (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/155.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.436 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000476 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/105.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.436 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000004e2 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/159.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.436 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000003b6 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/009.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.437 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x1000000048c (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/116.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.437 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000522 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/191.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.437 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000004e6 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/161.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.437 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000458 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/090.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.437 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000004f0 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/166.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.437 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000482 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/111.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.437 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000003c6 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/017.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.437 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000440 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/078.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.437 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000418 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/058.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.437 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000402 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/047.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.437 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000460 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/094.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.437 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000003e2 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/031.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.437 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000004c4 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/144.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.437 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000528 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/194.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.437 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000004f4 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/168.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.438 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000003ce (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/021.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.438 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000003c2 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/015.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.438 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x1000000052a (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/195.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.438 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000426 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/065.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.438 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x1000000046c (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/100.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.438 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000003f2 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/039.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.438 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000003f0 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/038.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.438 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000003e0 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/030.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.438 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000004a2 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/127.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.438 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000524 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/192.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.438 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000508 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/178.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.438 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000516 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/185.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.439 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000470 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/102.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.439 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x1000000052c (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/196.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.439 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x1000000047e (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/109.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.439 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x1000000047a (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/107.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.439 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000456 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/089.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.439 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000003cc (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/020.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.439 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x1000000042e (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/069.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.439 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000502 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/175.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.439 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x1000000051a (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/187.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.439 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000446 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/081.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.439 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000438 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/074.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.439 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x1000000040a (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/051.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.439 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000003da (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/027.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.439 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000518 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/186.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.439 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000422 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/063.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.440 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000414 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/056.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.440 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000004f2 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/167.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.440 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000538 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/202.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.440 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000004bc (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/140.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.440 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000004b8 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/138.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.440 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000003be (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/013.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.440 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x1000000045e (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/093.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.440 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000514 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/184.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.440 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000434 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/072.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.440 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000004ce (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/149.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.440 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000004e4 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/160.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.440 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000412 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/055.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.440 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000490 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/118.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.440 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000004e0 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/158.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.440 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000416 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/057.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.440 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000408 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/050.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.440 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000003a6 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/001.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.441 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000003e4 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/032.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.441 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000488 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/114.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.441 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000474 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/104.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.441 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000450 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/086.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.442 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000532 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/199.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.442 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000003e6 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/033.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.442 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000526 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/193.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.442 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000004ca (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/147.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.442 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000530 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/198.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.442 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x1000000052e (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/197.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.442 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x1000000043a (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/075.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.442 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000004be (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/141.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.442 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000003dc (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/028.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.442 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000536 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/201.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.442 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000462 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/095.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.442 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x1000000041a (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/059.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.442 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x1000000049a (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/123.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.442 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000632 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/cifs/001.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.443 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x1000000045c (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/092.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.443 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000681 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/037.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.443 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000689 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/041.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.443 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x1000000067b (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/034.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.443 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x1000000065b (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/018.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.443 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x1000000067d (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/035.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.443 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x1000000063f (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/004.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.443 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000647 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/008.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.443 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x1000000064d (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/011.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.443 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000641 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/005.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.443 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x1000000065d (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/019.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.443 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x1000000064f (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/012.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.444 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000657 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/016.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.444 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x1000000063d (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/003.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.444 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000006b7 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/304.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.444 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000687 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/040.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.444 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000006bd (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/307.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.444 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x1000000064b (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/010.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.444 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000669 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/025.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.444 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x1000000066b (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/026.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.444 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000006b5 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/303.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.444 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000653 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/014.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.445 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000006b9 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/305.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.445 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x1000000068b (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/042.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.445 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000667 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/024.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.445 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000665 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/023.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.445 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000645 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/007.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.445 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000006b1 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/301.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.445 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x1000000067f (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/036.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.445 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000663 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/022.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.445 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000006bf (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/308.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.445 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000643 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/006.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.445 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000649 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/009.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.446 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000671 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/029.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.446 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000006af (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/271.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.446 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000659 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/017.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.446 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000636 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/001.cfg) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.446 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000675 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/031.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.446 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000661 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/021.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.446 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000655 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/015.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.446 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000683 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/038.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.446 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000685 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/039.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.446 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000639 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/001.out.nozero) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.446 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000673 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/030.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.446 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x1000000065f (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/020.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.446 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x1000000066d (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/027.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.447 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000651 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/013.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.447 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000637 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/001.out.default) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.447 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000006b3 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/302.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.447 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000677 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/032.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.447 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000638 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/001.out.nodelalloc) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.447 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x10000000679 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/033.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.447 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x1000000066f (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/028.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.447 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: Scrub error on inode 0x100000006bb (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/306.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.447 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:22 smithi040 ceph-mon[27984]: scrub summary: idle+waiting paths [/] 2024-04-21T23:20:22.450 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000004c6 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/145.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.450 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x1000000050c (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/180.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.450 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x1000000040c (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/052.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.450 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000003f8 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/042.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.450 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000472 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/103.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.450 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000004d2 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/151.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.450 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000454 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/088.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.450 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000004ac (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/132.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.450 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000003c0 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/014.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.450 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000004cc (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/148.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.450 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000003d2 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/023.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.451 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x1000000047c (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/108.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.451 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000430 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/070.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.451 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000004ee (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/165.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.451 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000448 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/082.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.451 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000496 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/121.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.451 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x1000000042c (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/068.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.451 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000003b2 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/007.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.451 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000420 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/062.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.451 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000004ec (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/164.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.451 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000494 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/120.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.451 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000004a8 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/130.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.451 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000428 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/066.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.451 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x1000000044e (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/085.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.451 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000410 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/054.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.451 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000004ae (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/133.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.452 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x1000000040e (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/053.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.452 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000004d6 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/153.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.452 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x1000000041e (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/061.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.452 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000442 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/079.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.452 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x1000000044c (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/084.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.452 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000444 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/080.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.452 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000004aa (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/131.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.452 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000004b0 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/134.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.452 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000003a8 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/002.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.452 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000452 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/087.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.452 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x1000000046e (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/101.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.452 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000004c2 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/143.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.452 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000520 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/190.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.452 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000003ec (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/036.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.453 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000004dc (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/156.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.453 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000504 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/176.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.453 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000003d0 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/022.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.453 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000004da (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/155.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.453 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000476 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/105.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.453 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000004e2 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/159.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.453 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000003b6 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/009.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.453 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x1000000048c (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/116.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.453 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000522 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/191.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.453 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000004e6 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/161.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.453 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000458 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/090.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.453 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000004f0 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/166.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.453 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000482 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/111.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.453 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000003c6 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/017.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.453 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000440 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/078.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.453 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000418 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/058.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.454 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000402 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/047.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.454 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000460 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/094.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.454 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000003e2 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/031.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.454 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000004c4 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/144.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.454 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000528 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/194.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.454 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000004f4 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/168.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.454 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000003ce (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/021.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.454 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000003c2 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/015.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.454 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x1000000052a (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/195.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.454 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000426 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/065.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.454 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x1000000046c (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/100.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.454 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000003f2 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/039.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.454 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000003f0 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/038.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.454 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000003e0 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/030.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.455 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000004a2 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/127.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.455 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000524 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/192.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.455 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000508 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/178.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.455 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000516 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/185.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.455 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000470 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/102.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.455 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x1000000052c (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/196.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.455 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x1000000047e (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/109.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.455 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x1000000047a (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/107.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.455 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000456 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/089.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.455 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000003cc (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/020.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.455 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x1000000042e (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/069.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.455 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000502 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/175.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.455 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x1000000051a (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/187.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.455 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000446 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/081.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.456 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000438 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/074.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.456 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x1000000040a (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/051.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.456 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000003da (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/027.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.456 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000518 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/186.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.456 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000422 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/063.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.456 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000414 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/056.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.456 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000004f2 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/167.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.456 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000538 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/202.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.456 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000004bc (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/140.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.456 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000004b8 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/138.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.456 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000003be (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/013.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.456 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x1000000045e (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/093.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.456 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000514 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/184.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.456 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000434 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/072.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.456 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000004ce (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/149.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.457 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000004e4 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/160.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.457 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000412 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/055.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.457 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000490 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/118.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.457 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000004e0 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/158.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.457 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000416 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/057.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.457 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000408 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/050.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.457 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000003a6 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/001.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.457 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000003e4 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/032.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.457 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000488 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/114.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.457 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000474 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/104.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.457 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000450 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/086.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.457 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000532 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/199.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.457 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000003e6 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/033.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.457 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000526 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/193.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.457 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000004ca (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/147.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.457 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000530 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/198.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.458 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x1000000052e (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/197.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.458 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x1000000043a (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/075.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.458 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000004be (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/141.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.458 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000003dc (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/028.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.458 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000536 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/201.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.458 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000462 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/095.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.458 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x1000000041a (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/059.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.458 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x1000000049a (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/123.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.458 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000632 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/cifs/001.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.458 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x1000000045c (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/092.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.458 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000681 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/037.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.458 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000689 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/041.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.458 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x1000000067b (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/034.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.458 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x1000000065b (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/018.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.459 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x1000000067d (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/035.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.459 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x1000000063f (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/004.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.459 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000647 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/008.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.459 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x1000000064d (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/011.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.459 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000641 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/005.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.459 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x1000000065d (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/019.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.459 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x1000000064f (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/012.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.459 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000657 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/016.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.459 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x1000000063d (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/003.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.459 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000006b7 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/304.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.459 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000687 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/040.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.459 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000006bd (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/307.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.459 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x1000000064b (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/010.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.459 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000669 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/025.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.459 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x1000000066b (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/026.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.460 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000006b5 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/303.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.460 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000653 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/014.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.460 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000006b9 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/305.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.460 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x1000000068b (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/042.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.460 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000667 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/024.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.460 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000665 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/023.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.460 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000645 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/007.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.460 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000006b1 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/301.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.460 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x1000000067f (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/036.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.460 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000663 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/022.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.460 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000006bf (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/308.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.460 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000643 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/006.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.460 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000649 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/009.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.460 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000671 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/029.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.461 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000006af (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/271.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.461 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000659 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/017.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.461 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000636 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/001.cfg) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.461 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000675 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/031.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.461 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000661 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/021.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.461 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000655 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/015.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.461 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000683 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/038.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.461 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000685 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/039.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.461 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000639 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/001.out.nozero) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.461 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000673 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/030.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.461 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x1000000065f (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/020.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.461 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x1000000066d (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/027.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.461 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000651 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/013.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.461 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000637 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/001.out.default) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.461 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000006b3 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/302.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.462 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000677 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/032.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.462 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000638 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/001.out.nodelalloc) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.462 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x10000000679 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/033.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.462 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x1000000066f (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/028.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.462 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: Scrub error on inode 0x100000006bb (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/306.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.462 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:22 smithi111 ceph-mon[31085]: scrub summary: idle+waiting paths [/] 2024-04-21T23:20:22.462 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000004c6 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/145.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.462 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x1000000050c (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/180.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.462 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x1000000040c (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/052.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.462 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000003f8 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/042.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.462 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000472 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/103.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.462 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000004d2 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/151.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.462 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000454 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/088.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.462 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000004ac (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/132.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.463 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000003c0 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/014.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.463 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000004cc (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/148.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.463 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000003d2 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/023.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.463 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x1000000047c (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/108.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.463 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000430 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/070.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.463 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000004ee (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/165.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.463 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000448 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/082.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.463 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000496 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/121.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.463 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x1000000042c (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/068.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.463 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000003b2 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/007.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.463 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000420 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/062.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.463 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000004ec (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/164.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.463 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000494 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/120.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.463 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000004a8 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/130.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.463 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000428 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/066.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.464 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x1000000044e (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/085.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.464 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000410 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/054.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.464 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000004ae (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/133.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.464 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x1000000040e (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/053.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.464 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000004d6 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/153.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.464 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x1000000041e (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/061.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.464 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000442 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/079.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.464 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x1000000044c (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/084.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.464 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000444 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/080.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.464 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000004aa (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/131.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.464 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000004b0 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/134.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.464 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000003a8 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/002.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.464 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000452 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/087.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.464 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x1000000046e (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/101.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.465 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000004c2 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/143.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.465 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000520 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/190.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.465 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000003ec (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/036.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.465 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000004dc (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/156.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.465 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000504 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/176.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.465 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000003d0 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/022.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.465 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000004da (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/155.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.465 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000476 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/105.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.465 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000004e2 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/159.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.465 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000003b6 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/009.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.465 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x1000000048c (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/116.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.465 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000522 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/191.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.465 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000004e6 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/161.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.465 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000458 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/090.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.466 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000004f0 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/166.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.466 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000482 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/111.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.466 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000003c6 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/017.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.466 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000440 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/078.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.466 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000418 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/058.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.466 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000402 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/047.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.466 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000460 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/094.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.466 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000003e2 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/031.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.466 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000004c4 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/144.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.466 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000528 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/194.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.466 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000004f4 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/168.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.466 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000003ce (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/021.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.466 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000003c2 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/015.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x1000000052a (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/195.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000426 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/065.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x1000000046c (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/100.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000003f2 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/039.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000003f0 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/038.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000003e0 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/030.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000004a2 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/127.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000524 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/192.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000508 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/178.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000516 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/185.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000470 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/102.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x1000000052c (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/196.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x1000000047e (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/109.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x1000000047a (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/107.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000456 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/089.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000003cc (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/020.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x1000000042e (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/069.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000502 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/175.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x1000000051a (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/187.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000446 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/081.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000438 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/074.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x1000000040a (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/051.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000003da (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/027.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000518 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/186.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000422 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/063.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000414 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/056.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000004f2 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/167.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000538 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/202.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.469 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000004bc (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/140.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.469 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000004b8 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/138.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.469 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000003be (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/013.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.469 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x1000000045e (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/093.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.469 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000514 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/184.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.469 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000434 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/072.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.469 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000004ce (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/149.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.469 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000004e4 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/160.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.469 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000412 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/055.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.469 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000490 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/118.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.469 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000004e0 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/158.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.469 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000416 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/057.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.469 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000408 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/050.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.469 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000003a6 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/001.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.470 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000003e4 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/032.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.470 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000488 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/114.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.470 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000474 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/104.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.470 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000450 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/086.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.470 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000532 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/199.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.470 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000003e6 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/033.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.470 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000526 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/193.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.470 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000004ca (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/147.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.470 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000530 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/198.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.470 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x1000000052e (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/197.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.470 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x1000000043a (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/075.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.470 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000004be (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/141.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.470 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000003dc (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/028.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.471 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000536 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/201.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.471 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000462 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/095.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.471 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x1000000041a (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/059.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.471 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x1000000049a (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/123.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.471 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000632 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/cifs/001.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.471 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x1000000045c (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/092.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.471 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000681 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/037.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.471 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000689 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/041.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.471 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x1000000067b (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/034.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.471 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x1000000065b (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/018.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.471 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x1000000067d (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/035.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.471 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x1000000063f (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/004.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.471 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000647 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/008.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.472 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x1000000064d (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/011.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.472 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000641 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/005.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.472 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x1000000065d (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/019.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.472 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x1000000064f (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/012.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.472 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000657 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/016.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.472 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x1000000063d (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/003.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.472 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000006b7 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/304.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.472 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000687 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/040.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.472 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000006bd (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/307.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.472 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x1000000064b (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/010.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.472 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000669 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/025.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.472 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x1000000066b (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/026.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.472 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000006b5 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/303.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.472 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000653 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/014.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.473 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000006b9 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/305.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.473 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x1000000068b (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/042.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.473 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000667 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/024.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.473 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000665 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/023.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.473 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000645 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/007.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.473 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000006b1 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/301.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.473 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x1000000067f (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/036.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.473 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000663 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/022.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.473 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000006bf (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/308.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.473 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000643 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/006.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.473 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000649 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/009.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.473 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000671 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/029.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.473 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000006af (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/271.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.473 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000659 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/017.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.473 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000636 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/001.cfg) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.473 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000675 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/031.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.474 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000661 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/021.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.474 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000655 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/015.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.474 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000683 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/038.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.474 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000685 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/039.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.474 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000639 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/001.out.nozero) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.474 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000673 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/030.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.474 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x1000000065f (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/020.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.474 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x1000000066d (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/027.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.474 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000651 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/013.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.474 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000637 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/001.out.default) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.474 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000006b3 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/302.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.474 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000677 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/032.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.474 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000638 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/001.out.nodelalloc) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.474 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x10000000679 (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/033.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.475 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x1000000066f (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/028.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.475 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: Scrub error on inode 0x100000006bb (/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/306.out) see mds.h log and `damage ls` output for details 2024-04-21T23:20:22.475 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:22 smithi174 ceph-mon[31409]: scrub summary: idle+waiting paths [/] 2024-04-21T23:20:23.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:23 smithi040 ceph-mon[27984]: pgmap v62: 129 pgs: 129 active+clean; 86 MiB data, 711 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 4.2 MiB/s wr, 560 op/s 2024-04-21T23:20:23.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:23 smithi174 ceph-mon[31409]: pgmap v62: 129 pgs: 129 active+clean; 86 MiB data, 711 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 4.2 MiB/s wr, 560 op/s 2024-04-21T23:20:23.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:23 smithi111 ceph-mon[31085]: pgmap v62: 129 pgs: 129 active+clean; 86 MiB data, 711 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 4.2 MiB/s wr, 560 op/s 2024-04-21T23:20:23.470 INFO:tasks.workunit.client.0.smithi040.stdout:autoheader 2024-04-21T23:20:23.922 INFO:tasks.workunit.client.0.smithi040.stdout:autoconf 2024-04-21T23:20:24.249 INFO:tasks.workunit.client.0.smithi040.stdout:./configure \ 2024-04-21T23:20:24.249 INFO:tasks.workunit.client.0.smithi040.stdout: --libexecdir=/usr/lib \ 2024-04-21T23:20:24.249 INFO:tasks.workunit.client.0.smithi040.stdout: --exec_prefix=/var/lib 2024-04-21T23:20:24.402 INFO:tasks.workunit.client.0.smithi040.stdout:checking build system type... x86_64-pc-linux-gnu 2024-04-21T23:20:24.402 INFO:tasks.workunit.client.0.smithi040.stdout:checking host system type... x86_64-pc-linux-gnu 2024-04-21T23:20:24.403 INFO:tasks.workunit.client.0.smithi040.stdout:checking how to print strings... printf 2024-04-21T23:20:24.403 INFO:tasks.workunit.client.0.smithi040.stdout:checking for gcc... gcc 2024-04-21T23:20:24.506 INFO:tasks.workunit.client.0.smithi040.stdout:checking whether the C compiler works... yes 2024-04-21T23:20:24.506 INFO:tasks.workunit.client.0.smithi040.stdout:checking for C compiler default output file name... a.out 2024-04-21T23:20:24.556 INFO:tasks.workunit.client.0.smithi040.stdout:checking for suffix of executables... 2024-04-21T23:20:24.600 INFO:tasks.workunit.client.0.smithi040.stdout:checking whether we are cross compiling... no 2024-04-21T23:20:24.644 INFO:tasks.workunit.client.0.smithi040.stdout:checking for suffix of object files... o 2024-04-21T23:20:24.676 INFO:tasks.workunit.client.0.smithi040.stdout:checking whether we are using the GNU C compiler... yes 2024-04-21T23:20:24.709 INFO:tasks.workunit.client.0.smithi040.stdout:checking whether gcc accepts -g... yes 2024-04-21T23:20:24.796 INFO:tasks.workunit.client.0.smithi040.stdout:checking for gcc option to accept ISO C89... none needed 2024-04-21T23:20:24.800 INFO:tasks.workunit.client.0.smithi040.stdout:checking for a sed that does not truncate output... /usr/bin/sed 2024-04-21T23:20:24.805 INFO:tasks.workunit.client.0.smithi040.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2024-04-21T23:20:24.806 INFO:tasks.workunit.client.0.smithi040.stdout:checking for egrep... /usr/bin/grep -E 2024-04-21T23:20:24.807 INFO:tasks.workunit.client.0.smithi040.stdout:checking for fgrep... /usr/bin/grep -F 2024-04-21T23:20:24.818 INFO:tasks.workunit.client.0.smithi040.stdout:checking for ld used by gcc... /usr/bin/ld 2024-04-21T23:20:24.819 INFO:tasks.workunit.client.0.smithi040.stdout:checking if the linker (/usr/bin/ld) is GNU ld... yes 2024-04-21T23:20:24.826 INFO:tasks.workunit.client.0.smithi040.stdout:checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B 2024-04-21T23:20:25.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:25 smithi040 ceph-mon[27984]: pgmap v63: 129 pgs: 129 active+clean; 86 MiB data, 711 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 4.2 MiB/s wr, 560 op/s 2024-04-21T23:20:25.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:25 smithi174 ceph-mon[31409]: pgmap v63: 129 pgs: 129 active+clean; 86 MiB data, 711 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 4.2 MiB/s wr, 560 op/s 2024-04-21T23:20:25.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:25 smithi111 ceph-mon[31085]: pgmap v63: 129 pgs: 129 active+clean; 86 MiB data, 711 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 4.2 MiB/s wr, 560 op/s 2024-04-21T23:20:25.782 INFO:tasks.workunit.client.0.smithi040.stdout:checking the name lister (/usr/bin/nm -B) interface... BSD nm 2024-04-21T23:20:25.782 INFO:tasks.workunit.client.0.smithi040.stdout:checking whether ln -s works... yes 2024-04-21T23:20:25.799 INFO:tasks.workunit.client.0.smithi040.stdout:checking the maximum length of command line arguments... 1572864 2024-04-21T23:20:25.805 INFO:tasks.workunit.client.0.smithi040.stdout:checking how to convert x86_64-pc-linux-gnu file names to x86_64-pc-linux-gnu format... func_convert_file_noop 2024-04-21T23:20:25.805 INFO:tasks.workunit.client.0.smithi040.stdout:checking how to convert x86_64-pc-linux-gnu file names to toolchain format... func_convert_file_noop 2024-04-21T23:20:25.805 INFO:tasks.workunit.client.0.smithi040.stdout:checking for /usr/bin/ld option to reload object files... -r 2024-04-21T23:20:25.805 INFO:tasks.workunit.client.0.smithi040.stdout:checking for objdump... objdump 2024-04-21T23:20:25.805 INFO:tasks.workunit.client.0.smithi040.stdout:checking how to recognize dependent libraries... pass_all 2024-04-21T23:20:25.805 INFO:tasks.workunit.client.0.smithi040.stdout:checking for dlltool... no 2024-04-21T23:20:25.805 INFO:tasks.workunit.client.0.smithi040.stdout:checking how to associate runtime and link libraries... printf %s\n 2024-04-21T23:20:25.806 INFO:tasks.workunit.client.0.smithi040.stdout:checking for ar... ar 2024-04-21T23:20:25.914 INFO:tasks.workunit.client.0.smithi040.stdout:checking for archiver @FILE support... @ 2024-04-21T23:20:25.915 INFO:tasks.workunit.client.0.smithi040.stdout:checking for strip... strip 2024-04-21T23:20:25.915 INFO:tasks.workunit.client.0.smithi040.stdout:checking for ranlib... ranlib 2024-04-21T23:20:25.915 INFO:tasks.workunit.client.0.smithi040.stdout:checking for gawk... gawk 2024-04-21T23:20:26.039 INFO:tasks.workunit.client.0.smithi040.stdout:checking command to parse /usr/bin/nm -B output from gcc object... ok 2024-04-21T23:20:26.046 INFO:tasks.workunit.client.0.smithi040.stdout:checking for sysroot... no 2024-04-21T23:20:26.064 INFO:tasks.workunit.client.0.smithi040.stdout:checking for a working dd... /usr/bin/dd 2024-04-21T23:20:26.075 INFO:tasks.workunit.client.0.smithi040.stdout:checking how to truncate binary pipes... /usr/bin/dd bs=4096 count=1 2024-04-21T23:20:26.111 INFO:tasks.workunit.client.0.smithi040.stdout:checking for mt... no 2024-04-21T23:20:26.125 INFO:tasks.workunit.client.0.smithi040.stdout:checking if : is a manifest tool... no 2024-04-21T23:20:26.226 INFO:tasks.workunit.client.0.smithi040.stdout:checking how to run the C preprocessor... gcc -E 2024-04-21T23:20:26.443 INFO:tasks.workunit.client.0.smithi040.stdout:checking for ANSI C header files... yes 2024-04-21T23:20:26.486 INFO:tasks.workunit.client.0.smithi040.stdout:checking for sys/types.h... yes 2024-04-21T23:20:26.533 INFO:tasks.workunit.client.0.smithi040.stdout:checking for sys/stat.h... yes 2024-04-21T23:20:26.586 INFO:tasks.workunit.client.0.smithi040.stdout:checking for stdlib.h... yes 2024-04-21T23:20:26.636 INFO:tasks.workunit.client.0.smithi040.stdout:checking for string.h... yes 2024-04-21T23:20:26.684 INFO:tasks.workunit.client.0.smithi040.stdout:checking for memory.h... yes 2024-04-21T23:20:26.734 INFO:tasks.workunit.client.0.smithi040.stdout:checking for strings.h... yes 2024-04-21T23:20:26.785 INFO:tasks.workunit.client.0.smithi040.stdout:checking for inttypes.h... yes 2024-04-21T23:20:26.832 INFO:tasks.workunit.client.0.smithi040.stdout:checking for stdint.h... yes 2024-04-21T23:20:26.882 INFO:tasks.workunit.client.0.smithi040.stdout:checking for unistd.h... yes 2024-04-21T23:20:26.936 INFO:tasks.workunit.client.0.smithi040.stdout:checking for dlfcn.h... yes 2024-04-21T23:20:26.942 INFO:tasks.workunit.client.0.smithi040.stdout:checking for objdir... .libs 2024-04-21T23:20:27.085 INFO:tasks.workunit.client.0.smithi040.stdout:checking if gcc supports -fno-rtti -fno-exceptions... no 2024-04-21T23:20:27.086 INFO:tasks.workunit.client.0.smithi040.stdout:checking for gcc option to produce PIC... -fPIC -DPIC 2024-04-21T23:20:27.134 INFO:tasks.workunit.client.0.smithi040.stdout:checking if gcc PIC flag -fPIC -DPIC works... yes 2024-04-21T23:20:27.194 INFO:tasks.workunit.client.0.smithi040.stdout:checking if gcc static flag -static works... no 2024-04-21T23:20:27.263 INFO:tasks.workunit.client.0.smithi040.stdout:checking if gcc supports -c -o file.o... yes 2024-04-21T23:20:27.263 INFO:tasks.workunit.client.0.smithi040.stdout:checking if gcc supports -c -o file.o... (cached) yes 2024-04-21T23:20:27.269 INFO:tasks.workunit.client.0.smithi040.stdout:checking whether the gcc linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes 2024-04-21T23:20:27.326 INFO:tasks.workunit.client.0.smithi040.stdout:checking whether -lc should be explicitly linked in... no 2024-04-21T23:20:27.404 INFO:tasks.workunit.client.0.smithi040.stdout:checking dynamic linker characteristics... GNU/Linux ld.so 2024-04-21T23:20:27.405 INFO:tasks.workunit.client.0.smithi040.stdout:checking how to hardcode library paths into programs... immediate 2024-04-21T23:20:27.406 INFO:tasks.workunit.client.0.smithi040.stdout:checking whether stripping libraries is possible... yes 2024-04-21T23:20:27.406 INFO:tasks.workunit.client.0.smithi040.stdout:checking if libtool supports shared libraries... yes 2024-04-21T23:20:27.406 INFO:tasks.workunit.client.0.smithi040.stdout:checking whether to build shared libraries... yes 2024-04-21T23:20:27.406 INFO:tasks.workunit.client.0.smithi040.stdout:checking whether to build static libraries... yes 2024-04-21T23:20:27.412 INFO:tasks.workunit.client.0.smithi040.stdout:checking for gcc... (cached) gcc 2024-04-21T23:20:27.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:27 smithi040 ceph-mon[27984]: pgmap v64: 129 pgs: 129 active+clean; 90 MiB data, 767 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.6 MiB/s wr, 861 op/s 2024-04-21T23:20:27.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:27 smithi174 ceph-mon[31409]: pgmap v64: 129 pgs: 129 active+clean; 90 MiB data, 767 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.6 MiB/s wr, 861 op/s 2024-04-21T23:20:27.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:27 smithi111 ceph-mon[31085]: pgmap v64: 129 pgs: 129 active+clean; 90 MiB data, 767 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.6 MiB/s wr, 861 op/s 2024-04-21T23:20:27.468 INFO:tasks.workunit.client.0.smithi040.stdout:checking whether we are using the GNU C compiler... (cached) yes 2024-04-21T23:20:27.468 INFO:tasks.workunit.client.0.smithi040.stdout:checking whether gcc accepts -g... (cached) yes 2024-04-21T23:20:27.468 INFO:tasks.workunit.client.0.smithi040.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2024-04-21T23:20:27.469 INFO:tasks.workunit.client.0.smithi040.stdout:checking for gmake... /usr/bin/gmake 2024-04-21T23:20:27.469 INFO:tasks.workunit.client.0.smithi040.stderr:./configure: line 11968: test: $(SHELL): binary operator expected 2024-04-21T23:20:27.470 INFO:tasks.workunit.client.0.smithi040.stdout:checking for tar... /bin/tar 2024-04-21T23:20:27.470 INFO:tasks.workunit.client.0.smithi040.stdout:checking for gzip... /bin/gzip 2024-04-21T23:20:27.501 INFO:tasks.workunit.client.0.smithi040.stdout:checking whether gcc -MM is supported... yes 2024-04-21T23:20:27.501 INFO:tasks.workunit.client.0.smithi040.stdout:checking for sort... /bin/sort 2024-04-21T23:20:27.501 INFO:tasks.workunit.client.0.smithi040.stdout:checking whether ln -s works... yes 2024-04-21T23:20:27.501 INFO:tasks.workunit.client.0.smithi040.stdout:checking for rpm... /bin/rpm 2024-04-21T23:20:27.507 INFO:tasks.workunit.client.0.smithi040.stdout:checking for rpmbuild... no 2024-04-21T23:20:27.507 INFO:tasks.workunit.client.0.smithi040.stdout:checking for ANSI C header files... (cached) yes 2024-04-21T23:20:27.553 INFO:tasks.workunit.client.0.smithi040.stdout:checking assert.h usability... yes 2024-04-21T23:20:27.571 INFO:tasks.workunit.client.0.smithi040.stdout:checking assert.h presence... yes 2024-04-21T23:20:27.571 INFO:tasks.workunit.client.0.smithi040.stdout:checking for assert.h... yes 2024-04-21T23:20:27.632 INFO:tasks.workunit.client.0.smithi040.stdout:checking bstring.h usability... no 2024-04-21T23:20:27.666 INFO:tasks.workunit.client.0.smithi040.stdout:checking bstring.h presence... no 2024-04-21T23:20:27.666 INFO:tasks.workunit.client.0.smithi040.stdout:checking for bstring.h... no 2024-04-21T23:20:27.710 INFO:tasks.workunit.client.0.smithi040.stdout:checking libgen.h usability... yes 2024-04-21T23:20:27.727 INFO:tasks.workunit.client.0.smithi040.stdout:checking libgen.h presence... yes 2024-04-21T23:20:27.727 INFO:tasks.workunit.client.0.smithi040.stdout:checking for libgen.h... yes 2024-04-21T23:20:27.787 INFO:tasks.workunit.client.0.smithi040.stdout:checking dirent.h usability... yes 2024-04-21T23:20:27.805 INFO:tasks.workunit.client.0.smithi040.stdout:checking dirent.h presence... yes 2024-04-21T23:20:27.806 INFO:tasks.workunit.client.0.smithi040.stdout:checking for dirent.h... yes 2024-04-21T23:20:27.857 INFO:tasks.workunit.client.0.smithi040.stdout:checking errno.h usability... yes 2024-04-21T23:20:27.874 INFO:tasks.workunit.client.0.smithi040.stdout:checking errno.h presence... yes 2024-04-21T23:20:27.874 INFO:tasks.workunit.client.0.smithi040.stdout:checking for errno.h... yes 2024-04-21T23:20:27.928 INFO:tasks.workunit.client.0.smithi040.stdout:checking malloc.h usability... yes 2024-04-21T23:20:27.947 INFO:tasks.workunit.client.0.smithi040.stdout:checking malloc.h presence... yes 2024-04-21T23:20:27.948 INFO:tasks.workunit.client.0.smithi040.stdout:checking for malloc.h... yes 2024-04-21T23:20:28.009 INFO:tasks.workunit.client.0.smithi040.stdout:checking uuid.h usability... no 2024-04-21T23:20:28.039 INFO:tasks.workunit.client.0.smithi040.stdout:checking uuid.h presence... no 2024-04-21T23:20:28.039 INFO:tasks.workunit.client.0.smithi040.stdout:checking for uuid.h... no 2024-04-21T23:20:28.091 INFO:tasks.workunit.client.0.smithi040.stdout:checking uuid/uuid.h usability... yes 2024-04-21T23:20:28.110 INFO:tasks.workunit.client.0.smithi040.stdout:checking uuid/uuid.h presence... yes 2024-04-21T23:20:28.111 INFO:tasks.workunit.client.0.smithi040.stdout:checking for uuid/uuid.h... yes 2024-04-21T23:20:28.167 INFO:tasks.workunit.client.0.smithi040.stdout:checking sys/uuid.h usability... no 2024-04-21T23:20:28.197 INFO:tasks.workunit.client.0.smithi040.stdout:checking sys/uuid.h presence... no 2024-04-21T23:20:28.197 INFO:tasks.workunit.client.0.smithi040.stdout:checking for sys/uuid.h... no 2024-04-21T23:20:28.248 INFO:tasks.workunit.client.0.smithi040.stdout:checking sys/file.h usability... yes 2024-04-21T23:20:28.267 INFO:tasks.workunit.client.0.smithi040.stdout:checking sys/file.h presence... yes 2024-04-21T23:20:28.267 INFO:tasks.workunit.client.0.smithi040.stdout:checking for sys/file.h... yes 2024-04-21T23:20:28.323 INFO:tasks.workunit.client.0.smithi040.stdout:checking sys/fcntl.h usability... yes 2024-04-21T23:20:28.341 INFO:tasks.workunit.client.0.smithi040.stdout:checking sys/fcntl.h presence... yes 2024-04-21T23:20:28.341 INFO:tasks.workunit.client.0.smithi040.stdout:checking for sys/fcntl.h... yes 2024-04-21T23:20:28.428 INFO:tasks.workunit.client.0.smithi040.stdout:checking sys/syssgi.h usability... no 2024-04-21T23:20:28.459 INFO:tasks.workunit.client.0.smithi040.stdout:checking sys/syssgi.h presence... no 2024-04-21T23:20:28.459 INFO:tasks.workunit.client.0.smithi040.stdout:checking for sys/syssgi.h... no 2024-04-21T23:20:28.534 INFO:tasks.workunit.client.0.smithi040.stdout:checking sys/param.h usability... yes 2024-04-21T23:20:28.554 INFO:tasks.workunit.client.0.smithi040.stdout:checking sys/param.h presence... yes 2024-04-21T23:20:28.555 INFO:tasks.workunit.client.0.smithi040.stdout:checking for sys/param.h... yes 2024-04-21T23:20:28.569 INFO:tasks.workunit.client.0.smithi040.stdout:checking for sys/stat.h... (cached) yes 2024-04-21T23:20:28.612 INFO:tasks.workunit.client.0.smithi040.stdout:checking sys/statvfs.h usability... yes 2024-04-21T23:20:28.631 INFO:tasks.workunit.client.0.smithi040.stdout:checking sys/statvfs.h presence... yes 2024-04-21T23:20:28.631 INFO:tasks.workunit.client.0.smithi040.stdout:checking for sys/statvfs.h... yes 2024-04-21T23:20:28.688 INFO:tasks.workunit.client.0.smithi040.stdout:checking sys/time.h usability... yes 2024-04-21T23:20:28.705 INFO:tasks.workunit.client.0.smithi040.stdout:checking sys/time.h presence... yes 2024-04-21T23:20:28.705 INFO:tasks.workunit.client.0.smithi040.stdout:checking for sys/time.h... yes 2024-04-21T23:20:28.760 INFO:tasks.workunit.client.0.smithi040.stdout:checking sys/ioctl.h usability... yes 2024-04-21T23:20:28.800 INFO:tasks.workunit.client.0.smithi040.stdout:checking sys/ioctl.h presence... yes 2024-04-21T23:20:28.801 INFO:tasks.workunit.client.0.smithi040.stdout:checking for sys/ioctl.h... yes 2024-04-21T23:20:28.871 INFO:tasks.workunit.client.0.smithi040.stdout:checking sys/wait.h usability... yes 2024-04-21T23:20:28.895 INFO:tasks.workunit.client.0.smithi040.stdout:checking sys/wait.h presence... yes 2024-04-21T23:20:28.895 INFO:tasks.workunit.client.0.smithi040.stdout:checking for sys/wait.h... yes 2024-04-21T23:20:28.910 INFO:tasks.workunit.client.0.smithi040.stdout:checking for sys/types.h... (cached) yes 2024-04-21T23:20:28.915 INFO:tasks.workunit.client.0.smithi040.stdout:checking for strings.h... (cached) yes 2024-04-21T23:20:28.962 INFO:tasks.workunit.client.0.smithi040.stdout:checking err.h usability... yes 2024-04-21T23:20:28.984 INFO:tasks.workunit.client.0.smithi040.stdout:checking err.h presence... yes 2024-04-21T23:20:28.984 INFO:tasks.workunit.client.0.smithi040.stdout:checking for err.h... yes 2024-04-21T23:20:29.035 INFO:tasks.workunit.client.0.smithi040.stdout:checking linux/falloc.h usability... yes 2024-04-21T23:20:29.052 INFO:tasks.workunit.client.0.smithi040.stdout:checking linux/falloc.h presence... yes 2024-04-21T23:20:29.052 INFO:tasks.workunit.client.0.smithi040.stdout:checking for linux/falloc.h... yes 2024-04-21T23:20:29.110 INFO:tasks.workunit.client.0.smithi040.stdout:checking sys/fs/xfs_fsops.h usability... no 2024-04-21T23:20:29.147 INFO:tasks.workunit.client.0.smithi040.stdout:checking sys/fs/xfs_fsops.h presence... no 2024-04-21T23:20:29.147 INFO:tasks.workunit.client.0.smithi040.stdout:checking for sys/fs/xfs_fsops.h... no 2024-04-21T23:20:29.205 INFO:tasks.workunit.client.0.smithi040.stdout:checking sys/fs/xfs_itable.h usability... no 2024-04-21T23:20:29.241 INFO:tasks.workunit.client.0.smithi040.stdout:checking sys/fs/xfs_itable.h presence... no 2024-04-21T23:20:29.241 INFO:tasks.workunit.client.0.smithi040.stdout:checking for sys/fs/xfs_itable.h... no 2024-04-21T23:20:29.295 INFO:tasks.workunit.client.0.smithi040.stdout:checking xfs/platform_defs.h usability... no 2024-04-21T23:20:29.328 INFO:tasks.workunit.client.0.smithi040.stdout:checking xfs/platform_defs.h presence... no 2024-04-21T23:20:29.329 INFO:tasks.workunit.client.0.smithi040.stdout:checking for xfs/platform_defs.h... no 2024-04-21T23:20:29.380 INFO:tasks.workunit.client.0.smithi040.stdout:checking btrfs/ioctl.h usability... no 2024-04-21T23:20:29.410 INFO:tasks.workunit.client.0.smithi040.stdout:checking btrfs/ioctl.h presence... no 2024-04-21T23:20:29.410 INFO:tasks.workunit.client.0.smithi040.stdout:checking for btrfs/ioctl.h... no 2024-04-21T23:20:29.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:29 smithi040 ceph-mon[27984]: pgmap v65: 129 pgs: 129 active+clean; 90 MiB data, 767 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.5 MiB/s wr, 469 op/s 2024-04-21T23:20:29.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:29 smithi174 ceph-mon[31409]: pgmap v65: 129 pgs: 129 active+clean; 90 MiB data, 767 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.5 MiB/s wr, 469 op/s 2024-04-21T23:20:29.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:29 smithi111 ceph-mon[31085]: pgmap v65: 129 pgs: 129 active+clean; 90 MiB data, 767 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.5 MiB/s wr, 469 op/s 2024-04-21T23:20:29.460 INFO:tasks.workunit.client.0.smithi040.stdout:checking cifs/ioctl.h usability... no 2024-04-21T23:20:29.491 INFO:tasks.workunit.client.0.smithi040.stdout:checking cifs/ioctl.h presence... no 2024-04-21T23:20:29.491 INFO:tasks.workunit.client.0.smithi040.stdout:checking for cifs/ioctl.h... no 2024-04-21T23:20:29.560 INFO:tasks.workunit.client.0.smithi040.stdout:checking sys/mman.h usability... yes 2024-04-21T23:20:29.578 INFO:tasks.workunit.client.0.smithi040.stdout:checking sys/mman.h presence... yes 2024-04-21T23:20:29.579 INFO:tasks.workunit.client.0.smithi040.stdout:checking for sys/mman.h... yes 2024-04-21T23:20:29.623 INFO:tasks.workunit.client.0.smithi040.stdout:checking for xfs/xfs_log_format.h... no 2024-04-21T23:20:29.686 INFO:tasks.workunit.client.0.smithi040.stdout:checking for uuid_compare... no 2024-04-21T23:20:29.732 INFO:tasks.workunit.client.0.smithi040.stdout:checking for uuid_compare in -luuid... yes 2024-04-21T23:20:29.809 INFO:tasks.workunit.client.0.smithi040.stdout:checking for xfs/xfs.h... yes 2024-04-21T23:20:29.842 INFO:tasks.workunit.client.0.smithi040.stdout:checking for xfs/libxfs.h... no 2024-04-21T23:20:29.876 INFO:tasks.workunit.client.0.smithi040.stdout:checking whether xlog_assign_lsn is declared... no 2024-04-21T23:20:29.921 INFO:tasks.workunit.client.0.smithi040.stdout:checking for xfs/xqm.h... yes 2024-04-21T23:20:29.982 INFO:tasks.workunit.client.0.smithi040.stdout:checking xfsctl from xfs/xfs.h... ok 2024-04-21T23:20:30.032 INFO:tasks.workunit.client.0.smithi040.stdout:checking xfs/handle.h usability... yes 2024-04-21T23:20:30.055 INFO:tasks.workunit.client.0.smithi040.stdout:checking xfs/handle.h presence... yes 2024-04-21T23:20:30.055 INFO:tasks.workunit.client.0.smithi040.stdout:checking for xfs/handle.h... yes 2024-04-21T23:20:30.109 INFO:tasks.workunit.client.0.smithi040.stdout:checking for attr_list_by_handle in -lhandle... yes 2024-04-21T23:20:30.165 INFO:tasks.workunit.client.0.smithi040.stdout:checking attr/attributes.h usability... yes 2024-04-21T23:20:30.183 INFO:tasks.workunit.client.0.smithi040.stdout:checking attr/attributes.h presence... yes 2024-04-21T23:20:30.184 INFO:tasks.workunit.client.0.smithi040.stdout:checking for attr/attributes.h... yes 2024-04-21T23:20:30.239 INFO:tasks.workunit.client.0.smithi040.stdout:checking sys/attributes.h usability... no 2024-04-21T23:20:30.269 INFO:tasks.workunit.client.0.smithi040.stdout:checking sys/attributes.h presence... no 2024-04-21T23:20:30.269 INFO:tasks.workunit.client.0.smithi040.stdout:checking for sys/attributes.h... no 2024-04-21T23:20:30.323 INFO:tasks.workunit.client.0.smithi040.stdout:checking for attr_list in -lattr... yes 2024-04-21T23:20:30.371 INFO:tasks.workunit.client.0.smithi040.stdout:checking for attr_set in -lattr... yes 2024-04-21T23:20:30.421 INFO:tasks.workunit.client.0.smithi040.stdout:checking sys/acl.h usability... yes 2024-04-21T23:20:30.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:30 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:20:30.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:30 smithi040 ceph-mon[27984]: scrub summary: idle 2024-04-21T23:20:30.439 INFO:tasks.workunit.client.0.smithi040.stdout:checking sys/acl.h presence... yes 2024-04-21T23:20:30.439 INFO:tasks.workunit.client.0.smithi040.stdout:checking for sys/acl.h... yes 2024-04-21T23:20:30.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:30 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:20:30.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:30 smithi174 ceph-mon[31409]: scrub summary: idle 2024-04-21T23:20:30.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:30 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:20:30.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:30 smithi111 ceph-mon[31085]: scrub summary: idle 2024-04-21T23:20:30.488 INFO:tasks.workunit.client.0.smithi040.stdout:checking acl/libacl.h usability... yes 2024-04-21T23:20:30.505 INFO:tasks.workunit.client.0.smithi040.stdout:checking acl/libacl.h presence... yes 2024-04-21T23:20:30.505 INFO:tasks.workunit.client.0.smithi040.stdout:checking for acl/libacl.h... yes 2024-04-21T23:20:30.558 INFO:tasks.workunit.client.0.smithi040.stdout:checking for acl_init in -lacl... yes 2024-04-21T23:20:30.600 INFO:tasks.workunit.client.0.smithi040.stdout:checking gdbm-ndbm.h usability... no 2024-04-21T23:20:30.630 INFO:tasks.workunit.client.0.smithi040.stdout:checking gdbm-ndbm.h presence... no 2024-04-21T23:20:30.630 INFO:tasks.workunit.client.0.smithi040.stdout:checking for gdbm-ndbm.h... no 2024-04-21T23:20:30.672 INFO:tasks.workunit.client.0.smithi040.stdout:checking gdbm/ndbm.h usability... no 2024-04-21T23:20:30.707 INFO:tasks.workunit.client.0.smithi040.stdout:checking gdbm/ndbm.h presence... no 2024-04-21T23:20:30.707 INFO:tasks.workunit.client.0.smithi040.stdout:checking for gdbm/ndbm.h... no 2024-04-21T23:20:30.749 INFO:tasks.workunit.client.0.smithi040.stdout:checking ndbm.h usability... no 2024-04-21T23:20:30.780 INFO:tasks.workunit.client.0.smithi040.stdout:checking ndbm.h presence... no 2024-04-21T23:20:30.780 INFO:tasks.workunit.client.0.smithi040.stdout:checking for ndbm.h... no 2024-04-21T23:20:30.820 INFO:tasks.workunit.client.0.smithi040.stdout:checking libaio.h usability... yes 2024-04-21T23:20:30.840 INFO:tasks.workunit.client.0.smithi040.stdout:checking libaio.h presence... yes 2024-04-21T23:20:30.840 INFO:tasks.workunit.client.0.smithi040.stdout:checking for libaio.h... yes 2024-04-21T23:20:30.895 INFO:tasks.workunit.client.0.smithi040.stdout:checking for dm_make_handle in -ldm... no 2024-04-21T23:20:30.896 INFO:tasks.workunit.client.0.smithi040.stdout: 2024-04-21T23:20:30.896 INFO:tasks.workunit.client.0.smithi040.stdout:WARNING: could not find a valid DMAPI base library. 2024-04-21T23:20:30.896 INFO:tasks.workunit.client.0.smithi040.stdout:If you want DMAPI support please install the data migration 2024-04-21T23:20:30.896 INFO:tasks.workunit.client.0.smithi040.stdout:API (dmapi) library package. Alternatively, run "make install" 2024-04-21T23:20:30.896 INFO:tasks.workunit.client.0.smithi040.stdout:from the dmapi source. 2024-04-21T23:20:30.896 INFO:tasks.workunit.client.0.smithi040.stdout: 2024-04-21T23:20:30.934 INFO:tasks.workunit.client.0.smithi040.stdout:checking linux/fiemap.h usability... yes 2024-04-21T23:20:30.952 INFO:tasks.workunit.client.0.smithi040.stdout:checking linux/fiemap.h presence... yes 2024-04-21T23:20:30.952 INFO:tasks.workunit.client.0.smithi040.stdout:checking for linux/fiemap.h... yes 2024-04-21T23:20:31.000 INFO:tasks.workunit.client.0.smithi040.stdout:checking for fallocate... yes 2024-04-21T23:20:31.060 INFO:tasks.workunit.client.0.smithi040.stdout:checking for open_by_handle_at... yes 2024-04-21T23:20:31.117 INFO:tasks.workunit.client.0.smithi040.stdout:checking sys/prctl.h usability... yes 2024-04-21T23:20:31.134 INFO:tasks.workunit.client.0.smithi040.stdout:checking sys/prctl.h presence... yes 2024-04-21T23:20:31.134 INFO:tasks.workunit.client.0.smithi040.stdout:checking for sys/prctl.h... yes 2024-04-21T23:20:31.180 INFO:tasks.workunit.client.0.smithi040.stdout:checking linux/fs.h usability... yes 2024-04-21T23:20:31.197 INFO:tasks.workunit.client.0.smithi040.stdout:checking linux/fs.h presence... yes 2024-04-21T23:20:31.197 INFO:tasks.workunit.client.0.smithi040.stdout:checking for linux/fs.h... yes 2024-04-21T23:20:31.244 INFO:tasks.workunit.client.0.smithi040.stdout:checking btrfsutil.h usability... no 2024-04-21T23:20:31.275 INFO:tasks.workunit.client.0.smithi040.stdout:checking btrfsutil.h presence... no 2024-04-21T23:20:31.275 INFO:tasks.workunit.client.0.smithi040.stdout:checking for btrfsutil.h... no 2024-04-21T23:20:31.323 INFO:tasks.workunit.client.0.smithi040.stdout:checking for copy_file_range... yes 2024-04-21T23:20:31.378 INFO:tasks.workunit.client.0.smithi040.stdout:checking for renameat2... yes 2024-04-21T23:20:31.401 INFO:tasks.workunit.client.0.smithi040.stdout:configure: creating ./config.status 2024-04-21T23:20:31.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:31 smithi040 ceph-mon[27984]: pgmap v66: 129 pgs: 129 active+clean; 93 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.7 MiB/s wr, 488 op/s 2024-04-21T23:20:31.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:31 smithi040 ceph-mon[27984]: scrub summary: idle 2024-04-21T23:20:31.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:31 smithi040 ceph-mon[27984]: scrub summary: idle 2024-04-21T23:20:31.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:31 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:20:31.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:31 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:20:31.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:31 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:20:31.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:31 smithi174 ceph-mon[31409]: pgmap v66: 129 pgs: 129 active+clean; 93 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.7 MiB/s wr, 488 op/s 2024-04-21T23:20:31.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:31 smithi174 ceph-mon[31409]: scrub summary: idle 2024-04-21T23:20:31.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:31 smithi174 ceph-mon[31409]: scrub summary: idle 2024-04-21T23:20:31.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:31 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:20:31.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:31 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:20:31.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:31 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:20:31.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:31 smithi111 ceph-mon[31085]: pgmap v66: 129 pgs: 129 active+clean; 93 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.7 MiB/s wr, 488 op/s 2024-04-21T23:20:31.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:31 smithi111 ceph-mon[31085]: scrub summary: idle 2024-04-21T23:20:31.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:31 smithi111 ceph-mon[31085]: scrub summary: idle 2024-04-21T23:20:31.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:31 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:20:31.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:31 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:20:31.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:31 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:20:31.737 INFO:tasks.workunit.client.0.smithi040.stdout:config.status: creating include/builddefs 2024-04-21T23:20:31.751 INFO:tasks.workunit.client.0.smithi040.stdout:config.status: creating include/config.h 2024-04-21T23:20:31.763 INFO:tasks.workunit.client.0.smithi040.stdout:config.status: executing libtool commands 2024-04-21T23:20:31.845 INFO:tasks.workunit.client.0.smithi040.stdout:Building include 2024-04-21T23:20:31.846 INFO:tasks.workunit.client.0.smithi040.stdout:Building lib 2024-04-21T23:20:31.846 INFO:tasks.workunit.client.0.smithi040.stdout:Building tests 2024-04-21T23:20:31.850 INFO:tasks.workunit.client.0.smithi040.stdout: [LTDEP] 2024-04-21T23:20:32.020 INFO:tasks.workunit.client.0.smithi040.stdout: [CC] dataascii.lo 2024-04-21T23:20:32.021 INFO:tasks.workunit.client.0.smithi040.stdout: [CC] databin.lo 2024-04-21T23:20:32.021 INFO:tasks.workunit.client.0.smithi040.stdout: [CC] datapid.lo 2024-04-21T23:20:32.021 INFO:tasks.workunit.client.0.smithi040.stdout: [CC] file_lock.lo 2024-04-21T23:20:32.127 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.128+0000 7f8a113be640 1 -- 172.21.15.40:0/1459856554 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a0c1032f0 msgr2=0x7f8a0c10b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:32.127 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.128+0000 7f8a113be640 1 --2- 172.21.15.40:0/1459856554 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a0c1032f0 0x7f8a0c10b7d0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f89f4009900 tx=0x7f89f402bcd0 comp rx=0 tx=0).stop 2024-04-21T23:20:32.127 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.128+0000 7f8a113be640 1 -- 172.21.15.40:0/1459856554 shutdown_connections 2024-04-21T23:20:32.128 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.128+0000 7f8a113be640 1 --2- 172.21.15.40:0/1459856554 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a0c10bda0 0x7f8a0c10e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:32.128 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.128+0000 7f8a113be640 1 --2- 172.21.15.40:0/1459856554 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a0c1032f0 0x7f8a0c10b7d0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:32.128 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.128+0000 7f8a113be640 1 --2- 172.21.15.40:0/1459856554 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a0c1029d0 0x7f8a0c102db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:32.128 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.128+0000 7f8a113be640 1 -- 172.21.15.40:0/1459856554 >> 172.21.15.40:0/1459856554 conn(0x7f8a0c077790 msgr2=0x7f8a0c077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:32.128 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.128+0000 7f8a113be640 1 -- 172.21.15.40:0/1459856554 shutdown_connections 2024-04-21T23:20:32.128 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.128+0000 7f8a113be640 1 -- 172.21.15.40:0/1459856554 wait complete. 2024-04-21T23:20:32.130 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.129+0000 7f8a113be640 1 Processor -- start 2024-04-21T23:20:32.130 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.129+0000 7f8a113be640 1 -- start start 2024-04-21T23:20:32.130 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.129+0000 7f8a113be640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a0c1029d0 0x7f8a0c19a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:32.130 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.129+0000 7f8a113be640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a0c1032f0 0x7f8a0c19a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:32.130 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.129+0000 7f8a113be640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a0c10bda0 0x7f8a0c19ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:32.130 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.129+0000 7f8a113be640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8a0c113dc0 con 0x7f8a0c10bda0 2024-04-21T23:20:32.130 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.129+0000 7f8a113be640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f8a0c113c40 con 0x7f8a0c1032f0 2024-04-21T23:20:32.131 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.129+0000 7f8a113be640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8a0c113f40 con 0x7f8a0c1029d0 2024-04-21T23:20:32.131 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.129+0000 7f8a0a7fc640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a0c1032f0 0x7f8a0c19a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:32.131 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.129+0000 7f8a0affd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a0c1029d0 0x7f8a0c19a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:32.131 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.130+0000 7f8a0a7fc640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a0c1032f0 0x7f8a0c19a700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.40:33250/0 (socket says 172.21.15.40:33250) 2024-04-21T23:20:32.131 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.130+0000 7f8a0a7fc640 1 -- 172.21.15.40:0/569569389 learned_addr learned my addr 172.21.15.40:0/569569389 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:20:32.131 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.130+0000 7f8a0b7fe640 1 --2- 172.21.15.40:0/569569389 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a0c10bda0 0x7f8a0c19ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:32.131 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.130+0000 7f8a0affd640 1 -- 172.21.15.40:0/569569389 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a0c1032f0 msgr2=0x7f8a0c19a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:32.131 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.130+0000 7f8a0affd640 1 --2- 172.21.15.40:0/569569389 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a0c1032f0 0x7f8a0c19a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:32.131 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.130+0000 7f8a0affd640 1 -- 172.21.15.40:0/569569389 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a0c10bda0 msgr2=0x7f8a0c19ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:32.131 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.130+0000 7f8a0affd640 1 --2- 172.21.15.40:0/569569389 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a0c10bda0 0x7f8a0c19ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:32.131 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.130+0000 7f8a0affd640 1 -- 172.21.15.40:0/569569389 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a0c19f1b0 con 0x7f8a0c1029d0 2024-04-21T23:20:32.131 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.130+0000 7f8a0a7fc640 1 --2- 172.21.15.40:0/569569389 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a0c1032f0 0x7f8a0c19a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:20:32.131 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.130+0000 7f8a0b7fe640 1 --2- 172.21.15.40:0/569569389 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a0c10bda0 0x7f8a0c19ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:20:32.131 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.131+0000 7f8a0affd640 1 --2- 172.21.15.40:0/569569389 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a0c1029d0 0x7f8a0c19a1c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f89fc00e9f0 tx=0x7f89fc00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:32.132 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.131+0000 7f8a03fff640 1 -- 172.21.15.40:0/569569389 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89fc00ccc0 con 0x7f8a0c1029d0 2024-04-21T23:20:32.132 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.131+0000 7f8a03fff640 1 -- 172.21.15.40:0/569569389 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89fc010910 con 0x7f8a0c1029d0 2024-04-21T23:20:32.132 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.131+0000 7f8a03fff640 1 -- 172.21.15.40:0/569569389 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89fc018960 con 0x7f8a0c1029d0 2024-04-21T23:20:32.132 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.131+0000 7f8a113be640 1 -- 172.21.15.40:0/569569389 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a0c19f4a0 con 0x7f8a0c1029d0 2024-04-21T23:20:32.132 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.131+0000 7f8a113be640 1 -- 172.21.15.40:0/569569389 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a0c1a6d30 con 0x7f8a0c1029d0 2024-04-21T23:20:32.135 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.134+0000 7f8a03fff640 1 -- 172.21.15.40:0/569569389 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f89fc010430 con 0x7f8a0c1029d0 2024-04-21T23:20:32.135 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.134+0000 7f8a13fad740 1 Processor -- start 2024-04-21T23:20:32.135 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.134+0000 7f8a13fad740 1 -- start start 2024-04-21T23:20:32.135 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.135+0000 7f8a13fad740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a3793edb20 0x55a3793edf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:32.135 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.135+0000 7f8a13fad740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a3793ee440 0x55a3793f6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:32.135 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.135+0000 7f8a13fad740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3793f7210 0x55a3793f9600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:32.135 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.135+0000 7f8a13fad740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55a379351b70 con 0x55a3793f7210 2024-04-21T23:20:32.135 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.135+0000 7f8a13fad740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55a3792fa470 con 0x55a3793edb20 2024-04-21T23:20:32.135 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.135+0000 7f8a13fad740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55a37934db30 con 0x55a3793ee440 2024-04-21T23:20:32.135 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.135+0000 7f8a0a7fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3793f7210 0x55a3793f9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:32.135 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.135+0000 7f8a0a7fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3793f7210 0x55a3793f9600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42340/0 (socket says 172.21.15.40:42340) 2024-04-21T23:20:32.135 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.135+0000 7f8a0a7fc640 1 -- 172.21.15.40:0/4255232582 learned_addr learned my addr 172.21.15.40:0/4255232582 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:20:32.136 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.135+0000 7f8a0b7fe640 1 --2- 172.21.15.40:0/4255232582 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a3793edb20 0x55a3793edf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:32.136 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.135+0000 7f8a0affd640 1 --2- 172.21.15.40:0/4255232582 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a3793ee440 0x55a3793f6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:32.136 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.135+0000 7f8a0a7fc640 1 -- 172.21.15.40:0/4255232582 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a3793ee440 msgr2=0x55a3793f6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:32.136 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.135+0000 7f8a0a7fc640 1 --2- 172.21.15.40:0/4255232582 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a3793ee440 0x55a3793f6cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:32.136 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.135+0000 7f8a0a7fc640 1 -- 172.21.15.40:0/4255232582 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a3793edb20 msgr2=0x55a3793edf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:32.136 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.135+0000 7f8a0a7fc640 1 --2- 172.21.15.40:0/4255232582 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a3793edb20 0x55a3793edf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:32.136 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.135+0000 7f8a0a7fc640 1 -- 172.21.15.40:0/4255232582 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a379066210 con 0x55a3793f7210 2024-04-21T23:20:32.136 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.135+0000 7f8a0b7fe640 1 --2- 172.21.15.40:0/4255232582 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a3793edb20 0x55a3793edf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:20:32.136 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.136+0000 7f8a03fff640 1 --2- 172.21.15.40:0/569569389 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f89f80a2e20 0x7f89f80a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:32.136 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.136+0000 7f8a0a7fc640 1 --2- 172.21.15.40:0/4255232582 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3793f7210 0x55a3793f9600 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f89f4036040 tx=0x7f89f4005e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=303c194cb25372b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:32.136 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.136+0000 7f8a0b7fe640 1 --2- 172.21.15.40:0/569569389 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f89f80a2e20 0x7f89f80a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:32.136 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.136+0000 7f8a03fff640 1 -- 172.21.15.40:0/569569389 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f89fc0cfb00 con 0x7f8a0c1029d0 2024-04-21T23:20:32.140 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.136+0000 7f8a017fa640 1 -- 172.21.15.40:0/4255232582 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89f4004a70 con 0x55a3793f7210 2024-04-21T23:20:32.140 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.136+0000 7f8a017fa640 1 -- 172.21.15.40:0/4255232582 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89f4004c10 con 0x55a3793f7210 2024-04-21T23:20:32.140 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.136+0000 7f8a017fa640 1 -- 172.21.15.40:0/4255232582 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89f4005270 con 0x55a3793f7210 2024-04-21T23:20:32.140 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.137+0000 7f8a13fad740 1 -- 172.21.15.40:0/4255232582 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3793f7210 msgr2=0x55a3793f9600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:32.141 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.137+0000 7f8a0b7fe640 1 --2- 172.21.15.40:0/569569389 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f89f80a2e20 0x7f89f80a52e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8a04005fd0 tx=0x7f8a04009600 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:32.141 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.137+0000 7f8a13fad740 1 --2- 172.21.15.40:0/4255232582 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3793f7210 0x55a3793f9600 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f89f4036040 tx=0x7f89f4005e60 comp rx=0 tx=0).stop 2024-04-21T23:20:32.141 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.138+0000 7f8a13fad740 1 -- 172.21.15.40:0/4255232582 shutdown_connections 2024-04-21T23:20:32.141 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.138+0000 7f8a13fad740 1 --2- 172.21.15.40:0/4255232582 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3793f7210 0x55a3793f9600 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:32.141 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.138+0000 7f8a13fad740 1 --2- 172.21.15.40:0/4255232582 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a3793ee440 0x55a3793f6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:32.141 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.138+0000 7f8a13fad740 1 --2- 172.21.15.40:0/4255232582 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a3793edb20 0x55a3793edf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:32.141 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.138+0000 7f8a13fad740 1 -- 172.21.15.40:0/4255232582 >> 172.21.15.40:0/4255232582 conn(0x55a3793e8440 msgr2=0x55a3793e8820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:32.141 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.138+0000 7f8a13fad740 1 -- 172.21.15.40:0/4255232582 shutdown_connections 2024-04-21T23:20:32.141 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.138+0000 7f8a13fad740 1 -- 172.21.15.40:0/4255232582 wait complete. 2024-04-21T23:20:32.141 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.138+0000 7f8a13fad740 1 -- start start 2024-04-21T23:20:32.141 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.138+0000 7f8a13fad740 1 Processor -- start 2024-04-21T23:20:32.141 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.139+0000 7f8a13fad740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a3793edb20 0x55a3793f42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:32.141 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.139+0000 7f8a13fad740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a3793ee440 0x55a3793f47e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:32.141 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.139+0000 7f8a13fad740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3793f7210 0x55a3793f4d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:32.142 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.139+0000 7f8a13fad740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55a3793f9de0 con 0x55a3793f7210 2024-04-21T23:20:32.142 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.139+0000 7f8a0affd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a3793edb20 0x55a3793f42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:32.142 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.139+0000 7f8a0affd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a3793edb20 0x55a3793f42a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.40:33266/0 (socket says 172.21.15.40:33266) 2024-04-21T23:20:32.142 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.139+0000 7f8a0affd640 1 -- 172.21.15.40:0/1820370124 learned_addr learned my addr 172.21.15.40:0/1820370124 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:20:32.142 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.139+0000 7f8a13fad740 1 -- 172.21.15.40:0/1820370124 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55a37930da10 con 0x55a3793edb20 2024-04-21T23:20:32.142 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.139+0000 7f8a0b7fe640 1 --2- 172.21.15.40:0/1820370124 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3793f7210 0x55a3793f4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:32.142 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.139+0000 7f8a13fad740 1 -- 172.21.15.40:0/1820370124 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55a3793f9f60 con 0x55a3793ee440 2024-04-21T23:20:32.142 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.139+0000 7f8a0a7fc640 1 --2- 172.21.15.40:0/1820370124 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a3793ee440 0x55a3793f47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:32.142 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.140+0000 7f8a13fad740 10 client.0 _ll_register_callbacks cb 0x55a379348b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:20:32.142 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.140+0000 7f8a0affd640 1 -- 172.21.15.40:0/1820370124 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a3793ee440 msgr2=0x55a3793f47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:32.142 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.140+0000 7f8a0affd640 1 --2- 172.21.15.40:0/1820370124 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a3793ee440 0x55a3793f47e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:32.142 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.140+0000 7f8a0affd640 1 -- 172.21.15.40:0/1820370124 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3793f7210 msgr2=0x55a3793f4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:32.142 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.140+0000 7f8a0affd640 1 --2- 172.21.15.40:0/1820370124 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3793f7210 0x55a3793f4d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:32.143 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.140+0000 7f8a0affd640 1 -- 172.21.15.40:0/1820370124 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a379132620 con 0x55a3793edb20 2024-04-21T23:20:32.143 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.140+0000 7f8a0b7fe640 1 --2- 172.21.15.40:0/1820370124 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3793f7210 0x55a3793f4d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:20:32.143 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.140+0000 7f8a0affd640 1 --2- 172.21.15.40:0/1820370124 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a3793edb20 0x55a3793f42a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f89fc00c6a0 tx=0x7f89fc0d3240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b984d6a59a3fc979 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:32.143 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.140+0000 7f89f27fc640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-21T23:20:32.143 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.140+0000 7f89f27fc640 1 -- 172.21.15.40:0/1820370124 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89fc09b860 con 0x55a3793edb20 2024-04-21T23:20:32.143 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.140+0000 7f89f27fc640 1 -- 172.21.15.40:0/1820370124 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f89fc010ac0 con 0x55a3793edb20 2024-04-21T23:20:32.143 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.140+0000 7f8a13fad740 1 -- 172.21.15.40:0/1820370124 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f89f4004c10 con 0x55a3793edb20 2024-04-21T23:20:32.143 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.141+0000 7f89f27fc640 1 -- 172.21.15.40:0/1820370124 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f89fc09b680 con 0x55a3793edb20 2024-04-21T23:20:32.143 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.141+0000 7f8a13fad740 10 client.25745 fetch_fsmap learned FSMap version 29 2024-04-21T23:20:32.143 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.141+0000 7f8a13fad740 1 -- 172.21.15.40:0/1820370124 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a3794060d0 con 0x55a3793edb20 2024-04-21T23:20:32.143 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.141+0000 7f89f27fc640 1 -- 172.21.15.40:0/1820370124 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89f4004c10 con 0x55a3793edb20 2024-04-21T23:20:32.143 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.141+0000 7f89f27fc640 1 -- 172.21.15.40:0/1820370124 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f89fc09b170 con 0x55a3793edb20 2024-04-21T23:20:32.143 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.141+0000 7f8a13fad740 10 client.25745 fetch_fsmap finished waiting for FSMap version 29 2024-04-21T23:20:32.143 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.141+0000 7f8a13fad740 10 client.25745 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:20:32.143 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.141+0000 7f8a13fad740 20 client.25745 populate_metadata read hostname 'smithi040' 2024-04-21T23:20:32.144 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.141+0000 7f8a13fad740 1 --2- 172.21.15.40:0/1820370124 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55a379407590 0x55a3794099c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:32.144 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.141+0000 7f8a13fad740 4 client.25745 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:20:32.144 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.141+0000 7f8a13fad740 1 -- 172.21.15.40:0/1820370124 --> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a37935f0b0 con 0x55a379407590 2024-04-21T23:20:32.144 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.141+0000 7f8a0a7fc640 1 --2- 172.21.15.40:0/1820370124 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55a379407590 0x55a3794099c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:32.145 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.142+0000 7f89f27fc640 10 client.25745 ms_handle_connect on v2:172.21.15.111:6838/3712235548 2024-04-21T23:20:32.145 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.142+0000 7f8a0a7fc640 1 --2- 172.21.15.40:0/1820370124 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55a379407590 0x55a3794099c0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a3ef1a0fbd762c53 server_cookie=28623b66f0cb8d7f in_seq=0 out_seq=0 2024-04-21T23:20:32.145 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.143+0000 7f89f27fc640 1 -- 172.21.15.40:0/1820370124 <== mds.0 v2:172.21.15.111:6838/3712235548 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a37935f0b0 con 0x55a379407590 2024-04-21T23:20:32.145 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.143+0000 7f89f27fc640 10 client.25745 handle_command_reply: tid=0 2024-04-21T23:20:32.145 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.143+0000 7f8a13fad740 1 client.25745 shutdown 2024-04-21T23:20:32.145 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.143+0000 7f8a13fad740 1 -- 172.21.15.40:0/1820370124 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a3793edb20 msgr2=0x55a3793f42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:32.145 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.143+0000 7f8a13fad740 1 --2- 172.21.15.40:0/1820370124 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a3793edb20 0x55a3793f42a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f89fc00c6a0 tx=0x7f89fc0d3240 comp rx=0 tx=0).stop 2024-04-21T23:20:32.145 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.143+0000 7f8a13fad740 1 -- 172.21.15.40:0/1820370124 shutdown_connections 2024-04-21T23:20:32.145 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.143+0000 7f8a13fad740 1 --2- 172.21.15.40:0/1820370124 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55a379407590 0x55a3794099c0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:32.145 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.143+0000 7f8a13fad740 1 --2- 172.21.15.40:0/1820370124 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3793f7210 0x55a3793f4d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:32.146 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.143+0000 7f89f27fc640 0 client.25745 ms_handle_reset on v2:172.21.15.111:6838/3712235548 2024-04-21T23:20:32.146 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.143+0000 7f8a13fad740 1 --2- 172.21.15.40:0/1820370124 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a3793ee440 0x55a3793f47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:32.146 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.143+0000 7f8a13fad740 1 --2- 172.21.15.40:0/1820370124 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a3793edb20 0x55a3793f42a0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:32.146 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.143+0000 7f8a13fad740 1 -- 172.21.15.40:0/1820370124 >> 172.21.15.40:0/1820370124 conn(0x55a3793e8440 msgr2=0x55a3793ea190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:32.146 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.143+0000 7f8a13fad740 1 -- 172.21.15.40:0/1820370124 shutdown_connections 2024-04-21T23:20:32.146 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.143+0000 7f8a13fad740 1 -- 172.21.15.40:0/1820370124 wait complete. 2024-04-21T23:20:32.146 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.143+0000 7f8a13fad740 20 client.25745 trim_cache size 0 max 16384 2024-04-21T23:20:32.166 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.166+0000 7f8a13fad740 1 Processor -- start 2024-04-21T23:20:32.166 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.166+0000 7f8a13fad740 1 -- start start 2024-04-21T23:20:32.166 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.166+0000 7f8a13fad740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a3793edb20 0x55a3793f69d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:32.167 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.166+0000 7f8a13fad740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3793ee440 0x55a379402ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:32.167 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.166+0000 7f8a13fad740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a3793f7210 0x55a3793e9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:32.167 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.166+0000 7f8a13fad740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8a0c1b3670 con 0x55a3793ee440 2024-04-21T23:20:32.167 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.166+0000 7f8a13fad740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f8a0c1b0ee0 con 0x55a3793edb20 2024-04-21T23:20:32.167 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.166+0000 7f8a13fad740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8a0c1aed30 con 0x55a3793f7210 2024-04-21T23:20:32.167 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.166+0000 7f8a0a7fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3793ee440 0x55a379402ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:32.167 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.167+0000 7f8a0affd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a3793edb20 0x55a3793f69d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:32.167 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.167+0000 7f8a0a7fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3793ee440 0x55a379402ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42354/0 (socket says 172.21.15.40:42354) 2024-04-21T23:20:32.167 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.167+0000 7f8a0a7fc640 1 -- 172.21.15.40:0/2703219549 learned_addr learned my addr 172.21.15.40:0/2703219549 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:20:32.167 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.167+0000 7f8a0b7fe640 1 --2- 172.21.15.40:0/2703219549 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a3793f7210 0x55a3793e9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:32.167 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.167+0000 7f8a0affd640 1 -- 172.21.15.40:0/2703219549 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a3793f7210 msgr2=0x55a3793e9fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:32.167 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.167+0000 7f8a0affd640 1 --2- 172.21.15.40:0/2703219549 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a3793f7210 0x55a3793e9fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:32.167 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.167+0000 7f8a0affd640 1 -- 172.21.15.40:0/2703219549 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3793ee440 msgr2=0x55a379402ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:32.167 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.167+0000 7f8a0affd640 1 --2- 172.21.15.40:0/2703219549 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3793ee440 0x55a379402ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.167+0000 7f8a0affd640 1 -- 172.21.15.40:0/2703219549 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a3792d8540 con 0x55a3793edb20 2024-04-21T23:20:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.167+0000 7f8a0affd640 1 --2- 172.21.15.40:0/2703219549 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a3793edb20 0x55a3793f69d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f89fc099fd0 tx=0x7f89fc00ce60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=172f7716fda98b15 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.167+0000 7f8a00ff9640 1 -- 172.21.15.40:0/2703219549 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89fc09b860 con 0x55a3793edb20 2024-04-21T23:20:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.167+0000 7f8a00ff9640 1 -- 172.21.15.40:0/2703219549 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89fc097da0 con 0x55a3793edb20 2024-04-21T23:20:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.167+0000 7f8a00ff9640 1 -- 172.21.15.40:0/2703219549 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89fc0a48e0 con 0x55a3793edb20 2024-04-21T23:20:32.169 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.168+0000 7f8a13fad740 1 -- 172.21.15.40:0/2703219549 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a3793edb20 msgr2=0x55a3793f69d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:32.169 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.168+0000 7f8a13fad740 1 --2- 172.21.15.40:0/2703219549 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a3793edb20 0x55a3793f69d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f89fc099fd0 tx=0x7f89fc00ce60 comp rx=0 tx=0).stop 2024-04-21T23:20:32.169 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.168+0000 7f8a13fad740 1 -- 172.21.15.40:0/2703219549 shutdown_connections 2024-04-21T23:20:32.169 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.168+0000 7f8a13fad740 1 --2- 172.21.15.40:0/2703219549 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a3793f7210 0x55a3793e9fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:32.169 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.168+0000 7f8a13fad740 1 --2- 172.21.15.40:0/2703219549 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3793ee440 0x55a379402ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:32.169 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.168+0000 7f8a13fad740 1 --2- 172.21.15.40:0/2703219549 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a3793edb20 0x55a3793f69d0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:32.169 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.168+0000 7f8a13fad740 1 -- 172.21.15.40:0/2703219549 >> 172.21.15.40:0/2703219549 conn(0x55a3793e8440 msgr2=0x55a3793e9890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:32.170 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.169+0000 7f8a13fad740 1 -- 172.21.15.40:0/2703219549 shutdown_connections 2024-04-21T23:20:32.170 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.169+0000 7f8a13fad740 1 -- 172.21.15.40:0/2703219549 wait complete. 2024-04-21T23:20:32.170 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.169+0000 7f8a13fad740 1 -- start start 2024-04-21T23:20:32.170 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.169+0000 7f8a13fad740 1 Processor -- start 2024-04-21T23:20:32.170 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.169+0000 7f8a13fad740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3793edb20 0x55a3793fa6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:32.170 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.169+0000 7f8a13fad740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a3793ee440 0x55a3793fac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:32.170 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.169+0000 7f8a13fad740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a3793f7210 0x55a379465b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:32.172 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.170+0000 7f8a13fad740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8a0c1af3a0 con 0x55a3793edb20 2024-04-21T23:20:32.172 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.170+0000 7f8a13fad740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f8a0c1b0810 con 0x55a3793ee440 2024-04-21T23:20:32.172 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.170+0000 7f8a13fad740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8a0c1afa70 con 0x55a3793f7210 2024-04-21T23:20:32.172 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.170+0000 7f8a13fad740 10 client.0 _ll_register_callbacks cb 0x7f89fc0d3340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:20:32.172 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.170+0000 7f8a0affd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3793edb20 0x55a3793fa6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:32.172 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.171+0000 7f8a0affd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3793edb20 0x55a3793fa6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42362/0 (socket says 172.21.15.40:42362) 2024-04-21T23:20:32.172 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.171+0000 7f8a0affd640 1 -- 172.21.15.40:0/2569189864 learned_addr learned my addr 172.21.15.40:0/2569189864 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:20:32.172 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.170+0000 7f8a0a7fc640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a3793ee440 0x55a3793fac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:32.172 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.170+0000 7f8a0b7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a3793f7210 0x55a379465b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:32.172 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.171+0000 7f8a0a7fc640 1 -- 172.21.15.40:0/2569189864 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a3793f7210 msgr2=0x55a379465b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:32.172 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.171+0000 7f8a0a7fc640 1 --2- 172.21.15.40:0/2569189864 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a3793f7210 0x55a379465b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:32.173 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.171+0000 7f8a0a7fc640 1 -- 172.21.15.40:0/2569189864 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3793edb20 msgr2=0x55a3793fa6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:32.173 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.171+0000 7f8a0a7fc640 1 --2- 172.21.15.40:0/2569189864 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3793edb20 0x55a3793fa6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:32.173 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.171+0000 7f8a0a7fc640 1 -- 172.21.15.40:0/2569189864 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a3794661b0 con 0x55a3793ee440 2024-04-21T23:20:32.173 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.171+0000 7f8a0affd640 1 --2- 172.21.15.40:0/2569189864 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3793edb20 0x55a3793fa6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:20:32.173 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.171+0000 7f8a0a7fc640 1 --2- 172.21.15.40:0/2569189864 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a3793ee440 0x55a3793fac10 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f89f402ce50 tx=0x7f89f4004dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a3c018f2a26f9ac server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:32.173 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.171+0000 7f89f27fc640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-21T23:20:32.173 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.172+0000 7f89f27fc640 1 -- 172.21.15.40:0/2569189864 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89f4059060 con 0x55a3793ee440 2024-04-21T23:20:32.173 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.172+0000 7f89f27fc640 1 -- 172.21.15.40:0/2569189864 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f89f4004040 con 0x55a3793ee440 2024-04-21T23:20:32.173 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.172+0000 7f8a13fad740 1 -- 172.21.15.40:0/2569189864 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f89fc097da0 con 0x55a3793ee440 2024-04-21T23:20:32.173 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.172+0000 7f89f27fc640 1 -- 172.21.15.40:0/2569189864 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f89f4060050 con 0x55a3793ee440 2024-04-21T23:20:32.173 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.172+0000 7f8a13fad740 10 client.25751 fetch_fsmap learned FSMap version 29 2024-04-21T23:20:32.173 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.172+0000 7f8a13fad740 1 -- 172.21.15.40:0/2569189864 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a37946d5d0 con 0x55a3793ee440 2024-04-21T23:20:32.173 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.172+0000 7f89f27fc640 1 -- 172.21.15.40:0/2569189864 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89fc097da0 con 0x55a3793ee440 2024-04-21T23:20:32.173 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.172+0000 7f89f27fc640 1 -- 172.21.15.40:0/2569189864 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f89f4051070 con 0x55a3793ee440 2024-04-21T23:20:32.173 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.172+0000 7f8a13fad740 10 client.25751 fetch_fsmap finished waiting for FSMap version 29 2024-04-21T23:20:32.174 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.172+0000 7f8a13fad740 10 client.25751 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:20:32.174 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.172+0000 7f8a13fad740 20 client.25751 populate_metadata read hostname 'smithi040' 2024-04-21T23:20:32.174 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.172+0000 7f8a13fad740 1 --2- 172.21.15.40:0/2569189864 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55a37946da20 0x55a37946fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:32.174 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.172+0000 7f8a13fad740 4 client.25751 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T23:20:32.174 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.172+0000 7f8a13fad740 1 -- 172.21.15.40:0/2569189864 --> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a379359400 con 0x55a37946da20 2024-04-21T23:20:32.174 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.173+0000 7f8a0affd640 1 --2- 172.21.15.40:0/2569189864 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55a37946da20 0x55a37946fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:32.174 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.173+0000 7f89f27fc640 10 client.25751 ms_handle_connect on v2:172.21.15.111:6838/3712235548 2024-04-21T23:20:32.174 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.173+0000 7f8a0affd640 1 --2- 172.21.15.40:0/2569189864 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55a37946da20 0x55a37946fe10 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2a1fad4262d1cb3b server_cookie=8ec561dac623b84c in_seq=0 out_seq=0 2024-04-21T23:20:32.181 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.181+0000 7f89f27fc640 1 -- 172.21.15.40:0/2569189864 <== mds.0 v2:172.21.15.111:6838/3712235548 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a379359400 con 0x55a37946da20 2024-04-21T23:20:32.181 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.181+0000 7f89f27fc640 10 client.25751 handle_command_reply: tid=0 2024-04-21T23:20:32.181 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.181+0000 7f8a13fad740 1 client.25751 shutdown 2024-04-21T23:20:32.181 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.181+0000 7f8a13fad740 1 -- 172.21.15.40:0/2569189864 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a3793ee440 msgr2=0x55a3793fac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:32.181 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.181+0000 7f8a13fad740 1 --2- 172.21.15.40:0/2569189864 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a3793ee440 0x55a3793fac10 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f89f402ce50 tx=0x7f89f4004dc0 comp rx=0 tx=0).stop 2024-04-21T23:20:32.181 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.182+0000 7f8a13fad740 1 -- 172.21.15.40:0/2569189864 shutdown_connections 2024-04-21T23:20:32.181 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.182+0000 7f8a13fad740 1 --2- 172.21.15.40:0/2569189864 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55a37946da20 0x55a37946fe10 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:32.182 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.182+0000 7f8a13fad740 1 --2- 172.21.15.40:0/2569189864 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a3793ee440 0x55a3793fac10 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:32.182 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.182+0000 7f8a13fad740 1 --2- 172.21.15.40:0/2569189864 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a3793f7210 0x55a379465b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:32.182 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.182+0000 7f89f27fc640 0 client.25751 ms_handle_reset on v2:172.21.15.111:6838/3712235548 2024-04-21T23:20:32.182 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.182+0000 7f8a13fad740 1 --2- 172.21.15.40:0/2569189864 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3793edb20 0x55a3793fa6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:32.182 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.182+0000 7f8a13fad740 1 -- 172.21.15.40:0/2569189864 >> 172.21.15.40:0/2569189864 conn(0x55a3793e8440 msgr2=0x55a3793f2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:32.182 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.182+0000 7f8a13fad740 1 -- 172.21.15.40:0/2569189864 shutdown_connections 2024-04-21T23:20:32.182 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.182+0000 7f8a13fad740 1 -- 172.21.15.40:0/2569189864 wait complete. 2024-04-21T23:20:32.182 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.182+0000 7f8a13fad740 20 client.25751 trim_cache size 0 max 16384 2024-04-21T23:20:32.182 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:{ 2024-04-21T23:20:32.182 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "message": "", 2024-04-21T23:20:32.183 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "return_code": 0 2024-04-21T23:20:32.183 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:} 2024-04-21T23:20:32.183 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.182+0000 7f8a113be640 1 -- 172.21.15.40:0/569569389 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f89f80a2e20 msgr2=0x7f89f80a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:32.183 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.182+0000 7f8a113be640 1 --2- 172.21.15.40:0/569569389 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f89f80a2e20 0x7f89f80a52e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8a04005fd0 tx=0x7f8a04009600 comp rx=0 tx=0).stop 2024-04-21T23:20:32.183 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.182+0000 7f8a113be640 1 -- 172.21.15.40:0/569569389 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a0c1029d0 msgr2=0x7f8a0c19a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:32.183 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.182+0000 7f8a113be640 1 --2- 172.21.15.40:0/569569389 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a0c1029d0 0x7f8a0c19a1c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f89fc00e9f0 tx=0x7f89fc00eeb0 comp rx=0 tx=0).stop 2024-04-21T23:20:32.184 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.183+0000 7f8a113be640 1 -- 172.21.15.40:0/569569389 shutdown_connections 2024-04-21T23:20:32.184 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.183+0000 7f8a113be640 1 --2- 172.21.15.40:0/569569389 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f89f80a2e20 0x7f89f80a52e0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:32.184 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.183+0000 7f8a113be640 1 --2- 172.21.15.40:0/569569389 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a0c1032f0 0x7f8a0c19a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:32.184 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.183+0000 7f8a113be640 1 --2- 172.21.15.40:0/569569389 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a0c10bda0 0x7f8a0c19ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:32.184 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.183+0000 7f8a113be640 1 --2- 172.21.15.40:0/569569389 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a0c1029d0 0x7f8a0c19a1c0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:32.184 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.183+0000 7f8a113be640 1 -- 172.21.15.40:0/569569389 >> 172.21.15.40:0/569569389 conn(0x7f8a0c077790 msgr2=0x7f8a0c0fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:32.184 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.183+0000 7f8a113be640 1 -- 172.21.15.40:0/569569389 shutdown_connections 2024-04-21T23:20:32.184 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-21T23:20:32.183+0000 7f8a113be640 1 -- 172.21.15.40:0/569569389 wait complete. 2024-04-21T23:20:32.187 INFO:tasks.workunit.client.0.smithi040.stdout: [CC] pattern.lo 2024-04-21T23:20:32.188 INFO:tasks.workunit.client.0.smithi040.stdout: [CC] forker.lo 2024-04-21T23:20:32.211 INFO:tasks.workunit.client.0.smithi040.stdout: [CC] open_flags.lo 2024-04-21T23:20:32.222 INFO:tasks.workunit.client.0.smithi040.stdout: [CC] random_range.lo 2024-04-21T23:20:32.338 INFO:tasks.workunit.client.0.smithi040.stdout: [CC] string_to_tokens.lo 2024-04-21T23:20:32.360 INFO:tasks.workunit.client.0.smithi040.stdout: [CC] str_to_bytes.lo 2024-04-21T23:20:32.389 INFO:tasks.workunit.client.0.smithi040.stdout: [CC] tlibio.lo 2024-04-21T23:20:32.484 INFO:tasks.workunit.client.0.smithi040.stdout: [CC] write_log.lo 2024-04-21T23:20:32.496 INFO:tasks.workunit.client.0.smithi040.stdout: [CC] random.lo 2024-04-21T23:20:32.764 INFO:tasks.workunit.client.0.smithi040.stdout: [LD] 2024-04-21T23:20:32.885 INFO:tasks.workunit.client.0.smithi040.stdout:Building ltp 2024-04-21T23:20:32.886 INFO:tasks.workunit.client.0.smithi040.stdout:Building src 2024-04-21T23:20:32.886 INFO:tasks.workunit.client.0.smithi040.stdout:Building m4 2024-04-21T23:20:32.886 INFO:tasks.workunit.client.0.smithi040.stdout:Building common 2024-04-21T23:20:32.894 INFO:tasks.workunit.client.0.smithi040.stdout: [DEP] 2024-04-21T23:20:32.912 INFO:tasks.workunit.client.0.smithi040.stdout: [DEP] 2024-04-21T23:20:33.085 INFO:tasks.workunit.client.0.smithi040.stdout: [CC] doio 2024-04-21T23:20:33.085 INFO:tasks.workunit.client.0.smithi040.stdout: [CC] fsstress 2024-04-21T23:20:33.085 INFO:tasks.workunit.client.0.smithi040.stdout: [CC] fsx 2024-04-21T23:20:33.231 INFO:tasks.workunit.client.0.smithi040.stderr:fsstress.c: In function 'attr_list_path': 2024-04-21T23:20:33.231 INFO:tasks.workunit.client.0.smithi040.stderr:fsstress.c:859:9: warning: 'attr_list' is deprecated: Use listxattr or llistxattr instead [-Wdeprecated-declarations] 2024-04-21T23:20:33.231 INFO:tasks.workunit.client.0.smithi040.stderr: 859 | rval = attr_list(name->path, buffer, buffersize, flags, cursor); 2024-04-21T23:20:33.231 INFO:tasks.workunit.client.0.smithi040.stderr: | ^~~~ 2024-04-21T23:20:33.232 INFO:tasks.workunit.client.0.smithi040.stderr:In file included from ../src/global.h:25, 2024-04-21T23:20:33.232 INFO:tasks.workunit.client.0.smithi040.stderr: from fsstress.c:12: 2024-04-21T23:20:33.232 INFO:tasks.workunit.client.0.smithi040.stderr:/usr/include/attr/attributes.h:165:12: note: declared here 2024-04-21T23:20:33.232 INFO:tasks.workunit.client.0.smithi040.stderr: 165 | extern int attr_list(const char *__path, char *__buffer, const int __buffersize, 2024-04-21T23:20:33.232 INFO:tasks.workunit.client.0.smithi040.stderr: | ^~~~~~~~~ 2024-04-21T23:20:33.232 INFO:tasks.workunit.client.0.smithi040.stderr:fsstress.c: In function 'attr_remove_path': 2024-04-21T23:20:33.232 INFO:tasks.workunit.client.0.smithi040.stderr:fsstress.c:878:9: warning: 'attr_remove' is deprecated: Use removexattr or lremovexattr instead [-Wdeprecated-declarations] 2024-04-21T23:20:33.232 INFO:tasks.workunit.client.0.smithi040.stderr: 878 | rval = attr_remove(name->path, attrname, flags); 2024-04-21T23:20:33.232 INFO:tasks.workunit.client.0.smithi040.stderr: | ^~~~ 2024-04-21T23:20:33.232 INFO:tasks.workunit.client.0.smithi040.stderr:In file included from ../src/global.h:25, 2024-04-21T23:20:33.232 INFO:tasks.workunit.client.0.smithi040.stderr: from fsstress.c:12: 2024-04-21T23:20:33.232 INFO:tasks.workunit.client.0.smithi040.stderr:/usr/include/attr/attributes.h:152:12: note: declared here 2024-04-21T23:20:33.232 INFO:tasks.workunit.client.0.smithi040.stderr: 152 | extern int attr_remove (const char *__path, const char *__attrname, 2024-04-21T23:20:33.232 INFO:tasks.workunit.client.0.smithi040.stderr: | ^~~~~~~~~~~ 2024-04-21T23:20:33.232 INFO:tasks.workunit.client.0.smithi040.stderr:fsstress.c: In function 'attr_set_path': 2024-04-21T23:20:33.233 INFO:tasks.workunit.client.0.smithi040.stderr:fsstress.c:898:9: warning: 'attr_set' is deprecated: Use setxattr or lsetxattr instead [-Wdeprecated-declarations] 2024-04-21T23:20:33.233 INFO:tasks.workunit.client.0.smithi040.stderr: 898 | rval = attr_set(name->path, attrname, attrvalue, valuelength, flags); 2024-04-21T23:20:33.233 INFO:tasks.workunit.client.0.smithi040.stderr: | ^~~~ 2024-04-21T23:20:33.233 INFO:tasks.workunit.client.0.smithi040.stderr:In file included from ../src/global.h:25, 2024-04-21T23:20:33.233 INFO:tasks.workunit.client.0.smithi040.stderr: from fsstress.c:12: 2024-04-21T23:20:33.233 INFO:tasks.workunit.client.0.smithi040.stderr:/usr/include/attr/attributes.h:139:12: note: declared here 2024-04-21T23:20:33.233 INFO:tasks.workunit.client.0.smithi040.stderr: 139 | extern int attr_set (const char *__path, const char *__attrname, 2024-04-21T23:20:33.233 INFO:tasks.workunit.client.0.smithi040.stderr: | ^~~~~~~~ 2024-04-21T23:20:33.243 INFO:tasks.workunit.client.0.smithi040.stderr:fsstress.c: In function 'allocsp_f': 2024-04-21T23:20:33.243 INFO:tasks.workunit.client.0.smithi040.stderr:fsstress.c:2047:32: error: 'XFS_IOC_ALLOCSP64' undeclared (first use in this function); did you mean 'XFS_IOC_RESVSP64'? 2024-04-21T23:20:33.243 INFO:tasks.workunit.client.0.smithi040.stderr: 2047 | e = xfsctl(f.path, fd, XFS_IOC_ALLOCSP64, &fl) < 0 ? errno : 0; 2024-04-21T23:20:33.243 INFO:tasks.workunit.client.0.smithi040.stderr: | ^~~~~~~~~~~~~~~~~ 2024-04-21T23:20:33.243 INFO:tasks.workunit.client.0.smithi040.stderr: | XFS_IOC_RESVSP64 2024-04-21T23:20:33.243 INFO:tasks.workunit.client.0.smithi040.stderr:fsstress.c:2047:32: note: each undeclared identifier is reported only once for each function it appears in 2024-04-21T23:20:33.261 INFO:tasks.workunit.client.0.smithi040.stderr:fsstress.c: In function 'freesp_f': 2024-04-21T23:20:33.261 INFO:tasks.workunit.client.0.smithi040.stderr:fsstress.c:3640:32: error: 'XFS_IOC_FREESP64' undeclared (first use in this function); did you mean 'XFS_IOC_RESVSP64'? 2024-04-21T23:20:33.261 INFO:tasks.workunit.client.0.smithi040.stderr: 3640 | e = xfsctl(f.path, fd, XFS_IOC_FREESP64, &fl) < 0 ? errno : 0; 2024-04-21T23:20:33.261 INFO:tasks.workunit.client.0.smithi040.stderr: | ^~~~~~~~~~~~~~~~ 2024-04-21T23:20:33.261 INFO:tasks.workunit.client.0.smithi040.stderr: | XFS_IOC_RESVSP64 2024-04-21T23:20:33.295 INFO:tasks.workunit.client.0.smithi040.stderr:gmake[3]: *** [Makefile:47: fsstress] Error 1 2024-04-21T23:20:33.295 INFO:tasks.workunit.client.0.smithi040.stderr:gmake[3]: *** Waiting for unfinished jobs.... 2024-04-21T23:20:33.433 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:33 smithi040 ceph-mon[27984]: pgmap v67: 129 pgs: 129 active+clean; 93 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 757 KiB/s rd, 646 KiB/s wr, 320 op/s 2024-04-21T23:20:33.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:33 smithi174 ceph-mon[31409]: pgmap v67: 129 pgs: 129 active+clean; 93 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 757 KiB/s rd, 646 KiB/s wr, 320 op/s 2024-04-21T23:20:33.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:33 smithi111 ceph-mon[31085]: pgmap v67: 129 pgs: 129 active+clean; 93 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 757 KiB/s rd, 646 KiB/s wr, 320 op/s 2024-04-21T23:20:33.674 INFO:tasks.workunit.client.0.smithi040.stderr:doio.c: In function 'parse_cmdline': 2024-04-21T23:20:33.674 INFO:tasks.workunit.client.0.smithi040.stderr:doio.c:3113:33: warning: '%s' directive argument is null [-Wformat-overflow=] 2024-04-21T23:20:33.674 INFO:tasks.workunit.client.0.smithi040.stderr: 3113 | fprintf(stderr, 2024-04-21T23:20:33.674 INFO:tasks.workunit.client.0.smithi040.stderr: | ^~~~~~~~~~~~~~~ 2024-04-21T23:20:33.674 INFO:tasks.workunit.client.0.smithi040.stderr: 3114 | "%s%s: Illegal -C arg (%s). Must be one of: ", 2024-04-21T23:20:33.674 INFO:tasks.workunit.client.0.smithi040.stderr: | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-21T23:20:33.674 INFO:tasks.workunit.client.0.smithi040.stderr: 3115 | Prog, TagName, tok); 2024-04-21T23:20:33.674 INFO:tasks.workunit.client.0.smithi040.stderr: | ~~~~~~~~~~~~~~~~~~~ 2024-04-21T23:20:33.702 INFO:tasks.workunit.client.0.smithi040.stderr:fsx.c: In function 'main': 2024-04-21T23:20:33.702 INFO:tasks.workunit.client.0.smithi040.stderr:fsx.c:2662:33: warning: 'strncpy' specified bound 4096 equals destination size [-Wstringop-truncation] 2024-04-21T23:20:33.702 INFO:tasks.workunit.client.0.smithi040.stderr: 2662 | strncpy(opsfile, optarg, sizeof(opsfile)); 2024-04-21T23:20:33.702 INFO:tasks.workunit.client.0.smithi040.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-21T23:20:33.702 INFO:tasks.workunit.client.0.smithi040.stderr:fsx.c:2632:25: warning: 'strncpy' specified bound 1024 equals destination size [-Wstringop-truncation] 2024-04-21T23:20:33.702 INFO:tasks.workunit.client.0.smithi040.stderr: 2632 | strncpy(dname, optarg, sizeof(dname)); 2024-04-21T23:20:33.702 INFO:tasks.workunit.client.0.smithi040.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-21T23:20:33.817 INFO:tasks.workunit.client.0.smithi040.stderr:gmake[2]: *** [include/buildrules:30: ltp] Error 2 2024-04-21T23:20:33.817 INFO:tasks.workunit.client.0.smithi040.stderr:gmake[2]: *** Waiting for unfinished jobs.... 2024-04-21T23:20:34.457 INFO:tasks.workunit.client.0.smithi040.stdout: [DEP] 2024-04-21T23:20:34.458 INFO:tasks.workunit.client.0.smithi040.stdout: [DEP] 2024-04-21T23:20:34.458 INFO:tasks.workunit.client.0.smithi040.stdout: [DEP] 2024-04-21T23:20:34.685 INFO:tasks.workunit.client.0.smithi040.stdout: [CC] dirstress 2024-04-21T23:20:34.687 INFO:tasks.workunit.client.0.smithi040.stdout: [CC] fill 2024-04-21T23:20:34.687 INFO:tasks.workunit.client.0.smithi040.stdout: [CC] fill2 2024-04-21T23:20:34.688 INFO:tasks.workunit.client.0.smithi040.stdout: [CC] getpagesize 2024-04-21T23:20:34.821 INFO:tasks.workunit.client.0.smithi040.stdout: [CC] holes 2024-04-21T23:20:34.835 INFO:tasks.workunit.client.0.smithi040.stdout: [CC] lstat64 2024-04-21T23:20:34.916 INFO:tasks.workunit.client.0.smithi040.stdout: [CC] nametest 2024-04-21T23:20:34.944 INFO:tasks.workunit.client.0.smithi040.stdout: [CC] permname 2024-04-21T23:20:35.014 INFO:tasks.workunit.client.0.smithi040.stdout: [CC] randholes 2024-04-21T23:20:35.048 INFO:tasks.workunit.client.0.smithi040.stdout: [CC] runas 2024-04-21T23:20:35.146 INFO:tasks.workunit.client.0.smithi040.stdout: [CC] truncfile 2024-04-21T23:20:35.164 INFO:tasks.workunit.client.0.smithi040.stdout: [CC] usemem 2024-04-21T23:20:35.224 INFO:tasks.workunit.client.0.smithi040.stdout: [CC] mmapcat 2024-04-21T23:20:35.277 INFO:tasks.workunit.client.0.smithi040.stdout: [CC] append_reader 2024-04-21T23:20:35.326 INFO:tasks.workunit.client.0.smithi040.stdout: [CC] append_writer 2024-04-21T23:20:35.335 INFO:tasks.workunit.client.0.smithi040.stdout: [CC] dirperf 2024-04-21T23:20:35.390 INFO:tasks.workunit.client.0.smithi040.stdout: [CC] metaperf 2024-04-21T23:20:35.420 INFO:tasks.workunit.client.0.smithi040.stdout: [CC] devzero 2024-04-21T23:20:35.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:35 smithi040 ceph-mon[27984]: pgmap v68: 129 pgs: 129 active+clean; 93 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 757 KiB/s rd, 646 KiB/s wr, 320 op/s 2024-04-21T23:20:35.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:35 smithi174 ceph-mon[31409]: pgmap v68: 129 pgs: 129 active+clean; 93 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 757 KiB/s rd, 646 KiB/s wr, 320 op/s 2024-04-21T23:20:35.443 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:35 smithi111 ceph-mon[31085]: pgmap v68: 129 pgs: 129 active+clean; 93 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 757 KiB/s rd, 646 KiB/s wr, 320 op/s 2024-04-21T23:20:35.490 INFO:tasks.workunit.client.0.smithi040.stdout: [CC] feature 2024-04-21T23:20:35.595 INFO:tasks.workunit.client.0.smithi040.stdout: [CC] alloc 2024-04-21T23:20:35.635 INFO:tasks.workunit.client.0.smithi040.stdout: [CC] fault 2024-04-21T23:20:35.684 INFO:tasks.workunit.client.0.smithi040.stdout: [CC] fstest 2024-04-21T23:20:35.694 INFO:tasks.workunit.client.0.smithi040.stdout: [CC] t_access_root 2024-04-21T23:20:35.728 INFO:tasks.workunit.client.0.smithi040.stderr:alloc.c: In function 'main': 2024-04-21T23:20:35.728 INFO:tasks.workunit.client.0.smithi040.stderr:alloc.c:154:1: error: 'bozo' undeclared (first use in this function) 2024-04-21T23:20:35.729 INFO:tasks.workunit.client.0.smithi040.stderr: 154 | bozo! 2024-04-21T23:20:35.729 INFO:tasks.workunit.client.0.smithi040.stderr: | ^~~~ 2024-04-21T23:20:35.729 INFO:tasks.workunit.client.0.smithi040.stderr:alloc.c:154:1: note: each undeclared identifier is reported only once for each function it appears in 2024-04-21T23:20:35.729 INFO:tasks.workunit.client.0.smithi040.stderr:alloc.c:154:5: error: expected ';' before '!' token 2024-04-21T23:20:35.729 INFO:tasks.workunit.client.0.smithi040.stderr: 154 | bozo! 2024-04-21T23:20:35.729 INFO:tasks.workunit.client.0.smithi040.stderr: | ^ 2024-04-21T23:20:35.729 INFO:tasks.workunit.client.0.smithi040.stderr: | ; 2024-04-21T23:20:35.730 INFO:tasks.workunit.client.0.smithi040.stderr:alloc.c:184:25: error: 'rflag' undeclared (first use in this function); did you mean 'nflag'? 2024-04-21T23:20:35.730 INFO:tasks.workunit.client.0.smithi040.stderr: 184 | rflag = 1; 2024-04-21T23:20:35.730 INFO:tasks.workunit.client.0.smithi040.stderr: | ^~~~~ 2024-04-21T23:20:35.730 INFO:tasks.workunit.client.0.smithi040.stderr: | nflag 2024-04-21T23:20:35.730 INFO:tasks.workunit.client.0.smithi040.stderr:alloc.c:320:5: error: expected ';' before '!' token 2024-04-21T23:20:35.730 INFO:tasks.workunit.client.0.smithi040.stderr: 320 | bozo! 2024-04-21T23:20:35.730 INFO:tasks.workunit.client.0.smithi040.stderr: | ^ 2024-04-21T23:20:35.730 INFO:tasks.workunit.client.0.smithi040.stderr: | ; 2024-04-21T23:20:35.731 INFO:tasks.workunit.client.0.smithi040.stderr:alloc.c:128:25: warning: variable 'f' set but not used [-Wunused-but-set-variable] 2024-04-21T23:20:35.731 INFO:tasks.workunit.client.0.smithi040.stderr: 128 | struct flock64 f; 2024-04-21T23:20:35.731 INFO:tasks.workunit.client.0.smithi040.stderr: | ^ 2024-04-21T23:20:35.733 INFO:tasks.workunit.client.0.smithi040.stderr:gmake[3]: *** [Makefile:83: alloc] Error 1 2024-04-21T23:20:35.733 INFO:tasks.workunit.client.0.smithi040.stderr:gmake[3]: *** Waiting for unfinished jobs.... 2024-04-21T23:20:35.837 INFO:tasks.workunit.client.0.smithi040.stderr:fstest.c: In function 'run_child': 2024-04-21T23:20:35.837 INFO:tasks.workunit.client.0.smithi040.stderr:fstest.c:95:27: warning: '/file' directive writing 5 bytes into a region of size between 1 and 1024 [-Wformat-overflow=] 2024-04-21T23:20:35.837 INFO:tasks.workunit.client.0.smithi040.stderr: 95 | sprintf(fname, "%s/file%d", dir, fnum); 2024-04-21T23:20:35.837 INFO:tasks.workunit.client.0.smithi040.stderr: | ^~~~~ 2024-04-21T23:20:35.838 INFO:tasks.workunit.client.0.smithi040.stderr:fstest.c:95:24: note: directive argument in the range [0, 2147483647] 2024-04-21T23:20:35.838 INFO:tasks.workunit.client.0.smithi040.stderr: 95 | sprintf(fname, "%s/file%d", dir, fnum); 2024-04-21T23:20:35.838 INFO:tasks.workunit.client.0.smithi040.stderr: | ^~~~~~~~~~~ 2024-04-21T23:20:35.838 INFO:tasks.workunit.client.0.smithi040.stderr:fstest.c:95:9: note: 'sprintf' output between 7 and 1039 bytes into a destination of size 1024 2024-04-21T23:20:35.838 INFO:tasks.workunit.client.0.smithi040.stderr: 95 | sprintf(fname, "%s/file%d", dir, fnum); 2024-04-21T23:20:35.838 INFO:tasks.workunit.client.0.smithi040.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-21T23:20:35.838 INFO:tasks.workunit.client.0.smithi040.stderr:fstest.c:166:27: warning: '/file' directive writing 5 bytes into a region of size between 1 and 1024 [-Wformat-overflow=] 2024-04-21T23:20:35.838 INFO:tasks.workunit.client.0.smithi040.stderr: 166 | sprintf(fname, "%s/file%d", dir, fnum); 2024-04-21T23:20:35.838 INFO:tasks.workunit.client.0.smithi040.stderr: | ^~~~~ 2024-04-21T23:20:35.838 INFO:tasks.workunit.client.0.smithi040.stderr:fstest.c:166:24: note: directive argument in the range [0, 2147483647] 2024-04-21T23:20:35.838 INFO:tasks.workunit.client.0.smithi040.stderr: 166 | sprintf(fname, "%s/file%d", dir, fnum); 2024-04-21T23:20:35.839 INFO:tasks.workunit.client.0.smithi040.stderr: | ^~~~~~~~~~~ 2024-04-21T23:20:35.839 INFO:tasks.workunit.client.0.smithi040.stderr:fstest.c:166:9: note: 'sprintf' output between 7 and 1039 bytes into a destination of size 1024 2024-04-21T23:20:35.839 INFO:tasks.workunit.client.0.smithi040.stderr: 166 | sprintf(fname, "%s/file%d", dir, fnum); 2024-04-21T23:20:35.839 INFO:tasks.workunit.client.0.smithi040.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-21T23:20:35.910 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-04-21T23:20:35.910 INFO:tasks.workunit.client.0.smithi040.stderr:gmake[2]: *** [include/buildrules:30: src] Error 2 2024-04-21T23:20:35.910 INFO:tasks.workunit.client.0.smithi040.stderr:make[1]: *** [Makefile:53: default] Error 2 2024-04-21T23:20:35.910 INFO:tasks.workunit.client.0.smithi040.stderr:make: *** [Makefile:51: default] Error 2 2024-04-21T23:20:35.911 INFO:tasks.workunit:Stopping ['suites/fsx.sh'] on client.0... 2024-04-21T23:20:35.911 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-21T23:20:36.221 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_430e09df97c8fc7dc2b2ae424f68ed11366c540f/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph_430e09df97c8fc7dc2b2ae424f68ed11366c540f/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_430e09df97c8fc7dc2b2ae424f68ed11366c540f/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/fsx.sh) on smithi040 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=430e09df97c8fc7dc2b2ae424f68ed11366c540f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh' 2024-04-21T23:20:36.449 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=44d0a6eb22c24cdbae7c8e078c780657 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_430e09df97c8fc7dc2b2ae424f68ed11366c540f/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph_430e09df97c8fc7dc2b2ae424f68ed11366c540f/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_430e09df97c8fc7dc2b2ae424f68ed11366c540f/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/fsx.sh) on smithi040 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=430e09df97c8fc7dc2b2ae424f68ed11366c540f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh' 2024-04-21T23:20:36.452 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-04-21T23:20:36.462 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi040.front.sepia.ceph.com 2024-04-21T23:20:36.475 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-04-21T23:20:36.499 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-04-21T23:20:36.500 INFO:teuthology.task.exec:Executing custom commands... 2024-04-21T23:20:36.500 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi040.front.sepia.ceph.com 2024-04-21T23:20:36.500 DEBUG:teuthology.orchestra.run.smithi040:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-04-21T23:20:36.543 DEBUG:teuthology.orchestra.run.smithi040:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-04-21T23:20:37.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:37 smithi040 ceph-mon[27984]: pgmap v69: 129 pgs: 129 active+clean; 95 MiB data, 781 MiB used, 1.0 TiB / 1.0 TiB avail; 960 KiB/s rd, 802 KiB/s wr, 337 op/s 2024-04-21T23:20:37.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:37 smithi174 ceph-mon[31409]: pgmap v69: 129 pgs: 129 active+clean; 95 MiB data, 781 MiB used, 1.0 TiB / 1.0 TiB avail; 960 KiB/s rd, 802 KiB/s wr, 337 op/s 2024-04-21T23:20:37.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:37 smithi111 ceph-mon[31085]: pgmap v69: 129 pgs: 129 active+clean; 95 MiB data, 781 MiB used, 1.0 TiB / 1.0 TiB avail; 960 KiB/s rd, 802 KiB/s wr, 337 op/s 2024-04-21T23:20:38.757 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T23:20:38.757 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T23:20:38.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.878+0000 7f3f99242640 1 -- 172.21.15.40:0/1402804482 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f94100620 msgr2=0x7f3f94100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:38.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.878+0000 7f3f99242640 1 --2- 172.21.15.40:0/1402804482 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f94100620 0x7f3f94100aa0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f3f88009930 tx=0x7f3f8802bc30 comp rx=0 tx=0).stop 2024-04-21T23:20:38.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.878+0000 7f3f99242640 1 -- 172.21.15.40:0/1402804482 shutdown_connections 2024-04-21T23:20:38.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.878+0000 7f3f99242640 1 --2- 172.21.15.40:0/1402804482 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f94100fe0 0x7f3f9410d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:38.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.878+0000 7f3f99242640 1 --2- 172.21.15.40:0/1402804482 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f94100620 0x7f3f94100aa0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:38.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.878+0000 7f3f99242640 1 --2- 172.21.15.40:0/1402804482 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f94106630 0x7f3f94106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:38.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.878+0000 7f3f99242640 1 -- 172.21.15.40:0/1402804482 >> 172.21.15.40:0/1402804482 conn(0x7f3f940776a0 msgr2=0x7f3f94077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:38.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.878+0000 7f3f99242640 1 -- 172.21.15.40:0/1402804482 shutdown_connections 2024-04-21T23:20:38.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.878+0000 7f3f99242640 1 -- 172.21.15.40:0/1402804482 wait complete. 2024-04-21T23:20:38.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.878+0000 7f3f99242640 1 Processor -- start 2024-04-21T23:20:38.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.879+0000 7f3f99242640 1 -- start start 2024-04-21T23:20:38.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.879+0000 7f3f99242640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f94100620 0x7f3f940fd650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:38.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.879+0000 7f3f99242640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f94100fe0 0x7f3f940fdb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:38.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.879+0000 7f3f99242640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f94106630 0x7f3f940fe0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:38.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.879+0000 7f3f99242640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3f94110a40 con 0x7f3f94100fe0 2024-04-21T23:20:38.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.879+0000 7f3f99242640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3f941108c0 con 0x7f3f94100620 2024-04-21T23:20:38.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.879+0000 7f3f99242640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3f94110bc0 con 0x7f3f94106630 2024-04-21T23:20:38.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.879+0000 7f3f9259c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f94100fe0 0x7f3f940fdb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:38.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.879+0000 7f3f9259c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f94100fe0 0x7f3f940fdb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42378/0 (socket says 172.21.15.40:42378) 2024-04-21T23:20:38.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.879+0000 7f3f9259c640 1 -- 172.21.15.40:0/663592717 learned_addr learned my addr 172.21.15.40:0/663592717 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:20:38.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.879+0000 7f3f9359e640 1 --2- 172.21.15.40:0/663592717 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f94106630 0x7f3f940fe0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:38.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.879+0000 7f3f92d9d640 1 --2- 172.21.15.40:0/663592717 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f94100620 0x7f3f940fd650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:38.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.879+0000 7f3f9259c640 1 -- 172.21.15.40:0/663592717 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f94106630 msgr2=0x7f3f940fe0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:38.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.879+0000 7f3f9259c640 1 --2- 172.21.15.40:0/663592717 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f94106630 0x7f3f940fe0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:38.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.879+0000 7f3f9259c640 1 -- 172.21.15.40:0/663592717 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f94100620 msgr2=0x7f3f940fd650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:38.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.879+0000 7f3f9259c640 1 --2- 172.21.15.40:0/663592717 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f94100620 0x7f3f940fd650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:38.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.879+0000 7f3f9259c640 1 -- 172.21.15.40:0/663592717 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f941a65d0 con 0x7f3f94100fe0 2024-04-21T23:20:38.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.880+0000 7f3f9259c640 1 --2- 172.21.15.40:0/663592717 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f94100fe0 0x7f3f940fdb90 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f3f88009230 tx=0x7f3f8802cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:38.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.880+0000 7f3f7bfff640 1 -- 172.21.15.40:0/663592717 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f88041070 con 0x7f3f94100fe0 2024-04-21T23:20:38.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.880+0000 7f3f7bfff640 1 -- 172.21.15.40:0/663592717 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f88004450 con 0x7f3f94100fe0 2024-04-21T23:20:38.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.880+0000 7f3f99242640 1 -- 172.21.15.40:0/663592717 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f941a6860 con 0x7f3f94100fe0 2024-04-21T23:20:38.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.880+0000 7f3f7bfff640 1 -- 172.21.15.40:0/663592717 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f88034440 con 0x7f3f94100fe0 2024-04-21T23:20:38.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.880+0000 7f3f99242640 1 -- 172.21.15.40:0/663592717 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f941a6d50 con 0x7f3f94100fe0 2024-04-21T23:20:38.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.882+0000 7f3f7bfff640 1 -- 172.21.15.40:0/663592717 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f3f88044070 con 0x7f3f94100fe0 2024-04-21T23:20:38.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.883+0000 7f3f9be1f740 1 Processor -- start 2024-04-21T23:20:38.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.883+0000 7f3f9be1f740 1 -- start start 2024-04-21T23:20:38.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.883+0000 7f3f9be1f740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c56a96f30 0x555c56a97310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:38.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.883+0000 7f3f9be1f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555c56a97850 0x555c56a9fef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:38.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.883+0000 7f3f7bfff640 1 --2- 172.21.15.40:0/663592717 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f3f7c0a2d00 0x7f3f7c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:38.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.883+0000 7f3f9be1f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555c56aa0430 0x555c56aa2820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:38.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.883+0000 7f3f9be1f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x555c569fb560 con 0x555c56a97850 2024-04-21T23:20:38.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.883+0000 7f3f9be1f740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x555c569a3730 con 0x555c56a96f30 2024-04-21T23:20:38.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.883+0000 7f3f9be1f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x555c569f7520 con 0x555c56aa0430 2024-04-21T23:20:38.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.883+0000 7f3f92d9d640 1 --2- 172.21.15.40:0/663592717 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f3f7c0a2d00 0x7f3f7c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:38.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.883+0000 7f3f9359e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c56a96f30 0x555c56a97310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:38.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.883+0000 7f3f9259c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555c56a97850 0x555c56a9fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:38.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.883+0000 7f3f9359e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c56a96f30 0x555c56a97310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.40:33304/0 (socket says 172.21.15.40:33304) 2024-04-21T23:20:38.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.883+0000 7f3f9359e640 1 -- 172.21.15.40:0/3966205547 learned_addr learned my addr 172.21.15.40:0/3966205547 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:20:38.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.883+0000 7f3f7bfff640 1 -- 172.21.15.40:0/663592717 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f3f880f1240 con 0x7f3f94100fe0 2024-04-21T23:20:38.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.883+0000 7f3f9359e640 1 --2- 172.21.15.40:0/3966205547 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555c56aa0430 0x555c56aa2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:38.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.884+0000 7f3f9359e640 1 -- 172.21.15.40:0/3966205547 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555c56aa0430 msgr2=0x555c56aa2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:38.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.884+0000 7f3f9359e640 1 --2- 172.21.15.40:0/3966205547 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555c56aa0430 0x555c56aa2820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:38.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.884+0000 7f3f9359e640 1 -- 172.21.15.40:0/3966205547 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555c56a97850 msgr2=0x555c56a9fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:38.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.884+0000 7f3f9359e640 1 --2- 172.21.15.40:0/3966205547 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555c56a97850 0x555c56a9fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:38.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.884+0000 7f3f9359e640 1 -- 172.21.15.40:0/3966205547 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555c567d97d0 con 0x555c56a96f30 2024-04-21T23:20:38.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.884+0000 7f3f92d9d640 1 --2- 172.21.15.40:0/663592717 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f3f7c0a2d00 0x7f3f7c0a51c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3f84005fd0 tx=0x7f3f8400b510 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:38.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.884+0000 7f3f9359e640 1 --2- 172.21.15.40:0/3966205547 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c56a96f30 0x555c56a97310 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3f8c012820 tx=0x7f3f8c009b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f618380907b09c2b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:38.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.884+0000 7f3f797fa640 1 -- 172.21.15.40:0/3966205547 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f8c012230 con 0x555c56a96f30 2024-04-21T23:20:38.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.884+0000 7f3f797fa640 1 -- 172.21.15.40:0/3966205547 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f8c003e50 con 0x555c56a96f30 2024-04-21T23:20:38.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.884+0000 7f3f797fa640 1 -- 172.21.15.40:0/3966205547 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f8c0043f0 con 0x555c56a96f30 2024-04-21T23:20:38.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.885+0000 7f3f9be1f740 1 -- 172.21.15.40:0/3966205547 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c56a96f30 msgr2=0x555c56a97310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:38.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.885+0000 7f3f9be1f740 1 --2- 172.21.15.40:0/3966205547 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c56a96f30 0x555c56a97310 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3f8c012820 tx=0x7f3f8c009b20 comp rx=0 tx=0).stop 2024-04-21T23:20:38.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.885+0000 7f3f9be1f740 1 -- 172.21.15.40:0/3966205547 shutdown_connections 2024-04-21T23:20:38.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.885+0000 7f3f9be1f740 1 --2- 172.21.15.40:0/3966205547 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555c56aa0430 0x555c56aa2820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:38.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.885+0000 7f3f9be1f740 1 --2- 172.21.15.40:0/3966205547 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555c56a97850 0x555c56a9fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:38.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.885+0000 7f3f9be1f740 1 --2- 172.21.15.40:0/3966205547 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c56a96f30 0x555c56a97310 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:38.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.885+0000 7f3f9be1f740 1 -- 172.21.15.40:0/3966205547 >> 172.21.15.40:0/3966205547 conn(0x555c56a91660 msgr2=0x555c56a91a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:38.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.885+0000 7f3f9be1f740 1 -- 172.21.15.40:0/3966205547 shutdown_connections 2024-04-21T23:20:38.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.885+0000 7f3f9be1f740 1 -- 172.21.15.40:0/3966205547 wait complete. 2024-04-21T23:20:38.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.885+0000 7f3f9be1f740 1 -- start start 2024-04-21T23:20:38.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.885+0000 7f3f9be1f740 1 Processor -- start 2024-04-21T23:20:38.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.885+0000 7f3f9be1f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555c56a96f30 0x555c56a9d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:38.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.885+0000 7f3f9be1f740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c56a97850 0x555c56a9d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:38.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.885+0000 7f3f9be1f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555c56aa0430 0x555c56a9df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:38.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.885+0000 7f3f9be1f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x555c56aa3220 con 0x555c56a96f30 2024-04-21T23:20:38.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.885+0000 7f3f9be1f740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x555c569b6e00 con 0x555c56a97850 2024-04-21T23:20:38.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.886+0000 7f3f9be1f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x555c56aa33a0 con 0x555c56aa0430 2024-04-21T23:20:38.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.886+0000 7f3f92d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555c56a96f30 0x555c56a9d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:38.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.886+0000 7f3f9be1f740 10 client.0 _ll_register_callbacks cb 0x555c569f1eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:20:38.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.886+0000 7f3f92d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555c56a96f30 0x555c56a9d4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42388/0 (socket says 172.21.15.40:42388) 2024-04-21T23:20:38.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.886+0000 7f3f92d9d640 1 -- 172.21.15.40:0/577437399 learned_addr learned my addr 172.21.15.40:0/577437399 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:20:38.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.886+0000 7f3f9359e640 1 --2- 172.21.15.40:0/577437399 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555c56aa0430 0x555c56a9df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:38.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.886+0000 7f3f9259c640 1 --2- 172.21.15.40:0/577437399 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c56a97850 0x555c56a9d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:38.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.886+0000 7f3f92d9d640 1 -- 172.21.15.40:0/577437399 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555c56aa0430 msgr2=0x555c56a9df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:38.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.886+0000 7f3f92d9d640 1 --2- 172.21.15.40:0/577437399 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555c56aa0430 0x555c56a9df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:38.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.886+0000 7f3f92d9d640 1 -- 172.21.15.40:0/577437399 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c56a97850 msgr2=0x555c56a9d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:38.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.886+0000 7f3f92d9d640 1 --2- 172.21.15.40:0/577437399 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c56a97850 0x555c56a9d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:38.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.886+0000 7f3f92d9d640 1 -- 172.21.15.40:0/577437399 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555c5670f3f0 con 0x555c56a96f30 2024-04-21T23:20:38.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.886+0000 7f3f9259c640 1 --2- 172.21.15.40:0/577437399 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c56a97850 0x555c56a9d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:20:38.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.886+0000 7f3f92d9d640 1 --2- 172.21.15.40:0/577437399 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555c56a96f30 0x555c56a9d4b0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f3f8400ffb0 tx=0x7f3f840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e180f0caaa80f37b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:38.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.886+0000 7f3f76ff5640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-21T23:20:38.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.886+0000 7f3f76ff5640 1 -- 172.21.15.40:0/577437399 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f84022070 con 0x555c56a96f30 2024-04-21T23:20:38.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.886+0000 7f3f76ff5640 1 -- 172.21.15.40:0/577437399 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f3f8401e860 con 0x555c56a96f30 2024-04-21T23:20:38.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.886+0000 7f3f9be1f740 1 -- 172.21.15.40:0/577437399 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f8c003e50 con 0x555c56a96f30 2024-04-21T23:20:38.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.886+0000 7f3f76ff5640 1 -- 172.21.15.40:0/577437399 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3f8402e050 con 0x555c56a96f30 2024-04-21T23:20:38.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.887+0000 7f3f9be1f740 10 client.16245 fetch_fsmap learned FSMap version 29 2024-04-21T23:20:38.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.887+0000 7f3f9be1f740 1 -- 172.21.15.40:0/577437399 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x555c566cf0f0 con 0x555c56a96f30 2024-04-21T23:20:38.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.887+0000 7f3f76ff5640 1 -- 172.21.15.40:0/577437399 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f8c003e50 con 0x555c56a96f30 2024-04-21T23:20:38.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.887+0000 7f3f76ff5640 1 -- 172.21.15.40:0/577437399 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f3f84010bc0 con 0x555c56a96f30 2024-04-21T23:20:38.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.887+0000 7f3f9be1f740 10 client.16245 fetch_fsmap finished waiting for FSMap version 29 2024-04-21T23:20:38.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.887+0000 7f3f9be1f740 10 client.16245 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:20:38.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.887+0000 7f3f9be1f740 20 client.16245 populate_metadata read hostname 'smithi040' 2024-04-21T23:20:38.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.887+0000 7f3f9be1f740 1 --2- 172.21.15.40:0/577437399 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x555c56ab0850 0x555c56ab2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:38.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.887+0000 7f3f9be1f740 4 client.16245 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:20:38.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.887+0000 7f3f9be1f740 1 -- 172.21.15.40:0/577437399 --> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555c56a08aa0 con 0x555c56ab0850 2024-04-21T23:20:38.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.887+0000 7f3f9259c640 1 --2- 172.21.15.40:0/577437399 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x555c56ab0850 0x555c56ab2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:38.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.888+0000 7f3f76ff5640 10 client.16245 ms_handle_connect on v2:172.21.15.111:6838/3712235548 2024-04-21T23:20:38.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.888+0000 7f3f9259c640 1 --2- 172.21.15.40:0/577437399 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x555c56ab0850 0x555c56ab2cd0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d0573bc65cd14113 server_cookie=f4ffdbf7bf894b22 in_seq=0 out_seq=0 2024-04-21T23:20:38.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.888+0000 7f3f76ff5640 1 -- 172.21.15.40:0/577437399 <== mds.0 v2:172.21.15.111:6838/3712235548 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555c56a08aa0 con 0x555c56ab0850 2024-04-21T23:20:38.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.888+0000 7f3f76ff5640 10 client.16245 handle_command_reply: tid=0 2024-04-21T23:20:38.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.888+0000 7f3f9be1f740 1 client.16245 shutdown 2024-04-21T23:20:38.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.889+0000 7f3f9be1f740 1 -- 172.21.15.40:0/577437399 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555c56a96f30 msgr2=0x555c56a9d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:38.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.889+0000 7f3f9be1f740 1 --2- 172.21.15.40:0/577437399 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555c56a96f30 0x555c56a9d4b0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f3f8400ffb0 tx=0x7f3f840046c0 comp rx=0 tx=0).stop 2024-04-21T23:20:38.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.889+0000 7f3f9be1f740 1 -- 172.21.15.40:0/577437399 shutdown_connections 2024-04-21T23:20:38.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.889+0000 7f3f9be1f740 1 --2- 172.21.15.40:0/577437399 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x555c56ab0850 0x555c56ab2cd0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:38.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.889+0000 7f3f9be1f740 1 --2- 172.21.15.40:0/577437399 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c56a97850 0x555c56a9d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:38.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.889+0000 7f3f76ff5640 0 client.16245 ms_handle_reset on v2:172.21.15.111:6838/3712235548 2024-04-21T23:20:38.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.889+0000 7f3f9be1f740 1 --2- 172.21.15.40:0/577437399 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555c56aa0430 0x555c56a9df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:38.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.889+0000 7f3f9be1f740 1 --2- 172.21.15.40:0/577437399 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555c56a96f30 0x555c56a9d4b0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:38.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.889+0000 7f3f9be1f740 1 -- 172.21.15.40:0/577437399 >> 172.21.15.40:0/577437399 conn(0x555c56a91660 msgr2=0x555c56a933b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:38.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.889+0000 7f3f9be1f740 1 -- 172.21.15.40:0/577437399 shutdown_connections 2024-04-21T23:20:38.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.889+0000 7f3f9be1f740 1 -- 172.21.15.40:0/577437399 wait complete. 2024-04-21T23:20:38.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.889+0000 7f3f9be1f740 20 client.16245 trim_cache size 0 max 16384 2024-04-21T23:20:38.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.907+0000 7f3f9be1f740 1 Processor -- start 2024-04-21T23:20:38.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.907+0000 7f3f9be1f740 1 -- start start 2024-04-21T23:20:38.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.907+0000 7f3f9be1f740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c56a96f30 0x555c56a9fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:38.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.907+0000 7f3f9be1f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555c56a97850 0x555c56aac0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:38.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.907+0000 7f3f9be1f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555c56aa0430 0x555c56a90560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:38.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.907+0000 7f3f9be1f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3f941b37a0 con 0x555c56a97850 2024-04-21T23:20:38.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.907+0000 7f3f9be1f740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3f941b1010 con 0x555c56a96f30 2024-04-21T23:20:38.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.907+0000 7f3f9be1f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3f941aee60 con 0x555c56aa0430 2024-04-21T23:20:38.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.907+0000 7f3f9259c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555c56a97850 0x555c56aac0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:38.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.907+0000 7f3f9259c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555c56a97850 0x555c56aac0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42398/0 (socket says 172.21.15.40:42398) 2024-04-21T23:20:38.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.907+0000 7f3f9259c640 1 -- 172.21.15.40:0/826689211 learned_addr learned my addr 172.21.15.40:0/826689211 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:20:38.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.907+0000 7f3f92d9d640 1 --2- 172.21.15.40:0/826689211 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c56a96f30 0x555c56a9fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:38.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.907+0000 7f3f9259c640 1 -- 172.21.15.40:0/826689211 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555c56aa0430 msgr2=0x555c56a90560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:38.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.907+0000 7f3f9259c640 1 --2- 172.21.15.40:0/826689211 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555c56aa0430 0x555c56a90560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:38.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.907+0000 7f3f9259c640 1 -- 172.21.15.40:0/826689211 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c56a96f30 msgr2=0x555c56a9fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:38.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.907+0000 7f3f9259c640 1 --2- 172.21.15.40:0/826689211 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c56a96f30 0x555c56a9fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:38.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.907+0000 7f3f9259c640 1 -- 172.21.15.40:0/826689211 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555c56aa2820 con 0x555c56a97850 2024-04-21T23:20:38.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.907+0000 7f3f92d9d640 1 --2- 172.21.15.40:0/826689211 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c56a96f30 0x555c56a9fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:20:38.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.907+0000 7f3f9259c640 1 --2- 172.21.15.40:0/826689211 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555c56a97850 0x555c56aac0e0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f3f880b7fd0 tx=0x7f3f8807a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef2063e4327e5353 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:38.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.908+0000 7f3f78ff9640 1 -- 172.21.15.40:0/826689211 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f8804b040 con 0x555c56a97850 2024-04-21T23:20:38.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.908+0000 7f3f78ff9640 1 -- 172.21.15.40:0/826689211 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f880bdb60 con 0x555c56a97850 2024-04-21T23:20:38.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.908+0000 7f3f78ff9640 1 -- 172.21.15.40:0/826689211 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f880bb7f0 con 0x555c56a97850 2024-04-21T23:20:38.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.908+0000 7f3f9be1f740 1 -- 172.21.15.40:0/826689211 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555c56a97850 msgr2=0x555c56aac0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:38.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.908+0000 7f3f9be1f740 1 --2- 172.21.15.40:0/826689211 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555c56a97850 0x555c56aac0e0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f3f880b7fd0 tx=0x7f3f8807a040 comp rx=0 tx=0).stop 2024-04-21T23:20:38.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.908+0000 7f3f9be1f740 1 -- 172.21.15.40:0/826689211 shutdown_connections 2024-04-21T23:20:38.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.908+0000 7f3f9be1f740 1 --2- 172.21.15.40:0/826689211 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555c56aa0430 0x555c56a90560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:38.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.908+0000 7f3f9be1f740 1 --2- 172.21.15.40:0/826689211 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555c56a97850 0x555c56aac0e0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:38.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.908+0000 7f3f9be1f740 1 --2- 172.21.15.40:0/826689211 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c56a96f30 0x555c56a9fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:38.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.908+0000 7f3f9be1f740 1 -- 172.21.15.40:0/826689211 >> 172.21.15.40:0/826689211 conn(0x555c56a91660 msgr2=0x555c56a8fe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:38.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.908+0000 7f3f9be1f740 1 -- 172.21.15.40:0/826689211 shutdown_connections 2024-04-21T23:20:38.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.908+0000 7f3f9be1f740 1 -- 172.21.15.40:0/826689211 wait complete. 2024-04-21T23:20:38.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.908+0000 7f3f9be1f740 1 -- start start 2024-04-21T23:20:38.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.909+0000 7f3f9be1f740 1 Processor -- start 2024-04-21T23:20:38.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.909+0000 7f3f9be1f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555c56a96f30 0x555c56aa4310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:38.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.909+0000 7f3f9be1f740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c56a97850 0x555c56aa4850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:38.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.909+0000 7f3f9be1f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555c56aa0430 0x555c56b15e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:38.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.909+0000 7f3f9be1f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3f941af4d0 con 0x555c56aa0430 2024-04-21T23:20:38.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.909+0000 7f3f9be1f740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3f941b0940 con 0x555c56a97850 2024-04-21T23:20:38.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.909+0000 7f3f9be1f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3f941afba0 con 0x555c56a96f30 2024-04-21T23:20:38.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.909+0000 7f3f9be1f740 10 client.0 _ll_register_callbacks cb 0x7f3f84015070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:20:38.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.909+0000 7f3f92d9d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555c56a96f30 0x555c56aa4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:38.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.909+0000 7f3f92d9d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555c56a96f30 0x555c56aa4310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.40:40718/0 (socket says 172.21.15.40:40718) 2024-04-21T23:20:38.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.909+0000 7f3f92d9d640 1 -- 172.21.15.40:0/3457767835 learned_addr learned my addr 172.21.15.40:0/3457767835 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:20:38.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.909+0000 7f3f9359e640 1 --2- 172.21.15.40:0/3457767835 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555c56aa0430 0x555c56b15e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:38.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.909+0000 7f3f9259c640 1 --2- 172.21.15.40:0/3457767835 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c56a97850 0x555c56aa4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:38.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.909+0000 7f3f92d9d640 1 -- 172.21.15.40:0/3457767835 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c56a97850 msgr2=0x555c56aa4850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:38.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.909+0000 7f3f92d9d640 1 --2- 172.21.15.40:0/3457767835 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c56a97850 0x555c56aa4850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:38.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.909+0000 7f3f92d9d640 1 -- 172.21.15.40:0/3457767835 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555c56aa0430 msgr2=0x555c56b15e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:38.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.909+0000 7f3f92d9d640 1 --2- 172.21.15.40:0/3457767835 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555c56aa0430 0x555c56b15e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:38.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.909+0000 7f3f92d9d640 1 -- 172.21.15.40:0/3457767835 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555c56aa50e0 con 0x555c56a96f30 2024-04-21T23:20:38.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.910+0000 7f3f92d9d640 1 --2- 172.21.15.40:0/3457767835 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555c56a96f30 0x555c56aa4310 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3f8400b750 tx=0x7f3f84026700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e98c626f58f702c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:38.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.910+0000 7f3f76ff5640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-04-21T23:20:38.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.910+0000 7f3f76ff5640 1 -- 172.21.15.40:0/3457767835 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f84036070 con 0x555c56a96f30 2024-04-21T23:20:38.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.910+0000 7f3f9be1f740 1 -- 172.21.15.40:0/3457767835 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f880bdb60 con 0x555c56a96f30 2024-04-21T23:20:38.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.910+0000 7f3f76ff5640 1 -- 172.21.15.40:0/3457767835 <== mon.1 v2:172.21.15.174:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f3f84026d80 con 0x555c56a96f30 2024-04-21T23:20:38.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.910+0000 7f3f76ff5640 1 -- 172.21.15.40:0/3457767835 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3f84042050 con 0x555c56a96f30 2024-04-21T23:20:38.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.910+0000 7f3f9be1f740 10 client.25699 fetch_fsmap learned FSMap version 29 2024-04-21T23:20:38.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.910+0000 7f3f9be1f740 1 -- 172.21.15.40:0/3457767835 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x555c56b173a0 con 0x555c56a96f30 2024-04-21T23:20:38.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.910+0000 7f3f76ff5640 1 -- 172.21.15.40:0/3457767835 <== mon.1 v2:172.21.15.174:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f880bdb60 con 0x555c56a96f30 2024-04-21T23:20:38.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.910+0000 7f3f76ff5640 1 -- 172.21.15.40:0/3457767835 <== mon.1 v2:172.21.15.174:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f3f84004950 con 0x555c56a96f30 2024-04-21T23:20:38.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.910+0000 7f3f9be1f740 10 client.25699 fetch_fsmap finished waiting for FSMap version 29 2024-04-21T23:20:38.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.910+0000 7f3f9be1f740 10 client.25699 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:20:38.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.910+0000 7f3f9be1f740 20 client.25699 populate_metadata read hostname 'smithi040' 2024-04-21T23:20:38.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.910+0000 7f3f9be1f740 1 --2- 172.21.15.40:0/3457767835 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x555c56b17a90 0x555c56b19eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:38.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.910+0000 7f3f9be1f740 4 client.25699 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T23:20:38.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.910+0000 7f3f9be1f740 1 -- 172.21.15.40:0/3457767835 --> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] -- command(tid 0: {"prefix": "scrub status"}) -- 0x555c56a02df0 con 0x555c56b17a90 2024-04-21T23:20:38.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.911+0000 7f3f9259c640 1 --2- 172.21.15.40:0/3457767835 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x555c56b17a90 0x555c56b19eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:38.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.911+0000 7f3f76ff5640 10 client.25699 ms_handle_connect on v2:172.21.15.111:6838/3712235548 2024-04-21T23:20:38.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.911+0000 7f3f9259c640 1 --2- 172.21.15.40:0/3457767835 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x555c56b17a90 0x555c56b19eb0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e62f810ed8c977f0 server_cookie=843c834730ad9ab5 in_seq=0 out_seq=0 2024-04-21T23:20:38.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.911+0000 7f3f76ff5640 1 -- 172.21.15.40:0/3457767835 <== mds.0 v2:172.21.15.111:6838/3712235548 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x555c56a02df0 con 0x555c56b17a90 2024-04-21T23:20:38.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.911+0000 7f3f76ff5640 10 client.25699 handle_command_reply: tid=0 2024-04-21T23:20:38.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.911+0000 7f3f9be1f740 1 client.25699 shutdown 2024-04-21T23:20:38.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.911+0000 7f3f9be1f740 1 -- 172.21.15.40:0/3457767835 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555c56a96f30 msgr2=0x555c56aa4310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:38.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.911+0000 7f3f9be1f740 1 --2- 172.21.15.40:0/3457767835 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555c56a96f30 0x555c56aa4310 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3f8400b750 tx=0x7f3f84026700 comp rx=0 tx=0).stop 2024-04-21T23:20:38.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.911+0000 7f3f9be1f740 1 -- 172.21.15.40:0/3457767835 shutdown_connections 2024-04-21T23:20:38.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.911+0000 7f3f9be1f740 1 --2- 172.21.15.40:0/3457767835 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x555c56b17a90 0x555c56b19eb0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:38.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.911+0000 7f3f9be1f740 1 --2- 172.21.15.40:0/3457767835 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c56a97850 0x555c56aa4850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:38.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.911+0000 7f3f9be1f740 1 --2- 172.21.15.40:0/3457767835 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555c56aa0430 0x555c56b15e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:38.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.911+0000 7f3f9be1f740 1 --2- 172.21.15.40:0/3457767835 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555c56a96f30 0x555c56aa4310 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:38.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.911+0000 7f3f76ff5640 0 client.25699 ms_handle_reset on v2:172.21.15.111:6838/3712235548 2024-04-21T23:20:38.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.911+0000 7f3f9be1f740 1 -- 172.21.15.40:0/3457767835 >> 172.21.15.40:0/3457767835 conn(0x555c56a91660 msgr2=0x555c56a906e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:38.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.911+0000 7f3f9be1f740 1 -- 172.21.15.40:0/3457767835 shutdown_connections 2024-04-21T23:20:38.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.911+0000 7f3f9be1f740 1 -- 172.21.15.40:0/3457767835 wait complete. 2024-04-21T23:20:38.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.911+0000 7f3f9be1f740 20 client.25699 trim_cache size 0 max 16384 2024-04-21T23:20:38.922 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-21T23:20:38.922 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "no active scrubs running", 2024-04-21T23:20:38.922 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": {} 2024-04-21T23:20:38.922 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-21T23:20:38.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.912+0000 7f3f99242640 1 -- 172.21.15.40:0/663592717 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f3f7c0a2d00 msgr2=0x7f3f7c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:38.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.912+0000 7f3f99242640 1 --2- 172.21.15.40:0/663592717 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f3f7c0a2d00 0x7f3f7c0a51c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3f84005fd0 tx=0x7f3f8400b510 comp rx=0 tx=0).stop 2024-04-21T23:20:38.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.912+0000 7f3f99242640 1 -- 172.21.15.40:0/663592717 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f94100fe0 msgr2=0x7f3f940fdb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:38.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.912+0000 7f3f99242640 1 --2- 172.21.15.40:0/663592717 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f94100fe0 0x7f3f940fdb90 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f3f88009230 tx=0x7f3f8802cc70 comp rx=0 tx=0).stop 2024-04-21T23:20:38.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.912+0000 7f3f99242640 1 -- 172.21.15.40:0/663592717 shutdown_connections 2024-04-21T23:20:38.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.912+0000 7f3f99242640 1 --2- 172.21.15.40:0/663592717 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f3f7c0a2d00 0x7f3f7c0a51c0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:38.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.912+0000 7f3f99242640 1 --2- 172.21.15.40:0/663592717 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f94106630 0x7f3f940fe0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:38.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.912+0000 7f3f99242640 1 --2- 172.21.15.40:0/663592717 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f94100fe0 0x7f3f940fdb90 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:38.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.912+0000 7f3f99242640 1 --2- 172.21.15.40:0/663592717 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f94100620 0x7f3f940fd650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:38.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.912+0000 7f3f99242640 1 -- 172.21.15.40:0/663592717 >> 172.21.15.40:0/663592717 conn(0x7f3f940776a0 msgr2=0x7f3f9406a180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:38.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.912+0000 7f3f99242640 1 -- 172.21.15.40:0/663592717 shutdown_connections 2024-04-21T23:20:38.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:38.912+0000 7f3f99242640 1 -- 172.21.15.40:0/663592717 wait complete. 2024-04-21T23:20:38.925 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T23:20:38.926 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:20:39.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.029+0000 7fd8ce7a1640 1 -- 172.21.15.40:0/96507274 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8c8101710 msgr2=0x7fd8c8101b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:39.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.029+0000 7fd8ce7a1640 1 --2- 172.21.15.40:0/96507274 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8c8101710 0x7fd8c8101b90 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fd8b0009930 tx=0x7fd8b002bc30 comp rx=0 tx=0).stop 2024-04-21T23:20:39.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.029+0000 7fd8ce7a1640 1 -- 172.21.15.40:0/96507274 shutdown_connections 2024-04-21T23:20:39.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.029+0000 7fd8ce7a1640 1 --2- 172.21.15.40:0/96507274 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8c81020d0 0x7fd8c810e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.029+0000 7fd8ce7a1640 1 --2- 172.21.15.40:0/96507274 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8c8101710 0x7fd8c8101b90 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.029+0000 7fd8ce7a1640 1 --2- 172.21.15.40:0/96507274 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd8c8107720 0x7fd8c8107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.029+0000 7fd8ce7a1640 1 -- 172.21.15.40:0/96507274 >> 172.21.15.40:0/96507274 conn(0x7fd8c8078b00 msgr2=0x7fd8c8078f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:39.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.029+0000 7fd8ce7a1640 1 -- 172.21.15.40:0/96507274 shutdown_connections 2024-04-21T23:20:39.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.029+0000 7fd8ce7a1640 1 -- 172.21.15.40:0/96507274 wait complete. 2024-04-21T23:20:39.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.029+0000 7fd8ce7a1640 1 Processor -- start 2024-04-21T23:20:39.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.030+0000 7fd8ce7a1640 1 -- start start 2024-04-21T23:20:39.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.030+0000 7fd8ce7a1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd8c8101710 0x7fd8c8073a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:39.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.030+0000 7fd8ce7a1640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8c81020d0 0x7fd8c8073f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:39.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.030+0000 7fd8ce7a1640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8c8107720 0x7fd8c80744c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:39.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.030+0000 7fd8ce7a1640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c8111b30 con 0x7fd8c81020d0 2024-04-21T23:20:39.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.030+0000 7fd8ce7a1640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c81119b0 con 0x7fd8c8107720 2024-04-21T23:20:39.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.030+0000 7fd8ce7a1640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c8111cb0 con 0x7fd8c8101710 2024-04-21T23:20:39.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.030+0000 7fd8c77fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8c81020d0 0x7fd8c8073f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:39.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.030+0000 7fd8c77fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8c81020d0 0x7fd8c8073f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42430/0 (socket says 172.21.15.40:42430) 2024-04-21T23:20:39.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.030+0000 7fd8c77fe640 1 -- 172.21.15.40:0/2285967261 learned_addr learned my addr 172.21.15.40:0/2285967261 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:20:39.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.030+0000 7fd8c7fff640 1 --2- 172.21.15.40:0/2285967261 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd8c8101710 0x7fd8c8073a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:39.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.030+0000 7fd8c77fe640 1 -- 172.21.15.40:0/2285967261 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd8c8101710 msgr2=0x7fd8c8073a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:39.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.030+0000 7fd8c77fe640 1 --2- 172.21.15.40:0/2285967261 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd8c8101710 0x7fd8c8073a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.030+0000 7fd8c77fe640 1 -- 172.21.15.40:0/2285967261 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8c8107720 msgr2=0x7fd8c80744c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:39.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.030+0000 7fd8c77fe640 1 --2- 172.21.15.40:0/2285967261 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8c8107720 0x7fd8c80744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.030+0000 7fd8c77fe640 1 -- 172.21.15.40:0/2285967261 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8c81a6590 con 0x7fd8c81020d0 2024-04-21T23:20:39.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.030+0000 7fd8c77fe640 1 --2- 172.21.15.40:0/2285967261 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8c81020d0 0x7fd8c8073f80 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fd8b0009230 tx=0x7fd8b002cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:39.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.031+0000 7fd8c57fa640 1 -- 172.21.15.40:0/2285967261 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b0041070 con 0x7fd8c81020d0 2024-04-21T23:20:39.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.031+0000 7fd8c57fa640 1 -- 172.21.15.40:0/2285967261 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd8b0004450 con 0x7fd8c81020d0 2024-04-21T23:20:39.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.031+0000 7fd8ce7a1640 1 -- 172.21.15.40:0/2285967261 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8c81a6820 con 0x7fd8c81020d0 2024-04-21T23:20:39.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.031+0000 7fd8c57fa640 1 -- 172.21.15.40:0/2285967261 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b0034440 con 0x7fd8c81020d0 2024-04-21T23:20:39.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.031+0000 7fd8ce7a1640 1 -- 172.21.15.40:0/2285967261 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8c81a6d10 con 0x7fd8c81020d0 2024-04-21T23:20:39.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.032+0000 7fd8ce7a1640 1 -- 172.21.15.40:0/2285967261 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8c80689b0 con 0x7fd8c81020d0 2024-04-21T23:20:39.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.033+0000 7fd8c57fa640 1 -- 172.21.15.40:0/2285967261 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fd8b0044070 con 0x7fd8c81020d0 2024-04-21T23:20:39.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.033+0000 7fd8c57fa640 1 --2- 172.21.15.40:0/2285967261 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fd8b40a2d00 0x7fd8b40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:39.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.034+0000 7fd8c7fff640 1 --2- 172.21.15.40:0/2285967261 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fd8b40a2d00 0x7fd8b40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:39.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.034+0000 7fd8c57fa640 1 -- 172.21.15.40:0/2285967261 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7fd8b00f12e0 con 0x7fd8c81020d0 2024-04-21T23:20:39.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.034+0000 7fd8c7fff640 1 --2- 172.21.15.40:0/2285967261 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fd8b40a2d00 0x7fd8b40a51c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd8b8005fd0 tx=0x7fd8b800b510 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:39.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.038+0000 7fd8c57fa640 1 -- 172.21.15.40:0/2285967261 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fd8b00bdc00 con 0x7fd8c81020d0 2024-04-21T23:20:39.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.267+0000 7fd8ce7a1640 1 -- 172.21.15.40:0/2285967261 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd8c8074f70 con 0x7fd8c81020d0 2024-04-21T23:20:39.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.268+0000 7fd8c57fa640 1 -- 172.21.15.40:0/2285967261 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11888 (secure 0 0 0) 0x7fd8b00bd500 con 0x7fd8c81020d0 2024-04-21T23:20:39.267 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:20:39.268 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/1571948830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1571948830},{"type":"v1","addr":"172.21.15.40:6835","nonce":1571948830}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/969356642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":969356642},{"type":"v1","addr":"172.21.15.40:6837","nonce":969356642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6839/3025057845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":3025057845},{"type":"v1","addr":"172.21.15.40:6839","nonce":3025057845}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6841/3350488804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":3350488804},{"type":"v1","addr":"172.21.15.40:6841","nonce":3350488804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/3438493203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":3438493203},{"type":"v1","addr":"172.21.15.174:6833","nonce":3438493203}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/3105752678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":3105752678},{"type":"v1","addr":"172.21.15.111:6833","nonce":3105752678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T23:15:55.742233+0000","modified":"2024-04-21T23:20:17.512434+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24521,"mds_1":24502,"mds_2":24494},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24461":{"gid":24461,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6835/2704462092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2704462092},{"type":"v1","addr":"172.21.15.111:6835","nonce":2704462092}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/3905733045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3905733045},{"type":"v1","addr":"172.21.15.174:6835","nonce":3905733045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6837/465579746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":465579746},{"type":"v1","addr":"172.21.15.174:6837","nonce":465579746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":71,"addr":"172.21.15.111:6837/2703764182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":2703764182},{"type":"v1","addr":"172.21.15.111:6837","nonce":2703764182}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24502":{"gid":24502,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.174:6839/2467509274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":2467509274},{"type":"v1","addr":"172.21.15.174:6839","nonce":2467509274}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3712235548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3712235548},{"type":"v1","addr":"172.21.15.111:6839","nonce":3712235548}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24502,24521,24494]},"id":1}]} 2024-04-21T23:20:39.268 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 29 2024-04-21T23:20:39.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.270+0000 7fd8ce7a1640 1 -- 172.21.15.40:0/2285967261 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fd8b40a2d00 msgr2=0x7fd8b40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:39.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.270+0000 7fd8ce7a1640 1 --2- 172.21.15.40:0/2285967261 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fd8b40a2d00 0x7fd8b40a51c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd8b8005fd0 tx=0x7fd8b800b510 comp rx=0 tx=0).stop 2024-04-21T23:20:39.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.270+0000 7fd8ce7a1640 1 -- 172.21.15.40:0/2285967261 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8c81020d0 msgr2=0x7fd8c8073f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:39.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.270+0000 7fd8ce7a1640 1 --2- 172.21.15.40:0/2285967261 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8c81020d0 0x7fd8c8073f80 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fd8b0009230 tx=0x7fd8b002cc70 comp rx=0 tx=0).stop 2024-04-21T23:20:39.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.270+0000 7fd8ce7a1640 1 -- 172.21.15.40:0/2285967261 shutdown_connections 2024-04-21T23:20:39.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.270+0000 7fd8ce7a1640 1 --2- 172.21.15.40:0/2285967261 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fd8b40a2d00 0x7fd8b40a51c0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.270+0000 7fd8ce7a1640 1 --2- 172.21.15.40:0/2285967261 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8c8107720 0x7fd8c80744c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.270+0000 7fd8ce7a1640 1 --2- 172.21.15.40:0/2285967261 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8c81020d0 0x7fd8c8073f80 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.270+0000 7fd8ce7a1640 1 --2- 172.21.15.40:0/2285967261 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd8c8101710 0x7fd8c8073a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.270+0000 7fd8ce7a1640 1 -- 172.21.15.40:0/2285967261 >> 172.21.15.40:0/2285967261 conn(0x7fd8c8078b00 msgr2=0x7fd8c81042c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:39.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.270+0000 7fd8ce7a1640 1 -- 172.21.15.40:0/2285967261 shutdown_connections 2024-04-21T23:20:39.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.270+0000 7fd8ce7a1640 1 -- 172.21.15.40:0/2285967261 wait complete. 2024-04-21T23:20:39.281 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-21T23:20:39.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.387+0000 7f4a18d8c640 1 -- 172.21.15.40:0/3492794111 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a14106630 msgr2=0x7f4a14106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:39.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.387+0000 7f4a18d8c640 1 --2- 172.21.15.40:0/3492794111 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a14106630 0x7f4a14106a10 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f4a04009930 tx=0x7f4a0402bc30 comp rx=0 tx=0).stop 2024-04-21T23:20:39.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.387+0000 7f4a18d8c640 1 -- 172.21.15.40:0/3492794111 shutdown_connections 2024-04-21T23:20:39.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.387+0000 7f4a18d8c640 1 --2- 172.21.15.40:0/3492794111 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a14100fe0 0x7f4a1410d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.387+0000 7f4a18d8c640 1 --2- 172.21.15.40:0/3492794111 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4a14100620 0x7f4a14100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.387+0000 7f4a18d8c640 1 --2- 172.21.15.40:0/3492794111 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a14106630 0x7f4a14106a10 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.387+0000 7f4a18d8c640 1 -- 172.21.15.40:0/3492794111 >> 172.21.15.40:0/3492794111 conn(0x7f4a140776a0 msgr2=0x7f4a14077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:39.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.387+0000 7f4a18d8c640 1 -- 172.21.15.40:0/3492794111 shutdown_connections 2024-04-21T23:20:39.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.387+0000 7f4a18d8c640 1 -- 172.21.15.40:0/3492794111 wait complete. 2024-04-21T23:20:39.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.387+0000 7f4a18d8c640 1 Processor -- start 2024-04-21T23:20:39.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.387+0000 7f4a18d8c640 1 -- start start 2024-04-21T23:20:39.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.388+0000 7f4a18d8c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4a14100620 0x7f4a1410df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:39.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.388+0000 7f4a18d8c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a14100fe0 0x7f4a1410e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:39.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.388+0000 7f4a18d8c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a14106630 0x7f4a1410ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:39.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.388+0000 7f4a18d8c640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4a14111bb0 con 0x7f4a14106630 2024-04-21T23:20:39.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.388+0000 7f4a18d8c640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f4a14111a30 con 0x7f4a14100fe0 2024-04-21T23:20:39.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.388+0000 7f4a18d8c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4a14111d30 con 0x7f4a14100620 2024-04-21T23:20:39.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.388+0000 7f4a12d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a14106630 0x7f4a1410ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:39.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.388+0000 7f4a12d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a14106630 0x7f4a1410ea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42438/0 (socket says 172.21.15.40:42438) 2024-04-21T23:20:39.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.388+0000 7f4a12d9d640 1 -- 172.21.15.40:0/673949945 learned_addr learned my addr 172.21.15.40:0/673949945 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:20:39.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.388+0000 7f4a1259c640 1 --2- 172.21.15.40:0/673949945 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4a14100620 0x7f4a1410df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:39.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.388+0000 7f4a0bfff640 1 --2- 172.21.15.40:0/673949945 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a14100fe0 0x7f4a1410e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:39.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.388+0000 7f4a12d9d640 1 -- 172.21.15.40:0/673949945 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4a14100620 msgr2=0x7f4a1410df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:39.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.388+0000 7f4a12d9d640 1 --2- 172.21.15.40:0/673949945 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4a14100620 0x7f4a1410df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.388+0000 7f4a12d9d640 1 -- 172.21.15.40:0/673949945 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a14100fe0 msgr2=0x7f4a1410e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:39.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.388+0000 7f4a12d9d640 1 --2- 172.21.15.40:0/673949945 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a14100fe0 0x7f4a1410e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.388+0000 7f4a12d9d640 1 -- 172.21.15.40:0/673949945 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a141aa9f0 con 0x7f4a14106630 2024-04-21T23:20:39.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.388+0000 7f4a12d9d640 1 --2- 172.21.15.40:0/673949945 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a14106630 0x7f4a1410ea00 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f4a0c00b970 tx=0x7f4a0c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:39.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.388+0000 7f4a0b7fe640 1 -- 172.21.15.40:0/673949945 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a0c019070 con 0x7f4a14106630 2024-04-21T23:20:39.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.388+0000 7f4a0b7fe640 1 -- 172.21.15.40:0/673949945 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a0c00cb40 con 0x7f4a14106630 2024-04-21T23:20:39.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.388+0000 7f4a0b7fe640 1 -- 172.21.15.40:0/673949945 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a0c007650 con 0x7f4a14106630 2024-04-21T23:20:39.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.388+0000 7f4a18d8c640 1 -- 172.21.15.40:0/673949945 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a141aac20 con 0x7f4a14106630 2024-04-21T23:20:39.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.389+0000 7f4a18d8c640 1 -- 172.21.15.40:0/673949945 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a141ab170 con 0x7f4a14106630 2024-04-21T23:20:39.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.391+0000 7f4a0b7fe640 1 -- 172.21.15.40:0/673949945 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f4a0c01c070 con 0x7f4a14106630 2024-04-21T23:20:39.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.391+0000 7f4a1b97a740 1 Processor -- start 2024-04-21T23:20:39.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.391+0000 7f4a1b97a740 1 -- start start 2024-04-21T23:20:39.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.391+0000 7f4a1b97a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55984541c020 0x55984541c400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:39.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.391+0000 7f4a1b97a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55984541c940 0x559845424fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:39.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.391+0000 7f4a1b97a740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559845425520 0x559845427910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:39.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.391+0000 7f4a1b97a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x559845380530 con 0x55984541c020 2024-04-21T23:20:39.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.391+0000 7f4a1b97a740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x559845328830 con 0x559845425520 2024-04-21T23:20:39.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.391+0000 7f4a1b97a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55984537c4f0 con 0x55984541c940 2024-04-21T23:20:39.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.392+0000 7f4a0bfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55984541c020 0x55984541c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:39.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.392+0000 7f4a0bfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55984541c020 0x55984541c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42446/0 (socket says 172.21.15.40:42446) 2024-04-21T23:20:39.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.392+0000 7f4a0bfff640 1 -- 172.21.15.40:0/845425238 learned_addr learned my addr 172.21.15.40:0/845425238 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:20:39.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.392+0000 7f4a0b7fe640 1 --2- 172.21.15.40:0/673949945 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f4a000a2d80 0x7f4a000a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:39.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.392+0000 7f4a1259c640 1 --2- 172.21.15.40:0/845425238 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55984541c940 0x559845424fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:39.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.392+0000 7f4a12d9d640 1 --2- 172.21.15.40:0/673949945 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f4a000a2d80 0x7f4a000a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:39.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.392+0000 7f4a0bfff640 1 --2- 172.21.15.40:0/845425238 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559845425520 0x559845427910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:39.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.392+0000 7f4a0bfff640 1 -- 172.21.15.40:0/845425238 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55984541c940 msgr2=0x559845424fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:39.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.392+0000 7f4a0bfff640 1 --2- 172.21.15.40:0/845425238 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55984541c940 0x559845424fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.392+0000 7f4a0bfff640 1 -- 172.21.15.40:0/845425238 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559845425520 msgr2=0x559845427910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:39.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.392+0000 7f4a0bfff640 1 --2- 172.21.15.40:0/845425238 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559845425520 0x559845427910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.392+0000 7f4a0bfff640 1 -- 172.21.15.40:0/845425238 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55984534c2c0 con 0x55984541c020 2024-04-21T23:20:39.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.392+0000 7f4a0b7fe640 1 -- 172.21.15.40:0/673949945 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f4a0c0cec40 con 0x7f4a14106630 2024-04-21T23:20:39.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.392+0000 7f4a0bfff640 1 --2- 172.21.15.40:0/845425238 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55984541c020 0x55984541c400 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f49fc00cdb0 tx=0x7f49fc00c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8300bc5f3ed1711 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:39.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.392+0000 7f4a12d9d640 1 --2- 172.21.15.40:0/673949945 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f4a000a2d80 0x7f4a000a5240 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4a0c0cc1a0 tx=0x7f4a0c014d00 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:39.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.392+0000 7f4a08ff9640 1 -- 172.21.15.40:0/845425238 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49fc004220 con 0x55984541c020 2024-04-21T23:20:39.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.392+0000 7f4a08ff9640 1 -- 172.21.15.40:0/845425238 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49fc0043c0 con 0x55984541c020 2024-04-21T23:20:39.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.392+0000 7f4a08ff9640 1 -- 172.21.15.40:0/845425238 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49fc0035f0 con 0x55984541c020 2024-04-21T23:20:39.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.393+0000 7f4a1b97a740 1 -- 172.21.15.40:0/845425238 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55984541c020 msgr2=0x55984541c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:39.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.393+0000 7f4a1b97a740 1 --2- 172.21.15.40:0/845425238 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55984541c020 0x55984541c400 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f49fc00cdb0 tx=0x7f49fc00c560 comp rx=0 tx=0).stop 2024-04-21T23:20:39.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.393+0000 7f4a1b97a740 1 -- 172.21.15.40:0/845425238 shutdown_connections 2024-04-21T23:20:39.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.393+0000 7f4a1b97a740 1 --2- 172.21.15.40:0/845425238 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559845425520 0x559845427910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.393+0000 7f4a1b97a740 1 --2- 172.21.15.40:0/845425238 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55984541c940 0x559845424fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.393+0000 7f4a1b97a740 1 --2- 172.21.15.40:0/845425238 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55984541c020 0x55984541c400 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.393+0000 7f4a1b97a740 1 -- 172.21.15.40:0/845425238 >> 172.21.15.40:0/845425238 conn(0x559845416750 msgr2=0x559845416b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:39.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.393+0000 7f4a1b97a740 1 -- 172.21.15.40:0/845425238 shutdown_connections 2024-04-21T23:20:39.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.393+0000 7f4a1b97a740 1 -- 172.21.15.40:0/845425238 wait complete. 2024-04-21T23:20:39.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.394+0000 7f4a1b97a740 1 -- start start 2024-04-21T23:20:39.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.394+0000 7f4a1b97a740 1 Processor -- start 2024-04-21T23:20:39.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.394+0000 7f4a1b97a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55984541c020 0x559845424e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:39.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.394+0000 7f4a1b97a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55984541c940 0x559845422200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:39.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.394+0000 7f4a1b97a740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559845425520 0x559845422760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:39.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.394+0000 7f4a1b97a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x559845428310 con 0x55984541c940 2024-04-21T23:20:39.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.394+0000 7f4a1b97a740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55984533bf10 con 0x559845425520 2024-04-21T23:20:39.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.394+0000 7f4a1b97a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x559845428490 con 0x55984541c020 2024-04-21T23:20:39.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.394+0000 7f4a1b97a740 10 client.0 _ll_register_callbacks cb 0x559845093be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:20:39.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.394+0000 7f4a1259c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55984541c940 0x559845422200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:39.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.394+0000 7f4a1259c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55984541c940 0x559845422200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42452/0 (socket says 172.21.15.40:42452) 2024-04-21T23:20:39.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.394+0000 7f4a1259c640 1 -- 172.21.15.40:0/3329361473 learned_addr learned my addr 172.21.15.40:0/3329361473 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:20:39.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.394+0000 7f4a0bfff640 1 --2- 172.21.15.40:0/3329361473 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55984541c020 0x559845424e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:39.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.394+0000 7f4a1259c640 1 -- 172.21.15.40:0/3329361473 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55984541c020 msgr2=0x559845424e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:39.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.394+0000 7f4a1259c640 1 --2- 172.21.15.40:0/3329361473 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55984541c020 0x559845424e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.394+0000 7f4a1259c640 1 -- 172.21.15.40:0/3329361473 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559845425520 msgr2=0x559845422760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:39.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.394+0000 7f4a1259c640 1 --2- 172.21.15.40:0/3329361473 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559845425520 0x559845422760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.394+0000 7f4a1259c640 1 -- 172.21.15.40:0/3329361473 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55984530dc80 con 0x55984541c940 2024-04-21T23:20:39.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.395+0000 7f4a1259c640 1 --2- 172.21.15.40:0/3329361473 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55984541c940 0x559845422200 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f4a04005ab0 tx=0x7f4a0402c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77744e590bb7f380 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:39.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.395+0000 7f49f9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-21T23:20:39.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.395+0000 7f49f9ffb640 1 -- 172.21.15.40:0/3329361473 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a0402cc50 con 0x55984541c940 2024-04-21T23:20:39.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.395+0000 7f49f9ffb640 1 -- 172.21.15.40:0/3329361473 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f4a040046c0 con 0x55984541c940 2024-04-21T23:20:39.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.395+0000 7f4a1b97a740 1 -- 172.21.15.40:0/3329361473 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f49fc0043c0 con 0x55984541c940 2024-04-21T23:20:39.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.395+0000 7f49f9ffb640 1 -- 172.21.15.40:0/3329361473 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f4a040048c0 con 0x55984541c940 2024-04-21T23:20:39.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.395+0000 7f4a1b97a740 10 client.16269 fetch_fsmap learned FSMap version 29 2024-04-21T23:20:39.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.395+0000 7f4a1b97a740 1 -- 172.21.15.40:0/3329361473 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5598450943f0 con 0x55984541c940 2024-04-21T23:20:39.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.395+0000 7f49f9ffb640 1 -- 172.21.15.40:0/3329361473 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49fc0043c0 con 0x55984541c940 2024-04-21T23:20:39.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.395+0000 7f49f9ffb640 1 -- 172.21.15.40:0/3329361473 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f4a04003660 con 0x55984541c940 2024-04-21T23:20:39.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.395+0000 7f4a1b97a740 10 client.16269 fetch_fsmap finished waiting for FSMap version 29 2024-04-21T23:20:39.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.395+0000 7f4a1b97a740 10 client.16269 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-21T23:20:39.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.395+0000 7f4a1b97a740 20 client.16269 populate_metadata read hostname 'smithi040' 2024-04-21T23:20:39.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.395+0000 7f4a1b97a740 1 --2- 172.21.15.40:0/3329361473 >> [v2:172.21.15.111:6836/2703764182,v1:172.21.15.111:6837/2703764182] conn(0x559845435610 0x559845437a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:39.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.395+0000 7f4a1b97a740 4 client.16269 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:20:39.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.395+0000 7f4a1b97a740 1 -- 172.21.15.40:0/3329361473 --> [v2:172.21.15.111:6836/2703764182,v1:172.21.15.111:6837/2703764182] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55984538da70 con 0x559845435610 2024-04-21T23:20:39.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.396+0000 7f4a0bfff640 1 --2- 172.21.15.40:0/3329361473 >> [v2:172.21.15.111:6836/2703764182,v1:172.21.15.111:6837/2703764182] conn(0x559845435610 0x559845437a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:39.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.396+0000 7f4a0bfff640 1 --2- 172.21.15.40:0/3329361473 >> [v2:172.21.15.111:6836/2703764182,v1:172.21.15.111:6837/2703764182] conn(0x559845435610 0x559845437a00 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=195661be86fd1835 server_cookie=8e532e0a349cca6c in_seq=0 out_seq=0 2024-04-21T23:20:39.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.396+0000 7f49f9ffb640 10 client.16269 ms_handle_connect on v2:172.21.15.111:6836/2703764182 2024-04-21T23:20:39.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.397+0000 7f49f9ffb640 1 -- 172.21.15.40:0/3329361473 <== mds.2 v2:172.21.15.111:6836/2703764182 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55984538da70 con 0x559845435610 2024-04-21T23:20:39.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.397+0000 7f49f9ffb640 10 client.16269 handle_command_reply: tid=0 2024-04-21T23:20:39.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.397+0000 7f4a1b97a740 1 client.16269 shutdown 2024-04-21T23:20:39.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.397+0000 7f4a1b97a740 1 -- 172.21.15.40:0/3329361473 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55984541c940 msgr2=0x559845422200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:39.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.397+0000 7f4a1b97a740 1 --2- 172.21.15.40:0/3329361473 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55984541c940 0x559845422200 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f4a04005ab0 tx=0x7f4a0402c7e0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.397+0000 7f4a1b97a740 1 -- 172.21.15.40:0/3329361473 shutdown_connections 2024-04-21T23:20:39.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.397+0000 7f4a1b97a740 1 --2- 172.21.15.40:0/3329361473 >> [v2:172.21.15.111:6836/2703764182,v1:172.21.15.111:6837/2703764182] conn(0x559845435610 0x559845437a00 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.397+0000 7f4a1b97a740 1 --2- 172.21.15.40:0/3329361473 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559845425520 0x559845422760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.397+0000 7f4a1b97a740 1 --2- 172.21.15.40:0/3329361473 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55984541c940 0x559845422200 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.397+0000 7f49f9ffb640 0 client.16269 ms_handle_reset on v2:172.21.15.111:6836/2703764182 2024-04-21T23:20:39.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.397+0000 7f4a1b97a740 1 --2- 172.21.15.40:0/3329361473 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55984541c020 0x559845424e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.397+0000 7f4a1b97a740 1 -- 172.21.15.40:0/3329361473 >> 172.21.15.40:0/3329361473 conn(0x559845416750 msgr2=0x5598454184a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:39.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.397+0000 7f4a1b97a740 1 -- 172.21.15.40:0/3329361473 shutdown_connections 2024-04-21T23:20:39.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.397+0000 7f4a1b97a740 1 -- 172.21.15.40:0/3329361473 wait complete. 2024-04-21T23:20:39.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.397+0000 7f4a1b97a740 20 client.16269 trim_cache size 0 max 16384 2024-04-21T23:20:39.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.410+0000 7f4a1b97a740 1 Processor -- start 2024-04-21T23:20:39.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.410+0000 7f4a1b97a740 1 -- start start 2024-04-21T23:20:39.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.410+0000 7f4a1b97a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55984541c020 0x5598454311d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:39.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.410+0000 7f4a1b97a740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55984541c940 0x55984541d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:39.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.410+0000 7f4a1b97a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559845425520 0x559845424930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:39.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.410+0000 7f4a1b97a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4a141b7ba0 con 0x559845425520 2024-04-21T23:20:39.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.410+0000 7f4a1b97a740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f4a141b5410 con 0x55984541c940 2024-04-21T23:20:39.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.410+0000 7f4a1b97a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4a141b3260 con 0x55984541c020 2024-04-21T23:20:39.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.411+0000 7f4a0bfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559845425520 0x559845424930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:39.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.411+0000 7f4a0bfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559845425520 0x559845424930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42456/0 (socket says 172.21.15.40:42456) 2024-04-21T23:20:39.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.411+0000 7f4a0bfff640 1 -- 172.21.15.40:0/522009161 learned_addr learned my addr 172.21.15.40:0/522009161 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:20:39.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.411+0000 7f4a0bfff640 1 --2- 172.21.15.40:0/522009161 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55984541c020 0x5598454311d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:39.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.411+0000 7f4a1259c640 1 --2- 172.21.15.40:0/522009161 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55984541c940 0x55984541d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:39.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.411+0000 7f4a0bfff640 1 -- 172.21.15.40:0/522009161 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55984541c020 msgr2=0x5598454311d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:39.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.411+0000 7f4a0bfff640 1 --2- 172.21.15.40:0/522009161 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55984541c020 0x5598454311d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.411+0000 7f4a0bfff640 1 -- 172.21.15.40:0/522009161 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55984541c940 msgr2=0x55984541d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:39.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.411+0000 7f4a0bfff640 1 --2- 172.21.15.40:0/522009161 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55984541c940 0x55984541d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.411+0000 7f4a0bfff640 1 -- 172.21.15.40:0/522009161 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55984515e7d0 con 0x559845425520 2024-04-21T23:20:39.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.411+0000 7f4a0bfff640 1 --2- 172.21.15.40:0/522009161 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559845425520 0x559845424930 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f49fc004bb0 tx=0x7f49fc014040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ebf43f0a31012fc server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:39.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.411+0000 7f49fbfff640 1 -- 172.21.15.40:0/522009161 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49fc02a550 con 0x559845425520 2024-04-21T23:20:39.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.411+0000 7f49fbfff640 1 -- 172.21.15.40:0/522009161 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49fc02a6f0 con 0x559845425520 2024-04-21T23:20:39.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.411+0000 7f49fbfff640 1 -- 172.21.15.40:0/522009161 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49fc004eb0 con 0x559845425520 2024-04-21T23:20:39.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.412+0000 7f4a1b97a740 1 -- 172.21.15.40:0/522009161 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559845425520 msgr2=0x559845424930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:39.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.412+0000 7f4a1b97a740 1 --2- 172.21.15.40:0/522009161 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559845425520 0x559845424930 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f49fc004bb0 tx=0x7f49fc014040 comp rx=0 tx=0).stop 2024-04-21T23:20:39.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.412+0000 7f4a1b97a740 1 -- 172.21.15.40:0/522009161 shutdown_connections 2024-04-21T23:20:39.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.412+0000 7f4a1b97a740 1 --2- 172.21.15.40:0/522009161 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55984541c940 0x55984541d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.412+0000 7f4a1b97a740 1 --2- 172.21.15.40:0/522009161 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559845425520 0x559845424930 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.412+0000 7f4a1b97a740 1 --2- 172.21.15.40:0/522009161 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55984541c020 0x5598454311d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.412+0000 7f4a1b97a740 1 -- 172.21.15.40:0/522009161 >> 172.21.15.40:0/522009161 conn(0x559845416750 msgr2=0x559845415850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:39.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.412+0000 7f4a1b97a740 1 -- 172.21.15.40:0/522009161 shutdown_connections 2024-04-21T23:20:39.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.412+0000 7f4a1b97a740 1 -- 172.21.15.40:0/522009161 wait complete. 2024-04-21T23:20:39.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.412+0000 7f4a1b97a740 1 -- start start 2024-04-21T23:20:39.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.412+0000 7f4a1b97a740 1 Processor -- start 2024-04-21T23:20:39.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.413+0000 7f4a1b97a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55984541c020 0x55984542d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:39.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.413+0000 7f4a1b97a740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55984541c940 0x559845428d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:39.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.413+0000 7f4a1b97a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559845425520 0x559845429250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:39.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.413+0000 7f4a1b97a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4a141b38d0 con 0x559845425520 2024-04-21T23:20:39.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.413+0000 7f4a1b97a740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f4a141b4d40 con 0x55984541c940 2024-04-21T23:20:39.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.413+0000 7f4a1b97a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4a141b3fa0 con 0x55984541c020 2024-04-21T23:20:39.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.413+0000 7f4a1b97a740 10 client.0 _ll_register_callbacks cb 0x7f4a04009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:20:39.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.413+0000 7f4a0bfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559845425520 0x559845429250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:39.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.413+0000 7f4a0bfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559845425520 0x559845429250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42466/0 (socket says 172.21.15.40:42466) 2024-04-21T23:20:39.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.413+0000 7f4a0bfff640 1 -- 172.21.15.40:0/994917979 learned_addr learned my addr 172.21.15.40:0/994917979 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:20:39.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.413+0000 7f4a0bfff640 1 --2- 172.21.15.40:0/994917979 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55984541c020 0x55984542d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:39.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.413+0000 7f4a1259c640 1 --2- 172.21.15.40:0/994917979 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55984541c940 0x559845428d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:39.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.413+0000 7f4a0bfff640 1 -- 172.21.15.40:0/994917979 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55984541c020 msgr2=0x55984542d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:39.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.413+0000 7f4a0bfff640 1 --2- 172.21.15.40:0/994917979 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55984541c020 0x55984542d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.413+0000 7f4a0bfff640 1 -- 172.21.15.40:0/994917979 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55984541c940 msgr2=0x559845428d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:39.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.413+0000 7f4a0bfff640 1 --2- 172.21.15.40:0/994917979 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55984541c940 0x559845428d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.413+0000 7f4a0bfff640 1 -- 172.21.15.40:0/994917979 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559845429940 con 0x559845425520 2024-04-21T23:20:39.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.413+0000 7f4a1259c640 1 --2- 172.21.15.40:0/994917979 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55984541c940 0x559845428d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:20:39.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.413+0000 7f4a0bfff640 1 --2- 172.21.15.40:0/994917979 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559845425520 0x559845429250 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f49fc00cba0 tx=0x7f49fc005a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c445a6f060a45ea server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:39.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.413+0000 7f49f9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-21T23:20:39.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.413+0000 7f49f9ffb640 1 -- 172.21.15.40:0/994917979 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49fc02a550 con 0x559845425520 2024-04-21T23:20:39.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.413+0000 7f49f9ffb640 1 -- 172.21.15.40:0/994917979 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f49fc004020 con 0x559845425520 2024-04-21T23:20:39.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.413+0000 7f4a1b97a740 1 -- 172.21.15.40:0/994917979 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f49fc02a6f0 con 0x559845425520 2024-04-21T23:20:39.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.413+0000 7f49f9ffb640 1 -- 172.21.15.40:0/994917979 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f49fc02a920 con 0x559845425520 2024-04-21T23:20:39.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.413+0000 7f4a1b97a740 10 client.16275 fetch_fsmap learned FSMap version 29 2024-04-21T23:20:39.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.413+0000 7f4a1b97a740 1 -- 172.21.15.40:0/994917979 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55984549ba40 con 0x559845425520 2024-04-21T23:20:39.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.414+0000 7f49f9ffb640 1 -- 172.21.15.40:0/994917979 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49fc02a6f0 con 0x559845425520 2024-04-21T23:20:39.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.414+0000 7f49f9ffb640 1 -- 172.21.15.40:0/994917979 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f49fc01b480 con 0x559845425520 2024-04-21T23:20:39.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.414+0000 7f4a1b97a740 10 client.16275 fetch_fsmap finished waiting for FSMap version 29 2024-04-21T23:20:39.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.414+0000 7f4a1b97a740 10 client.16275 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-21T23:20:39.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.414+0000 7f4a1b97a740 20 client.16275 populate_metadata read hostname 'smithi040' 2024-04-21T23:20:39.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.414+0000 7f4a1b97a740 1 --2- 172.21.15.40:0/994917979 >> [v2:172.21.15.111:6836/2703764182,v1:172.21.15.111:6837/2703764182] conn(0x55984549bfa0 0x55984549e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:39.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.414+0000 7f4a1b97a740 4 client.16275 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T23:20:39.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.414+0000 7f4a1b97a740 1 -- 172.21.15.40:0/994917979 --> [v2:172.21.15.111:6836/2703764182,v1:172.21.15.111:6837/2703764182] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559845387dc0 con 0x55984549bfa0 2024-04-21T23:20:39.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.414+0000 7f4a1259c640 1 --2- 172.21.15.40:0/994917979 >> [v2:172.21.15.111:6836/2703764182,v1:172.21.15.111:6837/2703764182] conn(0x55984549bfa0 0x55984549e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:39.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.415+0000 7f49f9ffb640 10 client.16275 ms_handle_connect on v2:172.21.15.111:6836/2703764182 2024-04-21T23:20:39.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.415+0000 7f4a1259c640 1 --2- 172.21.15.40:0/994917979 >> [v2:172.21.15.111:6836/2703764182,v1:172.21.15.111:6837/2703764182] conn(0x55984549bfa0 0x55984549e3c0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=53e709e8fadc7503 server_cookie=ae9201b0ad148ec6 in_seq=0 out_seq=0 2024-04-21T23:20:39.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.416+0000 7f49f9ffb640 1 -- 172.21.15.40:0/994917979 <== mds.2 v2:172.21.15.111:6836/2703764182 1 ==== command_reply(tid 0: 0 ) ==== 8+0+71173 (crc 0 0 0) 0x559845387dc0 con 0x55984549bfa0 2024-04-21T23:20:39.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.416+0000 7f49f9ffb640 10 client.16275 handle_command_reply: tid=0 2024-04-21T23:20:39.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.416+0000 7f4a1b97a740 1 client.16275 shutdown 2024-04-21T23:20:39.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.416+0000 7f4a1b97a740 1 -- 172.21.15.40:0/994917979 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559845425520 msgr2=0x559845429250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:39.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.417+0000 7f4a1b97a740 1 --2- 172.21.15.40:0/994917979 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559845425520 0x559845429250 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f49fc00cba0 tx=0x7f49fc005a30 comp rx=0 tx=0).stop 2024-04-21T23:20:39.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.417+0000 7f4a1b97a740 1 -- 172.21.15.40:0/994917979 shutdown_connections 2024-04-21T23:20:39.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.417+0000 7f4a1b97a740 1 --2- 172.21.15.40:0/994917979 >> [v2:172.21.15.111:6836/2703764182,v1:172.21.15.111:6837/2703764182] conn(0x55984549bfa0 0x55984549e3c0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.417+0000 7f4a1b97a740 1 --2- 172.21.15.40:0/994917979 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55984541c940 0x559845428d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.417+0000 7f4a1b97a740 1 --2- 172.21.15.40:0/994917979 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559845425520 0x559845429250 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.417+0000 7f49f9ffb640 0 client.16275 ms_handle_reset on v2:172.21.15.111:6836/2703764182 2024-04-21T23:20:39.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.417+0000 7f4a1b97a740 1 --2- 172.21.15.40:0/994917979 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55984541c020 0x55984542d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.417+0000 7f4a1b97a740 1 -- 172.21.15.40:0/994917979 >> 172.21.15.40:0/994917979 conn(0x559845416750 msgr2=0x559845415850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:39.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.417+0000 7f4a1b97a740 1 -- 172.21.15.40:0/994917979 shutdown_connections 2024-04-21T23:20:39.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.417+0000 7f4a1b97a740 1 -- 172.21.15.40:0/994917979 wait complete. 2024-04-21T23:20:39.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.417+0000 7f4a1b97a740 20 client.16275 trim_cache size 0 max 16384 2024-04-21T23:20:39.428 INFO:teuthology.orchestra.run.smithi040.stdout:[ 2024-04-21T23:20:39.429 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.429 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.429 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 1558614, 2024-04-21T23:20:39.429 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629106, 2024-04-21T23:20:39.429 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/199.out" 2024-04-21T23:20:39.429 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.429 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.429 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.429 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 4040619, 2024-04-21T23:20:39.430 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629088, 2024-04-21T23:20:39.430 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/190.out" 2024-04-21T23:20:39.430 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.430 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.430 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.430 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 11948687, 2024-04-21T23:20:39.430 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628942, 2024-04-21T23:20:39.430 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/117.out" 2024-04-21T23:20:39.431 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.431 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.431 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.431 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 16069758, 2024-04-21T23:20:39.431 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629615, 2024-04-21T23:20:39.431 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/050.out.nojournal" 2024-04-21T23:20:39.431 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.431 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.432 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.432 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 23422806, 2024-04-21T23:20:39.432 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629607, 2024-04-21T23:20:39.432 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/047.out" 2024-04-21T23:20:39.432 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.432 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.432 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.432 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 41751727, 2024-04-21T23:20:39.433 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629513, 2024-04-21T23:20:39.433 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/001.out" 2024-04-21T23:20:39.433 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.433 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.433 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.433 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 50174551, 2024-04-21T23:20:39.434 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628968, 2024-04-21T23:20:39.434 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/130.out" 2024-04-21T23:20:39.434 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.434 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.434 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.434 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 62929613, 2024-04-21T23:20:39.434 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629497, 2024-04-21T23:20:39.434 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/305.out" 2024-04-21T23:20:39.435 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.435 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.435 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.435 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 69625376, 2024-04-21T23:20:39.435 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629098, 2024-04-21T23:20:39.435 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/195.out" 2024-04-21T23:20:39.435 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.435 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.436 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.436 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 80410373, 2024-04-21T23:20:39.436 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628940, 2024-04-21T23:20:39.436 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/116.out" 2024-04-21T23:20:39.436 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.436 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.436 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.437 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 89963991, 2024-04-21T23:20:39.437 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629561, 2024-04-21T23:20:39.437 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/025.out" 2024-04-21T23:20:39.437 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.437 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.437 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.437 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 111470969, 2024-04-21T23:20:39.437 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629369, 2024-04-21T23:20:39.437 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/001.out.nozero" 2024-04-21T23:20:39.437 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.438 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.438 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.438 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 123940486, 2024-04-21T23:20:39.438 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628852, 2024-04-21T23:20:39.438 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/072.out" 2024-04-21T23:20:39.438 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.438 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.438 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.439 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 125500820, 2024-04-21T23:20:39.439 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629547, 2024-04-21T23:20:39.439 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/018.out" 2024-04-21T23:20:39.439 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.439 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.439 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.439 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 155045648, 2024-04-21T23:20:39.439 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629094, 2024-04-21T23:20:39.439 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/193.out" 2024-04-21T23:20:39.440 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.440 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.440 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.440 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 165499561, 2024-04-21T23:20:39.440 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628896, 2024-04-21T23:20:39.440 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/094.out" 2024-04-21T23:20:39.440 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.440 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.440 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.441 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 187285845, 2024-04-21T23:20:39.441 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628754, 2024-04-21T23:20:39.441 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/023.out" 2024-04-21T23:20:39.441 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.441 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.441 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.441 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 190192161, 2024-04-21T23:20:39.441 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629611, 2024-04-21T23:20:39.442 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/049.out" 2024-04-21T23:20:39.442 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.442 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.442 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.442 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 206231709, 2024-04-21T23:20:39.442 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628828, 2024-04-21T23:20:39.442 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/060.out" 2024-04-21T23:20:39.443 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.443 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.443 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.443 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 206448034, 2024-04-21T23:20:39.443 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629407, 2024-04-21T23:20:39.443 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/020.out" 2024-04-21T23:20:39.443 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.443 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.443 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.444 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 226150877, 2024-04-21T23:20:39.444 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629395, 2024-04-21T23:20:39.444 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/014.out" 2024-04-21T23:20:39.444 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.444 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.444 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.444 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 228112919, 2024-04-21T23:20:39.445 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629010, 2024-04-21T23:20:39.445 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/151.out" 2024-04-21T23:20:39.445 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.445 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.445 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.445 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 232252921, 2024-04-21T23:20:39.445 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629593, 2024-04-21T23:20:39.445 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/040.out" 2024-04-21T23:20:39.446 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.446 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.446 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.446 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 235955808, 2024-04-21T23:20:39.446 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629024, 2024-04-21T23:20:39.446 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/158.out" 2024-04-21T23:20:39.446 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.446 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.447 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.447 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 236379226, 2024-04-21T23:20:39.447 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629545, 2024-04-21T23:20:39.447 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/017.out" 2024-04-21T23:20:39.447 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.447 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.447 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.447 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 238310727, 2024-04-21T23:20:39.447 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629445, 2024-04-21T23:20:39.448 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/039.out" 2024-04-21T23:20:39.448 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.448 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.448 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.448 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 282350104, 2024-04-21T23:20:39.448 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629652, 2024-04-21T23:20:39.448 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/068.out" 2024-04-21T23:20:39.448 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.449 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.449 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.449 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 282361384, 2024-04-21T23:20:39.449 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629523, 2024-04-21T23:20:39.449 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/006.out" 2024-04-21T23:20:39.449 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.449 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.449 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.449 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 316585539, 2024-04-21T23:20:39.450 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629521, 2024-04-21T23:20:39.450 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/005.out" 2024-04-21T23:20:39.450 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.450 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.450 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.450 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 317155038, 2024-04-21T23:20:39.450 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628938, 2024-04-21T23:20:39.450 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/115.out" 2024-04-21T23:20:39.451 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.451 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.451 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.451 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 319439684, 2024-04-21T23:20:39.451 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629102, 2024-04-21T23:20:39.451 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/197.out" 2024-04-21T23:20:39.451 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.451 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.452 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.452 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 354473611, 2024-04-21T23:20:39.452 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628834, 2024-04-21T23:20:39.452 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/063.out" 2024-04-21T23:20:39.452 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.452 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.452 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.452 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 389349991, 2024-04-21T23:20:39.452 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628752, 2024-04-21T23:20:39.453 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/022.out" 2024-04-21T23:20:39.453 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.453 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.453 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.453 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 407478123, 2024-04-21T23:20:39.453 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628788, 2024-04-21T23:20:39.453 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/040.out" 2024-04-21T23:20:39.453 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.453 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.454 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.454 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 419175760, 2024-04-21T23:20:39.454 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628918, 2024-04-21T23:20:39.454 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/105.out" 2024-04-21T23:20:39.454 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.454 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.454 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.454 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 424794259, 2024-04-21T23:20:39.455 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629595, 2024-04-21T23:20:39.455 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/041.out" 2024-04-21T23:20:39.455 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.455 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.455 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.455 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 443892279, 2024-04-21T23:20:39.455 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629582, 2024-04-21T23:20:39.455 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/035.out.default" 2024-04-21T23:20:39.456 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.456 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.456 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.456 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 460231992, 2024-04-21T23:20:39.456 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628778, 2024-04-21T23:20:39.456 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/035.out" 2024-04-21T23:20:39.456 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.456 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.457 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.457 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 482935294, 2024-04-21T23:20:39.457 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629632, 2024-04-21T23:20:39.457 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/058.out" 2024-04-21T23:20:39.457 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.457 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.457 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.457 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 491907449, 2024-04-21T23:20:39.457 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628916, 2024-04-21T23:20:39.458 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/104.out" 2024-04-21T23:20:39.458 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.458 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.458 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.458 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 492753485, 2024-04-21T23:20:39.458 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629405, 2024-04-21T23:20:39.458 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/019.out" 2024-04-21T23:20:39.458 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.459 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.459 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.459 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 531868019, 2024-04-21T23:20:39.459 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629112, 2024-04-21T23:20:39.459 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/202.out" 2024-04-21T23:20:39.459 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.459 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.459 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.459 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 547391765, 2024-04-21T23:20:39.460 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629533, 2024-04-21T23:20:39.460 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/011.out" 2024-04-21T23:20:39.460 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.460 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.460 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.460 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 548131885, 2024-04-21T23:20:39.460 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628850, 2024-04-21T23:20:39.460 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/071.out" 2024-04-21T23:20:39.461 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.461 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.461 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.461 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 595940767, 2024-04-21T23:20:39.461 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629064, 2024-04-21T23:20:39.461 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/178.out" 2024-04-21T23:20:39.461 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.461 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.461 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.462 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 612632336, 2024-04-21T23:20:39.462 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628910, 2024-04-21T23:20:39.462 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/101.out" 2024-04-21T23:20:39.462 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.462 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.462 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.462 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 617227265, 2024-04-21T23:20:39.462 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629014, 2024-04-21T23:20:39.463 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/153.out" 2024-04-21T23:20:39.463 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.463 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.463 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.463 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 681225366, 2024-04-21T23:20:39.463 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628950, 2024-04-21T23:20:39.463 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/121.out" 2024-04-21T23:20:39.463 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.463 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.464 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.464 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 696091375, 2024-04-21T23:20:39.464 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628770, 2024-04-21T23:20:39.464 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/031.out" 2024-04-21T23:20:39.464 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.464 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.464 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.464 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 696097808, 2024-04-21T23:20:39.465 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629583, 2024-04-21T23:20:39.465 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/035.out.nfs" 2024-04-21T23:20:39.465 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.465 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.465 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.465 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 698259075, 2024-04-21T23:20:39.465 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629626, 2024-04-21T23:20:39.465 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/055.out" 2024-04-21T23:20:39.466 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.466 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.466 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.466 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 703326954, 2024-04-21T23:20:39.466 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628946, 2024-04-21T23:20:39.466 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/119.out" 2024-04-21T23:20:39.466 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.466 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.466 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.467 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 705490526, 2024-04-21T23:20:39.467 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628716, 2024-04-21T23:20:39.467 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/004.out" 2024-04-21T23:20:39.467 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.467 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.467 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.467 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 724738175, 2024-04-21T23:20:39.467 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628742, 2024-04-21T23:20:39.467 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/017.out" 2024-04-21T23:20:39.468 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.468 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.468 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.468 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 725301930, 2024-04-21T23:20:39.468 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629040, 2024-04-21T23:20:39.468 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/166.out" 2024-04-21T23:20:39.468 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.468 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.468 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.469 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 743572308, 2024-04-21T23:20:39.469 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629409, 2024-04-21T23:20:39.469 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/021.out" 2024-04-21T23:20:39.469 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.469 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.469 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.469 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 760477146, 2024-04-21T23:20:39.469 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628736, 2024-04-21T23:20:39.469 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/014.out" 2024-04-21T23:20:39.470 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.470 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.470 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.470 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 762632660, 2024-04-21T23:20:39.470 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629634, 2024-04-21T23:20:39.470 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/059.out" 2024-04-21T23:20:39.470 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.470 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.471 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.471 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 773648391, 2024-04-21T23:20:39.471 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629431, 2024-04-21T23:20:39.471 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/032.out" 2024-04-21T23:20:39.471 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.471 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.471 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.471 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 789618812, 2024-04-21T23:20:39.471 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628870, 2024-04-21T23:20:39.472 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/081.out" 2024-04-21T23:20:39.472 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.472 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.472 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.472 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 814575750, 2024-04-21T23:20:39.472 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628890, 2024-04-21T23:20:39.472 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/091.out" 2024-04-21T23:20:39.472 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.472 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.472 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.473 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 868638728, 2024-04-21T23:20:39.473 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629599, 2024-04-21T23:20:39.473 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/043.out" 2024-04-21T23:20:39.473 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.473 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.473 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.473 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 921551364, 2024-04-21T23:20:39.473 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629529, 2024-04-21T23:20:39.473 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/009.out" 2024-04-21T23:20:39.474 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.474 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.474 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.474 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 935417871, 2024-04-21T23:20:39.474 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629591, 2024-04-21T23:20:39.474 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/039.out" 2024-04-21T23:20:39.474 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.474 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.475 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.475 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 942050961, 2024-04-21T23:20:39.475 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629644, 2024-04-21T23:20:39.475 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/064.out" 2024-04-21T23:20:39.475 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.475 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.475 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.475 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 948685900, 2024-04-21T23:20:39.475 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629618, 2024-04-21T23:20:39.476 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/051.out" 2024-04-21T23:20:39.476 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.476 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.476 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.476 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 958076935, 2024-04-21T23:20:39.476 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628738, 2024-04-21T23:20:39.476 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/015.out" 2024-04-21T23:20:39.476 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.477 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.477 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.477 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 967921024, 2024-04-21T23:20:39.477 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629012, 2024-04-21T23:20:39.477 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/152.out" 2024-04-21T23:20:39.477 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.477 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.477 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.477 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 1000672300, 2024-04-21T23:20:39.478 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629008, 2024-04-21T23:20:39.478 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/150.out" 2024-04-21T23:20:39.478 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.478 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.478 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.478 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 1011174001, 2024-04-21T23:20:39.478 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629383, 2024-04-21T23:20:39.478 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/008.out" 2024-04-21T23:20:39.479 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.479 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.479 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.479 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 1014444164, 2024-04-21T23:20:39.479 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628792, 2024-04-21T23:20:39.479 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/042.out" 2024-04-21T23:20:39.479 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.479 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.479 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.480 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 1031517687, 2024-04-21T23:20:39.480 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629613, 2024-04-21T23:20:39.480 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/050.cfg" 2024-04-21T23:20:39.480 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.480 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.480 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.480 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 1045952921, 2024-04-21T23:20:39.480 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629078, 2024-04-21T23:20:39.480 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/185.out" 2024-04-21T23:20:39.480 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.481 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.481 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.481 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 1048158583, 2024-04-21T23:20:39.481 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629421, 2024-04-21T23:20:39.481 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/027.out" 2024-04-21T23:20:39.481 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.481 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.481 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.482 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 1117141399, 2024-04-21T23:20:39.482 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629636, 2024-04-21T23:20:39.482 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/060.out" 2024-04-21T23:20:39.482 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.482 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.482 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.482 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 1119626783, 2024-04-21T23:20:39.482 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629006, 2024-04-21T23:20:39.482 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/149.out" 2024-04-21T23:20:39.483 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.483 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.483 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.483 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 1125954055, 2024-04-21T23:20:39.483 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628802, 2024-04-21T23:20:39.483 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/047.out" 2024-04-21T23:20:39.483 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.483 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.483 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.484 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 1133950306, 2024-04-21T23:20:39.484 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629487, 2024-04-21T23:20:39.484 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/271.out" 2024-04-21T23:20:39.484 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.484 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.484 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.484 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 1151021417, 2024-04-21T23:20:39.484 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629555, 2024-04-21T23:20:39.484 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/022.out" 2024-04-21T23:20:39.485 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.485 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.485 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.485 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 1176876284, 2024-04-21T23:20:39.485 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628902, 2024-04-21T23:20:39.485 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/097.out" 2024-04-21T23:20:39.485 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.485 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.486 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.486 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 1187700839, 2024-04-21T23:20:39.486 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629403, 2024-04-21T23:20:39.486 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/018.out" 2024-04-21T23:20:39.486 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.486 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.486 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.486 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 1196187190, 2024-04-21T23:20:39.486 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628726, 2024-04-21T23:20:39.487 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/009.out" 2024-04-21T23:20:39.487 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.487 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.487 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.487 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 1206023674, 2024-04-21T23:20:39.487 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629050, 2024-04-21T23:20:39.487 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/171.out" 2024-04-21T23:20:39.487 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.487 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.488 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.488 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 1244103982, 2024-04-21T23:20:39.488 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629628, 2024-04-21T23:20:39.488 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/056.out" 2024-04-21T23:20:39.488 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.488 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.488 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.488 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 1261685445, 2024-04-21T23:20:39.488 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628904, 2024-04-21T23:20:39.489 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/098.out" 2024-04-21T23:20:39.489 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.489 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.489 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.489 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 1279659048, 2024-04-21T23:20:39.489 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628728, 2024-04-21T23:20:39.489 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/010.out" 2024-04-21T23:20:39.489 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.490 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.490 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.490 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 1310071881, 2024-04-21T23:20:39.490 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628858, 2024-04-21T23:20:39.490 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/075.out" 2024-04-21T23:20:39.490 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.490 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.490 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.490 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 1313425542, 2024-04-21T23:20:39.491 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629505, 2024-04-21T23:20:39.491 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/f2fs" 2024-04-21T23:20:39.491 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.491 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.491 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.491 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 1317632627, 2024-04-21T23:20:39.491 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628812, 2024-04-21T23:20:39.491 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/052.out" 2024-04-21T23:20:39.492 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.492 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.492 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.492 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 1326014882, 2024-04-21T23:20:39.492 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629443, 2024-04-21T23:20:39.492 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/038.out" 2024-04-21T23:20:39.492 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.492 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.492 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.493 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 1365744435, 2024-04-21T23:20:39.493 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629535, 2024-04-21T23:20:39.493 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/012.out" 2024-04-21T23:20:39.493 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.493 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.493 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.493 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 1376291464, 2024-04-21T23:20:39.493 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628760, 2024-04-21T23:20:39.494 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/026.out" 2024-04-21T23:20:39.494 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.494 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.494 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.494 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 1386323312, 2024-04-21T23:20:39.494 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628846, 2024-04-21T23:20:39.494 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/069.out" 2024-04-21T23:20:39.494 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.495 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.495 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.495 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 1391449221, 2024-04-21T23:20:39.495 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628842, 2024-04-21T23:20:39.495 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/067.out" 2024-04-21T23:20:39.495 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.495 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.495 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.496 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 1404352791, 2024-04-21T23:20:39.496 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628864, 2024-04-21T23:20:39.496 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/078.out" 2024-04-21T23:20:39.496 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.496 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.496 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.496 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 1413686772, 2024-04-21T23:20:39.496 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629042, 2024-04-21T23:20:39.497 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/167.out" 2024-04-21T23:20:39.497 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.497 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.497 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.497 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 1413882331, 2024-04-21T23:20:39.497 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628966, 2024-04-21T23:20:39.497 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/129.out" 2024-04-21T23:20:39.497 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.497 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.498 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.498 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 1432017438, 2024-04-21T23:20:39.498 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628990, 2024-04-21T23:20:39.498 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/141.out" 2024-04-21T23:20:39.498 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.498 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.498 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.499 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 1436661491, 2024-04-21T23:20:39.499 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629038, 2024-04-21T23:20:39.499 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/165.out" 2024-04-21T23:20:39.499 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.499 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.499 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.499 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 1471084343, 2024-04-21T23:20:39.499 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629622, 2024-04-21T23:20:39.500 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/053.out" 2024-04-21T23:20:39.500 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.500 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.500 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.500 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 1481766915, 2024-04-21T23:20:39.500 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628798, 2024-04-21T23:20:39.500 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/045.out" 2024-04-21T23:20:39.500 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.500 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.501 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.501 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 1491597138, 2024-04-21T23:20:39.501 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629503, 2024-04-21T23:20:39.501 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/308.out" 2024-04-21T23:20:39.501 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.501 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.501 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.501 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 1493453197, 2024-04-21T23:20:39.501 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629491, 2024-04-21T23:20:39.502 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/302.out" 2024-04-21T23:20:39.502 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.502 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.502 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.502 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 1508193125, 2024-04-21T23:20:39.502 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629411, 2024-04-21T23:20:39.502 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/022.out" 2024-04-21T23:20:39.502 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.503 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.503 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.503 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 1550028308, 2024-04-21T23:20:39.503 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629660, 2024-04-21T23:20:39.503 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/072.out" 2024-04-21T23:20:39.503 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.503 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.503 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.504 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 1558571294, 2024-04-21T23:20:39.504 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628964, 2024-04-21T23:20:39.504 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/128.out" 2024-04-21T23:20:39.504 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.504 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.504 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.504 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 1588546831, 2024-04-21T23:20:39.504 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629090, 2024-04-21T23:20:39.504 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/191.out" 2024-04-21T23:20:39.505 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.505 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.505 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.505 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 1619408086, 2024-04-21T23:20:39.505 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628730, 2024-04-21T23:20:39.505 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/011.out" 2024-04-21T23:20:39.505 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.505 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.505 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.506 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 1634749315, 2024-04-21T23:20:39.506 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629018, 2024-04-21T23:20:39.506 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/155.out" 2024-04-21T23:20:39.506 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.506 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.506 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.506 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 1636897717, 2024-04-21T23:20:39.506 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629541, 2024-04-21T23:20:39.507 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/015.out" 2024-04-21T23:20:39.507 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.507 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.507 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.507 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 1637742148, 2024-04-21T23:20:39.507 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629393, 2024-04-21T23:20:39.507 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/013.out" 2024-04-21T23:20:39.507 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.507 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.508 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.508 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 1638512286, 2024-04-21T23:20:39.508 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629415, 2024-04-21T23:20:39.508 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/024.out" 2024-04-21T23:20:39.508 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.508 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.508 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.508 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 1645977210, 2024-04-21T23:20:39.508 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628848, 2024-04-21T23:20:39.509 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/070.out" 2024-04-21T23:20:39.509 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.509 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.509 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.509 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 1652846782, 2024-04-21T23:20:39.509 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628774, 2024-04-21T23:20:39.509 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/033.out" 2024-04-21T23:20:39.509 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.510 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.510 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.510 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 1658771688, 2024-04-21T23:20:39.510 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628862, 2024-04-21T23:20:39.510 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/077.out" 2024-04-21T23:20:39.510 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.510 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.510 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.511 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 1672429416, 2024-04-21T23:20:39.511 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628888, 2024-04-21T23:20:39.511 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/090.out" 2024-04-21T23:20:39.511 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.511 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.511 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.511 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 1678904195, 2024-04-21T23:20:39.511 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629638, 2024-04-21T23:20:39.512 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/061.out" 2024-04-21T23:20:39.512 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.512 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.512 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.512 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 1692989564, 2024-04-21T23:20:39.512 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628954, 2024-04-21T23:20:39.512 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/123.out" 2024-04-21T23:20:39.512 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.512 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.513 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.513 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 1716183323, 2024-04-21T23:20:39.513 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629620, 2024-04-21T23:20:39.513 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/052.out" 2024-04-21T23:20:39.513 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.513 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.513 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.513 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 1719651905, 2024-04-21T23:20:39.513 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629417, 2024-04-21T23:20:39.514 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/025.out" 2024-04-21T23:20:39.514 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.514 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.514 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.514 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 1748648819, 2024-04-21T23:20:39.514 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629447, 2024-04-21T23:20:39.514 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/040.out" 2024-04-21T23:20:39.514 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.515 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.515 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.515 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 1772138610, 2024-04-21T23:20:39.515 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629642, 2024-04-21T23:20:39.515 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/063.out" 2024-04-21T23:20:39.515 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.515 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.515 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.515 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 1772423620, 2024-04-21T23:20:39.516 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628804, 2024-04-21T23:20:39.516 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/048.out" 2024-04-21T23:20:39.516 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.516 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.516 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.516 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 1776993630, 2024-04-21T23:20:39.516 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628712, 2024-04-21T23:20:39.516 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/002.out" 2024-04-21T23:20:39.517 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.517 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.517 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.517 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 1783561178, 2024-04-21T23:20:39.517 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629563, 2024-04-21T23:20:39.517 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/026.out" 2024-04-21T23:20:39.517 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.517 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.517 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.518 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 1796643947, 2024-04-21T23:20:39.518 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628880, 2024-04-21T23:20:39.518 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/086.out" 2024-04-21T23:20:39.518 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.518 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.518 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.518 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 1812379488, 2024-04-21T23:20:39.518 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628908, 2024-04-21T23:20:39.519 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/100.out" 2024-04-21T23:20:39.519 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.519 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.519 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.519 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 1850619126, 2024-04-21T23:20:39.519 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628912, 2024-04-21T23:20:39.519 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/102.out" 2024-04-21T23:20:39.519 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.519 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.520 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.520 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 1857865683, 2024-04-21T23:20:39.520 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628952, 2024-04-21T23:20:39.520 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/122.out" 2024-04-21T23:20:39.520 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.520 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.520 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.520 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 1884629309, 2024-04-21T23:20:39.520 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628820, 2024-04-21T23:20:39.520 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/056.out" 2024-04-21T23:20:39.521 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.521 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.521 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.521 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 1895900760, 2024-04-21T23:20:39.521 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628874, 2024-04-21T23:20:39.521 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/083.out" 2024-04-21T23:20:39.521 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.521 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.521 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.522 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 1899980742, 2024-04-21T23:20:39.522 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629379, 2024-04-21T23:20:39.522 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/006.out" 2024-04-21T23:20:39.522 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.522 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.522 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.522 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 1921810541, 2024-04-21T23:20:39.522 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629435, 2024-04-21T23:20:39.523 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/034.out" 2024-04-21T23:20:39.523 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.523 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.523 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.523 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 1933134326, 2024-04-21T23:20:39.523 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629401, 2024-04-21T23:20:39.523 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/017.out" 2024-04-21T23:20:39.523 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.523 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.523 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.524 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 1939247438, 2024-04-21T23:20:39.524 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628826, 2024-04-21T23:20:39.524 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/059.out" 2024-04-21T23:20:39.524 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.524 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.524 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.524 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 1947445025, 2024-04-21T23:20:39.524 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629640, 2024-04-21T23:20:39.524 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/062.out" 2024-04-21T23:20:39.525 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.525 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.525 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.525 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 1978139716, 2024-04-21T23:20:39.525 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628762, 2024-04-21T23:20:39.525 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/027.out" 2024-04-21T23:20:39.525 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.525 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.526 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.526 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 1988471507, 2024-04-21T23:20:39.526 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628822, 2024-04-21T23:20:39.526 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/057.out" 2024-04-21T23:20:39.526 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.526 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.526 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.526 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 1996030224, 2024-04-21T23:20:39.527 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629630, 2024-04-21T23:20:39.527 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/057.out" 2024-04-21T23:20:39.527 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.527 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.527 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.527 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2012961059, 2024-04-21T23:20:39.527 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628740, 2024-04-21T23:20:39.527 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/016.out" 2024-04-21T23:20:39.528 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.528 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.528 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.528 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2026535283, 2024-04-21T23:20:39.528 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628884, 2024-04-21T23:20:39.528 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/088.out" 2024-04-21T23:20:39.528 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.528 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.528 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.529 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2052141854, 2024-04-21T23:20:39.529 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629557, 2024-04-21T23:20:39.529 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/023.out" 2024-04-21T23:20:39.529 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.529 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.529 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.529 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2058244142, 2024-04-21T23:20:39.529 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629423, 2024-04-21T23:20:39.530 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/028.out" 2024-04-21T23:20:39.530 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.530 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.530 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.530 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2075428835, 2024-04-21T23:20:39.530 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629585, 2024-04-21T23:20:39.530 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/036.out" 2024-04-21T23:20:39.530 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.531 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.531 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.531 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2090493886, 2024-04-21T23:20:39.531 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629076, 2024-04-21T23:20:39.531 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/184.out" 2024-04-21T23:20:39.531 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.531 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.531 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.532 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2094429648, 2024-04-21T23:20:39.532 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628924, 2024-04-21T23:20:39.532 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/108.out" 2024-04-21T23:20:39.532 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.532 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.532 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.532 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2097455575, 2024-04-21T23:20:39.532 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629662, 2024-04-21T23:20:39.532 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/073.out" 2024-04-21T23:20:39.533 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.533 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.533 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.533 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2099487225, 2024-04-21T23:20:39.533 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628866, 2024-04-21T23:20:39.533 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/079.out" 2024-04-21T23:20:39.533 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.533 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.534 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.534 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2101061275, 2024-04-21T23:20:39.534 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628986, 2024-04-21T23:20:39.534 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/139.out" 2024-04-21T23:20:39.534 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.534 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.534 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.534 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2106217442, 2024-04-21T23:20:39.535 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629387, 2024-04-21T23:20:39.535 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/010.out" 2024-04-21T23:20:39.535 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.535 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.535 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.535 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2115629005, 2024-04-21T23:20:39.535 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629609, 2024-04-21T23:20:39.535 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/048.out" 2024-04-21T23:20:39.536 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.536 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.536 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.536 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2119036718, 2024-04-21T23:20:39.536 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628962, 2024-04-21T23:20:39.536 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/127.out" 2024-04-21T23:20:39.536 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.536 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.537 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.537 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2122325282, 2024-04-21T23:20:39.537 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628750, 2024-04-21T23:20:39.537 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/021.out" 2024-04-21T23:20:39.537 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.537 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.537 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.537 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2152139856, 2024-04-21T23:20:39.537 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629074, 2024-04-21T23:20:39.538 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/183.out" 2024-04-21T23:20:39.538 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.538 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.538 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.538 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2152200962, 2024-04-21T23:20:39.538 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628844, 2024-04-21T23:20:39.538 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/068.out" 2024-04-21T23:20:39.538 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.539 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.539 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.539 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2153664556, 2024-04-21T23:20:39.539 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628886, 2024-04-21T23:20:39.539 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/089.out" 2024-04-21T23:20:39.539 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.539 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.539 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.540 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2171990935, 2024-04-21T23:20:39.540 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629092, 2024-04-21T23:20:39.540 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/192.out" 2024-04-21T23:20:39.540 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.540 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.540 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.540 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2172829491, 2024-04-21T23:20:39.540 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628708, 2024-04-21T23:20:39.540 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs" 2024-04-21T23:20:39.541 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.541 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.541 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.541 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2173959597, 2024-04-21T23:20:39.541 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629080, 2024-04-21T23:20:39.541 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/186.out" 2024-04-21T23:20:39.541 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.541 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.542 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.542 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2194637319, 2024-04-21T23:20:39.542 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629501, 2024-04-21T23:20:39.542 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/307.out" 2024-04-21T23:20:39.542 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.542 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.542 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.542 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2209431598, 2024-04-21T23:20:39.543 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629499, 2024-04-21T23:20:39.543 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/306.out" 2024-04-21T23:20:39.543 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.543 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.543 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.543 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2227960050, 2024-04-21T23:20:39.543 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628816, 2024-04-21T23:20:39.543 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/054.out" 2024-04-21T23:20:39.543 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.544 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.544 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.544 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2261653342, 2024-04-21T23:20:39.544 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629368, 2024-04-21T23:20:39.544 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/001.out.nodelalloc" 2024-04-21T23:20:39.544 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.544 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.544 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.544 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2284603045, 2024-04-21T23:20:39.545 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628900, 2024-04-21T23:20:39.545 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/096.out" 2024-04-21T23:20:39.545 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.545 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.545 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.545 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2296912231, 2024-04-21T23:20:39.545 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628722, 2024-04-21T23:20:39.545 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/007.out" 2024-04-21T23:20:39.546 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.546 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.546 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.546 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2309980982, 2024-04-21T23:20:39.546 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629108, 2024-04-21T23:20:39.546 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/200.out" 2024-04-21T23:20:39.546 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.546 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.547 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.547 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2313657654, 2024-04-21T23:20:39.547 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629413, 2024-04-21T23:20:39.547 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/023.out" 2024-04-21T23:20:39.547 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.547 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.547 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.547 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2318196145, 2024-04-21T23:20:39.547 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628982, 2024-04-21T23:20:39.548 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/137.out" 2024-04-21T23:20:39.548 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.548 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.548 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.548 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2319918976, 2024-04-21T23:20:39.548 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628878, 2024-04-21T23:20:39.548 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/085.out" 2024-04-21T23:20:39.548 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.549 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.549 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.549 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2325686070, 2024-04-21T23:20:39.549 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629004, 2024-04-21T23:20:39.549 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/148.out" 2024-04-21T23:20:39.549 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.549 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.549 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.550 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2330724208, 2024-04-21T23:20:39.550 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629567, 2024-04-21T23:20:39.550 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/028.out" 2024-04-21T23:20:39.550 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.550 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.550 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.550 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2343806663, 2024-04-21T23:20:39.550 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629419, 2024-04-21T23:20:39.551 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/026.out" 2024-04-21T23:20:39.551 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.551 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.551 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.551 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2355096927, 2024-04-21T23:20:39.551 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628898, 2024-04-21T23:20:39.551 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/095.out" 2024-04-21T23:20:39.551 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.551 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.552 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.552 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2360334191, 2024-04-21T23:20:39.552 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628922, 2024-04-21T23:20:39.552 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/107.out" 2024-04-21T23:20:39.552 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.552 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.552 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.552 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2391575786, 2024-04-21T23:20:39.552 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629048, 2024-04-21T23:20:39.553 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/170.out" 2024-04-21T23:20:39.553 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.553 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.553 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.553 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2403280645, 2024-04-21T23:20:39.553 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629658, 2024-04-21T23:20:39.553 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/071.out" 2024-04-21T23:20:39.553 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.553 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.554 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.554 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2425953607, 2024-04-21T23:20:39.554 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628790, 2024-04-21T23:20:39.554 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/041.out" 2024-04-21T23:20:39.554 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.554 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.554 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.554 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2426385437, 2024-04-21T23:20:39.555 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629614, 2024-04-21T23:20:39.555 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/050.out.default" 2024-04-21T23:20:39.555 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.555 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.555 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.555 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2438188003, 2024-04-21T23:20:39.555 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629527, 2024-04-21T23:20:39.555 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/008.out" 2024-04-21T23:20:39.555 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.556 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.556 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.556 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2461913685, 2024-04-21T23:20:39.556 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628758, 2024-04-21T23:20:39.556 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/025.out" 2024-04-21T23:20:39.556 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.556 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.556 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.556 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2469817610, 2024-04-21T23:20:39.557 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629489, 2024-04-21T23:20:39.557 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/301.out" 2024-04-21T23:20:39.557 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.557 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.557 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.557 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2483764671, 2024-04-21T23:20:39.557 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629531, 2024-04-21T23:20:39.557 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/010.out" 2024-04-21T23:20:39.558 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.558 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.558 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.558 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2521237913, 2024-04-21T23:20:39.558 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629537, 2024-04-21T23:20:39.558 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/013.out" 2024-04-21T23:20:39.558 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.558 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.559 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.559 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2546101797, 2024-04-21T23:20:39.559 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628994, 2024-04-21T23:20:39.559 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/143.out" 2024-04-21T23:20:39.559 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.559 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.559 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.559 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2567511520, 2024-04-21T23:20:39.560 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629601, 2024-04-21T23:20:39.560 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/044.out" 2024-04-21T23:20:39.560 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.560 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.560 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.560 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2603642814, 2024-04-21T23:20:39.560 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629581, 2024-04-21T23:20:39.560 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/035.cfg" 2024-04-21T23:20:39.560 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.561 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.561 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.561 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2604634485, 2024-04-21T23:20:39.561 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629559, 2024-04-21T23:20:39.561 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/024.out" 2024-04-21T23:20:39.561 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.561 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.561 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.561 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2614619407, 2024-04-21T23:20:39.561 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628780, 2024-04-21T23:20:39.562 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/036.out" 2024-04-21T23:20:39.562 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.562 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.562 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.562 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2631103196, 2024-04-21T23:20:39.562 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629046, 2024-04-21T23:20:39.562 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/169.out" 2024-04-21T23:20:39.562 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.562 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.563 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.563 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2639601367, 2024-04-21T23:20:39.563 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628784, 2024-04-21T23:20:39.563 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/038.out" 2024-04-21T23:20:39.563 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.563 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.563 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.563 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2650560097, 2024-04-21T23:20:39.563 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629451, 2024-04-21T23:20:39.564 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/042.out" 2024-04-21T23:20:39.564 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.564 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.564 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.564 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2666219652, 2024-04-21T23:20:39.564 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629511, 2024-04-21T23:20:39.564 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic" 2024-04-21T23:20:39.564 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.564 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.565 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.565 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2666399661, 2024-04-21T23:20:39.565 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629549, 2024-04-21T23:20:39.565 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/019.out" 2024-04-21T23:20:39.565 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.565 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.565 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.565 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2685322802, 2024-04-21T23:20:39.565 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629648, 2024-04-21T23:20:39.566 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/066.out" 2024-04-21T23:20:39.566 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.566 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.566 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.566 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2694509877, 2024-04-21T23:20:39.566 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629068, 2024-04-21T23:20:39.566 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/180.out" 2024-04-21T23:20:39.566 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.567 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.567 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.567 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2703900723, 2024-04-21T23:20:39.567 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628786, 2024-04-21T23:20:39.567 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/039.out" 2024-04-21T23:20:39.567 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.567 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.567 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.567 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2783856577, 2024-04-21T23:20:39.568 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628744, 2024-04-21T23:20:39.568 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/018.out" 2024-04-21T23:20:39.568 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.568 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.568 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.568 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2789497070, 2024-04-21T23:20:39.568 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629096, 2024-04-21T23:20:39.568 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/194.out" 2024-04-21T23:20:39.569 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.569 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.569 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.569 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2802126831, 2024-04-21T23:20:39.569 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629072, 2024-04-21T23:20:39.569 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/182.out" 2024-04-21T23:20:39.569 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.569 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.570 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.570 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2804384051, 2024-04-21T23:20:39.570 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629646, 2024-04-21T23:20:39.570 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/065.out" 2024-04-21T23:20:39.570 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.570 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.570 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.570 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2807731837, 2024-04-21T23:20:39.570 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629441, 2024-04-21T23:20:39.571 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/037.out" 2024-04-21T23:20:39.571 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.571 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.571 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.571 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2809648593, 2024-04-21T23:20:39.571 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628772, 2024-04-21T23:20:39.571 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/032.out" 2024-04-21T23:20:39.573 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.573 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.573 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.573 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2816739951, 2024-04-21T23:20:39.573 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629624, 2024-04-21T23:20:39.574 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/054.out" 2024-04-21T23:20:39.574 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.574 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.574 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.574 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2821857918, 2024-04-21T23:20:39.574 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629110, 2024-04-21T23:20:39.574 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/201.out" 2024-04-21T23:20:39.574 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.575 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.575 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.575 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2839341456, 2024-04-21T23:20:39.575 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628814, 2024-04-21T23:20:39.575 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/053.out" 2024-04-21T23:20:39.575 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.575 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.575 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.575 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2856732774, 2024-04-21T23:20:39.576 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629066, 2024-04-21T23:20:39.576 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/179.out" 2024-04-21T23:20:39.576 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.576 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.576 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.576 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2859013629, 2024-04-21T23:20:39.576 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629525, 2024-04-21T23:20:39.577 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/007.out" 2024-04-21T23:20:39.577 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.577 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.577 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.577 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2885577075, 2024-04-21T23:20:39.577 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628958, 2024-04-21T23:20:39.577 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/125.out" 2024-04-21T23:20:39.577 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.577 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.578 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.578 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2891092575, 2024-04-21T23:20:39.578 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628838, 2024-04-21T23:20:39.578 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/065.out" 2024-04-21T23:20:39.578 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.578 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.578 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.578 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2901881156, 2024-04-21T23:20:39.579 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628830, 2024-04-21T23:20:39.579 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/061.out" 2024-04-21T23:20:39.579 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.579 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.579 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.579 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2903983007, 2024-04-21T23:20:39.579 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629020, 2024-04-21T23:20:39.579 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/156.out" 2024-04-21T23:20:39.580 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.580 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.580 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.580 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2909604243, 2024-04-21T23:20:39.580 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629373, 2024-04-21T23:20:39.580 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/003.out" 2024-04-21T23:20:39.580 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.580 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.580 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.581 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2919803636, 2024-04-21T23:20:39.581 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629399, 2024-04-21T23:20:39.581 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/016.out" 2024-04-21T23:20:39.581 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.581 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.581 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.581 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2924120603, 2024-04-21T23:20:39.581 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628748, 2024-04-21T23:20:39.582 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/020.out" 2024-04-21T23:20:39.582 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.582 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.582 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.582 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2925289664, 2024-04-21T23:20:39.582 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628796, 2024-04-21T23:20:39.582 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/044.out" 2024-04-21T23:20:39.582 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.583 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.583 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.583 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2936396766, 2024-04-21T23:20:39.583 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628894, 2024-04-21T23:20:39.583 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/093.out" 2024-04-21T23:20:39.583 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.583 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.583 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.583 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2937772258, 2024-04-21T23:20:39.584 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629104, 2024-04-21T23:20:39.584 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/198.out" 2024-04-21T23:20:39.584 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.584 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.584 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.584 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2948127614, 2024-04-21T23:20:39.584 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628746, 2024-04-21T23:20:39.584 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/019.out" 2024-04-21T23:20:39.585 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.585 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.585 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.585 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2955609826, 2024-04-21T23:20:39.585 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629579, 2024-04-21T23:20:39.585 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/034.out" 2024-04-21T23:20:39.585 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.585 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.586 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.586 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2992898823, 2024-04-21T23:20:39.586 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628980, 2024-04-21T23:20:39.586 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/136.out" 2024-04-21T23:20:39.586 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.586 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.586 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.586 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2997894209, 2024-04-21T23:20:39.587 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629058, 2024-04-21T23:20:39.587 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/175.out" 2024-04-21T23:20:39.587 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.587 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.587 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.587 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 2998643511, 2024-04-21T23:20:39.587 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629589, 2024-04-21T23:20:39.587 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/038.out" 2024-04-21T23:20:39.588 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.588 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.588 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.588 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3038393378, 2024-04-21T23:20:39.588 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629389, 2024-04-21T23:20:39.588 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/011.out" 2024-04-21T23:20:39.588 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.588 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.588 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.589 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3041600235, 2024-04-21T23:20:39.589 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629433, 2024-04-21T23:20:39.589 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/033.out" 2024-04-21T23:20:39.589 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.589 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.589 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.589 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3042234196, 2024-04-21T23:20:39.589 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628906, 2024-04-21T23:20:39.590 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/099.out" 2024-04-21T23:20:39.590 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.590 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.590 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.590 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3046032168, 2024-04-21T23:20:39.590 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628998, 2024-04-21T23:20:39.590 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/145.out" 2024-04-21T23:20:39.590 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.591 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.591 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.591 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3067176649, 2024-04-21T23:20:39.591 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628970, 2024-04-21T23:20:39.591 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/131.out" 2024-04-21T23:20:39.591 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.591 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.591 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.591 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3068675093, 2024-04-21T23:20:39.592 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629385, 2024-04-21T23:20:39.592 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/009.out" 2024-04-21T23:20:39.592 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.592 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.592 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.592 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3077720606, 2024-04-21T23:20:39.592 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628996, 2024-04-21T23:20:39.592 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/144.out" 2024-04-21T23:20:39.592 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.592 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.593 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.593 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3103475690, 2024-04-21T23:20:39.593 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629026, 2024-04-21T23:20:39.593 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/159.out" 2024-04-21T23:20:39.593 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.593 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.593 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.593 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3112024784, 2024-04-21T23:20:39.593 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628934, 2024-04-21T23:20:39.594 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/113.out" 2024-04-21T23:20:39.594 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.594 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.594 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.594 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3155470778, 2024-04-21T23:20:39.594 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629437, 2024-04-21T23:20:39.594 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/035.out" 2024-04-21T23:20:39.594 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.594 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.595 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.595 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3157273716, 2024-04-21T23:20:39.595 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628710, 2024-04-21T23:20:39.595 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/001.out" 2024-04-21T23:20:39.595 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.595 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.595 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.595 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3172985907, 2024-04-21T23:20:39.595 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628724, 2024-04-21T23:20:39.596 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/008.out" 2024-04-21T23:20:39.596 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.596 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.596 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.596 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3193623495, 2024-04-21T23:20:39.596 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629597, 2024-04-21T23:20:39.596 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/042.out" 2024-04-21T23:20:39.596 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.596 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.597 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.597 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3239482274, 2024-04-21T23:20:39.597 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628764, 2024-04-21T23:20:39.597 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/028.out" 2024-04-21T23:20:39.597 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.597 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.597 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.597 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3265393820, 2024-04-21T23:20:39.597 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629553, 2024-04-21T23:20:39.597 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/021.out" 2024-04-21T23:20:39.598 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.598 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.598 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.598 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3267142831, 2024-04-21T23:20:39.598 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629656, 2024-04-21T23:20:39.598 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/070.out" 2024-04-21T23:20:39.598 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.598 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.599 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.599 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3267304707, 2024-04-21T23:20:39.599 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629360, 2024-04-21T23:20:39.599 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/cifs" 2024-04-21T23:20:39.599 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.599 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.599 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.599 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3277748402, 2024-04-21T23:20:39.599 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629425, 2024-04-21T23:20:39.600 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/029.out" 2024-04-21T23:20:39.600 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.600 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.600 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.600 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3278011930, 2024-04-21T23:20:39.600 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629605, 2024-04-21T23:20:39.600 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/046.out" 2024-04-21T23:20:39.600 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.600 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.601 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.601 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3280473302, 2024-04-21T23:20:39.601 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629375, 2024-04-21T23:20:39.601 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/004.out" 2024-04-21T23:20:39.601 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.601 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.601 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.601 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3304684766, 2024-04-21T23:20:39.601 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628984, 2024-04-21T23:20:39.602 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/138.out" 2024-04-21T23:20:39.602 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.602 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.602 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.602 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3327120062, 2024-04-21T23:20:39.602 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629493, 2024-04-21T23:20:39.602 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/303.out" 2024-04-21T23:20:39.603 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.603 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.603 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.603 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3344079257, 2024-04-21T23:20:39.603 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629366, 2024-04-21T23:20:39.603 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/001.cfg" 2024-04-21T23:20:39.603 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.603 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.604 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.604 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3365715262, 2024-04-21T23:20:39.604 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629429, 2024-04-21T23:20:39.604 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/031.out" 2024-04-21T23:20:39.604 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.604 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.604 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.604 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3368495633, 2024-04-21T23:20:39.604 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629650, 2024-04-21T23:20:39.605 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/067.out" 2024-04-21T23:20:39.605 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.605 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.605 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.605 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3372436831, 2024-04-21T23:20:39.605 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629022, 2024-04-21T23:20:39.605 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/157.out" 2024-04-21T23:20:39.605 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.606 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.606 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.606 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3372963410, 2024-04-21T23:20:39.606 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629439, 2024-04-21T23:20:39.606 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/036.out" 2024-04-21T23:20:39.606 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.606 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.606 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.606 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3386111793, 2024-04-21T23:20:39.607 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628782, 2024-04-21T23:20:39.607 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/037.out" 2024-04-21T23:20:39.607 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.607 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.607 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.607 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3398267245, 2024-04-21T23:20:39.607 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629032, 2024-04-21T23:20:39.607 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/162.out" 2024-04-21T23:20:39.607 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.608 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.608 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.608 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3416093719, 2024-04-21T23:20:39.608 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629381, 2024-04-21T23:20:39.608 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/007.out" 2024-04-21T23:20:39.608 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.608 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.608 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.608 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3440574853, 2024-04-21T23:20:39.609 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629082, 2024-04-21T23:20:39.609 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/187.out" 2024-04-21T23:20:39.609 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.609 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.609 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.609 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3465337178, 2024-04-21T23:20:39.609 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628976, 2024-04-21T23:20:39.609 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/134.out" 2024-04-21T23:20:39.609 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.610 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.610 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.610 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3466785126, 2024-04-21T23:20:39.610 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629362, 2024-04-21T23:20:39.610 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/cifs/001.out" 2024-04-21T23:20:39.610 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.610 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.610 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.610 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3469853750, 2024-04-21T23:20:39.611 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628818, 2024-04-21T23:20:39.611 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/055.out" 2024-04-21T23:20:39.611 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.611 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.611 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.611 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3482546632, 2024-04-21T23:20:39.611 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628928, 2024-04-21T23:20:39.611 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/110.out" 2024-04-21T23:20:39.611 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.612 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.612 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.612 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3525208496, 2024-04-21T23:20:39.612 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628714, 2024-04-21T23:20:39.612 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/003.out" 2024-04-21T23:20:39.612 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.612 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.612 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.613 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3528560221, 2024-04-21T23:20:39.613 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629030, 2024-04-21T23:20:39.613 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/161.out" 2024-04-21T23:20:39.613 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.613 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.613 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.613 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3534446033, 2024-04-21T23:20:39.613 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628936, 2024-04-21T23:20:39.614 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/114.out" 2024-04-21T23:20:39.614 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.614 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.614 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.614 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3534719075, 2024-04-21T23:20:39.614 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628978, 2024-04-21T23:20:39.614 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/135.out" 2024-04-21T23:20:39.614 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.614 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.615 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.615 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3539642818, 2024-04-21T23:20:39.615 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629391, 2024-04-21T23:20:39.615 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/012.out" 2024-04-21T23:20:39.615 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.615 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.615 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.615 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3551727301, 2024-04-21T23:20:39.616 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628824, 2024-04-21T23:20:39.616 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/058.out" 2024-04-21T23:20:39.616 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.616 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.616 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.616 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3560385251, 2024-04-21T23:20:39.616 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628768, 2024-04-21T23:20:39.616 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/030.out" 2024-04-21T23:20:39.617 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.617 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.617 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.617 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3560858111, 2024-04-21T23:20:39.617 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629364, 2024-04-21T23:20:39.617 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4" 2024-04-21T23:20:39.617 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.617 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.618 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.618 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3562334303, 2024-04-21T23:20:39.618 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628914, 2024-04-21T23:20:39.618 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/103.out" 2024-04-21T23:20:39.618 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.618 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.618 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.618 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3567007406, 2024-04-21T23:20:39.619 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629100, 2024-04-21T23:20:39.619 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/196.out" 2024-04-21T23:20:39.619 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.619 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.619 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.619 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3578937546, 2024-04-21T23:20:39.619 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629515, 2024-04-21T23:20:39.619 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/002.out" 2024-04-21T23:20:39.620 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.620 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.620 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.620 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3605575045, 2024-04-21T23:20:39.620 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629616, 2024-04-21T23:20:39.620 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/050.out.xfsquota" 2024-04-21T23:20:39.620 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.620 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.621 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.621 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3611961388, 2024-04-21T23:20:39.621 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628926, 2024-04-21T23:20:39.621 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/109.out" 2024-04-21T23:20:39.621 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.621 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.621 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.621 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3624524824, 2024-04-21T23:20:39.622 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628832, 2024-04-21T23:20:39.622 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/062.out" 2024-04-21T23:20:39.622 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.622 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.622 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.622 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3638881860, 2024-04-21T23:20:39.622 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629654, 2024-04-21T23:20:39.622 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/069.out" 2024-04-21T23:20:39.623 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.623 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.623 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.623 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3642278035, 2024-04-21T23:20:39.623 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628956, 2024-04-21T23:20:39.623 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/124.out" 2024-04-21T23:20:39.623 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.623 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.624 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.624 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3660908708, 2024-04-21T23:20:39.624 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628892, 2024-04-21T23:20:39.624 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/092.out" 2024-04-21T23:20:39.624 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.624 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.624 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.624 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3670984507, 2024-04-21T23:20:39.625 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629084, 2024-04-21T23:20:39.625 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/188.out" 2024-04-21T23:20:39.625 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.625 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.625 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.625 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3674730059, 2024-04-21T23:20:39.625 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628868, 2024-04-21T23:20:39.625 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/080.out" 2024-04-21T23:20:39.626 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.626 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.626 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.626 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3679083051, 2024-04-21T23:20:39.626 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629028, 2024-04-21T23:20:39.626 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/160.out" 2024-04-21T23:20:39.626 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.626 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.627 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.627 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3680870297, 2024-04-21T23:20:39.627 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629377, 2024-04-21T23:20:39.627 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/005.out" 2024-04-21T23:20:39.627 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.627 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.627 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.628 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3690121004, 2024-04-21T23:20:39.628 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628944, 2024-04-21T23:20:39.628 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/118.out" 2024-04-21T23:20:39.628 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.628 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.628 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.628 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3708539435, 2024-04-21T23:20:39.628 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629036, 2024-04-21T23:20:39.629 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/164.out" 2024-04-21T23:20:39.629 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.629 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.629 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.629 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3716610679, 2024-04-21T23:20:39.629 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629002, 2024-04-21T23:20:39.629 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/147.out" 2024-04-21T23:20:39.630 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.630 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.630 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.630 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3735993147, 2024-04-21T23:20:39.630 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628930, 2024-04-21T23:20:39.630 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/111.out" 2024-04-21T23:20:39.630 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.630 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.631 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.631 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3737589287, 2024-04-21T23:20:39.631 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628718, 2024-04-21T23:20:39.631 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/005.out" 2024-04-21T23:20:39.631 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.631 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.631 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.631 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3747609878, 2024-04-21T23:20:39.632 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629517, 2024-04-21T23:20:39.632 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/003.out" 2024-04-21T23:20:39.632 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.632 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.632 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.632 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3761720115, 2024-04-21T23:20:39.632 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628856, 2024-04-21T23:20:39.633 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/074.out" 2024-04-21T23:20:39.633 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.633 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.633 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.633 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3777513020, 2024-04-21T23:20:39.633 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628992, 2024-04-21T23:20:39.633 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/142.out" 2024-04-21T23:20:39.633 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.633 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.634 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.634 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3777997468, 2024-04-21T23:20:39.634 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629495, 2024-04-21T23:20:39.634 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/304.out" 2024-04-21T23:20:39.634 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.634 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.634 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.634 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3792958047, 2024-04-21T23:20:39.635 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628882, 2024-04-21T23:20:39.635 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/087.out" 2024-04-21T23:20:39.635 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.635 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.635 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.635 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3832577580, 2024-04-21T23:20:39.635 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628948, 2024-04-21T23:20:39.635 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/120.out" 2024-04-21T23:20:39.636 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.636 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.636 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.636 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3845789394, 2024-04-21T23:20:39.636 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628960, 2024-04-21T23:20:39.636 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/126.out" 2024-04-21T23:20:39.636 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.636 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.637 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.637 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3856070076, 2024-04-21T23:20:39.637 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629603, 2024-04-21T23:20:39.637 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/045.out" 2024-04-21T23:20:39.637 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.637 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.637 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.637 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3868057311, 2024-04-21T23:20:39.638 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629449, 2024-04-21T23:20:39.638 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/041.out" 2024-04-21T23:20:39.638 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.638 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.638 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.638 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3873700857, 2024-04-21T23:20:39.638 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628810, 2024-04-21T23:20:39.638 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/051.out" 2024-04-21T23:20:39.639 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.639 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.639 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.639 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3882688893, 2024-04-21T23:20:39.639 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628800, 2024-04-21T23:20:39.639 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/046.out" 2024-04-21T23:20:39.639 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.639 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.640 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.640 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3885219739, 2024-04-21T23:20:39.640 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629060, 2024-04-21T23:20:39.640 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/176.out" 2024-04-21T23:20:39.640 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.640 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.640 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.641 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3896143639, 2024-04-21T23:20:39.641 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629070, 2024-04-21T23:20:39.641 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/181.out" 2024-04-21T23:20:39.641 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.641 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.641 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.641 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3901885622, 2024-04-21T23:20:39.641 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629587, 2024-04-21T23:20:39.641 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/037.out" 2024-04-21T23:20:39.642 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.642 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.642 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.642 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3915810659, 2024-04-21T23:20:39.642 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629519, 2024-04-21T23:20:39.642 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/004.out" 2024-04-21T23:20:39.642 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.642 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.643 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.643 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3919592202, 2024-04-21T23:20:39.643 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628836, 2024-04-21T23:20:39.643 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/064.out" 2024-04-21T23:20:39.643 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.643 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.643 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.643 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3926235333, 2024-04-21T23:20:39.643 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629367, 2024-04-21T23:20:39.644 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/001.out.default" 2024-04-21T23:20:39.644 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.644 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.644 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.644 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3940932027, 2024-04-21T23:20:39.644 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628872, 2024-04-21T23:20:39.644 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/082.out" 2024-04-21T23:20:39.644 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.645 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.645 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.645 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3942722507, 2024-04-21T23:20:39.645 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629427, 2024-04-21T23:20:39.645 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/030.out" 2024-04-21T23:20:39.645 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.645 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.646 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.646 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3973632754, 2024-04-21T23:20:39.646 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629397, 2024-04-21T23:20:39.646 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/015.out" 2024-04-21T23:20:39.646 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.646 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.646 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.646 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 3975944065, 2024-04-21T23:20:39.647 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628972, 2024-04-21T23:20:39.647 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/132.out" 2024-04-21T23:20:39.647 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.647 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.647 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.647 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 4006756849, 2024-04-21T23:20:39.647 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628776, 2024-04-21T23:20:39.647 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/034.out" 2024-04-21T23:20:39.647 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.648 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.648 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.648 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 4009157213, 2024-04-21T23:20:39.648 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629086, 2024-04-21T23:20:39.648 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/189.out" 2024-04-21T23:20:39.648 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.648 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.648 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.649 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 4020334252, 2024-04-21T23:20:39.649 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628734, 2024-04-21T23:20:39.649 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/013.out" 2024-04-21T23:20:39.649 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.649 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.649 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.649 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 4043107295, 2024-04-21T23:20:39.650 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629044, 2024-04-21T23:20:39.650 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/168.out" 2024-04-21T23:20:39.650 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.650 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.650 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.650 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 4067333953, 2024-04-21T23:20:39.650 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628840, 2024-04-21T23:20:39.650 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/066.out" 2024-04-21T23:20:39.651 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.651 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.651 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.651 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 4069277858, 2024-04-21T23:20:39.651 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628974, 2024-04-21T23:20:39.651 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/133.out" 2024-04-21T23:20:39.651 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.651 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.652 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.652 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 4078638835, 2024-04-21T23:20:39.652 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629000, 2024-04-21T23:20:39.652 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/146.out" 2024-04-21T23:20:39.652 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.652 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.652 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.652 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 4100537424, 2024-04-21T23:20:39.652 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628794, 2024-04-21T23:20:39.653 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/043.out" 2024-04-21T23:20:39.653 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.653 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.653 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.653 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 4155134551, 2024-04-21T23:20:39.653 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628988, 2024-04-21T23:20:39.653 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/140.out" 2024-04-21T23:20:39.653 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.653 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.653 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.654 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 4172285384, 2024-04-21T23:20:39.654 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628876, 2024-04-21T23:20:39.654 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/084.out" 2024-04-21T23:20:39.654 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.654 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.654 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.654 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 4183354616, 2024-04-21T23:20:39.655 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628808, 2024-04-21T23:20:39.655 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/050.out" 2024-04-21T23:20:39.655 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.655 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.655 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.655 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 4203535408, 2024-04-21T23:20:39.655 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629539, 2024-04-21T23:20:39.655 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/014.out" 2024-04-21T23:20:39.655 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.656 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.656 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.656 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 4217424393, 2024-04-21T23:20:39.656 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629565, 2024-04-21T23:20:39.656 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/027.out" 2024-04-21T23:20:39.656 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.656 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.656 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.656 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 4275066116, 2024-04-21T23:20:39.657 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511628854, 2024-04-21T23:20:39.657 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/073.out" 2024-04-21T23:20:39.657 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:20:39.657 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:20:39.657 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-21T23:20:39.657 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 4294778982, 2024-04-21T23:20:39.657 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511629543, 2024-04-21T23:20:39.657 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/016.out" 2024-04-21T23:20:39.657 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:20:39.658 INFO:teuthology.orchestra.run.smithi040.stdout:] 2024-04-21T23:20:39.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.417+0000 7f4a18d8c640 1 -- 172.21.15.40:0/673949945 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f4a000a2d80 msgr2=0x7f4a000a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:39.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.417+0000 7f4a18d8c640 1 --2- 172.21.15.40:0/673949945 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f4a000a2d80 0x7f4a000a5240 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4a0c0cc1a0 tx=0x7f4a0c014d00 comp rx=0 tx=0).stop 2024-04-21T23:20:39.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.417+0000 7f4a18d8c640 1 -- 172.21.15.40:0/673949945 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a14106630 msgr2=0x7f4a1410ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:39.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.417+0000 7f4a18d8c640 1 --2- 172.21.15.40:0/673949945 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a14106630 0x7f4a1410ea00 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f4a0c00b970 tx=0x7f4a0c00be30 comp rx=0 tx=0).stop 2024-04-21T23:20:39.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.417+0000 7f4a18d8c640 1 -- 172.21.15.40:0/673949945 shutdown_connections 2024-04-21T23:20:39.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.417+0000 7f4a18d8c640 1 --2- 172.21.15.40:0/673949945 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f4a000a2d80 0x7f4a000a5240 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.417+0000 7f4a18d8c640 1 --2- 172.21.15.40:0/673949945 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a14100fe0 0x7f4a1410e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.417+0000 7f4a18d8c640 1 --2- 172.21.15.40:0/673949945 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a14106630 0x7f4a1410ea00 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.417+0000 7f4a18d8c640 1 --2- 172.21.15.40:0/673949945 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4a14100620 0x7f4a1410df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.417+0000 7f4a18d8c640 1 -- 172.21.15.40:0/673949945 >> 172.21.15.40:0/673949945 conn(0x7f4a140776a0 msgr2=0x7f4a141030f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:39.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.417+0000 7f4a18d8c640 1 -- 172.21.15.40:0/673949945 shutdown_connections 2024-04-21T23:20:39.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.417+0000 7f4a18d8c640 1 -- 172.21.15.40:0/673949945 wait complete. 2024-04-21T23:20:39.660 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:39 smithi040 ceph-mon[27984]: pgmap v70: 129 pgs: 129 active+clean; 95 MiB data, 781 MiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 393 KiB/s wr, 36 op/s 2024-04-21T23:20:39.661 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:39 smithi111 ceph-mon[31085]: pgmap v70: 129 pgs: 129 active+clean; 95 MiB data, 781 MiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 393 KiB/s wr, 36 op/s 2024-04-21T23:20:39.661 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:39 smithi174 ceph-mon[31409]: pgmap v70: 129 pgs: 129 active+clean; 95 MiB data, 781 MiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 393 KiB/s wr, 36 op/s 2024-04-21T23:20:39.662 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-21T23:20:39.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.770+0000 7f5f8e20e640 1 -- 172.21.15.40:0/608493619 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f88101710 msgr2=0x7f5f88101b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:39.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.770+0000 7f5f8e20e640 1 --2- 172.21.15.40:0/608493619 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f88101710 0x7f5f88101b90 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f5f7c009930 tx=0x7f5f7c02bc30 comp rx=0 tx=0).stop 2024-04-21T23:20:39.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.770+0000 7f5f8e20e640 1 -- 172.21.15.40:0/608493619 shutdown_connections 2024-04-21T23:20:39.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.770+0000 7f5f8e20e640 1 --2- 172.21.15.40:0/608493619 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f881020d0 0x7f5f8810e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.770+0000 7f5f8e20e640 1 --2- 172.21.15.40:0/608493619 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f88101710 0x7f5f88101b90 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.770+0000 7f5f8e20e640 1 --2- 172.21.15.40:0/608493619 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5f88107720 0x7f5f88107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.770+0000 7f5f8e20e640 1 -- 172.21.15.40:0/608493619 >> 172.21.15.40:0/608493619 conn(0x7f5f88078b90 msgr2=0x7f5f88078fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:39.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.770+0000 7f5f8e20e640 1 -- 172.21.15.40:0/608493619 shutdown_connections 2024-04-21T23:20:39.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.770+0000 7f5f8e20e640 1 -- 172.21.15.40:0/608493619 wait complete. 2024-04-21T23:20:39.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.770+0000 7f5f8e20e640 1 Processor -- start 2024-04-21T23:20:39.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.771+0000 7f5f8e20e640 1 -- start start 2024-04-21T23:20:39.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.771+0000 7f5f8e20e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5f88101710 0x7f5f88198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:39.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.771+0000 7f5f8e20e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f881020d0 0x7f5f88198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:39.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.771+0000 7f5f8e20e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f88107720 0x7f5f8819c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:39.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.771+0000 7f5f8e20e640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5f88111b30 con 0x7f5f881020d0 2024-04-21T23:20:39.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.771+0000 7f5f8e20e640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f5f881119b0 con 0x7f5f88107720 2024-04-21T23:20:39.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.771+0000 7f5f8e20e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5f88111cb0 con 0x7f5f88101710 2024-04-21T23:20:39.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.771+0000 7f5f86ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f881020d0 0x7f5f88198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:39.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.771+0000 7f5f86ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f881020d0 0x7f5f88198590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42478/0 (socket says 172.21.15.40:42478) 2024-04-21T23:20:39.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.771+0000 7f5f86ffd640 1 -- 172.21.15.40:0/1083372996 learned_addr learned my addr 172.21.15.40:0/1083372996 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:20:39.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.771+0000 7f5f87fff640 1 --2- 172.21.15.40:0/1083372996 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f88107720 0x7f5f8819c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:39.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.771+0000 7f5f877fe640 1 --2- 172.21.15.40:0/1083372996 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5f88101710 0x7f5f88198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:39.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.771+0000 7f5f86ffd640 1 -- 172.21.15.40:0/1083372996 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5f88101710 msgr2=0x7f5f88198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:39.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.771+0000 7f5f86ffd640 1 --2- 172.21.15.40:0/1083372996 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5f88101710 0x7f5f88198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.771+0000 7f5f86ffd640 1 -- 172.21.15.40:0/1083372996 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f88107720 msgr2=0x7f5f8819c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:39.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.771+0000 7f5f86ffd640 1 --2- 172.21.15.40:0/1083372996 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f88107720 0x7f5f8819c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.771+0000 7f5f86ffd640 1 -- 172.21.15.40:0/1083372996 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f8819d040 con 0x7f5f881020d0 2024-04-21T23:20:39.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.771+0000 7f5f87fff640 1 --2- 172.21.15.40:0/1083372996 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f88107720 0x7f5f8819c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:20:39.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.771+0000 7f5f86ffd640 1 --2- 172.21.15.40:0/1083372996 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f881020d0 0x7f5f88198590 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f5f7c004f50 tx=0x7f5f7c02ce20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:39.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.772+0000 7f5f84ff9640 1 -- 172.21.15.40:0/1083372996 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f7c041070 con 0x7f5f881020d0 2024-04-21T23:20:39.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.772+0000 7f5f84ff9640 1 -- 172.21.15.40:0/1083372996 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f7c0044d0 con 0x7f5f881020d0 2024-04-21T23:20:39.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.772+0000 7f5f84ff9640 1 -- 172.21.15.40:0/1083372996 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f7c034440 con 0x7f5f881020d0 2024-04-21T23:20:39.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.772+0000 7f5f8e20e640 1 -- 172.21.15.40:0/1083372996 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f8819d2d0 con 0x7f5f881020d0 2024-04-21T23:20:39.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.772+0000 7f5f8e20e640 1 -- 172.21.15.40:0/1083372996 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f881a4bc0 con 0x7f5f881020d0 2024-04-21T23:20:39.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.774+0000 7f5f84ff9640 1 -- 172.21.15.40:0/1083372996 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f5f7c044070 con 0x7f5f881020d0 2024-04-21T23:20:39.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.774+0000 7f5f90df6740 1 Processor -- start 2024-04-21T23:20:39.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.774+0000 7f5f90df6740 1 -- start start 2024-04-21T23:20:39.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.774+0000 7f5f90df6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a888e32020 0x55a888e32400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:39.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.774+0000 7f5f90df6740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a888e32940 0x55a888e3afe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:39.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.774+0000 7f5f90df6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a888e3b520 0x55a888e3d910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:39.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.774+0000 7f5f90df6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55a888d96530 con 0x55a888e3b520 2024-04-21T23:20:39.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.774+0000 7f5f90df6740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55a888d3e830 con 0x55a888e32940 2024-04-21T23:20:39.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.774+0000 7f5f90df6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55a888d924f0 con 0x55a888e32020 2024-04-21T23:20:39.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.775+0000 7f5f84ff9640 1 --2- 172.21.15.40:0/1083372996 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f5f700a2d00 0x7f5f700a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:39.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.775+0000 7f5f86ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a888e3b520 0x55a888e3d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:39.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.775+0000 7f5f86ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a888e3b520 0x55a888e3d910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42480/0 (socket says 172.21.15.40:42480) 2024-04-21T23:20:39.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.775+0000 7f5f86ffd640 1 -- 172.21.15.40:0/3066593979 learned_addr learned my addr 172.21.15.40:0/3066593979 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:20:39.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.775+0000 7f5f877fe640 1 --2- 172.21.15.40:0/3066593979 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a888e32940 0x55a888e3afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:39.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.775+0000 7f5f87fff640 1 --2- 172.21.15.40:0/1083372996 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f5f700a2d00 0x7f5f700a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:39.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.775+0000 7f5f87fff640 1 --2- 172.21.15.40:0/3066593979 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a888e32020 0x55a888e32400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:39.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.775+0000 7f5f84ff9640 1 -- 172.21.15.40:0/1083372996 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f5f7c0f1350 con 0x7f5f881020d0 2024-04-21T23:20:39.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.775+0000 7f5f86ffd640 1 -- 172.21.15.40:0/3066593979 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a888e32020 msgr2=0x55a888e32400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:39.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.775+0000 7f5f86ffd640 1 --2- 172.21.15.40:0/3066593979 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a888e32020 0x55a888e32400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.775+0000 7f5f86ffd640 1 -- 172.21.15.40:0/3066593979 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a888e32940 msgr2=0x55a888e3afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:39.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.775+0000 7f5f86ffd640 1 --2- 172.21.15.40:0/3066593979 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a888e32940 0x55a888e3afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.775+0000 7f5f86ffd640 1 -- 172.21.15.40:0/3066593979 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a888d622c0 con 0x55a888e3b520 2024-04-21T23:20:39.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.775+0000 7f5f877fe640 1 --2- 172.21.15.40:0/3066593979 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a888e32940 0x55a888e3afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:20:39.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.775+0000 7f5f86ffd640 1 --2- 172.21.15.40:0/3066593979 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a888e3b520 0x55a888e3d910 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f5f7c009900 tx=0x7f5f7c003ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb881886f9e78d49 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:39.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.775+0000 7f5f87fff640 1 --2- 172.21.15.40:0/1083372996 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f5f700a2d00 0x7f5f700a51c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5f80007fd0 tx=0x7f5f80007d60 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:39.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.775+0000 7f5f6dffb640 1 -- 172.21.15.40:0/3066593979 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f7c04b070 con 0x55a888e3b520 2024-04-21T23:20:39.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.775+0000 7f5f6dffb640 1 -- 172.21.15.40:0/3066593979 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f7c0bd5d0 con 0x55a888e3b520 2024-04-21T23:20:39.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.775+0000 7f5f6dffb640 1 -- 172.21.15.40:0/3066593979 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f7c003710 con 0x55a888e3b520 2024-04-21T23:20:39.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.776+0000 7f5f90df6740 1 -- 172.21.15.40:0/3066593979 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a888e3b520 msgr2=0x55a888e3d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:39.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.776+0000 7f5f90df6740 1 --2- 172.21.15.40:0/3066593979 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a888e3b520 0x55a888e3d910 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f5f7c009900 tx=0x7f5f7c003ab0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.776+0000 7f5f90df6740 1 -- 172.21.15.40:0/3066593979 shutdown_connections 2024-04-21T23:20:39.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.776+0000 7f5f90df6740 1 --2- 172.21.15.40:0/3066593979 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a888e32940 0x55a888e3afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.776+0000 7f5f90df6740 1 --2- 172.21.15.40:0/3066593979 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a888e3b520 0x55a888e3d910 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.776+0000 7f5f90df6740 1 --2- 172.21.15.40:0/3066593979 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a888e32020 0x55a888e32400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.776+0000 7f5f90df6740 1 -- 172.21.15.40:0/3066593979 >> 172.21.15.40:0/3066593979 conn(0x55a888e2c750 msgr2=0x55a888e2cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:39.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.776+0000 7f5f90df6740 1 -- 172.21.15.40:0/3066593979 shutdown_connections 2024-04-21T23:20:39.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.776+0000 7f5f90df6740 1 -- 172.21.15.40:0/3066593979 wait complete. 2024-04-21T23:20:39.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.777+0000 7f5f90df6740 1 -- start start 2024-04-21T23:20:39.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.777+0000 7f5f90df6740 1 Processor -- start 2024-04-21T23:20:39.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.777+0000 7f5f90df6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a888e32020 0x55a888e3af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:39.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.777+0000 7f5f90df6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a888e32940 0x55a888e382c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:39.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.777+0000 7f5f90df6740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a888e3b520 0x55a888e388d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:39.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.777+0000 7f5f90df6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55a888e3e0d0 con 0x55a888e32020 2024-04-21T23:20:39.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.777+0000 7f5f90df6740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55a888d51f10 con 0x55a888e3b520 2024-04-21T23:20:39.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.777+0000 7f5f90df6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55a888e3e250 con 0x55a888e32940 2024-04-21T23:20:39.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.777+0000 7f5f90df6740 10 client.0 _ll_register_callbacks cb 0x55a888aa9be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:20:39.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.777+0000 7f5f877fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a888e32020 0x55a888e3af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:39.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.777+0000 7f5f877fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a888e32020 0x55a888e3af30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42490/0 (socket says 172.21.15.40:42490) 2024-04-21T23:20:39.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.777+0000 7f5f877fe640 1 -- 172.21.15.40:0/2846602807 learned_addr learned my addr 172.21.15.40:0/2846602807 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:20:39.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.777+0000 7f5f86ffd640 1 --2- 172.21.15.40:0/2846602807 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a888e32940 0x55a888e382c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:39.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.777+0000 7f5f87fff640 1 --2- 172.21.15.40:0/2846602807 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a888e3b520 0x55a888e388d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:39.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.777+0000 7f5f877fe640 1 -- 172.21.15.40:0/2846602807 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a888e32940 msgr2=0x55a888e382c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:39.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.777+0000 7f5f877fe640 1 --2- 172.21.15.40:0/2846602807 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a888e32940 0x55a888e382c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.777+0000 7f5f877fe640 1 -- 172.21.15.40:0/2846602807 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a888e3b520 msgr2=0x55a888e388d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:39.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.777+0000 7f5f877fe640 1 --2- 172.21.15.40:0/2846602807 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a888e3b520 0x55a888e388d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.777+0000 7f5f877fe640 1 -- 172.21.15.40:0/2846602807 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a888d23c80 con 0x55a888e32020 2024-04-21T23:20:39.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.777+0000 7f5f87fff640 1 --2- 172.21.15.40:0/2846602807 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a888e3b520 0x55a888e388d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:20:39.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.778+0000 7f5f877fe640 1 --2- 172.21.15.40:0/2846602807 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a888e32020 0x55a888e3af30 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f5f7800d470 tx=0x7f5f7800d930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a6ed3a9ba5a3e71 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:39.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.778+0000 7f5f66ffd640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-21T23:20:39.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.778+0000 7f5f66ffd640 1 -- 172.21.15.40:0/2846602807 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f7801c060 con 0x55a888e32020 2024-04-21T23:20:39.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.778+0000 7f5f66ffd640 1 -- 172.21.15.40:0/2846602807 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f5f78015750 con 0x55a888e32020 2024-04-21T23:20:39.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.778+0000 7f5f90df6740 1 -- 172.21.15.40:0/2846602807 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5f7c0bd5d0 con 0x55a888e32020 2024-04-21T23:20:39.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.778+0000 7f5f66ffd640 1 -- 172.21.15.40:0/2846602807 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5f78023050 con 0x55a888e32020 2024-04-21T23:20:39.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.778+0000 7f5f90df6740 10 client.16287 fetch_fsmap learned FSMap version 29 2024-04-21T23:20:39.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.778+0000 7f5f90df6740 1 -- 172.21.15.40:0/2846602807 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a888aaa3f0 con 0x55a888e32020 2024-04-21T23:20:39.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.778+0000 7f5f66ffd640 1 -- 172.21.15.40:0/2846602807 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f7c0bd5d0 con 0x55a888e32020 2024-04-21T23:20:39.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.778+0000 7f5f66ffd640 1 -- 172.21.15.40:0/2846602807 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f5f78016bd0 con 0x55a888e32020 2024-04-21T23:20:39.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.778+0000 7f5f90df6740 10 client.16287 fetch_fsmap finished waiting for FSMap version 29 2024-04-21T23:20:39.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.778+0000 7f5f90df6740 10 client.16287 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T23:20:39.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.778+0000 7f5f90df6740 20 client.16287 populate_metadata read hostname 'smithi040' 2024-04-21T23:20:39.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.778+0000 7f5f90df6740 1 --2- 172.21.15.40:0/2846602807 >> [v2:172.21.15.174:6838/2467509274,v1:172.21.15.174:6839/2467509274] conn(0x55a888e4b3c0 0x55a888e4d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:39.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.778+0000 7f5f90df6740 4 client.16287 mds_command: new command op to 24502 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:20:39.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.778+0000 7f5f90df6740 1 -- 172.21.15.40:0/2846602807 --> [v2:172.21.15.174:6838/2467509274,v1:172.21.15.174:6839/2467509274] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a888da3a70 con 0x55a888e4b3c0 2024-04-21T23:20:39.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.779+0000 7f5f86ffd640 1 --2- 172.21.15.40:0/2846602807 >> [v2:172.21.15.174:6838/2467509274,v1:172.21.15.174:6839/2467509274] conn(0x55a888e4b3c0 0x55a888e4d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:39.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.779+0000 7f5f86ffd640 1 --2- 172.21.15.40:0/2846602807 >> [v2:172.21.15.174:6838/2467509274,v1:172.21.15.174:6839/2467509274] conn(0x55a888e4b3c0 0x55a888e4d7d0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5cb9007e78bc45c6 server_cookie=22d4002c8d60c1b in_seq=0 out_seq=0 2024-04-21T23:20:39.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.779+0000 7f5f66ffd640 10 client.16287 ms_handle_connect on v2:172.21.15.174:6838/2467509274 2024-04-21T23:20:39.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.780+0000 7f5f66ffd640 1 -- 172.21.15.40:0/2846602807 <== mds.1 v2:172.21.15.174:6838/2467509274 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a888da3a70 con 0x55a888e4b3c0 2024-04-21T23:20:39.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.780+0000 7f5f66ffd640 10 client.16287 handle_command_reply: tid=0 2024-04-21T23:20:39.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.780+0000 7f5f90df6740 1 client.16287 shutdown 2024-04-21T23:20:39.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.780+0000 7f5f90df6740 1 -- 172.21.15.40:0/2846602807 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a888e32020 msgr2=0x55a888e3af30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:39.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.780+0000 7f5f90df6740 1 --2- 172.21.15.40:0/2846602807 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a888e32020 0x55a888e3af30 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f5f7800d470 tx=0x7f5f7800d930 comp rx=0 tx=0).stop 2024-04-21T23:20:39.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.780+0000 7f5f90df6740 1 -- 172.21.15.40:0/2846602807 shutdown_connections 2024-04-21T23:20:39.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.780+0000 7f5f90df6740 1 --2- 172.21.15.40:0/2846602807 >> [v2:172.21.15.174:6838/2467509274,v1:172.21.15.174:6839/2467509274] conn(0x55a888e4b3c0 0x55a888e4d7d0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.780+0000 7f5f90df6740 1 --2- 172.21.15.40:0/2846602807 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a888e3b520 0x55a888e388d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.780+0000 7f5f90df6740 1 --2- 172.21.15.40:0/2846602807 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a888e32940 0x55a888e382c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.780+0000 7f5f66ffd640 0 client.16287 ms_handle_reset on v2:172.21.15.174:6838/2467509274 2024-04-21T23:20:39.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.780+0000 7f5f90df6740 1 --2- 172.21.15.40:0/2846602807 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a888e32020 0x55a888e3af30 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.780+0000 7f5f90df6740 1 -- 172.21.15.40:0/2846602807 >> 172.21.15.40:0/2846602807 conn(0x55a888e2c750 msgr2=0x55a888e2e4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:39.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.781+0000 7f5f90df6740 1 -- 172.21.15.40:0/2846602807 shutdown_connections 2024-04-21T23:20:39.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.781+0000 7f5f90df6740 1 -- 172.21.15.40:0/2846602807 wait complete. 2024-04-21T23:20:39.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.781+0000 7f5f90df6740 20 client.16287 trim_cache size 0 max 16384 2024-04-21T23:20:39.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.794+0000 7f5f90df6740 1 Processor -- start 2024-04-21T23:20:39.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.794+0000 7f5f90df6740 1 -- start start 2024-04-21T23:20:39.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.794+0000 7f5f90df6740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a888e32020 0x55a888e2b850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:39.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.794+0000 7f5f90df6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a888e32940 0x55a888e3a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:39.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.794+0000 7f5f90df6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a888e3b520 0x55a888e3af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:39.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.794+0000 7f5f90df6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5f881b1570 con 0x55a888e3b520 2024-04-21T23:20:39.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.794+0000 7f5f90df6740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f5f881aede0 con 0x55a888e32020 2024-04-21T23:20:39.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.794+0000 7f5f90df6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5f881acc30 con 0x55a888e32940 2024-04-21T23:20:39.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.794+0000 7f5f87fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a888e3b520 0x55a888e3af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:39.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.794+0000 7f5f87fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a888e3b520 0x55a888e3af30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42500/0 (socket says 172.21.15.40:42500) 2024-04-21T23:20:39.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.794+0000 7f5f87fff640 1 -- 172.21.15.40:0/3411208554 learned_addr learned my addr 172.21.15.40:0/3411208554 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:20:39.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.794+0000 7f5f877fe640 1 --2- 172.21.15.40:0/3411208554 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a888e32020 0x55a888e2b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:39.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.794+0000 7f5f87fff640 1 -- 172.21.15.40:0/3411208554 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a888e32940 msgr2=0x55a888e3a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:39.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.794+0000 7f5f86ffd640 1 --2- 172.21.15.40:0/3411208554 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a888e32940 0x55a888e3a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:39.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.794+0000 7f5f87fff640 1 --2- 172.21.15.40:0/3411208554 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a888e32940 0x55a888e3a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.794+0000 7f5f87fff640 1 -- 172.21.15.40:0/3411208554 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a888e32020 msgr2=0x55a888e2b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:39.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.794+0000 7f5f87fff640 1 --2- 172.21.15.40:0/3411208554 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a888e32020 0x55a888e2b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.794+0000 7f5f87fff640 1 -- 172.21.15.40:0/3411208554 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a888b747d0 con 0x55a888e3b520 2024-04-21T23:20:39.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.795+0000 7f5f877fe640 1 --2- 172.21.15.40:0/3411208554 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a888e32020 0x55a888e2b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:20:39.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.795+0000 7f5f87fff640 1 --2- 172.21.15.40:0/3411208554 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a888e3b520 0x55a888e3af30 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f5f80007c20 tx=0x7f5f80004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8336454cb3287fbf server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:39.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.795+0000 7f5f6d7fa640 1 -- 172.21.15.40:0/3411208554 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f80024070 con 0x55a888e3b520 2024-04-21T23:20:39.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.795+0000 7f5f6d7fa640 1 -- 172.21.15.40:0/3411208554 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f80020460 con 0x55a888e3b520 2024-04-21T23:20:39.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.795+0000 7f5f6d7fa640 1 -- 172.21.15.40:0/3411208554 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f8001f620 con 0x55a888e3b520 2024-04-21T23:20:39.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.795+0000 7f5f90df6740 1 -- 172.21.15.40:0/3411208554 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a888e3b520 msgr2=0x55a888e3af30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:39.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.795+0000 7f5f90df6740 1 --2- 172.21.15.40:0/3411208554 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a888e3b520 0x55a888e3af30 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f5f80007c20 tx=0x7f5f80004ab0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.795+0000 7f5f90df6740 1 -- 172.21.15.40:0/3411208554 shutdown_connections 2024-04-21T23:20:39.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.795+0000 7f5f90df6740 1 --2- 172.21.15.40:0/3411208554 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a888e3b520 0x55a888e3af30 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.795+0000 7f5f90df6740 1 --2- 172.21.15.40:0/3411208554 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a888e32940 0x55a888e3a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.795+0000 7f5f90df6740 1 --2- 172.21.15.40:0/3411208554 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a888e32020 0x55a888e2b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.795+0000 7f5f90df6740 1 -- 172.21.15.40:0/3411208554 >> 172.21.15.40:0/3411208554 conn(0x55a888e2c750 msgr2=0x55a888e32e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:39.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.796+0000 7f5f90df6740 1 -- 172.21.15.40:0/3411208554 shutdown_connections 2024-04-21T23:20:39.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.796+0000 7f5f90df6740 1 -- 172.21.15.40:0/3411208554 wait complete. 2024-04-21T23:20:39.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.796+0000 7f5f90df6740 1 -- start start 2024-04-21T23:20:39.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.796+0000 7f5f90df6740 1 Processor -- start 2024-04-21T23:20:39.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.796+0000 7f5f90df6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a888e32020 0x55a888e3edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:39.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.796+0000 7f5f90df6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a888e32940 0x55a888e3f330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:39.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.796+0000 7f5f90df6740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a888e3b520 0x55a888e3f870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:39.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.796+0000 7f5f90df6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5f881ad2a0 con 0x55a888e32020 2024-04-21T23:20:39.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.796+0000 7f5f90df6740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f5f881ae710 con 0x55a888e3b520 2024-04-21T23:20:39.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.796+0000 7f5f90df6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5f881ad970 con 0x55a888e32940 2024-04-21T23:20:39.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.796+0000 7f5f90df6740 10 client.0 _ll_register_callbacks cb 0x7f5f7800b500 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:20:39.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.796+0000 7f5f877fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a888e32020 0x55a888e3edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:39.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.796+0000 7f5f877fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a888e32020 0x55a888e3edf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42506/0 (socket says 172.21.15.40:42506) 2024-04-21T23:20:39.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.796+0000 7f5f877fe640 1 -- 172.21.15.40:0/2642234142 learned_addr learned my addr 172.21.15.40:0/2642234142 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:20:39.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.796+0000 7f5f87fff640 1 --2- 172.21.15.40:0/2642234142 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a888e3b520 0x55a888e3f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:39.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.797+0000 7f5f86ffd640 1 --2- 172.21.15.40:0/2642234142 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a888e32940 0x55a888e3f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:39.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.797+0000 7f5f877fe640 1 -- 172.21.15.40:0/2642234142 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a888e32940 msgr2=0x55a888e3f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:39.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.797+0000 7f5f877fe640 1 --2- 172.21.15.40:0/2642234142 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a888e32940 0x55a888e3f330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.797+0000 7f5f877fe640 1 -- 172.21.15.40:0/2642234142 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a888e3b520 msgr2=0x55a888e3f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:39.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.797+0000 7f5f877fe640 1 --2- 172.21.15.40:0/2642234142 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a888e3b520 0x55a888e3f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.797+0000 7f5f877fe640 1 -- 172.21.15.40:0/2642234142 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a888eb0960 con 0x55a888e32020 2024-04-21T23:20:39.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.797+0000 7f5f87fff640 1 --2- 172.21.15.40:0/2642234142 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a888e3b520 0x55a888e3f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:20:39.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.797+0000 7f5f877fe640 1 --2- 172.21.15.40:0/2642234142 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a888e32020 0x55a888e3edf0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f5f7800c010 tx=0x7f5f780257a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e75831ef810fbd2b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:39.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.797+0000 7f5f66ffd640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-21T23:20:39.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.797+0000 7f5f66ffd640 1 -- 172.21.15.40:0/2642234142 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f7802b070 con 0x55a888e32020 2024-04-21T23:20:39.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.797+0000 7f5f66ffd640 1 -- 172.21.15.40:0/2642234142 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f5f78025cd0 con 0x55a888e32020 2024-04-21T23:20:39.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.797+0000 7f5f90df6740 1 -- 172.21.15.40:0/2642234142 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5f80020460 con 0x55a888e32020 2024-04-21T23:20:39.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.797+0000 7f5f66ffd640 1 -- 172.21.15.40:0/2642234142 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5f78037050 con 0x55a888e32020 2024-04-21T23:20:39.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.797+0000 7f5f90df6740 10 client.16293 fetch_fsmap learned FSMap version 29 2024-04-21T23:20:39.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.797+0000 7f5f90df6740 1 -- 172.21.15.40:0/2642234142 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a888eb1c40 con 0x55a888e32020 2024-04-21T23:20:39.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.797+0000 7f5f66ffd640 1 -- 172.21.15.40:0/2642234142 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f80020460 con 0x55a888e32020 2024-04-21T23:20:39.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.797+0000 7f5f66ffd640 1 -- 172.21.15.40:0/2642234142 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f5f78016470 con 0x55a888e32020 2024-04-21T23:20:39.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.798+0000 7f5f90df6740 10 client.16293 fetch_fsmap finished waiting for FSMap version 29 2024-04-21T23:20:39.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.798+0000 7f5f90df6740 10 client.16293 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T23:20:39.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.798+0000 7f5f90df6740 20 client.16293 populate_metadata read hostname 'smithi040' 2024-04-21T23:20:39.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.798+0000 7f5f90df6740 1 --2- 172.21.15.40:0/2642234142 >> [v2:172.21.15.174:6838/2467509274,v1:172.21.15.174:6839/2467509274] conn(0x55a888eb2360 0x55a888eb4780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:39.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.798+0000 7f5f90df6740 4 client.16293 mds_command: new command op to 24502 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T23:20:39.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.798+0000 7f5f90df6740 1 -- 172.21.15.40:0/2642234142 --> [v2:172.21.15.174:6838/2467509274,v1:172.21.15.174:6839/2467509274] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a888d9ddc0 con 0x55a888eb2360 2024-04-21T23:20:39.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.798+0000 7f5f86ffd640 1 --2- 172.21.15.40:0/2642234142 >> [v2:172.21.15.174:6838/2467509274,v1:172.21.15.174:6839/2467509274] conn(0x55a888eb2360 0x55a888eb4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:39.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.798+0000 7f5f86ffd640 1 --2- 172.21.15.40:0/2642234142 >> [v2:172.21.15.174:6838/2467509274,v1:172.21.15.174:6839/2467509274] conn(0x55a888eb2360 0x55a888eb4780 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=14ce8c63b58de38 server_cookie=c17888f29aef048a in_seq=0 out_seq=0 2024-04-21T23:20:39.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.798+0000 7f5f66ffd640 10 client.16293 ms_handle_connect on v2:172.21.15.174:6838/2467509274 2024-04-21T23:20:39.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.798+0000 7f5f66ffd640 1 -- 172.21.15.40:0/2642234142 <== mds.1 v2:172.21.15.174:6838/2467509274 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a888d9ddc0 con 0x55a888eb2360 2024-04-21T23:20:39.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.798+0000 7f5f66ffd640 10 client.16293 handle_command_reply: tid=0 2024-04-21T23:20:39.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.798+0000 7f5f90df6740 1 client.16293 shutdown 2024-04-21T23:20:39.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.799+0000 7f5f90df6740 1 -- 172.21.15.40:0/2642234142 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a888e32020 msgr2=0x55a888e3edf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:39.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.799+0000 7f5f90df6740 1 --2- 172.21.15.40:0/2642234142 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a888e32020 0x55a888e3edf0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f5f7800c010 tx=0x7f5f780257a0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.799+0000 7f5f90df6740 1 -- 172.21.15.40:0/2642234142 shutdown_connections 2024-04-21T23:20:39.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.799+0000 7f5f90df6740 1 --2- 172.21.15.40:0/2642234142 >> [v2:172.21.15.174:6838/2467509274,v1:172.21.15.174:6839/2467509274] conn(0x55a888eb2360 0x55a888eb4780 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.799+0000 7f5f90df6740 1 --2- 172.21.15.40:0/2642234142 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a888e3b520 0x55a888e3f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.799+0000 7f5f90df6740 1 --2- 172.21.15.40:0/2642234142 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a888e32940 0x55a888e3f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.799+0000 7f5f66ffd640 0 client.16293 ms_handle_reset on v2:172.21.15.174:6838/2467509274 2024-04-21T23:20:39.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.799+0000 7f5f90df6740 1 --2- 172.21.15.40:0/2642234142 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a888e32020 0x55a888e3edf0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.799+0000 7f5f90df6740 1 -- 172.21.15.40:0/2642234142 >> 172.21.15.40:0/2642234142 conn(0x55a888e2c750 msgr2=0x55a888e2b6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:39.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.799+0000 7f5f90df6740 1 -- 172.21.15.40:0/2642234142 shutdown_connections 2024-04-21T23:20:39.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.799+0000 7f5f90df6740 1 -- 172.21.15.40:0/2642234142 wait complete. 2024-04-21T23:20:39.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.799+0000 7f5f90df6740 20 client.16293 trim_cache size 0 max 16384 2024-04-21T23:20:39.809 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-04-21T23:20:39.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.799+0000 7f5f8e20e640 1 -- 172.21.15.40:0/1083372996 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f5f700a2d00 msgr2=0x7f5f700a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:39.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.799+0000 7f5f8e20e640 1 --2- 172.21.15.40:0/1083372996 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f5f700a2d00 0x7f5f700a51c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5f80007fd0 tx=0x7f5f80007d60 comp rx=0 tx=0).stop 2024-04-21T23:20:39.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.799+0000 7f5f8e20e640 1 -- 172.21.15.40:0/1083372996 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f881020d0 msgr2=0x7f5f88198590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:39.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.799+0000 7f5f8e20e640 1 --2- 172.21.15.40:0/1083372996 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f881020d0 0x7f5f88198590 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f5f7c004f50 tx=0x7f5f7c02ce20 comp rx=0 tx=0).stop 2024-04-21T23:20:39.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.799+0000 7f5f8e20e640 1 -- 172.21.15.40:0/1083372996 shutdown_connections 2024-04-21T23:20:39.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.799+0000 7f5f8e20e640 1 --2- 172.21.15.40:0/1083372996 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f5f700a2d00 0x7f5f700a51c0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.799+0000 7f5f8e20e640 1 --2- 172.21.15.40:0/1083372996 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f88107720 0x7f5f8819c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.799+0000 7f5f8e20e640 1 --2- 172.21.15.40:0/1083372996 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f881020d0 0x7f5f88198590 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.799+0000 7f5f8e20e640 1 --2- 172.21.15.40:0/1083372996 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5f88101710 0x7f5f88198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.799+0000 7f5f8e20e640 1 -- 172.21.15.40:0/1083372996 >> 172.21.15.40:0/1083372996 conn(0x7f5f88078b90 msgr2=0x7f5f880ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:39.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.799+0000 7f5f8e20e640 1 -- 172.21.15.40:0/1083372996 shutdown_connections 2024-04-21T23:20:39.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.799+0000 7f5f8e20e640 1 -- 172.21.15.40:0/1083372996 wait complete. 2024-04-21T23:20:39.812 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T23:20:39.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.917+0000 7f7b545f5640 1 -- 172.21.15.40:0/1866183961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b4c069dc0 msgr2=0x7f7b4c06a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:39.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.917+0000 7f7b545f5640 1 --2- 172.21.15.40:0/1866183961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b4c069dc0 0x7f7b4c06a240 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c009930 tx=0x7f7b3c02bc30 comp rx=0 tx=0).stop 2024-04-21T23:20:39.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.917+0000 7f7b545f5640 1 -- 172.21.15.40:0/1866183961 shutdown_connections 2024-04-21T23:20:39.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.917+0000 7f7b545f5640 1 --2- 172.21.15.40:0/1866183961 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b4c06a780 0x7f7b4c10e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.917+0000 7f7b545f5640 1 --2- 172.21.15.40:0/1866183961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b4c069dc0 0x7f7b4c06a240 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.917+0000 7f7b545f5640 1 --2- 172.21.15.40:0/1866183961 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b4c107730 0x7f7b4c107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.917+0000 7f7b545f5640 1 -- 172.21.15.40:0/1866183961 >> 172.21.15.40:0/1866183961 conn(0x7f7b4c0776a0 msgr2=0x7f7b4c077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:39.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.917+0000 7f7b545f5640 1 -- 172.21.15.40:0/1866183961 shutdown_connections 2024-04-21T23:20:39.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.917+0000 7f7b545f5640 1 -- 172.21.15.40:0/1866183961 wait complete. 2024-04-21T23:20:39.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.918+0000 7f7b545f5640 1 Processor -- start 2024-04-21T23:20:39.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.918+0000 7f7b545f5640 1 -- start start 2024-04-21T23:20:39.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.918+0000 7f7b545f5640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b4c069dc0 0x7f7b4c198110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:39.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.918+0000 7f7b545f5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b4c06a780 0x7f7b4c198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:39.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.918+0000 7f7b545f5640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b4c107730 0x7f7b4c19c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:39.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.918+0000 7f7b545f5640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7b4c111b90 con 0x7f7b4c06a780 2024-04-21T23:20:39.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.918+0000 7f7b545f5640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f7b4c111a10 con 0x7f7b4c069dc0 2024-04-21T23:20:39.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.918+0000 7f7b545f5640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7b4c111d10 con 0x7f7b4c107730 2024-04-21T23:20:39.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.918+0000 7f7b51b90640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b4c06a780 0x7f7b4c198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:39.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.918+0000 7f7b51b90640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b4c06a780 0x7f7b4c198650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42530/0 (socket says 172.21.15.40:42530) 2024-04-21T23:20:39.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.918+0000 7f7b51b90640 1 -- 172.21.15.40:0/3993873682 learned_addr learned my addr 172.21.15.40:0/3993873682 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:20:39.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.918+0000 7f7b52b92640 1 --2- 172.21.15.40:0/3993873682 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b4c107730 0x7f7b4c19c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:39.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.918+0000 7f7b52391640 1 --2- 172.21.15.40:0/3993873682 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b4c069dc0 0x7f7b4c198110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:39.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.918+0000 7f7b51b90640 1 -- 172.21.15.40:0/3993873682 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b4c107730 msgr2=0x7f7b4c19c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:39.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.918+0000 7f7b51b90640 1 --2- 172.21.15.40:0/3993873682 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b4c107730 0x7f7b4c19c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.918+0000 7f7b51b90640 1 -- 172.21.15.40:0/3993873682 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b4c069dc0 msgr2=0x7f7b4c198110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:39.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.918+0000 7f7b51b90640 1 --2- 172.21.15.40:0/3993873682 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b4c069dc0 0x7f7b4c198110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.918+0000 7f7b51b90640 1 -- 172.21.15.40:0/3993873682 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b4c19d100 con 0x7f7b4c06a780 2024-04-21T23:20:39.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.918+0000 7f7b52b92640 1 --2- 172.21.15.40:0/3993873682 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b4c107730 0x7f7b4c19c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:20:39.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.918+0000 7f7b52391640 1 --2- 172.21.15.40:0/3993873682 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b4c069dc0 0x7f7b4c198110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:20:39.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.918+0000 7f7b51b90640 1 --2- 172.21.15.40:0/3993873682 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b4c06a780 0x7f7b4c198650 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c009900 tx=0x7f7b3c004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:39.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.919+0000 7f7b437fe640 1 -- 172.21.15.40:0/3993873682 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b3c043020 con 0x7f7b4c06a780 2024-04-21T23:20:39.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.919+0000 7f7b545f5640 1 -- 172.21.15.40:0/3993873682 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b4c19d390 con 0x7f7b4c06a780 2024-04-21T23:20:39.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.919+0000 7f7b437fe640 1 -- 172.21.15.40:0/3993873682 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b3c0054e0 con 0x7f7b4c06a780 2024-04-21T23:20:39.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.919+0000 7f7b437fe640 1 -- 172.21.15.40:0/3993873682 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b3c003b80 con 0x7f7b4c06a780 2024-04-21T23:20:39.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.919+0000 7f7b545f5640 1 -- 172.21.15.40:0/3993873682 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b4c1a4c80 con 0x7f7b4c06a780 2024-04-21T23:20:39.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.921+0000 7f7b437fe640 1 -- 172.21.15.40:0/3993873682 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f7b3c046020 con 0x7f7b4c06a780 2024-04-21T23:20:39.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.921+0000 7f7b571dc740 1 Processor -- start 2024-04-21T23:20:39.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.922+0000 7f7b571dc740 1 -- start start 2024-04-21T23:20:39.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.922+0000 7f7b571dc740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55835cdd5020 0x55835cdd5400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:39.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.922+0000 7f7b571dc740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55835cdd5940 0x55835cdddfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:39.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.922+0000 7f7b571dc740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55835cdde520 0x55835cde0910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:39.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.922+0000 7f7b571dc740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55835cd39530 con 0x55835cdd5020 2024-04-21T23:20:39.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.922+0000 7f7b571dc740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55835cce1830 con 0x55835cdde520 2024-04-21T23:20:39.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.922+0000 7f7b571dc740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55835cd354f0 con 0x55835cdd5940 2024-04-21T23:20:39.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.922+0000 7f7b437fe640 1 --2- 172.21.15.40:0/3993873682 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f7b380a2d00 0x7f7b380a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:39.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.922+0000 7f7b52b92640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55835cdd5020 0x55835cdd5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:39.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.922+0000 7f7b52b92640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55835cdd5020 0x55835cdd5400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42544/0 (socket says 172.21.15.40:42544) 2024-04-21T23:20:39.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.922+0000 7f7b52b92640 1 -- 172.21.15.40:0/520015778 learned_addr learned my addr 172.21.15.40:0/520015778 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:20:39.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.922+0000 7f7b52b92640 1 --2- 172.21.15.40:0/3993873682 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f7b380a2d00 0x7f7b380a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:39.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.922+0000 7f7b52391640 1 --2- 172.21.15.40:0/520015778 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55835cdd5940 0x55835cdddfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:39.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.922+0000 7f7b437fe640 1 -- 172.21.15.40:0/3993873682 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f7b3c0f1950 con 0x7f7b4c06a780 2024-04-21T23:20:39.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.922+0000 7f7b51b90640 1 --2- 172.21.15.40:0/520015778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55835cdde520 0x55835cde0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:39.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.923+0000 7f7b52391640 1 -- 172.21.15.40:0/520015778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55835cdde520 msgr2=0x55835cde0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:39.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.923+0000 7f7b52391640 1 --2- 172.21.15.40:0/520015778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55835cdde520 0x55835cde0910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.923+0000 7f7b52391640 1 -- 172.21.15.40:0/520015778 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55835cdd5020 msgr2=0x55835cdd5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:39.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.923+0000 7f7b52391640 1 --2- 172.21.15.40:0/520015778 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55835cdd5020 0x55835cdd5400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.923+0000 7f7b52391640 1 -- 172.21.15.40:0/520015778 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55835cd052c0 con 0x55835cdd5940 2024-04-21T23:20:39.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.923+0000 7f7b52b92640 1 --2- 172.21.15.40:0/3993873682 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f7b380a2d00 0x7f7b380a51c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7b44009270 tx=0x7f7b44009be0 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:39.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.923+0000 7f7b52391640 1 --2- 172.21.15.40:0/520015778 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55835cdd5940 0x55835cdddfe0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7b48009af0 tx=0x7f7b4800e2f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d7fc41968492260 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:39.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.923+0000 7f7b40bfe640 1 -- 172.21.15.40:0/520015778 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b48016070 con 0x55835cdd5940 2024-04-21T23:20:39.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.923+0000 7f7b40bfe640 1 -- 172.21.15.40:0/520015778 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b48005500 con 0x55835cdd5940 2024-04-21T23:20:39.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.923+0000 7f7b40bfe640 1 -- 172.21.15.40:0/520015778 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b48011470 con 0x55835cdd5940 2024-04-21T23:20:39.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.924+0000 7f7b571dc740 1 -- 172.21.15.40:0/520015778 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55835cdd5940 msgr2=0x55835cdddfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:39.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.924+0000 7f7b571dc740 1 --2- 172.21.15.40:0/520015778 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55835cdd5940 0x55835cdddfe0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7b48009af0 tx=0x7f7b4800e2f0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.924+0000 7f7b571dc740 1 -- 172.21.15.40:0/520015778 shutdown_connections 2024-04-21T23:20:39.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.924+0000 7f7b571dc740 1 --2- 172.21.15.40:0/520015778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55835cdde520 0x55835cde0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.924+0000 7f7b571dc740 1 --2- 172.21.15.40:0/520015778 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55835cdd5940 0x55835cdddfe0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.924+0000 7f7b571dc740 1 --2- 172.21.15.40:0/520015778 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55835cdd5020 0x55835cdd5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.924+0000 7f7b571dc740 1 -- 172.21.15.40:0/520015778 >> 172.21.15.40:0/520015778 conn(0x55835cdcf750 msgr2=0x55835cdcfb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:39.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.924+0000 7f7b571dc740 1 -- 172.21.15.40:0/520015778 shutdown_connections 2024-04-21T23:20:39.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.924+0000 7f7b571dc740 1 -- 172.21.15.40:0/520015778 wait complete. 2024-04-21T23:20:39.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.924+0000 7f7b571dc740 1 -- start start 2024-04-21T23:20:39.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.924+0000 7f7b571dc740 1 Processor -- start 2024-04-21T23:20:39.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.925+0000 7f7b571dc740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55835cdd5020 0x55835cddde80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:39.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.925+0000 7f7b571dc740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55835cdd5940 0x55835cddb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:39.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.925+0000 7f7b571dc740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55835cdde520 0x55835cddb770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:39.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.925+0000 7f7b571dc740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55835cde1310 con 0x55835cdd5020 2024-04-21T23:20:39.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.925+0000 7f7b571dc740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55835ccf4f10 con 0x55835cdd5940 2024-04-21T23:20:39.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.925+0000 7f7b571dc740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55835cde1490 con 0x55835cdde520 2024-04-21T23:20:39.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.925+0000 7f7b571dc740 10 client.0 _ll_register_callbacks cb 0x55835ca4cbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:20:39.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.925+0000 7f7b52391640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55835cdd5020 0x55835cddde80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:39.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.925+0000 7f7b52391640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55835cdd5020 0x55835cddde80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42548/0 (socket says 172.21.15.40:42548) 2024-04-21T23:20:39.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.925+0000 7f7b52391640 1 -- 172.21.15.40:0/3611515537 learned_addr learned my addr 172.21.15.40:0/3611515537 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:20:39.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.925+0000 7f7b52b92640 1 --2- 172.21.15.40:0/3611515537 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55835cdde520 0x55835cddb770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:39.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.925+0000 7f7b51b90640 1 --2- 172.21.15.40:0/3611515537 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55835cdd5940 0x55835cddb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:39.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.925+0000 7f7b52391640 1 -- 172.21.15.40:0/3611515537 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55835cdde520 msgr2=0x55835cddb770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:39.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.925+0000 7f7b52391640 1 --2- 172.21.15.40:0/3611515537 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55835cdde520 0x55835cddb770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.925+0000 7f7b52391640 1 -- 172.21.15.40:0/3611515537 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55835cdd5940 msgr2=0x55835cddb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:39.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.925+0000 7f7b52391640 1 --2- 172.21.15.40:0/3611515537 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55835cdd5940 0x55835cddb210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.925+0000 7f7b52391640 1 -- 172.21.15.40:0/3611515537 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55835ccc6c80 con 0x55835cdd5020 2024-04-21T23:20:39.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.925+0000 7f7b52391640 1 --2- 172.21.15.40:0/3611515537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55835cdd5020 0x55835cddde80 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f7b48017000 tx=0x7f7b48004f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47ee952c614e34bf server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:39.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.925+0000 7f7b35ffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-21T23:20:39.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.925+0000 7f7b35ffb640 1 -- 172.21.15.40:0/3611515537 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b48028060 con 0x55835cdd5020 2024-04-21T23:20:39.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.925+0000 7f7b35ffb640 1 -- 172.21.15.40:0/3611515537 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f7b4800ecb0 con 0x55835cdd5020 2024-04-21T23:20:39.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.925+0000 7f7b571dc740 1 -- 172.21.15.40:0/3611515537 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7b48005500 con 0x55835cdd5020 2024-04-21T23:20:39.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.925+0000 7f7b35ffb640 1 -- 172.21.15.40:0/3611515537 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f7b4802f050 con 0x55835cdd5020 2024-04-21T23:20:39.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.926+0000 7f7b571dc740 10 client.16305 fetch_fsmap learned FSMap version 29 2024-04-21T23:20:39.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.926+0000 7f7b571dc740 1 -- 172.21.15.40:0/3611515537 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55835ca4d3f0 con 0x55835cdd5020 2024-04-21T23:20:39.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.926+0000 7f7b35ffb640 1 -- 172.21.15.40:0/3611515537 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b48005500 con 0x55835cdd5020 2024-04-21T23:20:39.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.926+0000 7f7b35ffb640 1 -- 172.21.15.40:0/3611515537 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f7b4800f040 con 0x55835cdd5020 2024-04-21T23:20:39.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.926+0000 7f7b571dc740 10 client.16305 fetch_fsmap finished waiting for FSMap version 29 2024-04-21T23:20:39.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.926+0000 7f7b571dc740 10 client.16305 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:20:39.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.926+0000 7f7b571dc740 20 client.16305 populate_metadata read hostname 'smithi040' 2024-04-21T23:20:39.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.926+0000 7f7b571dc740 1 --2- 172.21.15.40:0/3611515537 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55835cdee510 0x55835cdf0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:39.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.926+0000 7f7b571dc740 4 client.16305 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:20:39.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.926+0000 7f7b571dc740 1 -- 172.21.15.40:0/3611515537 --> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55835cd46a70 con 0x55835cdee510 2024-04-21T23:20:39.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.926+0000 7f7b51b90640 1 --2- 172.21.15.40:0/3611515537 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55835cdee510 0x55835cdf0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:39.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.927+0000 7f7b51b90640 1 --2- 172.21.15.40:0/3611515537 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55835cdee510 0x55835cdf0900 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6be80dd6f02e119 server_cookie=edadfbad3c180cf1 in_seq=0 out_seq=0 2024-04-21T23:20:39.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.927+0000 7f7b35ffb640 10 client.16305 ms_handle_connect on v2:172.21.15.111:6838/3712235548 2024-04-21T23:20:39.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.927+0000 7f7b35ffb640 1 -- 172.21.15.40:0/3611515537 <== mds.0 v2:172.21.15.111:6838/3712235548 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55835cd46a70 con 0x55835cdee510 2024-04-21T23:20:39.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.927+0000 7f7b35ffb640 10 client.16305 handle_command_reply: tid=0 2024-04-21T23:20:39.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.927+0000 7f7b571dc740 1 client.16305 shutdown 2024-04-21T23:20:39.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.928+0000 7f7b571dc740 1 -- 172.21.15.40:0/3611515537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55835cdd5020 msgr2=0x55835cddde80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:39.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.928+0000 7f7b571dc740 1 --2- 172.21.15.40:0/3611515537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55835cdd5020 0x55835cddde80 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f7b48017000 tx=0x7f7b48004f60 comp rx=0 tx=0).stop 2024-04-21T23:20:39.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.928+0000 7f7b571dc740 1 -- 172.21.15.40:0/3611515537 shutdown_connections 2024-04-21T23:20:39.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.928+0000 7f7b571dc740 1 --2- 172.21.15.40:0/3611515537 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55835cdee510 0x55835cdf0900 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.928+0000 7f7b571dc740 1 --2- 172.21.15.40:0/3611515537 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55835cdd5940 0x55835cddb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.928+0000 7f7b571dc740 1 --2- 172.21.15.40:0/3611515537 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55835cdde520 0x55835cddb770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.928+0000 7f7b35ffb640 0 client.16305 ms_handle_reset on v2:172.21.15.111:6838/3712235548 2024-04-21T23:20:39.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.928+0000 7f7b571dc740 1 --2- 172.21.15.40:0/3611515537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55835cdd5020 0x55835cddde80 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.928+0000 7f7b571dc740 1 -- 172.21.15.40:0/3611515537 >> 172.21.15.40:0/3611515537 conn(0x55835cdcf750 msgr2=0x55835cdd14a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:39.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.928+0000 7f7b571dc740 1 -- 172.21.15.40:0/3611515537 shutdown_connections 2024-04-21T23:20:39.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.928+0000 7f7b571dc740 1 -- 172.21.15.40:0/3611515537 wait complete. 2024-04-21T23:20:39.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.928+0000 7f7b571dc740 20 client.16305 trim_cache size 0 max 16384 2024-04-21T23:20:39.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.941+0000 7f7b571dc740 1 Processor -- start 2024-04-21T23:20:39.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.941+0000 7f7b571dc740 1 -- start start 2024-04-21T23:20:39.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.941+0000 7f7b571dc740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55835cdd5020 0x55835cdea1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:39.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.941+0000 7f7b571dc740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55835cdd5940 0x55835cddd940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:39.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.941+0000 7f7b571dc740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55835cdde520 0x55835cddde80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:39.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.941+0000 7f7b571dc740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7b4c1b15f0 con 0x55835cdde520 2024-04-21T23:20:39.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.941+0000 7f7b571dc740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f7b4c1aee60 con 0x55835cdd5020 2024-04-21T23:20:39.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.941+0000 7f7b571dc740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7b4c1accb0 con 0x55835cdd5940 2024-04-21T23:20:39.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.941+0000 7f7b52b92640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55835cdde520 0x55835cddde80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:39.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.941+0000 7f7b52b92640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55835cdde520 0x55835cddde80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42556/0 (socket says 172.21.15.40:42556) 2024-04-21T23:20:39.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.941+0000 7f7b52b92640 1 -- 172.21.15.40:0/3560562659 learned_addr learned my addr 172.21.15.40:0/3560562659 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:20:39.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.941+0000 7f7b51b90640 1 --2- 172.21.15.40:0/3560562659 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55835cdd5940 0x55835cddd940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:39.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.942+0000 7f7b52391640 1 --2- 172.21.15.40:0/3560562659 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55835cdd5020 0x55835cdea1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:39.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.942+0000 7f7b52b92640 1 -- 172.21.15.40:0/3560562659 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55835cdd5940 msgr2=0x55835cddd940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:39.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.942+0000 7f7b52b92640 1 --2- 172.21.15.40:0/3560562659 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55835cdd5940 0x55835cddd940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.942+0000 7f7b52b92640 1 -- 172.21.15.40:0/3560562659 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55835cdd5020 msgr2=0x55835cdea1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:39.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.942+0000 7f7b52b92640 1 --2- 172.21.15.40:0/3560562659 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55835cdd5020 0x55835cdea1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.942+0000 7f7b52b92640 1 -- 172.21.15.40:0/3560562659 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55835cb177d0 con 0x55835cdde520 2024-04-21T23:20:39.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.942+0000 7f7b52b92640 1 --2- 172.21.15.40:0/3560562659 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55835cdde520 0x55835cddde80 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f7b44011c40 tx=0x7f7b4401f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1667a2760217b0be server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:39.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.942+0000 7f7b37fff640 1 -- 172.21.15.40:0/3560562659 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b440045b0 con 0x55835cdde520 2024-04-21T23:20:39.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.942+0000 7f7b37fff640 1 -- 172.21.15.40:0/3560562659 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b44004750 con 0x55835cdde520 2024-04-21T23:20:39.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.942+0000 7f7b37fff640 1 -- 172.21.15.40:0/3560562659 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b44020650 con 0x55835cdde520 2024-04-21T23:20:39.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.943+0000 7f7b571dc740 1 -- 172.21.15.40:0/3560562659 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55835cdde520 msgr2=0x55835cddde80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:39.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.943+0000 7f7b571dc740 1 --2- 172.21.15.40:0/3560562659 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55835cdde520 0x55835cddde80 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f7b44011c40 tx=0x7f7b4401f000 comp rx=0 tx=0).stop 2024-04-21T23:20:39.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.943+0000 7f7b571dc740 1 -- 172.21.15.40:0/3560562659 shutdown_connections 2024-04-21T23:20:39.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.943+0000 7f7b571dc740 1 --2- 172.21.15.40:0/3560562659 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55835cdde520 0x55835cddde80 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.943+0000 7f7b571dc740 1 --2- 172.21.15.40:0/3560562659 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55835cdd5940 0x55835cddd940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.943+0000 7f7b571dc740 1 --2- 172.21.15.40:0/3560562659 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55835cdd5020 0x55835cdea1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.943+0000 7f7b571dc740 1 -- 172.21.15.40:0/3560562659 >> 172.21.15.40:0/3560562659 conn(0x55835cdcf750 msgr2=0x55835cdce850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:39.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.943+0000 7f7b571dc740 1 -- 172.21.15.40:0/3560562659 shutdown_connections 2024-04-21T23:20:39.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.943+0000 7f7b571dc740 1 -- 172.21.15.40:0/3560562659 wait complete. 2024-04-21T23:20:39.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.943+0000 7f7b571dc740 1 -- start start 2024-04-21T23:20:39.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.943+0000 7f7b571dc740 1 Processor -- start 2024-04-21T23:20:39.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.943+0000 7f7b571dc740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55835cdd5020 0x55835cde6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:39.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.943+0000 7f7b571dc740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55835cdd5940 0x55835cde1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:39.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.943+0000 7f7b571dc740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55835cdde520 0x55835cde23d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:39.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.943+0000 7f7b571dc740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7b4c1ad320 con 0x55835cdd5020 2024-04-21T23:20:39.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.943+0000 7f7b571dc740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f7b4c1ae790 con 0x55835cdde520 2024-04-21T23:20:39.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.943+0000 7f7b571dc740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7b4c1ad9f0 con 0x55835cdd5940 2024-04-21T23:20:39.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.943+0000 7f7b571dc740 10 client.0 _ll_register_callbacks cb 0x7f7b44014470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:20:39.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.943+0000 7f7b52391640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55835cdd5020 0x55835cde6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:39.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.943+0000 7f7b52391640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55835cdd5020 0x55835cde6ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42568/0 (socket says 172.21.15.40:42568) 2024-04-21T23:20:39.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.943+0000 7f7b52391640 1 -- 172.21.15.40:0/3994452675 learned_addr learned my addr 172.21.15.40:0/3994452675 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:20:39.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.944+0000 7f7b52b92640 1 --2- 172.21.15.40:0/3994452675 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55835cdde520 0x55835cde23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:39.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.943+0000 7f7b51b90640 1 --2- 172.21.15.40:0/3994452675 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55835cdd5940 0x55835cde1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:39.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.944+0000 7f7b52391640 1 -- 172.21.15.40:0/3994452675 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55835cdd5940 msgr2=0x55835cde1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:39.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.944+0000 7f7b52391640 1 --2- 172.21.15.40:0/3994452675 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55835cdd5940 0x55835cde1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.944+0000 7f7b52391640 1 -- 172.21.15.40:0/3994452675 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55835cdde520 msgr2=0x55835cde23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:20:39.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.944+0000 7f7b52391640 1 --2- 172.21.15.40:0/3994452675 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55835cdde520 0x55835cde23d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.944+0000 7f7b52391640 1 -- 172.21.15.40:0/3994452675 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55835cde2bd0 con 0x55835cdd5020 2024-04-21T23:20:39.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.944+0000 7f7b52b92640 1 --2- 172.21.15.40:0/3994452675 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55835cdde520 0x55835cde23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:20:39.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.944+0000 7f7b51b90640 1 --2- 172.21.15.40:0/3994452675 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55835cdd5940 0x55835cde1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:20:39.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.944+0000 7f7b52391640 1 --2- 172.21.15.40:0/3994452675 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55835cdd5020 0x55835cde6ad0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f7b48011df0 tx=0x7f7b48023410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=824530138f9c314e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:20:39.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.944+0000 7f7b35ffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-21T23:20:39.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.944+0000 7f7b35ffb640 1 -- 172.21.15.40:0/3994452675 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b48028050 con 0x55835cdd5020 2024-04-21T23:20:39.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.944+0000 7f7b35ffb640 1 -- 172.21.15.40:0/3994452675 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f7b48005520 con 0x55835cdd5020 2024-04-21T23:20:39.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.944+0000 7f7b571dc740 1 -- 172.21.15.40:0/3994452675 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7b44004750 con 0x55835cdd5020 2024-04-21T23:20:39.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.944+0000 7f7b35ffb640 1 -- 172.21.15.40:0/3994452675 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f7b4803e050 con 0x55835cdd5020 2024-04-21T23:20:39.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.944+0000 7f7b571dc740 10 client.16311 fetch_fsmap learned FSMap version 29 2024-04-21T23:20:39.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.944+0000 7f7b571dc740 1 -- 172.21.15.40:0/3994452675 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55835ce548e0 con 0x55835cdd5020 2024-04-21T23:20:39.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.944+0000 7f7b35ffb640 1 -- 172.21.15.40:0/3994452675 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b44004750 con 0x55835cdd5020 2024-04-21T23:20:39.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.944+0000 7f7b35ffb640 1 -- 172.21.15.40:0/3994452675 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f7b480117e0 con 0x55835cdd5020 2024-04-21T23:20:39.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.944+0000 7f7b571dc740 10 client.16311 fetch_fsmap finished waiting for FSMap version 29 2024-04-21T23:20:39.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.944+0000 7f7b571dc740 10 client.16311 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:20:39.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.945+0000 7f7b571dc740 20 client.16311 populate_metadata read hostname 'smithi040' 2024-04-21T23:20:39.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.945+0000 7f7b571dc740 1 --2- 172.21.15.40:0/3994452675 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55835ce550e0 0x55835ce57500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:20:39.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.945+0000 7f7b571dc740 4 client.16311 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T23:20:39.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.945+0000 7f7b571dc740 1 -- 172.21.15.40:0/3994452675 --> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55835cd40dc0 con 0x55835ce550e0 2024-04-21T23:20:39.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.945+0000 7f7b51b90640 1 --2- 172.21.15.40:0/3994452675 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55835ce550e0 0x55835ce57500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:20:39.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.945+0000 7f7b51b90640 1 --2- 172.21.15.40:0/3994452675 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55835ce550e0 0x55835ce57500 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d3fc214f4bf2850 server_cookie=adc754c33c0108cf in_seq=0 out_seq=0 2024-04-21T23:20:39.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.945+0000 7f7b35ffb640 10 client.16311 ms_handle_connect on v2:172.21.15.111:6838/3712235548 2024-04-21T23:20:39.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.945+0000 7f7b35ffb640 1 -- 172.21.15.40:0/3994452675 <== mds.0 v2:172.21.15.111:6838/3712235548 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55835cd40dc0 con 0x55835ce550e0 2024-04-21T23:20:39.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.945+0000 7f7b35ffb640 10 client.16311 handle_command_reply: tid=0 2024-04-21T23:20:39.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.945+0000 7f7b571dc740 1 client.16311 shutdown 2024-04-21T23:20:39.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.945+0000 7f7b571dc740 1 -- 172.21.15.40:0/3994452675 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55835cdd5020 msgr2=0x55835cde6ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:39.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.945+0000 7f7b571dc740 1 --2- 172.21.15.40:0/3994452675 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55835cdd5020 0x55835cde6ad0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f7b48011df0 tx=0x7f7b48023410 comp rx=0 tx=0).stop 2024-04-21T23:20:39.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.945+0000 7f7b571dc740 1 -- 172.21.15.40:0/3994452675 shutdown_connections 2024-04-21T23:20:39.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.945+0000 7f7b571dc740 1 --2- 172.21.15.40:0/3994452675 >> [v2:172.21.15.111:6838/3712235548,v1:172.21.15.111:6839/3712235548] conn(0x55835ce550e0 0x55835ce57500 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.945+0000 7f7b571dc740 1 --2- 172.21.15.40:0/3994452675 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55835cdde520 0x55835cde23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.946+0000 7f7b571dc740 1 --2- 172.21.15.40:0/3994452675 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55835cdd5940 0x55835cde1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.946+0000 7f7b35ffb640 0 client.16311 ms_handle_reset on v2:172.21.15.111:6838/3712235548 2024-04-21T23:20:39.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.946+0000 7f7b571dc740 1 --2- 172.21.15.40:0/3994452675 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55835cdd5020 0x55835cde6ad0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.946+0000 7f7b571dc740 1 -- 172.21.15.40:0/3994452675 >> 172.21.15.40:0/3994452675 conn(0x55835cdcf750 msgr2=0x55835cdd5f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:39.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.946+0000 7f7b571dc740 1 -- 172.21.15.40:0/3994452675 shutdown_connections 2024-04-21T23:20:39.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.946+0000 7f7b571dc740 1 -- 172.21.15.40:0/3994452675 wait complete. 2024-04-21T23:20:39.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.946+0000 7f7b571dc740 20 client.16311 trim_cache size 0 max 16384 2024-04-21T23:20:39.955 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-04-21T23:20:39.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.946+0000 7f7b545f5640 1 -- 172.21.15.40:0/3993873682 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f7b380a2d00 msgr2=0x7f7b380a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:39.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.946+0000 7f7b545f5640 1 --2- 172.21.15.40:0/3993873682 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f7b380a2d00 0x7f7b380a51c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7b44009270 tx=0x7f7b44009be0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.946+0000 7f7b545f5640 1 -- 172.21.15.40:0/3993873682 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b4c06a780 msgr2=0x7f7b4c198650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:20:39.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.946+0000 7f7b545f5640 1 --2- 172.21.15.40:0/3993873682 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b4c06a780 0x7f7b4c198650 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c009900 tx=0x7f7b3c004360 comp rx=0 tx=0).stop 2024-04-21T23:20:39.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.946+0000 7f7b545f5640 1 -- 172.21.15.40:0/3993873682 shutdown_connections 2024-04-21T23:20:39.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.946+0000 7f7b545f5640 1 --2- 172.21.15.40:0/3993873682 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f7b380a2d00 0x7f7b380a51c0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.946+0000 7f7b545f5640 1 --2- 172.21.15.40:0/3993873682 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b4c107730 0x7f7b4c19c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.946+0000 7f7b545f5640 1 --2- 172.21.15.40:0/3993873682 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b4c06a780 0x7f7b4c198650 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.946+0000 7f7b545f5640 1 --2- 172.21.15.40:0/3993873682 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b4c069dc0 0x7f7b4c198110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:20:39.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.946+0000 7f7b545f5640 1 -- 172.21.15.40:0/3993873682 >> 172.21.15.40:0/3993873682 conn(0x7f7b4c0776a0 msgr2=0x7f7b4c0fe650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:20:39.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.946+0000 7f7b545f5640 1 -- 172.21.15.40:0/3993873682 shutdown_connections 2024-04-21T23:20:39.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:20:39.946+0000 7f7b545f5640 1 -- 172.21.15.40:0/3993873682 wait complete. 2024-04-21T23:20:39.959 ERROR:tasks.fwd_scrub:rank 2 damaged: [{'damage_type': 'backtrace', 'id': 1558614, 'ino': 1099511629106, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/199.out'}, {'damage_type': 'backtrace', 'id': 4040619, 'ino': 1099511629088, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/190.out'}, {'damage_type': 'backtrace', 'id': 11948687, 'ino': 1099511628942, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/117.out'}, {'damage_type': 'backtrace', 'id': 16069758, 'ino': 1099511629615, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/050.out.nojournal'}, {'damage_type': 'backtrace', 'id': 23422806, 'ino': 1099511629607, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/047.out'}, {'damage_type': 'backtrace', 'id': 41751727, 'ino': 1099511629513, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/001.out'}, {'damage_type': 'backtrace', 'id': 50174551, 'ino': 1099511628968, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/130.out'}, {'damage_type': 'backtrace', 'id': 62929613, 'ino': 1099511629497, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/305.out'}, {'damage_type': 'backtrace', 'id': 69625376, 'ino': 1099511629098, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/195.out'}, {'damage_type': 'backtrace', 'id': 80410373, 'ino': 1099511628940, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/116.out'}, {'damage_type': 'backtrace', 'id': 89963991, 'ino': 1099511629561, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/025.out'}, {'damage_type': 'backtrace', 'id': 111470969, 'ino': 1099511629369, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/001.out.nozero'}, {'damage_type': 'backtrace', 'id': 123940486, 'ino': 1099511628852, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/072.out'}, {'damage_type': 'backtrace', 'id': 125500820, 'ino': 1099511629547, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/018.out'}, {'damage_type': 'backtrace', 'id': 155045648, 'ino': 1099511629094, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/193.out'}, {'damage_type': 'backtrace', 'id': 165499561, 'ino': 1099511628896, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/094.out'}, {'damage_type': 'backtrace', 'id': 187285845, 'ino': 1099511628754, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/023.out'}, {'damage_type': 'backtrace', 'id': 190192161, 'ino': 1099511629611, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/049.out'}, {'damage_type': 'backtrace', 'id': 206231709, 'ino': 1099511628828, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/060.out'}, {'damage_type': 'backtrace', 'id': 206448034, 'ino': 1099511629407, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/020.out'}, {'damage_type': 'backtrace', 'id': 226150877, 'ino': 1099511629395, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/014.out'}, {'damage_type': 'backtrace', 'id': 228112919, 'ino': 1099511629010, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/151.out'}, {'damage_type': 'backtrace', 'id': 232252921, 'ino': 1099511629593, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/040.out'}, {'damage_type': 'backtrace', 'id': 235955808, 'ino': 1099511629024, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/158.out'}, {'damage_type': 'backtrace', 'id': 236379226, 'ino': 1099511629545, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/017.out'}, {'damage_type': 'backtrace', 'id': 238310727, 'ino': 1099511629445, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/039.out'}, {'damage_type': 'backtrace', 'id': 282350104, 'ino': 1099511629652, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/068.out'}, {'damage_type': 'backtrace', 'id': 282361384, 'ino': 1099511629523, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/006.out'}, {'damage_type': 'backtrace', 'id': 316585539, 'ino': 1099511629521, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/005.out'}, {'damage_type': 'backtrace', 'id': 317155038, 'ino': 1099511628938, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/115.out'}, {'damage_type': 'backtrace', 'id': 319439684, 'ino': 1099511629102, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/197.out'}, {'damage_type': 'backtrace', 'id': 354473611, 'ino': 1099511628834, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/063.out'}, {'damage_type': 'backtrace', 'id': 389349991, 'ino': 1099511628752, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/022.out'}, {'damage_type': 'backtrace', 'id': 407478123, 'ino': 1099511628788, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/040.out'}, {'damage_type': 'backtrace', 'id': 419175760, 'ino': 1099511628918, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/105.out'}, {'damage_type': 'backtrace', 'id': 424794259, 'ino': 1099511629595, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/041.out'}, {'damage_type': 'backtrace', 'id': 443892279, 'ino': 1099511629582, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/035.out.default'}, {'damage_type': 'backtrace', 'id': 460231992, 'ino': 1099511628778, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/035.out'}, {'damage_type': 'backtrace', 'id': 482935294, 'ino': 1099511629632, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/058.out'}, {'damage_type': 'backtrace', 'id': 491907449, 'ino': 1099511628916, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/104.out'}, {'damage_type': 'backtrace', 'id': 492753485, 'ino': 1099511629405, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/019.out'}, {'damage_type': 'backtrace', 'id': 531868019, 'ino': 1099511629112, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/202.out'}, {'damage_type': 'backtrace', 'id': 547391765, 'ino': 1099511629533, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/011.out'}, {'damage_type': 'backtrace', 'id': 548131885, 'ino': 1099511628850, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/071.out'}, {'damage_type': 'backtrace', 'id': 595940767, 'ino': 1099511629064, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/178.out'}, {'damage_type': 'backtrace', 'id': 612632336, 'ino': 1099511628910, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/101.out'}, {'damage_type': 'backtrace', 'id': 617227265, 'ino': 1099511629014, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/153.out'}, {'damage_type': 'backtrace', 'id': 681225366, 'ino': 1099511628950, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/121.out'}, {'damage_type': 'backtrace', 'id': 696091375, 'ino': 1099511628770, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/031.out'}, {'damage_type': 'backtrace', 'id': 696097808, 'ino': 1099511629583, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/035.out.nfs'}, {'damage_type': 'backtrace', 'id': 698259075, 'ino': 1099511629626, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/055.out'}, {'damage_type': 'backtrace', 'id': 703326954, 'ino': 1099511628946, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/119.out'}, {'damage_type': 'backtrace', 'id': 705490526, 'ino': 1099511628716, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/004.out'}, {'damage_type': 'backtrace', 'id': 724738175, 'ino': 1099511628742, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/017.out'}, {'damage_type': 'backtrace', 'id': 725301930, 'ino': 1099511629040, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/166.out'}, {'damage_type': 'backtrace', 'id': 743572308, 'ino': 1099511629409, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/021.out'}, {'damage_type': 'backtrace', 'id': 760477146, 'ino': 1099511628736, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/014.out'}, {'damage_type': 'backtrace', 'id': 762632660, 'ino': 1099511629634, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/059.out'}, {'damage_type': 'backtrace', 'id': 773648391, 'ino': 1099511629431, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/032.out'}, {'damage_type': 'backtrace', 'id': 789618812, 'ino': 1099511628870, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/081.out'}, {'damage_type': 'backtrace', 'id': 814575750, 'ino': 1099511628890, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/091.out'}, {'damage_type': 'backtrace', 'id': 868638728, 'ino': 1099511629599, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/043.out'}, {'damage_type': 'backtrace', 'id': 921551364, 'ino': 1099511629529, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/009.out'}, {'damage_type': 'backtrace', 'id': 935417871, 'ino': 1099511629591, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/039.out'}, {'damage_type': 'backtrace', 'id': 942050961, 'ino': 1099511629644, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/064.out'}, {'damage_type': 'backtrace', 'id': 948685900, 'ino': 1099511629618, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/051.out'}, {'damage_type': 'backtrace', 'id': 958076935, 'ino': 1099511628738, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/015.out'}, {'damage_type': 'backtrace', 'id': 967921024, 'ino': 1099511629012, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/152.out'}, {'damage_type': 'backtrace', 'id': 1000672300, 'ino': 1099511629008, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/150.out'}, {'damage_type': 'backtrace', 'id': 1011174001, 'ino': 1099511629383, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/008.out'}, {'damage_type': 'backtrace', 'id': 1014444164, 'ino': 1099511628792, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/042.out'}, {'damage_type': 'backtrace', 'id': 1031517687, 'ino': 1099511629613, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/050.cfg'}, {'damage_type': 'backtrace', 'id': 1045952921, 'ino': 1099511629078, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/185.out'}, {'damage_type': 'backtrace', 'id': 1048158583, 'ino': 1099511629421, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/027.out'}, {'damage_type': 'backtrace', 'id': 1117141399, 'ino': 1099511629636, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/060.out'}, {'damage_type': 'backtrace', 'id': 1119626783, 'ino': 1099511629006, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/149.out'}, {'damage_type': 'backtrace', 'id': 1125954055, 'ino': 1099511628802, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/047.out'}, {'damage_type': 'backtrace', 'id': 1133950306, 'ino': 1099511629487, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/271.out'}, {'damage_type': 'backtrace', 'id': 1151021417, 'ino': 1099511629555, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/022.out'}, {'damage_type': 'backtrace', 'id': 1176876284, 'ino': 1099511628902, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/097.out'}, {'damage_type': 'backtrace', 'id': 1187700839, 'ino': 1099511629403, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/018.out'}, {'damage_type': 'backtrace', 'id': 1196187190, 'ino': 1099511628726, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/009.out'}, {'damage_type': 'backtrace', 'id': 1206023674, 'ino': 1099511629050, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/171.out'}, {'damage_type': 'backtrace', 'id': 1244103982, 'ino': 1099511629628, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/056.out'}, {'damage_type': 'backtrace', 'id': 1261685445, 'ino': 1099511628904, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/098.out'}, {'damage_type': 'backtrace', 'id': 1279659048, 'ino': 1099511628728, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/010.out'}, {'damage_type': 'backtrace', 'id': 1310071881, 'ino': 1099511628858, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/075.out'}, {'damage_type': 'backtrace', 'id': 1313425542, 'ino': 1099511629505, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/f2fs'}, {'damage_type': 'backtrace', 'id': 1317632627, 'ino': 1099511628812, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/052.out'}, {'damage_type': 'backtrace', 'id': 1326014882, 'ino': 1099511629443, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/038.out'}, {'damage_type': 'backtrace', 'id': 1365744435, 'ino': 1099511629535, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/012.out'}, {'damage_type': 'backtrace', 'id': 1376291464, 'ino': 1099511628760, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/026.out'}, {'damage_type': 'backtrace', 'id': 1386323312, 'ino': 1099511628846, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/069.out'}, {'damage_type': 'backtrace', 'id': 1391449221, 'ino': 1099511628842, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/067.out'}, {'damage_type': 'backtrace', 'id': 1404352791, 'ino': 1099511628864, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/078.out'}, {'damage_type': 'backtrace', 'id': 1413686772, 'ino': 1099511629042, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/167.out'}, {'damage_type': 'backtrace', 'id': 1413882331, 'ino': 1099511628966, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/129.out'}, {'damage_type': 'backtrace', 'id': 1432017438, 'ino': 1099511628990, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/141.out'}, {'damage_type': 'backtrace', 'id': 1436661491, 'ino': 1099511629038, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/165.out'}, {'damage_type': 'backtrace', 'id': 1471084343, 'ino': 1099511629622, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/053.out'}, {'damage_type': 'backtrace', 'id': 1481766915, 'ino': 1099511628798, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/045.out'}, {'damage_type': 'backtrace', 'id': 1491597138, 'ino': 1099511629503, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/308.out'}, {'damage_type': 'backtrace', 'id': 1493453197, 'ino': 1099511629491, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/302.out'}, {'damage_type': 'backtrace', 'id': 1508193125, 'ino': 1099511629411, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/022.out'}, {'damage_type': 'backtrace', 'id': 1550028308, 'ino': 1099511629660, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/072.out'}, {'damage_type': 'backtrace', 'id': 1558571294, 'ino': 1099511628964, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/128.out'}, {'damage_type': 'backtrace', 'id': 1588546831, 'ino': 1099511629090, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/191.out'}, {'damage_type': 'backtrace', 'id': 1619408086, 'ino': 1099511628730, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/011.out'}, {'damage_type': 'backtrace', 'id': 1634749315, 'ino': 1099511629018, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/155.out'}, {'damage_type': 'backtrace', 'id': 1636897717, 'ino': 1099511629541, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/015.out'}, {'damage_type': 'backtrace', 'id': 1637742148, 'ino': 1099511629393, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/013.out'}, {'damage_type': 'backtrace', 'id': 1638512286, 'ino': 1099511629415, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/024.out'}, {'damage_type': 'backtrace', 'id': 1645977210, 'ino': 1099511628848, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/070.out'}, {'damage_type': 'backtrace', 'id': 1652846782, 'ino': 1099511628774, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/033.out'}, {'damage_type': 'backtrace', 'id': 1658771688, 'ino': 1099511628862, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/077.out'}, {'damage_type': 'backtrace', 'id': 1672429416, 'ino': 1099511628888, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/090.out'}, {'damage_type': 'backtrace', 'id': 1678904195, 'ino': 1099511629638, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/061.out'}, {'damage_type': 'backtrace', 'id': 1692989564, 'ino': 1099511628954, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/123.out'}, {'damage_type': 'backtrace', 'id': 1716183323, 'ino': 1099511629620, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/052.out'}, {'damage_type': 'backtrace', 'id': 1719651905, 'ino': 1099511629417, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/025.out'}, {'damage_type': 'backtrace', 'id': 1748648819, 'ino': 1099511629447, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/040.out'}, {'damage_type': 'backtrace', 'id': 1772138610, 'ino': 1099511629642, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/063.out'}, {'damage_type': 'backtrace', 'id': 1772423620, 'ino': 1099511628804, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/048.out'}, {'damage_type': 'backtrace', 'id': 1776993630, 'ino': 1099511628712, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/002.out'}, {'damage_type': 'backtrace', 'id': 1783561178, 'ino': 1099511629563, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/026.out'}, {'damage_type': 'backtrace', 'id': 1796643947, 'ino': 1099511628880, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/086.out'}, {'damage_type': 'backtrace', 'id': 1812379488, 'ino': 1099511628908, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/100.out'}, {'damage_type': 'backtrace', 'id': 1850619126, 'ino': 1099511628912, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/102.out'}, {'damage_type': 'backtrace', 'id': 1857865683, 'ino': 1099511628952, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/122.out'}, {'damage_type': 'backtrace', 'id': 1884629309, 'ino': 1099511628820, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/056.out'}, {'damage_type': 'backtrace', 'id': 1895900760, 'ino': 1099511628874, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/083.out'}, {'damage_type': 'backtrace', 'id': 1899980742, 'ino': 1099511629379, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/006.out'}, {'damage_type': 'backtrace', 'id': 1921810541, 'ino': 1099511629435, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/034.out'}, {'damage_type': 'backtrace', 'id': 1933134326, 'ino': 1099511629401, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/017.out'}, {'damage_type': 'backtrace', 'id': 1939247438, 'ino': 1099511628826, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/059.out'}, {'damage_type': 'backtrace', 'id': 1947445025, 'ino': 1099511629640, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/062.out'}, {'damage_type': 'backtrace', 'id': 1978139716, 'ino': 1099511628762, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/027.out'}, {'damage_type': 'backtrace', 'id': 1988471507, 'ino': 1099511628822, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/057.out'}, {'damage_type': 'backtrace', 'id': 1996030224, 'ino': 1099511629630, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/057.out'}, {'damage_type': 'backtrace', 'id': 2012961059, 'ino': 1099511628740, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/016.out'}, {'damage_type': 'backtrace', 'id': 2026535283, 'ino': 1099511628884, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/088.out'}, {'damage_type': 'backtrace', 'id': 2052141854, 'ino': 1099511629557, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/023.out'}, {'damage_type': 'backtrace', 'id': 2058244142, 'ino': 1099511629423, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/028.out'}, {'damage_type': 'backtrace', 'id': 2075428835, 'ino': 1099511629585, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/036.out'}, {'damage_type': 'backtrace', 'id': 2090493886, 'ino': 1099511629076, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/184.out'}, {'damage_type': 'backtrace', 'id': 2094429648, 'ino': 1099511628924, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/108.out'}, {'damage_type': 'backtrace', 'id': 2097455575, 'ino': 1099511629662, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/073.out'}, {'damage_type': 'backtrace', 'id': 2099487225, 'ino': 1099511628866, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/079.out'}, {'damage_type': 'backtrace', 'id': 2101061275, 'ino': 1099511628986, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/139.out'}, {'damage_type': 'backtrace', 'id': 2106217442, 'ino': 1099511629387, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/010.out'}, {'damage_type': 'backtrace', 'id': 2115629005, 'ino': 1099511629609, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/048.out'}, {'damage_type': 'backtrace', 'id': 2119036718, 'ino': 1099511628962, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/127.out'}, {'damage_type': 'backtrace', 'id': 2122325282, 'ino': 1099511628750, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/021.out'}, {'damage_type': 'backtrace', 'id': 2152139856, 'ino': 1099511629074, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/183.out'}, {'damage_type': 'backtrace', 'id': 2152200962, 'ino': 1099511628844, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/068.out'}, {'damage_type': 'backtrace', 'id': 2153664556, 'ino': 1099511628886, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/089.out'}, {'damage_type': 'backtrace', 'id': 2171990935, 'ino': 1099511629092, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/192.out'}, {'damage_type': 'backtrace', 'id': 2172829491, 'ino': 1099511628708, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs'}, {'damage_type': 'backtrace', 'id': 2173959597, 'ino': 1099511629080, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/186.out'}, {'damage_type': 'backtrace', 'id': 2194637319, 'ino': 1099511629501, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/307.out'}, {'damage_type': 'backtrace', 'id': 2209431598, 'ino': 1099511629499, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/306.out'}, {'damage_type': 'backtrace', 'id': 2227960050, 'ino': 1099511628816, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/054.out'}, {'damage_type': 'backtrace', 'id': 2261653342, 'ino': 1099511629368, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/001.out.nodelalloc'}, {'damage_type': 'backtrace', 'id': 2284603045, 'ino': 1099511628900, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/096.out'}, {'damage_type': 'backtrace', 'id': 2296912231, 'ino': 1099511628722, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/007.out'}, {'damage_type': 'backtrace', 'id': 2309980982, 'ino': 1099511629108, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/200.out'}, {'damage_type': 'backtrace', 'id': 2313657654, 'ino': 1099511629413, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/023.out'}, {'damage_type': 'backtrace', 'id': 2318196145, 'ino': 1099511628982, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/137.out'}, {'damage_type': 'backtrace', 'id': 2319918976, 'ino': 1099511628878, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/085.out'}, {'damage_type': 'backtrace', 'id': 2325686070, 'ino': 1099511629004, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/148.out'}, {'damage_type': 'backtrace', 'id': 2330724208, 'ino': 1099511629567, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/028.out'}, {'damage_type': 'backtrace', 'id': 2343806663, 'ino': 1099511629419, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/026.out'}, {'damage_type': 'backtrace', 'id': 2355096927, 'ino': 1099511628898, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/095.out'}, {'damage_type': 'backtrace', 'id': 2360334191, 'ino': 1099511628922, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/107.out'}, {'damage_type': 'backtrace', 'id': 2391575786, 'ino': 1099511629048, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/170.out'}, {'damage_type': 'backtrace', 'id': 2403280645, 'ino': 1099511629658, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/071.out'}, {'damage_type': 'backtrace', 'id': 2425953607, 'ino': 1099511628790, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/041.out'}, {'damage_type': 'backtrace', 'id': 2426385437, 'ino': 1099511629614, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/050.out.default'}, {'damage_type': 'backtrace', 'id': 2438188003, 'ino': 1099511629527, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/008.out'}, {'damage_type': 'backtrace', 'id': 2461913685, 'ino': 1099511628758, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/025.out'}, {'damage_type': 'backtrace', 'id': 2469817610, 'ino': 1099511629489, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/301.out'}, {'damage_type': 'backtrace', 'id': 2483764671, 'ino': 1099511629531, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/010.out'}, {'damage_type': 'backtrace', 'id': 2521237913, 'ino': 1099511629537, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/013.out'}, {'damage_type': 'backtrace', 'id': 2546101797, 'ino': 1099511628994, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/143.out'}, {'damage_type': 'backtrace', 'id': 2567511520, 'ino': 1099511629601, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/044.out'}, {'damage_type': 'backtrace', 'id': 2603642814, 'ino': 1099511629581, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/035.cfg'}, {'damage_type': 'backtrace', 'id': 2604634485, 'ino': 1099511629559, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/024.out'}, {'damage_type': 'backtrace', 'id': 2614619407, 'ino': 1099511628780, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/036.out'}, {'damage_type': 'backtrace', 'id': 2631103196, 'ino': 1099511629046, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/169.out'}, {'damage_type': 'backtrace', 'id': 2639601367, 'ino': 1099511628784, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/038.out'}, {'damage_type': 'backtrace', 'id': 2650560097, 'ino': 1099511629451, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/042.out'}, {'damage_type': 'backtrace', 'id': 2666219652, 'ino': 1099511629511, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic'}, {'damage_type': 'backtrace', 'id': 2666399661, 'ino': 1099511629549, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/019.out'}, {'damage_type': 'backtrace', 'id': 2685322802, 'ino': 1099511629648, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/066.out'}, {'damage_type': 'backtrace', 'id': 2694509877, 'ino': 1099511629068, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/180.out'}, {'damage_type': 'backtrace', 'id': 2703900723, 'ino': 1099511628786, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/039.out'}, {'damage_type': 'backtrace', 'id': 2783856577, 'ino': 1099511628744, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/018.out'}, {'damage_type': 'backtrace', 'id': 2789497070, 'ino': 1099511629096, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/194.out'}, {'damage_type': 'backtrace', 'id': 2802126831, 'ino': 1099511629072, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/182.out'}, {'damage_type': 'backtrace', 'id': 2804384051, 'ino': 1099511629646, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/065.out'}, {'damage_type': 'backtrace', 'id': 2807731837, 'ino': 1099511629441, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/037.out'}, {'damage_type': 'backtrace', 'id': 2809648593, 'ino': 1099511628772, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/032.out'}, {'damage_type': 'backtrace', 'id': 2816739951, 'ino': 1099511629624, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/054.out'}, {'damage_type': 'backtrace', 'id': 2821857918, 'ino': 1099511629110, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/201.out'}, {'damage_type': 'backtrace', 'id': 2839341456, 'ino': 1099511628814, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/053.out'}, {'damage_type': 'backtrace', 'id': 2856732774, 'ino': 1099511629066, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/179.out'}, {'damage_type': 'backtrace', 'id': 2859013629, 'ino': 1099511629525, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/007.out'}, {'damage_type': 'backtrace', 'id': 2885577075, 'ino': 1099511628958, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/125.out'}, {'damage_type': 'backtrace', 'id': 2891092575, 'ino': 1099511628838, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/065.out'}, {'damage_type': 'backtrace', 'id': 2901881156, 'ino': 1099511628830, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/061.out'}, {'damage_type': 'backtrace', 'id': 2903983007, 'ino': 1099511629020, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/156.out'}, {'damage_type': 'backtrace', 'id': 2909604243, 'ino': 1099511629373, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/003.out'}, {'damage_type': 'backtrace', 'id': 2919803636, 'ino': 1099511629399, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/016.out'}, {'damage_type': 'backtrace', 'id': 2924120603, 'ino': 1099511628748, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/020.out'}, {'damage_type': 'backtrace', 'id': 2925289664, 'ino': 1099511628796, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/044.out'}, {'damage_type': 'backtrace', 'id': 2936396766, 'ino': 1099511628894, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/093.out'}, {'damage_type': 'backtrace', 'id': 2937772258, 'ino': 1099511629104, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/198.out'}, {'damage_type': 'backtrace', 'id': 2948127614, 'ino': 1099511628746, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/019.out'}, {'damage_type': 'backtrace', 'id': 2955609826, 'ino': 1099511629579, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/034.out'}, {'damage_type': 'backtrace', 'id': 2992898823, 'ino': 1099511628980, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/136.out'}, {'damage_type': 'backtrace', 'id': 2997894209, 'ino': 1099511629058, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/175.out'}, {'damage_type': 'backtrace', 'id': 2998643511, 'ino': 1099511629589, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/038.out'}, {'damage_type': 'backtrace', 'id': 3038393378, 'ino': 1099511629389, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/011.out'}, {'damage_type': 'backtrace', 'id': 3041600235, 'ino': 1099511629433, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/033.out'}, {'damage_type': 'backtrace', 'id': 3042234196, 'ino': 1099511628906, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/099.out'}, {'damage_type': 'backtrace', 'id': 3046032168, 'ino': 1099511628998, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/145.out'}, {'damage_type': 'backtrace', 'id': 3067176649, 'ino': 1099511628970, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/131.out'}, {'damage_type': 'backtrace', 'id': 3068675093, 'ino': 1099511629385, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/009.out'}, {'damage_type': 'backtrace', 'id': 3077720606, 'ino': 1099511628996, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/144.out'}, {'damage_type': 'backtrace', 'id': 3103475690, 'ino': 1099511629026, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/159.out'}, {'damage_type': 'backtrace', 'id': 3112024784, 'ino': 1099511628934, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/113.out'}, {'damage_type': 'backtrace', 'id': 3155470778, 'ino': 1099511629437, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/035.out'}, {'damage_type': 'backtrace', 'id': 3157273716, 'ino': 1099511628710, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/001.out'}, {'damage_type': 'backtrace', 'id': 3172985907, 'ino': 1099511628724, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/008.out'}, {'damage_type': 'backtrace', 'id': 3193623495, 'ino': 1099511629597, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/042.out'}, {'damage_type': 'backtrace', 'id': 3239482274, 'ino': 1099511628764, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/028.out'}, {'damage_type': 'backtrace', 'id': 3265393820, 'ino': 1099511629553, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/021.out'}, {'damage_type': 'backtrace', 'id': 3267142831, 'ino': 1099511629656, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/070.out'}, {'damage_type': 'backtrace', 'id': 3267304707, 'ino': 1099511629360, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/cifs'}, {'damage_type': 'backtrace', 'id': 3277748402, 'ino': 1099511629425, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/029.out'}, {'damage_type': 'backtrace', 'id': 3278011930, 'ino': 1099511629605, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/046.out'}, {'damage_type': 'backtrace', 'id': 3280473302, 'ino': 1099511629375, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/004.out'}, {'damage_type': 'backtrace', 'id': 3304684766, 'ino': 1099511628984, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/138.out'}, {'damage_type': 'backtrace', 'id': 3327120062, 'ino': 1099511629493, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/303.out'}, {'damage_type': 'backtrace', 'id': 3344079257, 'ino': 1099511629366, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/001.cfg'}, {'damage_type': 'backtrace', 'id': 3365715262, 'ino': 1099511629429, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/031.out'}, {'damage_type': 'backtrace', 'id': 3368495633, 'ino': 1099511629650, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/067.out'}, {'damage_type': 'backtrace', 'id': 3372436831, 'ino': 1099511629022, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/157.out'}, {'damage_type': 'backtrace', 'id': 3372963410, 'ino': 1099511629439, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/036.out'}, {'damage_type': 'backtrace', 'id': 3386111793, 'ino': 1099511628782, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/037.out'}, {'damage_type': 'backtrace', 'id': 3398267245, 'ino': 1099511629032, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/162.out'}, {'damage_type': 'backtrace', 'id': 3416093719, 'ino': 1099511629381, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/007.out'}, {'damage_type': 'backtrace', 'id': 3440574853, 'ino': 1099511629082, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/187.out'}, {'damage_type': 'backtrace', 'id': 3465337178, 'ino': 1099511628976, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/134.out'}, {'damage_type': 'backtrace', 'id': 3466785126, 'ino': 1099511629362, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/cifs/001.out'}, {'damage_type': 'backtrace', 'id': 3469853750, 'ino': 1099511628818, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/055.out'}, {'damage_type': 'backtrace', 'id': 3482546632, 'ino': 1099511628928, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/110.out'}, {'damage_type': 'backtrace', 'id': 3525208496, 'ino': 1099511628714, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/003.out'}, {'damage_type': 'backtrace', 'id': 3528560221, 'ino': 1099511629030, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/161.out'}, {'damage_type': 'backtrace', 'id': 3534446033, 'ino': 1099511628936, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/114.out'}, {'damage_type': 'backtrace', 'id': 3534719075, 'ino': 1099511628978, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/135.out'}, {'damage_type': 'backtrace', 'id': 3539642818, 'ino': 1099511629391, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/012.out'}, {'damage_type': 'backtrace', 'id': 3551727301, 'ino': 1099511628824, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/058.out'}, {'damage_type': 'backtrace', 'id': 3560385251, 'ino': 1099511628768, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/030.out'}, {'damage_type': 'backtrace', 'id': 3560858111, 'ino': 1099511629364, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4'}, {'damage_type': 'backtrace', 'id': 3562334303, 'ino': 1099511628914, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/103.out'}, {'damage_type': 'backtrace', 'id': 3567007406, 'ino': 1099511629100, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/196.out'}, {'damage_type': 'backtrace', 'id': 3578937546, 'ino': 1099511629515, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/002.out'}, {'damage_type': 'backtrace', 'id': 3605575045, 'ino': 1099511629616, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/050.out.xfsquota'}, {'damage_type': 'backtrace', 'id': 3611961388, 'ino': 1099511628926, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/109.out'}, {'damage_type': 'backtrace', 'id': 3624524824, 'ino': 1099511628832, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/062.out'}, {'damage_type': 'backtrace', 'id': 3638881860, 'ino': 1099511629654, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/069.out'}, {'damage_type': 'backtrace', 'id': 3642278035, 'ino': 1099511628956, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/124.out'}, {'damage_type': 'backtrace', 'id': 3660908708, 'ino': 1099511628892, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/092.out'}, {'damage_type': 'backtrace', 'id': 3670984507, 'ino': 1099511629084, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/188.out'}, {'damage_type': 'backtrace', 'id': 3674730059, 'ino': 1099511628868, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/080.out'}, {'damage_type': 'backtrace', 'id': 3679083051, 'ino': 1099511629028, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/160.out'}, {'damage_type': 'backtrace', 'id': 3680870297, 'ino': 1099511629377, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/005.out'}, {'damage_type': 'backtrace', 'id': 3690121004, 'ino': 1099511628944, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/118.out'}, {'damage_type': 'backtrace', 'id': 3708539435, 'ino': 1099511629036, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/164.out'}, {'damage_type': 'backtrace', 'id': 3716610679, 'ino': 1099511629002, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/147.out'}, {'damage_type': 'backtrace', 'id': 3735993147, 'ino': 1099511628930, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/111.out'}, {'damage_type': 'backtrace', 'id': 3737589287, 'ino': 1099511628718, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/005.out'}, {'damage_type': 'backtrace', 'id': 3747609878, 'ino': 1099511629517, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/003.out'}, {'damage_type': 'backtrace', 'id': 3761720115, 'ino': 1099511628856, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/074.out'}, {'damage_type': 'backtrace', 'id': 3777513020, 'ino': 1099511628992, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/142.out'}, {'damage_type': 'backtrace', 'id': 3777997468, 'ino': 1099511629495, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/304.out'}, {'damage_type': 'backtrace', 'id': 3792958047, 'ino': 1099511628882, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/087.out'}, {'damage_type': 'backtrace', 'id': 3832577580, 'ino': 1099511628948, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/120.out'}, {'damage_type': 'backtrace', 'id': 3845789394, 'ino': 1099511628960, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/126.out'}, {'damage_type': 'backtrace', 'id': 3856070076, 'ino': 1099511629603, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/045.out'}, {'damage_type': 'backtrace', 'id': 3868057311, 'ino': 1099511629449, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/041.out'}, {'damage_type': 'backtrace', 'id': 3873700857, 'ino': 1099511628810, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/051.out'}, {'damage_type': 'backtrace', 'id': 3882688893, 'ino': 1099511628800, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/046.out'}, {'damage_type': 'backtrace', 'id': 3885219739, 'ino': 1099511629060, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/176.out'}, {'damage_type': 'backtrace', 'id': 3896143639, 'ino': 1099511629070, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/181.out'}, {'damage_type': 'backtrace', 'id': 3901885622, 'ino': 1099511629587, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/037.out'}, {'damage_type': 'backtrace', 'id': 3915810659, 'ino': 1099511629519, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/004.out'}, {'damage_type': 'backtrace', 'id': 3919592202, 'ino': 1099511628836, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/064.out'}, {'damage_type': 'backtrace', 'id': 3926235333, 'ino': 1099511629367, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/001.out.default'}, {'damage_type': 'backtrace', 'id': 3940932027, 'ino': 1099511628872, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/082.out'}, {'damage_type': 'backtrace', 'id': 3942722507, 'ino': 1099511629427, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/030.out'}, {'damage_type': 'backtrace', 'id': 3973632754, 'ino': 1099511629397, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/ext4/015.out'}, {'damage_type': 'backtrace', 'id': 3975944065, 'ino': 1099511628972, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/132.out'}, {'damage_type': 'backtrace', 'id': 4006756849, 'ino': 1099511628776, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/034.out'}, {'damage_type': 'backtrace', 'id': 4009157213, 'ino': 1099511629086, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/189.out'}, {'damage_type': 'backtrace', 'id': 4020334252, 'ino': 1099511628734, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/013.out'}, {'damage_type': 'backtrace', 'id': 4043107295, 'ino': 1099511629044, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/168.out'}, {'damage_type': 'backtrace', 'id': 4067333953, 'ino': 1099511628840, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/066.out'}, {'damage_type': 'backtrace', 'id': 4069277858, 'ino': 1099511628974, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/133.out'}, {'damage_type': 'backtrace', 'id': 4078638835, 'ino': 1099511629000, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/146.out'}, {'damage_type': 'backtrace', 'id': 4100537424, 'ino': 1099511628794, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/043.out'}, {'damage_type': 'backtrace', 'id': 4155134551, 'ino': 1099511628988, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/140.out'}, {'damage_type': 'backtrace', 'id': 4172285384, 'ino': 1099511628876, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/084.out'}, {'damage_type': 'backtrace', 'id': 4183354616, 'ino': 1099511628808, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/050.out'}, {'damage_type': 'backtrace', 'id': 4203535408, 'ino': 1099511629539, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/014.out'}, {'damage_type': 'backtrace', 'id': 4217424393, 'ino': 1099511629565, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/027.out'}, {'damage_type': 'backtrace', 'id': 4275066116, 'ino': 1099511628854, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/btrfs/073.out'}, {'damage_type': 'backtrace', 'id': 4294778982, 'ino': 1099511629543, 'path': '/volumes/qa/sv_1/aa392c15-a144-41c4-854f-c3936c335a66/client.0/tmp/xfstests-dev/tests/generic/016.out'}] 2024-04-21T23:20:39.960 ERROR:tasks.fwd_scrub.fs.[cephfs]:exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph_430e09df97c8fc7dc2b2ae424f68ed11366c540f/qa/tasks/fwd_scrub.py", line 38, in _run self.do_scrub() File "/home/teuthworker/src/git.ceph.com_ceph_430e09df97c8fc7dc2b2ae424f68ed11366c540f/qa/tasks/fwd_scrub.py", line 55, in do_scrub self._scrub() File "/home/teuthworker/src/git.ceph.com_ceph_430e09df97c8fc7dc2b2ae424f68ed11366c540f/qa/tasks/fwd_scrub.py", line 79, in _scrub self._check_damage() File "/home/teuthworker/src/git.ceph.com_ceph_430e09df97c8fc7dc2b2ae424f68ed11366c540f/qa/tasks/fwd_scrub.py", line 90, in _check_damage raise RuntimeError(f"rank damage found: {types}") RuntimeError: rank damage found: {'backtrace'} 2024-04-21T23:20:40.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:40 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/2285967261' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:20:40.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:40 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/2285967261' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:20:40.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:40 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/2285967261' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:20:41.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:41 smithi040 ceph-mon[27984]: pgmap v71: 129 pgs: 129 active+clean; 95 MiB data, 785 MiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 402 KiB/s wr, 40 op/s 2024-04-21T23:20:41.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:41 smithi174 ceph-mon[31409]: pgmap v71: 129 pgs: 129 active+clean; 95 MiB data, 785 MiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 402 KiB/s wr, 40 op/s 2024-04-21T23:20:41.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:41 smithi111 ceph-mon[31085]: pgmap v71: 129 pgs: 129 active+clean; 95 MiB data, 785 MiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 402 KiB/s wr, 40 op/s 2024-04-21T23:20:43.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:43 smithi040 ceph-mon[27984]: pgmap v72: 129 pgs: 129 active+clean; 95 MiB data, 785 MiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 165 KiB/s wr, 21 op/s 2024-04-21T23:20:43.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:43 smithi174 ceph-mon[31409]: pgmap v72: 129 pgs: 129 active+clean; 95 MiB data, 785 MiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 165 KiB/s wr, 21 op/s 2024-04-21T23:20:43.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:43 smithi111 ceph-mon[31085]: pgmap v72: 129 pgs: 129 active+clean; 95 MiB data, 785 MiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 165 KiB/s wr, 21 op/s 2024-04-21T23:20:45.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:45 smithi040 ceph-mon[27984]: pgmap v73: 129 pgs: 129 active+clean; 95 MiB data, 785 MiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 165 KiB/s wr, 21 op/s 2024-04-21T23:20:45.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:45 smithi174 ceph-mon[31409]: pgmap v73: 129 pgs: 129 active+clean; 95 MiB data, 785 MiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 165 KiB/s wr, 21 op/s 2024-04-21T23:20:45.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:45 smithi111 ceph-mon[31085]: pgmap v73: 129 pgs: 129 active+clean; 95 MiB data, 785 MiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 165 KiB/s wr, 21 op/s 2024-04-21T23:20:47.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:47 smithi040 ceph-mon[27984]: pgmap v74: 129 pgs: 129 active+clean; 101 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 883 KiB/s wr, 106 op/s 2024-04-21T23:20:47.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:47 smithi174 ceph-mon[31409]: pgmap v74: 129 pgs: 129 active+clean; 101 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 883 KiB/s wr, 106 op/s 2024-04-21T23:20:47.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:47 smithi111 ceph-mon[31085]: pgmap v74: 129 pgs: 129 active+clean; 101 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 883 KiB/s wr, 106 op/s 2024-04-21T23:20:49.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:49 smithi040 ceph-mon[27984]: pgmap v75: 129 pgs: 129 active+clean; 101 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 727 KiB/s wr, 89 op/s 2024-04-21T23:20:49.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:49 smithi174 ceph-mon[31409]: pgmap v75: 129 pgs: 129 active+clean; 101 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 727 KiB/s wr, 89 op/s 2024-04-21T23:20:49.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:49 smithi111 ceph-mon[31085]: pgmap v75: 129 pgs: 129 active+clean; 101 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 727 KiB/s wr, 89 op/s 2024-04-21T23:20:51.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:51 smithi040 ceph-mon[27984]: pgmap v76: 129 pgs: 129 active+clean; 101 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 730 KiB/s wr, 94 op/s 2024-04-21T23:20:51.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:51 smithi174 ceph-mon[31409]: pgmap v76: 129 pgs: 129 active+clean; 101 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 730 KiB/s wr, 94 op/s 2024-04-21T23:20:51.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:51 smithi111 ceph-mon[31085]: pgmap v76: 129 pgs: 129 active+clean; 101 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 730 KiB/s wr, 94 op/s 2024-04-21T23:20:53.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:53 smithi040 ceph-mon[27984]: pgmap v77: 129 pgs: 129 active+clean; 101 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 721 KiB/s wr, 89 op/s 2024-04-21T23:20:53.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:53 smithi174 ceph-mon[31409]: pgmap v77: 129 pgs: 129 active+clean; 101 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 721 KiB/s wr, 89 op/s 2024-04-21T23:20:53.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:53 smithi111 ceph-mon[31085]: pgmap v77: 129 pgs: 129 active+clean; 101 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 721 KiB/s wr, 89 op/s 2024-04-21T23:20:55.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:55 smithi040 ceph-mon[27984]: pgmap v78: 129 pgs: 129 active+clean; 101 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 721 KiB/s wr, 89 op/s 2024-04-21T23:20:55.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:55 smithi174 ceph-mon[31409]: pgmap v78: 129 pgs: 129 active+clean; 101 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 721 KiB/s wr, 89 op/s 2024-04-21T23:20:55.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:55 smithi111 ceph-mon[31085]: pgmap v78: 129 pgs: 129 active+clean; 101 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 721 KiB/s wr, 89 op/s 2024-04-21T23:20:57.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:57 smithi040 ceph-mon[27984]: pgmap v79: 129 pgs: 129 active+clean; 101 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 721 KiB/s wr, 92 op/s 2024-04-21T23:20:57.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:57 smithi174 ceph-mon[31409]: pgmap v79: 129 pgs: 129 active+clean; 101 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 721 KiB/s wr, 92 op/s 2024-04-21T23:20:57.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:57 smithi111 ceph-mon[31085]: pgmap v79: 129 pgs: 129 active+clean; 101 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 721 KiB/s wr, 92 op/s 2024-04-21T23:20:59.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:20:59 smithi040 ceph-mon[27984]: pgmap v80: 129 pgs: 129 active+clean; 101 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 3.5 KiB/s wr, 7 op/s 2024-04-21T23:20:59.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:20:59 smithi174 ceph-mon[31409]: pgmap v80: 129 pgs: 129 active+clean; 101 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 3.5 KiB/s wr, 7 op/s 2024-04-21T23:20:59.443 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:20:59 smithi111 ceph-mon[31085]: pgmap v80: 129 pgs: 129 active+clean; 101 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 3.5 KiB/s wr, 7 op/s 2024-04-21T23:21:01.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:21:01 smithi040 ceph-mon[27984]: pgmap v81: 129 pgs: 129 active+clean; 101 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 5.2 KiB/s wr, 11 op/s 2024-04-21T23:21:01.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:21:01 smithi174 ceph-mon[31409]: pgmap v81: 129 pgs: 129 active+clean; 101 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 5.2 KiB/s wr, 11 op/s 2024-04-21T23:21:01.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:21:01 smithi111 ceph-mon[31085]: pgmap v81: 129 pgs: 129 active+clean; 101 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 5.2 KiB/s wr, 11 op/s 2024-04-21T23:21:03.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:21:03 smithi040 ceph-mon[27984]: pgmap v82: 129 pgs: 129 active+clean; 101 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.3 KiB/s wr, 6 op/s 2024-04-21T23:21:03.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:21:03 smithi174 ceph-mon[31409]: pgmap v82: 129 pgs: 129 active+clean; 101 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.3 KiB/s wr, 6 op/s 2024-04-21T23:21:03.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:21:03 smithi111 ceph-mon[31085]: pgmap v82: 129 pgs: 129 active+clean; 101 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.3 KiB/s wr, 6 op/s 2024-04-21T23:21:05.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:21:05 smithi040 ceph-mon[27984]: pgmap v83: 129 pgs: 129 active+clean; 101 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.3 KiB/s wr, 6 op/s 2024-04-21T23:21:05.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:21:05 smithi174 ceph-mon[31409]: pgmap v83: 129 pgs: 129 active+clean; 101 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.3 KiB/s wr, 6 op/s 2024-04-21T23:21:05.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:21:05 smithi111 ceph-mon[31085]: pgmap v83: 129 pgs: 129 active+clean; 101 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.3 KiB/s wr, 6 op/s 2024-04-21T23:21:07.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:21:07 smithi040 ceph-mon[27984]: pgmap v84: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 25 KiB/s wr, 24 op/s 2024-04-21T23:21:07.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:21:07 smithi174 ceph-mon[31409]: pgmap v84: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 25 KiB/s wr, 24 op/s 2024-04-21T23:21:07.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:21:07 smithi111 ceph-mon[31085]: pgmap v84: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 25 KiB/s wr, 24 op/s 2024-04-21T23:21:09.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:21:09 smithi040 ceph-mon[27984]: pgmap v85: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 21 op/s 2024-04-21T23:21:09.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:21:09 smithi174 ceph-mon[31409]: pgmap v85: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 21 op/s 2024-04-21T23:21:09.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:21:09 smithi111 ceph-mon[31085]: pgmap v85: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 21 op/s 2024-04-21T23:21:11.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:21:11 smithi040 ceph-mon[27984]: pgmap v86: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 26 KiB/s wr, 24 op/s 2024-04-21T23:21:11.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:21:11 smithi174 ceph-mon[31409]: pgmap v86: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 26 KiB/s wr, 24 op/s 2024-04-21T23:21:11.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:21:11 smithi111 ceph-mon[31085]: pgmap v86: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 26 KiB/s wr, 24 op/s 2024-04-21T23:21:13.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:21:13 smithi040 ceph-mon[27984]: pgmap v87: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 20 op/s 2024-04-21T23:21:13.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:21:13 smithi174 ceph-mon[31409]: pgmap v87: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 20 op/s 2024-04-21T23:21:13.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:21:13 smithi111 ceph-mon[31085]: pgmap v87: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 20 op/s 2024-04-21T23:21:14.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:21:14 smithi040 ceph-mon[27984]: pgmap v88: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 20 op/s 2024-04-21T23:21:14.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:21:14 smithi174 ceph-mon[31409]: pgmap v88: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 20 op/s 2024-04-21T23:21:14.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:21:14 smithi111 ceph-mon[31085]: pgmap v88: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 20 op/s 2024-04-21T23:21:16.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:21:16 smithi040 ceph-mon[27984]: pgmap v89: 129 pgs: 129 active+clean; 101 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 24 KiB/s wr, 23 op/s 2024-04-21T23:21:16.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:21:16 smithi174 ceph-mon[31409]: pgmap v89: 129 pgs: 129 active+clean; 101 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 24 KiB/s wr, 23 op/s 2024-04-21T23:21:16.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:21:16 smithi111 ceph-mon[31085]: pgmap v89: 129 pgs: 129 active+clean; 101 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 24 KiB/s wr, 23 op/s 2024-04-21T23:21:18.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:21:18 smithi040 ceph-mon[27984]: pgmap v90: 129 pgs: 129 active+clean; 101 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 2.2 KiB/s wr, 6 op/s 2024-04-21T23:21:18.965 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:21:18 smithi111 ceph-mon[31085]: pgmap v90: 129 pgs: 129 active+clean; 101 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 2.2 KiB/s wr, 6 op/s 2024-04-21T23:21:18.965 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:21:18 smithi174 ceph-mon[31409]: pgmap v90: 129 pgs: 129 active+clean; 101 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 2.2 KiB/s wr, 6 op/s 2024-04-21T23:21:20.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:21:20 smithi040 ceph-mon[27984]: pgmap v91: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 3.3 KiB/s wr, 8 op/s 2024-04-21T23:21:20.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:21:20 smithi174 ceph-mon[31409]: pgmap v91: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 3.3 KiB/s wr, 8 op/s 2024-04-21T23:21:20.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:21:20 smithi111 ceph-mon[31085]: pgmap v91: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 3.3 KiB/s wr, 8 op/s 2024-04-21T23:21:22.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:21:22 smithi040 ceph-mon[27984]: pgmap v92: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.6 KiB/s wr, 5 op/s 2024-04-21T23:21:22.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:21:22 smithi174 ceph-mon[31409]: pgmap v92: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.6 KiB/s wr, 5 op/s 2024-04-21T23:21:22.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:21:22 smithi111 ceph-mon[31085]: pgmap v92: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.6 KiB/s wr, 5 op/s 2024-04-21T23:21:24.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:21:24 smithi040 ceph-mon[27984]: pgmap v93: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.6 KiB/s wr, 5 op/s 2024-04-21T23:21:24.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:21:24 smithi174 ceph-mon[31409]: pgmap v93: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.6 KiB/s wr, 5 op/s 2024-04-21T23:21:24.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:21:24 smithi111 ceph-mon[31085]: pgmap v93: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.6 KiB/s wr, 5 op/s 2024-04-21T23:21:26.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:21:26 smithi040 ceph-mon[27984]: pgmap v94: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2024-04-21T23:21:26.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:21:26 smithi174 ceph-mon[31409]: pgmap v94: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2024-04-21T23:21:26.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:21:26 smithi111 ceph-mon[31085]: pgmap v94: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2024-04-21T23:21:28.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:21:28 smithi040 ceph-mon[27984]: pgmap v95: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.3 KiB/s wr, 5 op/s 2024-04-21T23:21:28.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:21:28 smithi174 ceph-mon[31409]: pgmap v95: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.3 KiB/s wr, 5 op/s 2024-04-21T23:21:28.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:21:28 smithi111 ceph-mon[31085]: pgmap v95: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.3 KiB/s wr, 5 op/s 2024-04-21T23:21:30.931 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:21:30 smithi040 ceph-mon[27984]: pgmap v96: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.3 KiB/s wr, 7 op/s 2024-04-21T23:21:30.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:21:30 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:21:30.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:21:30 smithi174 ceph-mon[31409]: pgmap v96: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.3 KiB/s wr, 7 op/s 2024-04-21T23:21:30.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:21:30 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:21:30.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:21:30 smithi111 ceph-mon[31085]: pgmap v96: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.3 KiB/s wr, 7 op/s 2024-04-21T23:21:30.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:21:30 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:21:31.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:21:31 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:21:31.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:21:31 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:21:31.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:21:31 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:21:31.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:21:31 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:21:31.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:21:31 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:21:31.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:21:31 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:21:31.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:21:31 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:21:31.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:21:31 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:21:31.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:21:31 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:21:32.931 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:21:32 smithi040 ceph-mon[27984]: pgmap v97: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-21T23:21:32.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:21:32 smithi174 ceph-mon[31409]: pgmap v97: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-21T23:21:32.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:21:32 smithi111 ceph-mon[31085]: pgmap v97: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-21T23:21:34.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:21:34 smithi040 ceph-mon[27984]: pgmap v98: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-21T23:21:34.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:21:34 smithi174 ceph-mon[31409]: pgmap v98: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-21T23:21:34.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:21:34 smithi111 ceph-mon[31085]: pgmap v98: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-21T23:21:36.931 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:21:36 smithi040 ceph-mon[27984]: pgmap v99: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 170 B/s wr, 7 op/s 2024-04-21T23:21:36.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:21:36 smithi174 ceph-mon[31409]: pgmap v99: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 170 B/s wr, 7 op/s 2024-04-21T23:21:36.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:21:36 smithi111 ceph-mon[31085]: pgmap v99: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 170 B/s wr, 7 op/s 2024-04-21T23:21:38.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:21:38 smithi040 ceph-mon[27984]: pgmap v100: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:21:38.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:21:38 smithi174 ceph-mon[31409]: pgmap v100: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:21:38.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:21:38 smithi111 ceph-mon[31085]: pgmap v100: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:21:40.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:21:40 smithi040 ceph-mon[27984]: pgmap v101: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:21:40.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:21:40 smithi174 ceph-mon[31409]: pgmap v101: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:21:40.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:21:40 smithi111 ceph-mon[31085]: pgmap v101: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:21:42.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:21:42 smithi040 ceph-mon[27984]: pgmap v102: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:21:42.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:21:42 smithi174 ceph-mon[31409]: pgmap v102: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:21:42.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:21:42 smithi111 ceph-mon[31085]: pgmap v102: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:21:44.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:21:44 smithi040 ceph-mon[27984]: pgmap v103: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:21:44.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:21:44 smithi174 ceph-mon[31409]: pgmap v103: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:21:44.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:21:44 smithi111 ceph-mon[31085]: pgmap v103: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:21:46.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:21:46 smithi040 ceph-mon[27984]: pgmap v104: 129 pgs: 129 active+clean; 101 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.8 KiB/s wr, 8 op/s 2024-04-21T23:21:46.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:21:46 smithi174 ceph-mon[31409]: pgmap v104: 129 pgs: 129 active+clean; 101 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.8 KiB/s wr, 8 op/s 2024-04-21T23:21:46.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:21:46 smithi111 ceph-mon[31085]: pgmap v104: 129 pgs: 129 active+clean; 101 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.8 KiB/s wr, 8 op/s 2024-04-21T23:21:48.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:21:48 smithi040 ceph-mon[27984]: pgmap v105: 129 pgs: 129 active+clean; 101 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.8 KiB/s wr, 5 op/s 2024-04-21T23:21:48.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:21:48 smithi174 ceph-mon[31409]: pgmap v105: 129 pgs: 129 active+clean; 101 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.8 KiB/s wr, 5 op/s 2024-04-21T23:21:48.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:21:48 smithi111 ceph-mon[31085]: pgmap v105: 129 pgs: 129 active+clean; 101 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.8 KiB/s wr, 5 op/s 2024-04-21T23:21:50.931 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:21:50 smithi040 ceph-mon[27984]: pgmap v106: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 4.4 KiB/s wr, 8 op/s 2024-04-21T23:21:50.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:21:50 smithi174 ceph-mon[31409]: pgmap v106: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 4.4 KiB/s wr, 8 op/s 2024-04-21T23:21:50.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:21:50 smithi111 ceph-mon[31085]: pgmap v106: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 4.4 KiB/s wr, 8 op/s 2024-04-21T23:21:52.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:21:52 smithi040 ceph-mon[27984]: pgmap v107: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 4.4 KiB/s wr, 6 op/s 2024-04-21T23:21:52.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:21:52 smithi174 ceph-mon[31409]: pgmap v107: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 4.4 KiB/s wr, 6 op/s 2024-04-21T23:21:52.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:21:52 smithi111 ceph-mon[31085]: pgmap v107: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 4.4 KiB/s wr, 6 op/s 2024-04-21T23:21:54.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:21:54 smithi040 ceph-mon[27984]: pgmap v108: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 4.4 KiB/s wr, 6 op/s 2024-04-21T23:21:54.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:21:54 smithi174 ceph-mon[31409]: pgmap v108: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 4.4 KiB/s wr, 6 op/s 2024-04-21T23:21:54.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:21:54 smithi111 ceph-mon[31085]: pgmap v108: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 4.4 KiB/s wr, 6 op/s 2024-04-21T23:21:56.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:21:56 smithi040 ceph-mon[27984]: pgmap v109: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 4.7 KiB/s wr, 9 op/s 2024-04-21T23:21:56.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:21:56 smithi174 ceph-mon[31409]: pgmap v109: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 4.7 KiB/s wr, 9 op/s 2024-04-21T23:21:56.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:21:56 smithi111 ceph-mon[31085]: pgmap v109: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 4.7 KiB/s wr, 9 op/s 2024-04-21T23:21:58.931 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:21:58 smithi040 ceph-mon[27984]: pgmap v110: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 2.9 KiB/s wr, 5 op/s 2024-04-21T23:21:58.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:21:58 smithi174 ceph-mon[31409]: pgmap v110: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 2.9 KiB/s wr, 5 op/s 2024-04-21T23:21:58.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:21:58 smithi111 ceph-mon[31085]: pgmap v110: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 2.9 KiB/s wr, 5 op/s 2024-04-21T23:22:00.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:22:00 smithi040 ceph-mon[27984]: pgmap v111: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 4.1 KiB/s wr, 8 op/s 2024-04-21T23:22:00.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:22:00 smithi174 ceph-mon[31409]: pgmap v111: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 4.1 KiB/s wr, 8 op/s 2024-04-21T23:22:00.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:22:00 smithi111 ceph-mon[31085]: pgmap v111: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 4.1 KiB/s wr, 8 op/s 2024-04-21T23:22:02.931 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:22:02 smithi040 ceph-mon[27984]: pgmap v112: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-04-21T23:22:02.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:22:02 smithi174 ceph-mon[31409]: pgmap v112: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-04-21T23:22:02.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:22:02 smithi111 ceph-mon[31085]: pgmap v112: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-04-21T23:22:04.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:22:04 smithi040 ceph-mon[27984]: pgmap v113: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-04-21T23:22:04.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:22:04 smithi174 ceph-mon[31409]: pgmap v113: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-04-21T23:22:04.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:22:04 smithi111 ceph-mon[31085]: pgmap v113: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-04-21T23:22:06.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:22:06 smithi040 ceph-mon[27984]: pgmap v114: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 23 op/s 2024-04-21T23:22:06.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:22:06 smithi040 ceph-mon[27984]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-21T23:22:06.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:22:06 smithi174 ceph-mon[31409]: pgmap v114: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 23 op/s 2024-04-21T23:22:06.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:22:06 smithi174 ceph-mon[31409]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-21T23:22:06.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:22:06 smithi111 ceph-mon[31085]: pgmap v114: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 23 op/s 2024-04-21T23:22:06.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:22:06 smithi111 ceph-mon[31085]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-21T23:22:08.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:22:08 smithi040 ceph-mon[27984]: pgmap v115: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-04-21T23:22:08.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:22:08 smithi174 ceph-mon[31409]: pgmap v115: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-04-21T23:22:08.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:22:08 smithi111 ceph-mon[31085]: pgmap v115: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-04-21T23:22:10.931 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:22:10 smithi040 ceph-mon[27984]: pgmap v116: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 22 op/s 2024-04-21T23:22:10.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:22:10 smithi174 ceph-mon[31409]: pgmap v116: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 22 op/s 2024-04-21T23:22:10.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:22:10 smithi111 ceph-mon[31085]: pgmap v116: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 22 op/s 2024-04-21T23:22:12.931 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:22:12 smithi040 ceph-mon[27984]: pgmap v117: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-21T23:22:12.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:22:12 smithi174 ceph-mon[31409]: pgmap v117: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-21T23:22:12.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:22:12 smithi111 ceph-mon[31085]: pgmap v117: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-21T23:22:14.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:22:14 smithi040 ceph-mon[27984]: pgmap v118: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-21T23:22:14.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:22:14 smithi174 ceph-mon[31409]: pgmap v118: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-21T23:22:14.943 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:22:14 smithi111 ceph-mon[31085]: pgmap v118: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-21T23:22:16.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:22:16 smithi040 ceph-mon[27984]: pgmap v119: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 22 op/s 2024-04-21T23:22:16.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:22:16 smithi174 ceph-mon[31409]: pgmap v119: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 22 op/s 2024-04-21T23:22:16.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:22:16 smithi111 ceph-mon[31085]: pgmap v119: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 22 op/s 2024-04-21T23:22:18.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:22:18 smithi040 ceph-mon[27984]: pgmap v120: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-21T23:22:18.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:22:18 smithi174 ceph-mon[31409]: pgmap v120: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-21T23:22:18.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:22:18 smithi111 ceph-mon[31085]: pgmap v120: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-21T23:22:20.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:22:20 smithi040 ceph-mon[27984]: pgmap v121: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-21T23:22:20.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:22:20 smithi040 ceph-mon[27984]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-21T23:22:20.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:22:20 smithi174 ceph-mon[31409]: pgmap v121: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-21T23:22:20.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:22:20 smithi174 ceph-mon[31409]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-21T23:22:20.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:22:20 smithi111 ceph-mon[31085]: pgmap v121: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-21T23:22:20.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:22:20 smithi111 ceph-mon[31085]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-21T23:22:22.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:22:22 smithi040 ceph-mon[27984]: pgmap v122: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-21T23:22:22.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:22:22 smithi174 ceph-mon[31409]: pgmap v122: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-21T23:22:22.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:22:22 smithi111 ceph-mon[31085]: pgmap v122: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-21T23:22:24.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:22:24 smithi040 ceph-mon[27984]: pgmap v123: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-21T23:22:24.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:22:24 smithi174 ceph-mon[31409]: pgmap v123: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-21T23:22:24.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:22:24 smithi111 ceph-mon[31085]: pgmap v123: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-21T23:22:26.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:22:26 smithi040 ceph-mon[27984]: pgmap v124: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2024-04-21T23:22:26.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:22:26 smithi174 ceph-mon[31409]: pgmap v124: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2024-04-21T23:22:26.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:22:26 smithi111 ceph-mon[31085]: pgmap v124: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2024-04-21T23:22:28.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:22:28 smithi040 ceph-mon[27984]: pgmap v125: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:22:28.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:22:28 smithi174 ceph-mon[31409]: pgmap v125: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:22:28.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:22:28 smithi111 ceph-mon[31085]: pgmap v125: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:22:30.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:22:30 smithi040 ceph-mon[27984]: pgmap v126: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:22:30.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:22:30 smithi174 ceph-mon[31409]: pgmap v126: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:22:30.943 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:22:30 smithi111 ceph-mon[31085]: pgmap v126: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:22:31.931 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:22:31 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:22:31.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:22:31 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:22:31.943 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:22:31 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:22:32.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:22:32 smithi040 ceph-mon[27984]: pgmap v127: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:22:32.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:22:32 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:22:32.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:22:32 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:22:32.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:22:32 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:22:32.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:22:32 smithi174 ceph-mon[31409]: pgmap v127: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:22:32.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:22:32 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:22:32.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:22:32 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:22:32.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:22:32 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:22:32.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:22:32 smithi111 ceph-mon[31085]: pgmap v127: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:22:32.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:22:32 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:22:32.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:22:32 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:22:32.945 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:22:32 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:22:34.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:22:34 smithi040 ceph-mon[27984]: pgmap v128: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:22:34.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:22:34 smithi174 ceph-mon[31409]: pgmap v128: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:22:34.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:22:34 smithi111 ceph-mon[31085]: pgmap v128: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:22:36.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:22:36 smithi040 ceph-mon[27984]: pgmap v129: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:22:36.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:22:36 smithi174 ceph-mon[31409]: pgmap v129: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:22:36.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:22:36 smithi111 ceph-mon[31085]: pgmap v129: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:22:38.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:22:38 smithi040 ceph-mon[27984]: pgmap v130: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:22:38.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:22:38 smithi174 ceph-mon[31409]: pgmap v130: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:22:38.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:22:38 smithi111 ceph-mon[31085]: pgmap v130: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:22:40.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:22:40 smithi040 ceph-mon[27984]: pgmap v131: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:22:40.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:22:40 smithi174 ceph-mon[31409]: pgmap v131: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:22:40.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:22:40 smithi111 ceph-mon[31085]: pgmap v131: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:22:42.934 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:22:42 smithi040 ceph-mon[27984]: pgmap v132: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:22:42.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:22:42 smithi174 ceph-mon[31409]: pgmap v132: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:22:42.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:22:42 smithi111 ceph-mon[31085]: pgmap v132: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:22:44.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:22:44 smithi040 ceph-mon[27984]: pgmap v133: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:22:44.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:22:44 smithi174 ceph-mon[31409]: pgmap v133: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:22:44.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:22:44 smithi111 ceph-mon[31085]: pgmap v133: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:22:45.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:22:45 smithi040 ceph-mon[27984]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-21T23:22:45.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:22:45 smithi174 ceph-mon[31409]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-21T23:22:45.943 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:22:45 smithi111 ceph-mon[31085]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-21T23:22:46.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:22:46 smithi040 ceph-mon[27984]: pgmap v134: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:22:46.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:22:46 smithi111 ceph-mon[31085]: pgmap v134: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:22:46.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:22:46 smithi174 ceph-mon[31409]: pgmap v134: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:22:48.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:22:48 smithi040 ceph-mon[27984]: pgmap v135: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:22:48.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:22:48 smithi174 ceph-mon[31409]: pgmap v135: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:22:48.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:22:48 smithi111 ceph-mon[31085]: pgmap v135: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:22:50.931 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:22:50 smithi040 ceph-mon[27984]: pgmap v136: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:22:50.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:22:50 smithi174 ceph-mon[31409]: pgmap v136: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:22:50.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:22:50 smithi111 ceph-mon[31085]: pgmap v136: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:22:52.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:22:52 smithi040 ceph-mon[27984]: pgmap v137: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:22:52.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:22:52 smithi174 ceph-mon[31409]: pgmap v137: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:22:52.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:22:52 smithi111 ceph-mon[31085]: pgmap v137: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:22:54.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:22:54 smithi040 ceph-mon[27984]: pgmap v138: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:22:54.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:22:54 smithi174 ceph-mon[31409]: pgmap v138: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:22:54.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:22:54 smithi111 ceph-mon[31085]: pgmap v138: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:22:56.931 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:22:56 smithi040 ceph-mon[27984]: pgmap v139: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:22:56.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:22:56 smithi174 ceph-mon[31409]: pgmap v139: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:22:56.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:22:56 smithi111 ceph-mon[31085]: pgmap v139: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:22:58.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:22:58 smithi174 ceph-mon[31409]: pgmap v140: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:22:58.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:22:58 smithi111 ceph-mon[31085]: pgmap v140: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:22:59.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:22:58 smithi040 ceph-mon[27984]: pgmap v140: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:23:00.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:23:00 smithi174 ceph-mon[31409]: pgmap v141: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:23:00.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:23:00 smithi111 ceph-mon[31085]: pgmap v141: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:23:01.055 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:23:00 smithi040 ceph-mon[27984]: pgmap v141: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:23:02.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:23:02 smithi174 ceph-mon[31409]: pgmap v142: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:23:02.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:23:02 smithi111 ceph-mon[31085]: pgmap v142: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:23:03.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:23:02 smithi040 ceph-mon[27984]: pgmap v142: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:23:04.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:23:04 smithi174 ceph-mon[31409]: pgmap v143: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:23:04.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:23:04 smithi111 ceph-mon[31085]: pgmap v143: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:23:05.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:23:04 smithi040 ceph-mon[27984]: pgmap v143: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:23:07.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:23:06 smithi040 ceph-mon[27984]: pgmap v144: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-21T23:23:07.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:23:06 smithi040 ceph-mon[27984]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-21T23:23:07.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:23:06 smithi174 ceph-mon[31409]: pgmap v144: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-21T23:23:07.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:23:06 smithi174 ceph-mon[31409]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-21T23:23:07.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:23:06 smithi111 ceph-mon[31085]: pgmap v144: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-21T23:23:07.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:23:06 smithi111 ceph-mon[31085]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-21T23:23:08.996 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:23:08 smithi174 ceph-mon[31409]: pgmap v145: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-04-21T23:23:08.996 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:23:08 smithi111 ceph-mon[31085]: pgmap v145: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-04-21T23:23:09.181 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:23:08 smithi040 ceph-mon[27984]: pgmap v145: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-04-21T23:23:11.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:23:10 smithi040 ceph-mon[27984]: pgmap v146: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-21T23:23:11.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:23:10 smithi174 ceph-mon[31409]: pgmap v146: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-21T23:23:11.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:23:10 smithi111 ceph-mon[31085]: pgmap v146: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-21T23:23:13.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:23:12 smithi040 ceph-mon[27984]: pgmap v147: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-21T23:23:13.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:23:12 smithi174 ceph-mon[31409]: pgmap v147: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-21T23:23:13.193 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:23:12 smithi111 ceph-mon[31085]: pgmap v147: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-21T23:23:14.984 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:23:14 smithi040 ceph-mon[27984]: pgmap v148: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-21T23:23:15.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:23:14 smithi174 ceph-mon[31409]: pgmap v148: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-21T23:23:15.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:23:14 smithi111 ceph-mon[31085]: pgmap v148: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-21T23:23:17.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:23:16 smithi040 ceph-mon[27984]: pgmap v149: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-21T23:23:17.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:23:16 smithi174 ceph-mon[31409]: pgmap v149: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-21T23:23:17.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:23:16 smithi111 ceph-mon[31085]: pgmap v149: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-21T23:23:19.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:23:18 smithi040 ceph-mon[27984]: pgmap v150: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-21T23:23:19.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:23:18 smithi174 ceph-mon[31409]: pgmap v150: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-21T23:23:19.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:23:18 smithi111 ceph-mon[31085]: pgmap v150: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-21T23:23:21.181 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:23:20 smithi040 ceph-mon[27984]: pgmap v151: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 7 op/s 2024-04-21T23:23:21.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:23:20 smithi174 ceph-mon[31409]: pgmap v151: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 7 op/s 2024-04-21T23:23:21.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:23:20 smithi111 ceph-mon[31085]: pgmap v151: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 7 op/s 2024-04-21T23:23:23.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:23:22 smithi040 ceph-mon[27984]: pgmap v152: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-21T23:23:23.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:23:22 smithi174 ceph-mon[31409]: pgmap v152: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-21T23:23:23.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:23:22 smithi111 ceph-mon[31085]: pgmap v152: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-21T23:23:25.181 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:23:24 smithi040 ceph-mon[27984]: pgmap v153: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-21T23:23:25.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:23:24 smithi174 ceph-mon[31409]: pgmap v153: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-21T23:23:25.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:23:24 smithi111 ceph-mon[31085]: pgmap v153: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-21T23:23:27.181 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:23:26 smithi040 ceph-mon[27984]: pgmap v154: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2024-04-21T23:23:27.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:23:26 smithi174 ceph-mon[31409]: pgmap v154: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2024-04-21T23:23:27.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:23:26 smithi111 ceph-mon[31085]: pgmap v154: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2024-04-21T23:23:29.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:23:28 smithi040 ceph-mon[27984]: pgmap v155: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:23:29.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:23:28 smithi174 ceph-mon[31409]: pgmap v155: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:23:29.193 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:23:28 smithi111 ceph-mon[31085]: pgmap v155: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:23:31.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:23:30 smithi040 ceph-mon[27984]: pgmap v156: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:23:31.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:23:30 smithi174 ceph-mon[31409]: pgmap v156: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:23:31.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:23:30 smithi111 ceph-mon[31085]: pgmap v156: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:23:33.181 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:23:32 smithi040 ceph-mon[27984]: pgmap v157: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:23:33.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:23:32 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:23:33.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:23:32 smithi174 ceph-mon[31409]: pgmap v157: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:23:33.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:23:32 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:23:33.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:23:32 smithi111 ceph-mon[31085]: pgmap v157: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:23:33.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:23:32 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:23:34.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:23:33 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:23:34.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:23:33 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:23:34.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:23:33 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:23:34.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:23:33 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:23:34.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:23:33 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:23:34.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:23:33 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:23:34.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:23:33 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:23:34.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:23:33 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:23:34.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:23:33 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:23:35.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:23:34 smithi040 ceph-mon[27984]: pgmap v158: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:23:35.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:23:34 smithi174 ceph-mon[31409]: pgmap v158: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:23:35.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:23:34 smithi111 ceph-mon[31085]: pgmap v158: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:23:37.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:23:36 smithi040 ceph-mon[27984]: pgmap v159: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:23:37.208 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:23:36 smithi111 ceph-mon[31085]: pgmap v159: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:23:37.208 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:23:36 smithi174 ceph-mon[31409]: pgmap v159: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:23:39.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:23:38 smithi040 ceph-mon[27984]: pgmap v160: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:23:39.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:23:38 smithi174 ceph-mon[31409]: pgmap v160: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:23:39.193 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:23:38 smithi111 ceph-mon[31085]: pgmap v160: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:23:41.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:23:40 smithi040 ceph-mon[27984]: pgmap v161: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:23:41.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:23:40 smithi111 ceph-mon[31085]: pgmap v161: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:23:41.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:23:40 smithi174 ceph-mon[31409]: pgmap v161: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:23:43.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:23:42 smithi040 ceph-mon[27984]: pgmap v162: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:23:43.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:23:42 smithi174 ceph-mon[31409]: pgmap v162: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:23:43.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:23:42 smithi111 ceph-mon[31085]: pgmap v162: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:23:45.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:23:44 smithi040 ceph-mon[27984]: pgmap v163: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:23:45.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:23:44 smithi174 ceph-mon[31409]: pgmap v163: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:23:45.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:23:44 smithi111 ceph-mon[31085]: pgmap v163: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:23:47.181 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:23:46 smithi040 ceph-mon[27984]: pgmap v164: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:23:47.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:23:46 smithi174 ceph-mon[31409]: pgmap v164: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:23:47.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:23:46 smithi111 ceph-mon[31085]: pgmap v164: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:23:49.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:23:48 smithi040 ceph-mon[27984]: pgmap v165: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:23:49.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:23:48 smithi174 ceph-mon[31409]: pgmap v165: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:23:49.193 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:23:48 smithi111 ceph-mon[31085]: pgmap v165: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:23:51.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:23:50 smithi040 ceph-mon[27984]: pgmap v166: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:23:51.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:23:50 smithi174 ceph-mon[31409]: pgmap v166: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:23:51.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:23:50 smithi111 ceph-mon[31085]: pgmap v166: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:23:53.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:23:52 smithi040 ceph-mon[27984]: pgmap v167: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:23:53.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:23:52 smithi174 ceph-mon[31409]: pgmap v167: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:23:53.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:23:52 smithi111 ceph-mon[31085]: pgmap v167: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:23:55.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:23:54 smithi040 ceph-mon[27984]: pgmap v168: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:23:55.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:23:54 smithi174 ceph-mon[31409]: pgmap v168: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:23:55.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:23:54 smithi111 ceph-mon[31085]: pgmap v168: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:23:57.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:23:56 smithi040 ceph-mon[27984]: pgmap v169: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:23:57.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:23:56 smithi174 ceph-mon[31409]: pgmap v169: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:23:57.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:23:56 smithi111 ceph-mon[31085]: pgmap v169: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:23:59.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:23:58 smithi040 ceph-mon[27984]: pgmap v170: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:23:59.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:23:58 smithi174 ceph-mon[31409]: pgmap v170: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:23:59.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:23:58 smithi111 ceph-mon[31085]: pgmap v170: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:24:01.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:24:00 smithi040 ceph-mon[27984]: pgmap v171: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:24:01.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:24:00 smithi174 ceph-mon[31409]: pgmap v171: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:24:01.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:24:00 smithi111 ceph-mon[31085]: pgmap v171: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:24:03.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:24:02 smithi040 ceph-mon[27984]: pgmap v172: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:24:03.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:24:02 smithi174 ceph-mon[31409]: pgmap v172: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:24:03.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:24:02 smithi111 ceph-mon[31085]: pgmap v172: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:24:05.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:24:04 smithi040 ceph-mon[27984]: pgmap v173: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:24:05.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:24:04 smithi174 ceph-mon[31409]: pgmap v173: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:24:05.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:24:04 smithi111 ceph-mon[31085]: pgmap v173: 129 pgs: 129 active+clean; 101 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:24:07.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:24:06 smithi040 ceph-mon[27984]: pgmap v174: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-21T23:24:07.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:24:06 smithi174 ceph-mon[31409]: pgmap v174: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-21T23:24:07.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:24:06 smithi111 ceph-mon[31085]: pgmap v174: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-21T23:24:09.246 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:24:08 smithi174 ceph-mon[31409]: pgmap v175: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-04-21T23:24:09.246 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:24:08 smithi111 ceph-mon[31085]: pgmap v175: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-04-21T23:24:09.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:24:08 smithi040 ceph-mon[27984]: pgmap v175: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-04-21T23:24:11.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:24:10 smithi040 ceph-mon[27984]: pgmap v176: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-21T23:24:11.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:24:10 smithi174 ceph-mon[31409]: pgmap v176: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-21T23:24:11.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:24:10 smithi111 ceph-mon[31085]: pgmap v176: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-21T23:24:13.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:24:12 smithi040 ceph-mon[27984]: pgmap v177: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-21T23:24:13.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:24:12 smithi174 ceph-mon[31409]: pgmap v177: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-21T23:24:13.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:24:12 smithi111 ceph-mon[31085]: pgmap v177: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-21T23:24:15.234 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:24:14 smithi040 ceph-mon[27984]: pgmap v178: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-21T23:24:15.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:24:14 smithi174 ceph-mon[31409]: pgmap v178: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-21T23:24:15.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:24:14 smithi111 ceph-mon[31085]: pgmap v178: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-21T23:24:17.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:24:16 smithi040 ceph-mon[27984]: pgmap v179: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-21T23:24:17.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:24:16 smithi174 ceph-mon[31409]: pgmap v179: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-21T23:24:17.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:24:16 smithi111 ceph-mon[31085]: pgmap v179: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-21T23:24:19.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:24:18 smithi040 ceph-mon[27984]: pgmap v180: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-21T23:24:19.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:24:18 smithi174 ceph-mon[31409]: pgmap v180: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-21T23:24:19.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:24:18 smithi111 ceph-mon[31085]: pgmap v180: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-21T23:24:21.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:24:20 smithi040 ceph-mon[27984]: pgmap v181: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 7 op/s 2024-04-21T23:24:21.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:24:20 smithi174 ceph-mon[31409]: pgmap v181: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 7 op/s 2024-04-21T23:24:21.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:24:20 smithi111 ceph-mon[31085]: pgmap v181: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 7 op/s 2024-04-21T23:24:23.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:24:22 smithi040 ceph-mon[27984]: pgmap v182: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-21T23:24:23.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:24:22 smithi174 ceph-mon[31409]: pgmap v182: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-21T23:24:23.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:24:22 smithi111 ceph-mon[31085]: pgmap v182: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-21T23:24:25.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:24:25 smithi040 ceph-mon[27984]: pgmap v183: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-21T23:24:25.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:24:25 smithi174 ceph-mon[31409]: pgmap v183: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-21T23:24:25.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:24:25 smithi111 ceph-mon[31085]: pgmap v183: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-21T23:24:27.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:24:27 smithi040 ceph-mon[27984]: pgmap v184: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2024-04-21T23:24:27.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:24:27 smithi174 ceph-mon[31409]: pgmap v184: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2024-04-21T23:24:27.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:24:27 smithi111 ceph-mon[31085]: pgmap v184: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2024-04-21T23:24:29.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:24:29 smithi040 ceph-mon[27984]: pgmap v185: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:24:29.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:24:29 smithi174 ceph-mon[31409]: pgmap v185: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:24:29.443 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:24:29 smithi111 ceph-mon[31085]: pgmap v185: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:24:31.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:24:31 smithi040 ceph-mon[27984]: pgmap v186: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:24:31.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:24:31 smithi174 ceph-mon[31409]: pgmap v186: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:24:31.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:24:31 smithi111 ceph-mon[31085]: pgmap v186: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:24:33.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:24:33 smithi040 ceph-mon[27984]: pgmap v187: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:24:33.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:24:33 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:24:33.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:24:33 smithi174 ceph-mon[31409]: pgmap v187: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:24:33.571 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:24:33 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:24:33.572 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:24:33 smithi111 ceph-mon[31085]: pgmap v187: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:24:33.572 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:24:33 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:24:34.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:24:34 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:24:34.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:24:34 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:24:34.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:24:34 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:24:34.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:24:34 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:24:34.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:24:34 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:24:34.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:24:34 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:24:34.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:24:34 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:24:34.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:24:34 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:24:34.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:24:34 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:24:35.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:24:35 smithi040 ceph-mon[27984]: pgmap v188: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:24:35.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:24:35 smithi174 ceph-mon[31409]: pgmap v188: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:24:35.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:24:35 smithi111 ceph-mon[31085]: pgmap v188: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:24:37.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:24:37 smithi040 ceph-mon[27984]: pgmap v189: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:24:37.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:24:37 smithi174 ceph-mon[31409]: pgmap v189: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:24:37.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:24:37 smithi111 ceph-mon[31085]: pgmap v189: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:24:39.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:24:39 smithi040 ceph-mon[27984]: pgmap v190: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:24:39.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:24:39 smithi174 ceph-mon[31409]: pgmap v190: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:24:39.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:24:39 smithi111 ceph-mon[31085]: pgmap v190: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:24:41.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:24:41 smithi040 ceph-mon[27984]: pgmap v191: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:24:41.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:24:41 smithi174 ceph-mon[31409]: pgmap v191: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:24:41.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:24:41 smithi111 ceph-mon[31085]: pgmap v191: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:24:43.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:24:43 smithi040 ceph-mon[27984]: pgmap v192: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:24:43.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:24:43 smithi174 ceph-mon[31409]: pgmap v192: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:24:43.443 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:24:43 smithi111 ceph-mon[31085]: pgmap v192: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:24:45.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:24:45 smithi040 ceph-mon[27984]: pgmap v193: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:24:45.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:24:45 smithi174 ceph-mon[31409]: pgmap v193: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:24:45.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:24:45 smithi111 ceph-mon[31085]: pgmap v193: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:24:47.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:24:47 smithi040 ceph-mon[27984]: pgmap v194: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:24:47.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:24:47 smithi174 ceph-mon[31409]: pgmap v194: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:24:47.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:24:47 smithi111 ceph-mon[31085]: pgmap v194: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:24:49.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:24:49 smithi040 ceph-mon[27984]: pgmap v195: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:24:49.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:24:49 smithi174 ceph-mon[31409]: pgmap v195: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:24:49.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:24:49 smithi111 ceph-mon[31085]: pgmap v195: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:24:51.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:24:51 smithi040 ceph-mon[27984]: pgmap v196: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:24:51.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:24:51 smithi174 ceph-mon[31409]: pgmap v196: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:24:51.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:24:51 smithi111 ceph-mon[31085]: pgmap v196: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:24:53.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:24:53 smithi040 ceph-mon[27984]: pgmap v197: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:24:53.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:24:53 smithi174 ceph-mon[31409]: pgmap v197: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:24:53.443 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:24:53 smithi111 ceph-mon[31085]: pgmap v197: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:24:55.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:24:55 smithi040 ceph-mon[27984]: pgmap v198: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:24:55.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:24:55 smithi174 ceph-mon[31409]: pgmap v198: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:24:55.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:24:55 smithi111 ceph-mon[31085]: pgmap v198: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:24:57.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:24:57 smithi040 ceph-mon[27984]: pgmap v199: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:24:57.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:24:57 smithi174 ceph-mon[31409]: pgmap v199: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:24:57.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:24:57 smithi111 ceph-mon[31085]: pgmap v199: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:24:59.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:24:59 smithi040 ceph-mon[27984]: pgmap v200: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:24:59.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:24:59 smithi174 ceph-mon[31409]: pgmap v200: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:24:59.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:24:59 smithi111 ceph-mon[31085]: pgmap v200: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:25:01.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:25:01 smithi040 ceph-mon[27984]: pgmap v201: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:25:01.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:25:01 smithi174 ceph-mon[31409]: pgmap v201: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:25:01.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:25:01 smithi111 ceph-mon[31085]: pgmap v201: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:25:03.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:25:03 smithi040 ceph-mon[27984]: pgmap v202: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:25:03.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:25:03 smithi174 ceph-mon[31409]: pgmap v202: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:25:03.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:25:03 smithi111 ceph-mon[31085]: pgmap v202: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:25:05.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:25:05 smithi040 ceph-mon[27984]: pgmap v203: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:25:05.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:25:05 smithi174 ceph-mon[31409]: pgmap v203: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:25:05.443 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:25:05 smithi111 ceph-mon[31085]: pgmap v203: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:25:07.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:25:07 smithi040 ceph-mon[27984]: pgmap v204: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-21T23:25:07.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:25:07 smithi174 ceph-mon[31409]: pgmap v204: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-21T23:25:07.443 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:25:07 smithi111 ceph-mon[31085]: pgmap v204: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-21T23:25:09.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:25:09 smithi040 ceph-mon[27984]: pgmap v205: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-04-21T23:25:09.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:25:09 smithi174 ceph-mon[31409]: pgmap v205: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-04-21T23:25:09.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:25:09 smithi111 ceph-mon[31085]: pgmap v205: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-04-21T23:25:11.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:25:11 smithi040 ceph-mon[27984]: pgmap v206: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-21T23:25:11.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:25:11 smithi040 ceph-mon[27984]: osdmap e96: 12 total, 12 up, 12 in 2024-04-21T23:25:11.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:25:11 smithi174 ceph-mon[31409]: pgmap v206: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-21T23:25:11.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:25:11 smithi174 ceph-mon[31409]: osdmap e96: 12 total, 12 up, 12 in 2024-04-21T23:25:11.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:25:11 smithi111 ceph-mon[31085]: pgmap v206: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-21T23:25:11.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:25:11 smithi111 ceph-mon[31085]: osdmap e96: 12 total, 12 up, 12 in 2024-04-21T23:25:13.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:25:13 smithi040 ceph-mon[27984]: pgmap v208: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-21T23:25:13.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:25:13 smithi174 ceph-mon[31409]: pgmap v208: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-21T23:25:13.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:25:13 smithi111 ceph-mon[31085]: pgmap v208: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-21T23:25:15.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:25:15 smithi040 ceph-mon[27984]: pgmap v209: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-21T23:25:15.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:25:15 smithi174 ceph-mon[31409]: pgmap v209: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-21T23:25:15.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:25:15 smithi111 ceph-mon[31085]: pgmap v209: 129 pgs: 129 active+clean; 101 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-21T23:25:17.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:25:17 smithi040 ceph-mon[27984]: pgmap v210: 129 pgs: 129 active+clean; 101 MiB data, 785 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-21T23:25:17.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:25:17 smithi174 ceph-mon[31409]: pgmap v210: 129 pgs: 129 active+clean; 101 MiB data, 785 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-21T23:25:17.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:25:17 smithi111 ceph-mon[31085]: pgmap v210: 129 pgs: 129 active+clean; 101 MiB data, 785 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-21T23:25:19.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:25:18 smithi040 ceph-mon[27984]: pgmap v211: 129 pgs: 129 active+clean; 101 MiB data, 785 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-21T23:25:19.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:25:18 smithi040 ceph-mon[27984]: osdmap e97: 12 total, 12 up, 12 in 2024-04-21T23:25:19.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:25:18 smithi174 ceph-mon[31409]: pgmap v211: 129 pgs: 129 active+clean; 101 MiB data, 785 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-21T23:25:19.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:25:18 smithi174 ceph-mon[31409]: osdmap e97: 12 total, 12 up, 12 in 2024-04-21T23:25:19.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:25:18 smithi111 ceph-mon[31085]: pgmap v211: 129 pgs: 129 active+clean; 101 MiB data, 785 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-21T23:25:19.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:25:18 smithi111 ceph-mon[31085]: osdmap e97: 12 total, 12 up, 12 in 2024-04-21T23:25:21.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:25:20 smithi040 ceph-mon[27984]: pgmap v213: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 216 B/s wr, 6 op/s 2024-04-21T23:25:21.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:25:20 smithi174 ceph-mon[31409]: pgmap v213: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 216 B/s wr, 6 op/s 2024-04-21T23:25:21.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:25:20 smithi111 ceph-mon[31085]: pgmap v213: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 216 B/s wr, 6 op/s 2024-04-21T23:25:23.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:25:22 smithi040 ceph-mon[27984]: pgmap v214: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-21T23:25:23.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:25:22 smithi174 ceph-mon[31409]: pgmap v214: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-21T23:25:23.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:25:22 smithi111 ceph-mon[31085]: pgmap v214: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-21T23:25:25.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:25:24 smithi040 ceph-mon[27984]: pgmap v215: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-21T23:25:25.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:25:24 smithi174 ceph-mon[31409]: pgmap v215: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-21T23:25:25.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:25:24 smithi111 ceph-mon[31085]: pgmap v215: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-21T23:25:27.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:25:26 smithi040 ceph-mon[27984]: pgmap v216: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-21T23:25:27.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:25:26 smithi174 ceph-mon[31409]: pgmap v216: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-21T23:25:27.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:25:26 smithi111 ceph-mon[31085]: pgmap v216: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-21T23:25:29.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:25:28 smithi040 ceph-mon[27984]: pgmap v217: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-21T23:25:29.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:25:28 smithi174 ceph-mon[31409]: pgmap v217: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-21T23:25:29.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:25:28 smithi111 ceph-mon[31085]: pgmap v217: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-21T23:25:31.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:25:30 smithi040 ceph-mon[27984]: pgmap v218: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 88 B/s wr, 7 op/s 2024-04-21T23:25:31.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:25:30 smithi174 ceph-mon[31409]: pgmap v218: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 88 B/s wr, 7 op/s 2024-04-21T23:25:31.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:25:30 smithi111 ceph-mon[31085]: pgmap v218: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 88 B/s wr, 7 op/s 2024-04-21T23:25:33.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:25:32 smithi040 ceph-mon[27984]: pgmap v219: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:25:33.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:25:32 smithi174 ceph-mon[31409]: pgmap v219: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:25:33.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:25:32 smithi111 ceph-mon[31085]: pgmap v219: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:25:34.434 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:25:34 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:25:34.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:25:34 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:25:34.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:25:34 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:25:35.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:25:35 smithi040 ceph-mon[27984]: pgmap v220: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:25:35.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:25:35 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:25:35.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:25:35 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:25:35.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:25:35 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:25:35.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:25:35 smithi174 ceph-mon[31409]: pgmap v220: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:25:35.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:25:35 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:25:35.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:25:35 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:25:35.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:25:35 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:25:35.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:25:35 smithi111 ceph-mon[31085]: pgmap v220: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:25:35.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:25:35 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:25:35.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:25:35 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:25:35.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:25:35 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:25:37.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:25:37 smithi040 ceph-mon[27984]: pgmap v221: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:25:37.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:25:37 smithi174 ceph-mon[31409]: pgmap v221: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:25:37.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:25:37 smithi111 ceph-mon[31085]: pgmap v221: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:25:39.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:25:39 smithi040 ceph-mon[27984]: pgmap v222: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:25:39.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:25:39 smithi174 ceph-mon[31409]: pgmap v222: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:25:39.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:25:39 smithi111 ceph-mon[31085]: pgmap v222: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:25:41.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:25:41 smithi040 ceph-mon[27984]: pgmap v223: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:25:41.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:25:41 smithi174 ceph-mon[31409]: pgmap v223: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:25:41.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:25:41 smithi111 ceph-mon[31085]: pgmap v223: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:25:43.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:25:43 smithi040 ceph-mon[27984]: pgmap v224: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:25:43.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:25:43 smithi174 ceph-mon[31409]: pgmap v224: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:25:43.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:25:43 smithi111 ceph-mon[31085]: pgmap v224: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:25:45.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:25:45 smithi040 ceph-mon[27984]: pgmap v225: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:25:45.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:25:45 smithi174 ceph-mon[31409]: pgmap v225: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:25:45.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:25:45 smithi111 ceph-mon[31085]: pgmap v225: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:25:47.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:25:47 smithi040 ceph-mon[27984]: pgmap v226: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:25:47.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:25:47 smithi174 ceph-mon[31409]: pgmap v226: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:25:47.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:25:47 smithi111 ceph-mon[31085]: pgmap v226: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:25:49.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:25:49 smithi040 ceph-mon[27984]: pgmap v227: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:25:49.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:25:49 smithi174 ceph-mon[31409]: pgmap v227: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:25:49.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:25:49 smithi111 ceph-mon[31085]: pgmap v227: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:25:51.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:25:51 smithi040 ceph-mon[27984]: pgmap v228: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:25:51.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:25:51 smithi174 ceph-mon[31409]: pgmap v228: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:25:51.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:25:51 smithi111 ceph-mon[31085]: pgmap v228: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:25:53.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:25:53 smithi040 ceph-mon[27984]: pgmap v229: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-21T23:25:53.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:25:53 smithi111 ceph-mon[31085]: pgmap v229: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-21T23:25:53.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:25:53 smithi174 ceph-mon[31409]: pgmap v229: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-21T23:25:55.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:25:55 smithi040 ceph-mon[27984]: pgmap v230: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-21T23:25:55.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:25:55 smithi174 ceph-mon[31409]: pgmap v230: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-21T23:25:55.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:25:55 smithi111 ceph-mon[31085]: pgmap v230: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-21T23:25:57.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:25:57 smithi040 ceph-mon[27984]: pgmap v231: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:25:57.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:25:57 smithi174 ceph-mon[31409]: pgmap v231: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:25:57.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:25:57 smithi111 ceph-mon[31085]: pgmap v231: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:25:59.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:25:59 smithi040 ceph-mon[27984]: pgmap v232: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-21T23:25:59.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:25:59 smithi174 ceph-mon[31409]: pgmap v232: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-21T23:25:59.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:25:59 smithi111 ceph-mon[31085]: pgmap v232: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-21T23:26:01.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:26:01 smithi040 ceph-mon[27984]: pgmap v233: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-04-21T23:26:01.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:26:01 smithi174 ceph-mon[31409]: pgmap v233: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-04-21T23:26:01.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:26:01 smithi111 ceph-mon[31085]: pgmap v233: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-04-21T23:26:03.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:26:03 smithi040 ceph-mon[27984]: pgmap v234: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-21T23:26:03.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:26:03 smithi174 ceph-mon[31409]: pgmap v234: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-21T23:26:03.446 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:26:03 smithi111 ceph-mon[31085]: pgmap v234: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-21T23:26:05.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:26:05 smithi040 ceph-mon[27984]: pgmap v235: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-21T23:26:05.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:26:05 smithi174 ceph-mon[31409]: pgmap v235: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-21T23:26:05.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:26:05 smithi111 ceph-mon[31085]: pgmap v235: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-21T23:26:07.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:26:07 smithi040 ceph-mon[27984]: pgmap v236: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-21T23:26:07.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:26:07 smithi174 ceph-mon[31409]: pgmap v236: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-21T23:26:07.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:26:07 smithi111 ceph-mon[31085]: pgmap v236: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-21T23:26:09.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:26:09 smithi040 ceph-mon[27984]: pgmap v237: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 21 op/s 2024-04-21T23:26:09.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:26:09 smithi174 ceph-mon[31409]: pgmap v237: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 21 op/s 2024-04-21T23:26:09.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:26:09 smithi111 ceph-mon[31085]: pgmap v237: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 21 op/s 2024-04-21T23:26:11.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:26:11 smithi040 ceph-mon[27984]: pgmap v238: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-21T23:26:11.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:26:11 smithi040 ceph-mon[27984]: osdmap e98: 12 total, 12 up, 12 in 2024-04-21T23:26:11.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:26:11 smithi174 ceph-mon[31409]: pgmap v238: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-21T23:26:11.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:26:11 smithi174 ceph-mon[31409]: osdmap e98: 12 total, 12 up, 12 in 2024-04-21T23:26:11.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:26:11 smithi111 ceph-mon[31085]: pgmap v238: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-21T23:26:11.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:26:11 smithi111 ceph-mon[31085]: osdmap e98: 12 total, 12 up, 12 in 2024-04-21T23:26:13.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:26:13 smithi040 ceph-mon[27984]: pgmap v240: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-21T23:26:13.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:26:13 smithi174 ceph-mon[31409]: pgmap v240: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-21T23:26:13.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:26:13 smithi111 ceph-mon[31085]: pgmap v240: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-21T23:26:15.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:26:15 smithi040 ceph-mon[27984]: pgmap v241: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-21T23:26:15.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:26:15 smithi174 ceph-mon[31409]: pgmap v241: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-21T23:26:15.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:26:15 smithi111 ceph-mon[31085]: pgmap v241: 129 pgs: 129 active+clean; 101 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-21T23:26:17.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:26:17 smithi040 ceph-mon[27984]: pgmap v242: 129 pgs: 129 active+clean; 95 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-21T23:26:17.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:26:17 smithi174 ceph-mon[31409]: pgmap v242: 129 pgs: 129 active+clean; 95 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-21T23:26:17.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:26:17 smithi111 ceph-mon[31085]: pgmap v242: 129 pgs: 129 active+clean; 95 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-21T23:26:19.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:26:18 smithi040 ceph-mon[27984]: pgmap v243: 129 pgs: 129 active+clean; 95 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-21T23:26:19.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:26:18 smithi040 ceph-mon[27984]: osdmap e99: 12 total, 12 up, 12 in 2024-04-21T23:26:19.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:26:18 smithi174 ceph-mon[31409]: pgmap v243: 129 pgs: 129 active+clean; 95 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-21T23:26:19.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:26:18 smithi174 ceph-mon[31409]: osdmap e99: 12 total, 12 up, 12 in 2024-04-21T23:26:19.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:26:18 smithi111 ceph-mon[31085]: pgmap v243: 129 pgs: 129 active+clean; 95 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-21T23:26:19.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:26:18 smithi111 ceph-mon[31085]: osdmap e99: 12 total, 12 up, 12 in 2024-04-21T23:26:21.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:26:20 smithi040 ceph-mon[27984]: pgmap v245: 129 pgs: 129 active+clean; 95 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 216 B/s wr, 6 op/s 2024-04-21T23:26:21.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:26:20 smithi174 ceph-mon[31409]: pgmap v245: 129 pgs: 129 active+clean; 95 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 216 B/s wr, 6 op/s 2024-04-21T23:26:21.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:26:20 smithi111 ceph-mon[31085]: pgmap v245: 129 pgs: 129 active+clean; 95 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 216 B/s wr, 6 op/s 2024-04-21T23:26:23.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:26:22 smithi040 ceph-mon[27984]: pgmap v246: 129 pgs: 129 active+clean; 95 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-21T23:26:23.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:26:22 smithi174 ceph-mon[31409]: pgmap v246: 129 pgs: 129 active+clean; 95 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-21T23:26:23.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:26:22 smithi111 ceph-mon[31085]: pgmap v246: 129 pgs: 129 active+clean; 95 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-21T23:26:25.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:26:24 smithi040 ceph-mon[27984]: pgmap v247: 129 pgs: 129 active+clean; 95 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-21T23:26:25.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:26:24 smithi174 ceph-mon[31409]: pgmap v247: 129 pgs: 129 active+clean; 95 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-21T23:26:25.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:26:24 smithi111 ceph-mon[31085]: pgmap v247: 129 pgs: 129 active+clean; 95 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-21T23:26:27.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:26:26 smithi040 ceph-mon[27984]: pgmap v248: 129 pgs: 129 active+clean; 95 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-21T23:26:27.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:26:27 smithi174 ceph-mon[31409]: pgmap v248: 129 pgs: 129 active+clean; 95 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-21T23:26:27.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:26:27 smithi111 ceph-mon[31085]: pgmap v248: 129 pgs: 129 active+clean; 95 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-21T23:26:29.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:26:29 smithi040 ceph-mon[27984]: pgmap v249: 129 pgs: 129 active+clean; 95 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-21T23:26:29.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:26:29 smithi174 ceph-mon[31409]: pgmap v249: 129 pgs: 129 active+clean; 95 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-21T23:26:29.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:26:29 smithi111 ceph-mon[31085]: pgmap v249: 129 pgs: 129 active+clean; 95 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-21T23:26:31.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:26:31 smithi040 ceph-mon[27984]: pgmap v250: 129 pgs: 129 active+clean; 95 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 88 B/s wr, 7 op/s 2024-04-21T23:26:31.447 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:26:31 smithi111 ceph-mon[31085]: pgmap v250: 129 pgs: 129 active+clean; 95 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 88 B/s wr, 7 op/s 2024-04-21T23:26:31.447 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:26:31 smithi174 ceph-mon[31409]: pgmap v250: 129 pgs: 129 active+clean; 95 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 88 B/s wr, 7 op/s 2024-04-21T23:26:33.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:26:33 smithi040 ceph-mon[27984]: pgmap v251: 129 pgs: 129 active+clean; 95 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:26:33.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:26:33 smithi174 ceph-mon[31409]: pgmap v251: 129 pgs: 129 active+clean; 95 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:26:33.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:26:33 smithi111 ceph-mon[31085]: pgmap v251: 129 pgs: 129 active+clean; 95 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:26:35.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:26:35 smithi040 ceph-mon[27984]: pgmap v252: 129 pgs: 129 active+clean; 95 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:26:35.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:26:35 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:26:35.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:26:35 smithi174 ceph-mon[31409]: pgmap v252: 129 pgs: 129 active+clean; 95 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:26:35.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:26:35 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:26:35.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:26:35 smithi111 ceph-mon[31085]: pgmap v252: 129 pgs: 129 active+clean; 95 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:26:35.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:26:35 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:26:36.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:26:36 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:26:36.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:26:36 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:26:36.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:26:36 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:26:36.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:26:36 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:26:36.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:26:36 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:26:36.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:26:36 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:26:36.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:26:36 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:26:36.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:26:36 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:26:36.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:26:36 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:26:37.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:26:37 smithi040 ceph-mon[27984]: pgmap v253: 129 pgs: 129 active+clean; 95 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:26:37.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:26:37 smithi174 ceph-mon[31409]: pgmap v253: 129 pgs: 129 active+clean; 95 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:26:37.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:26:37 smithi111 ceph-mon[31085]: pgmap v253: 129 pgs: 129 active+clean; 95 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:26:39.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:26:39 smithi040 ceph-mon[27984]: pgmap v254: 129 pgs: 129 active+clean; 95 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:26:39.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:26:39 smithi174 ceph-mon[31409]: pgmap v254: 129 pgs: 129 active+clean; 95 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:26:39.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:26:39 smithi111 ceph-mon[31085]: pgmap v254: 129 pgs: 129 active+clean; 95 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:26:41.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:26:41 smithi040 ceph-mon[27984]: pgmap v255: 129 pgs: 129 active+clean; 95 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:26:41.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:26:41 smithi174 ceph-mon[31409]: pgmap v255: 129 pgs: 129 active+clean; 95 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:26:41.443 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:26:41 smithi111 ceph-mon[31085]: pgmap v255: 129 pgs: 129 active+clean; 95 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:26:43.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:26:43 smithi040 ceph-mon[27984]: pgmap v256: 129 pgs: 129 active+clean; 95 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:26:43.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:26:43 smithi174 ceph-mon[31409]: pgmap v256: 129 pgs: 129 active+clean; 95 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:26:43.443 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:26:43 smithi111 ceph-mon[31085]: pgmap v256: 129 pgs: 129 active+clean; 95 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:26:44.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:26:44 smithi174 ceph-mon[31409]: pgmap v257: 129 pgs: 129 active+clean; 95 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:26:44.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:26:44 smithi111 ceph-mon[31085]: pgmap v257: 129 pgs: 129 active+clean; 95 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:26:44.681 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:26:44 smithi040 ceph-mon[27984]: pgmap v257: 129 pgs: 129 active+clean; 95 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:26:46.931 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:26:46 smithi040 ceph-mon[27984]: pgmap v258: 129 pgs: 129 active+clean; 95 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:26:46.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:26:46 smithi174 ceph-mon[31409]: pgmap v258: 129 pgs: 129 active+clean; 95 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:26:46.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:26:46 smithi111 ceph-mon[31085]: pgmap v258: 129 pgs: 129 active+clean; 95 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:26:48.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:26:48 smithi040 ceph-mon[27984]: pgmap v259: 129 pgs: 129 active+clean; 95 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:26:48.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:26:48 smithi174 ceph-mon[31409]: pgmap v259: 129 pgs: 129 active+clean; 95 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:26:48.943 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:26:48 smithi111 ceph-mon[31085]: pgmap v259: 129 pgs: 129 active+clean; 95 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:26:50.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:26:50 smithi040 ceph-mon[27984]: pgmap v260: 129 pgs: 129 active+clean; 95 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:26:50.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:26:50 smithi174 ceph-mon[31409]: pgmap v260: 129 pgs: 129 active+clean; 95 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:26:50.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:26:50 smithi111 ceph-mon[31085]: pgmap v260: 129 pgs: 129 active+clean; 95 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:26:52.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:26:52 smithi040 ceph-mon[27984]: pgmap v261: 129 pgs: 129 active+clean; 95 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:26:52.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:26:52 smithi174 ceph-mon[31409]: pgmap v261: 129 pgs: 129 active+clean; 95 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:26:52.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:26:52 smithi111 ceph-mon[31085]: pgmap v261: 129 pgs: 129 active+clean; 95 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:26:54.931 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:26:54 smithi040 ceph-mon[27984]: pgmap v262: 129 pgs: 129 active+clean; 95 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:26:54.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:26:54 smithi174 ceph-mon[31409]: pgmap v262: 129 pgs: 129 active+clean; 95 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:26:54.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:26:54 smithi111 ceph-mon[31085]: pgmap v262: 129 pgs: 129 active+clean; 95 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:26:56.931 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:26:56 smithi040 ceph-mon[27984]: pgmap v263: 129 pgs: 129 active+clean; 95 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:26:56.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:26:56 smithi174 ceph-mon[31409]: pgmap v263: 129 pgs: 129 active+clean; 95 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:26:56.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:26:56 smithi111 ceph-mon[31085]: pgmap v263: 129 pgs: 129 active+clean; 95 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:26:58.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:26:58 smithi040 ceph-mon[27984]: pgmap v264: 129 pgs: 129 active+clean; 95 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:26:58.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:26:58 smithi174 ceph-mon[31409]: pgmap v264: 129 pgs: 129 active+clean; 95 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:26:58.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:26:58 smithi111 ceph-mon[31085]: pgmap v264: 129 pgs: 129 active+clean; 95 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:27:00.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:27:00 smithi040 ceph-mon[27984]: pgmap v265: 129 pgs: 129 active+clean; 95 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:27:00.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:27:00 smithi174 ceph-mon[31409]: pgmap v265: 129 pgs: 129 active+clean; 95 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:27:00.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:27:00 smithi111 ceph-mon[31085]: pgmap v265: 129 pgs: 129 active+clean; 95 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:27:02.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:27:02 smithi040 ceph-mon[27984]: pgmap v266: 129 pgs: 129 active+clean; 95 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:27:02.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:27:02 smithi174 ceph-mon[31409]: pgmap v266: 129 pgs: 129 active+clean; 95 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:27:02.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:27:02 smithi111 ceph-mon[31085]: pgmap v266: 129 pgs: 129 active+clean; 95 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:27:04.931 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:27:04 smithi040 ceph-mon[27984]: pgmap v267: 129 pgs: 129 active+clean; 95 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:27:04.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:27:04 smithi174 ceph-mon[31409]: pgmap v267: 129 pgs: 129 active+clean; 95 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:27:04.943 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:27:04 smithi111 ceph-mon[31085]: pgmap v267: 129 pgs: 129 active+clean; 95 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:27:06.931 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:27:06 smithi040 ceph-mon[27984]: pgmap v268: 129 pgs: 129 active+clean; 95 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-21T23:27:06.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:27:06 smithi174 ceph-mon[31409]: pgmap v268: 129 pgs: 129 active+clean; 95 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-21T23:27:06.943 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:27:06 smithi111 ceph-mon[31085]: pgmap v268: 129 pgs: 129 active+clean; 95 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-21T23:27:08.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:27:08 smithi040 ceph-mon[27984]: pgmap v269: 129 pgs: 129 active+clean; 95 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-04-21T23:27:08.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:27:08 smithi174 ceph-mon[31409]: pgmap v269: 129 pgs: 129 active+clean; 95 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-04-21T23:27:08.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:27:08 smithi111 ceph-mon[31085]: pgmap v269: 129 pgs: 129 active+clean; 95 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-04-21T23:27:10.934 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:27:10 smithi040 ceph-mon[27984]: pgmap v270: 129 pgs: 129 active+clean; 95 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-21T23:27:10.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:27:10 smithi174 ceph-mon[31409]: pgmap v270: 129 pgs: 129 active+clean; 95 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-21T23:27:10.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:27:10 smithi111 ceph-mon[31085]: pgmap v270: 129 pgs: 129 active+clean; 95 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-21T23:27:11.931 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:27:11 smithi040 ceph-mon[27984]: osdmap e100: 12 total, 12 up, 12 in 2024-04-21T23:27:11.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:27:11 smithi174 ceph-mon[31409]: osdmap e100: 12 total, 12 up, 12 in 2024-04-21T23:27:11.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:27:11 smithi111 ceph-mon[31085]: osdmap e100: 12 total, 12 up, 12 in 2024-04-21T23:27:12.931 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:27:12 smithi040 ceph-mon[27984]: pgmap v272: 129 pgs: 129 active+clean; 95 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-21T23:27:12.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:27:12 smithi174 ceph-mon[31409]: pgmap v272: 129 pgs: 129 active+clean; 95 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-21T23:27:12.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:27:12 smithi111 ceph-mon[31085]: pgmap v272: 129 pgs: 129 active+clean; 95 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-21T23:27:14.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:27:14 smithi040 ceph-mon[27984]: pgmap v273: 129 pgs: 129 active+clean; 95 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-21T23:27:14.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:27:14 smithi174 ceph-mon[31409]: pgmap v273: 129 pgs: 129 active+clean; 95 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-21T23:27:14.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:27:14 smithi111 ceph-mon[31085]: pgmap v273: 129 pgs: 129 active+clean; 95 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-21T23:27:16.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:27:16 smithi040 ceph-mon[27984]: pgmap v274: 129 pgs: 129 active+clean; 95 MiB data, 779 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-21T23:27:16.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:27:16 smithi174 ceph-mon[31409]: pgmap v274: 129 pgs: 129 active+clean; 95 MiB data, 779 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-21T23:27:16.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:27:16 smithi111 ceph-mon[31085]: pgmap v274: 129 pgs: 129 active+clean; 95 MiB data, 779 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-21T23:27:19.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:27:19 smithi040 ceph-mon[27984]: pgmap v275: 129 pgs: 129 active+clean; 95 MiB data, 779 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-21T23:27:19.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:27:19 smithi040 ceph-mon[27984]: osdmap e101: 12 total, 12 up, 12 in 2024-04-21T23:27:19.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:27:19 smithi174 ceph-mon[31409]: pgmap v275: 129 pgs: 129 active+clean; 95 MiB data, 779 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-21T23:27:19.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:27:19 smithi174 ceph-mon[31409]: osdmap e101: 12 total, 12 up, 12 in 2024-04-21T23:27:19.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:27:19 smithi111 ceph-mon[31085]: pgmap v275: 129 pgs: 129 active+clean; 95 MiB data, 779 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-21T23:27:19.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:27:19 smithi111 ceph-mon[31085]: osdmap e101: 12 total, 12 up, 12 in 2024-04-21T23:27:21.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:27:21 smithi040 ceph-mon[27984]: pgmap v277: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2024-04-21T23:27:21.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:27:21 smithi174 ceph-mon[31409]: pgmap v277: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2024-04-21T23:27:21.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:27:21 smithi111 ceph-mon[31085]: pgmap v277: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2024-04-21T23:27:23.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:27:23 smithi040 ceph-mon[27984]: pgmap v278: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-21T23:27:23.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:27:23 smithi111 ceph-mon[31085]: pgmap v278: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-21T23:27:23.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:27:23 smithi174 ceph-mon[31409]: pgmap v278: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-21T23:27:25.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:27:25 smithi040 ceph-mon[27984]: pgmap v279: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-21T23:27:25.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:27:25 smithi174 ceph-mon[31409]: pgmap v279: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-21T23:27:25.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:27:25 smithi111 ceph-mon[31085]: pgmap v279: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-21T23:27:27.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:27:27 smithi040 ceph-mon[27984]: pgmap v280: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-21T23:27:27.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:27:27 smithi174 ceph-mon[31409]: pgmap v280: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-21T23:27:27.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:27:27 smithi111 ceph-mon[31085]: pgmap v280: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-21T23:27:29.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:27:29 smithi040 ceph-mon[27984]: pgmap v281: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-21T23:27:29.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:27:29 smithi174 ceph-mon[31409]: pgmap v281: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-21T23:27:29.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:27:29 smithi111 ceph-mon[31085]: pgmap v281: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-21T23:27:31.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:27:31 smithi040 ceph-mon[27984]: pgmap v282: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 88 B/s wr, 7 op/s 2024-04-21T23:27:31.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:27:31 smithi174 ceph-mon[31409]: pgmap v282: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 88 B/s wr, 7 op/s 2024-04-21T23:27:31.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:27:31 smithi111 ceph-mon[31085]: pgmap v282: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 88 B/s wr, 7 op/s 2024-04-21T23:27:33.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:27:33 smithi040 ceph-mon[27984]: pgmap v283: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:27:33.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:27:33 smithi174 ceph-mon[31409]: pgmap v283: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:27:33.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:27:33 smithi111 ceph-mon[31085]: pgmap v283: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:27:35.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:27:35 smithi040 ceph-mon[27984]: pgmap v284: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:27:35.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:27:35 smithi174 ceph-mon[31409]: pgmap v284: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:27:35.443 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:27:35 smithi111 ceph-mon[31085]: pgmap v284: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:27:36.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:27:36 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:27:36.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:27:36 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:27:36.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:27:36 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:27:36.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:27:36 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:27:36.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:27:36 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:27:36.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:27:36 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:27:36.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:27:36 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:27:36.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:27:36 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:27:36.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:27:36 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:27:36.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:27:36 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:27:36.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:27:36 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:27:36.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:27:36 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:27:37.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:27:37 smithi040 ceph-mon[27984]: pgmap v285: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:27:37.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:27:37 smithi174 ceph-mon[31409]: pgmap v285: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:27:37.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:27:37 smithi111 ceph-mon[31085]: pgmap v285: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:27:39.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:27:39 smithi040 ceph-mon[27984]: pgmap v286: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:27:39.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:27:39 smithi174 ceph-mon[31409]: pgmap v286: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:27:39.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:27:39 smithi111 ceph-mon[31085]: pgmap v286: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:27:41.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:27:41 smithi040 ceph-mon[27984]: pgmap v287: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:27:41.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:27:41 smithi174 ceph-mon[31409]: pgmap v287: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:27:41.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:27:41 smithi111 ceph-mon[31085]: pgmap v287: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:27:43.433 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:27:43 smithi040 ceph-mon[27984]: pgmap v288: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:27:43.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:27:43 smithi174 ceph-mon[31409]: pgmap v288: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:27:43.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:27:43 smithi111 ceph-mon[31085]: pgmap v288: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:27:45.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:27:45 smithi040 ceph-mon[27984]: pgmap v289: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:27:45.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:27:45 smithi174 ceph-mon[31409]: pgmap v289: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:27:45.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:27:45 smithi111 ceph-mon[31085]: pgmap v289: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:27:47.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:27:47 smithi040 ceph-mon[27984]: pgmap v290: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:27:47.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:27:47 smithi174 ceph-mon[31409]: pgmap v290: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:27:47.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:27:47 smithi111 ceph-mon[31085]: pgmap v290: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:27:49.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:27:49 smithi040 ceph-mon[27984]: pgmap v291: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:27:49.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:27:49 smithi174 ceph-mon[31409]: pgmap v291: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:27:49.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:27:49 smithi111 ceph-mon[31085]: pgmap v291: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:27:51.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:27:51 smithi040 ceph-mon[27984]: pgmap v292: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:27:51.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:27:51 smithi174 ceph-mon[31409]: pgmap v292: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:27:51.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:27:51 smithi111 ceph-mon[31085]: pgmap v292: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:27:53.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:27:53 smithi040 ceph-mon[27984]: pgmap v293: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:27:53.445 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:27:53 smithi174 ceph-mon[31409]: pgmap v293: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:27:53.446 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:27:53 smithi111 ceph-mon[31085]: pgmap v293: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:27:55.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:27:55 smithi040 ceph-mon[27984]: pgmap v294: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:27:55.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:27:55 smithi174 ceph-mon[31409]: pgmap v294: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:27:55.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:27:55 smithi111 ceph-mon[31085]: pgmap v294: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:27:57.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:27:57 smithi040 ceph-mon[27984]: pgmap v295: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:27:57.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:27:57 smithi174 ceph-mon[31409]: pgmap v295: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:27:57.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:27:57 smithi111 ceph-mon[31085]: pgmap v295: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:27:59.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:27:59 smithi040 ceph-mon[27984]: pgmap v296: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:27:59.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:27:59 smithi174 ceph-mon[31409]: pgmap v296: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:27:59.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:27:59 smithi111 ceph-mon[31085]: pgmap v296: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:28:01.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:28:01 smithi040 ceph-mon[27984]: pgmap v297: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:28:01.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:28:01 smithi174 ceph-mon[31409]: pgmap v297: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:28:01.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:28:01 smithi111 ceph-mon[31085]: pgmap v297: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:28:03.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:28:03 smithi040 ceph-mon[27984]: pgmap v298: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:28:03.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:28:03 smithi174 ceph-mon[31409]: pgmap v298: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:28:03.443 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:28:03 smithi111 ceph-mon[31085]: pgmap v298: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:28:05.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:28:05 smithi040 ceph-mon[27984]: pgmap v299: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:28:05.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:28:05 smithi174 ceph-mon[31409]: pgmap v299: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:28:05.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:28:05 smithi111 ceph-mon[31085]: pgmap v299: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:28:07.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:28:07 smithi040 ceph-mon[27984]: pgmap v300: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-21T23:28:07.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:28:07 smithi174 ceph-mon[31409]: pgmap v300: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-21T23:28:07.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:28:07 smithi111 ceph-mon[31085]: pgmap v300: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-21T23:28:09.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:28:09 smithi040 ceph-mon[27984]: pgmap v301: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-04-21T23:28:09.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:28:09 smithi174 ceph-mon[31409]: pgmap v301: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-04-21T23:28:09.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:28:09 smithi111 ceph-mon[31085]: pgmap v301: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-04-21T23:28:11.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:28:11 smithi040 ceph-mon[27984]: pgmap v302: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-21T23:28:11.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:28:11 smithi040 ceph-mon[27984]: osdmap e102: 12 total, 12 up, 12 in 2024-04-21T23:28:11.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:28:11 smithi174 ceph-mon[31409]: pgmap v302: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-21T23:28:11.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:28:11 smithi174 ceph-mon[31409]: osdmap e102: 12 total, 12 up, 12 in 2024-04-21T23:28:11.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:28:11 smithi111 ceph-mon[31085]: pgmap v302: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-21T23:28:11.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:28:11 smithi111 ceph-mon[31085]: osdmap e102: 12 total, 12 up, 12 in 2024-04-21T23:28:13.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:28:13 smithi040 ceph-mon[27984]: pgmap v304: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-21T23:28:13.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:28:13 smithi174 ceph-mon[31409]: pgmap v304: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-21T23:28:13.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:28:13 smithi111 ceph-mon[31085]: pgmap v304: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-21T23:28:15.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:28:15 smithi040 ceph-mon[27984]: pgmap v305: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-21T23:28:15.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:28:15 smithi174 ceph-mon[31409]: pgmap v305: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-21T23:28:15.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:28:15 smithi111 ceph-mon[31085]: pgmap v305: 129 pgs: 129 active+clean; 95 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-21T23:28:17.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:28:17 smithi040 ceph-mon[27984]: pgmap v306: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-21T23:28:17.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:28:17 smithi174 ceph-mon[31409]: pgmap v306: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-21T23:28:17.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:28:17 smithi111 ceph-mon[31085]: pgmap v306: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-21T23:28:19.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:28:19 smithi040 ceph-mon[27984]: pgmap v307: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-21T23:28:19.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:28:19 smithi040 ceph-mon[27984]: osdmap e103: 12 total, 12 up, 12 in 2024-04-21T23:28:19.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:28:19 smithi174 ceph-mon[31409]: pgmap v307: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-21T23:28:19.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:28:19 smithi174 ceph-mon[31409]: osdmap e103: 12 total, 12 up, 12 in 2024-04-21T23:28:19.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:28:19 smithi111 ceph-mon[31085]: pgmap v307: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-21T23:28:19.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:28:19 smithi111 ceph-mon[31085]: osdmap e103: 12 total, 12 up, 12 in 2024-04-21T23:28:21.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:28:21 smithi040 ceph-mon[27984]: pgmap v309: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 215 B/s wr, 6 op/s 2024-04-21T23:28:21.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:28:21 smithi174 ceph-mon[31409]: pgmap v309: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 215 B/s wr, 6 op/s 2024-04-21T23:28:21.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:28:21 smithi111 ceph-mon[31085]: pgmap v309: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 215 B/s wr, 6 op/s 2024-04-21T23:28:23.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:28:23 smithi040 ceph-mon[27984]: pgmap v310: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-21T23:28:23.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:28:23 smithi174 ceph-mon[31409]: pgmap v310: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-21T23:28:23.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:28:23 smithi111 ceph-mon[31085]: pgmap v310: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-21T23:28:25.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:28:25 smithi040 ceph-mon[27984]: pgmap v311: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-21T23:28:25.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:28:25 smithi174 ceph-mon[31409]: pgmap v311: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-21T23:28:25.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:28:25 smithi111 ceph-mon[31085]: pgmap v311: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-21T23:28:27.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:28:27 smithi040 ceph-mon[27984]: pgmap v312: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-21T23:28:27.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:28:27 smithi174 ceph-mon[31409]: pgmap v312: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-21T23:28:27.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:28:27 smithi111 ceph-mon[31085]: pgmap v312: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-21T23:28:29.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:28:29 smithi040 ceph-mon[27984]: pgmap v313: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-21T23:28:29.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:28:29 smithi174 ceph-mon[31409]: pgmap v313: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-21T23:28:29.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:28:29 smithi111 ceph-mon[31085]: pgmap v313: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-21T23:28:31.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:28:31 smithi040 ceph-mon[27984]: pgmap v314: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 88 B/s wr, 7 op/s 2024-04-21T23:28:31.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:28:31 smithi174 ceph-mon[31409]: pgmap v314: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 88 B/s wr, 7 op/s 2024-04-21T23:28:31.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:28:31 smithi111 ceph-mon[31085]: pgmap v314: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 88 B/s wr, 7 op/s 2024-04-21T23:28:33.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:28:33 smithi040 ceph-mon[27984]: pgmap v315: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:28:33.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:28:33 smithi174 ceph-mon[31409]: pgmap v315: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:28:33.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:28:33 smithi111 ceph-mon[31085]: pgmap v315: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:28:33.615 DEBUG:teuthology.orchestra.run.smithi040:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-04-21T23:28:33.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:33.731+0000 7f52c0f85640 1 -- 172.21.15.40:0/3078682045 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f52bc101740 msgr2=0x7f52bc101bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:28:33.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:33.731+0000 7f52c0f85640 1 --2- 172.21.15.40:0/3078682045 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f52bc101740 0x7f52bc101bc0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f52b0009930 tx=0x7f52b002bc30 comp rx=0 tx=0).stop 2024-04-21T23:28:33.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:33.731+0000 7f52c0f85640 1 -- 172.21.15.40:0/3078682045 shutdown_connections 2024-04-21T23:28:33.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:33.731+0000 7f52c0f85640 1 --2- 172.21.15.40:0/3078682045 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52bc102100 0x7f52bc10e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:28:33.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:33.731+0000 7f52c0f85640 1 --2- 172.21.15.40:0/3078682045 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f52bc101740 0x7f52bc101bc0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:28:33.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:33.731+0000 7f52c0f85640 1 --2- 172.21.15.40:0/3078682045 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52bc107750 0x7f52bc107b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:28:33.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:33.731+0000 7f52c0f85640 1 -- 172.21.15.40:0/3078682045 >> 172.21.15.40:0/3078682045 conn(0x7f52bc0fd210 msgr2=0x7f52bc0ff630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:28:33.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:33.732+0000 7f52c0f85640 1 -- 172.21.15.40:0/3078682045 shutdown_connections 2024-04-21T23:28:33.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:33.732+0000 7f52c0f85640 1 -- 172.21.15.40:0/3078682045 wait complete. 2024-04-21T23:28:33.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:33.732+0000 7f52c0f85640 1 Processor -- start 2024-04-21T23:28:33.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:33.732+0000 7f52c0f85640 1 -- start start 2024-04-21T23:28:33.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:33.733+0000 7f52c0f85640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f52bc101740 0x7f52bc1980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:28:33.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:33.733+0000 7f52c0f85640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52bc102100 0x7f52bc1985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:28:33.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:33.733+0000 7f52c0f85640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52bc107750 0x7f52bc19c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:28:33.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:33.733+0000 7f52c0f85640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f52bc111b60 con 0x7f52bc101740 2024-04-21T23:28:33.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:33.733+0000 7f52c0f85640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f52bc1119e0 con 0x7f52bc107750 2024-04-21T23:28:33.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:33.733+0000 7f52c0f85640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f52bc111ce0 con 0x7f52bc102100 2024-04-21T23:28:33.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:33.733+0000 7f52ba59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f52bc101740 0x7f52bc1980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:28:33.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:33.733+0000 7f52ba59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f52bc101740 0x7f52bc1980a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58598/0 (socket says 172.21.15.40:58598) 2024-04-21T23:28:33.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:33.733+0000 7f52ba59c640 1 -- 172.21.15.40:0/604458276 learned_addr learned my addr 172.21.15.40:0/604458276 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:28:33.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:33.733+0000 7f52bad9d640 1 --2- 172.21.15.40:0/604458276 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52bc107750 0x7f52bc19c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:28:33.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:33.733+0000 7f52ba59c640 1 -- 172.21.15.40:0/604458276 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52bc102100 msgr2=0x7f52bc1985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:28:33.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:33.733+0000 7f52b9d9b640 1 --2- 172.21.15.40:0/604458276 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52bc102100 0x7f52bc1985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:28:33.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:33.733+0000 7f52ba59c640 1 --2- 172.21.15.40:0/604458276 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52bc102100 0x7f52bc1985e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:28:33.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:33.733+0000 7f52ba59c640 1 -- 172.21.15.40:0/604458276 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52bc107750 msgr2=0x7f52bc19c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:28:33.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:33.733+0000 7f52ba59c640 1 --2- 172.21.15.40:0/604458276 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52bc107750 0x7f52bc19c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:28:33.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:33.733+0000 7f52ba59c640 1 -- 172.21.15.40:0/604458276 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52bc19d040 con 0x7f52bc101740 2024-04-21T23:28:33.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:33.733+0000 7f52b9d9b640 1 --2- 172.21.15.40:0/604458276 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52bc102100 0x7f52bc1985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:28:33.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:33.733+0000 7f52bad9d640 1 --2- 172.21.15.40:0/604458276 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52bc107750 0x7f52bc19c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:28:33.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:33.733+0000 7f52ba59c640 1 --2- 172.21.15.40:0/604458276 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f52bc101740 0x7f52bc1980a0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f52ac00e9f0 tx=0x7f52ac00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:28:33.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:33.733+0000 7f52ab7fe640 1 -- 172.21.15.40:0/604458276 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52ac00cd80 con 0x7f52bc101740 2024-04-21T23:28:33.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:33.734+0000 7f52ab7fe640 1 -- 172.21.15.40:0/604458276 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f52ac005500 con 0x7f52bc101740 2024-04-21T23:28:33.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:33.734+0000 7f52c0f85640 1 -- 172.21.15.40:0/604458276 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52bc19d330 con 0x7f52bc101740 2024-04-21T23:28:33.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:33.734+0000 7f52ab7fe640 1 -- 172.21.15.40:0/604458276 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52ac010690 con 0x7f52bc101740 2024-04-21T23:28:33.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:33.734+0000 7f52c0f85640 1 -- 172.21.15.40:0/604458276 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52bc1a4bc0 con 0x7f52bc101740 2024-04-21T23:28:33.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:33.735+0000 7f52c0f85640 1 -- 172.21.15.40:0/604458276 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52bc04f500 con 0x7f52bc101740 2024-04-21T23:28:33.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:33.736+0000 7f52ab7fe640 1 -- 172.21.15.40:0/604458276 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f52ac0046c0 con 0x7f52bc101740 2024-04-21T23:28:33.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:33.736+0000 7f52ab7fe640 1 --2- 172.21.15.40:0/604458276 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f52a40a2da0 0x7f52a40a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:28:33.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:33.737+0000 7f52ab7fe640 1 -- 172.21.15.40:0/604458276 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9554+0+0 (secure 0 0 0) 0x7f52ac0cff00 con 0x7f52bc101740 2024-04-21T23:28:33.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:33.737+0000 7f52b9d9b640 1 --2- 172.21.15.40:0/604458276 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f52a40a2da0 0x7f52a40a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:28:33.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:33.737+0000 7f52b9d9b640 1 --2- 172.21.15.40:0/604458276 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f52a40a2da0 0x7f52a40a5260 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f52b0036040 tx=0x7f52b0000bc0 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:28:33.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:33.740+0000 7f52ab7fe640 1 -- 172.21.15.40:0/604458276 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f52ac0536f0 con 0x7f52bc101740 2024-04-21T23:28:33.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:33.948+0000 7f52c0f85640 1 -- 172.21.15.40:0/604458276 --> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f52bc109d20 con 0x7f52a40a2da0 2024-04-21T23:28:33.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:33.957+0000 7f52ab7fe640 1 -- 172.21.15.40:0/604458276 <== mgr.25111 v2:172.21.15.40:6800/478611259 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+350 (secure 0 0 0) 0x7f52b002c980 con 0x7f52a40a2da0 2024-04-21T23:28:33.956 INFO:teuthology.orchestra.run.smithi040.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-04-21T00:00:00", "created": "2024-04-21T23:18:31", "first": "2024-04-21T23:19:00", "last": "2024-04-21T23:28:00", "last_pruned": "2024-04-21T23:28:00", "created_count": 10, "pruned_count": 4, "active": true} 2024-04-21T23:28:33.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:33.959+0000 7f52c0f85640 1 -- 172.21.15.40:0/604458276 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f52a40a2da0 msgr2=0x7f52a40a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:28:33.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:33.959+0000 7f52c0f85640 1 --2- 172.21.15.40:0/604458276 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f52a40a2da0 0x7f52a40a5260 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f52b0036040 tx=0x7f52b0000bc0 comp rx=0 tx=0).stop 2024-04-21T23:28:33.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:33.959+0000 7f52c0f85640 1 -- 172.21.15.40:0/604458276 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f52bc101740 msgr2=0x7f52bc1980a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:28:33.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:33.959+0000 7f52c0f85640 1 --2- 172.21.15.40:0/604458276 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f52bc101740 0x7f52bc1980a0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f52ac00e9f0 tx=0x7f52ac00edf0 comp rx=0 tx=0).stop 2024-04-21T23:28:33.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:33.959+0000 7f52c0f85640 1 -- 172.21.15.40:0/604458276 shutdown_connections 2024-04-21T23:28:33.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:33.959+0000 7f52c0f85640 1 --2- 172.21.15.40:0/604458276 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f52a40a2da0 0x7f52a40a5260 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:28:33.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:33.959+0000 7f52c0f85640 1 --2- 172.21.15.40:0/604458276 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52bc107750 0x7f52bc19c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:28:33.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:33.959+0000 7f52c0f85640 1 --2- 172.21.15.40:0/604458276 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52bc102100 0x7f52bc1985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:28:33.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:33.959+0000 7f52c0f85640 1 --2- 172.21.15.40:0/604458276 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f52bc101740 0x7f52bc1980a0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:28:33.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:33.959+0000 7f52c0f85640 1 -- 172.21.15.40:0/604458276 >> 172.21.15.40:0/604458276 conn(0x7f52bc0fd210 msgr2=0x7f52bc0fd810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:28:33.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:33.959+0000 7f52c0f85640 1 -- 172.21.15.40:0/604458276 shutdown_connections 2024-04-21T23:28:33.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:33.959+0000 7f52c0f85640 1 -- 172.21.15.40:0/604458276 wait complete. 2024-04-21T23:28:33.968 DEBUG:teuthology.orchestra.run.smithi040:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-04-21T23:28:34.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.073+0000 7f4c5e042640 1 -- 172.21.15.40:0/246206751 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c5810e120 msgr2=0x7f4c58110510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:28:34.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.073+0000 7f4c5e042640 1 --2- 172.21.15.40:0/246206751 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c5810e120 0x7f4c58110510 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f4c50009ca0 tx=0x7f4c5002c0f0 comp rx=0 tx=0).stop 2024-04-21T23:28:34.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.073+0000 7f4c5e042640 1 -- 172.21.15.40:0/246206751 shutdown_connections 2024-04-21T23:28:34.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.073+0000 7f4c5e042640 1 --2- 172.21.15.40:0/246206751 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c5810e120 0x7f4c58110510 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:28:34.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.073+0000 7f4c5e042640 1 --2- 172.21.15.40:0/246206751 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4c58100e90 0x7f4c5810dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:28:34.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.073+0000 7f4c5e042640 1 --2- 172.21.15.40:0/246206751 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c58100570 0x7f4c58100950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:28:34.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.073+0000 7f4c5e042640 1 -- 172.21.15.40:0/246206751 >> 172.21.15.40:0/246206751 conn(0x7f4c580769c0 msgr2=0x7f4c58076dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:28:34.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.074+0000 7f4c5e042640 1 -- 172.21.15.40:0/246206751 shutdown_connections 2024-04-21T23:28:34.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.074+0000 7f4c5e042640 1 -- 172.21.15.40:0/246206751 wait complete. 2024-04-21T23:28:34.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.074+0000 7f4c5e042640 1 Processor -- start 2024-04-21T23:28:34.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.074+0000 7f4c5e042640 1 -- start start 2024-04-21T23:28:34.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.074+0000 7f4c5e042640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c58100570 0x7f4c5819a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:28:34.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.074+0000 7f4c5e042640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4c58100e90 0x7f4c5819a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:28:34.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.074+0000 7f4c5e042640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c5810e120 0x7f4c5819ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:28:34.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.075+0000 7f4c5e042640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4c58113d80 con 0x7f4c5810e120 2024-04-21T23:28:34.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.075+0000 7f4c5e042640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f4c58113c00 con 0x7f4c58100570 2024-04-21T23:28:34.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.075+0000 7f4c5e042640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4c58113f00 con 0x7f4c58100e90 2024-04-21T23:28:34.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.075+0000 7f4c57fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c5810e120 0x7f4c5819ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:28:34.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.075+0000 7f4c57fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c5810e120 0x7f4c5819ea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58624/0 (socket says 172.21.15.40:58624) 2024-04-21T23:28:34.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.075+0000 7f4c57fff640 1 -- 172.21.15.40:0/3214700036 learned_addr learned my addr 172.21.15.40:0/3214700036 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:28:34.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.075+0000 7f4c577fe640 1 --2- 172.21.15.40:0/3214700036 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c58100570 0x7f4c5819a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:28:34.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.075+0000 7f4c56ffd640 1 --2- 172.21.15.40:0/3214700036 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4c58100e90 0x7f4c5819a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:28:34.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.075+0000 7f4c57fff640 1 -- 172.21.15.40:0/3214700036 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4c58100e90 msgr2=0x7f4c5819a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:28:34.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.075+0000 7f4c57fff640 1 --2- 172.21.15.40:0/3214700036 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4c58100e90 0x7f4c5819a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:28:34.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.075+0000 7f4c57fff640 1 -- 172.21.15.40:0/3214700036 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c58100570 msgr2=0x7f4c5819a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:28:34.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.075+0000 7f4c57fff640 1 --2- 172.21.15.40:0/3214700036 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c58100570 0x7f4c5819a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:28:34.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.075+0000 7f4c57fff640 1 -- 172.21.15.40:0/3214700036 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c5819f190 con 0x7f4c5810e120 2024-04-21T23:28:34.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.075+0000 7f4c57fff640 1 --2- 172.21.15.40:0/3214700036 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c5810e120 0x7f4c5819ea60 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f4c50005cc0 tx=0x7f4c50032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:28:34.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.075+0000 7f4c54ff9640 1 -- 172.21.15.40:0/3214700036 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c50039040 con 0x7f4c5810e120 2024-04-21T23:28:34.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.075+0000 7f4c5e042640 1 -- 172.21.15.40:0/3214700036 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c5819f420 con 0x7f4c5810e120 2024-04-21T23:28:34.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.075+0000 7f4c54ff9640 1 -- 172.21.15.40:0/3214700036 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c50044070 con 0x7f4c5810e120 2024-04-21T23:28:34.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.075+0000 7f4c54ff9640 1 -- 172.21.15.40:0/3214700036 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c500042a0 con 0x7f4c5810e120 2024-04-21T23:28:34.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.075+0000 7f4c5e042640 1 -- 172.21.15.40:0/3214700036 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c581a6d10 con 0x7f4c5810e120 2024-04-21T23:28:34.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.076+0000 7f4c5e042640 1 -- 172.21.15.40:0/3214700036 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c5804f500 con 0x7f4c5810e120 2024-04-21T23:28:34.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.078+0000 7f4c54ff9640 1 -- 172.21.15.40:0/3214700036 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f4c50004440 con 0x7f4c5810e120 2024-04-21T23:28:34.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.078+0000 7f4c54ff9640 1 --2- 172.21.15.40:0/3214700036 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f4c400a2da0 0x7f4c400a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:28:34.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.079+0000 7f4c577fe640 1 --2- 172.21.15.40:0/3214700036 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f4c400a2da0 0x7f4c400a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:28:34.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.079+0000 7f4c54ff9640 1 -- 172.21.15.40:0/3214700036 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9554+0+0 (secure 0 0 0) 0x7f4c500f19c0 con 0x7f4c5810e120 2024-04-21T23:28:34.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.079+0000 7f4c577fe640 1 --2- 172.21.15.40:0/3214700036 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f4c400a2da0 0x7f4c400a5260 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c00bbe0 tx=0x7f4c4c00b390 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:28:34.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.083+0000 7f4c54ff9640 1 -- 172.21.15.40:0/3214700036 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f4c500f3e20 con 0x7f4c5810e120 2024-04-21T23:28:34.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.290+0000 7f4c5e042640 1 -- 172.21.15.40:0/3214700036 --> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f4c5810fbe0 con 0x7f4c400a2da0 2024-04-21T23:28:34.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.298+0000 7f4c54ff9640 1 -- 172.21.15.40:0/3214700036 <== mgr.25111 v2:172.21.15.40:6800/478611259 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f4c5810fbe0 con 0x7f4c400a2da0 2024-04-21T23:28:34.297 INFO:teuthology.orchestra.run.smithi040.stdout:/ 1m 6m3h 2024-04-21T23:28:34.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.300+0000 7f4c5e042640 1 -- 172.21.15.40:0/3214700036 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f4c400a2da0 msgr2=0x7f4c400a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:28:34.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.300+0000 7f4c5e042640 1 --2- 172.21.15.40:0/3214700036 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f4c400a2da0 0x7f4c400a5260 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c00bbe0 tx=0x7f4c4c00b390 comp rx=0 tx=0).stop 2024-04-21T23:28:34.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.300+0000 7f4c5e042640 1 -- 172.21.15.40:0/3214700036 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c5810e120 msgr2=0x7f4c5819ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:28:34.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.300+0000 7f4c5e042640 1 --2- 172.21.15.40:0/3214700036 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c5810e120 0x7f4c5819ea60 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f4c50005cc0 tx=0x7f4c50032f40 comp rx=0 tx=0).stop 2024-04-21T23:28:34.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.300+0000 7f4c5e042640 1 -- 172.21.15.40:0/3214700036 shutdown_connections 2024-04-21T23:28:34.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.300+0000 7f4c5e042640 1 --2- 172.21.15.40:0/3214700036 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f4c400a2da0 0x7f4c400a5260 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:28:34.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.300+0000 7f4c5e042640 1 --2- 172.21.15.40:0/3214700036 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c5810e120 0x7f4c5819ea60 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:28:34.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.300+0000 7f4c5e042640 1 --2- 172.21.15.40:0/3214700036 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4c58100e90 0x7f4c5819a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:28:34.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.300+0000 7f4c5e042640 1 --2- 172.21.15.40:0/3214700036 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c58100570 0x7f4c5819a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:28:34.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.300+0000 7f4c5e042640 1 -- 172.21.15.40:0/3214700036 >> 172.21.15.40:0/3214700036 conn(0x7f4c580769c0 msgr2=0x7f4c5810f4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:28:34.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.300+0000 7f4c5e042640 1 -- 172.21.15.40:0/3214700036 shutdown_connections 2024-04-21T23:28:34.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.300+0000 7f4c5e042640 1 -- 172.21.15.40:0/3214700036 wait complete. 2024-04-21T23:28:34.310 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-04-21T23:28:34.321 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-04-21T23:28:34.322 ERROR:teuthology.run_tasks:Manager failed: fwd_scrub Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_430e09df97c8fc7dc2b2ae424f68ed11366c540f/qa/tasks/fwd_scrub.py", line 164, in task stop_all_fwd_scrubbers(ctx.ceph[config['cluster']].thrashers) File "/home/teuthworker/src/git.ceph.com_ceph_430e09df97c8fc7dc2b2ae424f68ed11366c540f/qa/tasks/fwd_scrub.py", line 99, in stop_all_fwd_scrubbers raise RuntimeError(f"error during scrub thrashing: {thrasher.exception}") RuntimeError: error during scrub thrashing: rank damage found: {'backtrace'} 2024-04-21T23:28:34.327 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-04-21T23:28:34.352 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T23:28:34.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.459+0000 7f0b1f93a640 1 -- 172.21.15.40:0/1442536422 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b181020f0 msgr2=0x7f0b1810e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:28:34.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.459+0000 7f0b1f93a640 1 --2- 172.21.15.40:0/1442536422 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b181020f0 0x7f0b1810e600 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f0b10009cd0 tx=0x7f0b1002c0f0 comp rx=0 tx=0).stop 2024-04-21T23:28:34.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.459+0000 7f0b1f93a640 1 -- 172.21.15.40:0/1442536422 shutdown_connections 2024-04-21T23:28:34.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.459+0000 7f0b1f93a640 1 --2- 172.21.15.40:0/1442536422 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b18101730 0x7f0b18101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:28:34.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.459+0000 7f0b1f93a640 1 --2- 172.21.15.40:0/1442536422 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b181020f0 0x7f0b1810e600 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:28:34.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.459+0000 7f0b1f93a640 1 --2- 172.21.15.40:0/1442536422 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0b18107740 0x7f0b18107b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:28:34.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.459+0000 7f0b1f93a640 1 -- 172.21.15.40:0/1442536422 >> 172.21.15.40:0/1442536422 conn(0x7f0b180fd1e0 msgr2=0x7f0b180ff600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:28:34.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.459+0000 7f0b1f93a640 1 -- 172.21.15.40:0/1442536422 shutdown_connections 2024-04-21T23:28:34.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.459+0000 7f0b1f93a640 1 -- 172.21.15.40:0/1442536422 wait complete. 2024-04-21T23:28:34.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.460+0000 7f0b1f93a640 1 Processor -- start 2024-04-21T23:28:34.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.460+0000 7f0b1f93a640 1 -- start start 2024-04-21T23:28:34.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.460+0000 7f0b1f93a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0b18101730 0x7f0b18197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:28:34.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.460+0000 7f0b1f93a640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b181020f0 0x7f0b181984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:28:34.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.460+0000 7f0b1f93a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b18107740 0x7f0b1819c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:28:34.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.460+0000 7f0b1f93a640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0b18111b50 con 0x7f0b18107740 2024-04-21T23:28:34.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.460+0000 7f0b1f93a640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f0b181119d0 con 0x7f0b181020f0 2024-04-21T23:28:34.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.460+0000 7f0b1f93a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0b18111cd0 con 0x7f0b18101730 2024-04-21T23:28:34.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.460+0000 7f0b1ded7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b18107740 0x7f0b1819c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:28:34.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.460+0000 7f0b1ded7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b18107740 0x7f0b1819c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58644/0 (socket says 172.21.15.40:58644) 2024-04-21T23:28:34.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.460+0000 7f0b1ded7640 1 -- 172.21.15.40:0/360294288 learned_addr learned my addr 172.21.15.40:0/360294288 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:28:34.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.460+0000 7f0b1d6d6640 1 --2- 172.21.15.40:0/360294288 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0b18101730 0x7f0b18197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:28:34.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.460+0000 7f0b1ced5640 1 --2- 172.21.15.40:0/360294288 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b181020f0 0x7f0b181984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:28:34.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.460+0000 7f0b1ded7640 1 -- 172.21.15.40:0/360294288 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0b18101730 msgr2=0x7f0b18197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:28:34.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.460+0000 7f0b1ded7640 1 --2- 172.21.15.40:0/360294288 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0b18101730 0x7f0b18197f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:28:34.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.460+0000 7f0b1ded7640 1 -- 172.21.15.40:0/360294288 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b181020f0 msgr2=0x7f0b181984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:28:34.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.460+0000 7f0b1ded7640 1 --2- 172.21.15.40:0/360294288 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b181020f0 0x7f0b181984d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:28:34.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.460+0000 7f0b1ded7640 1 -- 172.21.15.40:0/360294288 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b1819cf80 con 0x7f0b18107740 2024-04-21T23:28:34.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.460+0000 7f0b1ced5640 1 --2- 172.21.15.40:0/360294288 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b181020f0 0x7f0b181984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:28:34.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.461+0000 7f0b1ded7640 1 --2- 172.21.15.40:0/360294288 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b18107740 0x7f0b1819c850 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f0b100348f0 tx=0x7f0b10032ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:28:34.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.461+0000 7f0b0a7fc640 1 -- 172.21.15.40:0/360294288 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b10041020 con 0x7f0b18107740 2024-04-21T23:28:34.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.461+0000 7f0b0a7fc640 1 -- 172.21.15.40:0/360294288 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b10034cd0 con 0x7f0b18107740 2024-04-21T23:28:34.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.461+0000 7f0b1f93a640 1 -- 172.21.15.40:0/360294288 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b1819d210 con 0x7f0b18107740 2024-04-21T23:28:34.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.461+0000 7f0b0a7fc640 1 -- 172.21.15.40:0/360294288 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b10039040 con 0x7f0b18107740 2024-04-21T23:28:34.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.461+0000 7f0b1f93a640 1 -- 172.21.15.40:0/360294288 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b181a4b00 con 0x7f0b18107740 2024-04-21T23:28:34.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.462+0000 7f0b1f93a640 1 -- 172.21.15.40:0/360294288 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b18068560 con 0x7f0b18107740 2024-04-21T23:28:34.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.463+0000 7f0b0a7fc640 1 -- 172.21.15.40:0/360294288 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f0b10044020 con 0x7f0b18107740 2024-04-21T23:28:34.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.464+0000 7f0b0a7fc640 1 --2- 172.21.15.40:0/360294288 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f0b040a2d00 0x7f0b040a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:28:34.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.464+0000 7f0b0a7fc640 1 -- 172.21.15.40:0/360294288 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9554+0+0 (secure 0 0 0) 0x7f0b100f1a30 con 0x7f0b18107740 2024-04-21T23:28:34.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.464+0000 7f0b1d6d6640 1 --2- 172.21.15.40:0/360294288 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f0b040a2d00 0x7f0b040a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:28:34.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.465+0000 7f0b1d6d6640 1 --2- 172.21.15.40:0/360294288 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f0b040a2d00 0x7f0b040a51c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f0b14005fd0 tx=0x7f0b1400b390 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:28:34.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.468+0000 7f0b0a7fc640 1 -- 172.21.15.40:0/360294288 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f0b1004b020 con 0x7f0b18107740 2024-04-21T23:28:34.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.655+0000 7f0b1f93a640 1 -- 172.21.15.40:0/360294288 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0b181a4d60 con 0x7f0b18107740 2024-04-21T23:28:34.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.657+0000 7f0b0a7fc640 1 -- 172.21.15.40:0/360294288 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v103) ==== 74+0+25417 (secure 0 0 0) 0x7f0b100be470 con 0x7f0b18107740 2024-04-21T23:28:34.656 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:28:34.656 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":103,"fsid":"d814e4dc-0033-11ef-bc93-c7b262605968","created":"2024-04-21T23:07:52.659207+0000","modified":"2024-04-21T23:28:18.013843+0000","last_up_change":"2024-04-21T23:15:05.308821+0000","last_in_change":"2024-04-21T23:14:45.447491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:11:14.824775+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T23:15:54.322571+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T23:15:55.164493+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":102,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T23:15:58.838266+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":102,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"0e8d692a-22fe-4a87-8d99-9151f96cbb02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6803","nonce":2009758784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6805","nonce":2009758784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6809","nonce":2009758784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6807","nonce":2009758784}]},"public_addr":"172.21.15.40:6803/2009758784","cluster_addr":"172.21.15.40:6805/2009758784","heartbeat_back_addr":"172.21.15.40:6809/2009758784","heartbeat_front_addr":"172.21.15.40:6807/2009758784","state":["exists","up"]},{"osd":1,"uuid":"fe5fe11b-0e84-4359-83a7-1ba45105755a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6801","nonce":1763903836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6803","nonce":1763903836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6807","nonce":1763903836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6805","nonce":1763903836}]},"public_addr":"172.21.15.111:6801/1763903836","cluster_addr":"172.21.15.111:6803/1763903836","heartbeat_back_addr":"172.21.15.111:6807/1763903836","heartbeat_front_addr":"172.21.15.111:6805/1763903836","state":["exists","up"]},{"osd":2,"uuid":"813ad735-5cc2-4da0-ae16-6db238395fbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6801","nonce":439695825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6803","nonce":439695825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6807","nonce":439695825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6805","nonce":439695825}]},"public_addr":"172.21.15.174:6801/439695825","cluster_addr":"172.21.15.174:6803/439695825","heartbeat_back_addr":"172.21.15.174:6807/439695825","heartbeat_front_addr":"172.21.15.174:6805/439695825","state":["exists","up"]},{"osd":3,"uuid":"09becb03-b7ce-4ffc-917d-b7900ec37e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6811","nonce":3454312436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6813","nonce":3454312436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6817","nonce":3454312436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6815","nonce":3454312436}]},"public_addr":"172.21.15.40:6811/3454312436","cluster_addr":"172.21.15.40:6813/3454312436","heartbeat_back_addr":"172.21.15.40:6817/3454312436","heartbeat_front_addr":"172.21.15.40:6815/3454312436","state":["exists","up"]},{"osd":4,"uuid":"46eabfd7-d88a-4efd-94eb-04354d3a9d7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6809","nonce":1810866070}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6811","nonce":1810866070}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6815","nonce":1810866070}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6813","nonce":1810866070}]},"public_addr":"172.21.15.111:6809/1810866070","cluster_addr":"172.21.15.111:6811/1810866070","heartbeat_back_addr":"172.21.15.111:6815/1810866070","heartbeat_front_addr":"172.21.15.111:6813/1810866070","state":["exists","up"]},{"osd":5,"uuid":"6f73b582-6c65-41db-aab0-9b360f83d5c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6809","nonce":3752986711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6811","nonce":3752986711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6815","nonce":3752986711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6813","nonce":3752986711}]},"public_addr":"172.21.15.174:6809/3752986711","cluster_addr":"172.21.15.174:6811/3752986711","heartbeat_back_addr":"172.21.15.174:6815/3752986711","heartbeat_front_addr":"172.21.15.174:6813/3752986711","state":["exists","up"]},{"osd":6,"uuid":"39f305f0-12d2-4732-aaa0-cd66700af538","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6819","nonce":1618855427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6821","nonce":1618855427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6825","nonce":1618855427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6823","nonce":1618855427}]},"public_addr":"172.21.15.40:6819/1618855427","cluster_addr":"172.21.15.40:6821/1618855427","heartbeat_back_addr":"172.21.15.40:6825/1618855427","heartbeat_front_addr":"172.21.15.40:6823/1618855427","state":["exists","up"]},{"osd":7,"uuid":"769f2468-5102-490b-89f8-d0444d3169a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6817","nonce":927054162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6819","nonce":927054162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6823","nonce":927054162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6821","nonce":927054162}]},"public_addr":"172.21.15.111:6817/927054162","cluster_addr":"172.21.15.111:6819/927054162","heartbeat_back_addr":"172.21.15.111:6823/927054162","heartbeat_front_addr":"172.21.15.111:6821/927054162","state":["exists","up"]},{"osd":8,"uuid":"1a444e80-007d-473b-b972-4c814a813c22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6817","nonce":3817848746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6819","nonce":3817848746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6823","nonce":3817848746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6821","nonce":3817848746}]},"public_addr":"172.21.15.174:6817/3817848746","cluster_addr":"172.21.15.174:6819/3817848746","heartbeat_back_addr":"172.21.15.174:6823/3817848746","heartbeat_front_addr":"172.21.15.174:6821/3817848746","state":["exists","up"]},{"osd":9,"uuid":"9a6b6a56-6ae5-49fe-9e5b-cb17d686e0dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6827","nonce":1649446531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6829","nonce":1649446531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6833","nonce":1649446531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6831","nonce":1649446531}]},"public_addr":"172.21.15.40:6827/1649446531","cluster_addr":"172.21.15.40:6829/1649446531","heartbeat_back_addr":"172.21.15.40:6833/1649446531","heartbeat_front_addr":"172.21.15.40:6831/1649446531","state":["exists","up"]},{"osd":10,"uuid":"a70038ba-e2ad-4170-85cc-ea0470a2edef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6825","nonce":1227538935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6827","nonce":1227538935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6831","nonce":1227538935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6829","nonce":1227538935}]},"public_addr":"172.21.15.111:6825/1227538935","cluster_addr":"172.21.15.111:6827/1227538935","heartbeat_back_addr":"172.21.15.111:6831/1227538935","heartbeat_front_addr":"172.21.15.111:6829/1227538935","state":["exists","up"]},{"osd":11,"uuid":"781532bb-7a5e-44e3-8eb7-b584b0054666","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6825","nonce":2045865553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6827","nonce":2045865553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6831","nonce":2045865553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6829","nonce":2045865553}]},"public_addr":"172.21.15.174:6825/2045865553","cluster_addr":"172.21.15.174:6827/2045865553","heartbeat_back_addr":"172.21.15.174:6831/2045865553","heartbeat_front_addr":"172.21.15.174:6829/2045865553","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:10:24.761290+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:10:48.565428+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:11:12.307818+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:11:37.951529+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:02.832523+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:27.568879+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:52.997707+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:13:18.937274+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:13:44.708657+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:14:10.833855+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:14:37.078272+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:15:03.878076+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":1}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.2a","mappings":[{"from":3,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"4.9","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":10}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:6801/2500760788":"2024-04-22T23:18:23.389674+0000","172.21.15.40:6800/2500760788":"2024-04-22T23:18:23.389674+0000","172.21.15.40:0/2742002992":"2024-04-22T23:18:23.389674+0000","172.21.15.40:0/2494672676":"2024-04-22T23:18:23.389674+0000","172.21.15.40:6801/526554319":"2024-04-22T23:08:50.305770+0000","172.21.15.40:6800/526554319":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/3178763471":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/4045911364":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/276300080":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/3689649814":"2024-04-22T23:08:23.743030+0000","172.21.15.40:0/2241116715":"2024-04-22T23:08:23.743030+0000","172.21.15.40:0/1406242364":"2024-04-22T23:18:23.389674+0000","172.21.15.40:0/1737227818":"2024-04-22T23:18:23.389674+0000","172.21.15.40:6801/1479040326":"2024-04-22T23:08:23.743030+0000","172.21.15.40:6800/1479040326":"2024-04-22T23:08:23.743030+0000","172.21.15.40:0/681298099":"2024-04-22T23:08:23.743030+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]},{"pool":4,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T23:28:34.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.659+0000 7f0b1f93a640 1 -- 172.21.15.40:0/360294288 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f0b040a2d00 msgr2=0x7f0b040a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:28:34.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.659+0000 7f0b1f93a640 1 --2- 172.21.15.40:0/360294288 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f0b040a2d00 0x7f0b040a51c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f0b14005fd0 tx=0x7f0b1400b390 comp rx=0 tx=0).stop 2024-04-21T23:28:34.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.659+0000 7f0b1f93a640 1 -- 172.21.15.40:0/360294288 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b18107740 msgr2=0x7f0b1819c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:28:34.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.659+0000 7f0b1f93a640 1 --2- 172.21.15.40:0/360294288 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b18107740 0x7f0b1819c850 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f0b100348f0 tx=0x7f0b10032ec0 comp rx=0 tx=0).stop 2024-04-21T23:28:34.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.659+0000 7f0b1f93a640 1 -- 172.21.15.40:0/360294288 shutdown_connections 2024-04-21T23:28:34.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.659+0000 7f0b1f93a640 1 --2- 172.21.15.40:0/360294288 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f0b040a2d00 0x7f0b040a51c0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:28:34.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.659+0000 7f0b1f93a640 1 --2- 172.21.15.40:0/360294288 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b181020f0 0x7f0b181984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:28:34.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.659+0000 7f0b1f93a640 1 --2- 172.21.15.40:0/360294288 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b18107740 0x7f0b1819c850 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:28:34.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.659+0000 7f0b1f93a640 1 --2- 172.21.15.40:0/360294288 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0b18101730 0x7f0b18197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:28:34.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.659+0000 7f0b1f93a640 1 -- 172.21.15.40:0/360294288 >> 172.21.15.40:0/360294288 conn(0x7f0b180fd1e0 msgr2=0x7f0b180fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:28:34.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.659+0000 7f0b1f93a640 1 -- 172.21.15.40:0/360294288 shutdown_connections 2024-04-21T23:28:34.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.659+0000 7f0b1f93a640 1 -- 172.21.15.40:0/360294288 wait complete. 2024-04-21T23:28:34.669 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T23:11:14.824775+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T23:15:54.322571+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T23:15:55.164493+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '102', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 102, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T23:15:58.838266+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '102', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 102, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T23:28:34.670 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T23:28:34.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.776+0000 7fe8f2408640 1 -- 172.21.15.40:0/2995284980 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8ec100990 msgr2=0x7fe8ec100df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:28:34.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.776+0000 7fe8f2408640 1 --2- 172.21.15.40:0/2995284980 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8ec100990 0x7fe8ec100df0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fe8e0009900 tx=0x7fe8e002bc30 comp rx=0 tx=0).stop 2024-04-21T23:28:34.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.776+0000 7fe8f2408640 1 -- 172.21.15.40:0/2995284980 shutdown_connections 2024-04-21T23:28:34.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.776+0000 7fe8f2408640 1 --2- 172.21.15.40:0/2995284980 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe8ec101330 0x7fe8ec10d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:28:34.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.776+0000 7fe8f2408640 1 --2- 172.21.15.40:0/2995284980 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8ec100990 0x7fe8ec100df0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:28:34.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.776+0000 7fe8f2408640 1 --2- 172.21.15.40:0/2995284980 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8ec106690 0x7fe8ec106a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:28:34.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.776+0000 7fe8f2408640 1 -- 172.21.15.40:0/2995284980 >> 172.21.15.40:0/2995284980 conn(0x7fe8ec076440 msgr2=0x7fe8ec076850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:28:34.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.776+0000 7fe8f2408640 1 -- 172.21.15.40:0/2995284980 shutdown_connections 2024-04-21T23:28:34.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.776+0000 7fe8f2408640 1 -- 172.21.15.40:0/2995284980 wait complete. 2024-04-21T23:28:34.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.777+0000 7fe8f2408640 1 Processor -- start 2024-04-21T23:28:34.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.777+0000 7fe8f2408640 1 -- start start 2024-04-21T23:28:34.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.777+0000 7fe8f2408640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe8ec100990 0x7fe8ec198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:28:34.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.777+0000 7fe8f2408640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8ec101330 0x7fe8ec198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:28:34.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.777+0000 7fe8f2408640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8ec106690 0x7fe8ec19c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:28:34.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.777+0000 7fe8f2408640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe8ec110a20 con 0x7fe8ec106690 2024-04-21T23:28:34.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.777+0000 7fe8f2408640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe8ec1108a0 con 0x7fe8ec101330 2024-04-21T23:28:34.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.777+0000 7fe8f2408640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe8ec110ba0 con 0x7fe8ec100990 2024-04-21T23:28:34.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.777+0000 7fe8f09a5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8ec106690 0x7fe8ec19c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:28:34.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.777+0000 7fe8f09a5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8ec106690 0x7fe8ec19c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58662/0 (socket says 172.21.15.40:58662) 2024-04-21T23:28:34.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.777+0000 7fe8f09a5640 1 -- 172.21.15.40:0/144278931 learned_addr learned my addr 172.21.15.40:0/144278931 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:28:34.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.777+0000 7fe8ebfff640 1 --2- 172.21.15.40:0/144278931 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe8ec100990 0x7fe8ec198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:28:34.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.777+0000 7fe8f09a5640 1 -- 172.21.15.40:0/144278931 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe8ec100990 msgr2=0x7fe8ec198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:28:34.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.778+0000 7fe8f09a5640 1 --2- 172.21.15.40:0/144278931 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe8ec100990 0x7fe8ec198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:28:34.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.778+0000 7fe8eb7fe640 1 --2- 172.21.15.40:0/144278931 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8ec101330 0x7fe8ec198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:28:34.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.778+0000 7fe8f09a5640 1 -- 172.21.15.40:0/144278931 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8ec101330 msgr2=0x7fe8ec198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:28:34.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.778+0000 7fe8f09a5640 1 --2- 172.21.15.40:0/144278931 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8ec101330 0x7fe8ec198590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:28:34.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.778+0000 7fe8f09a5640 1 -- 172.21.15.40:0/144278931 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8ec19d040 con 0x7fe8ec106690 2024-04-21T23:28:34.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.778+0000 7fe8f09a5640 1 --2- 172.21.15.40:0/144278931 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8ec106690 0x7fe8ec19c910 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fe8e400eaa0 tx=0x7fe8e400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:28:34.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.778+0000 7fe8e97fa640 1 -- 172.21.15.40:0/144278931 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8e400cdb0 con 0x7fe8ec106690 2024-04-21T23:28:34.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.778+0000 7fe8f2408640 1 -- 172.21.15.40:0/144278931 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8ec19d330 con 0x7fe8ec106690 2024-04-21T23:28:34.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.778+0000 7fe8e97fa640 1 -- 172.21.15.40:0/144278931 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe8e4005020 con 0x7fe8ec106690 2024-04-21T23:28:34.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.778+0000 7fe8e97fa640 1 -- 172.21.15.40:0/144278931 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8e40116b0 con 0x7fe8ec106690 2024-04-21T23:28:34.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.778+0000 7fe8f2408640 1 -- 172.21.15.40:0/144278931 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8ec1a4bc0 con 0x7fe8ec106690 2024-04-21T23:28:34.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.779+0000 7fe8f2408640 1 -- 172.21.15.40:0/144278931 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8ec101d80 con 0x7fe8ec106690 2024-04-21T23:28:34.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.780+0000 7fe8e97fa640 1 -- 172.21.15.40:0/144278931 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fe8e4005560 con 0x7fe8ec106690 2024-04-21T23:28:34.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.781+0000 7fe8e97fa640 1 --2- 172.21.15.40:0/144278931 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fe8d40a2d00 0x7fe8d40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:28:34.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.781+0000 7fe8ebfff640 1 --2- 172.21.15.40:0/144278931 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fe8d40a2d00 0x7fe8d40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:28:34.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.781+0000 7fe8e97fa640 1 -- 172.21.15.40:0/144278931 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9554+0+0 (secure 0 0 0) 0x7fe8e40cfb60 con 0x7fe8ec106690 2024-04-21T23:28:34.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.781+0000 7fe8ebfff640 1 --2- 172.21.15.40:0/144278931 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fe8d40a2d00 0x7fe8d40a51c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fe8dc009890 tx=0x7fe8dc008040 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:28:34.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.785+0000 7fe8e97fa640 1 -- 172.21.15.40:0/144278931 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fe8e4016030 con 0x7fe8ec106690 2024-04-21T23:28:34.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.968+0000 7fe8f2408640 1 -- 172.21.15.40:0/144278931 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe8ec1993a0 con 0x7fe8ec106690 2024-04-21T23:28:34.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.968+0000 7fe8e97fa640 1 -- 172.21.15.40:0/144278931 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v103) ==== 93+0+10 (secure 0 0 0) 0x7fe8e4086070 con 0x7fe8ec106690 2024-04-21T23:28:34.968 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-04-21T23:28:34.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.971+0000 7fe8f2408640 1 -- 172.21.15.40:0/144278931 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fe8d40a2d00 msgr2=0x7fe8d40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:28:34.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.971+0000 7fe8f2408640 1 --2- 172.21.15.40:0/144278931 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fe8d40a2d00 0x7fe8d40a51c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fe8dc009890 tx=0x7fe8dc008040 comp rx=0 tx=0).stop 2024-04-21T23:28:34.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.971+0000 7fe8f2408640 1 -- 172.21.15.40:0/144278931 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8ec106690 msgr2=0x7fe8ec19c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:28:34.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.971+0000 7fe8f2408640 1 --2- 172.21.15.40:0/144278931 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8ec106690 0x7fe8ec19c910 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fe8e400eaa0 tx=0x7fe8e400ef60 comp rx=0 tx=0).stop 2024-04-21T23:28:34.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.971+0000 7fe8f2408640 1 -- 172.21.15.40:0/144278931 shutdown_connections 2024-04-21T23:28:34.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.971+0000 7fe8f2408640 1 --2- 172.21.15.40:0/144278931 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fe8d40a2d00 0x7fe8d40a51c0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:28:34.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.971+0000 7fe8f2408640 1 --2- 172.21.15.40:0/144278931 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8ec101330 0x7fe8ec198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:28:34.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.971+0000 7fe8f2408640 1 --2- 172.21.15.40:0/144278931 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8ec106690 0x7fe8ec19c910 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:28:34.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.971+0000 7fe8f2408640 1 --2- 172.21.15.40:0/144278931 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe8ec100990 0x7fe8ec198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:28:34.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.971+0000 7fe8f2408640 1 -- 172.21.15.40:0/144278931 >> 172.21.15.40:0/144278931 conn(0x7fe8ec076440 msgr2=0x7fe8ec06a020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:28:34.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.971+0000 7fe8f2408640 1 -- 172.21.15.40:0/144278931 shutdown_connections 2024-04-21T23:28:34.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:34.971+0000 7fe8f2408640 1 -- 172.21.15.40:0/144278931 wait complete. 2024-04-21T23:28:34.980 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T23:28:35.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.087+0000 7fa77cbec640 1 -- 172.21.15.40:0/2287471400 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7781020d0 msgr2=0x7fa77810e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:28:35.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.087+0000 7fa77cbec640 1 --2- 172.21.15.40:0/2287471400 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7781020d0 0x7fa77810e5e0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fa770009cd0 tx=0x7fa77002c0f0 comp rx=0 tx=0).stop 2024-04-21T23:28:35.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.087+0000 7fa77cbec640 1 -- 172.21.15.40:0/2287471400 shutdown_connections 2024-04-21T23:28:35.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.087+0000 7fa77cbec640 1 --2- 172.21.15.40:0/2287471400 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa778101710 0x7fa778101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:28:35.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.087+0000 7fa77cbec640 1 --2- 172.21.15.40:0/2287471400 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7781020d0 0x7fa77810e5e0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:28:35.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.087+0000 7fa77cbec640 1 --2- 172.21.15.40:0/2287471400 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa778107720 0x7fa778107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:28:35.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.087+0000 7fa77cbec640 1 -- 172.21.15.40:0/2287471400 >> 172.21.15.40:0/2287471400 conn(0x7fa7780fd180 msgr2=0x7fa7780ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:28:35.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.087+0000 7fa77cbec640 1 -- 172.21.15.40:0/2287471400 shutdown_connections 2024-04-21T23:28:35.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.087+0000 7fa77cbec640 1 -- 172.21.15.40:0/2287471400 wait complete. 2024-04-21T23:28:35.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.088+0000 7fa77cbec640 1 Processor -- start 2024-04-21T23:28:35.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.088+0000 7fa77cbec640 1 -- start start 2024-04-21T23:28:35.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.088+0000 7fa77cbec640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa778101710 0x7fa778198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:28:35.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.088+0000 7fa77cbec640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7781020d0 0x7fa778198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:28:35.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.088+0000 7fa77cbec640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa778107720 0x7fa77819c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:28:35.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.088+0000 7fa77cbec640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa778111b30 con 0x7fa7781020d0 2024-04-21T23:28:35.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.088+0000 7fa77cbec640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fa7781119b0 con 0x7fa778101710 2024-04-21T23:28:35.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.088+0000 7fa77cbec640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa778111cb0 con 0x7fa778107720 2024-04-21T23:28:35.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.088+0000 7fa76ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7781020d0 0x7fa778198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:28:35.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.088+0000 7fa76ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7781020d0 0x7fa778198550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58678/0 (socket says 172.21.15.40:58678) 2024-04-21T23:28:35.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.088+0000 7fa76ffff640 1 -- 172.21.15.40:0/3667851487 learned_addr learned my addr 172.21.15.40:0/3667851487 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:28:35.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.088+0000 7fa77659c640 1 --2- 172.21.15.40:0/3667851487 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa778101710 0x7fa778198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:28:35.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.088+0000 7fa76ffff640 1 -- 172.21.15.40:0/3667851487 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa778107720 msgr2=0x7fa77819c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:28:35.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.088+0000 7fa776d9d640 1 --2- 172.21.15.40:0/3667851487 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa778107720 0x7fa77819c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:28:35.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.088+0000 7fa76ffff640 1 --2- 172.21.15.40:0/3667851487 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa778107720 0x7fa77819c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:28:35.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.088+0000 7fa76ffff640 1 -- 172.21.15.40:0/3667851487 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa778101710 msgr2=0x7fa778198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:28:35.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.088+0000 7fa76ffff640 1 --2- 172.21.15.40:0/3667851487 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa778101710 0x7fa778198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:28:35.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.088+0000 7fa76ffff640 1 -- 172.21.15.40:0/3667851487 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa77819d000 con 0x7fa7781020d0 2024-04-21T23:28:35.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.088+0000 7fa77659c640 1 --2- 172.21.15.40:0/3667851487 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa778101710 0x7fa778198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:28:35.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.088+0000 7fa76ffff640 1 --2- 172.21.15.40:0/3667851487 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7781020d0 0x7fa778198550 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fa76000d950 tx=0x7fa76000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:28:35.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.089+0000 7fa76f7fe640 1 -- 172.21.15.40:0/3667851487 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa760017030 con 0x7fa7781020d0 2024-04-21T23:28:35.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.089+0000 7fa77cbec640 1 -- 172.21.15.40:0/3667851487 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa77819d2f0 con 0x7fa7781020d0 2024-04-21T23:28:35.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.089+0000 7fa76f7fe640 1 -- 172.21.15.40:0/3667851487 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa760004020 con 0x7fa7781020d0 2024-04-21T23:28:35.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.089+0000 7fa76f7fe640 1 -- 172.21.15.40:0/3667851487 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7600047a0 con 0x7fa7781020d0 2024-04-21T23:28:35.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.089+0000 7fa77cbec640 1 -- 172.21.15.40:0/3667851487 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7781a4b80 con 0x7fa7781020d0 2024-04-21T23:28:35.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.090+0000 7fa77cbec640 1 -- 172.21.15.40:0/3667851487 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7780689b0 con 0x7fa7781020d0 2024-04-21T23:28:35.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.091+0000 7fa76f7fe640 1 -- 172.21.15.40:0/3667851487 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fa7600041d0 con 0x7fa7781020d0 2024-04-21T23:28:35.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.092+0000 7fa76f7fe640 1 --2- 172.21.15.40:0/3667851487 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fa7640a2d00 0x7fa7640a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:28:35.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.092+0000 7fa77659c640 1 --2- 172.21.15.40:0/3667851487 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fa7640a2d00 0x7fa7640a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:28:35.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.092+0000 7fa76f7fe640 1 -- 172.21.15.40:0/3667851487 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9554+0+0 (secure 0 0 0) 0x7fa7600cf970 con 0x7fa7781020d0 2024-04-21T23:28:35.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.092+0000 7fa77659c640 1 --2- 172.21.15.40:0/3667851487 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fa7640a2d00 0x7fa7640a51c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fa7680058c0 tx=0x7fa768005cb0 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:28:35.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.095+0000 7fa76f7fe640 1 -- 172.21.15.40:0/3667851487 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fa76009c180 con 0x7fa7781020d0 2024-04-21T23:28:35.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.280+0000 7fa77cbec640 1 -- 172.21.15.40:0/3667851487 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fa77802d170 con 0x7fa7781020d0 2024-04-21T23:28:35.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.281+0000 7fa76f7fe640 1 -- 172.21.15.40:0/3667851487 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v103) ==== 104+0+11 (secure 0 0 0) 0x7fa76009ba80 con 0x7fa7781020d0 2024-04-21T23:28:35.280 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-21T23:28:35.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.283+0000 7fa77cbec640 1 -- 172.21.15.40:0/3667851487 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fa7640a2d00 msgr2=0x7fa7640a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:28:35.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.283+0000 7fa77cbec640 1 --2- 172.21.15.40:0/3667851487 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fa7640a2d00 0x7fa7640a51c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fa7680058c0 tx=0x7fa768005cb0 comp rx=0 tx=0).stop 2024-04-21T23:28:35.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.283+0000 7fa77cbec640 1 -- 172.21.15.40:0/3667851487 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7781020d0 msgr2=0x7fa778198550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:28:35.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.283+0000 7fa77cbec640 1 --2- 172.21.15.40:0/3667851487 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7781020d0 0x7fa778198550 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fa76000d950 tx=0x7fa76000de10 comp rx=0 tx=0).stop 2024-04-21T23:28:35.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.283+0000 7fa77cbec640 1 -- 172.21.15.40:0/3667851487 shutdown_connections 2024-04-21T23:28:35.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.283+0000 7fa77cbec640 1 --2- 172.21.15.40:0/3667851487 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fa7640a2d00 0x7fa7640a51c0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:28:35.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.283+0000 7fa77cbec640 1 --2- 172.21.15.40:0/3667851487 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa778107720 0x7fa77819c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:28:35.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.283+0000 7fa77cbec640 1 --2- 172.21.15.40:0/3667851487 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7781020d0 0x7fa778198550 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:28:35.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.283+0000 7fa77cbec640 1 --2- 172.21.15.40:0/3667851487 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa778101710 0x7fa778198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:28:35.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.283+0000 7fa77cbec640 1 -- 172.21.15.40:0/3667851487 >> 172.21.15.40:0/3667851487 conn(0x7fa7780fd180 msgr2=0x7fa7780fd870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:28:35.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.283+0000 7fa77cbec640 1 -- 172.21.15.40:0/3667851487 shutdown_connections 2024-04-21T23:28:35.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.283+0000 7fa77cbec640 1 -- 172.21.15.40:0/3667851487 wait complete. 2024-04-21T23:28:35.291 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:28:35 smithi040 ceph-mon[27984]: pgmap v316: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:28:35.291 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:28:35 smithi040 ceph-mon[27984]: from='client.16317 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:28:35.292 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:28:35 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/360294288' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:28:35.292 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:28:35 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/144278931' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:28:35.293 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T23:28:35.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.398+0000 7f4e0994a640 1 -- 172.21.15.40:0/2168524390 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e04107720 msgr2=0x7f4e04107b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:28:35.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.398+0000 7f4e0994a640 1 --2- 172.21.15.40:0/2168524390 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e04107720 0x7f4e04107b00 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f4df4009930 tx=0x7f4df402bc30 comp rx=0 tx=0).stop 2024-04-21T23:28:35.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.398+0000 7f4e0994a640 1 -- 172.21.15.40:0/2168524390 shutdown_connections 2024-04-21T23:28:35.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.398+0000 7f4e0994a640 1 --2- 172.21.15.40:0/2168524390 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e041020d0 0x7f4e0410e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:28:35.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.398+0000 7f4e0994a640 1 --2- 172.21.15.40:0/2168524390 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4e04101710 0x7f4e04101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:28:35.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.398+0000 7f4e0994a640 1 --2- 172.21.15.40:0/2168524390 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e04107720 0x7f4e04107b00 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:28:35.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.398+0000 7f4e0994a640 1 -- 172.21.15.40:0/2168524390 >> 172.21.15.40:0/2168524390 conn(0x7f4e040fd180 msgr2=0x7f4e040ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:28:35.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.398+0000 7f4e0994a640 1 -- 172.21.15.40:0/2168524390 shutdown_connections 2024-04-21T23:28:35.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.398+0000 7f4e0994a640 1 -- 172.21.15.40:0/2168524390 wait complete. 2024-04-21T23:28:35.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.399+0000 7f4e0994a640 1 Processor -- start 2024-04-21T23:28:35.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.399+0000 7f4e0994a640 1 -- start start 2024-04-21T23:28:35.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.399+0000 7f4e0994a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4e04101710 0x7f4e04197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:28:35.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.399+0000 7f4e0994a640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e041020d0 0x7f4e04198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:28:35.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.399+0000 7f4e0994a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e04107720 0x7f4e0419c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:28:35.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.399+0000 7f4e0994a640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4e04111af0 con 0x7f4e04107720 2024-04-21T23:28:35.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.399+0000 7f4e0994a640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f4e04111970 con 0x7f4e041020d0 2024-04-21T23:28:35.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.399+0000 7f4e0994a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4e04111c70 con 0x7f4e04101710 2024-04-21T23:28:35.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.399+0000 7f4e037fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e04107720 0x7f4e0419c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:28:35.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.399+0000 7f4e037fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e04107720 0x7f4e0419c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58710/0 (socket says 172.21.15.40:58710) 2024-04-21T23:28:35.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.399+0000 7f4e037fe640 1 -- 172.21.15.40:0/2420046410 learned_addr learned my addr 172.21.15.40:0/2420046410 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:28:35.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.399+0000 7f4dfbfff640 1 --2- 172.21.15.40:0/2420046410 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e041020d0 0x7f4e04198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:28:35.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.399+0000 7f4e02ffd640 1 --2- 172.21.15.40:0/2420046410 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4e04101710 0x7f4e04197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:28:35.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.400+0000 7f4e037fe640 1 -- 172.21.15.40:0/2420046410 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4e04101710 msgr2=0x7f4e04197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:28:35.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.400+0000 7f4e037fe640 1 --2- 172.21.15.40:0/2420046410 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4e04101710 0x7f4e04197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:28:35.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.400+0000 7f4e037fe640 1 -- 172.21.15.40:0/2420046410 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e041020d0 msgr2=0x7f4e04198510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:28:35.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.400+0000 7f4e037fe640 1 --2- 172.21.15.40:0/2420046410 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e041020d0 0x7f4e04198510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:28:35.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.400+0000 7f4e037fe640 1 -- 172.21.15.40:0/2420046410 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e0419cfc0 con 0x7f4e04107720 2024-04-21T23:28:35.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.400+0000 7f4dfbfff640 1 --2- 172.21.15.40:0/2420046410 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e041020d0 0x7f4e04198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:28:35.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.400+0000 7f4e037fe640 1 --2- 172.21.15.40:0/2420046410 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e04107720 0x7f4e0419c890 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc00cce0 tx=0x7f4dfc00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:28:35.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.400+0000 7f4e08948640 1 -- 172.21.15.40:0/2420046410 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dfc00f450 con 0x7f4e04107720 2024-04-21T23:28:35.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.400+0000 7f4e08948640 1 -- 172.21.15.40:0/2420046410 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4dfc00fad0 con 0x7f4e04107720 2024-04-21T23:28:35.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.400+0000 7f4e0994a640 1 -- 172.21.15.40:0/2420046410 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e0419d2b0 con 0x7f4e04107720 2024-04-21T23:28:35.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.400+0000 7f4e08948640 1 -- 172.21.15.40:0/2420046410 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dfc018c10 con 0x7f4e04107720 2024-04-21T23:28:35.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.400+0000 7f4e0994a640 1 -- 172.21.15.40:0/2420046410 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e041a4b40 con 0x7f4e04107720 2024-04-21T23:28:35.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.401+0000 7f4e0994a640 1 -- 172.21.15.40:0/2420046410 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e040689b0 con 0x7f4e04107720 2024-04-21T23:28:35.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.402+0000 7f4e08948640 1 -- 172.21.15.40:0/2420046410 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f4dfc00f5f0 con 0x7f4e04107720 2024-04-21T23:28:35.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.403+0000 7f4e08948640 1 --2- 172.21.15.40:0/2420046410 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f4df00a2d00 0x7f4df00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:28:35.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.403+0000 7f4e02ffd640 1 --2- 172.21.15.40:0/2420046410 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f4df00a2d00 0x7f4df00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:28:35.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.403+0000 7f4e08948640 1 -- 172.21.15.40:0/2420046410 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9554+0+0 (secure 0 0 0) 0x7f4dfc0cfff0 con 0x7f4e04107720 2024-04-21T23:28:35.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.403+0000 7f4e02ffd640 1 --2- 172.21.15.40:0/2420046410 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f4df00a2d00 0x7f4df00a51c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4df4006010 tx=0x7f4df4000bc0 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:28:35.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.407+0000 7f4e08948640 1 -- 172.21.15.40:0/2420046410 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f4dfc014030 con 0x7f4e04107720 2024-04-21T23:28:35.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:28:35 smithi174 ceph-mon[31409]: pgmap v316: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:28:35.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:28:35 smithi174 ceph-mon[31409]: from='client.16317 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:28:35.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:28:35 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/360294288' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:28:35.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:28:35 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/144278931' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:28:35.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:28:35 smithi111 ceph-mon[31085]: pgmap v316: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:28:35.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:28:35 smithi111 ceph-mon[31085]: from='client.16317 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:28:35.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:28:35 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/360294288' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:28:35.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:28:35 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/144278931' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:28:35.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.591+0000 7f4e0994a640 1 -- 172.21.15.40:0/2420046410 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f4e04109cb0 con 0x7f4e04107720 2024-04-21T23:28:35.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.592+0000 7f4e08948640 1 -- 172.21.15.40:0/2420046410 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v103) ==== 100+0+11 (secure 0 0 0) 0x7f4dfc09c7c0 con 0x7f4e04107720 2024-04-21T23:28:35.591 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-04-21T23:28:35.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.594+0000 7f4e0994a640 1 -- 172.21.15.40:0/2420046410 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f4df00a2d00 msgr2=0x7f4df00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:28:35.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.594+0000 7f4e0994a640 1 --2- 172.21.15.40:0/2420046410 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f4df00a2d00 0x7f4df00a51c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4df4006010 tx=0x7f4df4000bc0 comp rx=0 tx=0).stop 2024-04-21T23:28:35.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.594+0000 7f4e0994a640 1 -- 172.21.15.40:0/2420046410 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e04107720 msgr2=0x7f4e0419c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:28:35.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.594+0000 7f4e0994a640 1 --2- 172.21.15.40:0/2420046410 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e04107720 0x7f4e0419c890 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc00cce0 tx=0x7f4dfc00a620 comp rx=0 tx=0).stop 2024-04-21T23:28:35.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.594+0000 7f4e0994a640 1 -- 172.21.15.40:0/2420046410 shutdown_connections 2024-04-21T23:28:35.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.594+0000 7f4e0994a640 1 --2- 172.21.15.40:0/2420046410 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f4df00a2d00 0x7f4df00a51c0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:28:35.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.594+0000 7f4e0994a640 1 --2- 172.21.15.40:0/2420046410 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e041020d0 0x7f4e04198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:28:35.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.594+0000 7f4e0994a640 1 --2- 172.21.15.40:0/2420046410 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e04107720 0x7f4e0419c890 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:28:35.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.594+0000 7f4e0994a640 1 --2- 172.21.15.40:0/2420046410 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4e04101710 0x7f4e04197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:28:35.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.594+0000 7f4e0994a640 1 -- 172.21.15.40:0/2420046410 >> 172.21.15.40:0/2420046410 conn(0x7f4e040fd180 msgr2=0x7f4e040fd910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:28:35.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.594+0000 7f4e0994a640 1 -- 172.21.15.40:0/2420046410 shutdown_connections 2024-04-21T23:28:35.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.594+0000 7f4e0994a640 1 -- 172.21.15.40:0/2420046410 wait complete. 2024-04-21T23:28:35.604 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T23:28:35.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.709+0000 7f79264c2640 1 -- 172.21.15.40:0/519288630 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f79201020d0 msgr2=0x7f792010e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:28:35.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.709+0000 7f79264c2640 1 --2- 172.21.15.40:0/519288630 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f79201020d0 0x7f792010e5e0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f7918009cd0 tx=0x7f791802c0f0 comp rx=0 tx=0).stop 2024-04-21T23:28:35.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.710+0000 7f79264c2640 1 -- 172.21.15.40:0/519288630 shutdown_connections 2024-04-21T23:28:35.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.710+0000 7f79264c2640 1 --2- 172.21.15.40:0/519288630 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7920101710 0x7f7920101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:28:35.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.710+0000 7f79264c2640 1 --2- 172.21.15.40:0/519288630 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f79201020d0 0x7f792010e5e0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:28:35.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.710+0000 7f79264c2640 1 --2- 172.21.15.40:0/519288630 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7920107720 0x7f7920107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:28:35.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.710+0000 7f79264c2640 1 -- 172.21.15.40:0/519288630 >> 172.21.15.40:0/519288630 conn(0x7f7920078b90 msgr2=0x7f7920078fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:28:35.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.710+0000 7f79264c2640 1 -- 172.21.15.40:0/519288630 shutdown_connections 2024-04-21T23:28:35.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.710+0000 7f79264c2640 1 -- 172.21.15.40:0/519288630 wait complete. 2024-04-21T23:28:35.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.710+0000 7f79264c2640 1 Processor -- start 2024-04-21T23:28:35.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.710+0000 7f79264c2640 1 -- start start 2024-04-21T23:28:35.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.710+0000 7f79264c2640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7920101710 0x7f7920198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:28:35.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.711+0000 7f79264c2640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f79201020d0 0x7f7920198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:28:35.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.711+0000 7f79264c2640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7920107720 0x7f792019c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:28:35.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.711+0000 7f79264c2640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7920111b30 con 0x7f79201020d0 2024-04-21T23:28:35.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.711+0000 7f79264c2640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f79201119b0 con 0x7f7920107720 2024-04-21T23:28:35.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.711+0000 7f79264c2640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7920111cb0 con 0x7f7920101710 2024-04-21T23:28:35.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.711+0000 7f791f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f79201020d0 0x7f7920198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:28:35.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.711+0000 7f791f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f79201020d0 0x7f7920198570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58730/0 (socket says 172.21.15.40:58730) 2024-04-21T23:28:35.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.711+0000 7f791f7fe640 1 -- 172.21.15.40:0/785783122 learned_addr learned my addr 172.21.15.40:0/785783122 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:28:35.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.711+0000 7f791ffff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7920101710 0x7f7920198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:28:35.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.711+0000 7f7924a5f640 1 --2- 172.21.15.40:0/785783122 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7920107720 0x7f792019c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:28:35.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.711+0000 7f791f7fe640 1 -- 172.21.15.40:0/785783122 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7920101710 msgr2=0x7f7920198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:28:35.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.711+0000 7f791f7fe640 1 --2- 172.21.15.40:0/785783122 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7920101710 0x7f7920198030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:28:35.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.711+0000 7f791f7fe640 1 -- 172.21.15.40:0/785783122 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7920107720 msgr2=0x7f792019c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:28:35.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.711+0000 7f791f7fe640 1 --2- 172.21.15.40:0/785783122 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7920107720 0x7f792019c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:28:35.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.711+0000 7f791f7fe640 1 -- 172.21.15.40:0/785783122 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f792019d020 con 0x7f79201020d0 2024-04-21T23:28:35.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.711+0000 7f7924a5f640 1 --2- 172.21.15.40:0/785783122 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7920107720 0x7f792019c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:28:35.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.711+0000 7f791f7fe640 1 --2- 172.21.15.40:0/785783122 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f79201020d0 0x7f7920198570 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f790800d950 tx=0x7f790800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:28:35.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.711+0000 7f791d7fa640 1 -- 172.21.15.40:0/785783122 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7908017030 con 0x7f79201020d0 2024-04-21T23:28:35.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.711+0000 7f79264c2640 1 -- 172.21.15.40:0/785783122 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f792019d310 con 0x7f79201020d0 2024-04-21T23:28:35.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.711+0000 7f791d7fa640 1 -- 172.21.15.40:0/785783122 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7908004020 con 0x7f79201020d0 2024-04-21T23:28:35.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.711+0000 7f791d7fa640 1 -- 172.21.15.40:0/785783122 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79080047a0 con 0x7f79201020d0 2024-04-21T23:28:35.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.711+0000 7f79264c2640 1 -- 172.21.15.40:0/785783122 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79201a4ba0 con 0x7f79201020d0 2024-04-21T23:28:35.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.712+0000 7f79264c2640 1 -- 172.21.15.40:0/785783122 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79200689b0 con 0x7f79201020d0 2024-04-21T23:28:35.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.713+0000 7f791d7fa640 1 -- 172.21.15.40:0/785783122 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f79080041d0 con 0x7f79201020d0 2024-04-21T23:28:35.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.714+0000 7f791d7fa640 1 --2- 172.21.15.40:0/785783122 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f790c0a2d00 0x7f790c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:28:35.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.714+0000 7f791ffff640 1 --2- 172.21.15.40:0/785783122 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f790c0a2d00 0x7f790c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:28:35.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.714+0000 7f791d7fa640 1 -- 172.21.15.40:0/785783122 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9554+0+0 (secure 0 0 0) 0x7f79080cf240 con 0x7f79201020d0 2024-04-21T23:28:35.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.714+0000 7f791ffff640 1 --2- 172.21.15.40:0/785783122 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f790c0a2d00 0x7f790c0a51c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f7910005fd0 tx=0x7f791000b390 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:28:35.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.718+0000 7f791d7fa640 1 -- 172.21.15.40:0/785783122 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f790809ba50 con 0x7f79201020d0 2024-04-21T23:28:35.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.903+0000 7f79264c2640 1 -- 172.21.15.40:0/785783122 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f79201096f0 con 0x7f79201020d0 2024-04-21T23:28:35.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.903+0000 7f791d7fa640 1 -- 172.21.15.40:0/785783122 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v103) ==== 103+0+11 (secure 0 0 0) 0x7f790809b350 con 0x7f79201020d0 2024-04-21T23:28:35.902 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-21T23:28:35.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.905+0000 7f79264c2640 1 -- 172.21.15.40:0/785783122 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f790c0a2d00 msgr2=0x7f790c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:28:35.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.905+0000 7f79264c2640 1 --2- 172.21.15.40:0/785783122 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f790c0a2d00 0x7f790c0a51c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f7910005fd0 tx=0x7f791000b390 comp rx=0 tx=0).stop 2024-04-21T23:28:35.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.905+0000 7f79264c2640 1 -- 172.21.15.40:0/785783122 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f79201020d0 msgr2=0x7f7920198570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:28:35.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.905+0000 7f79264c2640 1 --2- 172.21.15.40:0/785783122 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f79201020d0 0x7f7920198570 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f790800d950 tx=0x7f790800de10 comp rx=0 tx=0).stop 2024-04-21T23:28:35.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.905+0000 7f79264c2640 1 -- 172.21.15.40:0/785783122 shutdown_connections 2024-04-21T23:28:35.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.905+0000 7f79264c2640 1 --2- 172.21.15.40:0/785783122 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f790c0a2d00 0x7f790c0a51c0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:28:35.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.905+0000 7f79264c2640 1 --2- 172.21.15.40:0/785783122 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7920107720 0x7f792019c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:28:35.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.905+0000 7f79264c2640 1 --2- 172.21.15.40:0/785783122 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f79201020d0 0x7f7920198570 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:28:35.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.905+0000 7f79264c2640 1 --2- 172.21.15.40:0/785783122 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7920101710 0x7f7920198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:28:35.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.905+0000 7f79264c2640 1 -- 172.21.15.40:0/785783122 >> 172.21.15.40:0/785783122 conn(0x7f7920078b90 msgr2=0x7f79200ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:28:35.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.905+0000 7f79264c2640 1 -- 172.21.15.40:0/785783122 shutdown_connections 2024-04-21T23:28:35.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:35.905+0000 7f79264c2640 1 -- 172.21.15.40:0/785783122 wait complete. 2024-04-21T23:28:35.915 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-04-21T23:28:36.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:36.017+0000 7f28af496640 1 -- 172.21.15.40:0/1255523394 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28a8105b30 msgr2=0x7f28a8105fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:28:36.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:36.017+0000 7f28af496640 1 --2- 172.21.15.40:0/1255523394 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28a8105b30 0x7f28a8105fc0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f28a0009ca0 tx=0x7f28a002c180 comp rx=0 tx=0).stop 2024-04-21T23:28:36.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:36.017+0000 7f28af496640 1 -- 172.21.15.40:0/1255523394 shutdown_connections 2024-04-21T23:28:36.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:36.017+0000 7f28af496640 1 --2- 172.21.15.40:0/1255523394 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28a8101130 0x7f28a81055f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:28:36.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:36.017+0000 7f28af496640 1 --2- 172.21.15.40:0/1255523394 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28a8105b30 0x7f28a8105fc0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:28:36.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:36.017+0000 7f28af496640 1 --2- 172.21.15.40:0/1255523394 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28a8100810 0x7f28a8100bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:28:36.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:36.017+0000 7f28af496640 1 -- 172.21.15.40:0/1255523394 >> 172.21.15.40:0/1255523394 conn(0x7f28a80777a0 msgr2=0x7f28a8077bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:28:36.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:36.017+0000 7f28af496640 1 -- 172.21.15.40:0/1255523394 shutdown_connections 2024-04-21T23:28:36.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:36.017+0000 7f28af496640 1 -- 172.21.15.40:0/1255523394 wait complete. 2024-04-21T23:28:36.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:36.018+0000 7f28af496640 1 Processor -- start 2024-04-21T23:28:36.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:36.018+0000 7f28af496640 1 -- start start 2024-04-21T23:28:36.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:36.018+0000 7f28af496640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28a8100810 0x7f28a819e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:28:36.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:36.018+0000 7f28af496640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28a8101130 0x7f28a819eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:28:36.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:36.018+0000 7f28af496640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28a8105b30 0x7f28a81a2e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:28:36.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:36.018+0000 7f28af496640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f28a81180f0 con 0x7f28a8105b30 2024-04-21T23:28:36.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:36.018+0000 7f28af496640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f28a8117f70 con 0x7f28a8100810 2024-04-21T23:28:36.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:36.018+0000 7f28af496640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f28a8118270 con 0x7f28a8101130 2024-04-21T23:28:36.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:36.018+0000 7f28ada33640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28a8105b30 0x7f28a81a2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:28:36.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:36.018+0000 7f28ada33640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28a8105b30 0x7f28a81a2e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58748/0 (socket says 172.21.15.40:58748) 2024-04-21T23:28:36.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:36.018+0000 7f28ada33640 1 -- 172.21.15.40:0/291115050 learned_addr learned my addr 172.21.15.40:0/291115050 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:28:36.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:36.018+0000 7f28ad232640 1 --2- 172.21.15.40:0/291115050 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28a8100810 0x7f28a819e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:28:36.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:36.018+0000 7f28aca31640 1 --2- 172.21.15.40:0/291115050 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28a8101130 0x7f28a819eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:28:36.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:36.018+0000 7f28ada33640 1 -- 172.21.15.40:0/291115050 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28a8101130 msgr2=0x7f28a819eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:28:36.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:36.018+0000 7f28ada33640 1 --2- 172.21.15.40:0/291115050 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28a8101130 0x7f28a819eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:28:36.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:36.018+0000 7f28ada33640 1 -- 172.21.15.40:0/291115050 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28a8100810 msgr2=0x7f28a819e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:28:36.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:36.018+0000 7f28ada33640 1 --2- 172.21.15.40:0/291115050 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28a8100810 0x7f28a819e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:28:36.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:36.018+0000 7f28ada33640 1 -- 172.21.15.40:0/291115050 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28a81a35a0 con 0x7f28a8105b30 2024-04-21T23:28:36.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:36.018+0000 7f28ad232640 1 --2- 172.21.15.40:0/291115050 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28a8100810 0x7f28a819e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:28:36.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:36.019+0000 7f28ada33640 1 --2- 172.21.15.40:0/291115050 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28a8105b30 0x7f28a81a2e70 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f28a002c790 tx=0x7f28a0035ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:28:36.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:36.019+0000 7f289e7fc640 1 -- 172.21.15.40:0/291115050 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28a0039040 con 0x7f28a8105b30 2024-04-21T23:28:36.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:36.019+0000 7f289e7fc640 1 -- 172.21.15.40:0/291115050 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28a0044070 con 0x7f28a8105b30 2024-04-21T23:28:36.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:36.019+0000 7f28af496640 1 -- 172.21.15.40:0/291115050 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28a81a3830 con 0x7f28a8105b30 2024-04-21T23:28:36.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:36.019+0000 7f289e7fc640 1 -- 172.21.15.40:0/291115050 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28a00329f0 con 0x7f28a8105b30 2024-04-21T23:28:36.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:36.019+0000 7f28af496640 1 -- 172.21.15.40:0/291115050 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28a81ab120 con 0x7f28a8105b30 2024-04-21T23:28:36.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:36.020+0000 7f28af496640 1 -- 172.21.15.40:0/291115050 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28a806a570 con 0x7f28a8105b30 2024-04-21T23:28:36.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:36.021+0000 7f289e7fc640 1 -- 172.21.15.40:0/291115050 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f28a0032b90 con 0x7f28a8105b30 2024-04-21T23:28:36.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:36.022+0000 7f289e7fc640 1 --2- 172.21.15.40:0/291115050 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f28940a2d00 0x7f28940a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:28:36.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:36.022+0000 7f289e7fc640 1 -- 172.21.15.40:0/291115050 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9554+0+0 (secure 0 0 0) 0x7f28a00f2400 con 0x7f28a8105b30 2024-04-21T23:28:36.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:36.022+0000 7f28ad232640 1 --2- 172.21.15.40:0/291115050 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f28940a2d00 0x7f28940a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:28:36.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:36.022+0000 7f28ad232640 1 --2- 172.21.15.40:0/291115050 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f28940a2d00 0x7f28940a51c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f28a4005fd0 tx=0x7f28a400b5f0 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:28:36.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:36.025+0000 7f289e7fc640 1 -- 172.21.15.40:0/291115050 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f28a00bec80 con 0x7f28a8105b30 2024-04-21T23:28:36.181 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:28:36 smithi040 ceph-mon[27984]: from='client.16323 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:28:36.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:28:36 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/3667851487' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T23:28:36.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:28:36 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/2420046410' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T23:28:36.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:28:36 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/785783122' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T23:28:36.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:36.285+0000 7f28af496640 1 -- 172.21.15.40:0/291115050 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7f28a81ab2f0 con 0x7f28a8105b30 2024-04-21T23:28:36.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:36.300+0000 7f289e7fc640 1 -- 172.21.15.40:0/291115050 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v26) ==== 81+0+740900 (secure 0 0 0) 0x7f28a0003c70 con 0x7f28a8105b30 2024-04-21T23:28:36.309 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:28:36.310 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-21T23:28:36.310 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 26, 2024-04-21T23:28:36.310 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.310 INFO:teuthology.orchestra.run.smithi040.stdout: "active_gid": 25111, 2024-04-21T23:28:36.310 INFO:teuthology.orchestra.run.smithi040.stdout: "active_name": "x", 2024-04-21T23:28:36.310 INFO:teuthology.orchestra.run.smithi040.stdout: "active_addrs": { 2024-04-21T23:28:36.310 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-21T23:28:36.310 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:36.311 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-21T23:28:36.311 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6800", 2024-04-21T23:28:36.311 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 478611259 2024-04-21T23:28:36.311 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.311 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:36.311 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-21T23:28:36.311 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6801", 2024-04-21T23:28:36.311 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 478611259 2024-04-21T23:28:36.311 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:36.312 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-21T23:28:36.312 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.312 INFO:teuthology.orchestra.run.smithi040.stdout: "active_addr": "172.21.15.40:6801/478611259", 2024-04-21T23:28:36.312 INFO:teuthology.orchestra.run.smithi040.stdout: "active_change": "2024-04-21T23:18:23.389907+0000", 2024-04-21T23:28:36.312 INFO:teuthology.orchestra.run.smithi040.stdout: "active_mgr_features": 4540701547738038271, 2024-04-21T23:28:36.312 INFO:teuthology.orchestra.run.smithi040.stdout: "available": true, 2024-04-21T23:28:36.312 INFO:teuthology.orchestra.run.smithi040.stdout: "standbys": [ 2024-04-21T23:28:36.312 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:36.312 INFO:teuthology.orchestra.run.smithi040.stdout: "gid": 25061, 2024-04-21T23:28:36.313 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "y", 2024-04-21T23:28:36.313 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr_features": 4540701547738038271, 2024-04-21T23:28:36.313 INFO:teuthology.orchestra.run.smithi040.stdout: "available_modules": [ 2024-04-21T23:28:36.313 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:36.313 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "alerts", 2024-04-21T23:28:36.313 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:36.313 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:36.313 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:36.314 INFO:teuthology.orchestra.run.smithi040.stdout: "interval": { 2024-04-21T23:28:36.314 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "interval", 2024-04-21T23:28:36.314 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-21T23:28:36.314 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.314 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.314 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "60", 2024-04-21T23:28:36.314 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.314 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.314 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.315 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "How frequently to reexamine health status", 2024-04-21T23:28:36.315 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.315 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.315 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.315 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.315 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:36.315 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:36.315 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.315 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.316 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.316 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.316 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.316 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.316 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:36.316 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:36.316 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:36.316 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:36.317 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:36.317 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:36.317 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:36.317 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:36.317 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.317 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.317 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.317 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.317 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.318 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:36.318 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:36.318 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.318 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.318 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.318 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.318 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.318 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.318 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.319 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.319 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.319 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.319 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.319 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.319 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:36.319 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:36.319 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.320 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.320 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.320 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:36.320 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.320 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.320 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:36.320 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:36.320 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:36.321 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:36.321 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:36.321 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:36.321 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:36.321 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:36.321 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.321 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.321 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.321 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.322 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.322 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:36.322 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:36.322 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.322 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.322 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.322 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.322 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.323 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.323 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.323 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.323 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.323 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.323 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.323 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.323 INFO:teuthology.orchestra.run.smithi040.stdout: "smtp_destination": { 2024-04-21T23:28:36.323 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "smtp_destination", 2024-04-21T23:28:36.324 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.324 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.324 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.324 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.324 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.324 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.324 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.324 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Email address to send alerts to", 2024-04-21T23:28:36.325 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.325 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.325 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.325 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.325 INFO:teuthology.orchestra.run.smithi040.stdout: "smtp_from_name": { 2024-04-21T23:28:36.325 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "smtp_from_name", 2024-04-21T23:28:36.325 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.325 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.325 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.326 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "Ceph", 2024-04-21T23:28:36.326 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.326 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.326 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.326 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Email From: name", 2024-04-21T23:28:36.326 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.326 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.326 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.327 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.327 INFO:teuthology.orchestra.run.smithi040.stdout: "smtp_host": { 2024-04-21T23:28:36.327 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "smtp_host", 2024-04-21T23:28:36.327 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.327 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.327 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.327 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.327 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.327 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.328 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.328 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "SMTP server", 2024-04-21T23:28:36.328 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.328 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.328 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.328 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.328 INFO:teuthology.orchestra.run.smithi040.stdout: "smtp_password": { 2024-04-21T23:28:36.328 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "smtp_password", 2024-04-21T23:28:36.328 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.329 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.329 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.329 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.329 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.329 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.329 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.329 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Password to authenticate with", 2024-04-21T23:28:36.329 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.329 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.330 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.330 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.330 INFO:teuthology.orchestra.run.smithi040.stdout: "smtp_port": { 2024-04-21T23:28:36.330 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "smtp_port", 2024-04-21T23:28:36.330 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:36.330 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.330 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.330 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "465", 2024-04-21T23:28:36.330 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.331 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.331 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.331 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "SMTP port", 2024-04-21T23:28:36.331 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.331 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.331 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.331 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.331 INFO:teuthology.orchestra.run.smithi040.stdout: "smtp_sender": { 2024-04-21T23:28:36.331 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "smtp_sender", 2024-04-21T23:28:36.332 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.332 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.332 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.332 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.332 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.332 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.332 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.332 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "SMTP envelope sender", 2024-04-21T23:28:36.332 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.333 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.333 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.333 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.333 INFO:teuthology.orchestra.run.smithi040.stdout: "smtp_ssl": { 2024-04-21T23:28:36.333 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "smtp_ssl", 2024-04-21T23:28:36.333 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.333 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.334 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.334 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:36.334 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.334 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.334 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.334 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-21T23:28:36.334 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.334 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.335 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.335 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.335 INFO:teuthology.orchestra.run.smithi040.stdout: "smtp_user": { 2024-04-21T23:28:36.335 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "smtp_user", 2024-04-21T23:28:36.335 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.335 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.335 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.335 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.335 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.336 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.336 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.336 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "User to authenticate as", 2024-04-21T23:28:36.336 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.336 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.336 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.336 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:36.336 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:36.336 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.337 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:36.337 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "balancer", 2024-04-21T23:28:36.337 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:36.337 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:36.337 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:36.337 INFO:teuthology.orchestra.run.smithi040.stdout: "active": { 2024-04-21T23:28:36.337 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "active", 2024-04-21T23:28:36.337 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.337 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.338 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.338 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:36.338 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.338 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.338 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.338 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "automatically balance PGs across cluster", 2024-04-21T23:28:36.338 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.338 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.339 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.339 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.339 INFO:teuthology.orchestra.run.smithi040.stdout: "begin_time": { 2024-04-21T23:28:36.339 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "begin_time", 2024-04-21T23:28:36.339 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.339 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.339 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.339 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "0000", 2024-04-21T23:28:36.339 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.340 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.340 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.340 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "beginning time of day to automatically balance", 2024-04-21T23:28:36.340 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-21T23:28:36.340 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.340 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.340 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.340 INFO:teuthology.orchestra.run.smithi040.stdout: "begin_weekday": { 2024-04-21T23:28:36.341 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "begin_weekday", 2024-04-21T23:28:36.341 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "uint", 2024-04-21T23:28:36.341 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.341 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.341 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "0", 2024-04-21T23:28:36.341 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "0", 2024-04-21T23:28:36.341 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "6", 2024-04-21T23:28:36.341 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.341 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-21T23:28:36.342 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-21T23:28:36.342 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.342 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.342 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.342 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_compat_max_iterations": { 2024-04-21T23:28:36.342 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "crush_compat_max_iterations", 2024-04-21T23:28:36.342 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "uint", 2024-04-21T23:28:36.342 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.343 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.343 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "25", 2024-04-21T23:28:36.343 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "1", 2024-04-21T23:28:36.343 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "250", 2024-04-21T23:28:36.343 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.343 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-21T23:28:36.343 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.343 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.344 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.344 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.344 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_compat_metrics": { 2024-04-21T23:28:36.344 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "crush_compat_metrics", 2024-04-21T23:28:36.344 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.344 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.344 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.344 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "pgs,objects,bytes", 2024-04-21T23:28:36.345 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.345 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.345 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.345 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-21T23:28:36.345 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-21T23:28:36.345 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.345 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.345 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.346 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_compat_step": { 2024-04-21T23:28:36.346 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "crush_compat_step", 2024-04-21T23:28:36.346 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "float", 2024-04-21T23:28:36.346 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.346 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.346 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "0.5", 2024-04-21T23:28:36.346 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "0.001", 2024-04-21T23:28:36.346 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "0.999", 2024-04-21T23:28:36.347 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.347 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "aggressiveness of optimization", 2024-04-21T23:28:36.347 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-21T23:28:36.347 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.347 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.347 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.347 INFO:teuthology.orchestra.run.smithi040.stdout: "end_time": { 2024-04-21T23:28:36.347 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "end_time", 2024-04-21T23:28:36.348 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.348 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.348 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.348 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "2359", 2024-04-21T23:28:36.348 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.348 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.348 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.348 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "ending time of day to automatically balance", 2024-04-21T23:28:36.349 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-21T23:28:36.349 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.349 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.349 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.349 INFO:teuthology.orchestra.run.smithi040.stdout: "end_weekday": { 2024-04-21T23:28:36.349 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "end_weekday", 2024-04-21T23:28:36.349 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "uint", 2024-04-21T23:28:36.349 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.350 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.350 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "0", 2024-04-21T23:28:36.350 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "0", 2024-04-21T23:28:36.350 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "6", 2024-04-21T23:28:36.350 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.350 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-21T23:28:36.350 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-21T23:28:36.350 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.351 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.351 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.351 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:36.351 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:36.351 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.351 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.351 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.351 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.351 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.352 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.352 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:36.352 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:36.352 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:36.352 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:36.352 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:36.352 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:36.352 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:36.353 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:36.353 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.353 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.353 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.353 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.353 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.353 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:36.353 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:36.354 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.354 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.354 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.354 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.354 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.354 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.354 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.355 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.355 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.355 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.355 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.355 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.355 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:36.355 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:36.356 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.356 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.356 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.356 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:36.356 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.356 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.356 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:36.357 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:36.357 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:36.357 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:36.357 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:36.357 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:36.357 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:36.357 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:36.357 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.358 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.358 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.358 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.358 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.358 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:36.358 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:36.358 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.358 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.359 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.359 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.359 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.359 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.359 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.359 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.359 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.359 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.360 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.360 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.360 INFO:teuthology.orchestra.run.smithi040.stdout: "min_score": { 2024-04-21T23:28:36.360 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "min_score", 2024-04-21T23:28:36.360 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "float", 2024-04-21T23:28:36.360 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.360 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.360 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "0", 2024-04-21T23:28:36.361 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.361 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.361 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.361 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-21T23:28:36.361 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.361 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.361 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.361 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.361 INFO:teuthology.orchestra.run.smithi040.stdout: "mode": { 2024-04-21T23:28:36.362 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "mode", 2024-04-21T23:28:36.362 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.362 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.362 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.362 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "upmap", 2024-04-21T23:28:36.362 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.362 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.362 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:36.363 INFO:teuthology.orchestra.run.smithi040.stdout: "crush-compat", 2024-04-21T23:28:36.363 INFO:teuthology.orchestra.run.smithi040.stdout: "none", 2024-04-21T23:28:36.363 INFO:teuthology.orchestra.run.smithi040.stdout: "read", 2024-04-21T23:28:36.363 INFO:teuthology.orchestra.run.smithi040.stdout: "upmap", 2024-04-21T23:28:36.363 INFO:teuthology.orchestra.run.smithi040.stdout: "upmap-read" 2024-04-21T23:28:36.363 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:36.363 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Balancer mode", 2024-04-21T23:28:36.364 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.364 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.364 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.364 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.364 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_ids": { 2024-04-21T23:28:36.364 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "pool_ids", 2024-04-21T23:28:36.364 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.364 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.364 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.365 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.365 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.365 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.365 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.365 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-21T23:28:36.365 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.365 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.365 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.365 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.366 INFO:teuthology.orchestra.run.smithi040.stdout: "sleep_interval": { 2024-04-21T23:28:36.366 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "sleep_interval", 2024-04-21T23:28:36.366 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-21T23:28:36.366 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.366 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.366 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "60", 2024-04-21T23:28:36.366 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.366 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.367 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.367 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-21T23:28:36.367 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.367 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.367 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.367 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.367 INFO:teuthology.orchestra.run.smithi040.stdout: "upmap_max_deviation": { 2024-04-21T23:28:36.367 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "upmap_max_deviation", 2024-04-21T23:28:36.368 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:36.368 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.368 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.368 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "5", 2024-04-21T23:28:36.368 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "1", 2024-04-21T23:28:36.368 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.368 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.368 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-21T23:28:36.368 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-21T23:28:36.369 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.369 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.369 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.369 INFO:teuthology.orchestra.run.smithi040.stdout: "upmap_max_optimizations": { 2024-04-21T23:28:36.369 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "upmap_max_optimizations", 2024-04-21T23:28:36.369 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "uint", 2024-04-21T23:28:36.369 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.369 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.369 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10", 2024-04-21T23:28:36.370 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.370 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.370 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.370 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-21T23:28:36.370 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.370 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.370 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.370 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:36.371 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:36.371 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.371 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:36.371 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "cephadm", 2024-04-21T23:28:36.371 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:36.371 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:36.371 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:36.371 INFO:teuthology.orchestra.run.smithi040.stdout: "agent_down_multiplier": { 2024-04-21T23:28:36.372 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "agent_down_multiplier", 2024-04-21T23:28:36.372 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "float", 2024-04-21T23:28:36.372 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.372 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.372 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "3.0", 2024-04-21T23:28:36.372 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.372 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.372 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.372 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-21T23:28:36.373 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.373 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.373 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.373 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.373 INFO:teuthology.orchestra.run.smithi040.stdout: "agent_refresh_rate": { 2024-04-21T23:28:36.373 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "agent_refresh_rate", 2024-04-21T23:28:36.373 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-21T23:28:36.373 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.373 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.374 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "20", 2024-04-21T23:28:36.374 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.374 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.374 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.374 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-21T23:28:36.374 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.374 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.374 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.375 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.375 INFO:teuthology.orchestra.run.smithi040.stdout: "agent_starting_port": { 2024-04-21T23:28:36.375 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "agent_starting_port", 2024-04-21T23:28:36.375 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:36.375 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.375 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.375 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "4721", 2024-04-21T23:28:36.375 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.375 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.376 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.376 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-21T23:28:36.376 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.376 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.376 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.376 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.376 INFO:teuthology.orchestra.run.smithi040.stdout: "allow_ptrace": { 2024-04-21T23:28:36.377 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "allow_ptrace", 2024-04-21T23:28:36.377 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.377 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.377 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.377 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.377 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.377 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.377 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.377 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-21T23:28:36.378 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-21T23:28:36.378 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.378 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.378 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.378 INFO:teuthology.orchestra.run.smithi040.stdout: "autotune_interval": { 2024-04-21T23:28:36.378 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "autotune_interval", 2024-04-21T23:28:36.378 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-21T23:28:36.378 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.379 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.379 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "600", 2024-04-21T23:28:36.379 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.379 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.379 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.379 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-21T23:28:36.379 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.379 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.379 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.380 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.380 INFO:teuthology.orchestra.run.smithi040.stdout: "autotune_memory_target_ratio": { 2024-04-21T23:28:36.380 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "autotune_memory_target_ratio", 2024-04-21T23:28:36.380 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "float", 2024-04-21T23:28:36.380 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.380 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.380 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "0.7", 2024-04-21T23:28:36.380 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.380 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.381 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.381 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-21T23:28:36.381 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.381 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.381 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.381 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.381 INFO:teuthology.orchestra.run.smithi040.stdout: "cephadm_log_destination": { 2024-04-21T23:28:36.381 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "cephadm_log_destination", 2024-04-21T23:28:36.381 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.382 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.382 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.382 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.382 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.382 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.382 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:36.382 INFO:teuthology.orchestra.run.smithi040.stdout: "file", 2024-04-21T23:28:36.382 INFO:teuthology.orchestra.run.smithi040.stdout: "file,syslog", 2024-04-21T23:28:36.383 INFO:teuthology.orchestra.run.smithi040.stdout: "syslog" 2024-04-21T23:28:36.383 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:36.383 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-21T23:28:36.383 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.383 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.383 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.383 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.383 INFO:teuthology.orchestra.run.smithi040.stdout: "cgroups_split": { 2024-04-21T23:28:36.383 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "cgroups_split", 2024-04-21T23:28:36.384 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.384 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.384 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.384 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:36.384 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.384 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.384 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.384 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-21T23:28:36.384 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.385 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.385 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.385 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.385 INFO:teuthology.orchestra.run.smithi040.stdout: "config_checks_enabled": { 2024-04-21T23:28:36.385 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "config_checks_enabled", 2024-04-21T23:28:36.385 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.385 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.385 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.386 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.386 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.386 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.386 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.386 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-21T23:28:36.386 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.386 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.386 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.386 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.387 INFO:teuthology.orchestra.run.smithi040.stdout: "config_dashboard": { 2024-04-21T23:28:36.387 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "config_dashboard", 2024-04-21T23:28:36.387 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.387 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.387 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.387 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:36.387 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.387 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.388 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.388 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-21T23:28:36.388 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.388 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.388 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.388 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.388 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_alertmanager": { 2024-04-21T23:28:36.388 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_alertmanager", 2024-04-21T23:28:36.388 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.389 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.389 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.389 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-21T23:28:36.389 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.389 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.389 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.389 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Prometheus container image", 2024-04-21T23:28:36.389 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.390 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.390 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.390 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.390 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_base": { 2024-04-21T23:28:36.390 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_base", 2024-04-21T23:28:36.390 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.390 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.390 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.390 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-21T23:28:36.391 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.391 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.391 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.391 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Container image name, without the tag", 2024-04-21T23:28:36.391 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.391 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.391 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.391 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.392 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_elasticsearch": { 2024-04-21T23:28:36.392 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_elasticsearch", 2024-04-21T23:28:36.392 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.392 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.392 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.392 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-21T23:28:36.392 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.392 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.392 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.393 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "elasticsearch container image", 2024-04-21T23:28:36.393 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.393 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.393 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.393 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.393 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_grafana": { 2024-04-21T23:28:36.393 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_grafana", 2024-04-21T23:28:36.393 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.393 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.394 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.394 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-21T23:28:36.394 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.394 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.394 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.394 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Prometheus container image", 2024-04-21T23:28:36.394 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.394 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.395 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.395 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.395 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_haproxy": { 2024-04-21T23:28:36.395 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_haproxy", 2024-04-21T23:28:36.395 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.395 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.395 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.395 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-21T23:28:36.396 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.396 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.396 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.396 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "HAproxy container image", 2024-04-21T23:28:36.396 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.396 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.396 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.396 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.397 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_jaeger_agent": { 2024-04-21T23:28:36.397 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_jaeger_agent", 2024-04-21T23:28:36.397 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.397 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.397 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.397 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-21T23:28:36.397 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.397 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.397 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.398 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Jaeger agent container image", 2024-04-21T23:28:36.398 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.398 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.398 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.398 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.398 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_jaeger_collector": { 2024-04-21T23:28:36.398 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_jaeger_collector", 2024-04-21T23:28:36.399 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.399 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.399 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.399 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-21T23:28:36.399 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.399 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.399 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.399 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Jaeger collector container image", 2024-04-21T23:28:36.400 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.400 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.400 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.400 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.400 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_jaeger_query": { 2024-04-21T23:28:36.400 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_jaeger_query", 2024-04-21T23:28:36.400 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.400 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.401 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.401 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-21T23:28:36.401 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.401 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.401 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.401 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Jaeger query container image", 2024-04-21T23:28:36.401 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.401 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.402 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.402 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.402 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_keepalived": { 2024-04-21T23:28:36.402 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_keepalived", 2024-04-21T23:28:36.402 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.402 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.402 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.402 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-21T23:28:36.403 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.403 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.403 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.403 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Keepalived container image", 2024-04-21T23:28:36.403 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.403 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.403 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.403 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.403 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_loki": { 2024-04-21T23:28:36.404 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_loki", 2024-04-21T23:28:36.404 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.404 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.404 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.404 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-21T23:28:36.404 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.404 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.404 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.404 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Loki container image", 2024-04-21T23:28:36.405 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.405 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.405 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.405 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.405 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_node_exporter": { 2024-04-21T23:28:36.405 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_node_exporter", 2024-04-21T23:28:36.405 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.405 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.406 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.406 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-21T23:28:36.406 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.406 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.406 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.406 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Prometheus container image", 2024-04-21T23:28:36.406 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.406 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.407 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.407 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.407 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_nvmeof": { 2024-04-21T23:28:36.407 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_nvmeof", 2024-04-21T23:28:36.407 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.407 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.407 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.407 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-21T23:28:36.407 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.408 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.408 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.408 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Nvme-of container image", 2024-04-21T23:28:36.408 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.408 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.408 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.408 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.408 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_prometheus": { 2024-04-21T23:28:36.408 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_prometheus", 2024-04-21T23:28:36.409 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.409 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.409 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.409 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-21T23:28:36.409 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.409 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.409 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.409 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Prometheus container image", 2024-04-21T23:28:36.409 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.410 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.410 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.410 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.410 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_promtail": { 2024-04-21T23:28:36.410 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_promtail", 2024-04-21T23:28:36.410 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.410 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.411 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.411 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-21T23:28:36.411 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.411 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.411 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.411 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Promtail container image", 2024-04-21T23:28:36.411 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.411 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.411 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.412 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.412 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_samba": { 2024-04-21T23:28:36.412 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_samba", 2024-04-21T23:28:36.412 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.412 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.412 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.412 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-21T23:28:36.412 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.413 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.413 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.413 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Samba/SMB container image", 2024-04-21T23:28:36.413 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.413 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.413 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.413 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.413 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_snmp_gateway": { 2024-04-21T23:28:36.413 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_snmp_gateway", 2024-04-21T23:28:36.414 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.414 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.414 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.414 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-21T23:28:36.414 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.414 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.414 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.414 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "SNMP Gateway container image", 2024-04-21T23:28:36.415 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.415 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.415 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.415 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.415 INFO:teuthology.orchestra.run.smithi040.stdout: "container_init": { 2024-04-21T23:28:36.415 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_init", 2024-04-21T23:28:36.415 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.415 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.415 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.416 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:36.416 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.416 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.416 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.416 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Run podman/docker with `--init`", 2024-04-21T23:28:36.416 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.416 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.417 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.417 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.417 INFO:teuthology.orchestra.run.smithi040.stdout: "daemon_cache_timeout": { 2024-04-21T23:28:36.417 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "daemon_cache_timeout", 2024-04-21T23:28:36.417 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-21T23:28:36.417 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.417 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.417 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "600", 2024-04-21T23:28:36.417 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.418 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.418 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.418 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-21T23:28:36.418 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.418 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.418 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.418 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.418 INFO:teuthology.orchestra.run.smithi040.stdout: "default_cephadm_command_timeout": { 2024-04-21T23:28:36.419 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "default_cephadm_command_timeout", 2024-04-21T23:28:36.419 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:36.419 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.419 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.419 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "900", 2024-04-21T23:28:36.419 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.419 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.419 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.419 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-21T23:28:36.420 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.420 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.420 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.420 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.420 INFO:teuthology.orchestra.run.smithi040.stdout: "default_registry": { 2024-04-21T23:28:36.420 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "default_registry", 2024-04-21T23:28:36.420 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.420 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.420 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.421 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "docker.io", 2024-04-21T23:28:36.421 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.421 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.421 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.421 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-21T23:28:36.421 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.421 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.421 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.422 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.422 INFO:teuthology.orchestra.run.smithi040.stdout: "device_cache_timeout": { 2024-04-21T23:28:36.422 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "device_cache_timeout", 2024-04-21T23:28:36.422 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-21T23:28:36.422 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.422 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.422 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "1800", 2024-04-21T23:28:36.422 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.423 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.423 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.423 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "seconds to cache device inventory", 2024-04-21T23:28:36.423 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.423 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.423 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.423 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.423 INFO:teuthology.orchestra.run.smithi040.stdout: "device_enhanced_scan": { 2024-04-21T23:28:36.424 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "device_enhanced_scan", 2024-04-21T23:28:36.424 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.424 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.424 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.424 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.424 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.424 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.424 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.424 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-21T23:28:36.425 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.425 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.425 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.425 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.425 INFO:teuthology.orchestra.run.smithi040.stdout: "facts_cache_timeout": { 2024-04-21T23:28:36.425 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "facts_cache_timeout", 2024-04-21T23:28:36.425 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-21T23:28:36.425 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.426 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.426 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "60", 2024-04-21T23:28:36.426 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.426 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.426 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.426 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "seconds to cache host facts data", 2024-04-21T23:28:36.426 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.426 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.427 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.427 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.427 INFO:teuthology.orchestra.run.smithi040.stdout: "grafana_dashboards_path": { 2024-04-21T23:28:36.427 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "grafana_dashboards_path", 2024-04-21T23:28:36.427 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.427 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.427 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.427 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-21T23:28:36.427 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.428 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.428 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.428 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-21T23:28:36.428 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.428 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.428 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.428 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.428 INFO:teuthology.orchestra.run.smithi040.stdout: "host_check_interval": { 2024-04-21T23:28:36.429 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "host_check_interval", 2024-04-21T23:28:36.429 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-21T23:28:36.429 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.429 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.429 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "600", 2024-04-21T23:28:36.429 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.429 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.429 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.429 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "how frequently to perform a host check", 2024-04-21T23:28:36.430 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.430 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.430 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.430 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.430 INFO:teuthology.orchestra.run.smithi040.stdout: "hw_monitoring": { 2024-04-21T23:28:36.433 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "hw_monitoring", 2024-04-21T23:28:36.434 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.434 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.434 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.434 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.434 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.434 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.434 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.435 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-21T23:28:36.435 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.435 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.435 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.435 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.435 INFO:teuthology.orchestra.run.smithi040.stdout: "inventory_list_all": { 2024-04-21T23:28:36.435 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "inventory_list_all", 2024-04-21T23:28:36.435 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.436 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.436 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.436 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.436 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.436 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.436 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.436 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-21T23:28:36.436 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.436 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.437 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.437 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.437 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:36.437 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:36.437 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.437 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.437 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.437 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.438 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.438 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.438 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:36.438 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:36.438 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:36.438 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:36.438 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:36.438 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:36.439 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:36.439 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:36.439 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.439 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.439 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.439 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.439 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.439 INFO:teuthology.orchestra.run.smithi040.stdout: "log_refresh_metadata": { 2024-04-21T23:28:36.439 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_refresh_metadata", 2024-04-21T23:28:36.440 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.440 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.440 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.440 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.440 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.440 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.440 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.441 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-21T23:28:36.441 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.441 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.441 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.441 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.441 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:36.441 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:36.441 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.441 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.442 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.442 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:36.442 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.442 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.442 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.442 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-21T23:28:36.442 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.442 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.442 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.443 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.443 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:36.443 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:36.443 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.443 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.443 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.443 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:36.443 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.444 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.444 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:36.444 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:36.444 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:36.444 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:36.444 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:36.444 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:36.444 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:36.445 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:36.445 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.445 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.445 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.445 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.445 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.445 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:36.445 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:36.445 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.446 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.446 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.446 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.446 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.446 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.446 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.446 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.446 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.446 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.447 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.447 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.447 INFO:teuthology.orchestra.run.smithi040.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-21T23:28:36.447 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-21T23:28:36.447 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.447 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.447 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.447 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.448 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.448 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.448 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.448 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-21T23:28:36.448 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.448 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.448 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.448 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.448 INFO:teuthology.orchestra.run.smithi040.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-21T23:28:36.449 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-21T23:28:36.449 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.449 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.449 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.449 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "*", 2024-04-21T23:28:36.449 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.449 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.449 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.449 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-21T23:28:36.450 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.450 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.450 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.450 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.450 INFO:teuthology.orchestra.run.smithi040.stdout: "max_count_per_host": { 2024-04-21T23:28:36.450 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "max_count_per_host", 2024-04-21T23:28:36.450 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:36.450 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.450 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.451 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10", 2024-04-21T23:28:36.451 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.451 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.451 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.451 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "max number of daemons per service per host", 2024-04-21T23:28:36.451 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.451 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.451 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.451 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.452 INFO:teuthology.orchestra.run.smithi040.stdout: "max_osd_draining_count": { 2024-04-21T23:28:36.452 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "max_osd_draining_count", 2024-04-21T23:28:36.452 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:36.452 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.452 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.452 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10", 2024-04-21T23:28:36.452 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.452 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.452 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.453 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-21T23:28:36.453 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.453 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.453 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.453 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.453 INFO:teuthology.orchestra.run.smithi040.stdout: "migration_current": { 2024-04-21T23:28:36.453 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "migration_current", 2024-04-21T23:28:36.453 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:36.453 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.453 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.454 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.454 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.454 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.454 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.454 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "internal - do not modify", 2024-04-21T23:28:36.454 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.454 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.454 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.455 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.455 INFO:teuthology.orchestra.run.smithi040.stdout: "mode": { 2024-04-21T23:28:36.455 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "mode", 2024-04-21T23:28:36.455 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.455 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.455 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.455 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "root", 2024-04-21T23:28:36.455 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.456 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.456 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:36.456 INFO:teuthology.orchestra.run.smithi040.stdout: "cephadm-package", 2024-04-21T23:28:36.456 INFO:teuthology.orchestra.run.smithi040.stdout: "root" 2024-04-21T23:28:36.456 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:36.456 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "mode for remote execution of cephadm", 2024-04-21T23:28:36.456 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.456 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.456 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.456 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.457 INFO:teuthology.orchestra.run.smithi040.stdout: "oob_default_addr": { 2024-04-21T23:28:36.457 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "oob_default_addr", 2024-04-21T23:28:36.457 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.457 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.457 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.457 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "169.254.1.1", 2024-04-21T23:28:36.457 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.457 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.457 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.458 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-21T23:28:36.458 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.458 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.458 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.458 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.458 INFO:teuthology.orchestra.run.smithi040.stdout: "prometheus_alerts_path": { 2024-04-21T23:28:36.458 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "prometheus_alerts_path", 2024-04-21T23:28:36.458 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.458 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.459 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.459 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-21T23:28:36.459 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.459 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.459 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.459 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-21T23:28:36.459 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.459 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.459 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.460 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.460 INFO:teuthology.orchestra.run.smithi040.stdout: "registry_insecure": { 2024-04-21T23:28:36.460 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "registry_insecure", 2024-04-21T23:28:36.460 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.460 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.460 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.460 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.460 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.461 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.461 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.461 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-21T23:28:36.461 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.461 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.461 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.461 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.461 INFO:teuthology.orchestra.run.smithi040.stdout: "registry_password": { 2024-04-21T23:28:36.462 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "registry_password", 2024-04-21T23:28:36.462 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.462 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.462 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.462 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.462 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.462 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.462 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.463 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-21T23:28:36.463 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.463 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.463 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.463 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.463 INFO:teuthology.orchestra.run.smithi040.stdout: "registry_url": { 2024-04-21T23:28:36.463 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "registry_url", 2024-04-21T23:28:36.463 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.463 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.464 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.464 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.464 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.464 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.464 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.464 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-21T23:28:36.464 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.464 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.464 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.465 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.465 INFO:teuthology.orchestra.run.smithi040.stdout: "registry_username": { 2024-04-21T23:28:36.465 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "registry_username", 2024-04-21T23:28:36.465 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.465 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.465 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.465 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.465 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.466 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.466 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.466 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-21T23:28:36.466 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.466 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.466 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.466 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.466 INFO:teuthology.orchestra.run.smithi040.stdout: "secure_monitoring_stack": { 2024-04-21T23:28:36.467 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "secure_monitoring_stack", 2024-04-21T23:28:36.467 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.467 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.467 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.467 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.467 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.467 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.467 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.467 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-21T23:28:36.468 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.468 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.468 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.468 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.468 INFO:teuthology.orchestra.run.smithi040.stdout: "service_discovery_port": { 2024-04-21T23:28:36.468 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "service_discovery_port", 2024-04-21T23:28:36.468 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:36.468 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.468 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.469 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "8765", 2024-04-21T23:28:36.469 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.469 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.469 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.469 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "cephadm service discovery port", 2024-04-21T23:28:36.469 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.469 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.469 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.470 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.470 INFO:teuthology.orchestra.run.smithi040.stdout: "ssh_config_file": { 2024-04-21T23:28:36.470 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ssh_config_file", 2024-04-21T23:28:36.470 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.470 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.470 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.470 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.470 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.471 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.471 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.471 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-21T23:28:36.471 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.471 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.471 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.471 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.471 INFO:teuthology.orchestra.run.smithi040.stdout: "use_agent": { 2024-04-21T23:28:36.471 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "use_agent", 2024-04-21T23:28:36.472 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.472 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.472 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.472 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.472 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.472 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.472 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.472 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-21T23:28:36.473 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.473 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.473 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.473 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.473 INFO:teuthology.orchestra.run.smithi040.stdout: "use_repo_digest": { 2024-04-21T23:28:36.473 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "use_repo_digest", 2024-04-21T23:28:36.473 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.473 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.473 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.474 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:36.474 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.474 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.474 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.474 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-21T23:28:36.474 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.474 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.474 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.475 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.475 INFO:teuthology.orchestra.run.smithi040.stdout: "warn_on_failed_host_check": { 2024-04-21T23:28:36.475 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "warn_on_failed_host_check", 2024-04-21T23:28:36.475 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.475 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.475 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.475 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:36.475 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.475 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.476 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.476 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "raise a health warning if the host check fails", 2024-04-21T23:28:36.476 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.476 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.476 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.476 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.476 INFO:teuthology.orchestra.run.smithi040.stdout: "warn_on_stray_daemons": { 2024-04-21T23:28:36.476 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "warn_on_stray_daemons", 2024-04-21T23:28:36.476 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.477 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.477 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.477 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:36.477 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.477 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.477 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.477 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-21T23:28:36.477 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.477 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.478 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.478 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.478 INFO:teuthology.orchestra.run.smithi040.stdout: "warn_on_stray_hosts": { 2024-04-21T23:28:36.478 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "warn_on_stray_hosts", 2024-04-21T23:28:36.478 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.478 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.478 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.478 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:36.479 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.479 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.479 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.479 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-21T23:28:36.479 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.479 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.479 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.479 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:36.479 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:36.480 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.480 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:36.480 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "crash", 2024-04-21T23:28:36.480 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:36.480 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:36.480 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:36.480 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:36.480 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:36.481 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.481 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.481 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.481 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.481 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.481 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.481 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:36.481 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:36.482 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:36.482 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:36.482 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:36.482 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:36.482 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:36.482 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:36.482 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.483 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.483 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.483 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.483 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.483 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:36.483 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:36.483 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.483 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.484 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.484 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.484 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.484 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.484 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.484 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.484 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.484 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.484 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.485 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.485 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:36.485 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:36.485 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.485 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.485 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.485 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:36.485 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.486 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.486 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:36.486 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:36.486 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:36.486 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:36.486 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:36.486 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:36.486 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:36.487 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:36.487 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.487 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.487 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.487 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.487 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.487 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:36.487 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:36.487 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.488 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.488 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.488 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.488 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.488 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.488 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.488 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.488 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.489 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.489 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.489 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.489 INFO:teuthology.orchestra.run.smithi040.stdout: "retain_interval": { 2024-04-21T23:28:36.489 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "retain_interval", 2024-04-21T23:28:36.489 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-21T23:28:36.489 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.489 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.489 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "31536000", 2024-04-21T23:28:36.490 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.490 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.490 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.490 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-21T23:28:36.490 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.490 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.490 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.490 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.491 INFO:teuthology.orchestra.run.smithi040.stdout: "warn_recent_interval": { 2024-04-21T23:28:36.491 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "warn_recent_interval", 2024-04-21T23:28:36.491 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-21T23:28:36.491 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.491 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.491 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "1209600", 2024-04-21T23:28:36.491 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.491 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.491 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.492 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-21T23:28:36.492 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.492 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.492 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.492 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:36.492 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:36.492 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.492 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:36.492 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "dashboard", 2024-04-21T23:28:36.493 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:36.493 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:36.493 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:36.493 INFO:teuthology.orchestra.run.smithi040.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-21T23:28:36.493 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-21T23:28:36.493 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:36.493 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.493 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.494 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10", 2024-04-21T23:28:36.494 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.494 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.494 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.494 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.494 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.494 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.494 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.495 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.495 INFO:teuthology.orchestra.run.smithi040.stdout: "ALERTMANAGER_API_HOST": { 2024-04-21T23:28:36.495 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-21T23:28:36.495 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.495 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.495 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.495 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.495 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.496 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.496 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.496 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.496 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.496 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.496 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.496 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.496 INFO:teuthology.orchestra.run.smithi040.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-21T23:28:36.496 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-21T23:28:36.497 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.497 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.497 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.497 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:36.497 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.497 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.497 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.497 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.498 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.498 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.498 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.498 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.498 INFO:teuthology.orchestra.run.smithi040.stdout: "AUDIT_API_ENABLED": { 2024-04-21T23:28:36.498 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "AUDIT_API_ENABLED", 2024-04-21T23:28:36.498 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.498 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.498 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.499 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.499 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.499 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.499 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.499 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.499 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.499 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.499 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.499 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.500 INFO:teuthology.orchestra.run.smithi040.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-21T23:28:36.500 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-21T23:28:36.500 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.500 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.500 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.500 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:36.500 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.500 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.500 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.501 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.501 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.501 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.501 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.501 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.501 INFO:teuthology.orchestra.run.smithi040.stdout: "ENABLE_BROWSABLE_API": { 2024-04-21T23:28:36.501 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-21T23:28:36.501 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.501 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.502 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.502 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:36.502 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.502 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.502 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.502 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.502 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.502 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.503 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.503 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.503 INFO:teuthology.orchestra.run.smithi040.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-21T23:28:36.503 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-21T23:28:36.503 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.503 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.503 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.503 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:36.503 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.504 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.504 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.504 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.504 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.504 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.504 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.504 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.504 INFO:teuthology.orchestra.run.smithi040.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-21T23:28:36.504 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-21T23:28:36.505 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.505 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.505 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.505 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:36.505 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.505 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.505 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.505 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.506 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.506 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.506 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.506 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.506 INFO:teuthology.orchestra.run.smithi040.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-21T23:28:36.506 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-21T23:28:36.506 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.506 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.506 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.507 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:36.507 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.507 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.507 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.507 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.507 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.507 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.507 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.507 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.508 INFO:teuthology.orchestra.run.smithi040.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-21T23:28:36.508 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-21T23:28:36.508 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.508 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.508 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.508 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:36.508 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.508 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.509 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.509 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.509 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.509 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.509 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.509 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.509 INFO:teuthology.orchestra.run.smithi040.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-21T23:28:36.509 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-21T23:28:36.509 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.510 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.510 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.510 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:36.510 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.510 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.510 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.510 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.510 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.511 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.511 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.511 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.511 INFO:teuthology.orchestra.run.smithi040.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-21T23:28:36.511 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-21T23:28:36.511 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.511 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.511 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.511 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:36.512 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.512 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.512 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.512 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.512 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.512 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.512 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.512 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.512 INFO:teuthology.orchestra.run.smithi040.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-21T23:28:36.513 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-21T23:28:36.513 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.513 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.513 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.513 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:36.513 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.513 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.513 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.513 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.514 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.514 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.514 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.514 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.514 INFO:teuthology.orchestra.run.smithi040.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-21T23:28:36.514 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-21T23:28:36.514 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.514 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.515 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.515 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.515 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.515 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.515 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.515 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.515 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.515 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.515 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.516 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.516 INFO:teuthology.orchestra.run.smithi040.stdout: "GRAFANA_API_PASSWORD": { 2024-04-21T23:28:36.516 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-21T23:28:36.516 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.516 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.516 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.516 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "admin", 2024-04-21T23:28:36.516 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.516 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.517 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.517 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.517 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.517 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.517 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.517 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.517 INFO:teuthology.orchestra.run.smithi040.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-21T23:28:36.517 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-21T23:28:36.518 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.518 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.518 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.518 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:36.518 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.518 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.518 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.518 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.519 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.519 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.519 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.519 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.519 INFO:teuthology.orchestra.run.smithi040.stdout: "GRAFANA_API_URL": { 2024-04-21T23:28:36.519 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "GRAFANA_API_URL", 2024-04-21T23:28:36.519 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.519 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.519 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.520 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.520 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.520 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.520 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.520 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.520 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.520 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.520 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.520 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.521 INFO:teuthology.orchestra.run.smithi040.stdout: "GRAFANA_API_USERNAME": { 2024-04-21T23:28:36.521 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-21T23:28:36.521 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.521 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.521 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.521 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "admin", 2024-04-21T23:28:36.521 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.521 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.521 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.522 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.522 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.522 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.522 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.522 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.522 INFO:teuthology.orchestra.run.smithi040.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-21T23:28:36.522 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-21T23:28:36.522 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.523 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.523 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.523 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.523 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.523 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.523 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.523 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.523 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.523 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.524 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.524 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.524 INFO:teuthology.orchestra.run.smithi040.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-21T23:28:36.524 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-21T23:28:36.524 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.524 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.524 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.524 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.525 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.525 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.525 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.525 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.525 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.525 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.525 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.525 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.526 INFO:teuthology.orchestra.run.smithi040.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-21T23:28:36.526 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-21T23:28:36.526 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.526 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.526 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.526 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:36.526 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.526 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.526 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.527 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.527 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.527 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.527 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.527 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.527 INFO:teuthology.orchestra.run.smithi040.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-21T23:28:36.527 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-21T23:28:36.527 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.528 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.528 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.528 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.528 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.528 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.528 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.528 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.528 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.528 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.529 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.529 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.529 INFO:teuthology.orchestra.run.smithi040.stdout: "PROMETHEUS_API_HOST": { 2024-04-21T23:28:36.529 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-21T23:28:36.529 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.529 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.529 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.529 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.529 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.530 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.530 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.530 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.530 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.530 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.530 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.530 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.530 INFO:teuthology.orchestra.run.smithi040.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-21T23:28:36.531 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-21T23:28:36.531 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.531 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.531 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.531 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:36.531 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.531 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.531 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.531 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.532 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.532 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.532 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.532 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.532 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-21T23:28:36.532 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-21T23:28:36.532 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.532 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.533 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.533 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.533 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.533 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.533 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.533 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.533 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.533 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.533 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.534 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.534 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-21T23:28:36.534 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-21T23:28:36.534 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.534 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.534 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.534 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.534 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.534 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.535 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.535 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.535 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.535 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.535 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.535 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.535 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-21T23:28:36.535 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-21T23:28:36.536 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.536 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.536 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.536 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:36.536 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.536 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.536 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.536 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.536 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.537 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.537 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.537 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.537 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-21T23:28:36.537 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-21T23:28:36.537 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.537 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.537 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.537 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:36.538 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.538 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.538 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.538 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.538 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.538 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.538 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.538 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.539 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-21T23:28:36.539 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-21T23:28:36.539 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.539 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.539 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.539 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.539 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.539 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.539 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.540 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.540 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.540 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.540 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.540 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.540 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-21T23:28:36.540 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-21T23:28:36.540 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.540 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.541 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.541 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.541 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.541 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.541 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.541 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.541 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.541 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.542 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.542 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.542 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-21T23:28:36.542 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-21T23:28:36.542 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.542 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.542 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.542 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.543 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.543 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.543 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.543 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.543 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.543 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.543 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.543 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.543 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_ENABLED": { 2024-04-21T23:28:36.544 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-21T23:28:36.544 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.544 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.544 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.544 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:36.544 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.544 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.544 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.545 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.545 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.545 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.545 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.545 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.545 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-21T23:28:36.545 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-21T23:28:36.546 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.546 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.546 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.546 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-21T23:28:36.546 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.546 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.546 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.546 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.547 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.547 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.547 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.547 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.547 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-21T23:28:36.547 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-21T23:28:36.547 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:36.547 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.548 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.548 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10", 2024-04-21T23:28:36.548 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.548 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.548 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.548 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.548 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.548 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.548 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.549 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.549 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-21T23:28:36.549 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-21T23:28:36.549 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:36.549 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.549 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.549 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "8", 2024-04-21T23:28:36.549 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.550 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.550 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.550 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.550 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.550 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.550 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.550 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.550 INFO:teuthology.orchestra.run.smithi040.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-21T23:28:36.551 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-21T23:28:36.551 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:36.551 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.551 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.551 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "45", 2024-04-21T23:28:36.551 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.551 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.551 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.552 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.552 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.552 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.552 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.552 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.552 INFO:teuthology.orchestra.run.smithi040.stdout: "RGW_API_ACCESS_KEY": { 2024-04-21T23:28:36.552 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-21T23:28:36.552 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.552 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.553 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.553 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.553 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.553 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.553 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.553 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.553 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.553 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.553 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.554 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.554 INFO:teuthology.orchestra.run.smithi040.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-21T23:28:36.554 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-21T23:28:36.554 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.554 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.554 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.554 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "admin", 2024-04-21T23:28:36.554 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.555 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.555 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.555 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.555 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.555 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.555 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.555 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.555 INFO:teuthology.orchestra.run.smithi040.stdout: "RGW_API_SECRET_KEY": { 2024-04-21T23:28:36.556 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-21T23:28:36.556 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.556 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.556 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.556 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.556 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.556 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.556 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.557 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.557 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.557 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.557 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.557 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.557 INFO:teuthology.orchestra.run.smithi040.stdout: "RGW_API_SSL_VERIFY": { 2024-04-21T23:28:36.557 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-21T23:28:36.557 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.557 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.558 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.558 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:36.558 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.558 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.558 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.558 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.558 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.558 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.559 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.559 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.559 INFO:teuthology.orchestra.run.smithi040.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-21T23:28:36.559 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-21T23:28:36.559 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:36.559 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.559 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.559 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "0", 2024-04-21T23:28:36.559 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.560 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.560 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.560 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.560 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.560 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.560 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.560 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.560 INFO:teuthology.orchestra.run.smithi040.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-21T23:28:36.560 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-21T23:28:36.561 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:36.561 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.561 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.561 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10", 2024-04-21T23:28:36.561 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.561 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.561 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.561 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.562 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.562 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.562 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.562 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.562 INFO:teuthology.orchestra.run.smithi040.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-21T23:28:36.562 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-21T23:28:36.562 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:36.562 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.563 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.563 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "5", 2024-04-21T23:28:36.563 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.563 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.563 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.563 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.563 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.563 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.563 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.564 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.564 INFO:teuthology.orchestra.run.smithi040.stdout: "cross_origin_url": { 2024-04-21T23:28:36.564 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "cross_origin_url", 2024-04-21T23:28:36.564 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.564 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.564 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.564 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.564 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.564 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.565 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.565 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.565 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.565 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.565 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.565 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.565 INFO:teuthology.orchestra.run.smithi040.stdout: "crt_file": { 2024-04-21T23:28:36.565 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "crt_file", 2024-04-21T23:28:36.565 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.566 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.566 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.566 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.566 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.566 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.566 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.566 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.566 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.566 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.567 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.567 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.567 INFO:teuthology.orchestra.run.smithi040.stdout: "debug": { 2024-04-21T23:28:36.567 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "debug", 2024-04-21T23:28:36.567 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.567 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.567 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.568 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.568 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.568 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.568 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.568 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Enable/disable debug options", 2024-04-21T23:28:36.568 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.568 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.568 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.568 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.569 INFO:teuthology.orchestra.run.smithi040.stdout: "jwt_token_ttl": { 2024-04-21T23:28:36.569 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "jwt_token_ttl", 2024-04-21T23:28:36.569 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:36.569 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.569 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.569 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "28800", 2024-04-21T23:28:36.569 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.569 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.570 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.570 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.570 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.570 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.570 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.570 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.570 INFO:teuthology.orchestra.run.smithi040.stdout: "key_file": { 2024-04-21T23:28:36.570 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "key_file", 2024-04-21T23:28:36.570 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.571 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.571 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.571 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.571 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.571 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.571 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.571 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.571 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.572 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.572 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.572 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.572 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:36.572 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:36.572 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.572 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.572 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.573 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.573 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.573 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.573 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:36.573 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:36.573 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:36.573 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:36.573 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:36.573 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:36.574 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:36.574 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:36.574 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.574 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.574 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.574 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.574 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.574 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:36.575 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:36.575 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.575 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.575 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.575 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.575 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.575 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.575 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.575 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.576 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.576 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.576 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.576 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.576 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:36.576 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:36.576 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.576 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.577 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.577 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:36.577 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.577 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.577 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:36.577 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:36.577 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:36.577 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:36.577 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:36.578 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:36.578 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:36.578 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:36.578 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.578 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.578 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.578 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.578 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.579 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:36.579 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:36.579 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.579 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.579 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.579 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.579 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.579 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.579 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.580 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.580 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.580 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.580 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.580 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.580 INFO:teuthology.orchestra.run.smithi040.stdout: "motd": { 2024-04-21T23:28:36.580 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "motd", 2024-04-21T23:28:36.580 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.580 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.581 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.581 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.581 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.581 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.581 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.581 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "The message of the day", 2024-04-21T23:28:36.581 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.581 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.581 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.582 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.582 INFO:teuthology.orchestra.run.smithi040.stdout: "redirect_resolve_ip_addr": { 2024-04-21T23:28:36.582 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "redirect_resolve_ip_addr", 2024-04-21T23:28:36.582 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.582 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.582 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.582 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.582 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.583 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.583 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.583 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.583 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.583 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.583 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.583 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.583 INFO:teuthology.orchestra.run.smithi040.stdout: "server_addr": { 2024-04-21T23:28:36.584 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "server_addr", 2024-04-21T23:28:36.584 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.584 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.584 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.584 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "::", 2024-04-21T23:28:36.584 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.584 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.584 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.584 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.585 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.585 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.585 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.585 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.585 INFO:teuthology.orchestra.run.smithi040.stdout: "server_port": { 2024-04-21T23:28:36.585 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "server_port", 2024-04-21T23:28:36.585 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:36.585 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.586 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.586 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "8080", 2024-04-21T23:28:36.586 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.586 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.586 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.586 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.586 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.586 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.586 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.587 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.587 INFO:teuthology.orchestra.run.smithi040.stdout: "ssl": { 2024-04-21T23:28:36.587 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ssl", 2024-04-21T23:28:36.587 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.587 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.587 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.587 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:36.587 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.588 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.588 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.588 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.588 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.588 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.588 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.588 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.588 INFO:teuthology.orchestra.run.smithi040.stdout: "ssl_server_port": { 2024-04-21T23:28:36.589 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ssl_server_port", 2024-04-21T23:28:36.589 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:36.589 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.589 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.589 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "8443", 2024-04-21T23:28:36.589 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.589 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.589 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.589 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.590 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.590 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.590 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.590 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.590 INFO:teuthology.orchestra.run.smithi040.stdout: "standby_behaviour": { 2024-04-21T23:28:36.590 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "standby_behaviour", 2024-04-21T23:28:36.590 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.591 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.591 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.591 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "redirect", 2024-04-21T23:28:36.591 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.591 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.591 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:36.591 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:36.591 INFO:teuthology.orchestra.run.smithi040.stdout: "redirect" 2024-04-21T23:28:36.591 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:36.592 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.592 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.592 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.592 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.592 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.592 INFO:teuthology.orchestra.run.smithi040.stdout: "standby_error_status_code": { 2024-04-21T23:28:36.592 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "standby_error_status_code", 2024-04-21T23:28:36.592 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:36.593 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.593 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.593 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "500", 2024-04-21T23:28:36.593 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "400", 2024-04-21T23:28:36.593 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "599", 2024-04-21T23:28:36.593 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.593 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.593 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.593 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.594 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.594 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.594 INFO:teuthology.orchestra.run.smithi040.stdout: "url_prefix": { 2024-04-21T23:28:36.594 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "url_prefix", 2024-04-21T23:28:36.594 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.594 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.594 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.594 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.594 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.595 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.595 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.595 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.595 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.595 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.595 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.595 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:36.595 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:36.596 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.596 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:36.596 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "devicehealth", 2024-04-21T23:28:36.596 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:36.596 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:36.596 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:36.596 INFO:teuthology.orchestra.run.smithi040.stdout: "enable_monitoring": { 2024-04-21T23:28:36.596 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "enable_monitoring", 2024-04-21T23:28:36.596 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.597 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.597 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.597 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:36.597 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.597 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.597 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.597 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "monitor device health metrics", 2024-04-21T23:28:36.597 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.597 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.598 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.598 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.598 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:36.598 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:36.598 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.598 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.598 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.598 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.598 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.599 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.599 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:36.599 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:36.599 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:36.599 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:36.599 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:36.599 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:36.599 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:36.599 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:36.600 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.600 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.600 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.600 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.600 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.600 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:36.600 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:36.600 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.600 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.601 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.601 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.601 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.601 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.601 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.601 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.601 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.601 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.602 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.602 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.602 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:36.602 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:36.602 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.602 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.602 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.602 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:36.602 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.603 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.603 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:36.603 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:36.603 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:36.603 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:36.603 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:36.603 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:36.603 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:36.603 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:36.604 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.604 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.604 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.604 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.604 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.604 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:36.604 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:36.604 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.605 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.605 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.605 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.605 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.605 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.605 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.605 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.605 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.605 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.606 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.606 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.606 INFO:teuthology.orchestra.run.smithi040.stdout: "mark_out_threshold": { 2024-04-21T23:28:36.606 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "mark_out_threshold", 2024-04-21T23:28:36.606 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-21T23:28:36.606 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.606 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.606 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "2419200", 2024-04-21T23:28:36.607 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.607 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.607 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.607 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-21T23:28:36.607 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.607 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.607 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.607 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.607 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_name": { 2024-04-21T23:28:36.608 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "pool_name", 2024-04-21T23:28:36.608 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.608 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.608 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.608 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "device_health_metrics", 2024-04-21T23:28:36.608 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.608 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.608 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.608 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-21T23:28:36.609 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.609 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.609 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.609 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.609 INFO:teuthology.orchestra.run.smithi040.stdout: "retention_period": { 2024-04-21T23:28:36.609 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "retention_period", 2024-04-21T23:28:36.609 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-21T23:28:36.609 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.609 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.610 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "15552000", 2024-04-21T23:28:36.610 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.610 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.610 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.610 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "how long to retain device health metrics", 2024-04-21T23:28:36.610 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.610 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.610 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.611 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.611 INFO:teuthology.orchestra.run.smithi040.stdout: "scrape_frequency": { 2024-04-21T23:28:36.611 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "scrape_frequency", 2024-04-21T23:28:36.611 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-21T23:28:36.611 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.611 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.611 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "86400", 2024-04-21T23:28:36.611 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.612 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.612 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.612 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-21T23:28:36.612 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.612 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.612 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.612 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.612 INFO:teuthology.orchestra.run.smithi040.stdout: "self_heal": { 2024-04-21T23:28:36.613 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "self_heal", 2024-04-21T23:28:36.613 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.613 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.613 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.613 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:36.613 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.613 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.613 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.614 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-21T23:28:36.614 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.614 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.614 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.614 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.614 INFO:teuthology.orchestra.run.smithi040.stdout: "sleep_interval": { 2024-04-21T23:28:36.614 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "sleep_interval", 2024-04-21T23:28:36.614 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-21T23:28:36.615 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.615 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.615 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "600", 2024-04-21T23:28:36.615 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.615 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.615 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.615 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "how frequently to wake up and check device health", 2024-04-21T23:28:36.615 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.615 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.616 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.616 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.616 INFO:teuthology.orchestra.run.smithi040.stdout: "warn_threshold": { 2024-04-21T23:28:36.616 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "warn_threshold", 2024-04-21T23:28:36.616 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-21T23:28:36.616 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.616 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.616 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "7257600", 2024-04-21T23:28:36.617 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.617 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.617 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.617 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-21T23:28:36.617 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.617 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.617 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.617 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:36.617 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:36.618 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.618 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:36.618 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "diskprediction_local", 2024-04-21T23:28:36.618 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:36.618 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:36.618 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:36.618 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:36.618 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:36.619 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.619 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.619 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.619 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.619 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.619 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.619 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:36.619 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:36.619 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:36.620 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:36.620 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:36.620 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:36.620 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:36.620 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:36.620 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.620 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.620 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.621 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.621 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.621 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:36.621 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:36.621 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.621 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.621 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.621 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.621 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.622 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.622 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.622 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.622 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.622 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.622 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.622 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.622 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:36.623 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:36.623 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.623 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.623 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.623 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:36.623 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.623 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.623 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:36.623 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:36.624 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:36.624 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:36.624 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:36.624 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:36.624 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:36.624 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:36.624 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.624 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.624 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.625 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.625 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.625 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:36.625 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:36.625 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.625 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.625 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.625 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.626 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.626 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.626 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.626 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.626 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.626 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.626 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.626 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.627 INFO:teuthology.orchestra.run.smithi040.stdout: "predict_interval": { 2024-04-21T23:28:36.627 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "predict_interval", 2024-04-21T23:28:36.627 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.627 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.627 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.627 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "86400", 2024-04-21T23:28:36.627 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.627 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.627 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.628 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.628 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.628 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.628 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.628 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.628 INFO:teuthology.orchestra.run.smithi040.stdout: "predictor_model": { 2024-04-21T23:28:36.628 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "predictor_model", 2024-04-21T23:28:36.628 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.628 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.629 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.629 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "prophetstor", 2024-04-21T23:28:36.629 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.629 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.629 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.629 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.629 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.629 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.629 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.630 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.630 INFO:teuthology.orchestra.run.smithi040.stdout: "sleep_interval": { 2024-04-21T23:28:36.630 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "sleep_interval", 2024-04-21T23:28:36.630 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.630 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.630 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.630 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "600", 2024-04-21T23:28:36.631 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.631 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.631 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.631 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.631 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.631 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.631 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.631 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:36.631 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:36.631 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.632 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:36.632 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "influx", 2024-04-21T23:28:36.632 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": false, 2024-04-21T23:28:36.632 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "influxdb python module not found", 2024-04-21T23:28:36.632 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:36.632 INFO:teuthology.orchestra.run.smithi040.stdout: "batch_size": { 2024-04-21T23:28:36.632 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "batch_size", 2024-04-21T23:28:36.632 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:36.632 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.632 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.633 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "5000", 2024-04-21T23:28:36.633 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.633 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.633 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.633 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-21T23:28:36.633 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.633 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.633 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.634 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.634 INFO:teuthology.orchestra.run.smithi040.stdout: "database": { 2024-04-21T23:28:36.634 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "database", 2024-04-21T23:28:36.634 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.634 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.634 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.634 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "ceph", 2024-04-21T23:28:36.634 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.635 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.635 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.637 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-21T23:28:36.637 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.637 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.638 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.638 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.638 INFO:teuthology.orchestra.run.smithi040.stdout: "hostname": { 2024-04-21T23:28:36.638 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "hostname", 2024-04-21T23:28:36.638 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.638 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.638 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.638 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.639 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.639 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.639 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.639 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "InfluxDB server hostname", 2024-04-21T23:28:36.639 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.639 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.639 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.639 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.640 INFO:teuthology.orchestra.run.smithi040.stdout: "interval": { 2024-04-21T23:28:36.640 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "interval", 2024-04-21T23:28:36.640 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-21T23:28:36.640 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.640 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.640 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "30", 2024-04-21T23:28:36.640 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "5", 2024-04-21T23:28:36.640 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.640 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.641 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-21T23:28:36.641 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.641 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.641 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.641 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.641 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:36.641 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:36.641 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.641 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.642 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.642 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.642 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.642 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.642 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:36.642 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:36.642 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:36.642 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:36.643 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:36.643 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:36.643 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:36.643 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:36.643 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.643 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.643 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.643 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.643 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.644 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:36.644 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:36.644 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.644 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.644 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.644 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.644 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.644 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.644 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.645 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.645 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.645 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.645 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.645 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.645 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:36.645 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:36.645 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.645 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.646 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.646 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:36.646 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.646 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.646 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:36.646 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:36.646 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:36.646 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:36.647 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:36.647 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:36.647 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:36.647 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:36.647 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.647 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.647 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.647 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.647 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.647 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:36.648 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:36.648 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.648 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.648 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.648 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.648 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.648 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.648 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.649 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.649 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.649 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.649 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.649 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.649 INFO:teuthology.orchestra.run.smithi040.stdout: "password": { 2024-04-21T23:28:36.649 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "password", 2024-04-21T23:28:36.649 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.649 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.650 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.650 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.650 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.650 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.650 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.650 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "password of InfluxDB server user", 2024-04-21T23:28:36.650 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.650 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.650 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.651 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.651 INFO:teuthology.orchestra.run.smithi040.stdout: "port": { 2024-04-21T23:28:36.651 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "port", 2024-04-21T23:28:36.651 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:36.651 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.651 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.651 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "8086", 2024-04-21T23:28:36.652 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.652 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.652 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.652 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "InfluxDB server port", 2024-04-21T23:28:36.652 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.652 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.652 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.652 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.652 INFO:teuthology.orchestra.run.smithi040.stdout: "ssl": { 2024-04-21T23:28:36.653 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ssl", 2024-04-21T23:28:36.653 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.653 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.653 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.653 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "false", 2024-04-21T23:28:36.653 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.653 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.653 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.653 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-21T23:28:36.654 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.654 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.654 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.654 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.654 INFO:teuthology.orchestra.run.smithi040.stdout: "threads": { 2024-04-21T23:28:36.654 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "threads", 2024-04-21T23:28:36.654 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:36.654 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.654 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.655 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "5", 2024-04-21T23:28:36.655 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "1", 2024-04-21T23:28:36.655 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "32", 2024-04-21T23:28:36.655 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.655 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-21T23:28:36.655 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.655 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.655 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.656 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.656 INFO:teuthology.orchestra.run.smithi040.stdout: "username": { 2024-04-21T23:28:36.656 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "username", 2024-04-21T23:28:36.656 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.656 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.656 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.656 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.656 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.656 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.657 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.657 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "username of InfluxDB server user", 2024-04-21T23:28:36.657 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.657 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.657 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.657 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.657 INFO:teuthology.orchestra.run.smithi040.stdout: "verify_ssl": { 2024-04-21T23:28:36.657 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "verify_ssl", 2024-04-21T23:28:36.658 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.658 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.658 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.658 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "true", 2024-04-21T23:28:36.658 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.658 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.658 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.658 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-21T23:28:36.658 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.659 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.659 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.659 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:36.659 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:36.659 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.659 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:36.659 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "insights", 2024-04-21T23:28:36.659 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:36.659 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:36.660 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:36.660 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:36.660 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:36.660 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.660 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.660 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.660 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.660 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.660 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.661 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:36.661 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:36.661 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:36.661 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:36.661 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:36.661 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:36.661 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:36.661 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:36.662 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.662 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.662 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.662 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.662 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.662 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:36.662 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:36.662 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.662 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.663 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.663 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.663 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.663 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.663 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.663 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.663 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.663 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.663 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.664 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.664 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:36.664 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:36.664 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.664 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.664 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.664 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:36.664 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.664 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.665 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:36.665 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:36.665 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:36.665 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:36.665 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:36.665 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:36.665 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:36.665 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:36.665 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.666 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.666 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.666 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.666 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.666 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:36.666 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:36.666 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.666 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.667 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.667 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.667 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.667 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.667 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.667 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.667 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.667 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.667 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.667 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:36.668 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:36.668 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.668 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:36.668 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "iostat", 2024-04-21T23:28:36.668 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:36.668 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:36.668 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:36.668 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:36.669 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:36.669 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.669 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.669 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.669 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.669 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.669 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.669 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:36.669 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:36.670 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:36.670 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:36.670 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:36.670 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:36.670 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:36.670 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:36.670 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.670 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.671 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.671 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.671 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.671 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:36.671 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:36.671 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.671 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.671 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.671 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.672 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.672 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.672 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.672 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.672 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.672 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.672 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.672 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.672 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:36.673 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:36.673 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.673 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.673 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.673 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:36.673 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.673 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.673 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:36.673 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:36.674 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:36.674 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:36.674 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:36.674 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:36.674 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:36.674 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:36.674 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.674 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.675 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.675 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.675 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.675 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:36.675 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:36.675 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.675 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.675 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.675 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.676 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.676 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.676 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.676 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.676 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.676 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.676 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.676 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:36.676 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:36.677 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.677 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:36.677 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "k8sevents", 2024-04-21T23:28:36.677 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:36.677 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:36.677 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:36.677 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_event_retention_days": { 2024-04-21T23:28:36.677 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ceph_event_retention_days", 2024-04-21T23:28:36.678 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:36.678 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.678 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.678 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "7", 2024-04-21T23:28:36.678 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.678 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.678 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.678 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-21T23:28:36.679 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.679 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.679 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.679 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.679 INFO:teuthology.orchestra.run.smithi040.stdout: "config_check_secs": { 2024-04-21T23:28:36.679 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "config_check_secs", 2024-04-21T23:28:36.679 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:36.679 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.679 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.680 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10", 2024-04-21T23:28:36.680 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "10", 2024-04-21T23:28:36.680 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.680 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.680 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-21T23:28:36.680 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.680 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.680 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.681 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.681 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:36.681 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:36.681 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.681 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.681 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.681 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.681 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.682 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.682 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:36.682 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:36.682 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:36.682 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:36.682 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:36.682 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:36.682 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:36.683 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:36.683 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.683 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.683 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.683 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.683 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.683 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:36.683 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:36.683 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.684 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.684 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.684 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.684 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.684 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.684 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.684 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.684 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.684 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.685 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.685 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.685 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:36.685 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:36.685 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.685 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.685 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.685 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:36.685 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.686 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.686 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:36.686 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:36.686 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:36.686 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:36.686 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:36.686 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:36.686 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:36.687 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:36.687 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.687 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.687 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.687 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.687 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.687 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:36.687 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:36.687 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.688 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.688 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.688 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.688 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.688 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.688 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.688 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.688 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.688 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.689 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.689 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:36.689 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:36.689 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.689 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:36.689 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "localpool", 2024-04-21T23:28:36.689 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:36.689 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:36.689 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:36.690 INFO:teuthology.orchestra.run.smithi040.stdout: "failure_domain": { 2024-04-21T23:28:36.690 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "failure_domain", 2024-04-21T23:28:36.690 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.690 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.690 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.690 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "host", 2024-04-21T23:28:36.690 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.690 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.691 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.691 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "failure domain for any created local pool", 2024-04-21T23:28:36.691 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-21T23:28:36.691 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.691 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.691 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.691 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:36.691 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:36.691 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.692 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.692 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.692 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.692 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.692 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.692 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:36.692 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:36.692 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:36.692 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:36.693 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:36.693 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:36.693 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:36.693 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:36.693 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.693 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.693 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.693 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.694 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.694 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:36.694 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:36.694 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.694 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.694 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.694 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.694 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.695 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.695 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.695 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.695 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.695 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.695 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.695 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.695 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:36.695 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:36.696 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.696 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.696 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.696 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:36.696 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.696 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.696 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:36.696 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:36.696 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:36.697 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:36.697 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:36.697 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:36.697 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:36.697 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:36.697 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.697 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.697 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.697 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.698 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.698 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:36.698 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:36.698 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.698 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.698 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.698 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.699 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.699 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.699 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.699 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.699 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.699 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.699 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.699 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.699 INFO:teuthology.orchestra.run.smithi040.stdout: "min_size": { 2024-04-21T23:28:36.700 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "min_size", 2024-04-21T23:28:36.700 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:36.700 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.700 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.700 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.700 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.700 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.700 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.700 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "default min_size for any created local pool", 2024-04-21T23:28:36.701 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-21T23:28:36.701 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.701 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.701 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.701 INFO:teuthology.orchestra.run.smithi040.stdout: "num_rep": { 2024-04-21T23:28:36.701 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "num_rep", 2024-04-21T23:28:36.701 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:36.701 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.701 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.702 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "3", 2024-04-21T23:28:36.702 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.702 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.702 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.702 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "default replica count for any created local pool", 2024-04-21T23:28:36.702 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.702 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.702 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.703 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.703 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num": { 2024-04-21T23:28:36.703 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "pg_num", 2024-04-21T23:28:36.703 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:36.703 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.703 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.703 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "128", 2024-04-21T23:28:36.703 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.704 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.704 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.704 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "default pg_num for any created local pool", 2024-04-21T23:28:36.704 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.704 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.704 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.704 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.704 INFO:teuthology.orchestra.run.smithi040.stdout: "prefix": { 2024-04-21T23:28:36.704 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "prefix", 2024-04-21T23:28:36.705 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.705 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.705 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.705 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.705 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.705 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.705 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.705 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "name prefix for any created local pool", 2024-04-21T23:28:36.705 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.706 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.706 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.706 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.706 INFO:teuthology.orchestra.run.smithi040.stdout: "subtree": { 2024-04-21T23:28:36.706 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "subtree", 2024-04-21T23:28:36.706 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.706 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.706 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.707 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "rack", 2024-04-21T23:28:36.707 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.707 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.707 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.707 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-21T23:28:36.707 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-21T23:28:36.707 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.707 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.707 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:36.708 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:36.708 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.708 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:36.708 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "mds_autoscaler", 2024-04-21T23:28:36.708 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:36.708 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:36.708 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:36.708 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:36.708 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:36.709 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.709 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.709 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.709 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.709 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.709 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.709 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:36.709 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:36.710 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:36.710 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:36.710 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:36.710 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:36.710 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:36.710 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:36.710 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.710 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.710 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.711 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.711 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.711 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:36.711 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:36.711 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.711 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.711 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.711 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.712 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.712 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.712 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.712 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.712 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.712 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.712 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.712 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.712 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:36.713 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:36.713 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.713 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.713 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.713 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:36.713 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.713 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.713 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:36.713 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:36.714 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:36.714 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:36.714 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:36.714 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:36.714 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:36.714 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:36.714 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.714 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.715 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.715 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.715 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.715 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:36.715 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:36.715 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.715 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.715 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.716 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.716 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.716 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.716 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.716 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.716 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.716 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.716 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.717 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:36.717 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:36.717 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.717 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:36.717 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "mirroring", 2024-04-21T23:28:36.717 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:36.717 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:36.717 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:36.717 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:36.717 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:36.718 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.718 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.718 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.718 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.718 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.718 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.718 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:36.718 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:36.719 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:36.719 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:36.719 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:36.719 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:36.719 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:36.719 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:36.719 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.719 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.719 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.720 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.720 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.720 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:36.720 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:36.720 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.720 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.720 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.720 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.720 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.721 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.721 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.721 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.721 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.721 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.721 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.721 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.721 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:36.721 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:36.722 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.722 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.722 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.722 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:36.722 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.722 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.722 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:36.722 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:36.723 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:36.723 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:36.723 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:36.723 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:36.723 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:36.723 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:36.723 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.723 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.723 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.724 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.724 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.724 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:36.724 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:36.724 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.724 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.724 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.724 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.725 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.725 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.725 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.725 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.725 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.725 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.725 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.725 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:36.725 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:36.726 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.726 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:36.726 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "nfs", 2024-04-21T23:28:36.726 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:36.726 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:36.726 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:36.726 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:36.726 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:36.727 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.727 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.727 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.727 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.727 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.727 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.727 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:36.727 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:36.728 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:36.728 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:36.728 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:36.728 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:36.728 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:36.728 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:36.728 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.728 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.728 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.729 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.729 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.729 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:36.729 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:36.729 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.729 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.729 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.729 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.730 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.730 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.730 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.730 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.730 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.730 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.730 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.730 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.731 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:36.731 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:36.731 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.731 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.731 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.731 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:36.731 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.731 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.731 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:36.732 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:36.732 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:36.732 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:36.732 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:36.732 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:36.732 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:36.732 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:36.732 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.733 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.733 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.733 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.733 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.733 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:36.733 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:36.733 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.733 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.733 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.734 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.734 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.734 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.734 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.734 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.734 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.734 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.734 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.735 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:36.735 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:36.735 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.735 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:36.735 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "orchestrator", 2024-04-21T23:28:36.735 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:36.735 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:36.735 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:36.735 INFO:teuthology.orchestra.run.smithi040.stdout: "fail_fs": { 2024-04-21T23:28:36.735 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "fail_fs", 2024-04-21T23:28:36.736 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.736 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.736 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.736 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.736 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.736 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.736 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.736 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-21T23:28:36.737 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.737 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.737 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.737 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.737 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:36.737 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:36.737 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.737 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.737 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.737 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.738 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.738 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.738 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:36.738 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:36.738 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:36.738 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:36.738 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:36.738 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:36.739 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:36.739 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:36.739 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.739 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.739 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.739 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.739 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.739 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:36.739 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:36.740 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.740 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.740 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.740 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.740 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.740 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.740 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.740 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.741 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.741 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.741 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.741 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.741 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:36.741 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:36.741 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.741 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.741 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.742 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:36.742 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.742 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.742 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:36.742 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:36.742 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:36.742 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:36.742 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:36.743 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:36.743 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:36.743 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:36.743 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.743 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.743 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.743 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.743 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.743 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:36.744 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:36.744 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.744 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.744 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.744 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.744 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.744 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.744 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.744 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.745 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.745 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.745 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.745 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.745 INFO:teuthology.orchestra.run.smithi040.stdout: "orchestrator": { 2024-04-21T23:28:36.745 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "orchestrator", 2024-04-21T23:28:36.745 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.745 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.745 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.746 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.746 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.746 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.746 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:36.746 INFO:teuthology.orchestra.run.smithi040.stdout: "cephadm", 2024-04-21T23:28:36.746 INFO:teuthology.orchestra.run.smithi040.stdout: "rook", 2024-04-21T23:28:36.746 INFO:teuthology.orchestra.run.smithi040.stdout: "test_orchestrator" 2024-04-21T23:28:36.746 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:36.747 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Orchestrator backend", 2024-04-21T23:28:36.747 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.747 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.747 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.747 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:36.747 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:36.747 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.747 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:36.748 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "osd_perf_query", 2024-04-21T23:28:36.748 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:36.748 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:36.748 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:36.748 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:36.748 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:36.748 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.748 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.748 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.749 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.749 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.749 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.749 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:36.749 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:36.749 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:36.749 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:36.749 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:36.750 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:36.750 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:36.750 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:36.750 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.750 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.750 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.750 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.750 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.751 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:36.751 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:36.751 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.751 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.751 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.751 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.751 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.751 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.751 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.752 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.752 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.752 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.752 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.752 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.752 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:36.752 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:36.752 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.752 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.753 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.753 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:36.753 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.753 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.753 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:36.753 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:36.753 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:36.753 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:36.754 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:36.754 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:36.754 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:36.754 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:36.754 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.754 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.754 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.754 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.754 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.755 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:36.755 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:36.755 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.755 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.755 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.755 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.755 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.755 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.756 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.756 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.756 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.756 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.756 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.756 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:36.756 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:36.756 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.756 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:36.757 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "osd_support", 2024-04-21T23:28:36.757 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:36.757 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:36.757 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:36.757 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:36.757 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:36.757 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.757 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.757 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.758 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.758 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.758 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.758 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:36.758 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:36.758 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:36.758 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:36.758 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:36.759 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:36.759 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:36.759 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:36.759 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.759 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.759 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.759 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.759 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.759 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:36.760 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:36.760 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.760 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.760 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.760 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.760 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.760 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.760 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.760 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.761 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.761 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.761 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.761 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.761 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:36.761 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:36.761 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.761 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.762 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.762 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:36.762 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.762 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.762 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:36.762 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:36.762 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:36.762 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:36.762 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:36.763 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:36.763 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:36.763 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:36.763 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.763 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.763 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.763 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.763 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.764 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:36.764 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:36.764 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.764 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.764 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.764 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.764 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.764 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.764 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.765 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.765 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.765 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.765 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.765 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:36.765 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:36.765 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.765 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:36.765 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "pg_autoscaler", 2024-04-21T23:28:36.766 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:36.766 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:36.766 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:36.766 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:36.766 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:36.766 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.766 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.766 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.767 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.767 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.767 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.767 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:36.767 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:36.767 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:36.767 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:36.767 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:36.767 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:36.768 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:36.768 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:36.768 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.768 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.768 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.768 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.768 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.768 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:36.768 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:36.769 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.769 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.769 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.769 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.769 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.769 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.769 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.769 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.770 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.770 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.770 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.770 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.770 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:36.770 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:36.770 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.771 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.771 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.771 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:36.771 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.771 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.771 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:36.771 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:36.771 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:36.771 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:36.772 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:36.772 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:36.772 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:36.772 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:36.772 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.772 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.772 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.772 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.772 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.773 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:36.773 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:36.773 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.773 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.773 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.773 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.773 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.773 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.773 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.774 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.774 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.774 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.774 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.774 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.774 INFO:teuthology.orchestra.run.smithi040.stdout: "sleep_interval": { 2024-04-21T23:28:36.774 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "sleep_interval", 2024-04-21T23:28:36.774 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-21T23:28:36.775 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.775 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.775 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "60", 2024-04-21T23:28:36.775 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.775 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.775 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.775 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.775 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.775 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.776 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.776 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.776 INFO:teuthology.orchestra.run.smithi040.stdout: "threshold": { 2024-04-21T23:28:36.776 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "threshold", 2024-04-21T23:28:36.776 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "float", 2024-04-21T23:28:36.776 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.776 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.776 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "3.0", 2024-04-21T23:28:36.777 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "1.0", 2024-04-21T23:28:36.777 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.777 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.777 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "scaling threshold", 2024-04-21T23:28:36.777 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-21T23:28:36.777 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.777 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.777 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:36.777 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:36.778 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.778 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:36.778 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "progress", 2024-04-21T23:28:36.778 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:36.778 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:36.778 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:36.778 INFO:teuthology.orchestra.run.smithi040.stdout: "allow_pg_recovery_event": { 2024-04-21T23:28:36.778 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "allow_pg_recovery_event", 2024-04-21T23:28:36.779 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.779 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.779 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.779 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.779 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.779 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.779 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.779 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-21T23:28:36.779 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.780 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.780 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.780 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.780 INFO:teuthology.orchestra.run.smithi040.stdout: "enabled": { 2024-04-21T23:28:36.780 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "enabled", 2024-04-21T23:28:36.780 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.780 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.780 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.781 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:36.781 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.781 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.781 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.781 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.781 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.781 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.781 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.781 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.782 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:36.782 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:36.782 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.782 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.782 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.782 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.782 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.782 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.782 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:36.783 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:36.783 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:36.783 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:36.783 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:36.783 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:36.783 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:36.783 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:36.783 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.783 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.784 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.784 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.784 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.784 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:36.784 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:36.784 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.784 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.784 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.784 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.785 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.785 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.785 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.785 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.785 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.785 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.785 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.785 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.785 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:36.786 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:36.786 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.786 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.786 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.786 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:36.786 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.786 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.786 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:36.787 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:36.787 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:36.787 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:36.787 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:36.787 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:36.787 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:36.787 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:36.787 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.788 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.788 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.788 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.788 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.788 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:36.788 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:36.788 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.788 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.788 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.789 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.789 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.789 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.789 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.789 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.789 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.789 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.789 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.789 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.790 INFO:teuthology.orchestra.run.smithi040.stdout: "max_completed_events": { 2024-04-21T23:28:36.790 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "max_completed_events", 2024-04-21T23:28:36.790 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:36.790 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.790 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.790 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "50", 2024-04-21T23:28:36.790 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.790 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.790 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.791 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "number of past completed events to remember", 2024-04-21T23:28:36.791 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.791 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.791 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.791 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.791 INFO:teuthology.orchestra.run.smithi040.stdout: "sleep_interval": { 2024-04-21T23:28:36.791 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "sleep_interval", 2024-04-21T23:28:36.791 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-21T23:28:36.791 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.792 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.792 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "5", 2024-04-21T23:28:36.792 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.792 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.792 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.792 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "how long the module is going to sleep", 2024-04-21T23:28:36.792 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.792 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.793 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.793 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:36.793 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:36.793 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.793 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:36.793 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "prometheus", 2024-04-21T23:28:36.793 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:36.793 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:36.794 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:36.794 INFO:teuthology.orchestra.run.smithi040.stdout: "cache": { 2024-04-21T23:28:36.794 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "cache", 2024-04-21T23:28:36.794 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.794 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.794 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.794 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:36.794 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.794 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.795 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.795 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.795 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.795 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.795 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.795 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.795 INFO:teuthology.orchestra.run.smithi040.stdout: "exclude_perf_counters": { 2024-04-21T23:28:36.795 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "exclude_perf_counters", 2024-04-21T23:28:36.795 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.796 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.796 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.796 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:36.796 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.796 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.796 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.796 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-21T23:28:36.796 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-21T23:28:36.797 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.797 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.797 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.797 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:36.797 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:36.797 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.797 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.797 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.797 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.798 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.798 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.798 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:36.798 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:36.798 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:36.798 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:36.798 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:36.798 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:36.798 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:36.799 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:36.799 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.799 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.799 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.799 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.799 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.799 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:36.799 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:36.799 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.800 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.800 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.800 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.800 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.800 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.800 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.800 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.800 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.800 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.801 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.801 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.801 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:36.801 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:36.801 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.801 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.801 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.801 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:36.802 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.802 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.802 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:36.802 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:36.802 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:36.802 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:36.802 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:36.802 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:36.803 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:36.803 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:36.803 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.803 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.803 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.803 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.803 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.803 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:36.803 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:36.804 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.804 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.804 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.804 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.804 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.804 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.804 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.804 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.805 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.805 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.805 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.805 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.805 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_stats_pools": { 2024-04-21T23:28:36.805 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rbd_stats_pools", 2024-04-21T23:28:36.805 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.805 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.805 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.806 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.806 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.806 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.806 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.806 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.806 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.806 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.806 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.807 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.807 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-21T23:28:36.807 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-21T23:28:36.807 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:36.807 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.807 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.807 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "300", 2024-04-21T23:28:36.807 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.808 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.808 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.808 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.808 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.808 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.808 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.808 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.808 INFO:teuthology.orchestra.run.smithi040.stdout: "scrape_interval": { 2024-04-21T23:28:36.809 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "scrape_interval", 2024-04-21T23:28:36.809 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "float", 2024-04-21T23:28:36.809 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.809 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.809 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "15.0", 2024-04-21T23:28:36.809 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.809 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.809 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.809 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.810 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.810 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.810 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.810 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.810 INFO:teuthology.orchestra.run.smithi040.stdout: "server_addr": { 2024-04-21T23:28:36.810 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "server_addr", 2024-04-21T23:28:36.810 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.810 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.811 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.811 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "::", 2024-04-21T23:28:36.811 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.811 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.811 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.811 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-21T23:28:36.811 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.811 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.811 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.812 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.812 INFO:teuthology.orchestra.run.smithi040.stdout: "server_port": { 2024-04-21T23:28:36.812 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "server_port", 2024-04-21T23:28:36.812 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:36.812 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.812 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.812 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "9283", 2024-04-21T23:28:36.812 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.812 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.813 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.813 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-21T23:28:36.813 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.813 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.813 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.813 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.813 INFO:teuthology.orchestra.run.smithi040.stdout: "stale_cache_strategy": { 2024-04-21T23:28:36.813 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "stale_cache_strategy", 2024-04-21T23:28:36.814 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.814 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.814 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.814 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "log", 2024-04-21T23:28:36.814 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.814 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.814 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.814 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.815 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.815 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.815 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.815 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.815 INFO:teuthology.orchestra.run.smithi040.stdout: "standby_behaviour": { 2024-04-21T23:28:36.815 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "standby_behaviour", 2024-04-21T23:28:36.815 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.815 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.815 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.816 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "default", 2024-04-21T23:28:36.816 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.816 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.816 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:36.816 INFO:teuthology.orchestra.run.smithi040.stdout: "default", 2024-04-21T23:28:36.816 INFO:teuthology.orchestra.run.smithi040.stdout: "error" 2024-04-21T23:28:36.816 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:36.816 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.816 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.817 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.817 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.817 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.817 INFO:teuthology.orchestra.run.smithi040.stdout: "standby_error_status_code": { 2024-04-21T23:28:36.817 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "standby_error_status_code", 2024-04-21T23:28:36.817 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:36.817 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.817 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.818 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "500", 2024-04-21T23:28:36.818 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "400", 2024-04-21T23:28:36.818 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "599", 2024-04-21T23:28:36.818 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.818 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.818 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.818 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.818 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.818 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:36.819 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:36.819 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.819 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:36.819 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rbd_support", 2024-04-21T23:28:36.819 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:36.819 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:36.819 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:36.819 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:36.819 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:36.820 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.820 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.820 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.820 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.820 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.820 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.820 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:36.820 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:36.820 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:36.821 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:36.821 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:36.821 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:36.821 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:36.821 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:36.821 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.821 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.821 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.821 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.822 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.822 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:36.822 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:36.822 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.822 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.822 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.822 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.822 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.823 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.823 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.823 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.823 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.823 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.823 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.823 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.823 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:36.823 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:36.824 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.824 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.824 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.824 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:36.824 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.824 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.824 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:36.824 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:36.824 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:36.825 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:36.825 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:36.825 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:36.825 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:36.825 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:36.825 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.825 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.825 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.825 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.826 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.826 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:36.826 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:36.826 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.826 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.826 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.826 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.826 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.827 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.827 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.827 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.827 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.827 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.827 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.827 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.827 INFO:teuthology.orchestra.run.smithi040.stdout: "max_concurrent_snap_create": { 2024-04-21T23:28:36.828 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "max_concurrent_snap_create", 2024-04-21T23:28:36.828 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:36.828 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.828 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.828 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10", 2024-04-21T23:28:36.828 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.828 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.828 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.828 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.829 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.829 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.829 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.829 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.829 INFO:teuthology.orchestra.run.smithi040.stdout: "mirror_snapshot_schedule": { 2024-04-21T23:28:36.829 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "mirror_snapshot_schedule", 2024-04-21T23:28:36.829 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.829 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.830 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.830 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.830 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.830 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.830 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.830 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.830 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.830 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.831 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.831 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.831 INFO:teuthology.orchestra.run.smithi040.stdout: "trash_purge_schedule": { 2024-04-21T23:28:36.831 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "trash_purge_schedule", 2024-04-21T23:28:36.831 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.831 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.831 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.831 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.832 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.832 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.832 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.832 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.832 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.832 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.832 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.832 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:36.832 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:36.833 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.833 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:36.833 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "restful", 2024-04-21T23:28:36.833 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:36.833 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:36.833 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:36.833 INFO:teuthology.orchestra.run.smithi040.stdout: "enable_auth": { 2024-04-21T23:28:36.833 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "enable_auth", 2024-04-21T23:28:36.834 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.834 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.834 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.834 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:36.834 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.834 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.834 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.834 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.835 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.835 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.835 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.835 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.835 INFO:teuthology.orchestra.run.smithi040.stdout: "key_file": { 2024-04-21T23:28:36.835 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "key_file", 2024-04-21T23:28:36.835 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.835 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.836 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.836 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.836 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.836 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.836 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.836 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.836 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.837 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.837 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.837 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.837 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:36.837 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:36.837 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.837 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.837 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.838 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.838 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.838 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.838 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:36.838 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:36.838 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:36.838 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:36.838 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:36.838 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:36.839 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:36.839 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:36.839 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.839 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.839 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.839 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.839 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.839 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:36.840 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:36.840 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.841 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.841 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.841 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.842 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.842 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.842 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.842 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.842 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.842 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.842 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.842 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.843 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:36.843 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:36.843 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.843 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.843 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.843 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:36.843 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.843 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.844 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:36.844 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:36.844 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:36.844 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:36.844 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:36.844 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:36.844 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:36.844 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:36.844 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.845 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.845 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.845 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.845 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.845 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:36.845 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:36.845 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.845 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.845 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.846 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.846 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.846 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.846 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.846 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.846 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.846 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.846 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.847 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.847 INFO:teuthology.orchestra.run.smithi040.stdout: "server_addr": { 2024-04-21T23:28:36.847 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "server_addr", 2024-04-21T23:28:36.847 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.847 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.847 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.847 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.847 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.848 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.848 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.848 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.848 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.848 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.848 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.848 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.848 INFO:teuthology.orchestra.run.smithi040.stdout: "server_port": { 2024-04-21T23:28:36.849 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "server_port", 2024-04-21T23:28:36.849 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.849 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.849 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.849 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.849 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.849 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.849 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.850 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.850 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.850 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.850 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.850 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:36.850 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:36.850 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.851 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:36.851 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rgw", 2024-04-21T23:28:36.851 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:36.851 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:36.851 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:36.851 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:36.851 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:36.851 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.851 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.852 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.852 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.852 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.852 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.852 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:36.852 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:36.852 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:36.852 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:36.853 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:36.853 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:36.853 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:36.853 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:36.853 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.853 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.853 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.853 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.854 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.854 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:36.854 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:36.854 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.854 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.854 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.854 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.854 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.854 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.855 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.855 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.855 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.855 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.855 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.855 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.855 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:36.855 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:36.856 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.856 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.856 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.856 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:36.856 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.856 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.856 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:36.856 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:36.856 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:36.857 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:36.857 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:36.857 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:36.857 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:36.857 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:36.857 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.857 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.857 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.857 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.858 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.858 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:36.858 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:36.858 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.858 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.858 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.858 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.858 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.858 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.859 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.859 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.859 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.859 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.859 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.859 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:36.859 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:36.859 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.859 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:36.860 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rook", 2024-04-21T23:28:36.860 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:36.860 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:36.860 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:36.860 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:36.860 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:36.860 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.860 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.860 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.861 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.861 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.861 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.861 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:36.861 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:36.861 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:36.861 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:36.861 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:36.862 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:36.862 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:36.862 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:36.862 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.862 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.862 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.862 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.862 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.863 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:36.863 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:36.863 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.863 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.863 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.863 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.863 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.863 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.863 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.864 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.864 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.864 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.864 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.864 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.864 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:36.864 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:36.864 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.865 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.865 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.865 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:36.865 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.865 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.865 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:36.865 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:36.865 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:36.865 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:36.866 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:36.866 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:36.866 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:36.866 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:36.866 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.866 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.866 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.866 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.867 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.867 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:36.867 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:36.867 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.867 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.867 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.867 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.867 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.867 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.868 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.868 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.868 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.868 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.868 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.868 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.868 INFO:teuthology.orchestra.run.smithi040.stdout: "storage_class": { 2024-04-21T23:28:36.868 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "storage_class", 2024-04-21T23:28:36.868 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.869 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.869 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.869 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "local", 2024-04-21T23:28:36.869 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.869 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.869 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.869 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-21T23:28:36.869 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.869 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.870 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.870 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:36.870 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:36.870 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.870 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:36.870 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "selftest", 2024-04-21T23:28:36.870 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:36.870 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:36.871 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:36.871 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:36.871 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:36.871 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.871 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.871 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.871 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.871 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.871 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.872 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:36.872 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:36.872 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:36.872 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:36.872 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:36.872 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:36.872 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:36.872 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:36.873 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.873 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.873 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.873 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.873 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.873 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:36.873 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:36.873 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.873 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.874 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.874 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.874 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.874 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.874 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.874 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.874 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.874 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.875 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.875 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.875 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:36.875 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:36.875 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.875 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.875 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.875 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:36.875 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.876 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.876 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:36.876 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:36.876 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:36.876 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:36.876 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:36.876 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:36.876 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:36.876 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:36.877 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.877 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.877 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.877 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.877 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.877 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:36.877 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:36.877 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.878 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.878 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.878 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.878 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.878 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.878 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.878 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.878 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.878 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.879 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.879 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.879 INFO:teuthology.orchestra.run.smithi040.stdout: "roption1": { 2024-04-21T23:28:36.879 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "roption1", 2024-04-21T23:28:36.879 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.879 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.879 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.879 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.879 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.880 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.880 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.880 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.880 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.880 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.880 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.880 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.880 INFO:teuthology.orchestra.run.smithi040.stdout: "roption2": { 2024-04-21T23:28:36.880 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "roption2", 2024-04-21T23:28:36.881 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.881 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.881 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.881 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "xyz", 2024-04-21T23:28:36.881 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.881 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.881 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.881 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.881 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.882 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.882 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.882 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.882 INFO:teuthology.orchestra.run.smithi040.stdout: "rwoption1": { 2024-04-21T23:28:36.882 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rwoption1", 2024-04-21T23:28:36.882 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.882 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.882 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.882 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.883 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.883 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.883 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.883 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.883 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.883 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.883 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.883 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.883 INFO:teuthology.orchestra.run.smithi040.stdout: "rwoption2": { 2024-04-21T23:28:36.884 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rwoption2", 2024-04-21T23:28:36.884 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:36.884 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.884 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.884 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.884 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.884 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.884 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.885 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.885 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.885 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.885 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.885 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.885 INFO:teuthology.orchestra.run.smithi040.stdout: "rwoption3": { 2024-04-21T23:28:36.885 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rwoption3", 2024-04-21T23:28:36.885 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "float", 2024-04-21T23:28:36.885 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.886 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.886 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.886 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.886 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.886 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.886 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.886 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.886 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.887 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.887 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.887 INFO:teuthology.orchestra.run.smithi040.stdout: "rwoption4": { 2024-04-21T23:28:36.887 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rwoption4", 2024-04-21T23:28:36.887 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.887 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.887 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.887 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.887 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.888 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.888 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.888 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.888 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.888 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.888 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.888 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.888 INFO:teuthology.orchestra.run.smithi040.stdout: "rwoption5": { 2024-04-21T23:28:36.888 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rwoption5", 2024-04-21T23:28:36.889 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.889 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.889 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.889 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.889 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.889 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.889 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.889 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.889 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.890 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.890 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.890 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.890 INFO:teuthology.orchestra.run.smithi040.stdout: "rwoption6": { 2024-04-21T23:28:36.890 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rwoption6", 2024-04-21T23:28:36.890 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.890 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.890 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.890 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:36.891 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.891 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.891 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.891 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.891 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.891 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.891 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.891 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.891 INFO:teuthology.orchestra.run.smithi040.stdout: "rwoption7": { 2024-04-21T23:28:36.892 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rwoption7", 2024-04-21T23:28:36.892 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:36.892 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.892 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.892 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.892 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "1", 2024-04-21T23:28:36.892 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "42", 2024-04-21T23:28:36.892 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.893 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.893 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.893 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.893 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.893 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.893 INFO:teuthology.orchestra.run.smithi040.stdout: "testkey": { 2024-04-21T23:28:36.893 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "testkey", 2024-04-21T23:28:36.893 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.893 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.894 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.894 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.894 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.894 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.894 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.894 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.894 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.895 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.895 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.895 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.895 INFO:teuthology.orchestra.run.smithi040.stdout: "testlkey": { 2024-04-21T23:28:36.895 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "testlkey", 2024-04-21T23:28:36.895 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.895 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.895 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.895 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.896 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.896 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.896 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.896 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.896 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.896 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.896 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.896 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.896 INFO:teuthology.orchestra.run.smithi040.stdout: "testnewline": { 2024-04-21T23:28:36.897 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "testnewline", 2024-04-21T23:28:36.897 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.897 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.897 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.897 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.897 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.897 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.897 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.898 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.898 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.898 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.898 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.898 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:36.898 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:36.898 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.898 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:36.899 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "snap_schedule", 2024-04-21T23:28:36.899 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:36.899 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:36.899 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:36.899 INFO:teuthology.orchestra.run.smithi040.stdout: "allow_m_granularity": { 2024-04-21T23:28:36.899 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "allow_m_granularity", 2024-04-21T23:28:36.899 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.899 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.899 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.900 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.900 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.900 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.900 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.900 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "allow minute scheduled snapshots", 2024-04-21T23:28:36.900 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.900 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.900 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.900 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.901 INFO:teuthology.orchestra.run.smithi040.stdout: "dump_on_update": { 2024-04-21T23:28:36.901 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "dump_on_update", 2024-04-21T23:28:36.901 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.901 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.901 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.901 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.901 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.901 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.902 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.902 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "dump database to debug log on update", 2024-04-21T23:28:36.902 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.902 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.902 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.902 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.902 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:36.902 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:36.902 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.903 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.903 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.903 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.903 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.903 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.903 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:36.903 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:36.903 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:36.903 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:36.904 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:36.904 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:36.904 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:36.904 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:36.904 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.904 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.904 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.904 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.905 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.905 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:36.905 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:36.905 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.905 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.905 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.905 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.905 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.905 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.906 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.906 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.906 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.906 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.906 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.906 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.906 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:36.906 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:36.907 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.907 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.907 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.907 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:36.907 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.907 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.907 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:36.907 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:36.908 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:36.908 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:36.908 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:36.908 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:36.908 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:36.908 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:36.908 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.908 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.908 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.909 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.909 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.909 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:36.909 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:36.909 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.909 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.909 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.909 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.910 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.910 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.910 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.910 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.910 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.910 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.910 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.910 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:36.910 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:36.911 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.911 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:36.911 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "stats", 2024-04-21T23:28:36.911 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:36.911 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:36.911 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:36.911 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:36.911 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:36.912 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.912 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.912 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.912 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.912 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.912 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.912 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:36.912 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:36.913 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:36.913 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:36.913 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:36.913 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:36.913 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:36.913 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:36.913 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.913 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.913 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.914 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.914 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.914 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:36.914 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:36.914 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.914 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.914 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.914 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.915 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.915 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.915 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.915 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.915 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.915 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.915 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.915 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.915 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:36.916 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:36.916 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.916 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.916 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.916 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:36.916 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.916 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.916 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:36.917 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:36.917 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:36.917 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:36.917 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:36.917 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:36.917 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:36.917 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:36.917 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.917 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.918 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.918 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.918 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.918 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:36.918 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:36.918 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.918 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.918 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.918 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.919 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.919 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.919 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.919 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.919 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.919 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.919 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.919 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:36.920 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:36.920 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.920 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:36.920 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "status", 2024-04-21T23:28:36.920 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:36.920 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:36.920 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:36.920 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:36.920 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:36.921 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.921 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.921 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.921 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.921 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.921 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.921 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:36.921 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:36.921 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:36.922 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:36.922 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:36.922 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:36.922 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:36.922 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:36.922 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.922 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.922 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.923 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.923 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.923 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:36.923 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:36.923 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.923 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.923 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.923 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.923 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.924 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.924 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.924 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.924 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.924 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.924 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.924 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.924 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:36.925 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:36.925 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.925 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.925 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.925 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:36.925 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.925 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.925 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:36.925 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:36.926 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:36.926 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:36.926 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:36.926 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:36.926 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:36.926 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:36.926 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.926 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.927 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.927 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.927 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.927 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:36.927 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:36.927 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.927 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.927 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.927 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.928 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.928 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.928 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.928 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.928 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.928 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.928 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.928 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:36.928 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:36.929 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.929 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:36.929 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "telegraf", 2024-04-21T23:28:36.929 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:36.929 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:36.929 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:36.929 INFO:teuthology.orchestra.run.smithi040.stdout: "address": { 2024-04-21T23:28:36.929 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "address", 2024-04-21T23:28:36.929 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.930 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.930 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.930 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-21T23:28:36.930 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.930 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.930 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.930 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.930 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.931 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.931 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.931 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.931 INFO:teuthology.orchestra.run.smithi040.stdout: "interval": { 2024-04-21T23:28:36.931 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "interval", 2024-04-21T23:28:36.931 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-21T23:28:36.931 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.931 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.931 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "15", 2024-04-21T23:28:36.931 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.932 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.932 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.932 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.932 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.932 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.932 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.932 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.932 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:36.933 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:36.933 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.933 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.933 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.933 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.933 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.933 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.933 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:36.933 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:36.934 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:36.934 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:36.934 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:36.934 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:36.934 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:36.934 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:36.934 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.934 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.935 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.935 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.935 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.935 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:36.935 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:36.935 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.935 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.935 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.935 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.936 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.936 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.936 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.936 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.936 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.936 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.936 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.936 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.937 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:36.937 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:36.937 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.937 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.937 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.937 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:36.937 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.937 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.938 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:36.938 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:36.938 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:36.938 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:36.938 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:36.938 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:36.938 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:36.938 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:36.938 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.939 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.939 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.939 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.939 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.939 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:36.939 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:36.939 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.939 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.939 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.940 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.940 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.940 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.940 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.940 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.940 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.940 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.940 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.940 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:36.941 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:36.941 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.941 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:36.941 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "telemetry", 2024-04-21T23:28:36.941 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:36.941 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:36.941 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:36.941 INFO:teuthology.orchestra.run.smithi040.stdout: "channel_basic": { 2024-04-21T23:28:36.941 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "channel_basic", 2024-04-21T23:28:36.942 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.942 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.942 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.942 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:36.942 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.942 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.942 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.942 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-21T23:28:36.943 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.943 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.943 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.943 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.943 INFO:teuthology.orchestra.run.smithi040.stdout: "channel_crash": { 2024-04-21T23:28:36.943 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "channel_crash", 2024-04-21T23:28:36.943 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.943 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.943 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.944 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:36.944 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.944 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.944 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.944 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-21T23:28:36.944 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.944 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.944 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.944 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.945 INFO:teuthology.orchestra.run.smithi040.stdout: "channel_device": { 2024-04-21T23:28:36.945 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "channel_device", 2024-04-21T23:28:36.945 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.945 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.945 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.945 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:36.945 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.945 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.945 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.946 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-21T23:28:36.946 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.946 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.946 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.946 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.946 INFO:teuthology.orchestra.run.smithi040.stdout: "channel_ident": { 2024-04-21T23:28:36.946 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "channel_ident", 2024-04-21T23:28:36.946 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.947 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.947 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.947 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.947 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.947 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.947 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.947 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-21T23:28:36.947 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.947 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.948 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.948 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.948 INFO:teuthology.orchestra.run.smithi040.stdout: "channel_perf": { 2024-04-21T23:28:36.948 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "channel_perf", 2024-04-21T23:28:36.948 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.948 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.948 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.948 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.948 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.949 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.949 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.949 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-21T23:28:36.949 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.949 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.949 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.949 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.949 INFO:teuthology.orchestra.run.smithi040.stdout: "contact": { 2024-04-21T23:28:36.949 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "contact", 2024-04-21T23:28:36.950 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.950 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.950 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.950 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.950 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.950 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.950 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.950 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.951 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.951 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.951 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.951 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.951 INFO:teuthology.orchestra.run.smithi040.stdout: "description": { 2024-04-21T23:28:36.951 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "description", 2024-04-21T23:28:36.951 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.951 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.952 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.952 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.952 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.952 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.952 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.952 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.952 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.952 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.952 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.953 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.953 INFO:teuthology.orchestra.run.smithi040.stdout: "device_url": { 2024-04-21T23:28:36.953 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "device_url", 2024-04-21T23:28:36.953 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.953 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.953 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.953 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-21T23:28:36.953 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.953 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.954 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.954 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.954 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.954 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.954 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.954 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.954 INFO:teuthology.orchestra.run.smithi040.stdout: "enabled": { 2024-04-21T23:28:36.954 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "enabled", 2024-04-21T23:28:36.955 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.955 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.955 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.955 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.955 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.955 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.955 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.955 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.955 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.956 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.956 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.956 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.956 INFO:teuthology.orchestra.run.smithi040.stdout: "interval": { 2024-04-21T23:28:36.956 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "interval", 2024-04-21T23:28:36.956 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:36.956 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.956 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.957 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "24", 2024-04-21T23:28:36.957 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "8", 2024-04-21T23:28:36.957 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.957 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.957 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.957 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.957 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.957 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.957 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.958 INFO:teuthology.orchestra.run.smithi040.stdout: "last_opt_revision": { 2024-04-21T23:28:36.958 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "last_opt_revision", 2024-04-21T23:28:36.958 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:36.958 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.958 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.958 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "1", 2024-04-21T23:28:36.958 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.958 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.959 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.959 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.959 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.959 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.959 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.959 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.959 INFO:teuthology.orchestra.run.smithi040.stdout: "leaderboard": { 2024-04-21T23:28:36.959 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "leaderboard", 2024-04-21T23:28:36.960 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.960 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.960 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.960 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.960 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.960 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.960 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.960 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.960 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.961 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.961 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.961 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.961 INFO:teuthology.orchestra.run.smithi040.stdout: "leaderboard_description": { 2024-04-21T23:28:36.961 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "leaderboard_description", 2024-04-21T23:28:36.961 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.961 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.961 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.961 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.962 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.962 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.962 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.962 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.962 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.962 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.962 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.962 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.963 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:36.963 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:36.963 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.963 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.963 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.963 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.963 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.963 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.963 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:36.964 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:36.964 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:36.964 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:36.964 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:36.964 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:36.964 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:36.964 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:36.964 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.964 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.965 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.965 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.965 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.965 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:36.965 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:36.965 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.965 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.965 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.965 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.966 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.966 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.966 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.966 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.966 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.966 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.966 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.966 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.967 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:36.967 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:36.967 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.967 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.967 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.967 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:36.967 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.967 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.967 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:36.968 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:36.968 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:36.968 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:36.968 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:36.968 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:36.968 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:36.968 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:36.968 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.969 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.969 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.969 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.969 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.969 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:36.969 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:36.969 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.969 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.969 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.970 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.970 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.970 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.970 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.970 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.970 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.970 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.970 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.971 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.971 INFO:teuthology.orchestra.run.smithi040.stdout: "organization": { 2024-04-21T23:28:36.971 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "organization", 2024-04-21T23:28:36.971 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.971 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.971 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.971 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.971 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.972 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.972 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.972 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.972 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.972 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.972 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.972 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.972 INFO:teuthology.orchestra.run.smithi040.stdout: "proxy": { 2024-04-21T23:28:36.972 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "proxy", 2024-04-21T23:28:36.973 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.973 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.973 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.973 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.973 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.973 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.973 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.973 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.973 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.974 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.974 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.974 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.974 INFO:teuthology.orchestra.run.smithi040.stdout: "url": { 2024-04-21T23:28:36.974 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "url", 2024-04-21T23:28:36.974 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.974 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.975 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.975 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-21T23:28:36.975 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.975 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.975 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.975 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.975 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.975 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.975 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.976 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:36.976 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:36.976 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.976 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:36.976 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "test_orchestrator", 2024-04-21T23:28:36.976 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:36.976 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:36.976 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:36.976 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:36.977 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:36.977 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.977 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.977 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.977 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.977 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.977 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.977 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:36.977 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:36.978 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:36.978 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:36.978 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:36.978 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:36.978 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:36.978 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:36.978 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.978 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.979 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.979 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.979 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.979 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:36.979 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:36.979 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.979 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.979 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.980 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.980 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.980 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.980 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.980 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.980 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.980 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.980 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.980 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.981 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:36.981 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:36.981 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.981 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.981 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.981 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:36.981 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.981 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.982 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:36.982 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:36.982 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:36.982 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:36.982 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:36.982 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:36.982 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:36.982 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:36.982 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.983 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.983 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.983 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.983 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.983 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:36.983 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:36.983 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.983 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.984 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.984 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.984 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.984 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.984 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.984 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.984 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.984 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.985 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.985 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:36.985 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:36.985 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.985 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:36.985 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "volumes", 2024-04-21T23:28:36.985 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:36.985 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:36.985 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:36.986 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:36.986 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:36.986 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.986 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.986 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.986 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:36.986 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.986 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.987 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:36.987 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:36.987 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:36.987 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:36.987 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:36.987 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:36.987 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:36.987 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:36.987 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.988 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.988 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.988 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.988 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.988 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:36.988 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:36.988 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.988 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.989 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.989 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.989 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.989 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.989 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.989 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.989 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.989 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.989 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.990 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.990 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:36.990 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:36.990 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:36.990 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.990 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.990 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:36.990 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.990 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.991 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:36.991 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:36.991 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:36.991 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:36.991 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:36.991 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:36.991 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:36.991 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:36.992 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.992 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.992 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.992 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.992 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.992 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:36.992 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:36.992 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.993 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.993 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:36.993 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.993 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.993 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.993 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.993 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:36.993 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.994 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.994 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.994 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.994 INFO:teuthology.orchestra.run.smithi040.stdout: "max_concurrent_clones": { 2024-04-21T23:28:36.994 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "max_concurrent_clones", 2024-04-21T23:28:36.994 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:36.994 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.994 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.994 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "4", 2024-04-21T23:28:36.995 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.995 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.995 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.995 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-21T23:28:36.995 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.995 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.995 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.995 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.995 INFO:teuthology.orchestra.run.smithi040.stdout: "periodic_async_work": { 2024-04-21T23:28:36.996 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "periodic_async_work", 2024-04-21T23:28:36.996 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.996 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.996 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.996 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:36.996 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.996 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.996 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.997 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Periodically check for async work", 2024-04-21T23:28:36.997 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.997 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.997 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.997 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.997 INFO:teuthology.orchestra.run.smithi040.stdout: "snapshot_clone_delay": { 2024-04-21T23:28:36.997 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "snapshot_clone_delay", 2024-04-21T23:28:36.997 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:36.998 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.998 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.998 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "0", 2024-04-21T23:28:36.998 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:36.998 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:36.998 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:36.998 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-21T23:28:36.998 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:36.998 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:36.999 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:36.999 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:36.999 INFO:teuthology.orchestra.run.smithi040.stdout: "snapshot_clone_no_wait": { 2024-04-21T23:28:36.999 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "snapshot_clone_no_wait", 2024-04-21T23:28:36.999 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:36.999 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:36.999 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:36.999 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:37.000 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.000 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.000 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.000 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-21T23:28:37.000 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.000 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.000 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.000 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.000 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.001 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.001 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:37.001 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "zabbix", 2024-04-21T23:28:37.001 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:37.001 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:37.001 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:37.001 INFO:teuthology.orchestra.run.smithi040.stdout: "discovery_interval": { 2024-04-21T23:28:37.001 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "discovery_interval", 2024-04-21T23:28:37.002 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "uint", 2024-04-21T23:28:37.002 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.002 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.002 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "100", 2024-04-21T23:28:37.002 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.002 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.002 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.002 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.003 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.003 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.003 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.003 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.003 INFO:teuthology.orchestra.run.smithi040.stdout: "identifier": { 2024-04-21T23:28:37.003 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "identifier", 2024-04-21T23:28:37.003 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.003 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.003 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.004 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.004 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.004 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.004 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.004 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.004 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.004 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.004 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.004 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.005 INFO:teuthology.orchestra.run.smithi040.stdout: "interval": { 2024-04-21T23:28:37.005 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "interval", 2024-04-21T23:28:37.005 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-21T23:28:37.005 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.005 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.005 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "60", 2024-04-21T23:28:37.005 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.005 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.006 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.006 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.006 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.006 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.006 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.006 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.006 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:37.006 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:37.006 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.007 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.007 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.007 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.007 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.007 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.007 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.007 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:37.007 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:37.007 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:37.008 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:37.008 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:37.008 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:37.008 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.008 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.008 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.008 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.008 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.009 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.009 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:37.009 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:37.009 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.009 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.009 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.009 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.009 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.009 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.010 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.010 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.010 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.010 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.010 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.010 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.010 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:37.010 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:37.011 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.011 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.011 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.011 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:37.011 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.011 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.011 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.011 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:37.011 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:37.012 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:37.012 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:37.012 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:37.012 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:37.012 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.012 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.012 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.012 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.013 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.013 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.013 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:37.013 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:37.013 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.013 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.013 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.013 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.013 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.014 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.014 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.014 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.014 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.014 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.014 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.014 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.014 INFO:teuthology.orchestra.run.smithi040.stdout: "zabbix_host": { 2024-04-21T23:28:37.015 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "zabbix_host", 2024-04-21T23:28:37.015 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.015 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.015 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.015 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.015 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.015 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.015 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.016 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.016 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.016 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.016 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.016 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.016 INFO:teuthology.orchestra.run.smithi040.stdout: "zabbix_port": { 2024-04-21T23:28:37.016 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "zabbix_port", 2024-04-21T23:28:37.016 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:37.017 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.017 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.017 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10051", 2024-04-21T23:28:37.017 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.017 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.017 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.017 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.017 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.017 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.018 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.018 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.018 INFO:teuthology.orchestra.run.smithi040.stdout: "zabbix_sender": { 2024-04-21T23:28:37.018 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "zabbix_sender", 2024-04-21T23:28:37.018 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.018 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.018 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.018 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-21T23:28:37.018 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.019 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.019 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.019 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.019 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.019 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.019 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.019 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.019 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.020 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.020 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-21T23:28:37.020 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.020 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:37.020 INFO:teuthology.orchestra.run.smithi040.stdout: "gid": 25108, 2024-04-21T23:28:37.020 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "z", 2024-04-21T23:28:37.020 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr_features": 4540701547738038271, 2024-04-21T23:28:37.020 INFO:teuthology.orchestra.run.smithi040.stdout: "available_modules": [ 2024-04-21T23:28:37.021 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:37.021 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "alerts", 2024-04-21T23:28:37.021 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:37.021 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:37.021 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:37.021 INFO:teuthology.orchestra.run.smithi040.stdout: "interval": { 2024-04-21T23:28:37.021 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "interval", 2024-04-21T23:28:37.021 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-21T23:28:37.022 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.022 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.022 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "60", 2024-04-21T23:28:37.022 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.022 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.022 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.022 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "How frequently to reexamine health status", 2024-04-21T23:28:37.022 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.023 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.023 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.023 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.023 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:37.023 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:37.023 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.023 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.023 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.024 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.024 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.024 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.024 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.024 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:37.024 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:37.024 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:37.024 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:37.025 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:37.025 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:37.025 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.025 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.025 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.025 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.025 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.025 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.025 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:37.026 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:37.026 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.026 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.026 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.026 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.026 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.026 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.026 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.027 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.027 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.027 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.027 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.027 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.027 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:37.027 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:37.027 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.028 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.028 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.028 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:37.028 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.028 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.028 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.028 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:37.028 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:37.029 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:37.029 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:37.029 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:37.029 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:37.029 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.029 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.029 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.029 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.029 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.030 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.030 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:37.030 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:37.030 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.030 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.030 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.030 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.030 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.030 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.031 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.031 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.031 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.031 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.031 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.031 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.031 INFO:teuthology.orchestra.run.smithi040.stdout: "smtp_destination": { 2024-04-21T23:28:37.031 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "smtp_destination", 2024-04-21T23:28:37.031 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.032 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.032 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.032 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.032 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.032 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.032 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.032 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Email address to send alerts to", 2024-04-21T23:28:37.032 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.032 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.033 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.033 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.033 INFO:teuthology.orchestra.run.smithi040.stdout: "smtp_from_name": { 2024-04-21T23:28:37.033 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "smtp_from_name", 2024-04-21T23:28:37.033 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.033 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.033 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.033 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "Ceph", 2024-04-21T23:28:37.033 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.034 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.034 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.034 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Email From: name", 2024-04-21T23:28:37.034 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.034 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.034 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.034 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.035 INFO:teuthology.orchestra.run.smithi040.stdout: "smtp_host": { 2024-04-21T23:28:37.035 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "smtp_host", 2024-04-21T23:28:37.035 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.035 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.035 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.035 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.035 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.035 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.035 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.036 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "SMTP server", 2024-04-21T23:28:37.036 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.036 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.036 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.036 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.036 INFO:teuthology.orchestra.run.smithi040.stdout: "smtp_password": { 2024-04-21T23:28:37.036 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "smtp_password", 2024-04-21T23:28:37.036 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.037 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.037 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.037 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.037 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.037 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.037 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.037 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Password to authenticate with", 2024-04-21T23:28:37.037 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.038 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.038 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.038 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.038 INFO:teuthology.orchestra.run.smithi040.stdout: "smtp_port": { 2024-04-21T23:28:37.038 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "smtp_port", 2024-04-21T23:28:37.038 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:37.038 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.038 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.038 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "465", 2024-04-21T23:28:37.039 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.039 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.039 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.039 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "SMTP port", 2024-04-21T23:28:37.039 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.039 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.039 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.039 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.040 INFO:teuthology.orchestra.run.smithi040.stdout: "smtp_sender": { 2024-04-21T23:28:37.040 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "smtp_sender", 2024-04-21T23:28:37.040 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.040 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.040 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.040 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.040 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.040 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.040 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.041 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "SMTP envelope sender", 2024-04-21T23:28:37.041 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.041 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.041 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.041 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.041 INFO:teuthology.orchestra.run.smithi040.stdout: "smtp_ssl": { 2024-04-21T23:28:37.041 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "smtp_ssl", 2024-04-21T23:28:37.041 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.041 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.042 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.042 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:37.042 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.042 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.042 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.042 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-21T23:28:37.042 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.043 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.043 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.043 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.043 INFO:teuthology.orchestra.run.smithi040.stdout: "smtp_user": { 2024-04-21T23:28:37.043 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "smtp_user", 2024-04-21T23:28:37.043 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.043 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.043 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.043 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.044 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.044 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.044 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.044 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "User to authenticate as", 2024-04-21T23:28:37.044 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.044 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.044 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.044 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.044 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.045 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.045 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:37.045 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "balancer", 2024-04-21T23:28:37.045 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:37.045 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:37.045 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:37.045 INFO:teuthology.orchestra.run.smithi040.stdout: "active": { 2024-04-21T23:28:37.045 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "active", 2024-04-21T23:28:37.046 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.046 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.046 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.046 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:37.046 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.046 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.046 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.046 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "automatically balance PGs across cluster", 2024-04-21T23:28:37.046 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.047 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.047 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.047 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.047 INFO:teuthology.orchestra.run.smithi040.stdout: "begin_time": { 2024-04-21T23:28:37.047 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "begin_time", 2024-04-21T23:28:37.047 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.047 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.047 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.047 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "0000", 2024-04-21T23:28:37.048 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.050 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.050 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.050 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "beginning time of day to automatically balance", 2024-04-21T23:28:37.050 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-21T23:28:37.050 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.051 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.051 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.051 INFO:teuthology.orchestra.run.smithi040.stdout: "begin_weekday": { 2024-04-21T23:28:37.051 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "begin_weekday", 2024-04-21T23:28:37.051 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "uint", 2024-04-21T23:28:37.051 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.051 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.051 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "0", 2024-04-21T23:28:37.051 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "0", 2024-04-21T23:28:37.052 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "6", 2024-04-21T23:28:37.052 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.052 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-21T23:28:37.052 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-21T23:28:37.052 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.052 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.052 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.052 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_compat_max_iterations": { 2024-04-21T23:28:37.052 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "crush_compat_max_iterations", 2024-04-21T23:28:37.053 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "uint", 2024-04-21T23:28:37.053 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.053 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.053 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "25", 2024-04-21T23:28:37.053 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "1", 2024-04-21T23:28:37.053 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "250", 2024-04-21T23:28:37.053 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.053 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-21T23:28:37.054 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.054 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.054 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.054 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.054 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_compat_metrics": { 2024-04-21T23:28:37.054 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "crush_compat_metrics", 2024-04-21T23:28:37.054 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.054 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.054 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.055 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "pgs,objects,bytes", 2024-04-21T23:28:37.055 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.055 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.055 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.055 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-21T23:28:37.055 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-21T23:28:37.055 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.055 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.056 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.056 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_compat_step": { 2024-04-21T23:28:37.056 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "crush_compat_step", 2024-04-21T23:28:37.056 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "float", 2024-04-21T23:28:37.056 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.056 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.056 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "0.5", 2024-04-21T23:28:37.056 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "0.001", 2024-04-21T23:28:37.056 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "0.999", 2024-04-21T23:28:37.057 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.057 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "aggressiveness of optimization", 2024-04-21T23:28:37.057 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-21T23:28:37.057 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.057 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.057 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.057 INFO:teuthology.orchestra.run.smithi040.stdout: "end_time": { 2024-04-21T23:28:37.057 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "end_time", 2024-04-21T23:28:37.057 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.058 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.058 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.058 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "2359", 2024-04-21T23:28:37.058 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.058 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.058 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.058 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "ending time of day to automatically balance", 2024-04-21T23:28:37.058 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-21T23:28:37.059 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.059 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.059 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.059 INFO:teuthology.orchestra.run.smithi040.stdout: "end_weekday": { 2024-04-21T23:28:37.059 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "end_weekday", 2024-04-21T23:28:37.059 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "uint", 2024-04-21T23:28:37.059 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.059 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.059 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "0", 2024-04-21T23:28:37.060 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "0", 2024-04-21T23:28:37.060 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "6", 2024-04-21T23:28:37.060 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.060 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-21T23:28:37.060 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-21T23:28:37.060 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.060 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.060 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.061 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:37.061 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:37.061 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.061 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.061 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.061 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.061 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.061 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.061 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.062 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:37.062 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:37.062 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:37.062 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:37.062 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:37.062 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:37.062 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.062 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.062 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.063 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.063 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.063 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.063 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:37.063 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:37.063 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.063 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.063 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.064 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.064 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.064 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.064 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.064 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.064 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.064 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.064 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.064 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.065 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:37.065 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:37.065 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.065 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.065 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.065 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:37.065 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.066 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.066 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.066 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:37.066 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:37.066 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:37.066 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:37.066 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:37.066 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:37.066 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.067 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.067 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.067 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.067 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.067 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.067 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:37.067 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:37.067 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.068 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.068 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.068 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.068 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.068 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.068 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.068 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.068 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.069 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.069 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.069 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.069 INFO:teuthology.orchestra.run.smithi040.stdout: "min_score": { 2024-04-21T23:28:37.069 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "min_score", 2024-04-21T23:28:37.069 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "float", 2024-04-21T23:28:37.069 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.069 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.069 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "0", 2024-04-21T23:28:37.070 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.070 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.070 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.070 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-21T23:28:37.070 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.070 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.070 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.070 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.071 INFO:teuthology.orchestra.run.smithi040.stdout: "mode": { 2024-04-21T23:28:37.071 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "mode", 2024-04-21T23:28:37.071 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.071 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.071 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.071 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "upmap", 2024-04-21T23:28:37.071 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.071 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.071 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.072 INFO:teuthology.orchestra.run.smithi040.stdout: "crush-compat", 2024-04-21T23:28:37.072 INFO:teuthology.orchestra.run.smithi040.stdout: "none", 2024-04-21T23:28:37.072 INFO:teuthology.orchestra.run.smithi040.stdout: "read", 2024-04-21T23:28:37.072 INFO:teuthology.orchestra.run.smithi040.stdout: "upmap", 2024-04-21T23:28:37.072 INFO:teuthology.orchestra.run.smithi040.stdout: "upmap-read" 2024-04-21T23:28:37.072 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.072 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Balancer mode", 2024-04-21T23:28:37.072 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.073 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.073 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.073 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.073 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_ids": { 2024-04-21T23:28:37.073 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "pool_ids", 2024-04-21T23:28:37.073 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.073 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.073 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.073 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.074 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.074 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.074 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.074 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-21T23:28:37.074 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.074 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.074 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.074 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.075 INFO:teuthology.orchestra.run.smithi040.stdout: "sleep_interval": { 2024-04-21T23:28:37.075 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "sleep_interval", 2024-04-21T23:28:37.075 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-21T23:28:37.075 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.075 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.075 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "60", 2024-04-21T23:28:37.075 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.075 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.076 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.076 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-21T23:28:37.076 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.076 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.076 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.076 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.076 INFO:teuthology.orchestra.run.smithi040.stdout: "upmap_max_deviation": { 2024-04-21T23:28:37.076 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "upmap_max_deviation", 2024-04-21T23:28:37.077 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:37.077 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.077 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.077 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "5", 2024-04-21T23:28:37.077 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "1", 2024-04-21T23:28:37.077 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.077 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.077 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-21T23:28:37.078 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-21T23:28:37.078 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.078 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.078 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.078 INFO:teuthology.orchestra.run.smithi040.stdout: "upmap_max_optimizations": { 2024-04-21T23:28:37.078 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "upmap_max_optimizations", 2024-04-21T23:28:37.078 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "uint", 2024-04-21T23:28:37.078 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.078 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.079 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10", 2024-04-21T23:28:37.079 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.079 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.079 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.079 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-21T23:28:37.079 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.079 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.079 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.080 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.080 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.080 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.080 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:37.080 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "cephadm", 2024-04-21T23:28:37.080 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:37.080 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:37.080 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:37.081 INFO:teuthology.orchestra.run.smithi040.stdout: "agent_down_multiplier": { 2024-04-21T23:28:37.081 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "agent_down_multiplier", 2024-04-21T23:28:37.081 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "float", 2024-04-21T23:28:37.081 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.081 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.081 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "3.0", 2024-04-21T23:28:37.081 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.081 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.081 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.082 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-21T23:28:37.082 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.082 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.082 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.082 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.082 INFO:teuthology.orchestra.run.smithi040.stdout: "agent_refresh_rate": { 2024-04-21T23:28:37.082 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "agent_refresh_rate", 2024-04-21T23:28:37.082 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-21T23:28:37.083 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.083 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.083 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "20", 2024-04-21T23:28:37.083 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.083 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.083 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.083 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-21T23:28:37.083 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.083 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.084 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.084 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.084 INFO:teuthology.orchestra.run.smithi040.stdout: "agent_starting_port": { 2024-04-21T23:28:37.084 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "agent_starting_port", 2024-04-21T23:28:37.084 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:37.084 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.084 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.084 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "4721", 2024-04-21T23:28:37.084 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.085 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.085 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.085 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-21T23:28:37.085 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.085 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.085 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.085 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.085 INFO:teuthology.orchestra.run.smithi040.stdout: "allow_ptrace": { 2024-04-21T23:28:37.085 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "allow_ptrace", 2024-04-21T23:28:37.086 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.086 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.086 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.086 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.086 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.086 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.086 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.087 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-21T23:28:37.087 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-21T23:28:37.087 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.087 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.087 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.087 INFO:teuthology.orchestra.run.smithi040.stdout: "autotune_interval": { 2024-04-21T23:28:37.087 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "autotune_interval", 2024-04-21T23:28:37.087 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-21T23:28:37.087 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.088 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.088 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "600", 2024-04-21T23:28:37.088 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.088 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.088 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.088 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-21T23:28:37.088 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.088 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.088 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.089 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.089 INFO:teuthology.orchestra.run.smithi040.stdout: "autotune_memory_target_ratio": { 2024-04-21T23:28:37.089 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "autotune_memory_target_ratio", 2024-04-21T23:28:37.089 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "float", 2024-04-21T23:28:37.089 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.089 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.089 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "0.7", 2024-04-21T23:28:37.089 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.089 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.090 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.090 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-21T23:28:37.090 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.090 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.090 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.090 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.090 INFO:teuthology.orchestra.run.smithi040.stdout: "cephadm_log_destination": { 2024-04-21T23:28:37.090 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "cephadm_log_destination", 2024-04-21T23:28:37.091 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.091 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.091 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.091 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.091 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.091 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.091 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.091 INFO:teuthology.orchestra.run.smithi040.stdout: "file", 2024-04-21T23:28:37.091 INFO:teuthology.orchestra.run.smithi040.stdout: "file,syslog", 2024-04-21T23:28:37.092 INFO:teuthology.orchestra.run.smithi040.stdout: "syslog" 2024-04-21T23:28:37.092 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.092 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-21T23:28:37.092 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.092 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.092 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.092 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.092 INFO:teuthology.orchestra.run.smithi040.stdout: "cgroups_split": { 2024-04-21T23:28:37.093 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "cgroups_split", 2024-04-21T23:28:37.093 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.093 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.093 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.093 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:37.093 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.093 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.093 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.093 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-21T23:28:37.094 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.094 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.094 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.094 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.094 INFO:teuthology.orchestra.run.smithi040.stdout: "config_checks_enabled": { 2024-04-21T23:28:37.094 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "config_checks_enabled", 2024-04-21T23:28:37.094 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.094 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.095 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.095 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.095 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.095 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.095 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.095 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-21T23:28:37.095 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.095 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.096 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.096 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.096 INFO:teuthology.orchestra.run.smithi040.stdout: "config_dashboard": { 2024-04-21T23:28:37.096 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "config_dashboard", 2024-04-21T23:28:37.096 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.096 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.096 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.096 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:37.096 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.097 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.097 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.097 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-21T23:28:37.097 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.097 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.097 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.097 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.097 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_alertmanager": { 2024-04-21T23:28:37.098 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_alertmanager", 2024-04-21T23:28:37.098 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.098 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.098 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.098 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-21T23:28:37.098 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.098 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.098 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.098 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Prometheus container image", 2024-04-21T23:28:37.099 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.099 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.099 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.099 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.099 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_base": { 2024-04-21T23:28:37.099 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_base", 2024-04-21T23:28:37.099 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.099 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.099 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.100 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-21T23:28:37.100 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.100 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.100 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.100 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Container image name, without the tag", 2024-04-21T23:28:37.100 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.100 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.100 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.100 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.101 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_elasticsearch": { 2024-04-21T23:28:37.101 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_elasticsearch", 2024-04-21T23:28:37.101 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.101 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.101 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.101 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-21T23:28:37.101 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.101 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.102 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.102 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "elasticsearch container image", 2024-04-21T23:28:37.102 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.102 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.102 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.102 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.102 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_grafana": { 2024-04-21T23:28:37.102 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_grafana", 2024-04-21T23:28:37.103 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.103 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.103 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.103 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-21T23:28:37.103 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.103 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.103 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.103 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Prometheus container image", 2024-04-21T23:28:37.103 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.104 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.104 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.104 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.104 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_haproxy": { 2024-04-21T23:28:37.104 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_haproxy", 2024-04-21T23:28:37.104 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.104 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.104 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.105 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-21T23:28:37.105 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.105 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.105 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.105 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "HAproxy container image", 2024-04-21T23:28:37.105 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.105 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.105 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.105 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.106 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_jaeger_agent": { 2024-04-21T23:28:37.106 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_jaeger_agent", 2024-04-21T23:28:37.106 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.106 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.106 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.106 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-21T23:28:37.106 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.107 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.107 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.107 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Jaeger agent container image", 2024-04-21T23:28:37.107 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.107 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.107 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.107 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.107 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_jaeger_collector": { 2024-04-21T23:28:37.108 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_jaeger_collector", 2024-04-21T23:28:37.108 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.108 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.108 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.108 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-21T23:28:37.108 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.108 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.108 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.108 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Jaeger collector container image", 2024-04-21T23:28:37.109 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.109 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.109 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.109 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.109 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_jaeger_query": { 2024-04-21T23:28:37.109 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_jaeger_query", 2024-04-21T23:28:37.109 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.109 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.110 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.110 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-21T23:28:37.110 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.110 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.110 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.110 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Jaeger query container image", 2024-04-21T23:28:37.110 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.110 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.111 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.111 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.111 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_keepalived": { 2024-04-21T23:28:37.111 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_keepalived", 2024-04-21T23:28:37.111 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.111 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.111 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.111 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-21T23:28:37.111 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.112 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.112 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.112 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Keepalived container image", 2024-04-21T23:28:37.112 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.112 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.112 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.112 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.112 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_loki": { 2024-04-21T23:28:37.113 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_loki", 2024-04-21T23:28:37.113 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.113 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.113 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.113 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-21T23:28:37.113 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.113 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.113 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.113 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Loki container image", 2024-04-21T23:28:37.114 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.114 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.114 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.114 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.114 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_node_exporter": { 2024-04-21T23:28:37.114 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_node_exporter", 2024-04-21T23:28:37.114 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.114 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.115 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.115 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-21T23:28:37.115 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.115 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.115 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.115 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Prometheus container image", 2024-04-21T23:28:37.115 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.115 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.116 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.116 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.116 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_nvmeof": { 2024-04-21T23:28:37.116 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_nvmeof", 2024-04-21T23:28:37.116 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.116 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.116 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.116 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-21T23:28:37.117 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.117 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.117 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.117 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Nvme-of container image", 2024-04-21T23:28:37.117 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.117 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.117 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.117 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.117 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_prometheus": { 2024-04-21T23:28:37.118 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_prometheus", 2024-04-21T23:28:37.118 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.118 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.118 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.118 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-21T23:28:37.118 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.118 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.118 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.118 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Prometheus container image", 2024-04-21T23:28:37.119 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.119 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.119 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.119 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.119 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_promtail": { 2024-04-21T23:28:37.119 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_promtail", 2024-04-21T23:28:37.119 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.119 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.120 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.120 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-21T23:28:37.120 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.120 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.120 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.120 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Promtail container image", 2024-04-21T23:28:37.120 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.120 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.120 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.121 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.121 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_samba": { 2024-04-21T23:28:37.121 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_samba", 2024-04-21T23:28:37.121 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.121 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.121 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.121 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-21T23:28:37.121 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.121 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.122 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.122 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Samba/SMB container image", 2024-04-21T23:28:37.122 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.122 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.122 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.122 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.122 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_snmp_gateway": { 2024-04-21T23:28:37.122 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_snmp_gateway", 2024-04-21T23:28:37.123 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.123 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.123 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.123 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-21T23:28:37.123 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.123 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.123 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.123 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "SNMP Gateway container image", 2024-04-21T23:28:37.123 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.124 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.124 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.124 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.124 INFO:teuthology.orchestra.run.smithi040.stdout: "container_init": { 2024-04-21T23:28:37.124 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_init", 2024-04-21T23:28:37.124 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.124 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.124 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.124 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:37.125 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.125 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.125 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.125 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Run podman/docker with `--init`", 2024-04-21T23:28:37.125 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.125 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.125 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.125 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.125 INFO:teuthology.orchestra.run.smithi040.stdout: "daemon_cache_timeout": { 2024-04-21T23:28:37.126 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "daemon_cache_timeout", 2024-04-21T23:28:37.126 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-21T23:28:37.126 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.126 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.126 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "600", 2024-04-21T23:28:37.126 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.126 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.126 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.127 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-21T23:28:37.127 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.127 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.127 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.127 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.127 INFO:teuthology.orchestra.run.smithi040.stdout: "default_cephadm_command_timeout": { 2024-04-21T23:28:37.127 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "default_cephadm_command_timeout", 2024-04-21T23:28:37.127 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:37.127 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.128 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.128 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "900", 2024-04-21T23:28:37.128 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.128 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.128 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.128 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-21T23:28:37.128 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.128 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.128 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.129 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.129 INFO:teuthology.orchestra.run.smithi040.stdout: "default_registry": { 2024-04-21T23:28:37.129 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "default_registry", 2024-04-21T23:28:37.129 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.129 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.129 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.129 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "docker.io", 2024-04-21T23:28:37.129 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.130 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.130 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.130 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-21T23:28:37.130 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.130 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.130 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.130 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.130 INFO:teuthology.orchestra.run.smithi040.stdout: "device_cache_timeout": { 2024-04-21T23:28:37.130 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "device_cache_timeout", 2024-04-21T23:28:37.131 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-21T23:28:37.131 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.131 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.131 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "1800", 2024-04-21T23:28:37.131 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.131 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.131 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.131 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "seconds to cache device inventory", 2024-04-21T23:28:37.131 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.132 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.132 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.132 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.132 INFO:teuthology.orchestra.run.smithi040.stdout: "device_enhanced_scan": { 2024-04-21T23:28:37.132 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "device_enhanced_scan", 2024-04-21T23:28:37.132 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.132 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.132 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.133 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.133 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.133 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.133 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.133 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-21T23:28:37.133 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.133 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.133 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.133 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.134 INFO:teuthology.orchestra.run.smithi040.stdout: "facts_cache_timeout": { 2024-04-21T23:28:37.134 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "facts_cache_timeout", 2024-04-21T23:28:37.134 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-21T23:28:37.134 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.134 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.134 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "60", 2024-04-21T23:28:37.134 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.134 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.135 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.135 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "seconds to cache host facts data", 2024-04-21T23:28:37.135 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.135 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.135 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.135 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.135 INFO:teuthology.orchestra.run.smithi040.stdout: "grafana_dashboards_path": { 2024-04-21T23:28:37.135 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "grafana_dashboards_path", 2024-04-21T23:28:37.135 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.136 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.136 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.136 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-21T23:28:37.136 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.136 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.136 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.136 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-21T23:28:37.136 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.136 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.137 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.137 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.137 INFO:teuthology.orchestra.run.smithi040.stdout: "host_check_interval": { 2024-04-21T23:28:37.137 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "host_check_interval", 2024-04-21T23:28:37.137 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-21T23:28:37.137 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.137 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.137 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "600", 2024-04-21T23:28:37.138 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.138 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.138 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.138 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "how frequently to perform a host check", 2024-04-21T23:28:37.138 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.138 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.138 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.138 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.138 INFO:teuthology.orchestra.run.smithi040.stdout: "hw_monitoring": { 2024-04-21T23:28:37.139 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "hw_monitoring", 2024-04-21T23:28:37.139 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.139 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.139 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.139 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.139 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.139 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.139 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.139 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-21T23:28:37.140 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.140 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.140 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.140 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.140 INFO:teuthology.orchestra.run.smithi040.stdout: "inventory_list_all": { 2024-04-21T23:28:37.140 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "inventory_list_all", 2024-04-21T23:28:37.140 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.140 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.141 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.141 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.141 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.141 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.141 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.141 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-21T23:28:37.141 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.141 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.141 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.142 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.142 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:37.142 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:37.142 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.142 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.142 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.142 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.142 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.143 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.143 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.143 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:37.143 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:37.143 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:37.143 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:37.143 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:37.143 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:37.143 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.144 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.144 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.144 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.144 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.144 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.144 INFO:teuthology.orchestra.run.smithi040.stdout: "log_refresh_metadata": { 2024-04-21T23:28:37.144 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_refresh_metadata", 2024-04-21T23:28:37.144 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.145 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.145 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.145 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.145 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.145 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.145 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.145 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-21T23:28:37.145 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.146 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.146 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.146 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.146 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:37.146 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:37.146 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.146 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.146 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.147 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:37.147 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.147 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.147 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.147 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-21T23:28:37.147 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.147 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.147 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.148 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.148 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:37.148 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:37.148 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.148 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.148 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.148 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:37.148 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.149 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.149 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.149 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:37.149 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:37.149 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:37.149 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:37.149 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:37.149 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:37.149 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.150 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.150 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.150 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.150 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.150 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.150 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:37.150 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:37.150 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.151 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.151 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.151 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.151 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.151 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.151 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.151 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.151 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.152 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.152 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.152 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.152 INFO:teuthology.orchestra.run.smithi040.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-21T23:28:37.152 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-21T23:28:37.152 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.152 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.152 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.153 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.153 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.153 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.153 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.153 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-21T23:28:37.153 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.153 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.153 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.153 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.154 INFO:teuthology.orchestra.run.smithi040.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-21T23:28:37.154 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-21T23:28:37.154 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.154 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.154 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.154 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "*", 2024-04-21T23:28:37.154 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.155 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.155 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.155 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-21T23:28:37.155 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.155 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.155 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.155 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.155 INFO:teuthology.orchestra.run.smithi040.stdout: "max_count_per_host": { 2024-04-21T23:28:37.155 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "max_count_per_host", 2024-04-21T23:28:37.156 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:37.156 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.156 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.156 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10", 2024-04-21T23:28:37.156 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.156 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.156 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.156 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "max number of daemons per service per host", 2024-04-21T23:28:37.156 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.157 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.157 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.157 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.157 INFO:teuthology.orchestra.run.smithi040.stdout: "max_osd_draining_count": { 2024-04-21T23:28:37.157 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "max_osd_draining_count", 2024-04-21T23:28:37.157 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:37.157 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.157 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.157 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10", 2024-04-21T23:28:37.158 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.158 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.158 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.158 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-21T23:28:37.158 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.158 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.158 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.158 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.159 INFO:teuthology.orchestra.run.smithi040.stdout: "migration_current": { 2024-04-21T23:28:37.159 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "migration_current", 2024-04-21T23:28:37.159 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:37.159 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.159 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.159 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.159 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.159 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.159 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.160 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "internal - do not modify", 2024-04-21T23:28:37.160 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.160 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.160 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.160 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.160 INFO:teuthology.orchestra.run.smithi040.stdout: "mode": { 2024-04-21T23:28:37.160 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "mode", 2024-04-21T23:28:37.160 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.160 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.161 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.161 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "root", 2024-04-21T23:28:37.161 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.161 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.161 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.161 INFO:teuthology.orchestra.run.smithi040.stdout: "cephadm-package", 2024-04-21T23:28:37.161 INFO:teuthology.orchestra.run.smithi040.stdout: "root" 2024-04-21T23:28:37.161 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.161 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "mode for remote execution of cephadm", 2024-04-21T23:28:37.162 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.162 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.162 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.162 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.162 INFO:teuthology.orchestra.run.smithi040.stdout: "oob_default_addr": { 2024-04-21T23:28:37.162 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "oob_default_addr", 2024-04-21T23:28:37.162 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.162 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.163 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.163 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "169.254.1.1", 2024-04-21T23:28:37.163 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.163 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.163 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.163 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-21T23:28:37.163 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.163 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.163 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.164 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.164 INFO:teuthology.orchestra.run.smithi040.stdout: "prometheus_alerts_path": { 2024-04-21T23:28:37.164 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "prometheus_alerts_path", 2024-04-21T23:28:37.164 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.164 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.164 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.164 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-21T23:28:37.164 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.164 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.165 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.165 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-21T23:28:37.165 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.165 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.165 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.165 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.165 INFO:teuthology.orchestra.run.smithi040.stdout: "registry_insecure": { 2024-04-21T23:28:37.165 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "registry_insecure", 2024-04-21T23:28:37.165 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.166 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.166 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.166 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.166 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.166 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.166 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.166 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-21T23:28:37.166 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.167 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.167 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.167 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.167 INFO:teuthology.orchestra.run.smithi040.stdout: "registry_password": { 2024-04-21T23:28:37.167 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "registry_password", 2024-04-21T23:28:37.167 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.167 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.167 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.168 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.168 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.168 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.168 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.168 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-21T23:28:37.168 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.168 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.168 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.169 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.169 INFO:teuthology.orchestra.run.smithi040.stdout: "registry_url": { 2024-04-21T23:28:37.169 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "registry_url", 2024-04-21T23:28:37.169 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.169 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.169 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.169 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.169 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.170 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.170 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.170 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-21T23:28:37.170 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.170 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.170 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.170 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.170 INFO:teuthology.orchestra.run.smithi040.stdout: "registry_username": { 2024-04-21T23:28:37.171 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "registry_username", 2024-04-21T23:28:37.171 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.171 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.171 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.171 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.171 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.171 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.171 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.172 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-21T23:28:37.172 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.172 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.172 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.172 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.172 INFO:teuthology.orchestra.run.smithi040.stdout: "secure_monitoring_stack": { 2024-04-21T23:28:37.172 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "secure_monitoring_stack", 2024-04-21T23:28:37.172 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.172 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.173 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.173 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.173 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.173 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.173 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.173 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-21T23:28:37.173 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.173 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.174 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.174 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.174 INFO:teuthology.orchestra.run.smithi040.stdout: "service_discovery_port": { 2024-04-21T23:28:37.174 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "service_discovery_port", 2024-04-21T23:28:37.174 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:37.174 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.174 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.174 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "8765", 2024-04-21T23:28:37.174 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.175 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.175 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.175 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "cephadm service discovery port", 2024-04-21T23:28:37.175 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.175 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.175 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.175 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.175 INFO:teuthology.orchestra.run.smithi040.stdout: "ssh_config_file": { 2024-04-21T23:28:37.176 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ssh_config_file", 2024-04-21T23:28:37.176 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.176 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.176 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.176 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.176 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.176 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.176 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.177 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-21T23:28:37.177 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.177 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.177 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.177 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.177 INFO:teuthology.orchestra.run.smithi040.stdout: "use_agent": { 2024-04-21T23:28:37.177 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "use_agent", 2024-04-21T23:28:37.177 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.177 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.178 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.178 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.178 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.178 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.178 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.178 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-21T23:28:37.178 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.178 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.179 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.179 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.179 INFO:teuthology.orchestra.run.smithi040.stdout: "use_repo_digest": { 2024-04-21T23:28:37.179 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "use_repo_digest", 2024-04-21T23:28:37.179 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.179 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.179 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.179 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:37.180 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.180 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.180 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.180 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-21T23:28:37.180 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.180 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.180 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.180 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.180 INFO:teuthology.orchestra.run.smithi040.stdout: "warn_on_failed_host_check": { 2024-04-21T23:28:37.181 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "warn_on_failed_host_check", 2024-04-21T23:28:37.181 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.181 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.181 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.181 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:37.181 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.181 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.181 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.181 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "raise a health warning if the host check fails", 2024-04-21T23:28:37.182 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.182 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.182 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.182 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.182 INFO:teuthology.orchestra.run.smithi040.stdout: "warn_on_stray_daemons": { 2024-04-21T23:28:37.182 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "warn_on_stray_daemons", 2024-04-21T23:28:37.182 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.182 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.183 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.183 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:37.183 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.183 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.183 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.183 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-21T23:28:37.183 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.183 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.183 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.184 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.184 INFO:teuthology.orchestra.run.smithi040.stdout: "warn_on_stray_hosts": { 2024-04-21T23:28:37.184 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "warn_on_stray_hosts", 2024-04-21T23:28:37.184 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.184 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.184 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.184 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:37.184 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.184 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.185 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.185 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-21T23:28:37.185 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.185 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.185 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.185 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.185 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.185 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.185 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:37.186 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "crash", 2024-04-21T23:28:37.186 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:37.186 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:37.186 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:37.186 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:37.186 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:37.186 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.186 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.186 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.187 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.187 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.187 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.187 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.187 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:37.187 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:37.187 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:37.187 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:37.188 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:37.188 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:37.188 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.188 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.188 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.188 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.188 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.188 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.188 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:37.189 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:37.189 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.189 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.189 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.189 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.189 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.189 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.189 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.190 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.190 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.190 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.190 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.190 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.190 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:37.190 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:37.190 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.190 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.191 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.191 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:37.191 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.191 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.191 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.191 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:37.191 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:37.191 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:37.191 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:37.192 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:37.192 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:37.192 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.192 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.192 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.192 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.192 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.192 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.193 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:37.193 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:37.193 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.193 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.193 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.193 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.193 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.193 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.193 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.194 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.194 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.194 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.194 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.194 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.194 INFO:teuthology.orchestra.run.smithi040.stdout: "retain_interval": { 2024-04-21T23:28:37.194 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "retain_interval", 2024-04-21T23:28:37.194 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-21T23:28:37.194 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.195 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.195 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "31536000", 2024-04-21T23:28:37.195 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.195 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.195 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.195 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-21T23:28:37.195 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.195 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.196 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.196 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.196 INFO:teuthology.orchestra.run.smithi040.stdout: "warn_recent_interval": { 2024-04-21T23:28:37.196 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "warn_recent_interval", 2024-04-21T23:28:37.196 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-21T23:28:37.196 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.196 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.196 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "1209600", 2024-04-21T23:28:37.196 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.197 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.197 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.197 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-21T23:28:37.197 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.197 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.197 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.197 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.197 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.197 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.198 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:37.198 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "dashboard", 2024-04-21T23:28:37.198 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:37.198 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:37.198 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:37.198 INFO:teuthology.orchestra.run.smithi040.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-21T23:28:37.198 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-21T23:28:37.198 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:37.199 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.199 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.199 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10", 2024-04-21T23:28:37.199 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.199 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.199 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.199 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.199 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.199 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.200 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.200 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.200 INFO:teuthology.orchestra.run.smithi040.stdout: "ALERTMANAGER_API_HOST": { 2024-04-21T23:28:37.200 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-21T23:28:37.200 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.200 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.200 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.200 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.201 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.201 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.201 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.201 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.201 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.201 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.201 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.201 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.201 INFO:teuthology.orchestra.run.smithi040.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-21T23:28:37.202 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-21T23:28:37.202 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.202 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.202 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.202 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:37.202 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.202 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.202 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.202 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.203 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.203 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.203 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.203 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.203 INFO:teuthology.orchestra.run.smithi040.stdout: "AUDIT_API_ENABLED": { 2024-04-21T23:28:37.203 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "AUDIT_API_ENABLED", 2024-04-21T23:28:37.203 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.203 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.204 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.204 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.204 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.204 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.204 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.204 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.204 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.204 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.204 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.205 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.205 INFO:teuthology.orchestra.run.smithi040.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-21T23:28:37.205 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-21T23:28:37.205 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.205 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.205 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.205 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:37.205 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.205 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.206 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.206 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.206 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.206 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.206 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.206 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.206 INFO:teuthology.orchestra.run.smithi040.stdout: "ENABLE_BROWSABLE_API": { 2024-04-21T23:28:37.206 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-21T23:28:37.207 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.207 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.207 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.207 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:37.207 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.207 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.207 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.207 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.208 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.208 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.208 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.208 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.208 INFO:teuthology.orchestra.run.smithi040.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-21T23:28:37.208 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-21T23:28:37.208 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.208 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.209 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.209 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:37.209 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.209 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.209 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.209 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.209 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.209 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.210 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.210 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.210 INFO:teuthology.orchestra.run.smithi040.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-21T23:28:37.210 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-21T23:28:37.210 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.210 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.210 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.210 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:37.211 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.211 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.211 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.211 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.211 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.211 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.211 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.211 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.211 INFO:teuthology.orchestra.run.smithi040.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-21T23:28:37.212 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-21T23:28:37.212 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.212 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.212 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.212 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:37.212 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.212 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.212 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.213 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.213 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.213 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.213 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.213 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.213 INFO:teuthology.orchestra.run.smithi040.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-21T23:28:37.213 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-21T23:28:37.213 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.213 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.214 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.214 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:37.214 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.214 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.214 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.214 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.214 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.214 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.215 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.215 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.215 INFO:teuthology.orchestra.run.smithi040.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-21T23:28:37.215 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-21T23:28:37.215 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.215 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.215 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.215 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:37.215 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.216 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.216 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.216 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.216 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.216 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.216 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.216 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.216 INFO:teuthology.orchestra.run.smithi040.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-21T23:28:37.216 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-21T23:28:37.217 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.217 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.217 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.217 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:37.217 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.217 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.217 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.217 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.218 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.218 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.218 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.218 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.218 INFO:teuthology.orchestra.run.smithi040.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-21T23:28:37.218 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-21T23:28:37.218 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.218 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.219 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.219 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:37.219 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.219 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.219 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.219 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.219 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.219 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.219 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.220 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.220 INFO:teuthology.orchestra.run.smithi040.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-21T23:28:37.220 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-21T23:28:37.220 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.220 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.220 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.220 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.220 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.221 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.221 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.221 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.221 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.221 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.221 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.221 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.221 INFO:teuthology.orchestra.run.smithi040.stdout: "GRAFANA_API_PASSWORD": { 2024-04-21T23:28:37.222 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-21T23:28:37.222 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.222 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.222 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.222 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "admin", 2024-04-21T23:28:37.222 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.222 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.222 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.222 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.223 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.223 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.223 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.223 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.223 INFO:teuthology.orchestra.run.smithi040.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-21T23:28:37.223 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-21T23:28:37.223 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.223 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.224 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.224 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:37.224 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.224 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.224 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.224 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.224 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.224 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.224 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.225 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.225 INFO:teuthology.orchestra.run.smithi040.stdout: "GRAFANA_API_URL": { 2024-04-21T23:28:37.225 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "GRAFANA_API_URL", 2024-04-21T23:28:37.225 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.225 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.225 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.225 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.225 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.226 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.226 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.226 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.226 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.226 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.226 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.226 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.226 INFO:teuthology.orchestra.run.smithi040.stdout: "GRAFANA_API_USERNAME": { 2024-04-21T23:28:37.226 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-21T23:28:37.227 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.227 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.227 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.227 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "admin", 2024-04-21T23:28:37.227 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.227 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.227 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.227 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.227 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.228 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.228 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.228 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.228 INFO:teuthology.orchestra.run.smithi040.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-21T23:28:37.228 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-21T23:28:37.228 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.228 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.228 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.229 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.229 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.229 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.229 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.229 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.229 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.229 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.229 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.229 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.230 INFO:teuthology.orchestra.run.smithi040.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-21T23:28:37.230 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-21T23:28:37.230 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.230 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.230 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.230 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.230 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.230 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.230 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.231 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.231 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.231 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.231 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.231 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.231 INFO:teuthology.orchestra.run.smithi040.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-21T23:28:37.231 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-21T23:28:37.231 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.232 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.232 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.232 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:37.232 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.232 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.232 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.232 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.232 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.233 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.233 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.233 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.233 INFO:teuthology.orchestra.run.smithi040.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-21T23:28:37.233 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-21T23:28:37.233 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.233 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.233 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.233 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.234 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.234 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.234 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.234 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.234 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.234 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.234 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.234 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.235 INFO:teuthology.orchestra.run.smithi040.stdout: "PROMETHEUS_API_HOST": { 2024-04-21T23:28:37.235 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-21T23:28:37.235 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.235 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.235 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.235 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.235 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.235 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.235 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.236 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.236 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.236 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.236 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.236 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.236 INFO:teuthology.orchestra.run.smithi040.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-21T23:28:37.236 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-21T23:28:37.236 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.236 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.237 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.237 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:37.237 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.237 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.237 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.237 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.237 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.237 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.238 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.238 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.238 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-21T23:28:37.238 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-21T23:28:37.238 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.238 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.238 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.238 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.239 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.239 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.239 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.239 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.239 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.239 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.239 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.239 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.241 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-21T23:28:37.241 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-21T23:28:37.241 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.241 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.241 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.241 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.241 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.241 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.242 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.242 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.242 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.242 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.242 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.242 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.242 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-21T23:28:37.242 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-21T23:28:37.243 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.243 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.243 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.243 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:37.243 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.243 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.243 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.243 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.244 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.244 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.244 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.244 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.244 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-21T23:28:37.244 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-21T23:28:37.244 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.244 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.245 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.245 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:37.245 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.245 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.245 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.245 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.245 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.245 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.246 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.246 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.246 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-21T23:28:37.246 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-21T23:28:37.246 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.246 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.246 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.246 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.246 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.247 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.247 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.247 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.247 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.247 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.247 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.247 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.248 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-21T23:28:37.248 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-21T23:28:37.248 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.248 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.248 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.248 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.248 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.248 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.249 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.249 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.249 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.249 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.249 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.249 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.249 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-21T23:28:37.249 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-21T23:28:37.250 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.250 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.250 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.250 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.250 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.250 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.250 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.250 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.251 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.251 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.251 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.251 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.251 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_ENABLED": { 2024-04-21T23:28:37.251 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-21T23:28:37.251 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.251 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.252 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.252 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:37.252 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.252 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.252 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.252 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.252 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.252 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.253 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.253 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.253 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-21T23:28:37.253 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-21T23:28:37.253 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.253 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.253 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.253 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-21T23:28:37.253 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.254 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.254 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.254 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.254 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.254 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.254 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.254 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.254 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-21T23:28:37.255 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-21T23:28:37.255 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:37.255 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.255 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.255 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10", 2024-04-21T23:28:37.257 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.257 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.258 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.258 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.258 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.258 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.258 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.258 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.258 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-21T23:28:37.258 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-21T23:28:37.259 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:37.259 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.259 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.259 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "8", 2024-04-21T23:28:37.259 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.259 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.259 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.259 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.259 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.260 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.260 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.260 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.260 INFO:teuthology.orchestra.run.smithi040.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-21T23:28:37.260 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-21T23:28:37.260 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:37.260 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.260 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.260 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "45", 2024-04-21T23:28:37.261 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.261 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.261 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.261 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.261 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.261 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.261 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.261 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.262 INFO:teuthology.orchestra.run.smithi040.stdout: "RGW_API_ACCESS_KEY": { 2024-04-21T23:28:37.262 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-21T23:28:37.262 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.262 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.262 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.262 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.262 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.262 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.263 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.263 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.263 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.263 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.263 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.263 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.263 INFO:teuthology.orchestra.run.smithi040.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-21T23:28:37.263 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-21T23:28:37.263 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.264 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.264 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.264 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "admin", 2024-04-21T23:28:37.264 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.264 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.264 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.264 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.264 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.265 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.265 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.265 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.265 INFO:teuthology.orchestra.run.smithi040.stdout: "RGW_API_SECRET_KEY": { 2024-04-21T23:28:37.265 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-21T23:28:37.265 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.265 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.265 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.266 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.266 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.266 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.266 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.266 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.266 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.266 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.266 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.266 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.267 INFO:teuthology.orchestra.run.smithi040.stdout: "RGW_API_SSL_VERIFY": { 2024-04-21T23:28:37.267 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-21T23:28:37.267 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.267 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.267 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.267 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:37.267 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.267 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.268 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.268 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.268 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.268 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.268 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.268 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.268 INFO:teuthology.orchestra.run.smithi040.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-21T23:28:37.268 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-21T23:28:37.268 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:37.269 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.269 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.269 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "0", 2024-04-21T23:28:37.269 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.269 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.269 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.269 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.269 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.269 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.270 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.270 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.270 INFO:teuthology.orchestra.run.smithi040.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-21T23:28:37.270 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-21T23:28:37.270 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:37.270 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.270 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.270 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10", 2024-04-21T23:28:37.271 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.271 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.271 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.271 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.271 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.271 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.271 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.271 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.272 INFO:teuthology.orchestra.run.smithi040.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-21T23:28:37.272 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-21T23:28:37.272 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:37.272 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.272 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.272 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "5", 2024-04-21T23:28:37.272 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.272 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.272 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.273 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.273 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.273 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.273 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.273 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.273 INFO:teuthology.orchestra.run.smithi040.stdout: "cross_origin_url": { 2024-04-21T23:28:37.273 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "cross_origin_url", 2024-04-21T23:28:37.273 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.273 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.274 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.274 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.274 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.274 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.274 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.274 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.274 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.274 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.275 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.275 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.275 INFO:teuthology.orchestra.run.smithi040.stdout: "crt_file": { 2024-04-21T23:28:37.275 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "crt_file", 2024-04-21T23:28:37.275 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.275 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.275 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.275 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.275 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.276 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.276 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.276 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.276 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.276 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.276 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.276 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.276 INFO:teuthology.orchestra.run.smithi040.stdout: "debug": { 2024-04-21T23:28:37.276 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "debug", 2024-04-21T23:28:37.277 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.277 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.277 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.277 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.277 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.277 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.277 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.277 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Enable/disable debug options", 2024-04-21T23:28:37.277 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.278 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.278 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.278 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.278 INFO:teuthology.orchestra.run.smithi040.stdout: "jwt_token_ttl": { 2024-04-21T23:28:37.278 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "jwt_token_ttl", 2024-04-21T23:28:37.278 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:37.278 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.279 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.279 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "28800", 2024-04-21T23:28:37.279 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.279 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.279 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.279 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.279 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.279 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.280 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.280 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.280 INFO:teuthology.orchestra.run.smithi040.stdout: "key_file": { 2024-04-21T23:28:37.280 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "key_file", 2024-04-21T23:28:37.280 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.280 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.280 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.280 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.281 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.281 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.281 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.281 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.281 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.281 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.281 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.281 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.281 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:37.282 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:37.282 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.282 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.282 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.282 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.282 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.282 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.282 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.283 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:37.283 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:37.283 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:37.283 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:37.283 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:37.283 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:37.283 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.283 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.284 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.284 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.284 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.284 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.284 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:37.284 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:37.284 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.284 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.285 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.285 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.285 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.285 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.285 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.285 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.285 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.285 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.285 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.286 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.286 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:37.286 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:37.286 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.286 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.286 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.286 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:37.286 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.287 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.287 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.287 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:37.287 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:37.287 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:37.287 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:37.287 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:37.287 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:37.288 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.288 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.288 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.288 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.288 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.288 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.288 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:37.288 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:37.288 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.289 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.289 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.289 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.289 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.289 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.289 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.289 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.289 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.290 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.290 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.290 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.290 INFO:teuthology.orchestra.run.smithi040.stdout: "motd": { 2024-04-21T23:28:37.290 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "motd", 2024-04-21T23:28:37.290 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.290 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.290 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.290 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.291 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.291 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.291 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.291 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "The message of the day", 2024-04-21T23:28:37.291 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.291 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.291 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.291 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.291 INFO:teuthology.orchestra.run.smithi040.stdout: "redirect_resolve_ip_addr": { 2024-04-21T23:28:37.292 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "redirect_resolve_ip_addr", 2024-04-21T23:28:37.292 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.292 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.292 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.292 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.292 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.292 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.292 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.292 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.293 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.293 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.293 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.293 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.293 INFO:teuthology.orchestra.run.smithi040.stdout: "server_addr": { 2024-04-21T23:28:37.293 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "server_addr", 2024-04-21T23:28:37.293 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.293 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.294 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.294 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "::", 2024-04-21T23:28:37.294 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.294 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.294 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.294 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.294 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.294 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.294 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.295 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.295 INFO:teuthology.orchestra.run.smithi040.stdout: "server_port": { 2024-04-21T23:28:37.295 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "server_port", 2024-04-21T23:28:37.295 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:37.295 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.295 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.295 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "8080", 2024-04-21T23:28:37.295 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.295 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.296 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.296 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.296 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.296 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.296 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.296 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.296 INFO:teuthology.orchestra.run.smithi040.stdout: "ssl": { 2024-04-21T23:28:37.296 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ssl", 2024-04-21T23:28:37.297 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.297 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.297 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.297 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:37.297 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.297 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.297 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.297 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.297 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.298 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.298 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.298 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.298 INFO:teuthology.orchestra.run.smithi040.stdout: "ssl_server_port": { 2024-04-21T23:28:37.298 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ssl_server_port", 2024-04-21T23:28:37.298 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:37.298 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.298 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.298 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "8443", 2024-04-21T23:28:37.299 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.299 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.299 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.299 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.299 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.299 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.299 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.299 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.299 INFO:teuthology.orchestra.run.smithi040.stdout: "standby_behaviour": { 2024-04-21T23:28:37.300 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "standby_behaviour", 2024-04-21T23:28:37.300 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.300 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.300 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.300 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "redirect", 2024-04-21T23:28:37.300 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.300 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.300 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.301 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:37.301 INFO:teuthology.orchestra.run.smithi040.stdout: "redirect" 2024-04-21T23:28:37.301 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.301 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.301 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.301 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.301 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.301 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.301 INFO:teuthology.orchestra.run.smithi040.stdout: "standby_error_status_code": { 2024-04-21T23:28:37.302 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "standby_error_status_code", 2024-04-21T23:28:37.302 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:37.302 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.302 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.302 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "500", 2024-04-21T23:28:37.302 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "400", 2024-04-21T23:28:37.302 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "599", 2024-04-21T23:28:37.302 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.303 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.303 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.303 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.303 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.303 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.303 INFO:teuthology.orchestra.run.smithi040.stdout: "url_prefix": { 2024-04-21T23:28:37.303 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "url_prefix", 2024-04-21T23:28:37.303 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.304 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.304 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.304 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.304 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.304 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.304 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.304 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.304 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.304 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.305 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.305 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.305 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.305 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.305 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:37.305 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "devicehealth", 2024-04-21T23:28:37.305 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:37.305 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:37.306 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:37.306 INFO:teuthology.orchestra.run.smithi040.stdout: "enable_monitoring": { 2024-04-21T23:28:37.306 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "enable_monitoring", 2024-04-21T23:28:37.306 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.306 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.306 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.306 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:37.306 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.306 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.307 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.307 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "monitor device health metrics", 2024-04-21T23:28:37.307 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.307 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.307 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.307 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.307 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:37.307 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:37.308 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.308 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.308 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.308 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.308 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.308 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.308 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.308 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:37.308 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:37.309 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:37.309 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:37.309 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:37.309 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:37.309 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.309 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.309 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.309 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.310 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.310 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.310 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:37.310 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:37.310 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.310 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.310 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.310 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.311 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.311 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.311 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.311 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.311 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.311 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.311 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.311 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.312 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:37.312 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:37.312 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.312 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.312 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.312 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:37.312 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.312 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.313 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.313 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:37.313 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:37.313 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:37.313 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:37.313 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:37.313 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:37.313 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.313 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.314 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.314 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.314 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.314 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.314 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:37.314 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:37.314 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.314 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.315 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.315 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.315 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.315 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.315 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.315 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.315 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.315 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.315 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.316 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.316 INFO:teuthology.orchestra.run.smithi040.stdout: "mark_out_threshold": { 2024-04-21T23:28:37.316 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "mark_out_threshold", 2024-04-21T23:28:37.316 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-21T23:28:37.316 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.316 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.316 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "2419200", 2024-04-21T23:28:37.316 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.316 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.317 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.317 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-21T23:28:37.317 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.317 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.317 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.317 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.317 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_name": { 2024-04-21T23:28:37.317 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "pool_name", 2024-04-21T23:28:37.317 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.318 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.318 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.318 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "device_health_metrics", 2024-04-21T23:28:37.318 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.318 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.318 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.318 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-21T23:28:37.318 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.319 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.319 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.319 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.319 INFO:teuthology.orchestra.run.smithi040.stdout: "retention_period": { 2024-04-21T23:28:37.319 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "retention_period", 2024-04-21T23:28:37.319 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-21T23:28:37.319 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.319 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.320 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "15552000", 2024-04-21T23:28:37.320 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.320 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.320 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.320 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "how long to retain device health metrics", 2024-04-21T23:28:37.320 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.320 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.320 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.320 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.321 INFO:teuthology.orchestra.run.smithi040.stdout: "scrape_frequency": { 2024-04-21T23:28:37.321 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "scrape_frequency", 2024-04-21T23:28:37.321 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-21T23:28:37.321 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.321 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.321 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "86400", 2024-04-21T23:28:37.321 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.321 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.322 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.322 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-21T23:28:37.322 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.322 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.322 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.322 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.322 INFO:teuthology.orchestra.run.smithi040.stdout: "self_heal": { 2024-04-21T23:28:37.322 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "self_heal", 2024-04-21T23:28:37.322 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.323 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.323 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.323 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:37.323 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.323 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.323 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.323 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-21T23:28:37.323 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.324 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.324 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.324 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.324 INFO:teuthology.orchestra.run.smithi040.stdout: "sleep_interval": { 2024-04-21T23:28:37.324 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "sleep_interval", 2024-04-21T23:28:37.324 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-21T23:28:37.324 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.324 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.324 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "600", 2024-04-21T23:28:37.325 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.325 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.325 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.325 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "how frequently to wake up and check device health", 2024-04-21T23:28:37.325 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.325 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.325 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.325 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.326 INFO:teuthology.orchestra.run.smithi040.stdout: "warn_threshold": { 2024-04-21T23:28:37.326 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "warn_threshold", 2024-04-21T23:28:37.326 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-21T23:28:37.326 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.326 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.326 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "7257600", 2024-04-21T23:28:37.326 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.326 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.327 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.327 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-21T23:28:37.327 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.327 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.327 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.327 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.327 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.327 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.328 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:37.328 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "diskprediction_local", 2024-04-21T23:28:37.328 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:37.328 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:37.328 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:37.328 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:37.328 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:37.328 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.328 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.329 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.329 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.329 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.329 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.329 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.329 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:37.329 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:37.329 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:37.330 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:37.330 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:37.330 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:37.330 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.330 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.330 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.330 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.330 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.331 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.331 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:37.331 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:37.331 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.331 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.331 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.331 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.331 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.332 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.332 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.332 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.332 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.332 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.332 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.332 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.332 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:37.332 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:37.333 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.333 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.333 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.333 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:37.333 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.333 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.333 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.333 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:37.334 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:37.334 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:37.334 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:37.334 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:37.334 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:37.334 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.334 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.334 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.335 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.335 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.335 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.335 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:37.335 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:37.335 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.335 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.335 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.335 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.336 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.336 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.336 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.336 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.336 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.336 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.336 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.336 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.337 INFO:teuthology.orchestra.run.smithi040.stdout: "predict_interval": { 2024-04-21T23:28:37.337 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "predict_interval", 2024-04-21T23:28:37.337 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.337 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.337 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.337 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "86400", 2024-04-21T23:28:37.337 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.337 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.337 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.338 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.338 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.338 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.338 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.338 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.338 INFO:teuthology.orchestra.run.smithi040.stdout: "predictor_model": { 2024-04-21T23:28:37.338 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "predictor_model", 2024-04-21T23:28:37.339 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.339 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.339 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.339 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "prophetstor", 2024-04-21T23:28:37.339 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.339 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.339 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.339 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.340 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.340 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.340 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.340 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.340 INFO:teuthology.orchestra.run.smithi040.stdout: "sleep_interval": { 2024-04-21T23:28:37.340 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "sleep_interval", 2024-04-21T23:28:37.340 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.340 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.341 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.341 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "600", 2024-04-21T23:28:37.341 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.341 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.341 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.341 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.341 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.341 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.342 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.342 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.342 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.342 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.342 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:37.342 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "influx", 2024-04-21T23:28:37.342 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": false, 2024-04-21T23:28:37.342 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "influxdb python module not found", 2024-04-21T23:28:37.343 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:37.343 INFO:teuthology.orchestra.run.smithi040.stdout: "batch_size": { 2024-04-21T23:28:37.343 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "batch_size", 2024-04-21T23:28:37.343 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:37.343 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.343 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.343 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "5000", 2024-04-21T23:28:37.343 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.343 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.344 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.344 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-21T23:28:37.344 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.344 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.344 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.344 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.344 INFO:teuthology.orchestra.run.smithi040.stdout: "database": { 2024-04-21T23:28:37.344 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "database", 2024-04-21T23:28:37.344 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.345 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.345 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.345 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "ceph", 2024-04-21T23:28:37.345 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.345 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.345 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.345 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-21T23:28:37.345 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.346 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.346 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.346 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.346 INFO:teuthology.orchestra.run.smithi040.stdout: "hostname": { 2024-04-21T23:28:37.346 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "hostname", 2024-04-21T23:28:37.346 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.346 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.346 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.347 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.347 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.347 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.347 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.347 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "InfluxDB server hostname", 2024-04-21T23:28:37.347 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.347 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.347 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.347 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.348 INFO:teuthology.orchestra.run.smithi040.stdout: "interval": { 2024-04-21T23:28:37.348 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "interval", 2024-04-21T23:28:37.348 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-21T23:28:37.348 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.348 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.348 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "30", 2024-04-21T23:28:37.348 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "5", 2024-04-21T23:28:37.348 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.348 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.349 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-21T23:28:37.349 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.349 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.349 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.349 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.349 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:37.349 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:37.349 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.350 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.350 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.350 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.350 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.350 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.350 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.350 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:37.350 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:37.351 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:37.351 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:37.351 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:37.351 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:37.351 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.351 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.351 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.351 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.351 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.352 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.352 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:37.352 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:37.352 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.352 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.352 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.352 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.352 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.352 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.353 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.353 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.353 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.353 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.353 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.353 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.353 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:37.353 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:37.354 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.354 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.354 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.354 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:37.354 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.354 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.354 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.354 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:37.355 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:37.355 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:37.355 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:37.355 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:37.355 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:37.355 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.355 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.355 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.355 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.356 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.356 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.356 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:37.356 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:37.356 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.356 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.356 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.356 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.357 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.357 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.357 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.357 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.357 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.357 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.357 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.357 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.357 INFO:teuthology.orchestra.run.smithi040.stdout: "password": { 2024-04-21T23:28:37.358 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "password", 2024-04-21T23:28:37.358 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.358 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.358 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.358 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.358 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.358 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.358 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.359 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "password of InfluxDB server user", 2024-04-21T23:28:37.359 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.359 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.359 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.359 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.359 INFO:teuthology.orchestra.run.smithi040.stdout: "port": { 2024-04-21T23:28:37.359 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "port", 2024-04-21T23:28:37.359 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:37.359 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.360 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.360 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "8086", 2024-04-21T23:28:37.360 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.360 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.360 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.360 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "InfluxDB server port", 2024-04-21T23:28:37.360 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.360 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.361 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.361 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.361 INFO:teuthology.orchestra.run.smithi040.stdout: "ssl": { 2024-04-21T23:28:37.361 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ssl", 2024-04-21T23:28:37.361 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.361 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.361 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.361 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "false", 2024-04-21T23:28:37.361 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.362 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.362 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.362 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-21T23:28:37.362 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.362 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.362 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.362 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.362 INFO:teuthology.orchestra.run.smithi040.stdout: "threads": { 2024-04-21T23:28:37.363 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "threads", 2024-04-21T23:28:37.363 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:37.363 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.363 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.363 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "5", 2024-04-21T23:28:37.363 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "1", 2024-04-21T23:28:37.363 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "32", 2024-04-21T23:28:37.363 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.363 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-21T23:28:37.364 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.364 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.364 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.364 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.364 INFO:teuthology.orchestra.run.smithi040.stdout: "username": { 2024-04-21T23:28:37.364 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "username", 2024-04-21T23:28:37.364 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.364 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.365 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.365 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.365 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.365 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.365 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.365 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "username of InfluxDB server user", 2024-04-21T23:28:37.365 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.365 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.365 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.366 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.366 INFO:teuthology.orchestra.run.smithi040.stdout: "verify_ssl": { 2024-04-21T23:28:37.366 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "verify_ssl", 2024-04-21T23:28:37.366 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.366 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.366 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.366 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "true", 2024-04-21T23:28:37.366 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.367 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.367 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.367 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-21T23:28:37.367 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.367 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.367 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.367 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.367 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.368 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.368 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:37.368 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "insights", 2024-04-21T23:28:37.368 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:37.368 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:37.368 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:37.368 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:37.368 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:37.369 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.369 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.369 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.369 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.369 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.369 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.369 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.369 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:37.369 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:37.370 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:37.370 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:37.370 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:37.370 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:37.370 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.370 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.370 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.370 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.370 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.371 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.371 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:37.371 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:37.371 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.371 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.371 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.371 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.371 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.372 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.372 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.372 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.372 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.372 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.372 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.372 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.372 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:37.373 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:37.373 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.373 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.373 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.373 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:37.373 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.373 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.373 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.373 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:37.374 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:37.374 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:37.374 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:37.374 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:37.374 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:37.374 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.374 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.374 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.375 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.375 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.375 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.375 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:37.375 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:37.375 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.375 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.375 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.375 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.376 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.376 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.376 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.376 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.376 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.376 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.376 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.376 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.377 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.377 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.377 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:37.377 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "iostat", 2024-04-21T23:28:37.377 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:37.377 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:37.377 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:37.377 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:37.378 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:37.378 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.378 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.378 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.378 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.378 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.378 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.378 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.378 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:37.379 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:37.379 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:37.379 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:37.379 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:37.379 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:37.379 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.379 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.379 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.380 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.380 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.380 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.380 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:37.380 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:37.380 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.380 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.380 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.380 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.381 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.381 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.381 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.381 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.381 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.381 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.381 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.381 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.381 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:37.382 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:37.382 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.382 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.382 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.382 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:37.382 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.382 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.382 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.383 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:37.383 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:37.383 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:37.383 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:37.383 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:37.383 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:37.383 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.383 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.383 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.384 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.384 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.384 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.384 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:37.384 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:37.384 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.384 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.384 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.384 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.385 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.385 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.385 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.385 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.385 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.385 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.385 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.385 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.385 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.386 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.386 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:37.386 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "k8sevents", 2024-04-21T23:28:37.386 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:37.386 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:37.386 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:37.386 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_event_retention_days": { 2024-04-21T23:28:37.386 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ceph_event_retention_days", 2024-04-21T23:28:37.387 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:37.387 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.387 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.387 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "7", 2024-04-21T23:28:37.387 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.387 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.387 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.387 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-21T23:28:37.387 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.388 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.388 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.388 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.388 INFO:teuthology.orchestra.run.smithi040.stdout: "config_check_secs": { 2024-04-21T23:28:37.388 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "config_check_secs", 2024-04-21T23:28:37.388 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:37.388 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.388 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.389 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10", 2024-04-21T23:28:37.389 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "10", 2024-04-21T23:28:37.389 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.389 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.389 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-21T23:28:37.389 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.389 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.389 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.389 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.390 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:37.390 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:37.390 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.390 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.390 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.390 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.390 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.390 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.391 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.391 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:37.391 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:37.391 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:37.391 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:37.391 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:37.391 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:37.391 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.391 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.392 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.392 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.392 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.392 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.392 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:37.392 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:37.392 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.392 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.393 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.393 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.393 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.393 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.393 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.393 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.393 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.393 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.393 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.394 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.394 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:37.394 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:37.394 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.394 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.394 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.394 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:37.394 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.395 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.395 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.395 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:37.395 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:37.395 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:37.395 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:37.395 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:37.395 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:37.396 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.396 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.396 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.396 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.396 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.396 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.396 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:37.396 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:37.396 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.397 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.397 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.397 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.397 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.397 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.397 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.397 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.397 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.398 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.398 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.398 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.398 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.398 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.398 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:37.398 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "localpool", 2024-04-21T23:28:37.398 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:37.399 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:37.399 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:37.399 INFO:teuthology.orchestra.run.smithi040.stdout: "failure_domain": { 2024-04-21T23:28:37.399 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "failure_domain", 2024-04-21T23:28:37.399 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.399 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.399 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.399 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "host", 2024-04-21T23:28:37.400 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.400 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.400 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.400 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "failure domain for any created local pool", 2024-04-21T23:28:37.400 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-21T23:28:37.400 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.400 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.400 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.400 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:37.401 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:37.401 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.401 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.401 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.401 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.401 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.401 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.401 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.402 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:37.402 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:37.402 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:37.402 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:37.402 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:37.402 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:37.402 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.402 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.402 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.403 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.403 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.403 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.403 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:37.403 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:37.403 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.403 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.403 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.404 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.404 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.404 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.404 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.404 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.404 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.404 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.404 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.404 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.405 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:37.405 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:37.405 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.405 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.405 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.405 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:37.405 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.405 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.405 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.406 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:37.406 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:37.406 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:37.406 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:37.406 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:37.406 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:37.406 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.406 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.407 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.407 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.407 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.407 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.407 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:37.407 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:37.407 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.407 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.407 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.408 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.408 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.408 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.408 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.408 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.408 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.408 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.408 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.408 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.409 INFO:teuthology.orchestra.run.smithi040.stdout: "min_size": { 2024-04-21T23:28:37.409 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "min_size", 2024-04-21T23:28:37.409 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:37.409 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.409 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.409 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.409 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.409 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.410 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.410 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "default min_size for any created local pool", 2024-04-21T23:28:37.410 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-21T23:28:37.410 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.410 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.410 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.410 INFO:teuthology.orchestra.run.smithi040.stdout: "num_rep": { 2024-04-21T23:28:37.410 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "num_rep", 2024-04-21T23:28:37.410 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:37.411 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.411 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.411 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "3", 2024-04-21T23:28:37.411 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.411 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.411 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.411 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "default replica count for any created local pool", 2024-04-21T23:28:37.411 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.411 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.412 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.412 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.412 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num": { 2024-04-21T23:28:37.412 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "pg_num", 2024-04-21T23:28:37.412 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:37.412 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.412 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.413 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "128", 2024-04-21T23:28:37.413 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.413 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.413 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.413 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "default pg_num for any created local pool", 2024-04-21T23:28:37.413 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.413 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.413 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.414 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.414 INFO:teuthology.orchestra.run.smithi040.stdout: "prefix": { 2024-04-21T23:28:37.414 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "prefix", 2024-04-21T23:28:37.414 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.414 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.414 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.414 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.414 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.415 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.415 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.415 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "name prefix for any created local pool", 2024-04-21T23:28:37.415 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.415 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.415 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.415 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.415 INFO:teuthology.orchestra.run.smithi040.stdout: "subtree": { 2024-04-21T23:28:37.415 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "subtree", 2024-04-21T23:28:37.416 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.416 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.416 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.416 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "rack", 2024-04-21T23:28:37.416 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.416 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.416 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.416 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-21T23:28:37.417 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-21T23:28:37.417 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.417 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.417 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.417 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.417 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.417 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:37.417 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "mds_autoscaler", 2024-04-21T23:28:37.417 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:37.418 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:37.418 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:37.418 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:37.418 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:37.418 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.418 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.418 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.418 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.418 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.419 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.419 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.419 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:37.419 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:37.419 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:37.419 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:37.419 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:37.419 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:37.420 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.420 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.420 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.420 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.420 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.420 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.420 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:37.420 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:37.420 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.421 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.421 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.421 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.421 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.421 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.421 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.421 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.421 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.421 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.422 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.422 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.422 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:37.422 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:37.422 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.422 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.422 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.422 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:37.422 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.423 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.423 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.423 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:37.423 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:37.423 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:37.423 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:37.423 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:37.423 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:37.424 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.424 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.424 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.424 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.424 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.424 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.424 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:37.424 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:37.424 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.425 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.425 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.425 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.425 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.425 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.425 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.425 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.425 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.426 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.426 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.426 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.426 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.426 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.426 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:37.426 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "mirroring", 2024-04-21T23:28:37.426 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:37.427 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:37.427 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:37.427 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:37.427 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:37.427 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.427 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.427 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.427 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.428 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.428 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.428 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.428 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:37.428 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:37.428 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:37.428 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:37.429 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:37.429 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:37.429 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.429 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.429 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.429 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.429 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.429 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.430 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:37.430 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:37.430 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.430 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.430 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.430 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.431 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.431 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.431 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.431 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.431 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.431 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.431 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.432 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.432 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:37.432 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:37.432 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.432 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.432 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.433 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:37.433 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.433 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.433 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.433 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:37.433 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:37.433 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:37.433 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:37.434 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:37.434 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:37.434 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.434 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.434 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.434 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.434 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.435 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.435 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:37.435 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:37.435 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.435 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.436 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.436 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.436 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.436 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.436 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.436 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.437 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.437 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.437 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.437 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.437 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.437 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.438 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:37.438 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "nfs", 2024-04-21T23:28:37.438 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:37.439 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:37.439 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:37.439 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:37.439 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:37.439 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.440 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.440 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.440 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.440 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.440 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.441 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.441 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:37.441 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:37.441 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:37.441 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:37.441 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:37.441 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:37.442 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.442 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.442 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.442 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.442 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.442 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.442 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:37.443 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:37.443 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.443 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.443 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.443 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.443 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.444 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.444 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.444 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.444 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.444 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.444 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.445 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.445 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:37.445 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:37.445 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.445 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.446 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.446 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:37.446 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.446 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.446 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.446 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:37.447 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:37.447 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:37.447 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:37.447 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:37.447 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:37.448 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.448 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.448 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.448 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.448 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.449 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.449 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:37.449 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:37.449 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.449 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.449 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.450 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.450 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.450 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.450 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.450 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.450 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.451 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.451 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.451 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.451 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.451 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.451 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:37.451 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "orchestrator", 2024-04-21T23:28:37.452 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:37.452 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:37.452 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:37.452 INFO:teuthology.orchestra.run.smithi040.stdout: "fail_fs": { 2024-04-21T23:28:37.452 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "fail_fs", 2024-04-21T23:28:37.453 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.453 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.453 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.453 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.453 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.454 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.454 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.454 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-21T23:28:37.454 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.454 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.455 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.455 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.455 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:37.455 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:37.456 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.456 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.456 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.456 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.457 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.457 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.457 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.457 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:37.458 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:37.458 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:37.458 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:37.458 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:37.458 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:37.458 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.459 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.459 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.459 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.459 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.459 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.459 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:37.460 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:37.460 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.460 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.460 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.460 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.460 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.461 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.461 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.461 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.461 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.461 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.462 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.462 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.462 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:37.462 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:37.462 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.462 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.463 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.463 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:37.463 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.463 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.463 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.463 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:37.463 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:37.464 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:37.464 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:37.464 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:37.464 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:37.464 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.464 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.464 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.465 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.465 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.465 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.465 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:37.465 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:37.465 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.465 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.466 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.466 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.466 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.466 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.466 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.466 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.466 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.467 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.467 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.467 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.467 INFO:teuthology.orchestra.run.smithi040.stdout: "orchestrator": { 2024-04-21T23:28:37.467 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "orchestrator", 2024-04-21T23:28:37.467 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.468 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.468 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.468 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.468 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.468 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.468 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.468 INFO:teuthology.orchestra.run.smithi040.stdout: "cephadm", 2024-04-21T23:28:37.468 INFO:teuthology.orchestra.run.smithi040.stdout: "rook", 2024-04-21T23:28:37.469 INFO:teuthology.orchestra.run.smithi040.stdout: "test_orchestrator" 2024-04-21T23:28:37.469 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.469 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Orchestrator backend", 2024-04-21T23:28:37.469 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.469 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.469 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.469 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.470 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.470 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.470 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:37.470 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "osd_perf_query", 2024-04-21T23:28:37.470 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:37.470 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:37.470 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:37.471 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:37.471 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:37.471 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.471 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.471 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.471 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.471 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.471 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.472 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.472 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:37.472 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:37.472 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:37.472 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:37.472 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:37.473 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:37.473 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.473 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.473 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.473 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.473 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.474 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.474 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:37.474 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:37.474 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.474 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.474 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.475 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.475 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.475 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.475 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.475 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.475 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.475 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.476 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.476 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.476 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:37.476 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:37.476 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.476 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.477 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.477 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:37.477 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.477 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.477 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.477 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:37.478 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:37.478 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:37.478 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:37.478 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:37.478 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:37.479 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.479 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.479 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.479 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.479 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.479 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.480 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:37.480 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:37.480 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.480 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.480 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.481 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.481 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.481 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.481 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.482 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.482 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.482 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.482 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.482 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.483 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.483 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.483 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:37.483 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "osd_support", 2024-04-21T23:28:37.483 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:37.483 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:37.484 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:37.484 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:37.484 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:37.484 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.484 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.485 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.485 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.485 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.485 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.485 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.486 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:37.486 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:37.486 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:37.486 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:37.486 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:37.486 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:37.487 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.487 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.487 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.487 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.487 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.488 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.488 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:37.488 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:37.488 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.488 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.488 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.489 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.489 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.489 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.489 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.489 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.490 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.490 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.490 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.490 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.491 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:37.491 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:37.491 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.491 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.491 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.492 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:37.492 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.492 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.493 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.493 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:37.493 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:37.493 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:37.493 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:37.494 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:37.494 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:37.494 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.494 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.494 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.495 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.495 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.495 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.495 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:37.495 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:37.495 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.496 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.496 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.496 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.496 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.496 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.496 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.497 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.497 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.497 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.497 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.497 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.497 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.498 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.498 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:37.498 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "pg_autoscaler", 2024-04-21T23:28:37.498 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:37.498 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:37.499 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:37.499 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:37.499 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:37.499 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.499 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.499 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.500 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.500 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.500 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.500 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.500 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:37.500 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:37.501 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:37.501 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:37.501 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:37.501 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:37.501 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.501 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.501 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.502 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.502 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.502 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.502 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:37.502 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:37.503 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.503 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.503 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.503 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.503 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.504 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.504 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.504 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.504 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.504 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.504 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.505 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.505 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:37.505 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:37.505 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.505 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.505 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.506 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:37.506 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.506 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.506 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.507 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:37.507 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:37.507 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:37.507 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:37.507 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:37.508 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:37.508 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.508 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.508 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.509 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.509 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.509 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.509 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:37.509 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:37.510 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.510 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.510 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.510 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.510 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.511 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.511 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.511 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.511 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.511 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.511 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.512 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.512 INFO:teuthology.orchestra.run.smithi040.stdout: "sleep_interval": { 2024-04-21T23:28:37.512 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "sleep_interval", 2024-04-21T23:28:37.512 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-21T23:28:37.512 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.512 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.512 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "60", 2024-04-21T23:28:37.513 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.513 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.513 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.513 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.513 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.513 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.513 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.514 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.514 INFO:teuthology.orchestra.run.smithi040.stdout: "threshold": { 2024-04-21T23:28:37.515 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "threshold", 2024-04-21T23:28:37.515 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "float", 2024-04-21T23:28:37.515 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.516 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.516 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "3.0", 2024-04-21T23:28:37.516 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "1.0", 2024-04-21T23:28:37.516 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.516 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.516 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "scaling threshold", 2024-04-21T23:28:37.517 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-21T23:28:37.517 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.517 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.517 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.517 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.517 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.518 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:37.518 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "progress", 2024-04-21T23:28:37.518 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:37.518 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:37.518 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:37.518 INFO:teuthology.orchestra.run.smithi040.stdout: "allow_pg_recovery_event": { 2024-04-21T23:28:37.518 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "allow_pg_recovery_event", 2024-04-21T23:28:37.519 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.519 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.519 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.519 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.519 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.519 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.520 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.520 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-21T23:28:37.520 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.520 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.521 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.521 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.521 INFO:teuthology.orchestra.run.smithi040.stdout: "enabled": { 2024-04-21T23:28:37.521 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "enabled", 2024-04-21T23:28:37.521 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.521 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.521 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.522 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:37.522 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.522 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.522 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.522 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.522 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.523 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.523 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.523 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.524 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:37.524 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:37.524 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.524 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.525 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.525 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.525 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.525 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.526 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.526 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:37.526 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:37.526 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:37.527 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:37.527 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:37.527 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:37.527 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.528 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.528 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.528 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.529 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.529 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.529 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:37.529 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:37.529 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.530 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.530 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.530 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.531 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.531 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.531 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.531 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.531 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.532 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.532 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.532 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.532 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:37.532 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:37.532 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.532 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.533 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.533 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:37.533 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.533 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.534 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.534 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:37.534 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:37.534 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:37.535 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:37.535 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:37.535 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:37.535 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.536 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.536 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.536 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.536 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.537 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.537 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:37.537 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:37.537 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.537 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.537 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.537 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.537 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.538 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.538 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.538 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.538 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.538 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.539 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.539 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.539 INFO:teuthology.orchestra.run.smithi040.stdout: "max_completed_events": { 2024-04-21T23:28:37.539 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "max_completed_events", 2024-04-21T23:28:37.539 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:37.539 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.539 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.540 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "50", 2024-04-21T23:28:37.540 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.540 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.540 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.540 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "number of past completed events to remember", 2024-04-21T23:28:37.540 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.541 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.541 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.541 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.541 INFO:teuthology.orchestra.run.smithi040.stdout: "sleep_interval": { 2024-04-21T23:28:37.541 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "sleep_interval", 2024-04-21T23:28:37.541 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-21T23:28:37.542 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.542 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.542 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "5", 2024-04-21T23:28:37.542 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.542 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.543 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.543 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "how long the module is going to sleep", 2024-04-21T23:28:37.543 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.543 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.543 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.543 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.544 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.544 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.544 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:37.544 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "prometheus", 2024-04-21T23:28:37.544 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:37.545 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:37.545 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:37.545 INFO:teuthology.orchestra.run.smithi040.stdout: "cache": { 2024-04-21T23:28:37.545 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "cache", 2024-04-21T23:28:37.545 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.545 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.546 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.546 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:37.546 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.546 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.547 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.547 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.547 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.547 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.547 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.547 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.548 INFO:teuthology.orchestra.run.smithi040.stdout: "exclude_perf_counters": { 2024-04-21T23:28:37.548 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "exclude_perf_counters", 2024-04-21T23:28:37.548 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.548 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.548 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.548 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:37.549 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.549 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.549 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.549 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-21T23:28:37.549 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-21T23:28:37.550 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.550 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.550 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.550 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:37.550 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:37.551 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.551 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.551 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.551 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.551 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.552 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.552 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.552 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:37.552 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:37.552 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:37.552 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:37.553 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:37.553 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:37.553 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.553 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.553 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.553 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.554 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.554 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.554 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:37.554 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:37.554 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.554 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.554 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.555 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.555 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.555 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.555 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.555 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.555 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.556 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.556 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.556 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.556 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:37.556 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:37.556 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.557 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.557 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.557 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:37.557 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.557 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.557 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.558 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:37.558 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:37.558 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:37.559 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:37.559 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:37.559 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:37.559 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.559 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.559 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.559 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.560 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.560 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.560 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:37.560 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:37.560 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.560 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.561 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.561 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.561 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.561 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.561 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.561 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.561 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.562 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.562 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.562 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.562 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_stats_pools": { 2024-04-21T23:28:37.562 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rbd_stats_pools", 2024-04-21T23:28:37.562 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.563 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.563 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.563 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.563 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.564 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.564 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.564 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.564 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.564 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.565 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.565 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.565 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-21T23:28:37.565 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-21T23:28:37.565 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:37.565 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.565 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.566 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "300", 2024-04-21T23:28:37.566 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.566 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.566 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.566 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.566 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.566 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.567 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.567 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.567 INFO:teuthology.orchestra.run.smithi040.stdout: "scrape_interval": { 2024-04-21T23:28:37.567 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "scrape_interval", 2024-04-21T23:28:37.567 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "float", 2024-04-21T23:28:37.567 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.568 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.568 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "15.0", 2024-04-21T23:28:37.568 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.568 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.568 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.568 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.568 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.569 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.569 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.569 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.569 INFO:teuthology.orchestra.run.smithi040.stdout: "server_addr": { 2024-04-21T23:28:37.570 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "server_addr", 2024-04-21T23:28:37.570 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.570 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.570 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.570 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "::", 2024-04-21T23:28:37.571 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.571 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.571 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.571 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-21T23:28:37.571 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.572 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.572 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.572 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.572 INFO:teuthology.orchestra.run.smithi040.stdout: "server_port": { 2024-04-21T23:28:37.573 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "server_port", 2024-04-21T23:28:37.573 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:37.573 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.573 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.573 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "9283", 2024-04-21T23:28:37.573 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.574 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.574 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.574 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-21T23:28:37.574 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.574 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.574 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.575 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.575 INFO:teuthology.orchestra.run.smithi040.stdout: "stale_cache_strategy": { 2024-04-21T23:28:37.575 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "stale_cache_strategy", 2024-04-21T23:28:37.575 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.575 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.575 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.575 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "log", 2024-04-21T23:28:37.575 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.576 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.576 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.576 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.576 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.576 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.576 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.576 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.577 INFO:teuthology.orchestra.run.smithi040.stdout: "standby_behaviour": { 2024-04-21T23:28:37.577 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "standby_behaviour", 2024-04-21T23:28:37.577 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.577 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.577 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.577 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "default", 2024-04-21T23:28:37.578 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.578 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.578 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.578 INFO:teuthology.orchestra.run.smithi040.stdout: "default", 2024-04-21T23:28:37.578 INFO:teuthology.orchestra.run.smithi040.stdout: "error" 2024-04-21T23:28:37.578 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.578 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.579 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.579 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.579 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.579 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.580 INFO:teuthology.orchestra.run.smithi040.stdout: "standby_error_status_code": { 2024-04-21T23:28:37.580 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "standby_error_status_code", 2024-04-21T23:28:37.580 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:37.580 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.580 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.580 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "500", 2024-04-21T23:28:37.581 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "400", 2024-04-21T23:28:37.581 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "599", 2024-04-21T23:28:37.581 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.581 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.581 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.582 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.582 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.582 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.582 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.582 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.583 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:37.583 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rbd_support", 2024-04-21T23:28:37.583 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:37.583 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:37.583 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:37.583 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:37.584 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:37.584 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.584 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.584 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.585 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.585 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.585 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.585 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.586 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:37.586 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:37.586 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:37.587 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:37.587 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:37.587 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:37.587 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.587 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.587 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.588 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.588 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.588 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.588 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:37.588 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:37.588 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.589 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.589 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.589 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.589 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.589 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.590 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.590 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.590 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.590 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.591 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.591 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.591 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:37.591 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:37.591 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.591 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.592 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.592 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:37.592 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.592 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.592 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.593 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:37.593 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:37.593 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:37.593 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:37.593 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:37.593 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:37.594 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.594 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.594 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.594 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.595 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.595 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.595 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:37.595 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:37.595 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.596 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.596 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.596 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.596 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.597 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.597 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.597 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.597 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.598 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.598 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.598 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.599 INFO:teuthology.orchestra.run.smithi040.stdout: "max_concurrent_snap_create": { 2024-04-21T23:28:37.599 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "max_concurrent_snap_create", 2024-04-21T23:28:37.599 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:37.599 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.599 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.600 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10", 2024-04-21T23:28:37.600 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.600 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.600 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.601 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.601 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.601 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.601 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.602 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.602 INFO:teuthology.orchestra.run.smithi040.stdout: "mirror_snapshot_schedule": { 2024-04-21T23:28:37.602 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "mirror_snapshot_schedule", 2024-04-21T23:28:37.602 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.603 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.603 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.603 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.603 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.604 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.604 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.604 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.604 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.604 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.605 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.605 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.605 INFO:teuthology.orchestra.run.smithi040.stdout: "trash_purge_schedule": { 2024-04-21T23:28:37.605 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "trash_purge_schedule", 2024-04-21T23:28:37.606 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.606 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.606 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.606 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.606 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.606 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.607 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.607 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.607 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.607 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.607 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.607 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.608 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.608 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.608 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:37.608 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "restful", 2024-04-21T23:28:37.608 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:37.608 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:37.608 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:37.609 INFO:teuthology.orchestra.run.smithi040.stdout: "enable_auth": { 2024-04-21T23:28:37.609 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "enable_auth", 2024-04-21T23:28:37.609 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.609 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.609 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.609 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:37.609 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.610 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.610 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.610 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.610 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.610 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.611 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.611 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.611 INFO:teuthology.orchestra.run.smithi040.stdout: "key_file": { 2024-04-21T23:28:37.611 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "key_file", 2024-04-21T23:28:37.611 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.611 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.611 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.612 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.612 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.612 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.612 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.612 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.612 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.612 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.613 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.613 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.613 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:37.613 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:37.613 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.613 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.613 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.614 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.614 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.614 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.614 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.614 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:37.614 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:37.615 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:37.615 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:37.615 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:37.615 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:37.615 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.615 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.616 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.616 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.616 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.616 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.616 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:37.616 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:37.616 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.617 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.617 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.617 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.617 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.617 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.617 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.617 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.618 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.618 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.618 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.618 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.618 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:37.618 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:37.618 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.619 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.619 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.619 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:37.619 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.619 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.619 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.619 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:37.620 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:37.620 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:37.620 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:37.620 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:37.620 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:37.621 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.621 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.621 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.621 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.621 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.621 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.622 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:37.622 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:37.622 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.622 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.622 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.622 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.623 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.623 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.623 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.623 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.624 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.624 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.624 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.624 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.625 INFO:teuthology.orchestra.run.smithi040.stdout: "server_addr": { 2024-04-21T23:28:37.625 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "server_addr", 2024-04-21T23:28:37.625 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.625 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.626 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.626 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.626 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.626 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.627 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.627 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.627 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.627 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.628 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.628 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.628 INFO:teuthology.orchestra.run.smithi040.stdout: "server_port": { 2024-04-21T23:28:37.628 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "server_port", 2024-04-21T23:28:37.629 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.629 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.629 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.629 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.629 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.629 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.629 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.629 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.630 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.630 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.630 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.630 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.630 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.631 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.631 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:37.631 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rgw", 2024-04-21T23:28:37.631 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:37.631 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:37.631 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:37.632 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:37.632 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:37.632 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.632 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.632 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.633 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.633 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.633 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.633 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.634 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:37.634 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:37.634 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:37.634 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:37.634 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:37.635 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:37.635 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.635 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.635 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.635 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.635 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.635 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.636 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:37.636 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:37.636 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.636 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.636 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.636 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.637 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.637 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.637 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.637 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.637 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.638 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.638 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.638 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.638 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:37.638 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:37.638 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.639 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.639 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.639 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:37.639 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.639 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.639 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.640 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:37.640 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:37.640 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:37.640 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:37.640 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:37.640 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:37.640 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.641 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.641 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.641 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.641 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.641 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.641 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:37.642 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:37.642 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.642 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.642 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.642 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.642 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.643 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.643 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.643 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.643 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.643 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.644 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.644 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.644 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.644 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.645 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:37.645 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rook", 2024-04-21T23:28:37.645 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:37.645 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:37.645 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:37.645 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:37.646 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:37.646 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.646 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.646 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.646 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.647 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.647 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.647 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.647 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:37.647 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:37.647 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:37.647 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:37.648 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:37.648 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:37.648 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.648 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.648 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.648 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.648 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.649 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.649 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:37.649 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:37.649 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.649 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.649 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.650 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.650 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.650 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.650 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.650 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.650 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.651 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.651 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.651 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.651 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:37.651 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:37.651 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.651 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.651 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.652 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:37.652 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.652 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.652 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.652 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:37.652 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:37.653 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:37.653 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:37.653 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:37.653 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:37.653 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.653 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.654 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.654 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.654 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.654 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.654 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:37.655 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:37.655 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.655 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.655 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.655 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.656 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.656 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.656 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.656 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.656 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.656 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.656 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.657 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.657 INFO:teuthology.orchestra.run.smithi040.stdout: "storage_class": { 2024-04-21T23:28:37.657 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "storage_class", 2024-04-21T23:28:37.657 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.657 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.657 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.657 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "local", 2024-04-21T23:28:37.657 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.658 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.658 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.658 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-21T23:28:37.658 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.658 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.659 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.659 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.659 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.660 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.660 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:37.660 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "selftest", 2024-04-21T23:28:37.660 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:37.660 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:37.660 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:37.661 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:37.661 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:37.661 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.661 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.661 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.661 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.661 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.662 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.662 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.662 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:37.662 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:37.662 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:37.662 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:37.663 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:37.663 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:37.666 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.666 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.666 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.666 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.666 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.666 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.667 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:37.667 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:37.667 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.667 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.667 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.667 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.667 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.668 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.668 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.668 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.668 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.669 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.669 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.669 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.669 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:37.669 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:37.670 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.670 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.670 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.670 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:37.671 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.671 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.671 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.671 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:37.671 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:37.672 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:37.672 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:37.672 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:37.672 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:37.672 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.673 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.673 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.673 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.674 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.674 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.674 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:37.674 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:37.674 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.674 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.675 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.675 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.675 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.675 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.675 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.675 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.675 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.676 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.676 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.676 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.676 INFO:teuthology.orchestra.run.smithi040.stdout: "roption1": { 2024-04-21T23:28:37.676 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "roption1", 2024-04-21T23:28:37.677 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.677 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.677 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.677 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.678 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.678 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.678 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.678 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.679 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.679 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.679 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.680 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.680 INFO:teuthology.orchestra.run.smithi040.stdout: "roption2": { 2024-04-21T23:28:37.680 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "roption2", 2024-04-21T23:28:37.680 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.680 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.681 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.681 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "xyz", 2024-04-21T23:28:37.681 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.681 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.681 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.682 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.682 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.682 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.682 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.682 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.683 INFO:teuthology.orchestra.run.smithi040.stdout: "rwoption1": { 2024-04-21T23:28:37.683 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rwoption1", 2024-04-21T23:28:37.683 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.683 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.684 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.684 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.684 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.684 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.684 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.685 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.685 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.685 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.686 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.686 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.686 INFO:teuthology.orchestra.run.smithi040.stdout: "rwoption2": { 2024-04-21T23:28:37.686 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rwoption2", 2024-04-21T23:28:37.687 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:37.687 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.687 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.687 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.688 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.688 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.688 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.688 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.688 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.688 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.689 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.689 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.689 INFO:teuthology.orchestra.run.smithi040.stdout: "rwoption3": { 2024-04-21T23:28:37.689 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rwoption3", 2024-04-21T23:28:37.689 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "float", 2024-04-21T23:28:37.690 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.690 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.690 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.690 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.691 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.691 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.691 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.691 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.691 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.692 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.692 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.692 INFO:teuthology.orchestra.run.smithi040.stdout: "rwoption4": { 2024-04-21T23:28:37.692 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rwoption4", 2024-04-21T23:28:37.693 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.693 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.693 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.693 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.693 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.693 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.693 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.694 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.694 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.694 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.694 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.694 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.694 INFO:teuthology.orchestra.run.smithi040.stdout: "rwoption5": { 2024-04-21T23:28:37.695 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rwoption5", 2024-04-21T23:28:37.695 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.695 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.695 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.695 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.695 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.695 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.696 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.696 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.696 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.696 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.697 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.697 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.697 INFO:teuthology.orchestra.run.smithi040.stdout: "rwoption6": { 2024-04-21T23:28:37.697 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rwoption6", 2024-04-21T23:28:37.698 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.698 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.699 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.699 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:37.699 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.699 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.700 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.700 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.701 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.701 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.701 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.701 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.702 INFO:teuthology.orchestra.run.smithi040.stdout: "rwoption7": { 2024-04-21T23:28:37.702 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rwoption7", 2024-04-21T23:28:37.702 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:37.703 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.703 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.703 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.703 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "1", 2024-04-21T23:28:37.703 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "42", 2024-04-21T23:28:37.704 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.704 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.704 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.704 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.705 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.705 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.705 INFO:teuthology.orchestra.run.smithi040.stdout: "testkey": { 2024-04-21T23:28:37.706 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "testkey", 2024-04-21T23:28:37.706 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.706 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.706 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.707 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.707 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.707 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.707 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.708 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.708 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.708 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.708 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.709 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.709 INFO:teuthology.orchestra.run.smithi040.stdout: "testlkey": { 2024-04-21T23:28:37.709 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "testlkey", 2024-04-21T23:28:37.709 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.710 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.710 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.710 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.711 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.711 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.711 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.711 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.711 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.712 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.712 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.712 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.712 INFO:teuthology.orchestra.run.smithi040.stdout: "testnewline": { 2024-04-21T23:28:37.712 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "testnewline", 2024-04-21T23:28:37.713 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.713 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.713 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.713 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.714 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.714 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.714 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.714 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.715 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.715 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.715 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.715 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.715 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.715 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.716 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:37.716 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "snap_schedule", 2024-04-21T23:28:37.716 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:37.716 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:37.717 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:37.717 INFO:teuthology.orchestra.run.smithi040.stdout: "allow_m_granularity": { 2024-04-21T23:28:37.717 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "allow_m_granularity", 2024-04-21T23:28:37.717 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.718 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.718 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.718 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.718 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.719 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.719 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.719 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "allow minute scheduled snapshots", 2024-04-21T23:28:37.719 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.719 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.720 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.720 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.720 INFO:teuthology.orchestra.run.smithi040.stdout: "dump_on_update": { 2024-04-21T23:28:37.720 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "dump_on_update", 2024-04-21T23:28:37.720 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.721 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.721 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.721 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.721 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.722 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.722 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.722 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "dump database to debug log on update", 2024-04-21T23:28:37.722 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.723 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.723 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.723 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.723 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:37.724 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:37.724 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.724 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.724 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.725 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.725 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.725 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.725 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.726 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:37.726 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:37.726 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:37.726 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:37.726 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:37.727 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:37.727 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.727 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.727 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.727 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.727 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.728 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.728 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:37.728 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:37.728 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.729 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.729 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.729 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.729 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.730 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.730 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.730 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.730 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.730 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.731 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.731 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.731 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:37.731 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:37.732 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.732 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.732 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.732 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:37.732 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.733 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.733 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.733 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:37.733 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:37.734 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:37.734 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:37.734 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:37.735 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:37.735 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.735 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.736 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.736 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.736 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.736 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.736 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:37.737 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:37.737 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.737 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.737 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.737 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.738 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.738 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.738 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.738 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.739 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.739 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.739 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.740 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.740 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.740 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.740 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:37.741 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "stats", 2024-04-21T23:28:37.741 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:37.741 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:37.741 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:37.742 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:37.742 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:37.743 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.743 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.743 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.744 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.744 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.744 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.744 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.745 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:37.745 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:37.745 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:37.746 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:37.746 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:37.746 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:37.747 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.747 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.747 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.747 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.748 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.748 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.748 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:37.749 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:37.749 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.749 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.750 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.750 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.751 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.751 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.751 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.752 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.752 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.753 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.753 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.753 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.754 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:37.754 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:37.754 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.754 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.755 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.755 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:37.755 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.755 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.756 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.756 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:37.756 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:37.756 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:37.757 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:37.757 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:37.757 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:37.758 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.758 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.758 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.758 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.759 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.759 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.760 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:37.760 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:37.760 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.760 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.761 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.761 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.761 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.762 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.762 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.762 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.762 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.762 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.763 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.763 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.763 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.763 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.763 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:37.763 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "status", 2024-04-21T23:28:37.764 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:37.764 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:37.764 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:37.764 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:37.765 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:37.765 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.765 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.765 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.766 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.766 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.766 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.766 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.767 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:37.767 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:37.767 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:37.768 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:37.768 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:37.768 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:37.768 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.769 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.769 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.769 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.769 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.769 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.770 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:37.770 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:37.770 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.770 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.770 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.770 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.771 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.771 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.771 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.771 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.772 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.772 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.772 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.772 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.772 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:37.773 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:37.773 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.773 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.773 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.773 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:37.774 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.774 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.774 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.775 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:37.775 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:37.775 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:37.776 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:37.776 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:37.776 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:37.776 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.777 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.777 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.777 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.778 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.778 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.778 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:37.778 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:37.779 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.779 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.779 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.779 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.779 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.779 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.780 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.780 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.780 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.780 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.780 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.781 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.781 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.781 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.781 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:37.781 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "telegraf", 2024-04-21T23:28:37.782 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:37.782 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:37.782 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:37.783 INFO:teuthology.orchestra.run.smithi040.stdout: "address": { 2024-04-21T23:28:37.783 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "address", 2024-04-21T23:28:37.783 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.783 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.783 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.784 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-21T23:28:37.784 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.784 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.784 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.785 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.785 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.785 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.785 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.786 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.786 INFO:teuthology.orchestra.run.smithi040.stdout: "interval": { 2024-04-21T23:28:37.786 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "interval", 2024-04-21T23:28:37.787 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-21T23:28:37.787 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.787 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.787 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "15", 2024-04-21T23:28:37.787 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.788 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.788 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.788 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.788 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.788 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.788 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.788 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.789 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:37.789 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:37.789 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.789 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.789 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.790 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.790 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.790 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.790 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.790 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:37.791 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:37.791 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:37.791 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:37.791 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:37.791 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:37.792 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.792 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.792 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.792 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.792 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.792 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.792 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:37.793 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:37.793 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.793 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.793 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.794 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.794 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.794 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.794 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.795 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.795 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.795 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.795 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.795 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.796 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:37.796 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:37.796 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.796 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.796 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.797 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:37.797 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.797 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.797 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.798 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:37.798 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:37.798 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:37.798 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:37.798 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:37.798 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:37.798 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.799 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.799 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.799 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.799 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.799 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.799 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:37.799 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:37.799 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.800 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.800 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.800 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.800 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.800 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.800 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.800 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.800 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.801 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.801 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.801 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.801 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.801 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.801 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:37.801 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "telemetry", 2024-04-21T23:28:37.802 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:37.802 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:37.802 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:37.802 INFO:teuthology.orchestra.run.smithi040.stdout: "channel_basic": { 2024-04-21T23:28:37.802 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "channel_basic", 2024-04-21T23:28:37.802 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.803 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.803 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.803 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:37.803 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.803 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.803 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.803 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-21T23:28:37.804 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.804 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.804 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.804 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.804 INFO:teuthology.orchestra.run.smithi040.stdout: "channel_crash": { 2024-04-21T23:28:37.804 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "channel_crash", 2024-04-21T23:28:37.804 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.805 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.805 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.805 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:37.805 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.805 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.805 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.805 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-21T23:28:37.806 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.806 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.806 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.806 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.806 INFO:teuthology.orchestra.run.smithi040.stdout: "channel_device": { 2024-04-21T23:28:37.806 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "channel_device", 2024-04-21T23:28:37.807 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.807 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.807 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.807 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:37.807 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.808 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.808 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.808 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-21T23:28:37.808 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.808 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.808 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.809 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.809 INFO:teuthology.orchestra.run.smithi040.stdout: "channel_ident": { 2024-04-21T23:28:37.809 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "channel_ident", 2024-04-21T23:28:37.809 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.809 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.810 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.810 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.810 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.810 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.810 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.810 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-21T23:28:37.811 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.811 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.811 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.811 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.812 INFO:teuthology.orchestra.run.smithi040.stdout: "channel_perf": { 2024-04-21T23:28:37.812 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "channel_perf", 2024-04-21T23:28:37.812 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.812 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.812 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.813 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.813 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.813 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.813 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.813 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-21T23:28:37.814 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.814 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.814 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.814 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.814 INFO:teuthology.orchestra.run.smithi040.stdout: "contact": { 2024-04-21T23:28:37.814 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "contact", 2024-04-21T23:28:37.814 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.815 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.815 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.815 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.815 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.816 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.816 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.816 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.816 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.817 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.817 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.817 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.818 INFO:teuthology.orchestra.run.smithi040.stdout: "description": { 2024-04-21T23:28:37.818 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "description", 2024-04-21T23:28:37.818 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.818 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.819 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.819 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.819 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.819 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.820 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.820 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.820 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.820 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.821 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.821 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.821 INFO:teuthology.orchestra.run.smithi040.stdout: "device_url": { 2024-04-21T23:28:37.821 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "device_url", 2024-04-21T23:28:37.821 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.822 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.822 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.822 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-21T23:28:37.822 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.822 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.823 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.823 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.823 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.823 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.824 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.824 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.824 INFO:teuthology.orchestra.run.smithi040.stdout: "enabled": { 2024-04-21T23:28:37.825 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "enabled", 2024-04-21T23:28:37.825 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.825 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.825 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.825 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.826 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.826 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.826 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.826 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.826 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.827 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.827 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.827 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.827 INFO:teuthology.orchestra.run.smithi040.stdout: "interval": { 2024-04-21T23:28:37.827 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "interval", 2024-04-21T23:28:37.828 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:37.828 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.828 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.828 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "24", 2024-04-21T23:28:37.829 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "8", 2024-04-21T23:28:37.829 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.829 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.829 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.829 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.829 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.830 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.830 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.830 INFO:teuthology.orchestra.run.smithi040.stdout: "last_opt_revision": { 2024-04-21T23:28:37.830 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "last_opt_revision", 2024-04-21T23:28:37.831 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:37.831 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.831 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.831 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "1", 2024-04-21T23:28:37.832 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.832 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.832 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.832 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.833 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.833 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.833 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.833 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.834 INFO:teuthology.orchestra.run.smithi040.stdout: "leaderboard": { 2024-04-21T23:28:37.834 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "leaderboard", 2024-04-21T23:28:37.834 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.834 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.835 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.835 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.835 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.835 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.835 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.836 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.836 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.836 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.836 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.836 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.836 INFO:teuthology.orchestra.run.smithi040.stdout: "leaderboard_description": { 2024-04-21T23:28:37.837 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "leaderboard_description", 2024-04-21T23:28:37.837 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.837 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.837 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.837 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.837 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.838 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.838 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.838 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.838 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.839 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.839 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.839 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.839 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:37.839 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:37.840 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.840 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.840 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.840 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.840 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.840 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.841 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.841 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:37.841 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:37.841 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:37.841 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:37.841 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:37.841 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:37.842 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.842 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.842 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.842 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.842 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.842 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.842 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:37.843 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:37.843 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.843 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.843 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.843 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.844 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.844 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.844 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.844 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.844 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.845 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.845 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.845 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.845 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:37.845 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:37.845 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.845 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.846 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.846 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:37.846 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.846 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.847 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.847 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:37.847 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:37.848 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:37.848 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:37.848 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:37.848 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:37.848 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.849 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.849 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.849 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.849 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.849 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.850 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:37.850 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:37.850 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.850 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.850 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.851 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.851 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.851 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.851 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.851 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.851 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.852 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.852 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.852 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.852 INFO:teuthology.orchestra.run.smithi040.stdout: "organization": { 2024-04-21T23:28:37.852 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "organization", 2024-04-21T23:28:37.853 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.853 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.853 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.853 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.853 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.853 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.854 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.854 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.854 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.854 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.854 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.855 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.855 INFO:teuthology.orchestra.run.smithi040.stdout: "proxy": { 2024-04-21T23:28:37.855 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "proxy", 2024-04-21T23:28:37.855 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.856 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.856 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.856 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.856 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.856 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.856 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.856 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.857 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.857 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.857 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.857 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.857 INFO:teuthology.orchestra.run.smithi040.stdout: "url": { 2024-04-21T23:28:37.857 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "url", 2024-04-21T23:28:37.858 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.858 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.858 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.858 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-21T23:28:37.858 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.859 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.859 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.859 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.859 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.860 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.860 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.860 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.860 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.860 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.861 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:37.861 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "test_orchestrator", 2024-04-21T23:28:37.861 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:37.861 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:37.861 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:37.861 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:37.861 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:37.862 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.862 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.862 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.862 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.862 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.862 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.863 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.863 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:37.863 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:37.863 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:37.863 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:37.863 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:37.863 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:37.863 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.864 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.864 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.864 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.864 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.864 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.864 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:37.865 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:37.865 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.865 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.865 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.866 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.866 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.866 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.866 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.866 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.866 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.866 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.867 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.867 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.867 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:37.867 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:37.867 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.867 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.867 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.867 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:37.869 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.869 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.869 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.869 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:37.869 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:37.870 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:37.870 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:37.870 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:37.870 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:37.870 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.870 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.871 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.871 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.871 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.871 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.871 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:37.871 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:37.872 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.872 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.872 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.872 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.872 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.872 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.872 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.872 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.873 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.873 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.873 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.873 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.873 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.873 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.873 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:37.873 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "volumes", 2024-04-21T23:28:37.874 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:37.874 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:37.874 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:37.874 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:37.874 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:37.874 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.874 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.875 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.875 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.875 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.875 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.875 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.875 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:37.875 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:37.876 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:37.876 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:37.876 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:37.876 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:37.876 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.876 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.876 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.877 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.877 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.877 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.877 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:37.877 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:37.877 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.877 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.878 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.878 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.878 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.878 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.878 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.878 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.878 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.879 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.879 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.879 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.879 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:37.879 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:37.879 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.880 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.880 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.880 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:37.880 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.880 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.880 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.880 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:37.880 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:37.881 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:37.881 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:37.881 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:37.881 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:37.881 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.881 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.881 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.881 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.881 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.882 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.882 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:37.882 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:37.882 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.882 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.882 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.882 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.882 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.882 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.883 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.883 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.883 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.883 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.883 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.883 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.884 INFO:teuthology.orchestra.run.smithi040.stdout: "max_concurrent_clones": { 2024-04-21T23:28:37.884 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "max_concurrent_clones", 2024-04-21T23:28:37.884 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:37.884 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.884 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.884 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "4", 2024-04-21T23:28:37.884 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.885 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.885 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.885 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-21T23:28:37.885 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.885 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.885 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.885 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.886 INFO:teuthology.orchestra.run.smithi040.stdout: "periodic_async_work": { 2024-04-21T23:28:37.886 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "periodic_async_work", 2024-04-21T23:28:37.886 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.886 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.886 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.886 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.887 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.887 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.887 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.887 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Periodically check for async work", 2024-04-21T23:28:37.887 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.887 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.888 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.888 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.888 INFO:teuthology.orchestra.run.smithi040.stdout: "snapshot_clone_delay": { 2024-04-21T23:28:37.888 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "snapshot_clone_delay", 2024-04-21T23:28:37.888 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:37.888 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.888 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.889 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "0", 2024-04-21T23:28:37.889 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.889 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.889 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.889 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-21T23:28:37.889 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.890 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.890 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.890 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.890 INFO:teuthology.orchestra.run.smithi040.stdout: "snapshot_clone_no_wait": { 2024-04-21T23:28:37.890 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "snapshot_clone_no_wait", 2024-04-21T23:28:37.890 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.891 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.891 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.891 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:37.891 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.891 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.891 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.891 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-21T23:28:37.891 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.892 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.892 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.892 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.892 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.892 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.892 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:37.893 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "zabbix", 2024-04-21T23:28:37.893 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:37.893 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:37.893 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:37.893 INFO:teuthology.orchestra.run.smithi040.stdout: "discovery_interval": { 2024-04-21T23:28:37.893 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "discovery_interval", 2024-04-21T23:28:37.893 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "uint", 2024-04-21T23:28:37.894 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.894 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.894 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "100", 2024-04-21T23:28:37.894 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.894 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.894 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.894 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.895 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.895 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.895 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.895 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.895 INFO:teuthology.orchestra.run.smithi040.stdout: "identifier": { 2024-04-21T23:28:37.895 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "identifier", 2024-04-21T23:28:37.895 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.895 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.896 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.896 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.896 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.896 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.896 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.896 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.896 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.896 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.897 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.897 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.897 INFO:teuthology.orchestra.run.smithi040.stdout: "interval": { 2024-04-21T23:28:37.897 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "interval", 2024-04-21T23:28:37.897 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-21T23:28:37.897 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.897 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.897 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "60", 2024-04-21T23:28:37.898 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.898 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.898 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.898 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.898 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.898 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.898 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.899 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.899 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:37.899 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:37.899 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.899 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.899 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.899 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.900 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.900 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.900 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.900 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:37.900 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:37.900 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:37.900 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:37.901 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:37.901 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:37.901 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.901 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.901 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.901 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.901 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.901 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.902 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:37.902 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:37.902 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.902 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.902 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.902 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.902 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.903 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.903 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.903 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.903 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.903 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.903 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.903 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.903 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:37.904 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:37.904 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.904 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.904 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.904 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:37.904 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.904 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.905 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.905 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:37.905 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:37.905 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:37.905 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:37.905 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:37.906 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:37.906 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.906 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.906 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.906 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.906 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.906 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.906 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:37.907 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:37.907 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.907 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.907 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.907 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.907 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.907 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.908 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.908 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.908 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.908 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.908 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.908 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.909 INFO:teuthology.orchestra.run.smithi040.stdout: "zabbix_host": { 2024-04-21T23:28:37.909 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "zabbix_host", 2024-04-21T23:28:37.909 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.909 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.909 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.909 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.909 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.909 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.910 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.910 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.910 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.910 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.910 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.910 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.910 INFO:teuthology.orchestra.run.smithi040.stdout: "zabbix_port": { 2024-04-21T23:28:37.911 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "zabbix_port", 2024-04-21T23:28:37.911 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:37.911 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.911 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.911 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10051", 2024-04-21T23:28:37.911 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.911 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.912 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.912 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.912 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.912 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.912 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.912 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.912 INFO:teuthology.orchestra.run.smithi040.stdout: "zabbix_sender": { 2024-04-21T23:28:37.913 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "zabbix_sender", 2024-04-21T23:28:37.913 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.913 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.913 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.913 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-21T23:28:37.913 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.913 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.913 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.914 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.914 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.914 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.914 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.914 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.914 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.914 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.914 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-21T23:28:37.915 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.915 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.915 INFO:teuthology.orchestra.run.smithi040.stdout: "modules": [ 2024-04-21T23:28:37.915 INFO:teuthology.orchestra.run.smithi040.stdout: "cephadm", 2024-04-21T23:28:37.915 INFO:teuthology.orchestra.run.smithi040.stdout: "dashboard", 2024-04-21T23:28:37.915 INFO:teuthology.orchestra.run.smithi040.stdout: "iostat", 2024-04-21T23:28:37.915 INFO:teuthology.orchestra.run.smithi040.stdout: "nfs", 2024-04-21T23:28:37.916 INFO:teuthology.orchestra.run.smithi040.stdout: "restful", 2024-04-21T23:28:37.916 INFO:teuthology.orchestra.run.smithi040.stdout: "snap_schedule" 2024-04-21T23:28:37.916 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.916 INFO:teuthology.orchestra.run.smithi040.stdout: "available_modules": [ 2024-04-21T23:28:37.916 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:37.916 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "alerts", 2024-04-21T23:28:37.916 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:37.916 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:37.917 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:37.917 INFO:teuthology.orchestra.run.smithi040.stdout: "interval": { 2024-04-21T23:28:37.917 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "interval", 2024-04-21T23:28:37.917 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-21T23:28:37.917 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.917 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.918 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "60", 2024-04-21T23:28:37.918 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.918 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.918 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.918 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "How frequently to reexamine health status", 2024-04-21T23:28:37.918 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.918 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.918 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.919 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.919 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:37.919 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:37.919 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.919 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.919 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.919 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.919 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.920 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.920 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.920 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:37.920 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:37.920 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:37.920 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:37.920 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:37.921 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:37.921 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.921 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.921 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.921 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.921 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.921 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.921 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:37.922 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:37.922 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.922 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.922 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.922 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.922 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.922 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.922 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.923 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.923 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.923 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.923 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.923 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.923 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:37.923 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:37.924 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.924 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.924 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.924 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:37.924 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.924 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.924 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.925 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:37.925 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:37.925 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:37.925 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:37.925 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:37.925 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:37.925 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.926 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.926 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.926 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.926 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.926 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.926 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:37.927 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:37.927 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.927 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.927 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.927 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.927 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.927 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.928 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.928 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.928 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.928 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.928 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.928 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.928 INFO:teuthology.orchestra.run.smithi040.stdout: "smtp_destination": { 2024-04-21T23:28:37.929 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "smtp_destination", 2024-04-21T23:28:37.929 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.929 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.929 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.929 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.929 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.929 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.930 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.930 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Email address to send alerts to", 2024-04-21T23:28:37.930 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.930 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.930 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.930 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.930 INFO:teuthology.orchestra.run.smithi040.stdout: "smtp_from_name": { 2024-04-21T23:28:37.931 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "smtp_from_name", 2024-04-21T23:28:37.931 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.931 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.931 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.931 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "Ceph", 2024-04-21T23:28:37.931 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.932 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.932 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.932 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Email From: name", 2024-04-21T23:28:37.932 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.932 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.932 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.932 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.933 INFO:teuthology.orchestra.run.smithi040.stdout: "smtp_host": { 2024-04-21T23:28:37.933 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "smtp_host", 2024-04-21T23:28:37.933 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.933 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.933 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.933 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.933 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.933 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.934 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.934 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "SMTP server", 2024-04-21T23:28:37.934 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.934 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.934 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.934 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.935 INFO:teuthology.orchestra.run.smithi040.stdout: "smtp_password": { 2024-04-21T23:28:37.935 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "smtp_password", 2024-04-21T23:28:37.935 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.935 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.935 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.935 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.935 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.935 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.936 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.936 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Password to authenticate with", 2024-04-21T23:28:37.936 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.936 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.936 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.936 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.936 INFO:teuthology.orchestra.run.smithi040.stdout: "smtp_port": { 2024-04-21T23:28:37.937 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "smtp_port", 2024-04-21T23:28:37.937 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:37.937 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.937 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.937 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "465", 2024-04-21T23:28:37.937 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.937 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.937 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.938 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "SMTP port", 2024-04-21T23:28:37.938 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.938 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.938 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.938 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.938 INFO:teuthology.orchestra.run.smithi040.stdout: "smtp_sender": { 2024-04-21T23:28:37.938 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "smtp_sender", 2024-04-21T23:28:37.939 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.939 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.939 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.939 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.939 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.939 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.939 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.940 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "SMTP envelope sender", 2024-04-21T23:28:37.940 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.940 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.940 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.940 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.940 INFO:teuthology.orchestra.run.smithi040.stdout: "smtp_ssl": { 2024-04-21T23:28:37.940 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "smtp_ssl", 2024-04-21T23:28:37.941 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.941 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.941 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.941 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:37.941 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.941 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.941 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.942 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-21T23:28:37.942 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.942 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.942 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.942 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.942 INFO:teuthology.orchestra.run.smithi040.stdout: "smtp_user": { 2024-04-21T23:28:37.942 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "smtp_user", 2024-04-21T23:28:37.942 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.943 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.943 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.943 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.943 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.943 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.943 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.943 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "User to authenticate as", 2024-04-21T23:28:37.944 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.944 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.944 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.944 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.944 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.944 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.944 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:37.944 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "balancer", 2024-04-21T23:28:37.945 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:37.945 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:37.945 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:37.945 INFO:teuthology.orchestra.run.smithi040.stdout: "active": { 2024-04-21T23:28:37.945 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "active", 2024-04-21T23:28:37.945 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.945 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.946 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.946 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:37.946 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.946 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.946 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.946 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "automatically balance PGs across cluster", 2024-04-21T23:28:37.946 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.947 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.947 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.947 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.947 INFO:teuthology.orchestra.run.smithi040.stdout: "begin_time": { 2024-04-21T23:28:37.947 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "begin_time", 2024-04-21T23:28:37.947 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.947 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.947 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.948 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "0000", 2024-04-21T23:28:37.948 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.948 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.948 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.948 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "beginning time of day to automatically balance", 2024-04-21T23:28:37.948 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-21T23:28:37.948 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.949 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.949 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.949 INFO:teuthology.orchestra.run.smithi040.stdout: "begin_weekday": { 2024-04-21T23:28:37.949 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "begin_weekday", 2024-04-21T23:28:37.949 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "uint", 2024-04-21T23:28:37.949 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.949 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.949 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "0", 2024-04-21T23:28:37.950 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "0", 2024-04-21T23:28:37.950 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "6", 2024-04-21T23:28:37.950 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.950 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-21T23:28:37.950 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-21T23:28:37.950 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.951 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.951 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.951 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_compat_max_iterations": { 2024-04-21T23:28:37.951 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "crush_compat_max_iterations", 2024-04-21T23:28:37.951 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "uint", 2024-04-21T23:28:37.951 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.951 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.951 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "25", 2024-04-21T23:28:37.952 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "1", 2024-04-21T23:28:37.952 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "250", 2024-04-21T23:28:37.952 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.952 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-21T23:28:37.952 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.952 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.952 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.953 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.953 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_compat_metrics": { 2024-04-21T23:28:37.953 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "crush_compat_metrics", 2024-04-21T23:28:37.953 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.953 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.954 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.954 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "pgs,objects,bytes", 2024-04-21T23:28:37.954 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.954 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.954 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.954 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-21T23:28:37.954 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-21T23:28:37.955 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.955 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.955 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.955 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_compat_step": { 2024-04-21T23:28:37.955 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "crush_compat_step", 2024-04-21T23:28:37.955 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "float", 2024-04-21T23:28:37.955 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.955 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.956 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "0.5", 2024-04-21T23:28:37.956 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "0.001", 2024-04-21T23:28:37.956 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "0.999", 2024-04-21T23:28:37.956 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.956 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "aggressiveness of optimization", 2024-04-21T23:28:37.956 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-21T23:28:37.956 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.956 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.956 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.957 INFO:teuthology.orchestra.run.smithi040.stdout: "end_time": { 2024-04-21T23:28:37.957 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "end_time", 2024-04-21T23:28:37.957 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.957 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.957 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.957 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "2359", 2024-04-21T23:28:37.957 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.957 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.958 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.958 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "ending time of day to automatically balance", 2024-04-21T23:28:37.958 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-21T23:28:37.958 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.958 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.958 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.958 INFO:teuthology.orchestra.run.smithi040.stdout: "end_weekday": { 2024-04-21T23:28:37.959 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "end_weekday", 2024-04-21T23:28:37.959 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "uint", 2024-04-21T23:28:37.959 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.959 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.959 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "0", 2024-04-21T23:28:37.959 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "0", 2024-04-21T23:28:37.959 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "6", 2024-04-21T23:28:37.960 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.960 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-21T23:28:37.960 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-21T23:28:37.960 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.960 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.960 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.960 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:37.961 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:37.961 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.961 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.961 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.961 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.961 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.961 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.961 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.962 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:37.962 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:37.962 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:37.962 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:37.962 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:37.962 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:37.962 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.963 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.963 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.963 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.963 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.963 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.963 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:37.963 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:37.964 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.964 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.964 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.964 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.964 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.964 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.965 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.965 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.965 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.965 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.965 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.965 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.966 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:37.966 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:37.966 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.966 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.966 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.966 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:37.967 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.967 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.967 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.967 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:37.967 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:37.968 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:37.968 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:37.968 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:37.968 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:37.968 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.968 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.968 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.969 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.969 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.969 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.969 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:37.969 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:37.969 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.969 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.970 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.970 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.970 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.970 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.970 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.970 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:37.971 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.971 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.971 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.971 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.971 INFO:teuthology.orchestra.run.smithi040.stdout: "min_score": { 2024-04-21T23:28:37.971 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "min_score", 2024-04-21T23:28:37.972 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "float", 2024-04-21T23:28:37.972 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.972 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.972 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "0", 2024-04-21T23:28:37.972 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.972 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.972 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.973 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-21T23:28:37.973 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.973 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.973 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.973 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.973 INFO:teuthology.orchestra.run.smithi040.stdout: "mode": { 2024-04-21T23:28:37.973 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "mode", 2024-04-21T23:28:37.973 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.974 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.974 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.974 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "upmap", 2024-04-21T23:28:37.974 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.974 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.974 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:37.974 INFO:teuthology.orchestra.run.smithi040.stdout: "crush-compat", 2024-04-21T23:28:37.975 INFO:teuthology.orchestra.run.smithi040.stdout: "none", 2024-04-21T23:28:37.975 INFO:teuthology.orchestra.run.smithi040.stdout: "read", 2024-04-21T23:28:37.975 INFO:teuthology.orchestra.run.smithi040.stdout: "upmap", 2024-04-21T23:28:37.975 INFO:teuthology.orchestra.run.smithi040.stdout: "upmap-read" 2024-04-21T23:28:37.975 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:37.975 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Balancer mode", 2024-04-21T23:28:37.976 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.976 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.976 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.976 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.976 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_ids": { 2024-04-21T23:28:37.976 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "pool_ids", 2024-04-21T23:28:37.976 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:37.977 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.977 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.977 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:37.977 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.977 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.978 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.978 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-21T23:28:37.978 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.978 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.978 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.978 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.979 INFO:teuthology.orchestra.run.smithi040.stdout: "sleep_interval": { 2024-04-21T23:28:37.979 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "sleep_interval", 2024-04-21T23:28:37.979 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-21T23:28:37.979 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.979 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.979 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "60", 2024-04-21T23:28:37.979 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.979 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.980 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.980 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-21T23:28:37.980 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.980 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.980 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.980 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.980 INFO:teuthology.orchestra.run.smithi040.stdout: "upmap_max_deviation": { 2024-04-21T23:28:37.981 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "upmap_max_deviation", 2024-04-21T23:28:37.981 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:37.981 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.981 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.981 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "5", 2024-04-21T23:28:37.981 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "1", 2024-04-21T23:28:37.981 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.982 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.982 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-21T23:28:37.982 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-21T23:28:37.982 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.983 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.983 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.983 INFO:teuthology.orchestra.run.smithi040.stdout: "upmap_max_optimizations": { 2024-04-21T23:28:37.983 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "upmap_max_optimizations", 2024-04-21T23:28:37.983 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "uint", 2024-04-21T23:28:37.983 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.983 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:37.984 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10", 2024-04-21T23:28:37.984 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.984 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.984 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.984 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-21T23:28:37.984 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.985 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.985 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.985 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.985 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:37.985 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.985 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:37.985 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "cephadm", 2024-04-21T23:28:37.986 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:37.986 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:37.986 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:37.986 INFO:teuthology.orchestra.run.smithi040.stdout: "agent_down_multiplier": { 2024-04-21T23:28:37.986 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "agent_down_multiplier", 2024-04-21T23:28:37.986 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "float", 2024-04-21T23:28:37.987 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.987 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.987 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "3.0", 2024-04-21T23:28:37.987 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.987 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.987 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.987 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-21T23:28:37.988 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.988 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.988 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.988 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.989 INFO:teuthology.orchestra.run.smithi040.stdout: "agent_refresh_rate": { 2024-04-21T23:28:37.989 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "agent_refresh_rate", 2024-04-21T23:28:37.989 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-21T23:28:37.989 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.989 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.989 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "20", 2024-04-21T23:28:37.990 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.990 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.990 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.990 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-21T23:28:37.990 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.991 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.991 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.991 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.991 INFO:teuthology.orchestra.run.smithi040.stdout: "agent_starting_port": { 2024-04-21T23:28:37.991 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "agent_starting_port", 2024-04-21T23:28:37.992 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:37.992 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.992 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.992 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "4721", 2024-04-21T23:28:37.992 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.993 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.993 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.993 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-21T23:28:37.993 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.993 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.993 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.994 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.994 INFO:teuthology.orchestra.run.smithi040.stdout: "allow_ptrace": { 2024-04-21T23:28:37.994 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "allow_ptrace", 2024-04-21T23:28:37.994 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:37.994 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.994 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.995 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:37.995 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.995 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.995 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.995 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-21T23:28:37.996 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-21T23:28:37.996 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.996 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.996 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.996 INFO:teuthology.orchestra.run.smithi040.stdout: "autotune_interval": { 2024-04-21T23:28:37.997 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "autotune_interval", 2024-04-21T23:28:37.997 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-21T23:28:37.997 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.997 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:37.997 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "600", 2024-04-21T23:28:37.997 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:37.998 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:37.998 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:37.998 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-21T23:28:37.998 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:37.998 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:37.998 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:37.998 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:37.999 INFO:teuthology.orchestra.run.smithi040.stdout: "autotune_memory_target_ratio": { 2024-04-21T23:28:37.999 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "autotune_memory_target_ratio", 2024-04-21T23:28:37.999 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "float", 2024-04-21T23:28:37.999 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:37.999 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.000 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "0.7", 2024-04-21T23:28:38.000 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.000 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.000 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.000 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-21T23:28:38.001 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.001 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.001 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.001 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.001 INFO:teuthology.orchestra.run.smithi040.stdout: "cephadm_log_destination": { 2024-04-21T23:28:38.002 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "cephadm_log_destination", 2024-04-21T23:28:38.002 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.002 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.002 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.002 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.002 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.002 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.003 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:38.003 INFO:teuthology.orchestra.run.smithi040.stdout: "file", 2024-04-21T23:28:38.003 INFO:teuthology.orchestra.run.smithi040.stdout: "file,syslog", 2024-04-21T23:28:38.003 INFO:teuthology.orchestra.run.smithi040.stdout: "syslog" 2024-04-21T23:28:38.003 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:38.003 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-21T23:28:38.003 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.004 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.004 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.004 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.004 INFO:teuthology.orchestra.run.smithi040.stdout: "cgroups_split": { 2024-04-21T23:28:38.004 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "cgroups_split", 2024-04-21T23:28:38.004 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.005 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.005 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.005 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:38.005 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.005 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.006 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.006 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-21T23:28:38.006 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.006 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.006 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.007 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.007 INFO:teuthology.orchestra.run.smithi040.stdout: "config_checks_enabled": { 2024-04-21T23:28:38.007 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "config_checks_enabled", 2024-04-21T23:28:38.007 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.007 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.008 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.008 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.008 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.008 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.008 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.008 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-21T23:28:38.009 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.009 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.009 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.009 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.009 INFO:teuthology.orchestra.run.smithi040.stdout: "config_dashboard": { 2024-04-21T23:28:38.009 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "config_dashboard", 2024-04-21T23:28:38.010 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.010 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.010 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.010 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:38.010 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.010 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.011 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.011 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-21T23:28:38.011 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.011 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.011 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.012 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.012 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_alertmanager": { 2024-04-21T23:28:38.012 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_alertmanager", 2024-04-21T23:28:38.012 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.012 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.012 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.013 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-21T23:28:38.013 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.013 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.013 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.013 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Prometheus container image", 2024-04-21T23:28:38.013 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.014 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.014 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.014 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.014 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_base": { 2024-04-21T23:28:38.014 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_base", 2024-04-21T23:28:38.014 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.015 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.015 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.015 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-21T23:28:38.015 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.015 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.016 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.016 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Container image name, without the tag", 2024-04-21T23:28:38.016 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.016 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.016 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.017 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.017 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_elasticsearch": { 2024-04-21T23:28:38.017 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_elasticsearch", 2024-04-21T23:28:38.017 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.017 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.017 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.018 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-21T23:28:38.018 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.018 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.018 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.018 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "elasticsearch container image", 2024-04-21T23:28:38.019 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.019 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.019 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.019 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.020 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_grafana": { 2024-04-21T23:28:38.020 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_grafana", 2024-04-21T23:28:38.020 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.020 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.020 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.020 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-21T23:28:38.021 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.021 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.021 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.021 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Prometheus container image", 2024-04-21T23:28:38.021 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.022 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.022 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.022 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.022 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_haproxy": { 2024-04-21T23:28:38.022 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_haproxy", 2024-04-21T23:28:38.022 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.023 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.023 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.023 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-21T23:28:38.023 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.023 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.023 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.023 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "HAproxy container image", 2024-04-21T23:28:38.024 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.024 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.024 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.024 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.024 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_jaeger_agent": { 2024-04-21T23:28:38.024 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_jaeger_agent", 2024-04-21T23:28:38.025 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.025 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.025 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.026 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-21T23:28:38.026 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.026 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.026 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.026 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Jaeger agent container image", 2024-04-21T23:28:38.027 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.027 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.027 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.027 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.028 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_jaeger_collector": { 2024-04-21T23:28:38.028 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_jaeger_collector", 2024-04-21T23:28:38.028 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.028 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.028 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.028 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-21T23:28:38.029 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.029 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.029 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.029 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Jaeger collector container image", 2024-04-21T23:28:38.029 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.030 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.030 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.030 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.030 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_jaeger_query": { 2024-04-21T23:28:38.030 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_jaeger_query", 2024-04-21T23:28:38.031 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.031 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.031 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.031 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-21T23:28:38.032 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.032 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.032 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.032 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Jaeger query container image", 2024-04-21T23:28:38.032 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.033 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.033 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.033 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.033 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_keepalived": { 2024-04-21T23:28:38.033 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_keepalived", 2024-04-21T23:28:38.034 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.034 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.034 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.034 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-21T23:28:38.035 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.035 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.035 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.035 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Keepalived container image", 2024-04-21T23:28:38.036 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.036 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.036 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.036 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.036 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_loki": { 2024-04-21T23:28:38.037 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_loki", 2024-04-21T23:28:38.037 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.037 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.037 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.037 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-21T23:28:38.038 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.038 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.038 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.038 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Loki container image", 2024-04-21T23:28:38.038 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.039 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.039 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.039 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.039 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_node_exporter": { 2024-04-21T23:28:38.039 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_node_exporter", 2024-04-21T23:28:38.040 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.040 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.040 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.040 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-21T23:28:38.040 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.041 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.041 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.041 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Prometheus container image", 2024-04-21T23:28:38.041 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.041 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.042 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.042 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.042 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_nvmeof": { 2024-04-21T23:28:38.042 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_nvmeof", 2024-04-21T23:28:38.043 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.043 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.043 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.043 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-21T23:28:38.044 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.044 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.044 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.044 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Nvme-of container image", 2024-04-21T23:28:38.044 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.045 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.045 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.045 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.045 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_prometheus": { 2024-04-21T23:28:38.046 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_prometheus", 2024-04-21T23:28:38.046 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.046 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.046 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.046 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-21T23:28:38.047 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.047 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.047 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.047 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Prometheus container image", 2024-04-21T23:28:38.047 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.048 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.048 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.048 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.048 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_promtail": { 2024-04-21T23:28:38.048 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_promtail", 2024-04-21T23:28:38.048 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.049 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.049 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.049 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-21T23:28:38.049 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.049 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.050 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.050 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Promtail container image", 2024-04-21T23:28:38.050 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.050 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.050 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.051 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.051 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_samba": { 2024-04-21T23:28:38.051 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_samba", 2024-04-21T23:28:38.051 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.051 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.052 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.052 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-21T23:28:38.052 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.052 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.052 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.052 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Samba/SMB container image", 2024-04-21T23:28:38.053 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.053 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.053 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.053 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.053 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_snmp_gateway": { 2024-04-21T23:28:38.054 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_snmp_gateway", 2024-04-21T23:28:38.054 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.054 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.054 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.055 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-21T23:28:38.055 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.055 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.055 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.055 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "SNMP Gateway container image", 2024-04-21T23:28:38.056 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.056 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.056 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.056 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.057 INFO:teuthology.orchestra.run.smithi040.stdout: "container_init": { 2024-04-21T23:28:38.057 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_init", 2024-04-21T23:28:38.057 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.057 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.057 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.057 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:38.058 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.058 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.058 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.058 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Run podman/docker with `--init`", 2024-04-21T23:28:38.059 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.059 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.059 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.059 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.059 INFO:teuthology.orchestra.run.smithi040.stdout: "daemon_cache_timeout": { 2024-04-21T23:28:38.060 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "daemon_cache_timeout", 2024-04-21T23:28:38.060 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-21T23:28:38.060 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.060 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.060 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "600", 2024-04-21T23:28:38.061 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.061 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.061 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.061 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-21T23:28:38.061 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.061 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.062 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.062 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.062 INFO:teuthology.orchestra.run.smithi040.stdout: "default_cephadm_command_timeout": { 2024-04-21T23:28:38.062 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "default_cephadm_command_timeout", 2024-04-21T23:28:38.063 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:38.063 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.063 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.063 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "900", 2024-04-21T23:28:38.063 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.064 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.064 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.064 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-21T23:28:38.064 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.064 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.065 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.065 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.065 INFO:teuthology.orchestra.run.smithi040.stdout: "default_registry": { 2024-04-21T23:28:38.065 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "default_registry", 2024-04-21T23:28:38.065 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.065 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.066 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.066 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "docker.io", 2024-04-21T23:28:38.066 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.066 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.067 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.067 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-21T23:28:38.067 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.067 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.068 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.068 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.068 INFO:teuthology.orchestra.run.smithi040.stdout: "device_cache_timeout": { 2024-04-21T23:28:38.069 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "device_cache_timeout", 2024-04-21T23:28:38.069 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-21T23:28:38.069 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.069 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.069 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "1800", 2024-04-21T23:28:38.069 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.070 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.070 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.070 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "seconds to cache device inventory", 2024-04-21T23:28:38.070 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.071 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.071 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.071 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.071 INFO:teuthology.orchestra.run.smithi040.stdout: "device_enhanced_scan": { 2024-04-21T23:28:38.071 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "device_enhanced_scan", 2024-04-21T23:28:38.072 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.072 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.072 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.072 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.073 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.073 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.073 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.073 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-21T23:28:38.073 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.074 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.074 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.074 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.074 INFO:teuthology.orchestra.run.smithi040.stdout: "facts_cache_timeout": { 2024-04-21T23:28:38.074 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "facts_cache_timeout", 2024-04-21T23:28:38.075 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-21T23:28:38.075 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.075 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.075 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "60", 2024-04-21T23:28:38.075 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.077 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.078 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.078 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "seconds to cache host facts data", 2024-04-21T23:28:38.078 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.078 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.078 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.079 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.079 INFO:teuthology.orchestra.run.smithi040.stdout: "grafana_dashboards_path": { 2024-04-21T23:28:38.079 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "grafana_dashboards_path", 2024-04-21T23:28:38.079 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.079 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.079 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.080 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-21T23:28:38.080 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.080 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.080 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.080 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-21T23:28:38.081 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.081 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.081 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.081 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.081 INFO:teuthology.orchestra.run.smithi040.stdout: "host_check_interval": { 2024-04-21T23:28:38.081 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "host_check_interval", 2024-04-21T23:28:38.082 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-21T23:28:38.082 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.082 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.082 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "600", 2024-04-21T23:28:38.082 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.083 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.083 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.083 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "how frequently to perform a host check", 2024-04-21T23:28:38.083 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.084 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.084 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.084 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.084 INFO:teuthology.orchestra.run.smithi040.stdout: "hw_monitoring": { 2024-04-21T23:28:38.084 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "hw_monitoring", 2024-04-21T23:28:38.084 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.085 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.085 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.085 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.085 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.085 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.085 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.086 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-21T23:28:38.086 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.086 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.086 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.087 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.087 INFO:teuthology.orchestra.run.smithi040.stdout: "inventory_list_all": { 2024-04-21T23:28:38.087 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "inventory_list_all", 2024-04-21T23:28:38.087 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.087 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.088 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.088 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.088 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.088 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.088 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.089 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-21T23:28:38.089 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.089 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.089 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.089 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.090 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:38.090 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:38.090 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.090 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.090 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.091 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.091 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.091 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.091 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:38.091 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:38.091 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:38.092 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:38.092 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:38.092 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:38.092 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:38.092 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:38.092 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.093 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.093 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.093 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.093 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.093 INFO:teuthology.orchestra.run.smithi040.stdout: "log_refresh_metadata": { 2024-04-21T23:28:38.094 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_refresh_metadata", 2024-04-21T23:28:38.094 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.094 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.094 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.094 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.095 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.095 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.095 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.095 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-21T23:28:38.095 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.096 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.096 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.096 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.096 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:38.096 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:38.097 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.097 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.097 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.097 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:38.097 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.097 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.097 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.098 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-21T23:28:38.098 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.098 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.098 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.098 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.099 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:38.099 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:38.099 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.099 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.099 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.100 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:38.100 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.100 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.100 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:38.100 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:38.101 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:38.101 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:38.101 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:38.101 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:38.101 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:38.102 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:38.102 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.102 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.102 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.102 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.103 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.103 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:38.103 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:38.103 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.104 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.104 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.104 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.104 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.104 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.105 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.105 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.105 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.105 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.105 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.106 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.106 INFO:teuthology.orchestra.run.smithi040.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-21T23:28:38.106 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-21T23:28:38.106 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.106 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.107 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.107 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.107 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.107 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.107 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.107 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-21T23:28:38.108 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.108 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.108 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.108 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.109 INFO:teuthology.orchestra.run.smithi040.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-21T23:28:38.109 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-21T23:28:38.109 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.109 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.109 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.109 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "*", 2024-04-21T23:28:38.110 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.110 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.110 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.110 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-21T23:28:38.110 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.111 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.111 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.111 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.111 INFO:teuthology.orchestra.run.smithi040.stdout: "max_count_per_host": { 2024-04-21T23:28:38.111 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "max_count_per_host", 2024-04-21T23:28:38.111 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:38.112 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.112 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.112 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10", 2024-04-21T23:28:38.112 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.113 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.113 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.113 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "max number of daemons per service per host", 2024-04-21T23:28:38.113 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.114 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.114 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.114 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.114 INFO:teuthology.orchestra.run.smithi040.stdout: "max_osd_draining_count": { 2024-04-21T23:28:38.115 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "max_osd_draining_count", 2024-04-21T23:28:38.115 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:38.115 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.115 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.115 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10", 2024-04-21T23:28:38.116 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.116 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.116 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.116 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-21T23:28:38.117 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.117 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.117 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.117 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.117 INFO:teuthology.orchestra.run.smithi040.stdout: "migration_current": { 2024-04-21T23:28:38.118 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "migration_current", 2024-04-21T23:28:38.118 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:38.118 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.118 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.119 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.119 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.119 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.119 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.119 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "internal - do not modify", 2024-04-21T23:28:38.120 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.120 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.120 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.120 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.121 INFO:teuthology.orchestra.run.smithi040.stdout: "mode": { 2024-04-21T23:28:38.121 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "mode", 2024-04-21T23:28:38.121 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.121 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.121 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.121 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "root", 2024-04-21T23:28:38.122 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.122 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.122 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:38.122 INFO:teuthology.orchestra.run.smithi040.stdout: "cephadm-package", 2024-04-21T23:28:38.122 INFO:teuthology.orchestra.run.smithi040.stdout: "root" 2024-04-21T23:28:38.123 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:38.123 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "mode for remote execution of cephadm", 2024-04-21T23:28:38.123 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.123 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.123 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.124 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.124 INFO:teuthology.orchestra.run.smithi040.stdout: "oob_default_addr": { 2024-04-21T23:28:38.124 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "oob_default_addr", 2024-04-21T23:28:38.124 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.124 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.125 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.125 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "169.254.1.1", 2024-04-21T23:28:38.125 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.125 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.126 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.126 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-21T23:28:38.126 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.126 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.126 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.127 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.127 INFO:teuthology.orchestra.run.smithi040.stdout: "prometheus_alerts_path": { 2024-04-21T23:28:38.127 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "prometheus_alerts_path", 2024-04-21T23:28:38.127 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.127 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.128 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.128 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-21T23:28:38.128 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.128 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.128 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.129 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-21T23:28:38.129 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.129 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.129 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.129 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.130 INFO:teuthology.orchestra.run.smithi040.stdout: "registry_insecure": { 2024-04-21T23:28:38.130 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "registry_insecure", 2024-04-21T23:28:38.130 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.130 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.130 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.131 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.131 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.131 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.131 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.132 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-21T23:28:38.132 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.132 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.132 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.132 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.133 INFO:teuthology.orchestra.run.smithi040.stdout: "registry_password": { 2024-04-21T23:28:38.133 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "registry_password", 2024-04-21T23:28:38.133 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.133 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.134 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.134 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.134 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.135 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.135 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.135 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-21T23:28:38.135 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.136 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.136 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.136 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.136 INFO:teuthology.orchestra.run.smithi040.stdout: "registry_url": { 2024-04-21T23:28:38.136 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "registry_url", 2024-04-21T23:28:38.137 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.137 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.137 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.137 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.138 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.138 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.138 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.138 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-21T23:28:38.139 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.139 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.139 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.139 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.139 INFO:teuthology.orchestra.run.smithi040.stdout: "registry_username": { 2024-04-21T23:28:38.140 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "registry_username", 2024-04-21T23:28:38.140 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.140 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.140 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.140 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.141 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.141 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.141 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.141 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-21T23:28:38.141 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.142 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.142 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.142 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.142 INFO:teuthology.orchestra.run.smithi040.stdout: "secure_monitoring_stack": { 2024-04-21T23:28:38.142 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "secure_monitoring_stack", 2024-04-21T23:28:38.143 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.143 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.143 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.143 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.143 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.144 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.144 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.144 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-21T23:28:38.144 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.145 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.145 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.145 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.145 INFO:teuthology.orchestra.run.smithi040.stdout: "service_discovery_port": { 2024-04-21T23:28:38.146 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "service_discovery_port", 2024-04-21T23:28:38.146 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:38.146 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.146 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.146 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "8765", 2024-04-21T23:28:38.147 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.147 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.147 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.147 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "cephadm service discovery port", 2024-04-21T23:28:38.148 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.148 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.148 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.148 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.148 INFO:teuthology.orchestra.run.smithi040.stdout: "ssh_config_file": { 2024-04-21T23:28:38.149 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ssh_config_file", 2024-04-21T23:28:38.149 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.149 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.149 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.150 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.150 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.150 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.150 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.150 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-21T23:28:38.151 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.151 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.151 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.151 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.151 INFO:teuthology.orchestra.run.smithi040.stdout: "use_agent": { 2024-04-21T23:28:38.152 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "use_agent", 2024-04-21T23:28:38.152 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.152 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.152 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.152 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.153 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.153 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.153 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.153 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-21T23:28:38.153 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.154 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.154 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.154 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.154 INFO:teuthology.orchestra.run.smithi040.stdout: "use_repo_digest": { 2024-04-21T23:28:38.154 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "use_repo_digest", 2024-04-21T23:28:38.155 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.155 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.155 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.155 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:38.155 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.156 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.156 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.156 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-21T23:28:38.156 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.157 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.157 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.157 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.157 INFO:teuthology.orchestra.run.smithi040.stdout: "warn_on_failed_host_check": { 2024-04-21T23:28:38.157 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "warn_on_failed_host_check", 2024-04-21T23:28:38.158 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.158 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.158 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.158 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:38.158 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.159 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.159 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.159 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "raise a health warning if the host check fails", 2024-04-21T23:28:38.159 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.159 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.159 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.160 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.160 INFO:teuthology.orchestra.run.smithi040.stdout: "warn_on_stray_daemons": { 2024-04-21T23:28:38.160 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "warn_on_stray_daemons", 2024-04-21T23:28:38.160 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.160 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.160 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.161 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:38.161 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.161 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.161 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.161 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-21T23:28:38.162 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.162 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.162 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.162 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.162 INFO:teuthology.orchestra.run.smithi040.stdout: "warn_on_stray_hosts": { 2024-04-21T23:28:38.163 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "warn_on_stray_hosts", 2024-04-21T23:28:38.163 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.163 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.163 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.163 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:38.163 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.164 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.164 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.164 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-21T23:28:38.164 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.164 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.165 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.165 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:38.165 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:38.165 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.165 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:38.165 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "crash", 2024-04-21T23:28:38.165 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:38.166 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:38.166 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:38.166 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:38.166 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:38.167 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.167 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.167 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.167 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.167 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.167 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.168 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:38.168 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:38.168 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:38.168 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:38.168 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:38.169 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:38.169 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:38.169 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:38.169 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.169 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.170 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.170 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.170 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.170 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:38.170 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:38.171 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.171 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.171 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.171 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.171 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.172 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.172 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.172 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.172 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.173 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.173 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.173 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.174 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:38.174 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:38.174 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.174 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.175 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.175 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:38.175 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.175 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.175 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:38.175 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:38.176 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:38.176 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:38.176 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:38.176 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:38.176 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:38.176 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:38.177 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.177 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.177 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.177 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.177 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.178 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:38.178 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:38.178 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.178 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.178 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.179 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.179 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.179 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.179 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.179 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.179 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.180 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.180 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.180 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.180 INFO:teuthology.orchestra.run.smithi040.stdout: "retain_interval": { 2024-04-21T23:28:38.180 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "retain_interval", 2024-04-21T23:28:38.180 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-21T23:28:38.181 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.181 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.181 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "31536000", 2024-04-21T23:28:38.181 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.181 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.182 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.182 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-21T23:28:38.182 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.182 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.182 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.183 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.183 INFO:teuthology.orchestra.run.smithi040.stdout: "warn_recent_interval": { 2024-04-21T23:28:38.183 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "warn_recent_interval", 2024-04-21T23:28:38.183 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-21T23:28:38.184 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.184 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.184 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "1209600", 2024-04-21T23:28:38.184 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.184 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.184 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.185 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-21T23:28:38.185 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.185 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.185 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.185 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:38.186 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:38.186 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.186 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:38.186 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "dashboard", 2024-04-21T23:28:38.187 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:38.187 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:38.187 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:38.187 INFO:teuthology.orchestra.run.smithi040.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-21T23:28:38.187 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-21T23:28:38.187 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:38.188 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.188 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.188 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10", 2024-04-21T23:28:38.188 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.189 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.189 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.189 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.189 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.189 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.189 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.190 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.190 INFO:teuthology.orchestra.run.smithi040.stdout: "ALERTMANAGER_API_HOST": { 2024-04-21T23:28:38.190 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-21T23:28:38.190 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.190 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.191 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.191 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.191 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.191 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.192 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.192 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.192 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.192 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.192 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.193 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.193 INFO:teuthology.orchestra.run.smithi040.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-21T23:28:38.193 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-21T23:28:38.193 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.193 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.193 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.193 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:38.194 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.194 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.194 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.194 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.195 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.195 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.195 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.195 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.195 INFO:teuthology.orchestra.run.smithi040.stdout: "AUDIT_API_ENABLED": { 2024-04-21T23:28:38.195 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "AUDIT_API_ENABLED", 2024-04-21T23:28:38.196 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.196 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.196 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.196 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.196 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.197 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.197 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.197 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.197 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.197 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.197 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.197 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.198 INFO:teuthology.orchestra.run.smithi040.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-21T23:28:38.198 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-21T23:28:38.198 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.198 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.198 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.198 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:38.199 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.199 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.199 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.199 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.199 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.199 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.200 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.200 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.200 INFO:teuthology.orchestra.run.smithi040.stdout: "ENABLE_BROWSABLE_API": { 2024-04-21T23:28:38.200 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-21T23:28:38.200 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.201 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.201 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.201 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:38.201 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.201 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.201 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.202 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.202 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.202 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.202 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.202 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.202 INFO:teuthology.orchestra.run.smithi040.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-21T23:28:38.203 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-21T23:28:38.203 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.203 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.203 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.203 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:38.203 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.204 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.204 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.204 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.204 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.204 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.205 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.205 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.205 INFO:teuthology.orchestra.run.smithi040.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-21T23:28:38.205 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-21T23:28:38.205 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.205 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.206 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.206 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:38.206 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.206 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.206 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.206 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.207 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.207 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.207 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.207 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.207 INFO:teuthology.orchestra.run.smithi040.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-21T23:28:38.208 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-21T23:28:38.208 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.208 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.208 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.208 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:38.208 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.209 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.209 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.209 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.209 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.209 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.210 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.210 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.210 INFO:teuthology.orchestra.run.smithi040.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-21T23:28:38.210 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-21T23:28:38.210 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.210 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.210 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.211 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:38.211 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.211 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.211 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.211 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.212 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.212 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.212 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.212 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.213 INFO:teuthology.orchestra.run.smithi040.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-21T23:28:38.213 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-21T23:28:38.213 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.213 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.213 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.213 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:38.213 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.214 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.214 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.214 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.214 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.214 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.214 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.215 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.215 INFO:teuthology.orchestra.run.smithi040.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-21T23:28:38.215 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-21T23:28:38.215 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.215 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.215 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.216 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:38.216 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.216 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.216 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.217 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.217 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.217 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.217 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.217 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.218 INFO:teuthology.orchestra.run.smithi040.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-21T23:28:38.218 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-21T23:28:38.218 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.218 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.218 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.219 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:38.219 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.219 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.219 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.220 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.220 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.220 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.220 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.220 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.221 INFO:teuthology.orchestra.run.smithi040.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-21T23:28:38.221 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-21T23:28:38.221 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.221 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.221 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.221 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.222 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.222 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.222 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.222 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.222 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.223 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.223 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.223 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.223 INFO:teuthology.orchestra.run.smithi040.stdout: "GRAFANA_API_PASSWORD": { 2024-04-21T23:28:38.223 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-21T23:28:38.223 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.224 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.224 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.224 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "admin", 2024-04-21T23:28:38.224 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.224 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.224 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.224 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.225 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.225 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.225 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.225 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.225 INFO:teuthology.orchestra.run.smithi040.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-21T23:28:38.225 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-21T23:28:38.225 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.226 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.226 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.226 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:38.226 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.226 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.226 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.226 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.227 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.227 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.227 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.227 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.227 INFO:teuthology.orchestra.run.smithi040.stdout: "GRAFANA_API_URL": { 2024-04-21T23:28:38.227 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "GRAFANA_API_URL", 2024-04-21T23:28:38.227 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.228 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.228 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.228 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.228 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.228 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.229 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.229 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.229 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.229 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.229 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.229 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.230 INFO:teuthology.orchestra.run.smithi040.stdout: "GRAFANA_API_USERNAME": { 2024-04-21T23:28:38.230 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-21T23:28:38.230 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.230 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.230 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.230 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "admin", 2024-04-21T23:28:38.231 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.231 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.231 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.231 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.231 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.231 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.231 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.232 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.232 INFO:teuthology.orchestra.run.smithi040.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-21T23:28:38.232 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-21T23:28:38.232 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.232 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.232 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.233 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.233 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.233 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.233 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.233 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.233 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.234 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.234 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.234 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.234 INFO:teuthology.orchestra.run.smithi040.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-21T23:28:38.234 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-21T23:28:38.235 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.235 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.235 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.235 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.236 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.236 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.236 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.236 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.236 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.236 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.237 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.237 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.237 INFO:teuthology.orchestra.run.smithi040.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-21T23:28:38.237 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-21T23:28:38.237 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.238 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.238 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.238 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:38.238 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.238 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.239 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.239 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.239 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.239 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.239 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.240 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.240 INFO:teuthology.orchestra.run.smithi040.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-21T23:28:38.240 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-21T23:28:38.240 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.240 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.240 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.241 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.241 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.241 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.241 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.242 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.242 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.242 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.242 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.242 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.242 INFO:teuthology.orchestra.run.smithi040.stdout: "PROMETHEUS_API_HOST": { 2024-04-21T23:28:38.243 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-21T23:28:38.243 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.243 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.243 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.243 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.243 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.243 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.244 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.244 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.244 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.244 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.244 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.245 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.245 INFO:teuthology.orchestra.run.smithi040.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-21T23:28:38.245 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-21T23:28:38.245 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.245 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.245 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.246 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:38.246 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.246 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.246 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.247 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.247 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.247 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.247 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.247 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.248 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-21T23:28:38.248 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-21T23:28:38.248 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.248 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.248 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.248 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.249 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.249 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.249 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.249 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.249 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.249 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.250 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.250 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.250 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-21T23:28:38.250 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-21T23:28:38.251 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.251 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.251 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.251 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.251 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.251 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.251 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.252 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.252 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.252 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.252 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.252 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.252 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-21T23:28:38.253 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-21T23:28:38.253 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.253 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.253 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.253 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:38.253 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.253 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.254 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.254 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.254 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.254 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.254 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.254 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.255 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-21T23:28:38.255 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-21T23:28:38.255 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.255 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.256 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.256 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:38.256 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.256 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.256 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.257 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.257 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.257 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.257 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.257 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.258 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-21T23:28:38.258 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-21T23:28:38.258 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.258 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.258 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.259 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.259 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.259 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.259 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.259 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.260 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.260 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.260 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.260 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.261 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-21T23:28:38.261 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-21T23:28:38.261 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.261 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.261 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.262 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.262 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.262 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.262 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.262 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.262 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.263 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.263 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.263 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.263 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-21T23:28:38.263 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-21T23:28:38.264 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.264 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.264 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.264 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.264 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.265 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.265 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.265 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.265 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.265 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.265 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.266 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.266 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_ENABLED": { 2024-04-21T23:28:38.266 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-21T23:28:38.266 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.266 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.266 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.267 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:38.267 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.267 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.267 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.268 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.268 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.268 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.268 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.268 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.268 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-21T23:28:38.269 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-21T23:28:38.269 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.269 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.269 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.269 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-21T23:28:38.270 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.270 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.270 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.270 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.270 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.270 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.271 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.271 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.271 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-21T23:28:38.271 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-21T23:28:38.271 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:38.271 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.272 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.272 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10", 2024-04-21T23:28:38.272 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.272 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.272 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.272 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.273 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.273 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.273 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.273 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.273 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-21T23:28:38.274 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-21T23:28:38.274 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:38.274 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.274 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.274 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "8", 2024-04-21T23:28:38.275 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.275 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.275 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.275 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.276 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.276 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.276 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.276 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.276 INFO:teuthology.orchestra.run.smithi040.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-21T23:28:38.276 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-21T23:28:38.277 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:38.277 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.277 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.277 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "45", 2024-04-21T23:28:38.277 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.278 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.278 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.278 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.278 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.278 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.279 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.279 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.279 INFO:teuthology.orchestra.run.smithi040.stdout: "RGW_API_ACCESS_KEY": { 2024-04-21T23:28:38.279 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-21T23:28:38.279 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.280 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.280 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.280 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.280 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.280 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.280 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.281 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.281 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.281 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.281 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.282 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.282 INFO:teuthology.orchestra.run.smithi040.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-21T23:28:38.282 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-21T23:28:38.282 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.282 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.282 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.283 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "admin", 2024-04-21T23:28:38.283 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.283 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.283 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.283 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.283 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.283 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.284 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.284 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.284 INFO:teuthology.orchestra.run.smithi040.stdout: "RGW_API_SECRET_KEY": { 2024-04-21T23:28:38.284 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-21T23:28:38.284 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.284 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.284 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.284 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.285 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.285 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.285 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.285 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.285 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.285 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.286 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.286 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.286 INFO:teuthology.orchestra.run.smithi040.stdout: "RGW_API_SSL_VERIFY": { 2024-04-21T23:28:38.286 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-21T23:28:38.286 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.286 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.286 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.287 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:38.287 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.287 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.287 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.287 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.287 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.288 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.288 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.288 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.288 INFO:teuthology.orchestra.run.smithi040.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-21T23:28:38.288 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-21T23:28:38.289 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:38.289 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.289 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.289 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "0", 2024-04-21T23:28:38.290 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.290 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.290 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.290 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.290 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.290 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.291 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.291 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.291 INFO:teuthology.orchestra.run.smithi040.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-21T23:28:38.291 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-21T23:28:38.291 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:38.292 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.292 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.292 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10", 2024-04-21T23:28:38.292 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.292 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.292 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.292 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.293 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.293 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.293 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.293 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.293 INFO:teuthology.orchestra.run.smithi040.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-21T23:28:38.293 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-21T23:28:38.293 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:38.294 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.294 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.294 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "5", 2024-04-21T23:28:38.294 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.294 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.294 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.295 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.295 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.295 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.295 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.295 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.295 INFO:teuthology.orchestra.run.smithi040.stdout: "cross_origin_url": { 2024-04-21T23:28:38.295 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "cross_origin_url", 2024-04-21T23:28:38.296 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.296 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.296 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.296 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.296 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.296 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.297 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.297 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.297 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.297 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.297 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.298 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.298 INFO:teuthology.orchestra.run.smithi040.stdout: "crt_file": { 2024-04-21T23:28:38.298 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "crt_file", 2024-04-21T23:28:38.298 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.299 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.299 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.299 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.299 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.299 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.300 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.300 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.300 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.300 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.300 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.300 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.301 INFO:teuthology.orchestra.run.smithi040.stdout: "debug": { 2024-04-21T23:28:38.301 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "debug", 2024-04-21T23:28:38.301 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.301 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.301 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.302 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.302 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.302 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.302 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.302 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Enable/disable debug options", 2024-04-21T23:28:38.302 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.303 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.303 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.303 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.303 INFO:teuthology.orchestra.run.smithi040.stdout: "jwt_token_ttl": { 2024-04-21T23:28:38.303 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "jwt_token_ttl", 2024-04-21T23:28:38.303 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:38.304 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.304 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.304 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "28800", 2024-04-21T23:28:38.304 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.304 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.304 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.305 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.305 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.305 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.305 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.305 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.305 INFO:teuthology.orchestra.run.smithi040.stdout: "key_file": { 2024-04-21T23:28:38.306 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "key_file", 2024-04-21T23:28:38.306 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.306 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.306 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.307 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.307 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.307 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.307 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.307 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.308 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.308 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.308 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.308 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.308 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:38.308 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:38.309 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.309 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.309 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.309 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.309 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.310 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.310 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:38.310 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:38.310 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:38.310 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:38.311 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:38.311 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:38.311 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:38.311 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:38.311 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.311 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.312 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.312 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.312 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.312 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:38.312 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:38.313 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.313 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.313 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.313 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.313 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.313 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.313 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.314 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.314 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.314 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.314 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.314 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.314 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:38.315 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:38.315 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.315 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.315 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.315 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:38.315 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.316 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.316 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:38.316 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:38.316 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:38.316 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:38.316 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:38.317 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:38.317 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:38.317 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:38.317 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.317 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.317 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.318 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.318 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.318 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:38.318 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:38.319 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.319 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.319 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.319 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.319 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.319 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.320 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.320 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.320 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.320 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.320 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.321 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.321 INFO:teuthology.orchestra.run.smithi040.stdout: "motd": { 2024-04-21T23:28:38.321 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "motd", 2024-04-21T23:28:38.321 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.321 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.322 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.322 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.322 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.322 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.323 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.323 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "The message of the day", 2024-04-21T23:28:38.323 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.323 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.323 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.323 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.323 INFO:teuthology.orchestra.run.smithi040.stdout: "redirect_resolve_ip_addr": { 2024-04-21T23:28:38.324 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "redirect_resolve_ip_addr", 2024-04-21T23:28:38.324 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.324 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.324 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.324 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.324 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.325 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.325 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.325 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.325 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.325 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.325 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.326 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.326 INFO:teuthology.orchestra.run.smithi040.stdout: "server_addr": { 2024-04-21T23:28:38.326 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "server_addr", 2024-04-21T23:28:38.326 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.326 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.326 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.327 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "::", 2024-04-21T23:28:38.327 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.327 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.327 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.327 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.327 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.328 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.328 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.328 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.328 INFO:teuthology.orchestra.run.smithi040.stdout: "server_port": { 2024-04-21T23:28:38.328 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "server_port", 2024-04-21T23:28:38.329 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:38.329 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.329 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.329 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "8080", 2024-04-21T23:28:38.329 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.329 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.329 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.330 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.330 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.330 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.330 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.330 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.330 INFO:teuthology.orchestra.run.smithi040.stdout: "ssl": { 2024-04-21T23:28:38.331 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ssl", 2024-04-21T23:28:38.331 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.331 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.331 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.331 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:38.332 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.332 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.332 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.332 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.332 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.332 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.333 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.333 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.333 INFO:teuthology.orchestra.run.smithi040.stdout: "ssl_server_port": { 2024-04-21T23:28:38.333 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ssl_server_port", 2024-04-21T23:28:38.333 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:38.333 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.333 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.334 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "8443", 2024-04-21T23:28:38.334 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.334 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.334 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.334 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.334 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.334 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.334 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.335 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.335 INFO:teuthology.orchestra.run.smithi040.stdout: "standby_behaviour": { 2024-04-21T23:28:38.335 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "standby_behaviour", 2024-04-21T23:28:38.335 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.335 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.335 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.336 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "redirect", 2024-04-21T23:28:38.336 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.336 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.336 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:38.336 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:38.336 INFO:teuthology.orchestra.run.smithi040.stdout: "redirect" 2024-04-21T23:28:38.336 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:38.337 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.337 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.337 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.337 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.337 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.337 INFO:teuthology.orchestra.run.smithi040.stdout: "standby_error_status_code": { 2024-04-21T23:28:38.337 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "standby_error_status_code", 2024-04-21T23:28:38.337 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:38.338 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.338 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.338 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "500", 2024-04-21T23:28:38.338 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "400", 2024-04-21T23:28:38.338 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "599", 2024-04-21T23:28:38.338 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.338 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.339 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.339 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.339 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.339 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.339 INFO:teuthology.orchestra.run.smithi040.stdout: "url_prefix": { 2024-04-21T23:28:38.340 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "url_prefix", 2024-04-21T23:28:38.340 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.340 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.340 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.340 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.340 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.340 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.340 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.341 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.341 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.341 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.341 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.341 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:38.341 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:38.342 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.342 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:38.342 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "devicehealth", 2024-04-21T23:28:38.342 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:38.342 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:38.342 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:38.342 INFO:teuthology.orchestra.run.smithi040.stdout: "enable_monitoring": { 2024-04-21T23:28:38.343 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "enable_monitoring", 2024-04-21T23:28:38.343 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.343 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.343 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.343 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:38.343 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.344 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.344 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.344 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "monitor device health metrics", 2024-04-21T23:28:38.344 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.344 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.344 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.344 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.344 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:38.345 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:38.345 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.345 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.345 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.345 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.345 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.345 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.345 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:38.346 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:38.346 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:38.346 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:38.346 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:38.346 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:38.346 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:38.346 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:38.347 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.347 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.347 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.347 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.347 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.347 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:38.347 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:38.348 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.348 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.348 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.348 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.348 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.348 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.348 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.348 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.349 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.349 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.349 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.349 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.349 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:38.349 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:38.349 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.350 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.350 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.350 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:38.350 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.350 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.350 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:38.350 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:38.350 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:38.351 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:38.351 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:38.351 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:38.351 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:38.351 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:38.351 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.351 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.351 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.352 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.352 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.352 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:38.352 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:38.352 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.352 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.352 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.353 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.353 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.353 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.353 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.353 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.353 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.353 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.353 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.354 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.354 INFO:teuthology.orchestra.run.smithi040.stdout: "mark_out_threshold": { 2024-04-21T23:28:38.354 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "mark_out_threshold", 2024-04-21T23:28:38.354 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-21T23:28:38.354 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.354 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.354 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "2419200", 2024-04-21T23:28:38.354 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.355 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.355 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.355 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-21T23:28:38.355 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.355 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.355 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.355 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.355 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_name": { 2024-04-21T23:28:38.356 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "pool_name", 2024-04-21T23:28:38.356 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.356 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.356 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.356 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "device_health_metrics", 2024-04-21T23:28:38.356 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.356 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.356 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.357 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-21T23:28:38.357 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.357 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.357 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.357 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.357 INFO:teuthology.orchestra.run.smithi040.stdout: "retention_period": { 2024-04-21T23:28:38.357 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "retention_period", 2024-04-21T23:28:38.358 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-21T23:28:38.358 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.358 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.358 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "15552000", 2024-04-21T23:28:38.358 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.358 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.358 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.359 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "how long to retain device health metrics", 2024-04-21T23:28:38.359 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.359 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.359 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.359 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.359 INFO:teuthology.orchestra.run.smithi040.stdout: "scrape_frequency": { 2024-04-21T23:28:38.359 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "scrape_frequency", 2024-04-21T23:28:38.359 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-21T23:28:38.360 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.360 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.360 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "86400", 2024-04-21T23:28:38.360 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.360 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.360 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.360 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-21T23:28:38.361 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.361 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.361 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.361 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.361 INFO:teuthology.orchestra.run.smithi040.stdout: "self_heal": { 2024-04-21T23:28:38.361 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "self_heal", 2024-04-21T23:28:38.361 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.361 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.362 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.362 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:38.362 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.362 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.362 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.362 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-21T23:28:38.362 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.363 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.363 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.363 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.363 INFO:teuthology.orchestra.run.smithi040.stdout: "sleep_interval": { 2024-04-21T23:28:38.363 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "sleep_interval", 2024-04-21T23:28:38.363 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-21T23:28:38.363 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.363 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.363 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "600", 2024-04-21T23:28:38.364 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.364 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.364 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.364 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "how frequently to wake up and check device health", 2024-04-21T23:28:38.364 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.364 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.364 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.365 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.365 INFO:teuthology.orchestra.run.smithi040.stdout: "warn_threshold": { 2024-04-21T23:28:38.365 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "warn_threshold", 2024-04-21T23:28:38.365 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-21T23:28:38.365 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.365 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.365 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "7257600", 2024-04-21T23:28:38.365 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.366 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.366 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.366 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-21T23:28:38.366 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.366 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.366 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.366 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:38.366 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:38.367 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.367 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:38.367 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "diskprediction_local", 2024-04-21T23:28:38.367 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:38.367 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:38.367 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:38.367 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:38.367 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:38.368 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.368 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.368 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.368 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.368 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.368 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.368 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:38.368 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:38.369 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:38.369 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:38.369 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:38.369 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:38.369 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:38.369 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:38.369 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.369 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.370 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.370 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.370 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.370 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:38.370 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:38.370 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.370 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.370 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.371 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.371 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.371 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.371 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.371 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.371 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.371 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.371 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.372 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.372 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:38.372 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:38.372 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.372 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.372 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.372 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:38.372 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.373 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.373 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:38.373 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:38.373 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:38.373 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:38.373 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:38.373 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:38.373 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:38.374 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:38.374 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.374 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.374 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.374 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.374 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.375 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:38.375 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:38.375 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.375 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.375 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.375 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.375 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.375 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.376 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.376 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.376 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.376 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.376 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.376 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.376 INFO:teuthology.orchestra.run.smithi040.stdout: "predict_interval": { 2024-04-21T23:28:38.377 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "predict_interval", 2024-04-21T23:28:38.377 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.377 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.377 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.377 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "86400", 2024-04-21T23:28:38.378 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.378 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.378 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.378 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.378 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.378 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.379 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.379 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.379 INFO:teuthology.orchestra.run.smithi040.stdout: "predictor_model": { 2024-04-21T23:28:38.379 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "predictor_model", 2024-04-21T23:28:38.379 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.379 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.379 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.380 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "prophetstor", 2024-04-21T23:28:38.380 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.380 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.380 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.380 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.380 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.380 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.381 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.381 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.381 INFO:teuthology.orchestra.run.smithi040.stdout: "sleep_interval": { 2024-04-21T23:28:38.381 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "sleep_interval", 2024-04-21T23:28:38.381 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.381 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.382 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.382 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "600", 2024-04-21T23:28:38.382 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.382 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.382 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.382 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.382 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.382 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.383 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.383 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:38.383 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:38.383 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.383 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:38.384 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "influx", 2024-04-21T23:28:38.384 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": false, 2024-04-21T23:28:38.384 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "influxdb python module not found", 2024-04-21T23:28:38.384 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:38.384 INFO:teuthology.orchestra.run.smithi040.stdout: "batch_size": { 2024-04-21T23:28:38.385 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "batch_size", 2024-04-21T23:28:38.385 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:38.385 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.385 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.385 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "5000", 2024-04-21T23:28:38.385 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.386 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.386 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.386 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-21T23:28:38.386 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.386 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.387 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.387 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.387 INFO:teuthology.orchestra.run.smithi040.stdout: "database": { 2024-04-21T23:28:38.387 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "database", 2024-04-21T23:28:38.387 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.388 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.388 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.388 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "ceph", 2024-04-21T23:28:38.388 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.388 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.389 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.389 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-21T23:28:38.389 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.389 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.389 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.390 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.390 INFO:teuthology.orchestra.run.smithi040.stdout: "hostname": { 2024-04-21T23:28:38.390 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "hostname", 2024-04-21T23:28:38.390 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.391 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.391 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.391 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.391 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.391 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.392 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.392 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "InfluxDB server hostname", 2024-04-21T23:28:38.392 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.392 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.392 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.393 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.393 INFO:teuthology.orchestra.run.smithi040.stdout: "interval": { 2024-04-21T23:28:38.393 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "interval", 2024-04-21T23:28:38.393 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-21T23:28:38.393 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.394 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.394 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "30", 2024-04-21T23:28:38.394 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "5", 2024-04-21T23:28:38.394 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.394 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.395 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-21T23:28:38.395 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.395 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.395 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.395 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.395 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:38.396 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:38.396 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.396 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.396 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.396 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.396 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.397 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.397 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:38.397 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:38.397 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:38.397 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:38.398 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:38.398 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:38.398 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:38.398 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:38.399 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.399 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.399 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.399 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.399 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.400 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:38.400 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:38.400 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.400 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.400 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.401 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.401 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.401 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.401 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.401 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.402 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.402 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.402 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.402 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.402 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:38.403 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:38.403 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.403 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.403 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.403 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:38.404 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.404 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.404 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:38.404 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:38.404 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:38.405 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:38.405 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:38.405 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:38.405 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:38.406 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:38.406 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.406 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.406 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.406 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.407 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.407 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:38.407 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:38.407 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.407 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.407 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.407 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.408 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.408 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.408 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.408 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.408 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.408 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.408 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.409 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.409 INFO:teuthology.orchestra.run.smithi040.stdout: "password": { 2024-04-21T23:28:38.409 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "password", 2024-04-21T23:28:38.409 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.409 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.409 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.409 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.410 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.410 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.410 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.410 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "password of InfluxDB server user", 2024-04-21T23:28:38.410 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.410 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.410 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.411 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.411 INFO:teuthology.orchestra.run.smithi040.stdout: "port": { 2024-04-21T23:28:38.411 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "port", 2024-04-21T23:28:38.411 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:38.411 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.411 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.412 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "8086", 2024-04-21T23:28:38.412 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.412 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.412 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.412 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "InfluxDB server port", 2024-04-21T23:28:38.413 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.413 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.413 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.413 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.413 INFO:teuthology.orchestra.run.smithi040.stdout: "ssl": { 2024-04-21T23:28:38.414 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ssl", 2024-04-21T23:28:38.414 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.414 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.414 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.414 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "false", 2024-04-21T23:28:38.414 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.415 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.415 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.415 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-21T23:28:38.415 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.415 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.416 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.416 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.416 INFO:teuthology.orchestra.run.smithi040.stdout: "threads": { 2024-04-21T23:28:38.416 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "threads", 2024-04-21T23:28:38.416 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:38.417 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.417 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.417 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "5", 2024-04-21T23:28:38.417 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "1", 2024-04-21T23:28:38.418 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "32", 2024-04-21T23:28:38.418 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.418 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-21T23:28:38.418 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.419 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.419 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.419 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.419 INFO:teuthology.orchestra.run.smithi040.stdout: "username": { 2024-04-21T23:28:38.419 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "username", 2024-04-21T23:28:38.420 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.420 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.420 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.420 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.420 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.420 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.421 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.421 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "username of InfluxDB server user", 2024-04-21T23:28:38.421 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.421 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.421 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.421 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.422 INFO:teuthology.orchestra.run.smithi040.stdout: "verify_ssl": { 2024-04-21T23:28:38.422 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "verify_ssl", 2024-04-21T23:28:38.422 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.422 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.422 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.423 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "true", 2024-04-21T23:28:38.423 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.423 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.423 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.423 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-21T23:28:38.423 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.424 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.424 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.424 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:38.424 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:38.424 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.425 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:38.425 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "insights", 2024-04-21T23:28:38.425 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:38.425 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:38.425 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:38.425 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:38.426 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:38.426 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.426 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.426 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.426 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.427 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.427 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.427 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:38.428 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:38.428 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:38.428 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:38.428 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:38.428 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:38.429 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:38.429 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:38.429 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.429 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.429 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.430 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.430 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.430 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:38.430 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:38.430 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.431 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.431 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.431 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.431 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.432 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.432 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.432 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.432 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.432 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.432 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.432 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.433 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:38.433 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:38.433 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.433 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.433 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.433 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:38.434 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.434 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.434 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:38.434 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:38.434 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:38.434 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:38.435 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:38.435 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:38.435 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:38.435 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:38.436 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.436 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.436 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.436 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.436 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.436 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:38.437 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:38.437 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.437 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.437 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.437 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.438 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.438 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.438 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.438 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.439 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.439 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.439 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.440 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:38.440 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:38.440 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.440 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:38.440 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "iostat", 2024-04-21T23:28:38.440 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:38.440 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:38.441 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:38.441 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:38.441 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:38.442 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.442 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.442 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.443 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.443 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.443 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.443 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:38.443 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:38.443 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:38.444 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:38.444 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:38.444 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:38.444 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:38.445 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:38.445 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.445 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.445 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.445 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.446 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.446 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:38.446 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:38.446 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.446 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.447 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.447 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.447 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.447 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.448 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.448 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.448 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.448 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.448 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.449 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.449 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:38.449 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:38.449 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.449 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.450 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.450 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:38.450 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.450 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.450 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:38.450 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:38.451 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:38.451 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:38.451 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:38.451 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:38.451 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:38.451 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:38.451 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.452 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.452 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.452 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.452 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.452 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:38.452 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:38.453 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.453 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.453 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.453 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.453 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.453 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.454 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.454 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.454 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.454 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.455 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.455 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:38.455 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:38.455 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.455 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:38.456 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "k8sevents", 2024-04-21T23:28:38.456 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:38.456 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:38.456 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:38.457 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_event_retention_days": { 2024-04-21T23:28:38.457 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ceph_event_retention_days", 2024-04-21T23:28:38.457 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:38.457 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.457 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.458 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "7", 2024-04-21T23:28:38.458 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.458 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.458 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.458 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-21T23:28:38.458 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.458 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.459 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.459 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.459 INFO:teuthology.orchestra.run.smithi040.stdout: "config_check_secs": { 2024-04-21T23:28:38.459 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "config_check_secs", 2024-04-21T23:28:38.459 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:38.459 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.459 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.460 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10", 2024-04-21T23:28:38.460 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "10", 2024-04-21T23:28:38.460 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.460 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.460 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-21T23:28:38.460 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.460 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.460 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.461 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.461 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:38.461 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:38.461 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.461 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.461 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.462 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.462 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.462 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.462 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:38.462 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:38.462 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:38.463 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:38.463 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:38.463 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:38.463 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:38.463 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:38.463 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.463 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.464 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.464 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.464 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.464 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:38.464 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:38.464 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.464 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.465 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.465 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.465 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.465 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.465 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.465 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.465 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.466 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.466 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.466 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.466 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:38.466 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:38.466 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.467 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.467 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.467 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:38.467 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.467 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.467 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:38.467 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:38.468 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:38.468 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:38.468 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:38.468 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:38.468 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:38.468 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:38.469 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.469 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.469 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.469 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.469 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.469 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:38.469 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:38.470 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.470 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.470 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.470 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.470 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.470 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.470 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.471 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.471 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.471 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.471 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.471 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:38.472 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:38.472 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.472 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:38.472 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "localpool", 2024-04-21T23:28:38.472 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:38.473 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:38.473 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:38.473 INFO:teuthology.orchestra.run.smithi040.stdout: "failure_domain": { 2024-04-21T23:28:38.473 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "failure_domain", 2024-04-21T23:28:38.473 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.474 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.474 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.474 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "host", 2024-04-21T23:28:38.474 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.474 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.474 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.475 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "failure domain for any created local pool", 2024-04-21T23:28:38.475 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-21T23:28:38.475 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.475 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.475 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.475 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:38.475 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:38.475 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.476 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.476 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.476 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.476 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.476 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.476 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:38.476 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:38.476 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:38.477 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:38.477 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:38.477 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:38.477 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:38.477 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:38.477 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.477 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.478 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.478 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.478 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.478 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:38.478 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:38.478 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.478 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.479 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.479 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.479 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.479 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.479 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.479 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.479 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.479 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.480 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.480 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.480 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:38.480 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:38.480 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.480 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.480 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.480 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:38.481 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.481 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.481 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:38.481 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:38.481 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:38.481 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:38.481 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:38.481 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:38.482 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:38.482 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:38.482 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.482 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.482 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.482 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.482 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.482 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:38.483 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:38.483 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.483 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.483 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.483 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.485 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.485 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.485 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.486 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.486 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.486 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.486 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.486 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.486 INFO:teuthology.orchestra.run.smithi040.stdout: "min_size": { 2024-04-21T23:28:38.487 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "min_size", 2024-04-21T23:28:38.487 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:38.487 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.487 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.488 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.488 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.488 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.488 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.488 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "default min_size for any created local pool", 2024-04-21T23:28:38.489 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-21T23:28:38.489 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.489 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.489 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.489 INFO:teuthology.orchestra.run.smithi040.stdout: "num_rep": { 2024-04-21T23:28:38.490 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "num_rep", 2024-04-21T23:28:38.490 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:38.490 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.490 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.490 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "3", 2024-04-21T23:28:38.490 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.491 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.491 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.491 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "default replica count for any created local pool", 2024-04-21T23:28:38.491 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.491 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.491 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.492 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.492 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num": { 2024-04-21T23:28:38.492 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "pg_num", 2024-04-21T23:28:38.492 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:38.492 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.492 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.492 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "128", 2024-04-21T23:28:38.493 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.493 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.493 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.493 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "default pg_num for any created local pool", 2024-04-21T23:28:38.493 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.493 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.493 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.494 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.494 INFO:teuthology.orchestra.run.smithi040.stdout: "prefix": { 2024-04-21T23:28:38.494 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "prefix", 2024-04-21T23:28:38.494 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.494 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.494 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.494 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.495 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.495 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.495 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.495 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "name prefix for any created local pool", 2024-04-21T23:28:38.495 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.496 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.496 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.496 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.496 INFO:teuthology.orchestra.run.smithi040.stdout: "subtree": { 2024-04-21T23:28:38.496 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "subtree", 2024-04-21T23:28:38.496 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.496 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.496 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.497 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "rack", 2024-04-21T23:28:38.497 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.497 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.497 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.497 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-21T23:28:38.497 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-21T23:28:38.497 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.497 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.498 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:38.498 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:38.498 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.498 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:38.498 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "mds_autoscaler", 2024-04-21T23:28:38.498 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:38.498 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:38.498 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:38.499 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:38.499 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:38.499 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.499 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.499 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.499 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.499 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.499 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.500 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:38.500 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:38.500 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:38.500 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:38.500 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:38.500 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:38.500 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:38.500 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:38.501 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.501 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.501 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.501 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.501 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.501 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:38.501 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:38.502 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.502 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.502 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.502 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.502 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.503 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.503 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.503 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.503 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.503 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.503 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.504 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.504 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:38.504 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:38.504 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.504 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.504 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.505 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:38.505 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.505 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.505 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:38.505 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:38.505 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:38.506 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:38.506 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:38.506 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:38.506 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:38.506 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:38.506 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.506 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.507 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.507 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.507 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.507 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:38.507 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:38.507 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.507 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.507 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.508 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.508 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.508 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.508 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.508 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.508 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.508 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.508 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.508 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:38.509 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:38.509 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.509 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:38.509 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "mirroring", 2024-04-21T23:28:38.509 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:38.509 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:38.509 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:38.509 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:38.510 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:38.510 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.510 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.510 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.510 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.510 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.510 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.510 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:38.511 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:38.511 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:38.511 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:38.511 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:38.511 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:38.511 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:38.511 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:38.511 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.511 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.512 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.512 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.512 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.512 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:38.512 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:38.512 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.512 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.512 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.513 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.513 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.513 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.513 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.513 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.513 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.513 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.513 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.514 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.514 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:38.514 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:38.514 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.514 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.514 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.514 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:38.514 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.515 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.515 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:38.515 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:38.515 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:38.515 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:38.515 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:38.515 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:38.515 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:38.516 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:38.516 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.516 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.516 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.516 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.516 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.516 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:38.516 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:38.517 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.517 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.517 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.517 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.517 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.517 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.518 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.518 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.518 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.518 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.518 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.518 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:38.519 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:38.519 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.519 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:38.519 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "nfs", 2024-04-21T23:28:38.519 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:38.519 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:38.519 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:38.519 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:38.520 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:38.520 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.520 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.520 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.520 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.520 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.520 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.520 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:38.521 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:38.521 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:38.521 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:38.521 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:38.521 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:38.521 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:38.521 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:38.521 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.521 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.522 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.522 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.522 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.522 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:38.522 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:38.522 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.522 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.523 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.523 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.523 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.523 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.523 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.523 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.523 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.523 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.523 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.524 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.524 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:38.524 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:38.524 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.524 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.524 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.524 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:38.524 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.525 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.525 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:38.525 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:38.525 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:38.525 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:38.525 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:38.525 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:38.525 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:38.525 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:38.526 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.526 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.526 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.526 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.526 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.526 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:38.526 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:38.526 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.527 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.527 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.527 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.527 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.527 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.527 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.527 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.527 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.528 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.528 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.528 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:38.528 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:38.528 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.528 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:38.528 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "orchestrator", 2024-04-21T23:28:38.528 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:38.528 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:38.529 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:38.529 INFO:teuthology.orchestra.run.smithi040.stdout: "fail_fs": { 2024-04-21T23:28:38.529 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "fail_fs", 2024-04-21T23:28:38.529 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.529 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.529 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.529 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.529 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.529 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.530 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.530 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-21T23:28:38.530 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.530 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.530 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.530 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.530 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:38.530 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:38.530 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.531 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.531 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.531 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.531 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.531 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.531 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:38.531 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:38.531 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:38.531 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:38.532 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:38.532 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:38.532 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:38.532 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:38.532 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.532 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.532 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.532 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.533 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.533 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:38.533 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:38.533 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.533 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.533 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.533 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.533 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.533 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.534 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.534 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.534 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.534 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.534 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.534 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.534 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:38.534 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:38.535 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.535 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.535 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.535 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:38.535 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.535 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.535 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:38.535 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:38.535 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:38.536 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:38.536 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:38.536 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:38.536 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:38.536 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:38.536 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.536 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.536 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.536 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.537 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.537 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:38.537 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:38.537 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.537 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.537 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.537 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.537 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.537 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.538 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.538 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.538 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.538 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.538 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.538 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.538 INFO:teuthology.orchestra.run.smithi040.stdout: "orchestrator": { 2024-04-21T23:28:38.538 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "orchestrator", 2024-04-21T23:28:38.538 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.539 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.539 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.539 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.539 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.539 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.539 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:38.539 INFO:teuthology.orchestra.run.smithi040.stdout: "cephadm", 2024-04-21T23:28:38.539 INFO:teuthology.orchestra.run.smithi040.stdout: "rook", 2024-04-21T23:28:38.539 INFO:teuthology.orchestra.run.smithi040.stdout: "test_orchestrator" 2024-04-21T23:28:38.540 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:38.540 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Orchestrator backend", 2024-04-21T23:28:38.540 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.540 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.540 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.540 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:38.540 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:38.540 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.541 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:38.541 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "osd_perf_query", 2024-04-21T23:28:38.541 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:38.541 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:38.541 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:38.541 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:38.541 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:38.541 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.541 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.542 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.542 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.542 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.542 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.542 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:38.542 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:38.542 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:38.542 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:38.543 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:38.543 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:38.543 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:38.543 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:38.543 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.543 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.543 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.543 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.544 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.544 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:38.544 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:38.544 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.544 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.544 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.544 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.544 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.544 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.545 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.545 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.545 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.545 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.545 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.545 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.545 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:38.545 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:38.545 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.546 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.546 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.546 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:38.546 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.546 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.546 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:38.546 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:38.546 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:38.547 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:38.547 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:38.547 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:38.547 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:38.547 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:38.547 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.547 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.547 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.548 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.548 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.548 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:38.548 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:38.548 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.548 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.548 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.548 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.549 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.549 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.549 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.549 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.549 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.549 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.549 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.549 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:38.550 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:38.550 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.550 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:38.550 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "osd_support", 2024-04-21T23:28:38.550 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:38.550 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:38.550 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:38.550 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:38.551 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:38.551 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.551 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.551 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.551 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.551 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.551 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.551 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:38.552 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:38.552 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:38.552 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:38.552 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:38.552 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:38.552 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:38.552 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:38.552 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.552 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.553 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.553 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.553 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.553 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:38.553 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:38.553 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.553 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.553 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.554 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.554 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.554 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.554 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.554 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.554 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.554 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.555 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.555 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.555 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:38.555 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:38.555 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.555 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.556 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.556 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:38.556 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.556 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.556 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:38.557 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:38.557 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:38.557 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:38.557 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:38.557 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:38.557 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:38.557 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:38.557 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.558 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.558 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.558 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.558 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.558 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:38.558 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:38.558 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.558 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.559 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.559 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.559 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.559 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.559 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.559 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.559 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.559 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.559 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.560 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:38.560 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:38.560 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.560 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:38.560 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "pg_autoscaler", 2024-04-21T23:28:38.560 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:38.560 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:38.560 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:38.561 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:38.561 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:38.561 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.561 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.561 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.561 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.561 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.561 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.561 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:38.562 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:38.562 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:38.562 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:38.562 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:38.562 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:38.562 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:38.562 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:38.562 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.563 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.563 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.563 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.563 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.563 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:38.563 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:38.563 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.563 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.563 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.564 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.564 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.564 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.564 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.564 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.564 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.564 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.564 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.564 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.565 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:38.565 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:38.565 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.565 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.565 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.565 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:38.565 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.565 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.565 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:38.566 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:38.566 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:38.566 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:38.566 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:38.566 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:38.566 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:38.566 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:38.566 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.567 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.567 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.567 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.567 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.567 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:38.567 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:38.567 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.567 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.568 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.568 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.568 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.568 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.568 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.568 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.568 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.568 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.568 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.569 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.569 INFO:teuthology.orchestra.run.smithi040.stdout: "sleep_interval": { 2024-04-21T23:28:38.569 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "sleep_interval", 2024-04-21T23:28:38.569 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-21T23:28:38.569 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.569 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.569 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "60", 2024-04-21T23:28:38.569 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.569 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.570 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.570 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.570 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.570 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.570 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.570 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.570 INFO:teuthology.orchestra.run.smithi040.stdout: "threshold": { 2024-04-21T23:28:38.570 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "threshold", 2024-04-21T23:28:38.571 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "float", 2024-04-21T23:28:38.571 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.571 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.571 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "3.0", 2024-04-21T23:28:38.571 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "1.0", 2024-04-21T23:28:38.571 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.571 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.571 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "scaling threshold", 2024-04-21T23:28:38.572 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-21T23:28:38.572 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.572 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.572 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:38.572 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:38.572 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.572 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:38.572 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "progress", 2024-04-21T23:28:38.573 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:38.573 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:38.573 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:38.573 INFO:teuthology.orchestra.run.smithi040.stdout: "allow_pg_recovery_event": { 2024-04-21T23:28:38.573 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "allow_pg_recovery_event", 2024-04-21T23:28:38.573 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.573 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.573 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.573 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.574 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.574 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.574 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.574 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-21T23:28:38.574 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.574 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.574 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.574 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.575 INFO:teuthology.orchestra.run.smithi040.stdout: "enabled": { 2024-04-21T23:28:38.575 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "enabled", 2024-04-21T23:28:38.575 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.575 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.575 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.575 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:38.575 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.575 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.576 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.576 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.576 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.576 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.576 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.576 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.576 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:38.576 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:38.576 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.577 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.577 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.577 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.577 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.577 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.577 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:38.577 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:38.577 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:38.577 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:38.578 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:38.578 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:38.578 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:38.578 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:38.578 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.578 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.578 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.578 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.579 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.579 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:38.579 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:38.579 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.579 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.579 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.580 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.580 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.580 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.580 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.580 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.580 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.580 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.580 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.580 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.581 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:38.581 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:38.581 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.581 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.581 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.581 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:38.581 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.581 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.582 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:38.582 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:38.582 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:38.582 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:38.582 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:38.582 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:38.582 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:38.583 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:38.583 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.583 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.583 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.583 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.583 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.583 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:38.583 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:38.584 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.584 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.584 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.584 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.584 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.584 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.584 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.585 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.585 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.585 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.585 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.585 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.585 INFO:teuthology.orchestra.run.smithi040.stdout: "max_completed_events": { 2024-04-21T23:28:38.585 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "max_completed_events", 2024-04-21T23:28:38.585 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:38.586 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.586 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.586 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "50", 2024-04-21T23:28:38.586 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.586 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.586 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.587 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "number of past completed events to remember", 2024-04-21T23:28:38.587 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.587 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.587 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.587 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.587 INFO:teuthology.orchestra.run.smithi040.stdout: "sleep_interval": { 2024-04-21T23:28:38.587 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "sleep_interval", 2024-04-21T23:28:38.588 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-21T23:28:38.588 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.588 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.588 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "5", 2024-04-21T23:28:38.588 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.588 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.588 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.589 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "how long the module is going to sleep", 2024-04-21T23:28:38.589 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.589 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.589 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.589 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:38.589 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:38.589 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.590 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:38.590 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "prometheus", 2024-04-21T23:28:38.590 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:38.590 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:38.590 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:38.590 INFO:teuthology.orchestra.run.smithi040.stdout: "cache": { 2024-04-21T23:28:38.590 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "cache", 2024-04-21T23:28:38.591 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.591 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.591 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.591 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:38.591 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.591 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.591 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.591 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.591 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.592 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.592 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.592 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.592 INFO:teuthology.orchestra.run.smithi040.stdout: "exclude_perf_counters": { 2024-04-21T23:28:38.592 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "exclude_perf_counters", 2024-04-21T23:28:38.592 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.592 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.592 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.593 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:38.593 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.593 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.593 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.593 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-21T23:28:38.593 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-21T23:28:38.593 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.593 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.594 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.594 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:38.594 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:38.594 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.594 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.594 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.594 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.595 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.595 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.595 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:38.595 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:38.595 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:38.595 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:38.595 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:38.595 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:38.595 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:38.596 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:38.596 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.596 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.596 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.596 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.596 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.596 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:38.596 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:38.597 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.597 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.597 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.597 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.597 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.597 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.597 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.597 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.597 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.598 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.598 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.598 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.598 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:38.598 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:38.598 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.598 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.599 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.599 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:38.599 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.599 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.599 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:38.599 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:38.599 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:38.599 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:38.599 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:38.600 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:38.600 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:38.600 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:38.600 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.600 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.600 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.600 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.601 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.601 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:38.601 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:38.601 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.601 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.601 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.601 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.601 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.601 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.602 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.602 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.602 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.602 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.602 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.602 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.602 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_stats_pools": { 2024-04-21T23:28:38.602 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rbd_stats_pools", 2024-04-21T23:28:38.603 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.603 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.603 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.603 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.603 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.603 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.603 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.603 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.603 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.604 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.604 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.604 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.604 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-21T23:28:38.604 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-21T23:28:38.604 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:38.604 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.604 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.605 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "300", 2024-04-21T23:28:38.605 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.605 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.605 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.605 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.605 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.605 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.605 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.606 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.606 INFO:teuthology.orchestra.run.smithi040.stdout: "scrape_interval": { 2024-04-21T23:28:38.606 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "scrape_interval", 2024-04-21T23:28:38.606 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "float", 2024-04-21T23:28:38.606 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.606 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.606 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "15.0", 2024-04-21T23:28:38.607 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.607 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.607 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.607 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.607 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.607 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.607 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.608 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.608 INFO:teuthology.orchestra.run.smithi040.stdout: "server_addr": { 2024-04-21T23:28:38.608 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "server_addr", 2024-04-21T23:28:38.608 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.608 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.608 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.608 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "::", 2024-04-21T23:28:38.608 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.609 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.609 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.609 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-21T23:28:38.609 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.609 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.609 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.609 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.609 INFO:teuthology.orchestra.run.smithi040.stdout: "server_port": { 2024-04-21T23:28:38.609 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "server_port", 2024-04-21T23:28:38.610 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:38.610 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.610 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.610 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "9283", 2024-04-21T23:28:38.610 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.610 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.610 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.610 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-21T23:28:38.611 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.611 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.611 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.611 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.611 INFO:teuthology.orchestra.run.smithi040.stdout: "stale_cache_strategy": { 2024-04-21T23:28:38.611 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "stale_cache_strategy", 2024-04-21T23:28:38.611 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.611 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.612 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.612 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "log", 2024-04-21T23:28:38.612 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.612 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.612 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.612 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.612 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.612 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.613 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.613 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.613 INFO:teuthology.orchestra.run.smithi040.stdout: "standby_behaviour": { 2024-04-21T23:28:38.613 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "standby_behaviour", 2024-04-21T23:28:38.613 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.613 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.613 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.614 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "default", 2024-04-21T23:28:38.614 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.614 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.614 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:38.614 INFO:teuthology.orchestra.run.smithi040.stdout: "default", 2024-04-21T23:28:38.614 INFO:teuthology.orchestra.run.smithi040.stdout: "error" 2024-04-21T23:28:38.614 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:38.614 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.614 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.615 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.615 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.615 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.615 INFO:teuthology.orchestra.run.smithi040.stdout: "standby_error_status_code": { 2024-04-21T23:28:38.615 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "standby_error_status_code", 2024-04-21T23:28:38.615 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:38.615 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.615 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.616 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "500", 2024-04-21T23:28:38.616 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "400", 2024-04-21T23:28:38.616 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "599", 2024-04-21T23:28:38.616 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.616 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.616 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.616 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.616 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.616 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:38.617 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:38.617 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.617 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:38.617 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rbd_support", 2024-04-21T23:28:38.617 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:38.617 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:38.617 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:38.617 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:38.618 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:38.618 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.618 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.618 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.618 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.618 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.618 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.618 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:38.619 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:38.619 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:38.619 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:38.619 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:38.619 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:38.619 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:38.619 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:38.619 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.620 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.620 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.620 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.620 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.620 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:38.620 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:38.620 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.620 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.620 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.621 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.621 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.621 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.621 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.621 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.621 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.621 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.621 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.621 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.622 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:38.622 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:38.622 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.622 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.622 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.622 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:38.622 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.622 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.623 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:38.623 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:38.623 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:38.623 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:38.623 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:38.623 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:38.623 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:38.623 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:38.624 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.624 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.624 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.624 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.624 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.624 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:38.624 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:38.624 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.624 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.625 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.625 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.625 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.625 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.625 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.625 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.625 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.625 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.625 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.626 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.626 INFO:teuthology.orchestra.run.smithi040.stdout: "max_concurrent_snap_create": { 2024-04-21T23:28:38.626 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "max_concurrent_snap_create", 2024-04-21T23:28:38.626 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:38.626 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.626 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.626 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10", 2024-04-21T23:28:38.627 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.627 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.627 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.627 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.627 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.627 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.627 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.627 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.627 INFO:teuthology.orchestra.run.smithi040.stdout: "mirror_snapshot_schedule": { 2024-04-21T23:28:38.628 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "mirror_snapshot_schedule", 2024-04-21T23:28:38.628 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.628 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.628 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.628 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.628 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.628 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.628 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.629 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.629 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.629 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.629 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.629 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.629 INFO:teuthology.orchestra.run.smithi040.stdout: "trash_purge_schedule": { 2024-04-21T23:28:38.629 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "trash_purge_schedule", 2024-04-21T23:28:38.629 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.630 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.630 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.630 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.630 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.630 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.630 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.630 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.630 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.631 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.631 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.631 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:38.631 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:38.631 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.631 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:38.631 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "restful", 2024-04-21T23:28:38.631 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:38.632 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:38.632 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:38.632 INFO:teuthology.orchestra.run.smithi040.stdout: "enable_auth": { 2024-04-21T23:28:38.632 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "enable_auth", 2024-04-21T23:28:38.632 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.632 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.632 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.632 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:38.632 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.633 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.633 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.633 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.633 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.633 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.633 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.633 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.633 INFO:teuthology.orchestra.run.smithi040.stdout: "key_file": { 2024-04-21T23:28:38.633 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "key_file", 2024-04-21T23:28:38.634 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.634 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.634 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.634 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.634 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.634 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.634 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.634 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.635 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.635 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.635 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.635 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.635 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:38.635 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:38.635 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.635 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.635 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.636 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.636 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.636 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.636 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:38.636 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:38.636 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:38.636 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:38.636 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:38.636 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:38.637 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:38.637 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:38.637 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.637 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.637 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.637 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.637 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.637 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:38.638 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:38.638 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.638 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.638 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.638 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.638 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.638 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.638 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.639 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.639 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.639 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.639 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.639 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.639 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:38.639 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:38.639 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.639 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.640 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.640 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:38.640 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.640 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.640 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:38.640 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:38.640 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:38.640 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:38.640 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:38.641 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:38.641 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:38.641 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:38.641 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.641 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.641 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.641 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.641 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.641 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:38.642 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:38.642 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.642 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.642 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.642 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.642 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.642 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.642 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.643 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.643 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.643 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.643 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.643 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.643 INFO:teuthology.orchestra.run.smithi040.stdout: "server_addr": { 2024-04-21T23:28:38.643 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "server_addr", 2024-04-21T23:28:38.643 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.643 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.644 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.644 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.644 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.644 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.644 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.644 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.644 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.644 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.644 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.645 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.645 INFO:teuthology.orchestra.run.smithi040.stdout: "server_port": { 2024-04-21T23:28:38.645 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "server_port", 2024-04-21T23:28:38.645 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.645 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.645 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.645 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.645 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.645 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.646 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.646 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.646 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.646 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.646 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.646 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:38.646 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:38.646 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.647 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:38.647 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rgw", 2024-04-21T23:28:38.647 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:38.647 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:38.647 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:38.647 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:38.647 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:38.647 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.647 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.648 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.648 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.648 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.648 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.648 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:38.648 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:38.648 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:38.648 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:38.648 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:38.649 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:38.649 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:38.649 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:38.649 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.649 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.649 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.649 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.649 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.650 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:38.650 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:38.650 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.650 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.650 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.650 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.650 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.650 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.650 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.651 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.651 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.651 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.651 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.651 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.651 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:38.651 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:38.651 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.651 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.652 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.652 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:38.652 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.652 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.652 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:38.652 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:38.652 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:38.652 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:38.653 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:38.653 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:38.653 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:38.653 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:38.653 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.653 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.653 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.653 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.653 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.654 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:38.654 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:38.654 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.654 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.654 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.654 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.654 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.654 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.655 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.655 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.655 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.655 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.655 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.655 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:38.655 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:38.655 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.656 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:38.656 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rook", 2024-04-21T23:28:38.656 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:38.656 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:38.656 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:38.656 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:38.656 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:38.656 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.657 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.657 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.657 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.657 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.657 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.657 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:38.657 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:38.657 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:38.657 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:38.658 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:38.658 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:38.658 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:38.658 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:38.658 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.658 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.658 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.658 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.659 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.659 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:38.659 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:38.659 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.659 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.659 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.659 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.659 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.660 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.660 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.660 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.660 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.660 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.660 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.660 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.660 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:38.660 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:38.661 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.661 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.661 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.661 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:38.661 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.661 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.661 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:38.661 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:38.661 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:38.662 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:38.662 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:38.662 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:38.662 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:38.662 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:38.662 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.662 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.662 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.663 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.663 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.663 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:38.663 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:38.663 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.663 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.663 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.664 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.664 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.664 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.664 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.664 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.664 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.664 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.664 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.664 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.665 INFO:teuthology.orchestra.run.smithi040.stdout: "storage_class": { 2024-04-21T23:28:38.665 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "storage_class", 2024-04-21T23:28:38.665 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.665 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.665 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.665 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "local", 2024-04-21T23:28:38.665 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.665 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.665 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.666 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-21T23:28:38.666 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.666 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.666 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.666 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:38.666 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:38.666 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.666 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:38.667 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "selftest", 2024-04-21T23:28:38.667 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:38.667 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:38.667 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:38.667 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:38.667 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:38.667 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.667 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.667 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.668 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.668 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.668 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.668 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:38.668 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:38.668 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:38.668 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:38.668 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:38.669 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:38.669 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:38.669 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:38.669 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.669 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.669 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.669 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.669 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.670 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:38.670 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:38.670 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.670 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.670 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.670 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.670 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.670 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.670 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.671 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.671 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.671 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.671 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.671 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.671 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:38.671 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:38.671 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.672 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.672 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.672 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:38.672 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.672 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.672 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:38.672 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:38.672 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:38.673 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:38.673 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:38.673 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:38.673 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:38.673 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:38.673 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.673 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.673 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.674 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.674 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.674 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:38.674 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:38.674 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.674 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.674 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.674 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.674 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.675 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.675 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.675 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.675 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.675 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.675 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.675 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.675 INFO:teuthology.orchestra.run.smithi040.stdout: "roption1": { 2024-04-21T23:28:38.676 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "roption1", 2024-04-21T23:28:38.676 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.676 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.676 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.676 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.676 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.676 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.676 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.676 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.677 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.677 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.677 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.677 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.677 INFO:teuthology.orchestra.run.smithi040.stdout: "roption2": { 2024-04-21T23:28:38.677 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "roption2", 2024-04-21T23:28:38.677 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.677 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.678 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.678 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "xyz", 2024-04-21T23:28:38.678 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.678 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.678 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.678 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.678 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.678 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.679 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.679 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.679 INFO:teuthology.orchestra.run.smithi040.stdout: "rwoption1": { 2024-04-21T23:28:38.679 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rwoption1", 2024-04-21T23:28:38.679 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.679 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.679 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.679 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.680 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.680 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.680 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.680 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.680 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.680 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.680 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.680 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.680 INFO:teuthology.orchestra.run.smithi040.stdout: "rwoption2": { 2024-04-21T23:28:38.681 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rwoption2", 2024-04-21T23:28:38.681 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:38.681 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.681 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.681 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.681 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.681 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.681 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.681 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.682 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.682 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.682 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.682 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.682 INFO:teuthology.orchestra.run.smithi040.stdout: "rwoption3": { 2024-04-21T23:28:38.682 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rwoption3", 2024-04-21T23:28:38.682 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "float", 2024-04-21T23:28:38.682 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.683 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.683 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.683 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.683 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.683 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.683 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.683 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.683 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.683 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.684 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.684 INFO:teuthology.orchestra.run.smithi040.stdout: "rwoption4": { 2024-04-21T23:28:38.684 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rwoption4", 2024-04-21T23:28:38.684 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.684 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.684 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.684 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.684 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.685 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.685 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.685 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.685 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.685 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.685 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.685 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.685 INFO:teuthology.orchestra.run.smithi040.stdout: "rwoption5": { 2024-04-21T23:28:38.686 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rwoption5", 2024-04-21T23:28:38.686 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.686 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.686 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.686 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.686 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.686 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.690 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.690 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.690 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.690 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.690 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.690 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.691 INFO:teuthology.orchestra.run.smithi040.stdout: "rwoption6": { 2024-04-21T23:28:38.691 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rwoption6", 2024-04-21T23:28:38.691 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.691 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.691 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.691 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:38.691 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.691 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.692 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.692 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.692 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.692 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.692 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.692 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.692 INFO:teuthology.orchestra.run.smithi040.stdout: "rwoption7": { 2024-04-21T23:28:38.693 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rwoption7", 2024-04-21T23:28:38.693 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:38.693 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.693 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.693 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.693 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "1", 2024-04-21T23:28:38.693 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "42", 2024-04-21T23:28:38.693 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.694 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.694 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.694 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.694 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.694 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.695 INFO:teuthology.orchestra.run.smithi040.stdout: "testkey": { 2024-04-21T23:28:38.695 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "testkey", 2024-04-21T23:28:38.695 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.695 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.695 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.695 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.695 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.695 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.696 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.696 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.696 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.696 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.696 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.696 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.696 INFO:teuthology.orchestra.run.smithi040.stdout: "testlkey": { 2024-04-21T23:28:38.696 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "testlkey", 2024-04-21T23:28:38.697 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.697 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.697 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.697 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.697 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.697 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.697 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.697 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.697 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.698 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.698 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.698 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.698 INFO:teuthology.orchestra.run.smithi040.stdout: "testnewline": { 2024-04-21T23:28:38.698 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "testnewline", 2024-04-21T23:28:38.698 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.698 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.699 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.699 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.699 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.699 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.699 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.699 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.699 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.699 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.699 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.700 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:38.700 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:38.700 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.700 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:38.700 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "snap_schedule", 2024-04-21T23:28:38.700 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:38.700 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:38.700 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:38.701 INFO:teuthology.orchestra.run.smithi040.stdout: "allow_m_granularity": { 2024-04-21T23:28:38.701 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "allow_m_granularity", 2024-04-21T23:28:38.701 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.701 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.701 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.701 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.701 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.701 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.702 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.702 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "allow minute scheduled snapshots", 2024-04-21T23:28:38.702 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.702 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.702 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.702 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.703 INFO:teuthology.orchestra.run.smithi040.stdout: "dump_on_update": { 2024-04-21T23:28:38.703 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "dump_on_update", 2024-04-21T23:28:38.703 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.703 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.704 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.704 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.704 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.704 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.704 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.704 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "dump database to debug log on update", 2024-04-21T23:28:38.704 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.705 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.705 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.705 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.705 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:38.705 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:38.705 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.706 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.706 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.706 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.706 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.706 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.706 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:38.706 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:38.706 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:38.707 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:38.707 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:38.707 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:38.707 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:38.707 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:38.707 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.707 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.708 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.708 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.708 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.708 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:38.708 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:38.708 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.708 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.708 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.709 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.709 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.709 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.709 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.709 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.709 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.709 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.710 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.710 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.710 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:38.710 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:38.710 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.710 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.710 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.711 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:38.711 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.711 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.711 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:38.711 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:38.711 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:38.711 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:38.711 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:38.712 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:38.712 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:38.712 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:38.712 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.712 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.712 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.712 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.713 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.713 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:38.713 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:38.713 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.713 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.713 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.713 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.713 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.714 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.714 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.714 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.714 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.714 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.714 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.714 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:38.714 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:38.715 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.715 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:38.715 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "stats", 2024-04-21T23:28:38.715 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:38.716 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:38.716 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:38.716 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:38.716 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:38.716 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.717 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.717 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.717 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.717 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.717 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.718 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:38.718 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:38.718 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:38.718 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:38.719 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:38.719 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:38.719 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:38.719 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:38.720 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.720 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.720 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.720 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.720 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.721 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:38.721 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:38.721 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.721 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.722 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.722 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.722 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.722 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.722 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.722 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.722 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.723 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.723 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.723 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.723 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:38.723 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:38.723 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.723 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.723 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.724 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:38.724 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.724 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.724 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:38.724 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:38.724 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:38.724 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:38.724 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:38.725 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:38.725 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:38.725 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:38.725 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.725 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.725 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.725 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.725 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.726 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:38.726 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:38.726 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.726 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.726 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.726 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.726 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.726 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.727 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.727 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.727 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.727 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.727 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.727 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:38.727 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:38.727 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.728 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:38.728 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "status", 2024-04-21T23:28:38.728 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:38.728 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:38.728 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:38.728 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:38.728 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:38.728 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.728 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.729 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.729 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.729 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.729 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.729 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:38.729 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:38.729 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:38.729 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:38.730 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:38.730 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:38.730 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:38.730 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:38.731 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.731 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.731 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.731 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.731 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.732 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:38.732 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:38.732 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.732 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.732 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.732 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.732 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.732 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.732 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.733 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.733 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.733 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.733 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.733 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.733 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:38.733 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:38.733 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.734 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.734 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.734 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:38.734 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.734 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.734 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:38.735 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:38.735 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:38.735 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:38.735 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:38.735 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:38.735 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:38.735 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:38.735 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.736 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.736 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.736 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.736 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.736 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:38.736 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:38.736 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.736 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.737 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.737 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.737 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.737 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.737 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.737 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.737 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.738 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.738 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.738 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:38.738 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:38.738 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.738 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:38.738 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "telegraf", 2024-04-21T23:28:38.738 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:38.739 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:38.739 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:38.739 INFO:teuthology.orchestra.run.smithi040.stdout: "address": { 2024-04-21T23:28:38.739 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "address", 2024-04-21T23:28:38.739 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.739 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.739 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.739 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-21T23:28:38.739 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.740 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.740 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.740 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.740 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.740 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.740 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.740 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.740 INFO:teuthology.orchestra.run.smithi040.stdout: "interval": { 2024-04-21T23:28:38.741 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "interval", 2024-04-21T23:28:38.741 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-21T23:28:38.741 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.741 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.741 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "15", 2024-04-21T23:28:38.741 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.741 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.741 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.742 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.742 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.742 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.742 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.742 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.742 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:38.742 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:38.743 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.743 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.743 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.743 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.743 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.743 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.743 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:38.743 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:38.744 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:38.744 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:38.744 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:38.744 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:38.744 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:38.745 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:38.745 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.745 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.745 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.745 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.745 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.745 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:38.745 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:38.746 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.746 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.746 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.746 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.746 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.746 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.746 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.747 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.747 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.747 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.747 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.747 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.747 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:38.747 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:38.747 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.748 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.748 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.748 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:38.748 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.748 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.748 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:38.748 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:38.748 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:38.749 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:38.749 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:38.749 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:38.749 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:38.749 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:38.749 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.749 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.750 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.750 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.750 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.750 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:38.750 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:38.750 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.750 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.751 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.751 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.751 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.751 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.751 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.751 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.751 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.751 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.752 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.752 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:38.752 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:38.752 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.752 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:38.752 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "telemetry", 2024-04-21T23:28:38.752 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:38.752 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:38.753 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:38.753 INFO:teuthology.orchestra.run.smithi040.stdout: "channel_basic": { 2024-04-21T23:28:38.753 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "channel_basic", 2024-04-21T23:28:38.753 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.753 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.753 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.753 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:38.753 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.754 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.754 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.754 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-21T23:28:38.754 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.754 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.754 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.754 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.754 INFO:teuthology.orchestra.run.smithi040.stdout: "channel_crash": { 2024-04-21T23:28:38.755 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "channel_crash", 2024-04-21T23:28:38.755 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.755 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.755 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.755 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:38.755 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.755 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.756 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.756 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-21T23:28:38.756 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.756 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.756 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.756 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.756 INFO:teuthology.orchestra.run.smithi040.stdout: "channel_device": { 2024-04-21T23:28:38.756 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "channel_device", 2024-04-21T23:28:38.756 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.757 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.757 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.757 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:38.757 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.757 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.757 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.757 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-21T23:28:38.757 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.758 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.758 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.758 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.758 INFO:teuthology.orchestra.run.smithi040.stdout: "channel_ident": { 2024-04-21T23:28:38.758 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "channel_ident", 2024-04-21T23:28:38.758 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.758 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.758 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.759 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.759 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.759 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.759 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.759 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-21T23:28:38.759 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.759 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.759 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.760 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.760 INFO:teuthology.orchestra.run.smithi040.stdout: "channel_perf": { 2024-04-21T23:28:38.760 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "channel_perf", 2024-04-21T23:28:38.760 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.760 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.760 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.760 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.760 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.760 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.761 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.761 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-21T23:28:38.761 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.761 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.761 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.761 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.761 INFO:teuthology.orchestra.run.smithi040.stdout: "contact": { 2024-04-21T23:28:38.761 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "contact", 2024-04-21T23:28:38.762 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.762 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.762 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.762 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.762 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.762 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.762 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.763 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.763 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.763 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.763 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.763 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.763 INFO:teuthology.orchestra.run.smithi040.stdout: "description": { 2024-04-21T23:28:38.763 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "description", 2024-04-21T23:28:38.763 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.763 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.764 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.764 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.764 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.764 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.764 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.764 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.764 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.764 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.765 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.765 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.765 INFO:teuthology.orchestra.run.smithi040.stdout: "device_url": { 2024-04-21T23:28:38.765 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "device_url", 2024-04-21T23:28:38.765 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.766 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.766 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.766 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-21T23:28:38.766 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.766 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.766 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.767 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.767 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.767 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.767 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.767 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.767 INFO:teuthology.orchestra.run.smithi040.stdout: "enabled": { 2024-04-21T23:28:38.767 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "enabled", 2024-04-21T23:28:38.767 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.768 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.768 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.768 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.768 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.768 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.768 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.768 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.768 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.769 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.769 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.769 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.769 INFO:teuthology.orchestra.run.smithi040.stdout: "interval": { 2024-04-21T23:28:38.769 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "interval", 2024-04-21T23:28:38.769 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:38.769 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.769 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.770 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "24", 2024-04-21T23:28:38.770 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "8", 2024-04-21T23:28:38.770 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.770 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.770 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.770 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.770 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.770 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.771 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.771 INFO:teuthology.orchestra.run.smithi040.stdout: "last_opt_revision": { 2024-04-21T23:28:38.771 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "last_opt_revision", 2024-04-21T23:28:38.771 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:38.771 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.771 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.771 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "1", 2024-04-21T23:28:38.771 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.772 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.772 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.772 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.772 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.772 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.772 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.772 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.772 INFO:teuthology.orchestra.run.smithi040.stdout: "leaderboard": { 2024-04-21T23:28:38.772 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "leaderboard", 2024-04-21T23:28:38.773 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.773 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.773 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.773 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.773 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.773 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.773 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.773 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.774 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.774 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.774 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.774 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.774 INFO:teuthology.orchestra.run.smithi040.stdout: "leaderboard_description": { 2024-04-21T23:28:38.774 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "leaderboard_description", 2024-04-21T23:28:38.774 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.774 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.775 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.775 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.775 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:36.306+0000 7f28af496640 1 -- 172.21.15.40:0/291115050 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f28940a2d00 msgr2=0x7f28940a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:28:38.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:36.306+0000 7f28af496640 1 --2- 172.21.15.40:0/291115050 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f28940a2d00 0x7f28940a51c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f28a4005fd0 tx=0x7f28a400b5f0 comp rx=0 tx=0).stop 2024-04-21T23:28:38.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:36.306+0000 7f28af496640 1 -- 172.21.15.40:0/291115050 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28a8105b30 msgr2=0x7f28a81a2e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:28:38.776 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.776 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.776 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.776 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.777 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.777 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.777 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.777 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:38.777 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:38.777 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.777 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.777 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.778 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.778 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.778 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.778 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:38.778 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:38.778 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:38.778 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:38.778 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:38.779 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:38.779 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:38.779 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:38.779 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.779 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.779 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.779 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.780 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.780 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:38.780 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:38.780 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.780 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.780 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.780 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.781 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.781 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.781 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.781 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.781 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.781 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.781 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.781 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.781 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:38.782 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:38.782 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.782 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.782 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.782 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:38.782 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.782 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.782 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:38.783 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:38.783 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:38.783 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:38.783 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:38.783 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:38.783 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:38.783 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:38.783 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.783 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.784 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.784 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.784 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.784 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:38.784 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:38.784 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.784 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.784 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.785 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.785 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.785 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.785 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.785 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.785 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.785 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.785 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.786 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.786 INFO:teuthology.orchestra.run.smithi040.stdout: "organization": { 2024-04-21T23:28:38.786 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "organization", 2024-04-21T23:28:38.786 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.786 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.786 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.786 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.787 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.787 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.787 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.787 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.787 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.787 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.787 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.787 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.788 INFO:teuthology.orchestra.run.smithi040.stdout: "proxy": { 2024-04-21T23:28:38.788 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "proxy", 2024-04-21T23:28:38.788 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.788 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.788 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.788 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.788 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.788 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.788 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.789 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.789 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.789 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.789 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.789 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.789 INFO:teuthology.orchestra.run.smithi040.stdout: "url": { 2024-04-21T23:28:38.789 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "url", 2024-04-21T23:28:38.789 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.790 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.790 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.790 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-21T23:28:38.790 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.790 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.790 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.791 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.791 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.791 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.791 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.792 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:38.792 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:38.792 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.792 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:38.792 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "test_orchestrator", 2024-04-21T23:28:38.792 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:38.792 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:38.792 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:38.793 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:38.793 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:38.793 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.793 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.793 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.793 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.794 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.794 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.794 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:38.794 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:38.794 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:38.794 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:38.794 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:38.794 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:38.795 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:38.795 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:38.795 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.795 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.795 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.795 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.796 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.796 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:38.796 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:38.796 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.797 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.797 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.797 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.797 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.798 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.798 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.798 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.798 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.798 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.798 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.799 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.799 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:38.799 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:38.799 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.799 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.799 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.799 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:38.799 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.800 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.800 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:38.800 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:38.800 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:38.800 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:38.801 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:38.801 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:38.801 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:38.801 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:38.802 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.802 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.802 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.802 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.803 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.803 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:38.803 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:38.803 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.804 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.804 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.804 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.804 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.804 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.804 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.804 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.805 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.805 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.805 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.805 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:38.805 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:38.805 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.805 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:38.805 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "volumes", 2024-04-21T23:28:38.806 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:38.806 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:38.806 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:38.806 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:38.806 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:38.806 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.806 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.806 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.807 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.807 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.807 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.807 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:38.807 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:38.807 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:38.807 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:38.808 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:38.808 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:38.808 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:38.808 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:38.808 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.808 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.808 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.808 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.809 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.809 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:38.809 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:38.809 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.809 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.810 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.810 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.810 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.810 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.810 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.811 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.811 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.811 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.811 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.811 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.811 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:38.811 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:38.811 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.812 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.812 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.812 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:38.812 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.812 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.812 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:38.812 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:38.812 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:38.813 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:38.813 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:38.813 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:38.813 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:38.813 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:38.813 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.813 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.813 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.814 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.814 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.814 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:38.814 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:38.814 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.814 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.814 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.814 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.815 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.815 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.815 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.815 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.815 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.815 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.816 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.816 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.816 INFO:teuthology.orchestra.run.smithi040.stdout: "max_concurrent_clones": { 2024-04-21T23:28:38.816 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "max_concurrent_clones", 2024-04-21T23:28:38.816 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:38.817 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.817 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.817 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "4", 2024-04-21T23:28:38.817 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.817 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.817 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.817 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-21T23:28:38.818 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.818 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.818 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.818 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.818 INFO:teuthology.orchestra.run.smithi040.stdout: "periodic_async_work": { 2024-04-21T23:28:38.818 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "periodic_async_work", 2024-04-21T23:28:38.818 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.819 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.819 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.819 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.819 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.819 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.819 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.819 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Periodically check for async work", 2024-04-21T23:28:38.819 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.819 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.820 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.820 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.820 INFO:teuthology.orchestra.run.smithi040.stdout: "snapshot_clone_delay": { 2024-04-21T23:28:38.820 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "snapshot_clone_delay", 2024-04-21T23:28:38.820 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:38.820 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.820 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.820 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "0", 2024-04-21T23:28:38.821 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.821 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.821 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.821 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-21T23:28:38.822 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.822 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.822 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.822 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.822 INFO:teuthology.orchestra.run.smithi040.stdout: "snapshot_clone_no_wait": { 2024-04-21T23:28:38.822 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "snapshot_clone_no_wait", 2024-04-21T23:28:38.822 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.822 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.823 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.823 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-21T23:28:38.823 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.823 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.823 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.823 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-21T23:28:38.823 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.823 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.824 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.824 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:38.824 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:38.824 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.824 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:38.824 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "zabbix", 2024-04-21T23:28:38.824 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-21T23:28:38.824 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-21T23:28:38.824 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-21T23:28:38.825 INFO:teuthology.orchestra.run.smithi040.stdout: "discovery_interval": { 2024-04-21T23:28:38.825 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "discovery_interval", 2024-04-21T23:28:38.825 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "uint", 2024-04-21T23:28:38.825 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.825 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.825 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "100", 2024-04-21T23:28:38.825 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.825 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.826 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.826 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.826 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.826 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.826 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.827 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.827 INFO:teuthology.orchestra.run.smithi040.stdout: "identifier": { 2024-04-21T23:28:38.827 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "identifier", 2024-04-21T23:28:38.827 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.827 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.827 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.827 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.828 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.828 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.828 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.828 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.828 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.828 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.828 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.828 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.829 INFO:teuthology.orchestra.run.smithi040.stdout: "interval": { 2024-04-21T23:28:38.829 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "interval", 2024-04-21T23:28:38.829 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-21T23:28:38.829 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.830 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.830 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "60", 2024-04-21T23:28:38.830 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.830 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.831 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.831 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.831 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.832 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.832 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.832 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.832 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-21T23:28:38.833 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-21T23:28:38.833 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.833 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.834 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.834 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.834 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.834 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.835 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:38.835 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:38.835 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:38.835 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:38.836 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:38.836 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:38.836 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:38.837 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:38.837 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.837 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.837 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.838 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.838 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.838 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-21T23:28:38.839 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-21T23:28:38.839 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.839 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.839 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.840 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.840 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.840 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.840 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.841 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.841 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.841 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.842 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.842 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.842 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-21T23:28:38.842 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-21T23:28:38.843 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.843 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.843 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.844 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-21T23:28:38.844 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.844 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.844 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-21T23:28:38.845 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-21T23:28:38.845 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-21T23:28:38.845 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-21T23:28:38.845 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-21T23:28:38.846 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-21T23:28:38.846 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-21T23:28:38.846 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:38.847 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.847 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.847 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.847 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.848 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.848 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-21T23:28:38.848 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-21T23:28:38.848 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-21T23:28:38.849 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.849 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-21T23:28:38.849 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-21T23:28:38.849 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.850 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.850 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.850 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.850 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.850 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.850 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.850 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.850 INFO:teuthology.orchestra.run.smithi040.stdout: "zabbix_host": { 2024-04-21T23:28:38.851 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "zabbix_host", 2024-04-21T23:28:38.851 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.851 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.851 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.851 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-21T23:28:38.851 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.851 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.852 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.852 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.852 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.852 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.852 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.852 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.852 INFO:teuthology.orchestra.run.smithi040.stdout: "zabbix_port": { 2024-04-21T23:28:38.852 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "zabbix_port", 2024-04-21T23:28:38.852 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-21T23:28:38.853 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.853 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.853 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10051", 2024-04-21T23:28:38.853 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.853 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.853 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.853 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.853 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.854 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.854 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.854 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.854 INFO:teuthology.orchestra.run.smithi040.stdout: "zabbix_sender": { 2024-04-21T23:28:38.854 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "zabbix_sender", 2024-04-21T23:28:38.854 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-21T23:28:38.854 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-21T23:28:38.855 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-21T23:28:38.855 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-21T23:28:38.855 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-21T23:28:38.855 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-21T23:28:38.855 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-21T23:28:38.855 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-21T23:28:38.855 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-21T23:28:38.855 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-21T23:28:38.855 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-21T23:28:38.856 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:38.856 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:38.856 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:38.856 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:38.856 INFO:teuthology.orchestra.run.smithi040.stdout: "services": { 2024-04-21T23:28:38.857 INFO:teuthology.orchestra.run.smithi040.stdout: "dashboard": "https://172.21.15.40:8443/" 2024-04-21T23:28:38.857 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.857 INFO:teuthology.orchestra.run.smithi040.stdout: "always_on_modules": { 2024-04-21T23:28:38.857 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus": [ 2024-04-21T23:28:38.858 INFO:teuthology.orchestra.run.smithi040.stdout: "balancer", 2024-04-21T23:28:38.858 INFO:teuthology.orchestra.run.smithi040.stdout: "crash", 2024-04-21T23:28:38.858 INFO:teuthology.orchestra.run.smithi040.stdout: "devicehealth", 2024-04-21T23:28:38.858 INFO:teuthology.orchestra.run.smithi040.stdout: "orchestrator", 2024-04-21T23:28:38.859 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_autoscaler", 2024-04-21T23:28:38.859 INFO:teuthology.orchestra.run.smithi040.stdout: "progress", 2024-04-21T23:28:38.859 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_support", 2024-04-21T23:28:38.859 INFO:teuthology.orchestra.run.smithi040.stdout: "status", 2024-04-21T23:28:38.859 INFO:teuthology.orchestra.run.smithi040.stdout: "telemetry", 2024-04-21T23:28:38.859 INFO:teuthology.orchestra.run.smithi040.stdout: "volumes" 2024-04-21T23:28:38.859 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:38.859 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific": [ 2024-04-21T23:28:38.860 INFO:teuthology.orchestra.run.smithi040.stdout: "balancer", 2024-04-21T23:28:38.860 INFO:teuthology.orchestra.run.smithi040.stdout: "crash", 2024-04-21T23:28:38.860 INFO:teuthology.orchestra.run.smithi040.stdout: "devicehealth", 2024-04-21T23:28:38.860 INFO:teuthology.orchestra.run.smithi040.stdout: "orchestrator", 2024-04-21T23:28:38.860 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_autoscaler", 2024-04-21T23:28:38.860 INFO:teuthology.orchestra.run.smithi040.stdout: "progress", 2024-04-21T23:28:38.860 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_support", 2024-04-21T23:28:38.860 INFO:teuthology.orchestra.run.smithi040.stdout: "status", 2024-04-21T23:28:38.860 INFO:teuthology.orchestra.run.smithi040.stdout: "telemetry", 2024-04-21T23:28:38.861 INFO:teuthology.orchestra.run.smithi040.stdout: "volumes" 2024-04-21T23:28:38.861 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:38.861 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy": [ 2024-04-21T23:28:38.861 INFO:teuthology.orchestra.run.smithi040.stdout: "balancer", 2024-04-21T23:28:38.861 INFO:teuthology.orchestra.run.smithi040.stdout: "crash", 2024-04-21T23:28:38.861 INFO:teuthology.orchestra.run.smithi040.stdout: "devicehealth", 2024-04-21T23:28:38.861 INFO:teuthology.orchestra.run.smithi040.stdout: "orchestrator", 2024-04-21T23:28:38.861 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_autoscaler", 2024-04-21T23:28:38.862 INFO:teuthology.orchestra.run.smithi040.stdout: "progress", 2024-04-21T23:28:38.862 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_support", 2024-04-21T23:28:38.862 INFO:teuthology.orchestra.run.smithi040.stdout: "status", 2024-04-21T23:28:38.862 INFO:teuthology.orchestra.run.smithi040.stdout: "telemetry", 2024-04-21T23:28:38.863 INFO:teuthology.orchestra.run.smithi040.stdout: "volumes" 2024-04-21T23:28:38.863 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:38.863 INFO:teuthology.orchestra.run.smithi040.stdout: "reef": [ 2024-04-21T23:28:38.863 INFO:teuthology.orchestra.run.smithi040.stdout: "balancer", 2024-04-21T23:28:38.863 INFO:teuthology.orchestra.run.smithi040.stdout: "crash", 2024-04-21T23:28:38.863 INFO:teuthology.orchestra.run.smithi040.stdout: "devicehealth", 2024-04-21T23:28:38.863 INFO:teuthology.orchestra.run.smithi040.stdout: "orchestrator", 2024-04-21T23:28:38.863 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_autoscaler", 2024-04-21T23:28:38.863 INFO:teuthology.orchestra.run.smithi040.stdout: "progress", 2024-04-21T23:28:38.864 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_support", 2024-04-21T23:28:38.864 INFO:teuthology.orchestra.run.smithi040.stdout: "status", 2024-04-21T23:28:38.864 INFO:teuthology.orchestra.run.smithi040.stdout: "telemetry", 2024-04-21T23:28:38.864 INFO:teuthology.orchestra.run.smithi040.stdout: "volumes" 2024-04-21T23:28:38.864 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-21T23:28:38.864 INFO:teuthology.orchestra.run.smithi040.stdout: "squid": [ 2024-04-21T23:28:38.864 INFO:teuthology.orchestra.run.smithi040.stdout: "balancer", 2024-04-21T23:28:38.864 INFO:teuthology.orchestra.run.smithi040.stdout: "crash", 2024-04-21T23:28:38.865 INFO:teuthology.orchestra.run.smithi040.stdout: "devicehealth", 2024-04-21T23:28:38.865 INFO:teuthology.orchestra.run.smithi040.stdout: "orchestrator", 2024-04-21T23:28:38.865 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_autoscaler", 2024-04-21T23:28:38.865 INFO:teuthology.orchestra.run.smithi040.stdout: "progress", 2024-04-21T23:28:38.865 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_support", 2024-04-21T23:28:38.865 INFO:teuthology.orchestra.run.smithi040.stdout: "status", 2024-04-21T23:28:38.865 INFO:teuthology.orchestra.run.smithi040.stdout: "telemetry", 2024-04-21T23:28:38.865 INFO:teuthology.orchestra.run.smithi040.stdout: "volumes" 2024-04-21T23:28:38.865 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-21T23:28:38.866 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.866 INFO:teuthology.orchestra.run.smithi040.stdout: "last_failure_osd_epoch": 95, 2024-04-21T23:28:38.866 INFO:teuthology.orchestra.run.smithi040.stdout: "active_clients": [ 2024-04-21T23:28:38.866 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:38.866 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "devicehealth", 2024-04-21T23:28:38.866 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-21T23:28:38.867 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:38.867 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-21T23:28:38.867 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:0", 2024-04-21T23:28:38.867 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3433216413 2024-04-21T23:28:38.867 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:38.867 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-21T23:28:38.867 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.867 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:38.867 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "libcephsqlite", 2024-04-21T23:28:38.868 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-21T23:28:38.868 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:38.868 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-21T23:28:38.868 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:0", 2024-04-21T23:28:38.869 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3800836898 2024-04-21T23:28:38.869 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:38.869 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-21T23:28:38.869 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.869 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:38.870 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rbd_support", 2024-04-21T23:28:38.870 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-21T23:28:38.870 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:38.870 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-21T23:28:38.870 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:0", 2024-04-21T23:28:38.870 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4069450104 2024-04-21T23:28:38.870 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:38.870 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-21T23:28:38.871 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.871 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:38.871 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "snap_schedule", 2024-04-21T23:28:38.871 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-21T23:28:38.871 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:38.871 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-21T23:28:38.871 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:0", 2024-04-21T23:28:38.872 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3914710555 2024-04-21T23:28:38.872 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:38.872 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-21T23:28:38.872 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.872 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:38.872 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "snap_schedule", 2024-04-21T23:28:38.872 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-21T23:28:38.873 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:38.873 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-21T23:28:38.873 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:0", 2024-04-21T23:28:38.873 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3625423513 2024-04-21T23:28:38.873 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:38.873 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-21T23:28:38.873 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:38.873 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:38.874 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "volumes", 2024-04-21T23:28:38.874 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-21T23:28:38.874 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-21T23:28:38.874 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-21T23:28:38.874 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:0", 2024-04-21T23:28:38.875 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 2525246267 2024-04-21T23:28:38.875 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:38.875 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-21T23:28:38.875 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:38.875 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-21T23:28:38.876 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-21T23:28:38.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:36.306+0000 7f28af496640 1 --2- 172.21.15.40:0/291115050 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28a8105b30 0x7f28a81a2e70 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f28a002c790 tx=0x7f28a0035ed0 comp rx=0 tx=0).stop 2024-04-21T23:28:38.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:36.307+0000 7f28af496640 1 -- 172.21.15.40:0/291115050 shutdown_connections 2024-04-21T23:28:38.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:36.307+0000 7f28af496640 1 --2- 172.21.15.40:0/291115050 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f28940a2d00 0x7f28940a51c0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:28:38.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:36.307+0000 7f28af496640 1 --2- 172.21.15.40:0/291115050 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28a8105b30 0x7f28a81a2e70 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:28:38.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:36.307+0000 7f28af496640 1 --2- 172.21.15.40:0/291115050 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28a8101130 0x7f28a819eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:28:38.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:36.307+0000 7f28af496640 1 --2- 172.21.15.40:0/291115050 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28a8100810 0x7f28a819e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:28:38.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:36.307+0000 7f28af496640 1 -- 172.21.15.40:0/291115050 >> 172.21.15.40:0/291115050 conn(0x7f28a80777a0 msgr2=0x7f28a81035c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:28:38.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:36.307+0000 7f28af496640 1 -- 172.21.15.40:0/291115050 shutdown_connections 2024-04-21T23:28:38.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:28:36.307+0000 7f28af496640 1 -- 172.21.15.40:0/291115050 wait complete. 2024-04-21T23:28:38.880 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:28:36 smithi111 ceph-mon[31085]: from='client.16323 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:28:38.880 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:28:36 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/3667851487' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T23:28:38.880 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:28:36 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/2420046410' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T23:28:38.880 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:28:36 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/785783122' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T23:28:38.880 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:28:37 smithi111 ceph-mon[31085]: pgmap v317: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-21T23:28:38.881 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:28:37 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:28:38.881 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:28:37 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/291115050' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-21T23:28:38.881 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:28:36 smithi174 ceph-mon[31409]: from='client.16323 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:28:38.881 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:28:36 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/3667851487' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T23:28:38.881 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:28:36 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/2420046410' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T23:28:38.881 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:28:36 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/785783122' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T23:28:38.881 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:28:37 smithi174 ceph-mon[31409]: pgmap v317: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-21T23:28:38.882 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:28:37 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:28:38.882 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:28:37 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/291115050' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-21T23:28:38.888 DEBUG:tasks.check_counter:Getting stats from a 2024-04-21T23:28:38.889 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph daemon mds.a perf dump 2024-04-21T23:28:38.889 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:28:37 smithi040 ceph-mon[27984]: pgmap v317: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-21T23:28:38.890 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:28:37 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:28:38.890 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:28:37 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/291115050' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-21T23:28:39.136 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:28:39.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:28:38 smithi040 ceph-mon[27984]: pgmap v318: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 0 B/s wr, 7 op/s 2024-04-21T23:28:39.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:28:38 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:28:39.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:28:38 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:28:39.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:28:38 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:28:39.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:28:38 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:28:39.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:28:38 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:28:39.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:28:38 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:28:39.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:28:38 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:28:39.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:28:38 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:28:39.184 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:28:38 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:28:39.184 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:28:38 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:28:39.184 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:28:38 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:28:39.184 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:28:38 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:28:39.185 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:28:38 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:28:39.185 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:28:38 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:28:39.185 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:28:38 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:28:39.185 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:28:38 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:28:39.186 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:28:38 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:28:39.186 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:28:38 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:28:39.186 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:28:38 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:28:39.187 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:28:38 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:28:39.187 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:28:38 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:28:39.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:28:38 smithi111 ceph-mon[31085]: pgmap v318: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 0 B/s wr, 7 op/s 2024-04-21T23:28:39.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:28:38 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:28:39.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:28:38 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:28:39.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:28:38 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:28:39.196 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:28:38 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:28:39.196 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:28:38 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:28:39.196 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:28:38 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:28:39.196 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:28:38 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:28:39.197 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:28:38 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:28:39.197 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:28:38 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:28:39.197 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:28:38 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:28:39.197 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:28:38 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:28:39.198 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:28:38 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:28:39.198 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:28:38 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:28:39.198 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:28:38 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:28:39.199 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:28:38 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:28:39.199 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:28:38 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:28:39.199 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:28:38 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:28:39.199 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:28:38 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:28:39.200 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:28:38 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:28:39.200 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:28:38 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:28:39.200 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:28:38 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:28:39.201 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:28:38 smithi174 ceph-mon[31409]: pgmap v318: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 0 B/s wr, 7 op/s 2024-04-21T23:28:39.201 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:28:38 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:28:39.201 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:28:38 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:28:39.202 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:28:38 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:28:39.202 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:28:38 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:28:39.202 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:28:38 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:28:39.202 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:28:38 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:28:39.203 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:28:38 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:28:39.203 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:28:38 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:28:39.203 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:28:38 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:28:39.204 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:28:38 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:28:39.204 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:28:38 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:28:39.204 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:28:38 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:28:39.204 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:28:38 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:28:39.205 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:28:38 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:28:39.205 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:28:38 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:28:39.205 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:28:38 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:28:39.205 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:28:38 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:28:39.206 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:28:38 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:28:39.206 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:28:38 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:28:39.206 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:28:38 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:28:39.207 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:28:38 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' 2024-04-21T23:28:39.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:28:39 smithi040 ceph-mon[27984]: Adjusting osd_memory_target on smithi111 to 176.2M 2024-04-21T23:28:39.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:28:39 smithi040 ceph-mon[27984]: Adjusting osd_memory_target on smithi174 to 176.2M 2024-04-21T23:28:39.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:28:39 smithi040 ceph-mon[27984]: Unable to set osd_memory_target on smithi111 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-04-21T23:28:39.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:28:39 smithi040 ceph-mon[27984]: Unable to set osd_memory_target on smithi174 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-04-21T23:28:39.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:28:39 smithi040 ceph-mon[27984]: Adjusting osd_memory_target on smithi040 to 177.2M 2024-04-21T23:28:39.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:28:39 smithi040 ceph-mon[27984]: Unable to set osd_memory_target on smithi040 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-04-21T23:28:40.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:28:39 smithi174 ceph-mon[31409]: Adjusting osd_memory_target on smithi111 to 176.2M 2024-04-21T23:28:40.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:28:39 smithi174 ceph-mon[31409]: Adjusting osd_memory_target on smithi174 to 176.2M 2024-04-21T23:28:40.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:28:39 smithi174 ceph-mon[31409]: Unable to set osd_memory_target on smithi111 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-04-21T23:28:40.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:28:39 smithi174 ceph-mon[31409]: Unable to set osd_memory_target on smithi174 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-04-21T23:28:40.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:28:39 smithi174 ceph-mon[31409]: Adjusting osd_memory_target on smithi040 to 177.2M 2024-04-21T23:28:40.195 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:28:39 smithi174 ceph-mon[31409]: Unable to set osd_memory_target on smithi040 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-04-21T23:28:40.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:28:39 smithi111 ceph-mon[31085]: Adjusting osd_memory_target on smithi111 to 176.2M 2024-04-21T23:28:40.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:28:39 smithi111 ceph-mon[31085]: Adjusting osd_memory_target on smithi174 to 176.2M 2024-04-21T23:28:40.196 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:28:39 smithi111 ceph-mon[31085]: Unable to set osd_memory_target on smithi111 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-04-21T23:28:40.196 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:28:39 smithi111 ceph-mon[31085]: Unable to set osd_memory_target on smithi174 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-04-21T23:28:40.196 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:28:39 smithi111 ceph-mon[31085]: Adjusting osd_memory_target on smithi040 to 177.2M 2024-04-21T23:28:40.196 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:28:39 smithi111 ceph-mon[31085]: Unable to set osd_memory_target on smithi040 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-04-21T23:28:40.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:28:40 smithi040 ceph-mon[27984]: pgmap v319: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-21T23:28:41.095 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:28:40 smithi174 ceph-mon[31409]: pgmap v319: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-21T23:28:41.095 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:28:40 smithi111 ceph-mon[31085]: pgmap v319: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-21T23:28:41.179 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-21T23:28:41.179 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-0": { 2024-04-21T23:28:41.179 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 4, 2024-04-21T23:28:41.180 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 4, 2024-04-21T23:28:41.180 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 3024, 2024-04-21T23:28:41.180 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 448, 2024-04-21T23:28:41.180 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 4, 2024-04-21T23:28:41.180 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 0, 2024-04-21T23:28:41.180 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.001932758, 2024-04-21T23:28:41.180 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.000036932, 2024-04-21T23:28:41.181 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.000812112, 2024-04-21T23:28:41.181 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T23:28:41.181 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T23:28:41.181 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 3, 2024-04-21T23:28:41.181 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000895171, 2024-04-21T23:28:41.181 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000298390 2024-04-21T23:28:41.181 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:41.181 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-04-21T23:28:41.182 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-21T23:28:41.182 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-21T23:28:41.182 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-21T23:28:41.182 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:41.182 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-04-21T23:28:41.182 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 448 2024-04-21T23:28:41.182 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:41.183 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-1": { 2024-04-21T23:28:41.183 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 220, 2024-04-21T23:28:41.183 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 209, 2024-04-21T23:28:41.183 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 1580448, 2024-04-21T23:28:41.183 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 128064, 2024-04-21T23:28:41.183 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 2, 2024-04-21T23:28:41.183 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 1, 2024-04-21T23:28:41.183 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.043444554, 2024-04-21T23:28:41.184 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.014457552, 2024-04-21T23:28:41.184 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.033018187, 2024-04-21T23:28:41.184 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.004534937, 2024-04-21T23:28:41.184 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T23:28:41.184 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 208, 2024-04-21T23:28:41.184 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.011103393, 2024-04-21T23:28:41.184 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000053381 2024-04-21T23:28:41.184 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:41.185 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-04-21T23:28:41.185 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-21T23:28:41.186 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-21T23:28:41.186 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-21T23:28:41.186 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:41.186 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 1580448, 2024-04-21T23:28:41.186 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 128064 2024-04-21T23:28:41.186 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:41.186 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-2": { 2024-04-21T23:28:41.186 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 2, 2024-04-21T23:28:41.187 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 159, 2024-04-21T23:28:41.187 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 320, 2024-04-21T23:28:41.187 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 183392, 2024-04-21T23:28:41.187 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 5, 2024-04-21T23:28:41.187 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 3, 2024-04-21T23:28:41.187 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.014064544, 2024-04-21T23:28:41.187 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.009404764, 2024-04-21T23:28:41.187 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.002193883, 2024-04-21T23:28:41.188 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T23:28:41.188 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T23:28:41.188 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 156, 2024-04-21T23:28:41.188 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.011940360, 2024-04-21T23:28:41.188 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000076540 2024-04-21T23:28:41.188 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:41.188 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-04-21T23:28:41.189 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-21T23:28:41.189 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-21T23:28:41.189 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-21T23:28:41.189 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:41.189 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-21T23:28:41.189 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 183392 2024-04-21T23:28:41.189 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:41.190 INFO:teuthology.orchestra.run.smithi040.stdout: "cct": { 2024-04-21T23:28:41.190 INFO:teuthology.orchestra.run.smithi040.stdout: "total_workers": 0, 2024-04-21T23:28:41.190 INFO:teuthology.orchestra.run.smithi040.stdout: "unhealthy_workers": 0 2024-04-21T23:28:41.190 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:41.190 INFO:teuthology.orchestra.run.smithi040.stdout: "mempool": { 2024-04-21T23:28:41.191 INFO:teuthology.orchestra.run.smithi040.stdout: "bloom_filter_bytes": 0, 2024-04-21T23:28:41.191 INFO:teuthology.orchestra.run.smithi040.stdout: "bloom_filter_items": 0, 2024-04-21T23:28:41.191 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_alloc_bytes": 0, 2024-04-21T23:28:41.191 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_alloc_items": 0, 2024-04-21T23:28:41.191 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_data_bytes": 0, 2024-04-21T23:28:41.191 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_data_items": 0, 2024-04-21T23:28:41.191 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-21T23:28:41.192 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_onode_items": 0, 2024-04-21T23:28:41.192 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-21T23:28:41.192 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_meta_items": 0, 2024-04-21T23:28:41.192 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_other_bytes": 0, 2024-04-21T23:28:41.192 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_other_items": 0, 2024-04-21T23:28:41.192 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-21T23:28:41.193 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_buffer_items": 0, 2024-04-21T23:28:41.193 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_bytes": 0, 2024-04-21T23:28:41.193 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_items": 0, 2024-04-21T23:28:41.193 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_blob_bytes": 0, 2024-04-21T23:28:41.193 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_blob_items": 0, 2024-04-21T23:28:41.193 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-21T23:28:41.193 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_shared_blob_items": 0, 2024-04-21T23:28:41.194 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-21T23:28:41.194 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_inline_bl_items": 0, 2024-04-21T23:28:41.194 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_bytes": 0, 2024-04-21T23:28:41.194 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_items": 0, 2024-04-21T23:28:41.194 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_txc_bytes": 0, 2024-04-21T23:28:41.195 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_txc_items": 0, 2024-04-21T23:28:41.195 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-21T23:28:41.195 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_deferred_items": 0, 2024-04-21T23:28:41.195 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_bytes": 0, 2024-04-21T23:28:41.195 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_items": 0, 2024-04-21T23:28:41.195 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_bytes": 0, 2024-04-21T23:28:41.195 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_items": 0, 2024-04-21T23:28:41.196 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_reader_bytes": 0, 2024-04-21T23:28:41.196 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_reader_items": 0, 2024-04-21T23:28:41.196 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_writer_bytes": 0, 2024-04-21T23:28:41.196 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_writer_items": 0, 2024-04-21T23:28:41.196 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_anon_bytes": 149017, 2024-04-21T23:28:41.197 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_anon_items": 31, 2024-04-21T23:28:41.197 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_meta_bytes": 0, 2024-04-21T23:28:41.197 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_meta_items": 0, 2024-04-21T23:28:41.197 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_bytes": 0, 2024-04-21T23:28:41.197 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_items": 0, 2024-04-21T23:28:41.197 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mapbl_bytes": 0, 2024-04-21T23:28:41.197 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mapbl_items": 0, 2024-04-21T23:28:41.198 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pglog_bytes": 0, 2024-04-21T23:28:41.198 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pglog_items": 0, 2024-04-21T23:28:41.198 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_bytes": 0, 2024-04-21T23:28:41.198 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_items": 0, 2024-04-21T23:28:41.198 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_mapping_bytes": 0, 2024-04-21T23:28:41.198 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_mapping_items": 0, 2024-04-21T23:28:41.199 INFO:teuthology.orchestra.run.smithi040.stdout: "pgmap_bytes": 0, 2024-04-21T23:28:41.199 INFO:teuthology.orchestra.run.smithi040.stdout: "pgmap_items": 0, 2024-04-21T23:28:41.199 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_co_bytes": 928, 2024-04-21T23:28:41.199 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_co_items": 2, 2024-04-21T23:28:41.199 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_1_bytes": 0, 2024-04-21T23:28:41.199 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_1_items": 0, 2024-04-21T23:28:41.200 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_2_bytes": 0, 2024-04-21T23:28:41.200 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_2_items": 0 2024-04-21T23:28:41.200 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:41.200 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-21T23:28:41.200 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-04-21T23:28:41.200 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 104857600, 2024-04-21T23:28:41.201 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-04-21T23:28:41.201 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 222, 2024-04-21T23:28:41.201 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 1549460, 2024-04-21T23:28:41.201 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-04-21T23:28:41.201 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 222, 2024-04-21T23:28:41.202 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 0, 2024-04-21T23:28:41.202 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 0, 2024-04-21T23:28:41.202 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 222, 2024-04-21T23:28:41.202 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 1549460, 2024-04-21T23:28:41.202 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-04-21T23:28:41.202 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-21T23:28:41.203 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-21T23:28:41.203 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-21T23:28:41.203 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:41.203 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:41.203 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-21T23:28:41.941 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.a 2024-04-21T23:28:41.941 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.a 2024-04-21T23:28:41.941 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.a 2024-04-21T23:28:41.941 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.a 2024-04-21T23:28:41.941 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.a 2024-04-21T23:28:41.942 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.a 2024-04-21T23:28:41.942 DEBUG:tasks.check_counter:Getting stats from d 2024-04-21T23:28:41.942 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph daemon mds.d perf dump 2024-04-21T23:28:42.191 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:28:42.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:28:42 smithi040 ceph-mon[27984]: pgmap v320: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 0 B/s wr, 7 op/s 2024-04-21T23:28:43.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:28:42 smithi174 ceph-mon[31409]: pgmap v320: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 0 B/s wr, 7 op/s 2024-04-21T23:28:43.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:28:42 smithi111 ceph-mon[31085]: pgmap v320: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 0 B/s wr, 7 op/s 2024-04-21T23:28:44.117 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-21T23:28:44.118 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-0": { 2024-04-21T23:28:44.118 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 226, 2024-04-21T23:28:44.118 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 215, 2024-04-21T23:28:44.118 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 1582080, 2024-04-21T23:28:44.118 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 131904, 2024-04-21T23:28:44.118 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 4, 2024-04-21T23:28:44.119 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 0, 2024-04-21T23:28:44.119 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.047898436, 2024-04-21T23:28:44.119 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.013659090, 2024-04-21T23:28:44.119 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.037277820, 2024-04-21T23:28:44.119 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.004460240, 2024-04-21T23:28:44.119 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T23:28:44.119 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 214, 2024-04-21T23:28:44.120 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.011022142, 2024-04-21T23:28:44.120 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000051505 2024-04-21T23:28:44.120 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:44.120 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-04-21T23:28:44.120 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-21T23:28:44.120 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-21T23:28:44.120 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-21T23:28:44.121 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:44.121 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 1582080, 2024-04-21T23:28:44.121 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 131904 2024-04-21T23:28:44.121 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:44.121 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-1": { 2024-04-21T23:28:44.121 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 6, 2024-04-21T23:28:44.122 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 165, 2024-04-21T23:28:44.122 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 3344, 2024-04-21T23:28:44.122 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 184544, 2024-04-21T23:28:44.122 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 5, 2024-04-21T23:28:44.122 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 3, 2024-04-21T23:28:44.123 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.015436899, 2024-04-21T23:28:44.123 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.010504270, 2024-04-21T23:28:44.123 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.002295461, 2024-04-21T23:28:44.123 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T23:28:44.123 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T23:28:44.123 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 163, 2024-04-21T23:28:44.123 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.011249271, 2024-04-21T23:28:44.124 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000069013 2024-04-21T23:28:44.124 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:44.124 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-04-21T23:28:44.124 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-21T23:28:44.124 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-21T23:28:44.125 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-21T23:28:44.125 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:44.125 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 3344, 2024-04-21T23:28:44.125 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 184544 2024-04-21T23:28:44.125 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:44.126 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-2": { 2024-04-21T23:28:44.126 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 0, 2024-04-21T23:28:44.126 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 2, 2024-04-21T23:28:44.126 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 0, 2024-04-21T23:28:44.126 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 0, 2024-04-21T23:28:44.127 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 2, 2024-04-21T23:28:44.127 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 1, 2024-04-21T23:28:44.127 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.000349982, 2024-04-21T23:28:44.127 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.000000000, 2024-04-21T23:28:44.127 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.000263705, 2024-04-21T23:28:44.127 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T23:28:44.127 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T23:28:44.128 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-21T23:28:44.128 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-21T23:28:44.128 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-21T23:28:44.128 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:44.128 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-04-21T23:28:44.129 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-21T23:28:44.129 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-21T23:28:44.129 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-21T23:28:44.129 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:44.129 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-21T23:28:44.129 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 0 2024-04-21T23:28:44.129 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:44.130 INFO:teuthology.orchestra.run.smithi040.stdout: "cct": { 2024-04-21T23:28:44.130 INFO:teuthology.orchestra.run.smithi040.stdout: "total_workers": 0, 2024-04-21T23:28:44.130 INFO:teuthology.orchestra.run.smithi040.stdout: "unhealthy_workers": 0 2024-04-21T23:28:44.130 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:44.130 INFO:teuthology.orchestra.run.smithi040.stdout: "mempool": { 2024-04-21T23:28:44.130 INFO:teuthology.orchestra.run.smithi040.stdout: "bloom_filter_bytes": 0, 2024-04-21T23:28:44.131 INFO:teuthology.orchestra.run.smithi040.stdout: "bloom_filter_items": 0, 2024-04-21T23:28:44.131 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_alloc_bytes": 0, 2024-04-21T23:28:44.131 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_alloc_items": 0, 2024-04-21T23:28:44.131 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_data_bytes": 0, 2024-04-21T23:28:44.131 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_data_items": 0, 2024-04-21T23:28:44.131 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-21T23:28:44.131 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_onode_items": 0, 2024-04-21T23:28:44.131 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-21T23:28:44.132 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_meta_items": 0, 2024-04-21T23:28:44.132 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_other_bytes": 0, 2024-04-21T23:28:44.132 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_other_items": 0, 2024-04-21T23:28:44.132 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-21T23:28:44.132 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_buffer_items": 0, 2024-04-21T23:28:44.133 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_bytes": 0, 2024-04-21T23:28:44.133 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_items": 0, 2024-04-21T23:28:44.133 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_blob_bytes": 0, 2024-04-21T23:28:44.133 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_blob_items": 0, 2024-04-21T23:28:44.133 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-21T23:28:44.133 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_shared_blob_items": 0, 2024-04-21T23:28:44.133 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-21T23:28:44.133 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_inline_bl_items": 0, 2024-04-21T23:28:44.134 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_bytes": 0, 2024-04-21T23:28:44.134 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_items": 0, 2024-04-21T23:28:44.134 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_txc_bytes": 0, 2024-04-21T23:28:44.134 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_txc_items": 0, 2024-04-21T23:28:44.134 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-21T23:28:44.134 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_deferred_items": 0, 2024-04-21T23:28:44.134 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_bytes": 0, 2024-04-21T23:28:44.135 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_items": 0, 2024-04-21T23:28:44.135 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_bytes": 0, 2024-04-21T23:28:44.135 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_items": 0, 2024-04-21T23:28:44.135 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_reader_bytes": 0, 2024-04-21T23:28:44.135 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_reader_items": 0, 2024-04-21T23:28:44.135 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_writer_bytes": 0, 2024-04-21T23:28:44.135 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_writer_items": 0, 2024-04-21T23:28:44.136 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_anon_bytes": 148977, 2024-04-21T23:28:44.136 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_anon_items": 30, 2024-04-21T23:28:44.136 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_meta_bytes": 0, 2024-04-21T23:28:44.136 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_meta_items": 0, 2024-04-21T23:28:44.136 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_bytes": 0, 2024-04-21T23:28:44.136 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_items": 0, 2024-04-21T23:28:44.136 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mapbl_bytes": 0, 2024-04-21T23:28:44.136 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mapbl_items": 0, 2024-04-21T23:28:44.137 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pglog_bytes": 0, 2024-04-21T23:28:44.137 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pglog_items": 0, 2024-04-21T23:28:44.137 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_bytes": 0, 2024-04-21T23:28:44.137 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_items": 0, 2024-04-21T23:28:44.137 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_mapping_bytes": 0, 2024-04-21T23:28:44.137 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_mapping_items": 0, 2024-04-21T23:28:44.138 INFO:teuthology.orchestra.run.smithi040.stdout: "pgmap_bytes": 0, 2024-04-21T23:28:44.138 INFO:teuthology.orchestra.run.smithi040.stdout: "pgmap_items": 0, 2024-04-21T23:28:44.138 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_co_bytes": 928, 2024-04-21T23:28:44.138 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_co_items": 2, 2024-04-21T23:28:44.138 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_1_bytes": 0, 2024-04-21T23:28:44.138 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_1_items": 0, 2024-04-21T23:28:44.138 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_2_bytes": 0, 2024-04-21T23:28:44.139 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_2_items": 0 2024-04-21T23:28:44.139 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:44.139 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-21T23:28:44.139 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-04-21T23:28:44.139 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 104857600, 2024-04-21T23:28:44.139 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-04-21T23:28:44.139 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 228, 2024-04-21T23:28:44.140 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 1550240, 2024-04-21T23:28:44.140 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-04-21T23:28:44.140 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 228, 2024-04-21T23:28:44.140 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 0, 2024-04-21T23:28:44.140 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 0, 2024-04-21T23:28:44.140 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 228, 2024-04-21T23:28:44.140 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 1550240, 2024-04-21T23:28:44.140 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-04-21T23:28:44.141 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-21T23:28:44.141 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-21T23:28:44.141 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-21T23:28:44.141 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:44.141 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:44.141 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-21T23:28:44.770 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-04-21T23:28:44.770 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-04-21T23:28:44.771 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2024-04-21T23:28:44.771 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2024-04-21T23:28:44.771 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2024-04-21T23:28:44.771 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.d 2024-04-21T23:28:44.771 DEBUG:tasks.check_counter:Getting stats from g 2024-04-21T23:28:44.771 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph daemon mds.g perf dump 2024-04-21T23:28:45.018 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:28:45.140 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:28:44 smithi040 ceph-mon[27984]: pgmap v321: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 0 B/s wr, 7 op/s 2024-04-21T23:28:45.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:28:44 smithi174 ceph-mon[31409]: pgmap v321: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 0 B/s wr, 7 op/s 2024-04-21T23:28:45.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:28:44 smithi111 ceph-mon[31085]: pgmap v321: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 0 B/s wr, 7 op/s 2024-04-21T23:28:46.792 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-21T23:28:46.792 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-0": { 2024-04-21T23:28:46.792 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 0, 2024-04-21T23:28:46.793 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 2, 2024-04-21T23:28:46.793 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 0, 2024-04-21T23:28:46.793 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 0, 2024-04-21T23:28:46.793 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 4, 2024-04-21T23:28:46.793 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 0, 2024-04-21T23:28:46.794 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.001314944, 2024-04-21T23:28:46.794 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.000000000, 2024-04-21T23:28:46.794 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.000201071, 2024-04-21T23:28:46.794 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T23:28:46.794 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T23:28:46.794 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-21T23:28:46.795 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-21T23:28:46.795 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-21T23:28:46.795 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:46.795 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-04-21T23:28:46.796 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-21T23:28:46.796 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-21T23:28:46.796 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-21T23:28:46.796 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:46.796 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-21T23:28:46.797 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 0 2024-04-21T23:28:46.797 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:46.797 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-1": { 2024-04-21T23:28:46.797 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 6, 2024-04-21T23:28:46.797 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 163, 2024-04-21T23:28:46.797 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 3344, 2024-04-21T23:28:46.798 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 184192, 2024-04-21T23:28:46.798 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 5, 2024-04-21T23:28:46.798 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 3, 2024-04-21T23:28:46.799 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.015373177, 2024-04-21T23:28:46.799 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.010338305, 2024-04-21T23:28:46.799 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.002409994, 2024-04-21T23:28:46.799 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T23:28:46.799 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T23:28:46.799 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 161, 2024-04-21T23:28:46.800 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.011092212, 2024-04-21T23:28:46.800 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000068895 2024-04-21T23:28:46.800 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:46.800 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-04-21T23:28:46.800 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-21T23:28:46.801 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-21T23:28:46.801 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-21T23:28:46.801 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:46.801 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 3344, 2024-04-21T23:28:46.801 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 184192 2024-04-21T23:28:46.802 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:46.802 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-2": { 2024-04-21T23:28:46.802 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 222, 2024-04-21T23:28:46.802 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 211, 2024-04-21T23:28:46.802 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 1580992, 2024-04-21T23:28:46.802 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 129344, 2024-04-21T23:28:46.803 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 2, 2024-04-21T23:28:46.803 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 1, 2024-04-21T23:28:46.803 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.048092201, 2024-04-21T23:28:46.803 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.014100146, 2024-04-21T23:28:46.803 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.039566206, 2024-04-21T23:28:46.803 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.004527746, 2024-04-21T23:28:46.803 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T23:28:46.803 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 210, 2024-04-21T23:28:46.804 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.009972654, 2024-04-21T23:28:46.804 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000047488 2024-04-21T23:28:46.804 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:46.804 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-04-21T23:28:46.804 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-21T23:28:46.804 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-21T23:28:46.805 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-21T23:28:46.805 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:46.805 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 1580992, 2024-04-21T23:28:46.805 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 129344 2024-04-21T23:28:46.805 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:46.805 INFO:teuthology.orchestra.run.smithi040.stdout: "cct": { 2024-04-21T23:28:46.805 INFO:teuthology.orchestra.run.smithi040.stdout: "total_workers": 0, 2024-04-21T23:28:46.806 INFO:teuthology.orchestra.run.smithi040.stdout: "unhealthy_workers": 0 2024-04-21T23:28:46.806 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:46.806 INFO:teuthology.orchestra.run.smithi040.stdout: "mempool": { 2024-04-21T23:28:46.806 INFO:teuthology.orchestra.run.smithi040.stdout: "bloom_filter_bytes": 0, 2024-04-21T23:28:46.807 INFO:teuthology.orchestra.run.smithi040.stdout: "bloom_filter_items": 0, 2024-04-21T23:28:46.807 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_alloc_bytes": 0, 2024-04-21T23:28:46.807 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_alloc_items": 0, 2024-04-21T23:28:46.807 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_data_bytes": 0, 2024-04-21T23:28:46.808 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_data_items": 0, 2024-04-21T23:28:46.808 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-21T23:28:46.808 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_onode_items": 0, 2024-04-21T23:28:46.808 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-21T23:28:46.808 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_meta_items": 0, 2024-04-21T23:28:46.809 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_other_bytes": 0, 2024-04-21T23:28:46.809 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_other_items": 0, 2024-04-21T23:28:46.809 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-21T23:28:46.809 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_buffer_items": 0, 2024-04-21T23:28:46.809 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_bytes": 0, 2024-04-21T23:28:46.810 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_items": 0, 2024-04-21T23:28:46.810 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_blob_bytes": 0, 2024-04-21T23:28:46.810 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_blob_items": 0, 2024-04-21T23:28:46.810 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-21T23:28:46.811 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_shared_blob_items": 0, 2024-04-21T23:28:46.811 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-21T23:28:46.811 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_inline_bl_items": 0, 2024-04-21T23:28:46.811 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_bytes": 0, 2024-04-21T23:28:46.811 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_items": 0, 2024-04-21T23:28:46.812 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_txc_bytes": 0, 2024-04-21T23:28:46.812 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_txc_items": 0, 2024-04-21T23:28:46.812 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-21T23:28:46.812 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_deferred_items": 0, 2024-04-21T23:28:46.812 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_bytes": 0, 2024-04-21T23:28:46.813 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_items": 0, 2024-04-21T23:28:46.813 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_bytes": 0, 2024-04-21T23:28:46.813 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_items": 0, 2024-04-21T23:28:46.813 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_reader_bytes": 0, 2024-04-21T23:28:46.814 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_reader_items": 0, 2024-04-21T23:28:46.814 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_writer_bytes": 0, 2024-04-21T23:28:46.814 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_writer_items": 0, 2024-04-21T23:28:46.815 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_anon_bytes": 149017, 2024-04-21T23:28:46.815 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_anon_items": 31, 2024-04-21T23:28:46.815 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_meta_bytes": 0, 2024-04-21T23:28:46.815 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_meta_items": 0, 2024-04-21T23:28:46.815 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_bytes": 0, 2024-04-21T23:28:46.816 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_items": 0, 2024-04-21T23:28:46.816 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mapbl_bytes": 0, 2024-04-21T23:28:46.816 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mapbl_items": 0, 2024-04-21T23:28:46.816 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pglog_bytes": 0, 2024-04-21T23:28:46.816 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pglog_items": 0, 2024-04-21T23:28:46.817 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_bytes": 0, 2024-04-21T23:28:46.817 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_items": 0, 2024-04-21T23:28:46.817 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_mapping_bytes": 0, 2024-04-21T23:28:46.817 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_mapping_items": 0, 2024-04-21T23:28:46.817 INFO:teuthology.orchestra.run.smithi040.stdout: "pgmap_bytes": 0, 2024-04-21T23:28:46.818 INFO:teuthology.orchestra.run.smithi040.stdout: "pgmap_items": 0, 2024-04-21T23:28:46.818 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_co_bytes": 928, 2024-04-21T23:28:46.818 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_co_items": 2, 2024-04-21T23:28:46.818 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_1_bytes": 0, 2024-04-21T23:28:46.819 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_1_items": 0, 2024-04-21T23:28:46.819 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_2_bytes": 0, 2024-04-21T23:28:46.819 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_2_items": 0 2024-04-21T23:28:46.819 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:46.819 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-21T23:28:46.820 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-04-21T23:28:46.820 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 104857600, 2024-04-21T23:28:46.820 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-04-21T23:28:46.820 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 224, 2024-04-21T23:28:46.821 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 1549720, 2024-04-21T23:28:46.821 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-04-21T23:28:46.821 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 224, 2024-04-21T23:28:46.821 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 0, 2024-04-21T23:28:46.821 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 0, 2024-04-21T23:28:46.822 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 224, 2024-04-21T23:28:46.822 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 1549720, 2024-04-21T23:28:46.822 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-04-21T23:28:46.822 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-21T23:28:46.823 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-21T23:28:46.823 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-21T23:28:46.823 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:46.823 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:46.823 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-21T23:28:47.167 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:28:46 smithi040 ceph-mon[27984]: pgmap v322: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-21T23:28:47.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:28:46 smithi174 ceph-mon[31409]: pgmap v322: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-21T23:28:47.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:28:46 smithi111 ceph-mon[31085]: pgmap v322: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-21T23:28:47.517 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.g 2024-04-21T23:28:47.517 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.g 2024-04-21T23:28:47.517 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.g 2024-04-21T23:28:47.517 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.g 2024-04-21T23:28:47.517 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.g 2024-04-21T23:28:47.517 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.g 2024-04-21T23:28:47.518 DEBUG:tasks.check_counter:Getting stats from j 2024-04-21T23:28:47.518 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph daemon mds.j perf dump 2024-04-21T23:28:47.766 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:28:48.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:28:48 smithi040 ceph-mon[27984]: pgmap v323: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:28:49.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:28:48 smithi174 ceph-mon[31409]: pgmap v323: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:28:49.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:28:48 smithi111 ceph-mon[31085]: pgmap v323: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:28:49.444 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-21T23:28:49.445 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-0": { 2024-04-21T23:28:49.445 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 4, 2024-04-21T23:28:49.445 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 4, 2024-04-21T23:28:49.445 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 3024, 2024-04-21T23:28:49.445 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 448, 2024-04-21T23:28:49.445 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 4, 2024-04-21T23:28:49.445 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 0, 2024-04-21T23:28:49.446 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.001698793, 2024-04-21T23:28:49.446 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.000045299, 2024-04-21T23:28:49.446 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.000724918, 2024-04-21T23:28:49.446 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T23:28:49.446 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T23:28:49.446 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 3, 2024-04-21T23:28:49.446 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000752929, 2024-04-21T23:28:49.446 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000250976 2024-04-21T23:28:49.447 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:49.447 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-04-21T23:28:49.447 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-21T23:28:49.447 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-21T23:28:49.447 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-21T23:28:49.447 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:49.447 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-04-21T23:28:49.447 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 448 2024-04-21T23:28:49.447 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:49.448 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-1": { 2024-04-21T23:28:49.448 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 2, 2024-04-21T23:28:49.448 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 166, 2024-04-21T23:28:49.448 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 320, 2024-04-21T23:28:49.448 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 184624, 2024-04-21T23:28:49.448 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 5, 2024-04-21T23:28:49.448 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 3, 2024-04-21T23:28:49.448 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.014942014, 2024-04-21T23:28:49.448 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.010656363, 2024-04-21T23:28:49.449 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.001756389, 2024-04-21T23:28:49.449 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T23:28:49.449 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T23:28:49.449 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 163, 2024-04-21T23:28:49.449 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.010002767, 2024-04-21T23:28:49.449 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000061366 2024-04-21T23:28:49.449 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:49.449 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-04-21T23:28:49.449 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-21T23:28:49.450 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-21T23:28:49.450 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-21T23:28:49.450 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:49.450 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-21T23:28:49.450 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 184624 2024-04-21T23:28:49.450 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:49.450 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-2": { 2024-04-21T23:28:49.450 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 228, 2024-04-21T23:28:49.451 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 217, 2024-04-21T23:28:49.451 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 1582624, 2024-04-21T23:28:49.451 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 133184, 2024-04-21T23:28:49.451 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 2, 2024-04-21T23:28:49.451 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 1, 2024-04-21T23:28:49.451 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.046476989, 2024-04-21T23:28:49.451 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.014371280, 2024-04-21T23:28:49.451 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.036354728, 2024-04-21T23:28:49.451 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.004698258, 2024-04-21T23:28:49.452 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T23:28:49.452 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 216, 2024-04-21T23:28:49.452 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.010479662, 2024-04-21T23:28:49.452 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000048516 2024-04-21T23:28:49.452 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:49.452 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-04-21T23:28:49.452 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-21T23:28:49.452 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-21T23:28:49.453 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-21T23:28:49.453 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:49.453 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 1582624, 2024-04-21T23:28:49.453 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 133184 2024-04-21T23:28:49.453 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:49.453 INFO:teuthology.orchestra.run.smithi040.stdout: "cct": { 2024-04-21T23:28:49.453 INFO:teuthology.orchestra.run.smithi040.stdout: "total_workers": 0, 2024-04-21T23:28:49.453 INFO:teuthology.orchestra.run.smithi040.stdout: "unhealthy_workers": 0 2024-04-21T23:28:49.454 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:49.454 INFO:teuthology.orchestra.run.smithi040.stdout: "mempool": { 2024-04-21T23:28:49.454 INFO:teuthology.orchestra.run.smithi040.stdout: "bloom_filter_bytes": 0, 2024-04-21T23:28:49.454 INFO:teuthology.orchestra.run.smithi040.stdout: "bloom_filter_items": 0, 2024-04-21T23:28:49.454 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_alloc_bytes": 0, 2024-04-21T23:28:49.454 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_alloc_items": 0, 2024-04-21T23:28:49.454 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_data_bytes": 0, 2024-04-21T23:28:49.455 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_data_items": 0, 2024-04-21T23:28:49.455 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-21T23:28:49.455 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_onode_items": 0, 2024-04-21T23:28:49.455 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-21T23:28:49.455 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_meta_items": 0, 2024-04-21T23:28:49.455 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_other_bytes": 0, 2024-04-21T23:28:49.455 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_other_items": 0, 2024-04-21T23:28:49.455 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-21T23:28:49.455 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_buffer_items": 0, 2024-04-21T23:28:49.456 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_bytes": 0, 2024-04-21T23:28:49.456 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_items": 0, 2024-04-21T23:28:49.456 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_blob_bytes": 0, 2024-04-21T23:28:49.456 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_blob_items": 0, 2024-04-21T23:28:49.456 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-21T23:28:49.456 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_shared_blob_items": 0, 2024-04-21T23:28:49.456 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-21T23:28:49.456 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_inline_bl_items": 0, 2024-04-21T23:28:49.457 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_bytes": 0, 2024-04-21T23:28:49.457 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_items": 0, 2024-04-21T23:28:49.457 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_txc_bytes": 0, 2024-04-21T23:28:49.457 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_txc_items": 0, 2024-04-21T23:28:49.457 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-21T23:28:49.457 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_deferred_items": 0, 2024-04-21T23:28:49.457 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_bytes": 0, 2024-04-21T23:28:49.457 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_items": 0, 2024-04-21T23:28:49.457 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_bytes": 0, 2024-04-21T23:28:49.458 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_items": 0, 2024-04-21T23:28:49.458 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_reader_bytes": 0, 2024-04-21T23:28:49.458 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_reader_items": 0, 2024-04-21T23:28:49.458 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_writer_bytes": 0, 2024-04-21T23:28:49.458 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_writer_items": 0, 2024-04-21T23:28:49.458 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_anon_bytes": 148754, 2024-04-21T23:28:49.458 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_anon_items": 31, 2024-04-21T23:28:49.458 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_meta_bytes": 0, 2024-04-21T23:28:49.459 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_meta_items": 0, 2024-04-21T23:28:49.459 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_bytes": 0, 2024-04-21T23:28:49.459 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_items": 0, 2024-04-21T23:28:49.459 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mapbl_bytes": 0, 2024-04-21T23:28:49.459 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mapbl_items": 0, 2024-04-21T23:28:49.459 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pglog_bytes": 0, 2024-04-21T23:28:49.459 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pglog_items": 0, 2024-04-21T23:28:49.459 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_bytes": 0, 2024-04-21T23:28:49.459 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_items": 0, 2024-04-21T23:28:49.460 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_mapping_bytes": 0, 2024-04-21T23:28:49.460 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_mapping_items": 0, 2024-04-21T23:28:49.460 INFO:teuthology.orchestra.run.smithi040.stdout: "pgmap_bytes": 0, 2024-04-21T23:28:49.460 INFO:teuthology.orchestra.run.smithi040.stdout: "pgmap_items": 0, 2024-04-21T23:28:49.460 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_co_bytes": 928, 2024-04-21T23:28:49.460 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_co_items": 2, 2024-04-21T23:28:49.460 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_1_bytes": 0, 2024-04-21T23:28:49.460 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_1_items": 0, 2024-04-21T23:28:49.460 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_2_bytes": 0, 2024-04-21T23:28:49.461 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_2_items": 0 2024-04-21T23:28:49.461 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-21T23:28:49.461 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-21T23:28:49.461 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-04-21T23:28:49.461 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 104857600, 2024-04-21T23:28:49.461 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-04-21T23:28:49.461 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 230, 2024-04-21T23:28:49.461 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 1550500, 2024-04-21T23:28:49.461 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-04-21T23:28:49.462 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 230, 2024-04-21T23:28:49.462 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 0, 2024-04-21T23:28:49.462 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 0, 2024-04-21T23:28:49.462 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 230, 2024-04-21T23:28:49.462 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 1550500, 2024-04-21T23:28:49.462 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-04-21T23:28:49.462 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-21T23:28:49.462 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-21T23:28:49.463 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-21T23:28:49.463 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:49.463 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-21T23:28:49.463 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-21T23:28:50.165 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-04-21T23:28:50.165 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-04-21T23:28:50.165 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-04-21T23:28:50.165 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-04-21T23:28:50.165 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-04-21T23:28:50.165 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.j 2024-04-21T23:28:50.165 DEBUG:tasks.check_counter:Getting stats from b 2024-04-21T23:28:50.166 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph daemon mds.b perf dump 2024-04-21T23:28:50.417 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.b/config 2024-04-21T23:28:51.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:28:50 smithi040 ceph-mon[27984]: pgmap v324: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:28:51.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:28:50 smithi174 ceph-mon[31409]: pgmap v324: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:28:51.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:28:50 smithi111 ceph-mon[31085]: pgmap v324: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:28:52.382 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-21T23:28:52.383 INFO:teuthology.orchestra.run.smithi111.stdout: "AsyncMessenger::Worker-0": { 2024-04-21T23:28:52.383 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_messages": 2443, 2024-04-21T23:28:52.383 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages": 1601, 2024-04-21T23:28:52.383 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_bytes": 916768, 2024-04-21T23:28:52.383 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_bytes": 7857159, 2024-04-21T23:28:52.383 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_created_connections": 19, 2024-04-21T23:28:52.383 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_active_connections": 4, 2024-04-21T23:28:52.384 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_total_time": 0.254293754, 2024-04-21T23:28:52.384 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_send_time": 0.076228499, 2024-04-21T23:28:52.384 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_recv_time": 0.077720558, 2024-04-21T23:28:52.384 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_fast_dispatch_time": 0.088273469, 2024-04-21T23:28:52.384 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T23:28:52.384 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 1603, 2024-04-21T23:28:52.384 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 1.053904679, 2024-04-21T23:28:52.384 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000657457 2024-04-21T23:28:52.384 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:52.385 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_handle_ack_lat": { 2024-04-21T23:28:52.385 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 1789, 2024-04-21T23:28:52.385 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000339423, 2024-04-21T23:28:52.385 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000189 2024-04-21T23:28:52.385 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:52.385 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_encrypted_bytes": 354816, 2024-04-21T23:28:52.385 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_encrypted_bytes": 87024 2024-04-21T23:28:52.385 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:52.385 INFO:teuthology.orchestra.run.smithi111.stdout: "AsyncMessenger::Worker-1": { 2024-04-21T23:28:52.386 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_messages": 5482, 2024-04-21T23:28:52.386 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages": 13671, 2024-04-21T23:28:52.386 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_bytes": 1806142, 2024-04-21T23:28:52.386 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_bytes": 7783313, 2024-04-21T23:28:52.386 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_created_connections": 57, 2024-04-21T23:28:52.386 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_active_connections": 6, 2024-04-21T23:28:52.386 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_total_time": 0.470036283, 2024-04-21T23:28:52.386 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_send_time": 0.186671394, 2024-04-21T23:28:52.387 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_recv_time": 0.196373387, 2024-04-21T23:28:52.387 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_fast_dispatch_time": 0.075123184, 2024-04-21T23:28:52.387 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T23:28:52.387 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 13668, 2024-04-21T23:28:52.387 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 2.348849426, 2024-04-21T23:28:52.387 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000171850 2024-04-21T23:28:52.387 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:52.387 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_handle_ack_lat": { 2024-04-21T23:28:52.387 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 16657, 2024-04-21T23:28:52.388 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.002291372, 2024-04-21T23:28:52.388 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000137 2024-04-21T23:28:52.388 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:52.388 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_encrypted_bytes": 99312, 2024-04-21T23:28:52.388 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_encrypted_bytes": 484816 2024-04-21T23:28:52.388 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:52.388 INFO:teuthology.orchestra.run.smithi111.stdout: "AsyncMessenger::Worker-2": { 2024-04-21T23:28:52.388 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_messages": 3757, 2024-04-21T23:28:52.388 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages": 2557, 2024-04-21T23:28:52.389 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_bytes": 3196242, 2024-04-21T23:28:52.389 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_bytes": 5066476, 2024-04-21T23:28:52.389 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_created_connections": 14, 2024-04-21T23:28:52.389 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_active_connections": 7, 2024-04-21T23:28:52.389 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_total_time": 0.264331610, 2024-04-21T23:28:52.389 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_send_time": 0.099093058, 2024-04-21T23:28:52.389 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_recv_time": 0.136750367, 2024-04-21T23:28:52.389 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_fast_dispatch_time": 0.051082004, 2024-04-21T23:28:52.389 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T23:28:52.390 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 2553, 2024-04-21T23:28:52.390 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 1.401675981, 2024-04-21T23:28:52.390 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000549030 2024-04-21T23:28:52.390 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:52.390 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_handle_ack_lat": { 2024-04-21T23:28:52.390 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 3722, 2024-04-21T23:28:52.390 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000511378, 2024-04-21T23:28:52.391 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000137 2024-04-21T23:28:52.391 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:52.391 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_encrypted_bytes": 1715392, 2024-04-21T23:28:52.391 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_encrypted_bytes": 148976 2024-04-21T23:28:52.391 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:52.391 INFO:teuthology.orchestra.run.smithi111.stdout: "cct": { 2024-04-21T23:28:52.394 INFO:teuthology.orchestra.run.smithi111.stdout: "total_workers": 1, 2024-04-21T23:28:52.394 INFO:teuthology.orchestra.run.smithi111.stdout: "unhealthy_workers": 0 2024-04-21T23:28:52.394 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:52.394 INFO:teuthology.orchestra.run.smithi111.stdout: "finisher-MDSRank": { 2024-04-21T23:28:52.394 INFO:teuthology.orchestra.run.smithi111.stdout: "queue_len": 0, 2024-04-21T23:28:52.394 INFO:teuthology.orchestra.run.smithi111.stdout: "complete_latency": { 2024-04-21T23:28:52.395 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 683, 2024-04-21T23:28:52.395 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 2.337422024, 2024-04-21T23:28:52.395 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.003422287 2024-04-21T23:28:52.395 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-21T23:28:52.395 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:52.395 INFO:teuthology.orchestra.run.smithi111.stdout: "finisher-PurgeQueue": { 2024-04-21T23:28:52.395 INFO:teuthology.orchestra.run.smithi111.stdout: "queue_len": 0, 2024-04-21T23:28:52.395 INFO:teuthology.orchestra.run.smithi111.stdout: "complete_latency": { 2024-04-21T23:28:52.396 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 60, 2024-04-21T23:28:52.396 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.018339230, 2024-04-21T23:28:52.396 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000305653 2024-04-21T23:28:52.396 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-21T23:28:52.396 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:52.396 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": { 2024-04-21T23:28:52.396 INFO:teuthology.orchestra.run.smithi111.stdout: "request": 3560, 2024-04-21T23:28:52.396 INFO:teuthology.orchestra.run.smithi111.stdout: "reply": 3498, 2024-04-21T23:28:52.397 INFO:teuthology.orchestra.run.smithi111.stdout: "reply_latency": { 2024-04-21T23:28:52.397 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 3498, 2024-04-21T23:28:52.397 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 7.508496124, 2024-04-21T23:28:52.397 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.002146511 2024-04-21T23:28:52.397 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:52.397 INFO:teuthology.orchestra.run.smithi111.stdout: "slow_reply": 0, 2024-04-21T23:28:52.397 INFO:teuthology.orchestra.run.smithi111.stdout: "forward": 62, 2024-04-21T23:28:52.397 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_fetch_complete": 5, 2024-04-21T23:28:52.398 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_fetch_keys": 2, 2024-04-21T23:28:52.398 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_commit": 169, 2024-04-21T23:28:52.398 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_split": 12, 2024-04-21T23:28:52.398 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_merge": 0, 2024-04-21T23:28:52.398 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes": 50, 2024-04-21T23:28:52.398 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_top": 11, 2024-04-21T23:28:52.398 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_bottom": 25, 2024-04-21T23:28:52.398 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_pin_tail": 14, 2024-04-21T23:28:52.399 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_pinned": 27, 2024-04-21T23:28:52.399 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_expired": 3275, 2024-04-21T23:28:52.399 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_with_caps": 6, 2024-04-21T23:28:52.399 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": 7, 2024-04-21T23:28:52.399 INFO:teuthology.orchestra.run.smithi111.stdout: "subtrees": 6, 2024-04-21T23:28:52.399 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse": 3829, 2024-04-21T23:28:52.399 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_hit": 3218, 2024-04-21T23:28:52.400 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_forward": 62, 2024-04-21T23:28:52.400 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_discover": 1, 2024-04-21T23:28:52.400 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_dir_fetch": 0, 2024-04-21T23:28:52.400 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_remote_ino": 0, 2024-04-21T23:28:52.400 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_lock": 4, 2024-04-21T23:28:52.400 INFO:teuthology.orchestra.run.smithi111.stdout: "load_cent": 0, 2024-04-21T23:28:52.401 INFO:teuthology.orchestra.run.smithi111.stdout: "q": 0, 2024-04-21T23:28:52.401 INFO:teuthology.orchestra.run.smithi111.stdout: "exported": 13, 2024-04-21T23:28:52.401 INFO:teuthology.orchestra.run.smithi111.stdout: "exported_inodes": 3214, 2024-04-21T23:28:52.401 INFO:teuthology.orchestra.run.smithi111.stdout: "imported": 1, 2024-04-21T23:28:52.401 INFO:teuthology.orchestra.run.smithi111.stdout: "imported_inodes": 8, 2024-04-21T23:28:52.401 INFO:teuthology.orchestra.run.smithi111.stdout: "openino_dir_fetch": 0, 2024-04-21T23:28:52.401 INFO:teuthology.orchestra.run.smithi111.stdout: "openino_backtrace_fetch": 54, 2024-04-21T23:28:52.401 INFO:teuthology.orchestra.run.smithi111.stdout: "openino_peer_discover": 0, 2024-04-21T23:28:52.402 INFO:teuthology.orchestra.run.smithi111.stdout: "root_rfiles": 3454, 2024-04-21T23:28:52.402 INFO:teuthology.orchestra.run.smithi111.stdout: "root_rbytes": 26625020, 2024-04-21T23:28:52.402 INFO:teuthology.orchestra.run.smithi111.stdout: "root_rsnaps": 6, 2024-04-21T23:28:52.402 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_backtrace_fetch": 54, 2024-04-21T23:28:52.402 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_set_tag": 0, 2024-04-21T23:28:52.402 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_backtrace_repaired": 0, 2024-04-21T23:28:52.402 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_inotable_repaired": 0, 2024-04-21T23:28:52.402 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_dir_inodes": 31, 2024-04-21T23:28:52.402 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_dir_base_inodes": 4, 2024-04-21T23:28:52.403 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_dirfrag_rstats": 27, 2024-04-21T23:28:52.403 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_file_inodes": 23, 2024-04-21T23:28:52.403 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_inode_file_caps": 1, 2024-04-21T23:28:52.403 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_cap_op_revoke": 45, 2024-04-21T23:28:52.403 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_cap_op_grant": 2941, 2024-04-21T23:28:52.403 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_cap_op_trunc": 0, 2024-04-21T23:28:52.403 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-21T23:28:52.403 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-21T23:28:52.404 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_client_caps": 87, 2024-04-21T23:28:52.404 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_client_caps_dirty": 28, 2024-04-21T23:28:52.405 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_client_cap_release": 15, 2024-04-21T23:28:52.405 INFO:teuthology.orchestra.run.smithi111.stdout: "process_request_cap_release": 166 2024-04-21T23:28:52.405 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:52.405 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_cache": { 2024-04-21T23:28:52.405 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_update": 0, 2024-04-21T23:28:52.405 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_update_receipt": 0, 2024-04-21T23:28:52.405 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_try_discover": 0, 2024-04-21T23:28:52.405 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_send_discover": 1, 2024-04-21T23:28:52.406 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_handle_discover": 15, 2024-04-21T23:28:52.406 INFO:teuthology.orchestra.run.smithi111.stdout: "num_strays": 0, 2024-04-21T23:28:52.406 INFO:teuthology.orchestra.run.smithi111.stdout: "num_strays_delayed": 0, 2024-04-21T23:28:52.406 INFO:teuthology.orchestra.run.smithi111.stdout: "num_strays_enqueuing": 0, 2024-04-21T23:28:52.406 INFO:teuthology.orchestra.run.smithi111.stdout: "strays_created": 20, 2024-04-21T23:28:52.406 INFO:teuthology.orchestra.run.smithi111.stdout: "strays_enqueued": 17, 2024-04-21T23:28:52.406 INFO:teuthology.orchestra.run.smithi111.stdout: "strays_reintegrated": 3, 2024-04-21T23:28:52.406 INFO:teuthology.orchestra.run.smithi111.stdout: "strays_migrated": 0, 2024-04-21T23:28:52.407 INFO:teuthology.orchestra.run.smithi111.stdout: "num_recovering_processing": 0, 2024-04-21T23:28:52.407 INFO:teuthology.orchestra.run.smithi111.stdout: "num_recovering_enqueued": 0, 2024-04-21T23:28:52.407 INFO:teuthology.orchestra.run.smithi111.stdout: "num_recovering_prioritized": 0, 2024-04-21T23:28:52.407 INFO:teuthology.orchestra.run.smithi111.stdout: "recovery_started": 0, 2024-04-21T23:28:52.407 INFO:teuthology.orchestra.run.smithi111.stdout: "recovery_completed": 0, 2024-04-21T23:28:52.407 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_quiesce_path": 0, 2024-04-21T23:28:52.407 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_quiesce_inode": 0, 2024-04-21T23:28:52.407 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_enqueue_scrub": 4, 2024-04-21T23:28:52.407 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_exportdir": 13, 2024-04-21T23:28:52.408 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_flush": 0, 2024-04-21T23:28:52.408 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_fragmentdir": 13, 2024-04-21T23:28:52.408 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_fragstats": 0, 2024-04-21T23:28:52.408 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_inodestats": 0 2024-04-21T23:28:52.408 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:52.408 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_log": { 2024-04-21T23:28:52.408 INFO:teuthology.orchestra.run.smithi111.stdout: "evlrg": 0, 2024-04-21T23:28:52.408 INFO:teuthology.orchestra.run.smithi111.stdout: "evadd": 3206, 2024-04-21T23:28:52.408 INFO:teuthology.orchestra.run.smithi111.stdout: "evex": 3146, 2024-04-21T23:28:52.409 INFO:teuthology.orchestra.run.smithi111.stdout: "evtrm": 3146, 2024-04-21T23:28:52.409 INFO:teuthology.orchestra.run.smithi111.stdout: "ev": 60, 2024-04-21T23:28:52.409 INFO:teuthology.orchestra.run.smithi111.stdout: "evexg": 0, 2024-04-21T23:28:52.409 INFO:teuthology.orchestra.run.smithi111.stdout: "evexd": 5, 2024-04-21T23:28:52.409 INFO:teuthology.orchestra.run.smithi111.stdout: "segadd": 14, 2024-04-21T23:28:52.409 INFO:teuthology.orchestra.run.smithi111.stdout: "segex": 13, 2024-04-21T23:28:52.409 INFO:teuthology.orchestra.run.smithi111.stdout: "segtrm": 13, 2024-04-21T23:28:52.409 INFO:teuthology.orchestra.run.smithi111.stdout: "seg": 1, 2024-04-21T23:28:52.409 INFO:teuthology.orchestra.run.smithi111.stdout: "segmjr": 1, 2024-04-21T23:28:52.410 INFO:teuthology.orchestra.run.smithi111.stdout: "segexg": 0, 2024-04-21T23:28:52.410 INFO:teuthology.orchestra.run.smithi111.stdout: "segexd": 1, 2024-04-21T23:28:52.410 INFO:teuthology.orchestra.run.smithi111.stdout: "expos": 10953535, 2024-04-21T23:28:52.410 INFO:teuthology.orchestra.run.smithi111.stdout: "wrpos": 11089950, 2024-04-21T23:28:52.410 INFO:teuthology.orchestra.run.smithi111.stdout: "rdpos": 0, 2024-04-21T23:28:52.410 INFO:teuthology.orchestra.run.smithi111.stdout: "jlat": { 2024-04-21T23:28:52.410 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 174, 2024-04-21T23:28:52.411 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.829288911, 2024-04-21T23:28:52.411 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.004766028 2024-04-21T23:28:52.411 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:52.411 INFO:teuthology.orchestra.run.smithi111.stdout: "replayed": 0 2024-04-21T23:28:52.411 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:52.411 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_mem": { 2024-04-21T23:28:52.411 INFO:teuthology.orchestra.run.smithi111.stdout: "ino": 53, 2024-04-21T23:28:52.411 INFO:teuthology.orchestra.run.smithi111.stdout: "ino+": 2982, 2024-04-21T23:28:52.411 INFO:teuthology.orchestra.run.smithi111.stdout: "ino-": 2929, 2024-04-21T23:28:52.412 INFO:teuthology.orchestra.run.smithi111.stdout: "dir": 115, 2024-04-21T23:28:52.412 INFO:teuthology.orchestra.run.smithi111.stdout: "dir+": 178, 2024-04-21T23:28:52.412 INFO:teuthology.orchestra.run.smithi111.stdout: "dir-": 63, 2024-04-21T23:28:52.412 INFO:teuthology.orchestra.run.smithi111.stdout: "dn": 50, 2024-04-21T23:28:52.412 INFO:teuthology.orchestra.run.smithi111.stdout: "dn+": 3349, 2024-04-21T23:28:52.412 INFO:teuthology.orchestra.run.smithi111.stdout: "dn-": 3299, 2024-04-21T23:28:52.412 INFO:teuthology.orchestra.run.smithi111.stdout: "cap": 7, 2024-04-21T23:28:52.412 INFO:teuthology.orchestra.run.smithi111.stdout: "cap+": 2962, 2024-04-21T23:28:52.412 INFO:teuthology.orchestra.run.smithi111.stdout: "cap-": 2955, 2024-04-21T23:28:52.413 INFO:teuthology.orchestra.run.smithi111.stdout: "rss": 78860, 2024-04-21T23:28:52.413 INFO:teuthology.orchestra.run.smithi111.stdout: "heap": 231804 2024-04-21T23:28:52.413 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:52.413 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_server": { 2024-04-21T23:28:52.413 INFO:teuthology.orchestra.run.smithi111.stdout: "dispatch_client_request": 3723, 2024-04-21T23:28:52.413 INFO:teuthology.orchestra.run.smithi111.stdout: "dispatch_server_request": 1, 2024-04-21T23:28:52.413 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_client_request": 3560, 2024-04-21T23:28:52.413 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_client_session": 54, 2024-04-21T23:28:52.413 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_peer_request": 81, 2024-04-21T23:28:52.414 INFO:teuthology.orchestra.run.smithi111.stdout: "req_create_latency": { 2024-04-21T23:28:52.414 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 2890, 2024-04-21T23:28:52.414 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 7.147672329, 2024-04-21T23:28:52.414 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.002473243 2024-04-21T23:28:52.414 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:52.414 INFO:teuthology.orchestra.run.smithi111.stdout: "req_getattr_latency": { 2024-04-21T23:28:52.414 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 111, 2024-04-21T23:28:52.414 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.023827170, 2024-04-21T23:28:52.415 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000214659 2024-04-21T23:28:52.415 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:52.415 INFO:teuthology.orchestra.run.smithi111.stdout: "req_getfilelock_latency": { 2024-04-21T23:28:52.415 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:52.415 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:52.415 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:52.415 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:52.415 INFO:teuthology.orchestra.run.smithi111.stdout: "req_link_latency": { 2024-04-21T23:28:52.416 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 3, 2024-04-21T23:28:52.416 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.003864273, 2024-04-21T23:28:52.416 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.001288091 2024-04-21T23:28:52.416 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:52.416 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookup_latency": { 2024-04-21T23:28:52.416 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 71, 2024-04-21T23:28:52.416 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.031661169, 2024-04-21T23:28:52.416 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000445931 2024-04-21T23:28:52.417 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:52.417 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookuphash_latency": { 2024-04-21T23:28:52.417 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:52.417 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:52.417 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:52.417 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:52.417 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookupino_latency": { 2024-04-21T23:28:52.417 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 2, 2024-04-21T23:28:52.417 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000253926, 2024-04-21T23:28:52.418 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000126963 2024-04-21T23:28:52.418 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:52.418 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookupname_latency": { 2024-04-21T23:28:52.418 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:52.418 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:52.418 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:52.418 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:52.419 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookupparent_latency": { 2024-04-21T23:28:52.419 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:52.419 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:52.419 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:52.419 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:52.419 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookupsnap_latency": { 2024-04-21T23:28:52.419 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:52.419 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:52.419 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:52.420 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:52.420 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lssnap_latency": { 2024-04-21T23:28:52.420 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 10, 2024-04-21T23:28:52.421 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.001510594, 2024-04-21T23:28:52.421 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000151059 2024-04-21T23:28:52.421 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:52.421 INFO:teuthology.orchestra.run.smithi111.stdout: "req_mkdir_latency": { 2024-04-21T23:28:52.421 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 53, 2024-04-21T23:28:52.421 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.048904463, 2024-04-21T23:28:52.421 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000922725 2024-04-21T23:28:52.421 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:52.422 INFO:teuthology.orchestra.run.smithi111.stdout: "req_mknod_latency": { 2024-04-21T23:28:52.422 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:52.422 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:52.422 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:52.422 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:52.422 INFO:teuthology.orchestra.run.smithi111.stdout: "req_mksnap_latency": { 2024-04-21T23:28:52.422 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 10, 2024-04-21T23:28:52.422 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.058213491, 2024-04-21T23:28:52.423 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.005821349 2024-04-21T23:28:52.423 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:52.423 INFO:teuthology.orchestra.run.smithi111.stdout: "req_open_latency": { 2024-04-21T23:28:52.423 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 297, 2024-04-21T23:28:52.423 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.051625433, 2024-04-21T23:28:52.423 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000173823 2024-04-21T23:28:52.423 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:52.423 INFO:teuthology.orchestra.run.smithi111.stdout: "req_readdir_latency": { 2024-04-21T23:28:52.423 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 4, 2024-04-21T23:28:52.424 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.017503950, 2024-04-21T23:28:52.424 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.004375987 2024-04-21T23:28:52.424 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:52.424 INFO:teuthology.orchestra.run.smithi111.stdout: "req_rename_latency": { 2024-04-21T23:28:52.424 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 20, 2024-04-21T23:28:52.424 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.067325396, 2024-04-21T23:28:52.424 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.003366269 2024-04-21T23:28:52.424 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:52.424 INFO:teuthology.orchestra.run.smithi111.stdout: "req_renamesnap_latency": { 2024-04-21T23:28:52.425 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:52.425 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:52.425 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:52.425 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:52.425 INFO:teuthology.orchestra.run.smithi111.stdout: "req_snapdiff_latency": { 2024-04-21T23:28:52.425 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:52.425 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:52.425 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:52.426 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:52.426 INFO:teuthology.orchestra.run.smithi111.stdout: "req_rmdir_latency": { 2024-04-21T23:28:52.426 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:52.426 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:52.426 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:52.426 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:52.426 INFO:teuthology.orchestra.run.smithi111.stdout: "req_rmsnap_latency": { 2024-04-21T23:28:52.426 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 4, 2024-04-21T23:28:52.427 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.017696872, 2024-04-21T23:28:52.427 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.004424218 2024-04-21T23:28:52.427 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:52.427 INFO:teuthology.orchestra.run.smithi111.stdout: "req_rmxattr_latency": { 2024-04-21T23:28:52.427 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:52.427 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:52.427 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:52.427 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:52.428 INFO:teuthology.orchestra.run.smithi111.stdout: "req_setattr_latency": { 2024-04-21T23:28:52.428 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 2, 2024-04-21T23:28:52.428 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.001259765, 2024-04-21T23:28:52.428 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000629882 2024-04-21T23:28:52.428 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:52.428 INFO:teuthology.orchestra.run.smithi111.stdout: "req_setdirlayout_latency": { 2024-04-21T23:28:52.428 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:52.428 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:52.428 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:52.429 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:52.429 INFO:teuthology.orchestra.run.smithi111.stdout: "req_setfilelock_latency": { 2024-04-21T23:28:52.429 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:52.429 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:52.429 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:52.429 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:52.429 INFO:teuthology.orchestra.run.smithi111.stdout: "req_setlayout_latency": { 2024-04-21T23:28:52.429 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:52.430 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:52.430 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:52.430 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:52.430 INFO:teuthology.orchestra.run.smithi111.stdout: "req_setxattr_latency": { 2024-04-21T23:28:52.430 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 9, 2024-04-21T23:28:52.430 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.021261043, 2024-04-21T23:28:52.430 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.002362338 2024-04-21T23:28:52.430 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:52.431 INFO:teuthology.orchestra.run.smithi111.stdout: "req_symlink_latency": { 2024-04-21T23:28:52.431 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 1, 2024-04-21T23:28:52.431 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000669903, 2024-04-21T23:28:52.431 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000669903 2024-04-21T23:28:52.431 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:52.431 INFO:teuthology.orchestra.run.smithi111.stdout: "req_unlink_latency": { 2024-04-21T23:28:52.431 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 11, 2024-04-21T23:28:52.431 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.015246347, 2024-04-21T23:28:52.431 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.001386031 2024-04-21T23:28:52.432 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:52.432 INFO:teuthology.orchestra.run.smithi111.stdout: "cap_revoke_eviction": 0, 2024-04-21T23:28:52.432 INFO:teuthology.orchestra.run.smithi111.stdout: "cap_acquisition_throttle": 0, 2024-04-21T23:28:52.432 INFO:teuthology.orchestra.run.smithi111.stdout: "req_getvxattr_latency": { 2024-04-21T23:28:52.432 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:52.432 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:52.432 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:52.432 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-21T23:28:52.433 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:52.433 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_sessions": { 2024-04-21T23:28:52.433 INFO:teuthology.orchestra.run.smithi111.stdout: "session_count": 2, 2024-04-21T23:28:52.433 INFO:teuthology.orchestra.run.smithi111.stdout: "session_add": 7, 2024-04-21T23:28:52.433 INFO:teuthology.orchestra.run.smithi111.stdout: "session_remove": 5, 2024-04-21T23:28:52.433 INFO:teuthology.orchestra.run.smithi111.stdout: "sessions_open": 2, 2024-04-21T23:28:52.433 INFO:teuthology.orchestra.run.smithi111.stdout: "sessions_stale": 0, 2024-04-21T23:28:52.433 INFO:teuthology.orchestra.run.smithi111.stdout: "total_load": 39, 2024-04-21T23:28:52.433 INFO:teuthology.orchestra.run.smithi111.stdout: "average_load": 19, 2024-04-21T23:28:52.434 INFO:teuthology.orchestra.run.smithi111.stdout: "avg_session_uptime": 609, 2024-04-21T23:28:52.434 INFO:teuthology.orchestra.run.smithi111.stdout: "mdthresh_evicted": 0 2024-04-21T23:28:52.434 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:52.434 INFO:teuthology.orchestra.run.smithi111.stdout: "mempool": { 2024-04-21T23:28:52.434 INFO:teuthology.orchestra.run.smithi111.stdout: "bloom_filter_bytes": 0, 2024-04-21T23:28:52.434 INFO:teuthology.orchestra.run.smithi111.stdout: "bloom_filter_items": 0, 2024-04-21T23:28:52.435 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_alloc_bytes": 0, 2024-04-21T23:28:52.435 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_alloc_items": 0, 2024-04-21T23:28:52.435 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_data_bytes": 0, 2024-04-21T23:28:52.435 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_data_items": 0, 2024-04-21T23:28:52.435 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-21T23:28:52.435 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_onode_items": 0, 2024-04-21T23:28:52.435 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-21T23:28:52.435 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_meta_items": 0, 2024-04-21T23:28:52.435 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_other_bytes": 0, 2024-04-21T23:28:52.436 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_other_items": 0, 2024-04-21T23:28:52.436 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-21T23:28:52.436 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_buffer_items": 0, 2024-04-21T23:28:52.436 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_extent_bytes": 0, 2024-04-21T23:28:52.436 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_extent_items": 0, 2024-04-21T23:28:52.436 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_blob_bytes": 0, 2024-04-21T23:28:52.436 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_blob_items": 0, 2024-04-21T23:28:52.436 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-21T23:28:52.436 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_shared_blob_items": 0, 2024-04-21T23:28:52.437 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-21T23:28:52.437 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_inline_bl_items": 0, 2024-04-21T23:28:52.437 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_fsck_bytes": 0, 2024-04-21T23:28:52.437 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_fsck_items": 0, 2024-04-21T23:28:52.437 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_txc_bytes": 0, 2024-04-21T23:28:52.437 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_txc_items": 0, 2024-04-21T23:28:52.437 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-21T23:28:52.438 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_deferred_items": 0, 2024-04-21T23:28:52.438 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_bytes": 0, 2024-04-21T23:28:52.438 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_items": 0, 2024-04-21T23:28:52.438 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_bytes": 0, 2024-04-21T23:28:52.438 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_items": 0, 2024-04-21T23:28:52.439 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_reader_bytes": 0, 2024-04-21T23:28:52.439 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_reader_items": 0, 2024-04-21T23:28:52.439 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_writer_bytes": 0, 2024-04-21T23:28:52.439 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_writer_items": 0, 2024-04-21T23:28:52.439 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_anon_bytes": 317095, 2024-04-21T23:28:52.439 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_anon_items": 169, 2024-04-21T23:28:52.439 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_meta_bytes": 1672, 2024-04-21T23:28:52.439 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_meta_items": 19, 2024-04-21T23:28:52.440 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_bytes": 0, 2024-04-21T23:28:52.440 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_items": 0, 2024-04-21T23:28:52.440 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_mapbl_bytes": 0, 2024-04-21T23:28:52.440 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_mapbl_items": 0, 2024-04-21T23:28:52.440 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pglog_bytes": 0, 2024-04-21T23:28:52.440 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pglog_items": 0, 2024-04-21T23:28:52.440 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_bytes": 9704, 2024-04-21T23:28:52.440 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_items": 186, 2024-04-21T23:28:52.440 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_mapping_bytes": 0, 2024-04-21T23:28:52.441 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_mapping_items": 0, 2024-04-21T23:28:52.441 INFO:teuthology.orchestra.run.smithi111.stdout: "pgmap_bytes": 0, 2024-04-21T23:28:52.441 INFO:teuthology.orchestra.run.smithi111.stdout: "pgmap_items": 0, 2024-04-21T23:28:52.441 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_co_bytes": 340987, 2024-04-21T23:28:52.441 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_co_items": 2765, 2024-04-21T23:28:52.441 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_1_bytes": 0, 2024-04-21T23:28:52.441 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_1_items": 0, 2024-04-21T23:28:52.441 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_2_bytes": 0, 2024-04-21T23:28:52.442 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_2_items": 0 2024-04-21T23:28:52.442 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:52.442 INFO:teuthology.orchestra.run.smithi111.stdout: "objecter": { 2024-04-21T23:28:52.442 INFO:teuthology.orchestra.run.smithi111.stdout: "op_active": 0, 2024-04-21T23:28:52.442 INFO:teuthology.orchestra.run.smithi111.stdout: "op_laggy": 0, 2024-04-21T23:28:52.442 INFO:teuthology.orchestra.run.smithi111.stdout: "op_send": 3084, 2024-04-21T23:28:52.442 INFO:teuthology.orchestra.run.smithi111.stdout: "op_send_bytes": 9694495, 2024-04-21T23:28:52.442 INFO:teuthology.orchestra.run.smithi111.stdout: "op_resend": 37, 2024-04-21T23:28:52.442 INFO:teuthology.orchestra.run.smithi111.stdout: "op_reply": 3084, 2024-04-21T23:28:52.443 INFO:teuthology.orchestra.run.smithi111.stdout: "op_latency": { 2024-04-21T23:28:52.443 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 3084, 2024-04-21T23:28:52.443 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 288.406989758, 2024-04-21T23:28:52.443 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.093517182 2024-04-21T23:28:52.444 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:52.444 INFO:teuthology.orchestra.run.smithi111.stdout: "op_inflight": 0, 2024-04-21T23:28:52.444 INFO:teuthology.orchestra.run.smithi111.stdout: "oplen_avg": { 2024-04-21T23:28:52.444 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 3084, 2024-04-21T23:28:52.445 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 6343 2024-04-21T23:28:52.445 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:52.445 INFO:teuthology.orchestra.run.smithi111.stdout: "op": 3084, 2024-04-21T23:28:52.445 INFO:teuthology.orchestra.run.smithi111.stdout: "op_r": 75, 2024-04-21T23:28:52.446 INFO:teuthology.orchestra.run.smithi111.stdout: "op_w": 3009, 2024-04-21T23:28:52.446 INFO:teuthology.orchestra.run.smithi111.stdout: "op_rmw": 0, 2024-04-21T23:28:52.446 INFO:teuthology.orchestra.run.smithi111.stdout: "op_pg": 0, 2024-04-21T23:28:52.446 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_stat": 43, 2024-04-21T23:28:52.447 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_create": 1997, 2024-04-21T23:28:52.447 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_read": 10, 2024-04-21T23:28:52.447 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_write": 180, 2024-04-21T23:28:52.447 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_writefull": 61, 2024-04-21T23:28:52.448 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_writesame": 0, 2024-04-21T23:28:52.448 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_append": 0, 2024-04-21T23:28:52.448 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_zero": 0, 2024-04-21T23:28:52.448 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_truncate": 4, 2024-04-21T23:28:52.449 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_delete": 555, 2024-04-21T23:28:52.449 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_mapext": 0, 2024-04-21T23:28:52.449 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_sparse_read": 0, 2024-04-21T23:28:52.449 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_clonerange": 0, 2024-04-21T23:28:52.449 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_getxattr": 65, 2024-04-21T23:28:52.450 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_setxattr": 3009, 2024-04-21T23:28:52.450 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_cmpxattr": 0, 2024-04-21T23:28:52.450 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_rmxattr": 0, 2024-04-21T23:28:52.451 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_resetxattrs": 0, 2024-04-21T23:28:52.451 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_call": 0, 2024-04-21T23:28:52.451 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_watch": 0, 2024-04-21T23:28:52.451 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_notify": 0, 2024-04-21T23:28:52.452 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_src_cmpxattr": 0, 2024-04-21T23:28:52.452 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_pgls": 0, 2024-04-21T23:28:52.452 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_pgls_filter": 0, 2024-04-21T23:28:52.452 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_other": 0, 2024-04-21T23:28:52.452 INFO:teuthology.orchestra.run.smithi111.stdout: "linger_active": 0, 2024-04-21T23:28:52.453 INFO:teuthology.orchestra.run.smithi111.stdout: "linger_send": 0, 2024-04-21T23:28:52.453 INFO:teuthology.orchestra.run.smithi111.stdout: "linger_resend": 0, 2024-04-21T23:28:52.453 INFO:teuthology.orchestra.run.smithi111.stdout: "linger_ping": 0, 2024-04-21T23:28:52.453 INFO:teuthology.orchestra.run.smithi111.stdout: "poolop_active": 0, 2024-04-21T23:28:52.454 INFO:teuthology.orchestra.run.smithi111.stdout: "poolop_send": 0, 2024-04-21T23:28:52.454 INFO:teuthology.orchestra.run.smithi111.stdout: "poolop_resend": 0, 2024-04-21T23:28:52.454 INFO:teuthology.orchestra.run.smithi111.stdout: "poolstat_active": 0, 2024-04-21T23:28:52.455 INFO:teuthology.orchestra.run.smithi111.stdout: "poolstat_send": 0, 2024-04-21T23:28:52.455 INFO:teuthology.orchestra.run.smithi111.stdout: "poolstat_resend": 0, 2024-04-21T23:28:52.455 INFO:teuthology.orchestra.run.smithi111.stdout: "statfs_active": 0, 2024-04-21T23:28:52.455 INFO:teuthology.orchestra.run.smithi111.stdout: "statfs_send": 0, 2024-04-21T23:28:52.456 INFO:teuthology.orchestra.run.smithi111.stdout: "statfs_resend": 0, 2024-04-21T23:28:52.456 INFO:teuthology.orchestra.run.smithi111.stdout: "command_active": 0, 2024-04-21T23:28:52.456 INFO:teuthology.orchestra.run.smithi111.stdout: "command_send": 0, 2024-04-21T23:28:52.456 INFO:teuthology.orchestra.run.smithi111.stdout: "command_resend": 0, 2024-04-21T23:28:52.456 INFO:teuthology.orchestra.run.smithi111.stdout: "map_epoch": 103, 2024-04-21T23:28:52.457 INFO:teuthology.orchestra.run.smithi111.stdout: "map_full": 0, 2024-04-21T23:28:52.457 INFO:teuthology.orchestra.run.smithi111.stdout: "map_inc": 18, 2024-04-21T23:28:52.457 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_sessions": 12, 2024-04-21T23:28:52.457 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_session_open": 12, 2024-04-21T23:28:52.458 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_session_close": 0, 2024-04-21T23:28:52.458 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_laggy": 0, 2024-04-21T23:28:52.458 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_wr": 383, 2024-04-21T23:28:52.459 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_rd": 14, 2024-04-21T23:28:52.459 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_del": 22 2024-04-21T23:28:52.459 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:52.459 INFO:teuthology.orchestra.run.smithi111.stdout: "oft": { 2024-04-21T23:28:52.460 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_total_objs": 1, 2024-04-21T23:28:52.460 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_total_kv_pairs": 0, 2024-04-21T23:28:52.460 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_total_updates": 0, 2024-04-21T23:28:52.460 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_total_removes": 0 2024-04-21T23:28:52.461 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:52.461 INFO:teuthology.orchestra.run.smithi111.stdout: "osd-slow-ops": { 2024-04-21T23:28:52.461 INFO:teuthology.orchestra.run.smithi111.stdout: "slow_ops_count": 0 2024-04-21T23:28:52.461 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:52.461 INFO:teuthology.orchestra.run.smithi111.stdout: "purge_queue": { 2024-04-21T23:28:52.462 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executing_ops": 0, 2024-04-21T23:28:52.462 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executing_ops_high_water": 16, 2024-04-21T23:28:52.462 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executing": 0, 2024-04-21T23:28:52.462 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executing_high_water": 8, 2024-04-21T23:28:52.463 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executed_ops": 33, 2024-04-21T23:28:52.463 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executed": 17, 2024-04-21T23:28:52.463 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_item_in_journal": 0 2024-04-21T23:28:52.464 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:52.464 INFO:teuthology.orchestra.run.smithi111.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-21T23:28:52.464 INFO:teuthology.orchestra.run.smithi111.stdout: "val": 0, 2024-04-21T23:28:52.464 INFO:teuthology.orchestra.run.smithi111.stdout: "max": 104857600, 2024-04-21T23:28:52.465 INFO:teuthology.orchestra.run.smithi111.stdout: "get_started": 0, 2024-04-21T23:28:52.465 INFO:teuthology.orchestra.run.smithi111.stdout: "get": 11678, 2024-04-21T23:28:52.465 INFO:teuthology.orchestra.run.smithi111.stdout: "get_sum": 4798150, 2024-04-21T23:28:52.465 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_fail": 0, 2024-04-21T23:28:52.465 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_success": 11678, 2024-04-21T23:28:52.465 INFO:teuthology.orchestra.run.smithi111.stdout: "take": 0, 2024-04-21T23:28:52.466 INFO:teuthology.orchestra.run.smithi111.stdout: "take_sum": 0, 2024-04-21T23:28:52.466 INFO:teuthology.orchestra.run.smithi111.stdout: "put": 11678, 2024-04-21T23:28:52.466 INFO:teuthology.orchestra.run.smithi111.stdout: "put_sum": 4798150, 2024-04-21T23:28:52.466 INFO:teuthology.orchestra.run.smithi111.stdout: "wait": { 2024-04-21T23:28:52.466 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:52.466 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:52.466 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:52.466 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-21T23:28:52.467 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:52.467 INFO:teuthology.orchestra.run.smithi111.stdout: "throttle-objecter_bytes": { 2024-04-21T23:28:52.467 INFO:teuthology.orchestra.run.smithi111.stdout: "val": 0, 2024-04-21T23:28:52.467 INFO:teuthology.orchestra.run.smithi111.stdout: "max": 104857600, 2024-04-21T23:28:52.467 INFO:teuthology.orchestra.run.smithi111.stdout: "get_started": 0, 2024-04-21T23:28:52.468 INFO:teuthology.orchestra.run.smithi111.stdout: "get": 0, 2024-04-21T23:28:52.468 INFO:teuthology.orchestra.run.smithi111.stdout: "get_sum": 0, 2024-04-21T23:28:52.468 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_fail": 0, 2024-04-21T23:28:52.469 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_success": 0, 2024-04-21T23:28:52.469 INFO:teuthology.orchestra.run.smithi111.stdout: "take": 3084, 2024-04-21T23:28:52.469 INFO:teuthology.orchestra.run.smithi111.stdout: "take_sum": 9696252, 2024-04-21T23:28:52.469 INFO:teuthology.orchestra.run.smithi111.stdout: "put": 3084, 2024-04-21T23:28:52.469 INFO:teuthology.orchestra.run.smithi111.stdout: "put_sum": 9696252, 2024-04-21T23:28:52.470 INFO:teuthology.orchestra.run.smithi111.stdout: "wait": { 2024-04-21T23:28:52.470 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:52.470 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:52.470 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:52.470 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-21T23:28:52.470 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:52.470 INFO:teuthology.orchestra.run.smithi111.stdout: "throttle-objecter_ops": { 2024-04-21T23:28:52.471 INFO:teuthology.orchestra.run.smithi111.stdout: "val": 0, 2024-04-21T23:28:52.471 INFO:teuthology.orchestra.run.smithi111.stdout: "max": 1024, 2024-04-21T23:28:52.471 INFO:teuthology.orchestra.run.smithi111.stdout: "get_started": 0, 2024-04-21T23:28:52.471 INFO:teuthology.orchestra.run.smithi111.stdout: "get": 0, 2024-04-21T23:28:52.471 INFO:teuthology.orchestra.run.smithi111.stdout: "get_sum": 0, 2024-04-21T23:28:52.471 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_fail": 0, 2024-04-21T23:28:52.471 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_success": 0, 2024-04-21T23:28:52.471 INFO:teuthology.orchestra.run.smithi111.stdout: "take": 3084, 2024-04-21T23:28:52.471 INFO:teuthology.orchestra.run.smithi111.stdout: "take_sum": 3084, 2024-04-21T23:28:52.472 INFO:teuthology.orchestra.run.smithi111.stdout: "put": 3084, 2024-04-21T23:28:52.472 INFO:teuthology.orchestra.run.smithi111.stdout: "put_sum": 3084, 2024-04-21T23:28:52.472 INFO:teuthology.orchestra.run.smithi111.stdout: "wait": { 2024-04-21T23:28:52.472 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:52.472 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:52.472 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:52.472 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-21T23:28:52.472 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:52.473 INFO:teuthology.orchestra.run.smithi111.stdout: "throttle-write_buf_throttle": { 2024-04-21T23:28:52.473 INFO:teuthology.orchestra.run.smithi111.stdout: "val": 0, 2024-04-21T23:28:52.473 INFO:teuthology.orchestra.run.smithi111.stdout: "max": 3758096384, 2024-04-21T23:28:52.473 INFO:teuthology.orchestra.run.smithi111.stdout: "get_started": 0, 2024-04-21T23:28:52.473 INFO:teuthology.orchestra.run.smithi111.stdout: "get": 17, 2024-04-21T23:28:52.473 INFO:teuthology.orchestra.run.smithi111.stdout: "get_sum": 1845, 2024-04-21T23:28:52.473 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_fail": 0, 2024-04-21T23:28:52.473 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_success": 17, 2024-04-21T23:28:52.473 INFO:teuthology.orchestra.run.smithi111.stdout: "take": 0, 2024-04-21T23:28:52.474 INFO:teuthology.orchestra.run.smithi111.stdout: "take_sum": 0, 2024-04-21T23:28:52.474 INFO:teuthology.orchestra.run.smithi111.stdout: "put": 6, 2024-04-21T23:28:52.474 INFO:teuthology.orchestra.run.smithi111.stdout: "put_sum": 1845, 2024-04-21T23:28:52.474 INFO:teuthology.orchestra.run.smithi111.stdout: "wait": { 2024-04-21T23:28:52.474 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:52.474 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:52.474 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:52.474 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-21T23:28:52.474 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:52.475 INFO:teuthology.orchestra.run.smithi111.stdout: "throttle-write_buf_throttle-0x56429df7e500": { 2024-04-21T23:28:52.475 INFO:teuthology.orchestra.run.smithi111.stdout: "val": 0, 2024-04-21T23:28:52.475 INFO:teuthology.orchestra.run.smithi111.stdout: "max": 3758096384, 2024-04-21T23:28:52.475 INFO:teuthology.orchestra.run.smithi111.stdout: "get_started": 0, 2024-04-21T23:28:52.475 INFO:teuthology.orchestra.run.smithi111.stdout: "get": 3206, 2024-04-21T23:28:52.475 INFO:teuthology.orchestra.run.smithi111.stdout: "get_sum": 6895646, 2024-04-21T23:28:52.475 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_fail": 0, 2024-04-21T23:28:52.475 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_success": 3206, 2024-04-21T23:28:52.475 INFO:teuthology.orchestra.run.smithi111.stdout: "take": 0, 2024-04-21T23:28:52.476 INFO:teuthology.orchestra.run.smithi111.stdout: "take_sum": 0, 2024-04-21T23:28:52.476 INFO:teuthology.orchestra.run.smithi111.stdout: "put": 174, 2024-04-21T23:28:52.476 INFO:teuthology.orchestra.run.smithi111.stdout: "put_sum": 6895646, 2024-04-21T23:28:52.476 INFO:teuthology.orchestra.run.smithi111.stdout: "wait": { 2024-04-21T23:28:52.476 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:52.476 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:52.476 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:52.476 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-21T23:28:52.476 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-21T23:28:52.477 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-21T23:28:53.181 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:28:52 smithi040 ceph-mon[27984]: pgmap v325: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:28:53.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:28:52 smithi174 ceph-mon[31409]: pgmap v325: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:28:53.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:28:52 smithi111 ceph-mon[31085]: pgmap v325: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:28:53.254 INFO:tasks.check_counter:Daemon mds.b mds.exported=13 2024-04-21T23:28:53.255 INFO:tasks.check_counter:Daemon mds.b mds.imported=1 2024-04-21T23:28:53.255 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=6 2024-04-21T23:28:53.255 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=10 2024-04-21T23:28:53.255 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=4 2024-04-21T23:28:53.255 INFO:tasks.check_counter:Daemon mds.b mds.dir_split=12 2024-04-21T23:28:53.255 DEBUG:tasks.check_counter:Getting stats from e 2024-04-21T23:28:53.255 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph daemon mds.e perf dump 2024-04-21T23:28:53.492 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.b/config 2024-04-21T23:28:54.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:28:54 smithi111 ceph-mon[31085]: pgmap v326: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:28:55.152 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-21T23:28:55.153 INFO:teuthology.orchestra.run.smithi111.stdout: "AsyncMessenger::Worker-0": { 2024-04-21T23:28:55.153 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_messages": 301, 2024-04-21T23:28:55.153 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages": 281, 2024-04-21T23:28:55.153 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_bytes": 10051108, 2024-04-21T23:28:55.153 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_bytes": 150938, 2024-04-21T23:28:55.153 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_created_connections": 6, 2024-04-21T23:28:55.154 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_active_connections": 2, 2024-04-21T23:28:55.154 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_total_time": 0.054573974, 2024-04-21T23:28:55.154 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_send_time": 0.015378188, 2024-04-21T23:28:55.154 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_recv_time": 0.040512942, 2024-04-21T23:28:55.154 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_fast_dispatch_time": 0.006571351, 2024-04-21T23:28:55.154 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T23:28:55.154 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 280, 2024-04-21T23:28:55.155 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.013956423, 2024-04-21T23:28:55.155 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000049844 2024-04-21T23:28:55.155 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:55.155 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_handle_ack_lat": { 2024-04-21T23:28:55.155 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:55.155 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:55.155 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:55.156 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:55.156 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_encrypted_bytes": 1645840, 2024-04-21T23:28:55.156 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_encrypted_bytes": 136304 2024-04-21T23:28:55.156 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:55.156 INFO:teuthology.orchestra.run.smithi111.stdout: "AsyncMessenger::Worker-1": { 2024-04-21T23:28:55.156 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_messages": 681, 2024-04-21T23:28:55.156 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages": 843, 2024-04-21T23:28:55.156 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_bytes": 29532484, 2024-04-21T23:28:55.157 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_bytes": 660687, 2024-04-21T23:28:55.157 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_created_connections": 8, 2024-04-21T23:28:55.157 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_active_connections": 6, 2024-04-21T23:28:55.157 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_total_time": 0.094040797, 2024-04-21T23:28:55.157 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_send_time": 0.024509535, 2024-04-21T23:28:55.157 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_recv_time": 0.034117109, 2024-04-21T23:28:55.157 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_fast_dispatch_time": 0.027597871, 2024-04-21T23:28:55.158 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T23:28:55.158 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 841, 2024-04-21T23:28:55.158 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.044196110, 2024-04-21T23:28:55.158 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000052551 2024-04-21T23:28:55.158 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:55.158 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_handle_ack_lat": { 2024-04-21T23:28:55.158 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:55.158 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:55.159 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:55.159 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:55.159 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_encrypted_bytes": 3344, 2024-04-21T23:28:55.159 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_encrypted_bytes": 446032 2024-04-21T23:28:55.160 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:55.160 INFO:teuthology.orchestra.run.smithi111.stdout: "AsyncMessenger::Worker-2": { 2024-04-21T23:28:55.160 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_messages": 1598, 2024-04-21T23:28:55.160 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages": 1600, 2024-04-21T23:28:55.160 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_bytes": 31040299, 2024-04-21T23:28:55.160 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_bytes": 508222, 2024-04-21T23:28:55.160 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_created_connections": 6, 2024-04-21T23:28:55.160 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_active_connections": 5, 2024-04-21T23:28:55.160 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_total_time": 0.179076048, 2024-04-21T23:28:55.161 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_send_time": 0.052075894, 2024-04-21T23:28:55.161 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_recv_time": 0.059279288, 2024-04-21T23:28:55.161 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_fast_dispatch_time": 0.053115325, 2024-04-21T23:28:55.161 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T23:28:55.161 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 1598, 2024-04-21T23:28:55.161 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.063842631, 2024-04-21T23:28:55.161 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000039951 2024-04-21T23:28:55.161 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:55.162 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_handle_ack_lat": { 2024-04-21T23:28:55.162 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:55.162 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:55.162 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:55.162 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:55.162 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-21T23:28:55.162 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_encrypted_bytes": 0 2024-04-21T23:28:55.162 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:55.162 INFO:teuthology.orchestra.run.smithi111.stdout: "cct": { 2024-04-21T23:28:55.163 INFO:teuthology.orchestra.run.smithi111.stdout: "total_workers": 1, 2024-04-21T23:28:55.163 INFO:teuthology.orchestra.run.smithi111.stdout: "unhealthy_workers": 0 2024-04-21T23:28:55.163 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:55.163 INFO:teuthology.orchestra.run.smithi111.stdout: "finisher-MDSRank": { 2024-04-21T23:28:55.163 INFO:teuthology.orchestra.run.smithi111.stdout: "queue_len": 0, 2024-04-21T23:28:55.163 INFO:teuthology.orchestra.run.smithi111.stdout: "complete_latency": { 2024-04-21T23:28:55.163 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 3799, 2024-04-21T23:28:55.163 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.118686269, 2024-04-21T23:28:55.164 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000031241 2024-04-21T23:28:55.164 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-21T23:28:55.164 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:55.164 INFO:teuthology.orchestra.run.smithi111.stdout: "finisher-PurgeQueue": { 2024-04-21T23:28:55.164 INFO:teuthology.orchestra.run.smithi111.stdout: "queue_len": 0, 2024-04-21T23:28:55.164 INFO:teuthology.orchestra.run.smithi111.stdout: "complete_latency": { 2024-04-21T23:28:55.164 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:55.164 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:55.164 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:55.165 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-21T23:28:55.165 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:55.165 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": { 2024-04-21T23:28:55.165 INFO:teuthology.orchestra.run.smithi111.stdout: "request": 0, 2024-04-21T23:28:55.165 INFO:teuthology.orchestra.run.smithi111.stdout: "reply": 0, 2024-04-21T23:28:55.165 INFO:teuthology.orchestra.run.smithi111.stdout: "reply_latency": { 2024-04-21T23:28:55.165 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:55.165 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:55.166 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:55.166 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:55.166 INFO:teuthology.orchestra.run.smithi111.stdout: "slow_reply": 0, 2024-04-21T23:28:55.166 INFO:teuthology.orchestra.run.smithi111.stdout: "forward": 0, 2024-04-21T23:28:55.166 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_fetch_complete": 0, 2024-04-21T23:28:55.166 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_fetch_keys": 0, 2024-04-21T23:28:55.166 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_commit": 0, 2024-04-21T23:28:55.167 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_split": 0, 2024-04-21T23:28:55.167 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_merge": 0, 2024-04-21T23:28:55.167 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes": 12119, 2024-04-21T23:28:55.167 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_top": 199, 2024-04-21T23:28:55.167 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_bottom": 11920, 2024-04-21T23:28:55.167 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_pin_tail": 0, 2024-04-21T23:28:55.167 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_pinned": 11835, 2024-04-21T23:28:55.168 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_expired": 0, 2024-04-21T23:28:55.168 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_with_caps": 0, 2024-04-21T23:28:55.168 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": 0, 2024-04-21T23:28:55.168 INFO:teuthology.orchestra.run.smithi111.stdout: "subtrees": 10, 2024-04-21T23:28:55.168 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse": 0, 2024-04-21T23:28:55.168 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_hit": 0, 2024-04-21T23:28:55.169 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_forward": 0, 2024-04-21T23:28:55.169 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_discover": 0, 2024-04-21T23:28:55.169 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_dir_fetch": 0, 2024-04-21T23:28:55.169 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_remote_ino": 0, 2024-04-21T23:28:55.169 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_lock": 0, 2024-04-21T23:28:55.169 INFO:teuthology.orchestra.run.smithi111.stdout: "load_cent": 0, 2024-04-21T23:28:55.169 INFO:teuthology.orchestra.run.smithi111.stdout: "q": 0, 2024-04-21T23:28:55.169 INFO:teuthology.orchestra.run.smithi111.stdout: "exported": 0, 2024-04-21T23:28:55.170 INFO:teuthology.orchestra.run.smithi111.stdout: "exported_inodes": 0, 2024-04-21T23:28:55.170 INFO:teuthology.orchestra.run.smithi111.stdout: "imported": 0, 2024-04-21T23:28:55.170 INFO:teuthology.orchestra.run.smithi111.stdout: "imported_inodes": 0, 2024-04-21T23:28:55.170 INFO:teuthology.orchestra.run.smithi111.stdout: "openino_dir_fetch": 0, 2024-04-21T23:28:55.170 INFO:teuthology.orchestra.run.smithi111.stdout: "openino_backtrace_fetch": 0, 2024-04-21T23:28:55.170 INFO:teuthology.orchestra.run.smithi111.stdout: "openino_peer_discover": 0, 2024-04-21T23:28:55.170 INFO:teuthology.orchestra.run.smithi111.stdout: "root_rfiles": 0, 2024-04-21T23:28:55.170 INFO:teuthology.orchestra.run.smithi111.stdout: "root_rbytes": 0, 2024-04-21T23:28:55.170 INFO:teuthology.orchestra.run.smithi111.stdout: "root_rsnaps": 0, 2024-04-21T23:28:55.171 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_backtrace_fetch": 0, 2024-04-21T23:28:55.171 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_set_tag": 0, 2024-04-21T23:28:55.171 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_backtrace_repaired": 0, 2024-04-21T23:28:55.171 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_inotable_repaired": 0, 2024-04-21T23:28:55.171 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_dir_inodes": 0, 2024-04-21T23:28:55.171 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_dir_base_inodes": 0, 2024-04-21T23:28:55.171 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_dirfrag_rstats": 0, 2024-04-21T23:28:55.171 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_file_inodes": 0, 2024-04-21T23:28:55.172 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_inode_file_caps": 0, 2024-04-21T23:28:55.172 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_cap_op_revoke": 0, 2024-04-21T23:28:55.172 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_cap_op_grant": 0, 2024-04-21T23:28:55.172 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_cap_op_trunc": 0, 2024-04-21T23:28:55.172 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-21T23:28:55.172 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-21T23:28:55.172 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_client_caps": 0, 2024-04-21T23:28:55.172 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_client_caps_dirty": 0, 2024-04-21T23:28:55.172 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_client_cap_release": 0, 2024-04-21T23:28:55.173 INFO:teuthology.orchestra.run.smithi111.stdout: "process_request_cap_release": 0 2024-04-21T23:28:55.173 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:55.173 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_cache": { 2024-04-21T23:28:55.173 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_update": 0, 2024-04-21T23:28:55.173 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_update_receipt": 0, 2024-04-21T23:28:55.173 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_try_discover": 0, 2024-04-21T23:28:55.174 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_send_discover": 0, 2024-04-21T23:28:55.174 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_handle_discover": 0, 2024-04-21T23:28:55.174 INFO:teuthology.orchestra.run.smithi111.stdout: "num_strays": 0, 2024-04-21T23:28:55.174 INFO:teuthology.orchestra.run.smithi111.stdout: "num_strays_delayed": 0, 2024-04-21T23:28:55.174 INFO:teuthology.orchestra.run.smithi111.stdout: "num_strays_enqueuing": 0, 2024-04-21T23:28:55.174 INFO:teuthology.orchestra.run.smithi111.stdout: "strays_created": 0, 2024-04-21T23:28:55.174 INFO:teuthology.orchestra.run.smithi111.stdout: "strays_enqueued": 0, 2024-04-21T23:28:55.174 INFO:teuthology.orchestra.run.smithi111.stdout: "strays_reintegrated": 0, 2024-04-21T23:28:55.175 INFO:teuthology.orchestra.run.smithi111.stdout: "strays_migrated": 0, 2024-04-21T23:28:55.175 INFO:teuthology.orchestra.run.smithi111.stdout: "num_recovering_processing": 0, 2024-04-21T23:28:55.175 INFO:teuthology.orchestra.run.smithi111.stdout: "num_recovering_enqueued": 0, 2024-04-21T23:28:55.175 INFO:teuthology.orchestra.run.smithi111.stdout: "num_recovering_prioritized": 0, 2024-04-21T23:28:55.175 INFO:teuthology.orchestra.run.smithi111.stdout: "recovery_started": 0, 2024-04-21T23:28:55.175 INFO:teuthology.orchestra.run.smithi111.stdout: "recovery_completed": 0, 2024-04-21T23:28:55.175 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_quiesce_path": 0, 2024-04-21T23:28:55.175 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_quiesce_inode": 0, 2024-04-21T23:28:55.176 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_enqueue_scrub": 0, 2024-04-21T23:28:55.176 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_exportdir": 0, 2024-04-21T23:28:55.176 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_flush": 0, 2024-04-21T23:28:55.176 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_fragmentdir": 0, 2024-04-21T23:28:55.176 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_fragstats": 0, 2024-04-21T23:28:55.176 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_inodestats": 0 2024-04-21T23:28:55.176 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:55.176 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_log": { 2024-04-21T23:28:55.176 INFO:teuthology.orchestra.run.smithi111.stdout: "evlrg": 0, 2024-04-21T23:28:55.177 INFO:teuthology.orchestra.run.smithi111.stdout: "evadd": 0, 2024-04-21T23:28:55.177 INFO:teuthology.orchestra.run.smithi111.stdout: "evex": 0, 2024-04-21T23:28:55.177 INFO:teuthology.orchestra.run.smithi111.stdout: "evtrm": 0, 2024-04-21T23:28:55.177 INFO:teuthology.orchestra.run.smithi111.stdout: "ev": 24888, 2024-04-21T23:28:55.177 INFO:teuthology.orchestra.run.smithi111.stdout: "evexg": 0, 2024-04-21T23:28:55.177 INFO:teuthology.orchestra.run.smithi111.stdout: "evexd": 0, 2024-04-21T23:28:55.177 INFO:teuthology.orchestra.run.smithi111.stdout: "segadd": 0, 2024-04-21T23:28:55.177 INFO:teuthology.orchestra.run.smithi111.stdout: "segex": 0, 2024-04-21T23:28:55.177 INFO:teuthology.orchestra.run.smithi111.stdout: "segtrm": 0, 2024-04-21T23:28:55.178 INFO:teuthology.orchestra.run.smithi111.stdout: "seg": 38, 2024-04-21T23:28:55.178 INFO:teuthology.orchestra.run.smithi111.stdout: "segmjr": 4, 2024-04-21T23:28:55.178 INFO:teuthology.orchestra.run.smithi111.stdout: "segexg": 0, 2024-04-21T23:28:55.178 INFO:teuthology.orchestra.run.smithi111.stdout: "segexd": 0, 2024-04-21T23:28:55.178 INFO:teuthology.orchestra.run.smithi111.stdout: "expos": 4194304, 2024-04-21T23:28:55.178 INFO:teuthology.orchestra.run.smithi111.stdout: "wrpos": 72503015, 2024-04-21T23:28:55.178 INFO:teuthology.orchestra.run.smithi111.stdout: "rdpos": 72502441, 2024-04-21T23:28:55.178 INFO:teuthology.orchestra.run.smithi111.stdout: "jlat": { 2024-04-21T23:28:55.178 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:55.179 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:55.179 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:55.179 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:55.179 INFO:teuthology.orchestra.run.smithi111.stdout: "replayed": 24888 2024-04-21T23:28:55.179 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:55.179 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_mem": { 2024-04-21T23:28:55.179 INFO:teuthology.orchestra.run.smithi111.stdout: "ino": 7424, 2024-04-21T23:28:55.179 INFO:teuthology.orchestra.run.smithi111.stdout: "ino+": 10783, 2024-04-21T23:28:55.180 INFO:teuthology.orchestra.run.smithi111.stdout: "ino-": 3359, 2024-04-21T23:28:55.180 INFO:teuthology.orchestra.run.smithi111.stdout: "dir": 286, 2024-04-21T23:28:55.180 INFO:teuthology.orchestra.run.smithi111.stdout: "dir+": 333, 2024-04-21T23:28:55.180 INFO:teuthology.orchestra.run.smithi111.stdout: "dir-": 47, 2024-04-21T23:28:55.180 INFO:teuthology.orchestra.run.smithi111.stdout: "dn": 12119, 2024-04-21T23:28:55.180 INFO:teuthology.orchestra.run.smithi111.stdout: "dn+": 12273, 2024-04-21T23:28:55.180 INFO:teuthology.orchestra.run.smithi111.stdout: "dn-": 154, 2024-04-21T23:28:55.180 INFO:teuthology.orchestra.run.smithi111.stdout: "cap": 0, 2024-04-21T23:28:55.180 INFO:teuthology.orchestra.run.smithi111.stdout: "cap+": 0, 2024-04-21T23:28:55.181 INFO:teuthology.orchestra.run.smithi111.stdout: "cap-": 0, 2024-04-21T23:28:55.181 INFO:teuthology.orchestra.run.smithi111.stdout: "rss": 91468, 2024-04-21T23:28:55.181 INFO:teuthology.orchestra.run.smithi111.stdout: "heap": 215420 2024-04-21T23:28:55.181 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:55.181 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_server": { 2024-04-21T23:28:55.181 INFO:teuthology.orchestra.run.smithi111.stdout: "dispatch_client_request": 0, 2024-04-21T23:28:55.181 INFO:teuthology.orchestra.run.smithi111.stdout: "dispatch_server_request": 0, 2024-04-21T23:28:55.181 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_client_request": 0, 2024-04-21T23:28:55.181 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_client_session": 0, 2024-04-21T23:28:55.182 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_peer_request": 0, 2024-04-21T23:28:55.182 INFO:teuthology.orchestra.run.smithi111.stdout: "req_create_latency": { 2024-04-21T23:28:55.182 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:55.182 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:55.182 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:55.182 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:55.182 INFO:teuthology.orchestra.run.smithi111.stdout: "req_getattr_latency": { 2024-04-21T23:28:55.182 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:55.183 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:55.183 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:55.183 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:55.183 INFO:teuthology.orchestra.run.smithi111.stdout: "req_getfilelock_latency": { 2024-04-21T23:28:55.183 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:55.183 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:55.183 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:55.183 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:55.183 INFO:teuthology.orchestra.run.smithi111.stdout: "req_link_latency": { 2024-04-21T23:28:55.184 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:55.184 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:55.184 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:55.184 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:55.184 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookup_latency": { 2024-04-21T23:28:55.184 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:55.184 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:55.184 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:55.184 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:55.185 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookuphash_latency": { 2024-04-21T23:28:55.185 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:55.185 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:55.185 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:55.185 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:55.185 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookupino_latency": { 2024-04-21T23:28:55.185 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:55.185 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:55.185 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:55.186 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:55.186 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookupname_latency": { 2024-04-21T23:28:55.186 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:55.186 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:55.186 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:55.186 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:55.186 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookupparent_latency": { 2024-04-21T23:28:55.186 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:55.187 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:55.187 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:55.187 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:55.187 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookupsnap_latency": { 2024-04-21T23:28:55.187 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:55.187 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:55.187 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:55.187 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:55.187 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lssnap_latency": { 2024-04-21T23:28:55.188 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:55.188 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:55.188 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:55.188 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:55.189 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:28:54 smithi040 ceph-mon[27984]: pgmap v326: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:28:55.189 INFO:teuthology.orchestra.run.smithi111.stdout: "req_mkdir_latency": { 2024-04-21T23:28:55.189 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:55.189 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:55.189 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:55.190 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:55.190 INFO:teuthology.orchestra.run.smithi111.stdout: "req_mknod_latency": { 2024-04-21T23:28:55.190 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:55.190 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:55.190 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:55.190 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:55.190 INFO:teuthology.orchestra.run.smithi111.stdout: "req_mksnap_latency": { 2024-04-21T23:28:55.190 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:55.191 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:55.191 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:55.191 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:55.191 INFO:teuthology.orchestra.run.smithi111.stdout: "req_open_latency": { 2024-04-21T23:28:55.191 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:55.191 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:55.191 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:55.191 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:55.192 INFO:teuthology.orchestra.run.smithi111.stdout: "req_readdir_latency": { 2024-04-21T23:28:55.192 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:55.192 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:55.192 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:55.192 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:55.192 INFO:teuthology.orchestra.run.smithi111.stdout: "req_rename_latency": { 2024-04-21T23:28:55.192 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:55.192 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:55.192 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:55.193 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:55.193 INFO:teuthology.orchestra.run.smithi111.stdout: "req_renamesnap_latency": { 2024-04-21T23:28:55.193 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:55.193 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:55.193 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:55.193 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:55.193 INFO:teuthology.orchestra.run.smithi111.stdout: "req_snapdiff_latency": { 2024-04-21T23:28:55.193 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:55.193 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:55.194 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:55.194 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:55.194 INFO:teuthology.orchestra.run.smithi111.stdout: "req_rmdir_latency": { 2024-04-21T23:28:55.194 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:55.194 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:55.194 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:55.194 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:55.194 INFO:teuthology.orchestra.run.smithi111.stdout: "req_rmsnap_latency": { 2024-04-21T23:28:55.195 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:55.195 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:55.195 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:55.195 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:55.195 INFO:teuthology.orchestra.run.smithi111.stdout: "req_rmxattr_latency": { 2024-04-21T23:28:55.195 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:55.195 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:55.195 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:55.195 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:55.196 INFO:teuthology.orchestra.run.smithi111.stdout: "req_setattr_latency": { 2024-04-21T23:28:55.196 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:55.196 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:55.196 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:55.196 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:55.196 INFO:teuthology.orchestra.run.smithi111.stdout: "req_setdirlayout_latency": { 2024-04-21T23:28:55.196 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:55.196 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:55.197 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:55.197 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:55.197 INFO:teuthology.orchestra.run.smithi111.stdout: "req_setfilelock_latency": { 2024-04-21T23:28:55.197 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:55.197 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:55.197 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:55.197 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:55.197 INFO:teuthology.orchestra.run.smithi111.stdout: "req_setlayout_latency": { 2024-04-21T23:28:55.197 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:55.198 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:55.198 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:55.198 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:55.198 INFO:teuthology.orchestra.run.smithi111.stdout: "req_setxattr_latency": { 2024-04-21T23:28:55.198 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:55.198 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:55.198 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:55.198 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:55.199 INFO:teuthology.orchestra.run.smithi111.stdout: "req_symlink_latency": { 2024-04-21T23:28:55.199 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:55.199 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:55.199 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:55.199 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:55.199 INFO:teuthology.orchestra.run.smithi111.stdout: "req_unlink_latency": { 2024-04-21T23:28:55.199 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:55.199 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:55.199 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:55.200 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:55.200 INFO:teuthology.orchestra.run.smithi111.stdout: "cap_revoke_eviction": 0, 2024-04-21T23:28:55.200 INFO:teuthology.orchestra.run.smithi111.stdout: "cap_acquisition_throttle": 0, 2024-04-21T23:28:55.200 INFO:teuthology.orchestra.run.smithi111.stdout: "req_getvxattr_latency": { 2024-04-21T23:28:55.200 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:55.200 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:55.200 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:55.200 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-21T23:28:55.200 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:55.201 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_sessions": { 2024-04-21T23:28:55.201 INFO:teuthology.orchestra.run.smithi111.stdout: "session_count": 2, 2024-04-21T23:28:55.201 INFO:teuthology.orchestra.run.smithi111.stdout: "session_add": 6, 2024-04-21T23:28:55.201 INFO:teuthology.orchestra.run.smithi111.stdout: "session_remove": 4, 2024-04-21T23:28:55.201 INFO:teuthology.orchestra.run.smithi111.stdout: "sessions_open": 5, 2024-04-21T23:28:55.201 INFO:teuthology.orchestra.run.smithi111.stdout: "sessions_stale": 0, 2024-04-21T23:28:55.201 INFO:teuthology.orchestra.run.smithi111.stdout: "total_load": 0, 2024-04-21T23:28:55.201 INFO:teuthology.orchestra.run.smithi111.stdout: "average_load": 0, 2024-04-21T23:28:55.202 INFO:teuthology.orchestra.run.smithi111.stdout: "avg_session_uptime": 3424, 2024-04-21T23:28:55.202 INFO:teuthology.orchestra.run.smithi111.stdout: "mdthresh_evicted": 0 2024-04-21T23:28:55.202 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:55.202 INFO:teuthology.orchestra.run.smithi111.stdout: "mempool": { 2024-04-21T23:28:55.202 INFO:teuthology.orchestra.run.smithi111.stdout: "bloom_filter_bytes": 0, 2024-04-21T23:28:55.202 INFO:teuthology.orchestra.run.smithi111.stdout: "bloom_filter_items": 0, 2024-04-21T23:28:55.203 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_alloc_bytes": 0, 2024-04-21T23:28:55.203 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_alloc_items": 0, 2024-04-21T23:28:55.203 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_data_bytes": 0, 2024-04-21T23:28:55.203 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_data_items": 0, 2024-04-21T23:28:55.203 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-21T23:28:55.203 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_onode_items": 0, 2024-04-21T23:28:55.203 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-21T23:28:55.203 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_meta_items": 0, 2024-04-21T23:28:55.204 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_other_bytes": 0, 2024-04-21T23:28:55.204 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_other_items": 0, 2024-04-21T23:28:55.204 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-21T23:28:55.204 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_buffer_items": 0, 2024-04-21T23:28:55.204 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_extent_bytes": 0, 2024-04-21T23:28:55.204 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_extent_items": 0, 2024-04-21T23:28:55.204 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_blob_bytes": 0, 2024-04-21T23:28:55.204 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_blob_items": 0, 2024-04-21T23:28:55.204 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-21T23:28:55.205 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_shared_blob_items": 0, 2024-04-21T23:28:55.205 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-21T23:28:55.205 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_inline_bl_items": 0, 2024-04-21T23:28:55.205 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_fsck_bytes": 0, 2024-04-21T23:28:55.205 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_fsck_items": 0, 2024-04-21T23:28:55.205 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_txc_bytes": 0, 2024-04-21T23:28:55.205 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_txc_items": 0, 2024-04-21T23:28:55.205 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-21T23:28:55.206 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_deferred_items": 0, 2024-04-21T23:28:55.206 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_bytes": 0, 2024-04-21T23:28:55.206 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_items": 0, 2024-04-21T23:28:55.206 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_bytes": 0, 2024-04-21T23:28:55.206 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_items": 0, 2024-04-21T23:28:55.206 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_reader_bytes": 0, 2024-04-21T23:28:55.206 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_reader_items": 0, 2024-04-21T23:28:55.206 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_writer_bytes": 0, 2024-04-21T23:28:55.206 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_writer_items": 0, 2024-04-21T23:28:55.207 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_anon_bytes": 498443, 2024-04-21T23:28:55.207 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_anon_items": 7527, 2024-04-21T23:28:55.207 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_meta_bytes": 0, 2024-04-21T23:28:55.207 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_meta_items": 0, 2024-04-21T23:28:55.207 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_bytes": 0, 2024-04-21T23:28:55.207 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_items": 0, 2024-04-21T23:28:55.207 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_mapbl_bytes": 0, 2024-04-21T23:28:55.207 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_mapbl_items": 0, 2024-04-21T23:28:55.207 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pglog_bytes": 0, 2024-04-21T23:28:55.208 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pglog_items": 0, 2024-04-21T23:28:55.208 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_bytes": 9704, 2024-04-21T23:28:55.208 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_items": 186, 2024-04-21T23:28:55.208 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_mapping_bytes": 0, 2024-04-21T23:28:55.208 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_mapping_items": 0, 2024-04-21T23:28:55.208 INFO:teuthology.orchestra.run.smithi111.stdout: "pgmap_bytes": 0, 2024-04-21T23:28:55.208 INFO:teuthology.orchestra.run.smithi111.stdout: "pgmap_items": 0, 2024-04-21T23:28:55.208 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_co_bytes": 26092391, 2024-04-21T23:28:55.209 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_co_items": 233312, 2024-04-21T23:28:55.209 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_1_bytes": 0, 2024-04-21T23:28:55.209 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_1_items": 0, 2024-04-21T23:28:55.209 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_2_bytes": 0, 2024-04-21T23:28:55.209 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_2_items": 0 2024-04-21T23:28:55.209 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:55.209 INFO:teuthology.orchestra.run.smithi111.stdout: "objecter": { 2024-04-21T23:28:55.209 INFO:teuthology.orchestra.run.smithi111.stdout: "op_active": 0, 2024-04-21T23:28:55.209 INFO:teuthology.orchestra.run.smithi111.stdout: "op_laggy": 0, 2024-04-21T23:28:55.210 INFO:teuthology.orchestra.run.smithi111.stdout: "op_send": 2319, 2024-04-21T23:28:55.210 INFO:teuthology.orchestra.run.smithi111.stdout: "op_send_bytes": 21, 2024-04-21T23:28:55.210 INFO:teuthology.orchestra.run.smithi111.stdout: "op_resend": 3, 2024-04-21T23:28:55.210 INFO:teuthology.orchestra.run.smithi111.stdout: "op_reply": 2319, 2024-04-21T23:28:55.210 INFO:teuthology.orchestra.run.smithi111.stdout: "op_latency": { 2024-04-21T23:28:55.210 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 2319, 2024-04-21T23:28:55.210 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 1.527973446, 2024-04-21T23:28:55.210 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000658893 2024-04-21T23:28:55.211 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:55.211 INFO:teuthology.orchestra.run.smithi111.stdout: "op_inflight": 0, 2024-04-21T23:28:55.211 INFO:teuthology.orchestra.run.smithi111.stdout: "oplen_avg": { 2024-04-21T23:28:55.211 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 2319, 2024-04-21T23:28:55.211 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 2320 2024-04-21T23:28:55.211 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:55.211 INFO:teuthology.orchestra.run.smithi111.stdout: "op": 2319, 2024-04-21T23:28:55.211 INFO:teuthology.orchestra.run.smithi111.stdout: "op_r": 2319, 2024-04-21T23:28:55.211 INFO:teuthology.orchestra.run.smithi111.stdout: "op_w": 0, 2024-04-21T23:28:55.212 INFO:teuthology.orchestra.run.smithi111.stdout: "op_rmw": 0, 2024-04-21T23:28:55.212 INFO:teuthology.orchestra.run.smithi111.stdout: "op_pg": 0, 2024-04-21T23:28:55.212 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_stat": 1487, 2024-04-21T23:28:55.212 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_create": 0, 2024-04-21T23:28:55.212 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_read": 830, 2024-04-21T23:28:55.212 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_write": 0, 2024-04-21T23:28:55.212 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_writefull": 0, 2024-04-21T23:28:55.212 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_writesame": 0, 2024-04-21T23:28:55.212 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_append": 0, 2024-04-21T23:28:55.213 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_zero": 0, 2024-04-21T23:28:55.213 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_truncate": 0, 2024-04-21T23:28:55.213 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_delete": 0, 2024-04-21T23:28:55.213 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_mapext": 0, 2024-04-21T23:28:55.213 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_sparse_read": 0, 2024-04-21T23:28:55.213 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_clonerange": 0, 2024-04-21T23:28:55.213 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_getxattr": 1, 2024-04-21T23:28:55.213 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_setxattr": 0, 2024-04-21T23:28:55.214 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_cmpxattr": 0, 2024-04-21T23:28:55.214 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_rmxattr": 0, 2024-04-21T23:28:55.214 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_resetxattrs": 0, 2024-04-21T23:28:55.214 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_call": 0, 2024-04-21T23:28:55.214 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_watch": 0, 2024-04-21T23:28:55.214 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_notify": 0, 2024-04-21T23:28:55.214 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_src_cmpxattr": 0, 2024-04-21T23:28:55.214 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_pgls": 0, 2024-04-21T23:28:55.215 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_pgls_filter": 0, 2024-04-21T23:28:55.215 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_other": 0, 2024-04-21T23:28:55.215 INFO:teuthology.orchestra.run.smithi111.stdout: "linger_active": 0, 2024-04-21T23:28:55.215 INFO:teuthology.orchestra.run.smithi111.stdout: "linger_send": 0, 2024-04-21T23:28:55.215 INFO:teuthology.orchestra.run.smithi111.stdout: "linger_resend": 0, 2024-04-21T23:28:55.215 INFO:teuthology.orchestra.run.smithi111.stdout: "linger_ping": 0, 2024-04-21T23:28:55.215 INFO:teuthology.orchestra.run.smithi111.stdout: "poolop_active": 0, 2024-04-21T23:28:55.215 INFO:teuthology.orchestra.run.smithi111.stdout: "poolop_send": 0, 2024-04-21T23:28:55.215 INFO:teuthology.orchestra.run.smithi111.stdout: "poolop_resend": 0, 2024-04-21T23:28:55.216 INFO:teuthology.orchestra.run.smithi111.stdout: "poolstat_active": 0, 2024-04-21T23:28:55.216 INFO:teuthology.orchestra.run.smithi111.stdout: "poolstat_send": 0, 2024-04-21T23:28:55.216 INFO:teuthology.orchestra.run.smithi111.stdout: "poolstat_resend": 0, 2024-04-21T23:28:55.216 INFO:teuthology.orchestra.run.smithi111.stdout: "statfs_active": 0, 2024-04-21T23:28:55.216 INFO:teuthology.orchestra.run.smithi111.stdout: "statfs_send": 0, 2024-04-21T23:28:55.216 INFO:teuthology.orchestra.run.smithi111.stdout: "statfs_resend": 0, 2024-04-21T23:28:55.216 INFO:teuthology.orchestra.run.smithi111.stdout: "command_active": 0, 2024-04-21T23:28:55.216 INFO:teuthology.orchestra.run.smithi111.stdout: "command_send": 0, 2024-04-21T23:28:55.216 INFO:teuthology.orchestra.run.smithi111.stdout: "command_resend": 0, 2024-04-21T23:28:55.217 INFO:teuthology.orchestra.run.smithi111.stdout: "map_epoch": 103, 2024-04-21T23:28:55.217 INFO:teuthology.orchestra.run.smithi111.stdout: "map_full": 0, 2024-04-21T23:28:55.217 INFO:teuthology.orchestra.run.smithi111.stdout: "map_inc": 13, 2024-04-21T23:28:55.217 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_sessions": 9, 2024-04-21T23:28:55.217 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_session_open": 9, 2024-04-21T23:28:55.217 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_session_close": 0, 2024-04-21T23:28:55.217 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_laggy": 0, 2024-04-21T23:28:55.217 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_wr": 0, 2024-04-21T23:28:55.217 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_rd": 2, 2024-04-21T23:28:55.218 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_del": 0 2024-04-21T23:28:55.218 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:55.218 INFO:teuthology.orchestra.run.smithi111.stdout: "oft": { 2024-04-21T23:28:55.218 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_total_objs": 0, 2024-04-21T23:28:55.218 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_total_kv_pairs": 0, 2024-04-21T23:28:55.218 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_total_updates": 0, 2024-04-21T23:28:55.218 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_total_removes": 0 2024-04-21T23:28:55.218 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:55.219 INFO:teuthology.orchestra.run.smithi111.stdout: "osd-slow-ops": { 2024-04-21T23:28:55.219 INFO:teuthology.orchestra.run.smithi111.stdout: "slow_ops_count": 0 2024-04-21T23:28:55.219 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:55.219 INFO:teuthology.orchestra.run.smithi111.stdout: "purge_queue": { 2024-04-21T23:28:55.219 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executing_ops": 0, 2024-04-21T23:28:55.219 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executing_ops_high_water": 0, 2024-04-21T23:28:55.219 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executing": 0, 2024-04-21T23:28:55.219 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executing_high_water": 0, 2024-04-21T23:28:55.219 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executed_ops": 0, 2024-04-21T23:28:55.220 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executed": 0, 2024-04-21T23:28:55.220 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_item_in_journal": 0 2024-04-21T23:28:55.220 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:55.220 INFO:teuthology.orchestra.run.smithi111.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-21T23:28:55.220 INFO:teuthology.orchestra.run.smithi111.stdout: "val": 0, 2024-04-21T23:28:55.220 INFO:teuthology.orchestra.run.smithi111.stdout: "max": 104857600, 2024-04-21T23:28:55.220 INFO:teuthology.orchestra.run.smithi111.stdout: "get_started": 0, 2024-04-21T23:28:55.220 INFO:teuthology.orchestra.run.smithi111.stdout: "get": 2576, 2024-04-21T23:28:55.221 INFO:teuthology.orchestra.run.smithi111.stdout: "get_sum": 70375993, 2024-04-21T23:28:55.221 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_fail": 0, 2024-04-21T23:28:55.221 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_success": 2576, 2024-04-21T23:28:55.221 INFO:teuthology.orchestra.run.smithi111.stdout: "take": 0, 2024-04-21T23:28:55.221 INFO:teuthology.orchestra.run.smithi111.stdout: "take_sum": 0, 2024-04-21T23:28:55.221 INFO:teuthology.orchestra.run.smithi111.stdout: "put": 2576, 2024-04-21T23:28:55.221 INFO:teuthology.orchestra.run.smithi111.stdout: "put_sum": 70375993, 2024-04-21T23:28:55.221 INFO:teuthology.orchestra.run.smithi111.stdout: "wait": { 2024-04-21T23:28:55.221 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:55.222 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:55.222 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:55.222 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-21T23:28:55.222 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:55.222 INFO:teuthology.orchestra.run.smithi111.stdout: "throttle-objecter_bytes": { 2024-04-21T23:28:55.222 INFO:teuthology.orchestra.run.smithi111.stdout: "val": 0, 2024-04-21T23:28:55.222 INFO:teuthology.orchestra.run.smithi111.stdout: "max": 104857600, 2024-04-21T23:28:55.222 INFO:teuthology.orchestra.run.smithi111.stdout: "get_started": 0, 2024-04-21T23:28:55.222 INFO:teuthology.orchestra.run.smithi111.stdout: "get": 0, 2024-04-21T23:28:55.223 INFO:teuthology.orchestra.run.smithi111.stdout: "get_sum": 0, 2024-04-21T23:28:55.223 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_fail": 0, 2024-04-21T23:28:55.223 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_success": 0, 2024-04-21T23:28:55.223 INFO:teuthology.orchestra.run.smithi111.stdout: "take": 2319, 2024-04-21T23:28:55.223 INFO:teuthology.orchestra.run.smithi111.stdout: "take_sum": 68322279, 2024-04-21T23:28:55.223 INFO:teuthology.orchestra.run.smithi111.stdout: "put": 2319, 2024-04-21T23:28:55.223 INFO:teuthology.orchestra.run.smithi111.stdout: "put_sum": 68322279, 2024-04-21T23:28:55.223 INFO:teuthology.orchestra.run.smithi111.stdout: "wait": { 2024-04-21T23:28:55.224 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:55.224 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:55.224 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:55.224 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-21T23:28:55.224 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:55.224 INFO:teuthology.orchestra.run.smithi111.stdout: "throttle-objecter_ops": { 2024-04-21T23:28:55.224 INFO:teuthology.orchestra.run.smithi111.stdout: "val": 0, 2024-04-21T23:28:55.224 INFO:teuthology.orchestra.run.smithi111.stdout: "max": 1024, 2024-04-21T23:28:55.224 INFO:teuthology.orchestra.run.smithi111.stdout: "get_started": 0, 2024-04-21T23:28:55.225 INFO:teuthology.orchestra.run.smithi111.stdout: "get": 0, 2024-04-21T23:28:55.225 INFO:teuthology.orchestra.run.smithi111.stdout: "get_sum": 0, 2024-04-21T23:28:55.225 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_fail": 0, 2024-04-21T23:28:55.225 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_success": 0, 2024-04-21T23:28:55.225 INFO:teuthology.orchestra.run.smithi111.stdout: "take": 2319, 2024-04-21T23:28:55.225 INFO:teuthology.orchestra.run.smithi111.stdout: "take_sum": 2319, 2024-04-21T23:28:55.225 INFO:teuthology.orchestra.run.smithi111.stdout: "put": 2319, 2024-04-21T23:28:55.225 INFO:teuthology.orchestra.run.smithi111.stdout: "put_sum": 2319, 2024-04-21T23:28:55.226 INFO:teuthology.orchestra.run.smithi111.stdout: "wait": { 2024-04-21T23:28:55.226 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:55.226 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:55.226 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:55.226 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-21T23:28:55.226 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:55.226 INFO:teuthology.orchestra.run.smithi111.stdout: "throttle-write_buf_throttle": { 2024-04-21T23:28:55.226 INFO:teuthology.orchestra.run.smithi111.stdout: "val": 0, 2024-04-21T23:28:55.227 INFO:teuthology.orchestra.run.smithi111.stdout: "max": 3758096384, 2024-04-21T23:28:55.227 INFO:teuthology.orchestra.run.smithi111.stdout: "get_started": 0, 2024-04-21T23:28:55.227 INFO:teuthology.orchestra.run.smithi111.stdout: "get": 0, 2024-04-21T23:28:55.227 INFO:teuthology.orchestra.run.smithi111.stdout: "get_sum": 0, 2024-04-21T23:28:55.227 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_fail": 0, 2024-04-21T23:28:55.227 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_success": 0, 2024-04-21T23:28:55.227 INFO:teuthology.orchestra.run.smithi111.stdout: "take": 0, 2024-04-21T23:28:55.227 INFO:teuthology.orchestra.run.smithi111.stdout: "take_sum": 0, 2024-04-21T23:28:55.227 INFO:teuthology.orchestra.run.smithi111.stdout: "put": 0, 2024-04-21T23:28:55.228 INFO:teuthology.orchestra.run.smithi111.stdout: "put_sum": 0, 2024-04-21T23:28:55.228 INFO:teuthology.orchestra.run.smithi111.stdout: "wait": { 2024-04-21T23:28:55.228 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:55.228 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:55.228 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:55.228 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-21T23:28:55.228 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:55.228 INFO:teuthology.orchestra.run.smithi111.stdout: "throttle-write_buf_throttle-0x555bd4c6c6e0": { 2024-04-21T23:28:55.228 INFO:teuthology.orchestra.run.smithi111.stdout: "val": 0, 2024-04-21T23:28:55.229 INFO:teuthology.orchestra.run.smithi111.stdout: "max": 3758096384, 2024-04-21T23:28:55.229 INFO:teuthology.orchestra.run.smithi111.stdout: "get_started": 0, 2024-04-21T23:28:55.229 INFO:teuthology.orchestra.run.smithi111.stdout: "get": 0, 2024-04-21T23:28:55.229 INFO:teuthology.orchestra.run.smithi111.stdout: "get_sum": 0, 2024-04-21T23:28:55.229 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_fail": 0, 2024-04-21T23:28:55.229 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_success": 0, 2024-04-21T23:28:55.229 INFO:teuthology.orchestra.run.smithi111.stdout: "take": 0, 2024-04-21T23:28:55.229 INFO:teuthology.orchestra.run.smithi111.stdout: "take_sum": 0, 2024-04-21T23:28:55.229 INFO:teuthology.orchestra.run.smithi111.stdout: "put": 0, 2024-04-21T23:28:55.230 INFO:teuthology.orchestra.run.smithi111.stdout: "put_sum": 0, 2024-04-21T23:28:55.230 INFO:teuthology.orchestra.run.smithi111.stdout: "wait": { 2024-04-21T23:28:55.230 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:55.230 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:55.230 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:55.230 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-21T23:28:55.230 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-21T23:28:55.230 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-21T23:28:55.231 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:28:54 smithi174 ceph-mon[31409]: pgmap v326: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:28:55.938 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-04-21T23:28:55.939 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-04-21T23:28:55.939 INFO:tasks.check_counter:Daemon mds.e mds.root_rsnaps=0 2024-04-21T23:28:55.939 INFO:tasks.check_counter:Daemon mds.e mds_server.req_mksnap_latency.avgcount=0 2024-04-21T23:28:55.939 INFO:tasks.check_counter:Daemon mds.e mds_server.req_rmsnap_latency.avgcount=0 2024-04-21T23:28:55.939 INFO:tasks.check_counter:Daemon mds.e mds.dir_split=0 2024-04-21T23:28:55.939 DEBUG:tasks.check_counter:Getting stats from h 2024-04-21T23:28:55.939 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph daemon mds.h perf dump 2024-04-21T23:28:56.178 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.b/config 2024-04-21T23:28:56.944 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:28:56 smithi111 ceph-mon[31085]: pgmap v327: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:28:57.181 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:28:56 smithi040 ceph-mon[27984]: pgmap v327: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:28:57.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:28:56 smithi174 ceph-mon[31409]: pgmap v327: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:28:57.909 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-21T23:28:57.909 INFO:teuthology.orchestra.run.smithi111.stdout: "AsyncMessenger::Worker-0": { 2024-04-21T23:28:57.909 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_messages": 6277, 2024-04-21T23:28:57.909 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages": 7369, 2024-04-21T23:28:57.910 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_bytes": 4643133, 2024-04-21T23:28:57.910 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_bytes": 25096071, 2024-04-21T23:28:57.910 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_created_connections": 11, 2024-04-21T23:28:57.910 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_active_connections": 4, 2024-04-21T23:28:57.910 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_total_time": 0.553675875, 2024-04-21T23:28:57.910 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_send_time": 0.250183317, 2024-04-21T23:28:57.910 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_recv_time": 0.142806007, 2024-04-21T23:28:57.910 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_fast_dispatch_time": 0.137886573, 2024-04-21T23:28:57.911 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T23:28:57.911 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 7371, 2024-04-21T23:28:57.911 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.537632641, 2024-04-21T23:28:57.911 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000072938 2024-04-21T23:28:57.911 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:57.911 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_handle_ack_lat": { 2024-04-21T23:28:57.911 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 3514, 2024-04-21T23:28:57.912 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000747021, 2024-04-21T23:28:57.912 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000212 2024-04-21T23:28:57.912 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:57.912 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_encrypted_bytes": 928, 2024-04-21T23:28:57.912 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_encrypted_bytes": 976 2024-04-21T23:28:57.912 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:57.912 INFO:teuthology.orchestra.run.smithi111.stdout: "AsyncMessenger::Worker-1": { 2024-04-21T23:28:57.913 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_messages": 4567, 2024-04-21T23:28:57.913 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages": 5105, 2024-04-21T23:28:57.913 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_bytes": 1201305, 2024-04-21T23:28:57.913 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_bytes": 28639720, 2024-04-21T23:28:57.913 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_created_connections": 16, 2024-04-21T23:28:57.913 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_active_connections": 6, 2024-04-21T23:28:57.913 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_total_time": 0.379499065, 2024-04-21T23:28:57.913 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_send_time": 0.149927892, 2024-04-21T23:28:57.913 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_recv_time": 0.093967390, 2024-04-21T23:28:57.914 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_fast_dispatch_time": 0.114418547, 2024-04-21T23:28:57.914 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T23:28:57.914 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 5102, 2024-04-21T23:28:57.914 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.446904197, 2024-04-21T23:28:57.914 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000087593 2024-04-21T23:28:57.914 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:57.914 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_handle_ack_lat": { 2024-04-21T23:28:57.914 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 1781, 2024-04-21T23:28:57.915 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000301475, 2024-04-21T23:28:57.915 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000169 2024-04-21T23:28:57.915 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:57.915 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_encrypted_bytes": 6400, 2024-04-21T23:28:57.915 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_encrypted_bytes": 145136 2024-04-21T23:28:57.915 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:57.915 INFO:teuthology.orchestra.run.smithi111.stdout: "AsyncMessenger::Worker-2": { 2024-04-21T23:28:57.915 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_messages": 28586, 2024-04-21T23:28:57.915 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages": 42119, 2024-04-21T23:28:57.916 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_bytes": 10623527, 2024-04-21T23:28:57.916 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_bytes": 24830411, 2024-04-21T23:28:57.916 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_created_connections": 17, 2024-04-21T23:28:57.916 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_active_connections": 7, 2024-04-21T23:28:57.916 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_total_time": 1.386187601, 2024-04-21T23:28:57.916 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_send_time": 0.641769601, 2024-04-21T23:28:57.916 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_recv_time": 3.554008391, 2024-04-21T23:28:57.916 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_fast_dispatch_time": 0.055282842, 2024-04-21T23:28:57.916 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T23:28:57.917 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 42115, 2024-04-21T23:28:57.917 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 1.765635940, 2024-04-21T23:28:57.917 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000041924 2024-04-21T23:28:57.917 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:57.917 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_handle_ack_lat": { 2024-04-21T23:28:57.917 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 62722, 2024-04-21T23:28:57.917 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.008429952, 2024-04-21T23:28:57.917 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000134 2024-04-21T23:28:57.917 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:57.918 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_encrypted_bytes": 1661344, 2024-04-21T23:28:57.918 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_encrypted_bytes": 567072 2024-04-21T23:28:57.918 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:57.918 INFO:teuthology.orchestra.run.smithi111.stdout: "cct": { 2024-04-21T23:28:57.918 INFO:teuthology.orchestra.run.smithi111.stdout: "total_workers": 1, 2024-04-21T23:28:57.918 INFO:teuthology.orchestra.run.smithi111.stdout: "unhealthy_workers": 0 2024-04-21T23:28:57.918 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:57.918 INFO:teuthology.orchestra.run.smithi111.stdout: "finisher-MDSRank": { 2024-04-21T23:28:57.919 INFO:teuthology.orchestra.run.smithi111.stdout: "queue_len": 0, 2024-04-21T23:28:57.919 INFO:teuthology.orchestra.run.smithi111.stdout: "complete_latency": { 2024-04-21T23:28:57.919 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 2477, 2024-04-21T23:28:57.919 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 19.043400489, 2024-04-21T23:28:57.919 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.007688090 2024-04-21T23:28:57.919 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-21T23:28:57.919 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:57.919 INFO:teuthology.orchestra.run.smithi111.stdout: "finisher-PurgeQueue": { 2024-04-21T23:28:57.920 INFO:teuthology.orchestra.run.smithi111.stdout: "queue_len": 0, 2024-04-21T23:28:57.920 INFO:teuthology.orchestra.run.smithi111.stdout: "complete_latency": { 2024-04-21T23:28:57.920 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 5362, 2024-04-21T23:28:57.920 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 1.935867770, 2024-04-21T23:28:57.920 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000361034 2024-04-21T23:28:57.921 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-21T23:28:57.921 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:57.921 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": { 2024-04-21T23:28:57.921 INFO:teuthology.orchestra.run.smithi111.stdout: "request": 9195, 2024-04-21T23:28:57.921 INFO:teuthology.orchestra.run.smithi111.stdout: "reply": 9152, 2024-04-21T23:28:57.921 INFO:teuthology.orchestra.run.smithi111.stdout: "reply_latency": { 2024-04-21T23:28:57.922 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 9152, 2024-04-21T23:28:57.922 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 27.253713092, 2024-04-21T23:28:57.922 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.002977896 2024-04-21T23:28:57.922 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:57.922 INFO:teuthology.orchestra.run.smithi111.stdout: "slow_reply": 0, 2024-04-21T23:28:57.922 INFO:teuthology.orchestra.run.smithi111.stdout: "forward": 43, 2024-04-21T23:28:57.922 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_fetch_complete": 0, 2024-04-21T23:28:57.922 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_fetch_keys": 0, 2024-04-21T23:28:57.922 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_commit": 165, 2024-04-21T23:28:57.923 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_split": 19, 2024-04-21T23:28:57.923 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_merge": 1, 2024-04-21T23:28:57.923 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes": 9291, 2024-04-21T23:28:57.923 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_top": 1092, 2024-04-21T23:28:57.923 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_bottom": 8157, 2024-04-21T23:28:57.923 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_pin_tail": 42, 2024-04-21T23:28:57.923 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_pinned": 7702, 2024-04-21T23:28:57.923 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_expired": 176, 2024-04-21T23:28:57.924 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_with_caps": 2777, 2024-04-21T23:28:57.924 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": 2777, 2024-04-21T23:28:57.924 INFO:teuthology.orchestra.run.smithi111.stdout: "subtrees": 15, 2024-04-21T23:28:57.924 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse": 11545, 2024-04-21T23:28:57.924 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_hit": 10229, 2024-04-21T23:28:57.924 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_forward": 37, 2024-04-21T23:28:57.924 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_discover": 3, 2024-04-21T23:28:57.925 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_dir_fetch": 0, 2024-04-21T23:28:57.925 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_remote_ino": 0, 2024-04-21T23:28:57.925 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_lock": 39, 2024-04-21T23:28:57.925 INFO:teuthology.orchestra.run.smithi111.stdout: "load_cent": 0, 2024-04-21T23:28:57.925 INFO:teuthology.orchestra.run.smithi111.stdout: "q": 0, 2024-04-21T23:28:57.925 INFO:teuthology.orchestra.run.smithi111.stdout: "exported": 9, 2024-04-21T23:28:57.925 INFO:teuthology.orchestra.run.smithi111.stdout: "exported_inodes": 160, 2024-04-21T23:28:57.926 INFO:teuthology.orchestra.run.smithi111.stdout: "imported": 12, 2024-04-21T23:28:57.926 INFO:teuthology.orchestra.run.smithi111.stdout: "imported_inodes": 3211, 2024-04-21T23:28:57.926 INFO:teuthology.orchestra.run.smithi111.stdout: "openino_dir_fetch": 0, 2024-04-21T23:28:57.926 INFO:teuthology.orchestra.run.smithi111.stdout: "openino_backtrace_fetch": 3256, 2024-04-21T23:28:57.926 INFO:teuthology.orchestra.run.smithi111.stdout: "openino_peer_discover": 0, 2024-04-21T23:28:57.926 INFO:teuthology.orchestra.run.smithi111.stdout: "root_rfiles": 1313, 2024-04-21T23:28:57.926 INFO:teuthology.orchestra.run.smithi111.stdout: "root_rbytes": 2097386, 2024-04-21T23:28:57.927 INFO:teuthology.orchestra.run.smithi111.stdout: "root_rsnaps": 2, 2024-04-21T23:28:57.927 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_backtrace_fetch": 3256, 2024-04-21T23:28:57.927 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_set_tag": 0, 2024-04-21T23:28:57.927 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_backtrace_repaired": 0, 2024-04-21T23:28:57.927 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_inotable_repaired": 0, 2024-04-21T23:28:57.927 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_dir_inodes": 46, 2024-04-21T23:28:57.927 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_dir_base_inodes": 0, 2024-04-21T23:28:57.927 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_dirfrag_rstats": 46, 2024-04-21T23:28:57.928 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_file_inodes": 3210, 2024-04-21T23:28:57.928 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_inode_file_caps": 4, 2024-04-21T23:28:57.928 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_cap_op_revoke": 769, 2024-04-21T23:28:57.928 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_cap_op_grant": 9302, 2024-04-21T23:28:57.928 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_cap_op_trunc": 8, 2024-04-21T23:28:57.928 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-21T23:28:57.928 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-21T23:28:57.928 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_client_caps": 16503, 2024-04-21T23:28:57.929 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_client_caps_dirty": 11003, 2024-04-21T23:28:57.929 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_client_cap_release": 2204, 2024-04-21T23:28:57.929 INFO:teuthology.orchestra.run.smithi111.stdout: "process_request_cap_release": 4190 2024-04-21T23:28:57.929 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:57.929 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_cache": { 2024-04-21T23:28:57.929 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_update": 0, 2024-04-21T23:28:57.929 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_update_receipt": 0, 2024-04-21T23:28:57.929 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_try_discover": 0, 2024-04-21T23:28:57.930 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_send_discover": 9, 2024-04-21T23:28:57.930 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_handle_discover": 13, 2024-04-21T23:28:57.930 INFO:teuthology.orchestra.run.smithi111.stdout: "num_strays": 4, 2024-04-21T23:28:57.930 INFO:teuthology.orchestra.run.smithi111.stdout: "num_strays_delayed": 0, 2024-04-21T23:28:57.930 INFO:teuthology.orchestra.run.smithi111.stdout: "num_strays_enqueuing": 0, 2024-04-21T23:28:57.930 INFO:teuthology.orchestra.run.smithi111.stdout: "strays_created": 3223, 2024-04-21T23:28:57.930 INFO:teuthology.orchestra.run.smithi111.stdout: "strays_enqueued": 3219, 2024-04-21T23:28:57.930 INFO:teuthology.orchestra.run.smithi111.stdout: "strays_reintegrated": 0, 2024-04-21T23:28:57.931 INFO:teuthology.orchestra.run.smithi111.stdout: "strays_migrated": 0, 2024-04-21T23:28:57.931 INFO:teuthology.orchestra.run.smithi111.stdout: "num_recovering_processing": 0, 2024-04-21T23:28:57.931 INFO:teuthology.orchestra.run.smithi111.stdout: "num_recovering_enqueued": 0, 2024-04-21T23:28:57.931 INFO:teuthology.orchestra.run.smithi111.stdout: "num_recovering_prioritized": 0, 2024-04-21T23:28:57.931 INFO:teuthology.orchestra.run.smithi111.stdout: "recovery_started": 0, 2024-04-21T23:28:57.931 INFO:teuthology.orchestra.run.smithi111.stdout: "recovery_completed": 0, 2024-04-21T23:28:57.931 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_quiesce_path": 0, 2024-04-21T23:28:57.932 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_quiesce_inode": 0, 2024-04-21T23:28:57.932 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_enqueue_scrub": 0, 2024-04-21T23:28:57.932 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_exportdir": 9, 2024-04-21T23:28:57.932 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_flush": 0, 2024-04-21T23:28:57.932 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_fragmentdir": 20, 2024-04-21T23:28:57.932 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_fragstats": 0, 2024-04-21T23:28:57.932 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_inodestats": 0 2024-04-21T23:28:57.932 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:57.932 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_log": { 2024-04-21T23:28:57.933 INFO:teuthology.orchestra.run.smithi111.stdout: "evlrg": 1, 2024-04-21T23:28:57.933 INFO:teuthology.orchestra.run.smithi111.stdout: "evadd": 24888, 2024-04-21T23:28:57.933 INFO:teuthology.orchestra.run.smithi111.stdout: "evex": 0, 2024-04-21T23:28:57.933 INFO:teuthology.orchestra.run.smithi111.stdout: "evtrm": 0, 2024-04-21T23:28:57.933 INFO:teuthology.orchestra.run.smithi111.stdout: "ev": 24888, 2024-04-21T23:28:57.933 INFO:teuthology.orchestra.run.smithi111.stdout: "evexg": 0, 2024-04-21T23:28:57.933 INFO:teuthology.orchestra.run.smithi111.stdout: "evexd": 0, 2024-04-21T23:28:57.933 INFO:teuthology.orchestra.run.smithi111.stdout: "segadd": 38, 2024-04-21T23:28:57.934 INFO:teuthology.orchestra.run.smithi111.stdout: "segex": 0, 2024-04-21T23:28:57.934 INFO:teuthology.orchestra.run.smithi111.stdout: "segtrm": 0, 2024-04-21T23:28:57.934 INFO:teuthology.orchestra.run.smithi111.stdout: "seg": 38, 2024-04-21T23:28:57.934 INFO:teuthology.orchestra.run.smithi111.stdout: "segmjr": 4, 2024-04-21T23:28:57.934 INFO:teuthology.orchestra.run.smithi111.stdout: "segexg": 0, 2024-04-21T23:28:57.934 INFO:teuthology.orchestra.run.smithi111.stdout: "segexd": 0, 2024-04-21T23:28:57.934 INFO:teuthology.orchestra.run.smithi111.stdout: "expos": 4194304, 2024-04-21T23:28:57.935 INFO:teuthology.orchestra.run.smithi111.stdout: "wrpos": 72503015, 2024-04-21T23:28:57.935 INFO:teuthology.orchestra.run.smithi111.stdout: "rdpos": 0, 2024-04-21T23:28:57.935 INFO:teuthology.orchestra.run.smithi111.stdout: "jlat": { 2024-04-21T23:28:57.935 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 508, 2024-04-21T23:28:57.935 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 20.691521842, 2024-04-21T23:28:57.935 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.040731342 2024-04-21T23:28:57.935 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:57.935 INFO:teuthology.orchestra.run.smithi111.stdout: "replayed": 0 2024-04-21T23:28:57.936 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:57.936 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_mem": { 2024-04-21T23:28:57.936 INFO:teuthology.orchestra.run.smithi111.stdout: "ino": 7430, 2024-04-21T23:28:57.936 INFO:teuthology.orchestra.run.smithi111.stdout: "ino+": 10783, 2024-04-21T23:28:57.936 INFO:teuthology.orchestra.run.smithi111.stdout: "ino-": 3353, 2024-04-21T23:28:57.936 INFO:teuthology.orchestra.run.smithi111.stdout: "dir": 289, 2024-04-21T23:28:57.936 INFO:teuthology.orchestra.run.smithi111.stdout: "dir+": 334, 2024-04-21T23:28:57.936 INFO:teuthology.orchestra.run.smithi111.stdout: "dir-": 45, 2024-04-21T23:28:57.937 INFO:teuthology.orchestra.run.smithi111.stdout: "dn": 9291, 2024-04-21T23:28:57.937 INFO:teuthology.orchestra.run.smithi111.stdout: "dn+": 12493, 2024-04-21T23:28:57.937 INFO:teuthology.orchestra.run.smithi111.stdout: "dn-": 3202, 2024-04-21T23:28:57.937 INFO:teuthology.orchestra.run.smithi111.stdout: "cap": 2777, 2024-04-21T23:28:57.937 INFO:teuthology.orchestra.run.smithi111.stdout: "cap+": 6741, 2024-04-21T23:28:57.937 INFO:teuthology.orchestra.run.smithi111.stdout: "cap-": 3964, 2024-04-21T23:28:57.937 INFO:teuthology.orchestra.run.smithi111.stdout: "rss": 130292, 2024-04-21T23:28:57.937 INFO:teuthology.orchestra.run.smithi111.stdout: "heap": 223612 2024-04-21T23:28:57.937 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:57.938 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_server": { 2024-04-21T23:28:57.938 INFO:teuthology.orchestra.run.smithi111.stdout: "dispatch_client_request": 10114, 2024-04-21T23:28:57.938 INFO:teuthology.orchestra.run.smithi111.stdout: "dispatch_server_request": 90, 2024-04-21T23:28:57.938 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_client_request": 9195, 2024-04-21T23:28:57.938 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_client_session": 42, 2024-04-21T23:28:57.938 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_peer_request": 181, 2024-04-21T23:28:57.938 INFO:teuthology.orchestra.run.smithi111.stdout: "req_create_latency": { 2024-04-21T23:28:57.938 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 3784, 2024-04-21T23:28:57.939 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 13.531182979, 2024-04-21T23:28:57.939 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.003575894 2024-04-21T23:28:57.939 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:57.939 INFO:teuthology.orchestra.run.smithi111.stdout: "req_getattr_latency": { 2024-04-21T23:28:57.939 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 268, 2024-04-21T23:28:57.939 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 1.591700654, 2024-04-21T23:28:57.939 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.005939181 2024-04-21T23:28:57.939 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:57.939 INFO:teuthology.orchestra.run.smithi111.stdout: "req_getfilelock_latency": { 2024-04-21T23:28:57.940 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:57.940 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:57.940 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:57.940 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:57.940 INFO:teuthology.orchestra.run.smithi111.stdout: "req_link_latency": { 2024-04-21T23:28:57.940 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:57.940 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:57.940 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:57.941 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:57.941 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookup_latency": { 2024-04-21T23:28:57.941 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 1545, 2024-04-21T23:28:57.941 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.863675477, 2024-04-21T23:28:57.941 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000559013 2024-04-21T23:28:57.941 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:57.942 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookuphash_latency": { 2024-04-21T23:28:57.942 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:57.942 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:57.942 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:57.942 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:57.942 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookupino_latency": { 2024-04-21T23:28:57.942 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:57.942 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:57.942 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:57.943 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:57.943 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookupname_latency": { 2024-04-21T23:28:57.943 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:57.943 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:57.943 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:57.943 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:57.943 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookupparent_latency": { 2024-04-21T23:28:57.943 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:57.944 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:57.944 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:57.944 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:57.944 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookupsnap_latency": { 2024-04-21T23:28:57.944 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:57.944 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:57.944 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:57.944 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:57.945 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lssnap_latency": { 2024-04-21T23:28:57.945 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:57.946 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:57.946 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:57.946 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:57.946 INFO:teuthology.orchestra.run.smithi111.stdout: "req_mkdir_latency": { 2024-04-21T23:28:57.946 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 32, 2024-04-21T23:28:57.946 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.036172856, 2024-04-21T23:28:57.946 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.001130401 2024-04-21T23:28:57.947 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:57.947 INFO:teuthology.orchestra.run.smithi111.stdout: "req_mknod_latency": { 2024-04-21T23:28:57.947 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:57.947 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:57.947 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:57.947 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:57.947 INFO:teuthology.orchestra.run.smithi111.stdout: "req_mksnap_latency": { 2024-04-21T23:28:57.947 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:57.947 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:57.948 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:57.948 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:57.948 INFO:teuthology.orchestra.run.smithi111.stdout: "req_open_latency": { 2024-04-21T23:28:57.948 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 144, 2024-04-21T23:28:57.948 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.043038000, 2024-04-21T23:28:57.948 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000298875 2024-04-21T23:28:57.948 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:57.948 INFO:teuthology.orchestra.run.smithi111.stdout: "req_readdir_latency": { 2024-04-21T23:28:57.948 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 100, 2024-04-21T23:28:57.949 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.136192213, 2024-04-21T23:28:57.949 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.001361922 2024-04-21T23:28:57.949 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:57.949 INFO:teuthology.orchestra.run.smithi111.stdout: "req_rename_latency": { 2024-04-21T23:28:57.949 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 62, 2024-04-21T23:28:57.949 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.127235024, 2024-04-21T23:28:57.949 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.002052177 2024-04-21T23:28:57.949 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:57.950 INFO:teuthology.orchestra.run.smithi111.stdout: "req_renamesnap_latency": { 2024-04-21T23:28:57.950 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:57.950 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:57.950 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:57.950 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:57.950 INFO:teuthology.orchestra.run.smithi111.stdout: "req_snapdiff_latency": { 2024-04-21T23:28:57.950 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:57.950 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:57.950 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:57.951 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:57.951 INFO:teuthology.orchestra.run.smithi111.stdout: "req_rmdir_latency": { 2024-04-21T23:28:57.951 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 22, 2024-04-21T23:28:57.951 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.774047647, 2024-04-21T23:28:57.951 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.035183983 2024-04-21T23:28:57.951 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:57.951 INFO:teuthology.orchestra.run.smithi111.stdout: "req_rmsnap_latency": { 2024-04-21T23:28:57.951 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:57.952 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:57.952 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:57.952 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:57.952 INFO:teuthology.orchestra.run.smithi111.stdout: "req_rmxattr_latency": { 2024-04-21T23:28:57.952 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:57.952 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:57.952 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:57.952 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:57.952 INFO:teuthology.orchestra.run.smithi111.stdout: "req_setattr_latency": { 2024-04-21T23:28:57.953 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 8, 2024-04-21T23:28:57.953 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.045672926, 2024-04-21T23:28:57.953 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.005709115 2024-04-21T23:28:57.953 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:57.953 INFO:teuthology.orchestra.run.smithi111.stdout: "req_setdirlayout_latency": { 2024-04-21T23:28:57.953 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:57.953 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:57.953 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:57.953 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:57.954 INFO:teuthology.orchestra.run.smithi111.stdout: "req_setfilelock_latency": { 2024-04-21T23:28:57.954 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 8, 2024-04-21T23:28:57.954 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.002631224, 2024-04-21T23:28:57.954 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000328903 2024-04-21T23:28:57.954 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:57.954 INFO:teuthology.orchestra.run.smithi111.stdout: "req_setlayout_latency": { 2024-04-21T23:28:57.954 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:57.954 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:57.955 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:57.955 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:57.955 INFO:teuthology.orchestra.run.smithi111.stdout: "req_setxattr_latency": { 2024-04-21T23:28:57.955 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:57.955 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:57.955 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:57.955 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:57.955 INFO:teuthology.orchestra.run.smithi111.stdout: "req_symlink_latency": { 2024-04-21T23:28:57.955 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 6, 2024-04-21T23:28:57.956 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.005226557, 2024-04-21T23:28:57.956 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000871092 2024-04-21T23:28:57.956 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:57.956 INFO:teuthology.orchestra.run.smithi111.stdout: "req_unlink_latency": { 2024-04-21T23:28:57.956 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 3173, 2024-04-21T23:28:57.957 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 10.096937535, 2024-04-21T23:28:57.957 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.003182142 2024-04-21T23:28:57.957 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:57.957 INFO:teuthology.orchestra.run.smithi111.stdout: "cap_revoke_eviction": 0, 2024-04-21T23:28:57.957 INFO:teuthology.orchestra.run.smithi111.stdout: "cap_acquisition_throttle": 0, 2024-04-21T23:28:57.957 INFO:teuthology.orchestra.run.smithi111.stdout: "req_getvxattr_latency": { 2024-04-21T23:28:57.957 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:57.957 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:57.958 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:57.958 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-21T23:28:57.958 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:57.958 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_sessions": { 2024-04-21T23:28:57.958 INFO:teuthology.orchestra.run.smithi111.stdout: "session_count": 2, 2024-04-21T23:28:57.958 INFO:teuthology.orchestra.run.smithi111.stdout: "session_add": 6, 2024-04-21T23:28:57.958 INFO:teuthology.orchestra.run.smithi111.stdout: "session_remove": 4, 2024-04-21T23:28:57.959 INFO:teuthology.orchestra.run.smithi111.stdout: "sessions_open": 2, 2024-04-21T23:28:57.959 INFO:teuthology.orchestra.run.smithi111.stdout: "sessions_stale": 0, 2024-04-21T23:28:57.959 INFO:teuthology.orchestra.run.smithi111.stdout: "total_load": 6830, 2024-04-21T23:28:57.959 INFO:teuthology.orchestra.run.smithi111.stdout: "average_load": 3415, 2024-04-21T23:28:57.959 INFO:teuthology.orchestra.run.smithi111.stdout: "avg_session_uptime": 613, 2024-04-21T23:28:57.959 INFO:teuthology.orchestra.run.smithi111.stdout: "mdthresh_evicted": 0 2024-04-21T23:28:57.959 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:57.959 INFO:teuthology.orchestra.run.smithi111.stdout: "mempool": { 2024-04-21T23:28:57.959 INFO:teuthology.orchestra.run.smithi111.stdout: "bloom_filter_bytes": 0, 2024-04-21T23:28:57.960 INFO:teuthology.orchestra.run.smithi111.stdout: "bloom_filter_items": 0, 2024-04-21T23:28:57.960 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_alloc_bytes": 0, 2024-04-21T23:28:57.960 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_alloc_items": 0, 2024-04-21T23:28:57.960 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_data_bytes": 0, 2024-04-21T23:28:57.960 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_data_items": 0, 2024-04-21T23:28:57.960 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-21T23:28:57.960 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_onode_items": 0, 2024-04-21T23:28:57.960 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-21T23:28:57.960 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_meta_items": 0, 2024-04-21T23:28:57.961 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_other_bytes": 0, 2024-04-21T23:28:57.961 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_other_items": 0, 2024-04-21T23:28:57.961 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-21T23:28:57.961 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_buffer_items": 0, 2024-04-21T23:28:57.961 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_extent_bytes": 0, 2024-04-21T23:28:57.961 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_extent_items": 0, 2024-04-21T23:28:57.961 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_blob_bytes": 0, 2024-04-21T23:28:57.961 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_blob_items": 0, 2024-04-21T23:28:57.961 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-21T23:28:57.962 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_shared_blob_items": 0, 2024-04-21T23:28:57.962 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-21T23:28:57.962 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_inline_bl_items": 0, 2024-04-21T23:28:57.962 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_fsck_bytes": 0, 2024-04-21T23:28:57.962 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_fsck_items": 0, 2024-04-21T23:28:57.962 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_txc_bytes": 0, 2024-04-21T23:28:57.962 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_txc_items": 0, 2024-04-21T23:28:57.962 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-21T23:28:57.963 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_deferred_items": 0, 2024-04-21T23:28:57.963 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_bytes": 0, 2024-04-21T23:28:57.963 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_items": 0, 2024-04-21T23:28:57.963 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_bytes": 0, 2024-04-21T23:28:57.963 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_items": 0, 2024-04-21T23:28:57.963 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_reader_bytes": 0, 2024-04-21T23:28:57.963 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_reader_items": 0, 2024-04-21T23:28:57.963 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_writer_bytes": 0, 2024-04-21T23:28:57.963 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_writer_items": 0, 2024-04-21T23:28:57.964 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_anon_bytes": 548229, 2024-04-21T23:28:57.964 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_anon_items": 6016, 2024-04-21T23:28:57.964 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_meta_bytes": 1232, 2024-04-21T23:28:57.964 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_meta_items": 14, 2024-04-21T23:28:57.964 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_bytes": 0, 2024-04-21T23:28:57.964 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_items": 0, 2024-04-21T23:28:57.964 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_mapbl_bytes": 0, 2024-04-21T23:28:57.965 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_mapbl_items": 0, 2024-04-21T23:28:57.965 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pglog_bytes": 0, 2024-04-21T23:28:57.965 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pglog_items": 0, 2024-04-21T23:28:57.965 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_bytes": 9704, 2024-04-21T23:28:57.965 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_items": 186, 2024-04-21T23:28:57.965 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_mapping_bytes": 0, 2024-04-21T23:28:57.965 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_mapping_items": 0, 2024-04-21T23:28:57.965 INFO:teuthology.orchestra.run.smithi111.stdout: "pgmap_bytes": 0, 2024-04-21T23:28:57.966 INFO:teuthology.orchestra.run.smithi111.stdout: "pgmap_items": 0, 2024-04-21T23:28:57.966 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_co_bytes": 25346909, 2024-04-21T23:28:57.966 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_co_items": 263734, 2024-04-21T23:28:57.966 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_1_bytes": 0, 2024-04-21T23:28:57.966 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_1_items": 0, 2024-04-21T23:28:57.966 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_2_bytes": 0, 2024-04-21T23:28:57.966 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_2_items": 0 2024-04-21T23:28:57.966 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:57.967 INFO:teuthology.orchestra.run.smithi111.stdout: "objecter": { 2024-04-21T23:28:57.967 INFO:teuthology.orchestra.run.smithi111.stdout: "op_active": 0, 2024-04-21T23:28:57.967 INFO:teuthology.orchestra.run.smithi111.stdout: "op_laggy": 0, 2024-04-21T23:28:57.967 INFO:teuthology.orchestra.run.smithi111.stdout: "op_send": 10556, 2024-04-21T23:28:57.967 INFO:teuthology.orchestra.run.smithi111.stdout: "op_send_bytes": 71348208, 2024-04-21T23:28:57.967 INFO:teuthology.orchestra.run.smithi111.stdout: "op_resend": 28, 2024-04-21T23:28:57.967 INFO:teuthology.orchestra.run.smithi111.stdout: "op_reply": 10556, 2024-04-21T23:28:57.967 INFO:teuthology.orchestra.run.smithi111.stdout: "op_latency": { 2024-04-21T23:28:57.967 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 10556, 2024-04-21T23:28:57.968 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 35.450386145, 2024-04-21T23:28:57.968 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.003358316 2024-04-21T23:28:57.968 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:57.968 INFO:teuthology.orchestra.run.smithi111.stdout: "op_inflight": 0, 2024-04-21T23:28:57.968 INFO:teuthology.orchestra.run.smithi111.stdout: "oplen_avg": { 2024-04-21T23:28:57.968 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 10556, 2024-04-21T23:28:57.968 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 10801 2024-04-21T23:28:57.968 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:57.968 INFO:teuthology.orchestra.run.smithi111.stdout: "op": 10556, 2024-04-21T23:28:57.969 INFO:teuthology.orchestra.run.smithi111.stdout: "op_r": 3281, 2024-04-21T23:28:57.969 INFO:teuthology.orchestra.run.smithi111.stdout: "op_w": 7275, 2024-04-21T23:28:57.969 INFO:teuthology.orchestra.run.smithi111.stdout: "op_rmw": 0, 2024-04-21T23:28:57.969 INFO:teuthology.orchestra.run.smithi111.stdout: "op_pg": 0, 2024-04-21T23:28:57.969 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_stat": 0, 2024-04-21T23:28:57.969 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_create": 10, 2024-04-21T23:28:57.969 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_read": 25, 2024-04-21T23:28:57.969 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_write": 533, 2024-04-21T23:28:57.969 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_writefull": 23, 2024-04-21T23:28:57.970 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_writesame": 0, 2024-04-21T23:28:57.970 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_append": 0, 2024-04-21T23:28:57.970 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_zero": 0, 2024-04-21T23:28:57.970 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_truncate": 3, 2024-04-21T23:28:57.970 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_delete": 6480, 2024-04-21T23:28:57.970 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_mapext": 0, 2024-04-21T23:28:57.970 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_sparse_read": 0, 2024-04-21T23:28:57.970 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_clonerange": 0, 2024-04-21T23:28:57.970 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_getxattr": 3256, 2024-04-21T23:28:57.971 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_setxattr": 20, 2024-04-21T23:28:57.971 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_cmpxattr": 0, 2024-04-21T23:28:57.971 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_rmxattr": 0, 2024-04-21T23:28:57.971 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_resetxattrs": 0, 2024-04-21T23:28:57.971 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_call": 0, 2024-04-21T23:28:57.971 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_watch": 0, 2024-04-21T23:28:57.971 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_notify": 0, 2024-04-21T23:28:57.972 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_src_cmpxattr": 0, 2024-04-21T23:28:57.972 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_pgls": 0, 2024-04-21T23:28:57.972 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_pgls_filter": 0, 2024-04-21T23:28:57.972 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_other": 8, 2024-04-21T23:28:57.972 INFO:teuthology.orchestra.run.smithi111.stdout: "linger_active": 0, 2024-04-21T23:28:57.972 INFO:teuthology.orchestra.run.smithi111.stdout: "linger_send": 0, 2024-04-21T23:28:57.972 INFO:teuthology.orchestra.run.smithi111.stdout: "linger_resend": 0, 2024-04-21T23:28:57.972 INFO:teuthology.orchestra.run.smithi111.stdout: "linger_ping": 0, 2024-04-21T23:28:57.972 INFO:teuthology.orchestra.run.smithi111.stdout: "poolop_active": 0, 2024-04-21T23:28:57.973 INFO:teuthology.orchestra.run.smithi111.stdout: "poolop_send": 0, 2024-04-21T23:28:57.973 INFO:teuthology.orchestra.run.smithi111.stdout: "poolop_resend": 0, 2024-04-21T23:28:57.973 INFO:teuthology.orchestra.run.smithi111.stdout: "poolstat_active": 0, 2024-04-21T23:28:57.973 INFO:teuthology.orchestra.run.smithi111.stdout: "poolstat_send": 0, 2024-04-21T23:28:57.973 INFO:teuthology.orchestra.run.smithi111.stdout: "poolstat_resend": 0, 2024-04-21T23:28:57.973 INFO:teuthology.orchestra.run.smithi111.stdout: "statfs_active": 0, 2024-04-21T23:28:57.973 INFO:teuthology.orchestra.run.smithi111.stdout: "statfs_send": 0, 2024-04-21T23:28:57.973 INFO:teuthology.orchestra.run.smithi111.stdout: "statfs_resend": 0, 2024-04-21T23:28:57.973 INFO:teuthology.orchestra.run.smithi111.stdout: "command_active": 0, 2024-04-21T23:28:57.974 INFO:teuthology.orchestra.run.smithi111.stdout: "command_send": 0, 2024-04-21T23:28:57.974 INFO:teuthology.orchestra.run.smithi111.stdout: "command_resend": 0, 2024-04-21T23:28:57.974 INFO:teuthology.orchestra.run.smithi111.stdout: "map_epoch": 103, 2024-04-21T23:28:57.974 INFO:teuthology.orchestra.run.smithi111.stdout: "map_full": 0, 2024-04-21T23:28:57.974 INFO:teuthology.orchestra.run.smithi111.stdout: "map_inc": 13, 2024-04-21T23:28:57.974 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_sessions": 12, 2024-04-21T23:28:57.974 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_session_open": 12, 2024-04-21T23:28:57.974 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_session_close": 0, 2024-04-21T23:28:57.975 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_laggy": 0, 2024-04-21T23:28:57.975 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_wr": 417, 2024-04-21T23:28:57.975 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_rd": 0, 2024-04-21T23:28:57.975 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_del": 26 2024-04-21T23:28:57.975 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:57.975 INFO:teuthology.orchestra.run.smithi111.stdout: "oft": { 2024-04-21T23:28:57.975 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_total_objs": 1, 2024-04-21T23:28:57.975 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_total_kv_pairs": 0, 2024-04-21T23:28:57.975 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_total_updates": 0, 2024-04-21T23:28:57.976 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_total_removes": 0 2024-04-21T23:28:57.976 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:57.976 INFO:teuthology.orchestra.run.smithi111.stdout: "osd-slow-ops": { 2024-04-21T23:28:57.976 INFO:teuthology.orchestra.run.smithi111.stdout: "slow_ops_count": 2 2024-04-21T23:28:57.976 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:57.976 INFO:teuthology.orchestra.run.smithi111.stdout: "purge_queue": { 2024-04-21T23:28:57.976 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executing_ops": 0, 2024-04-21T23:28:57.976 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executing_ops_high_water": 17, 2024-04-21T23:28:57.977 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executing": 0, 2024-04-21T23:28:57.977 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executing_high_water": 10, 2024-04-21T23:28:57.977 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executed_ops": 6428, 2024-04-21T23:28:57.977 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executed": 3219, 2024-04-21T23:28:57.977 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_item_in_journal": 0 2024-04-21T23:28:57.977 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:57.977 INFO:teuthology.orchestra.run.smithi111.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-21T23:28:57.977 INFO:teuthology.orchestra.run.smithi111.stdout: "val": 0, 2024-04-21T23:28:57.977 INFO:teuthology.orchestra.run.smithi111.stdout: "max": 104857600, 2024-04-21T23:28:57.978 INFO:teuthology.orchestra.run.smithi111.stdout: "get_started": 0, 2024-04-21T23:28:57.978 INFO:teuthology.orchestra.run.smithi111.stdout: "get": 39426, 2024-04-21T23:28:57.978 INFO:teuthology.orchestra.run.smithi111.stdout: "get_sum": 12900410, 2024-04-21T23:28:57.978 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_fail": 0, 2024-04-21T23:28:57.978 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_success": 39426, 2024-04-21T23:28:57.978 INFO:teuthology.orchestra.run.smithi111.stdout: "take": 0, 2024-04-21T23:28:57.978 INFO:teuthology.orchestra.run.smithi111.stdout: "take_sum": 0, 2024-04-21T23:28:57.978 INFO:teuthology.orchestra.run.smithi111.stdout: "put": 39426, 2024-04-21T23:28:57.978 INFO:teuthology.orchestra.run.smithi111.stdout: "put_sum": 12900410, 2024-04-21T23:28:57.979 INFO:teuthology.orchestra.run.smithi111.stdout: "wait": { 2024-04-21T23:28:57.979 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:57.979 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:57.979 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:57.979 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-21T23:28:57.979 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:57.979 INFO:teuthology.orchestra.run.smithi111.stdout: "throttle-objecter_bytes": { 2024-04-21T23:28:57.979 INFO:teuthology.orchestra.run.smithi111.stdout: "val": 0, 2024-04-21T23:28:57.980 INFO:teuthology.orchestra.run.smithi111.stdout: "max": 104857600, 2024-04-21T23:28:57.980 INFO:teuthology.orchestra.run.smithi111.stdout: "get_started": 0, 2024-04-21T23:28:57.980 INFO:teuthology.orchestra.run.smithi111.stdout: "get": 0, 2024-04-21T23:28:57.980 INFO:teuthology.orchestra.run.smithi111.stdout: "get_sum": 0, 2024-04-21T23:28:57.980 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_fail": 0, 2024-04-21T23:28:57.980 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_success": 0, 2024-04-21T23:28:57.980 INFO:teuthology.orchestra.run.smithi111.stdout: "take": 10556, 2024-04-21T23:28:57.980 INFO:teuthology.orchestra.run.smithi111.stdout: "take_sum": 71718967, 2024-04-21T23:28:57.980 INFO:teuthology.orchestra.run.smithi111.stdout: "put": 10556, 2024-04-21T23:28:57.981 INFO:teuthology.orchestra.run.smithi111.stdout: "put_sum": 71718967, 2024-04-21T23:28:57.981 INFO:teuthology.orchestra.run.smithi111.stdout: "wait": { 2024-04-21T23:28:57.981 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:57.981 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:57.981 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:57.981 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-21T23:28:57.982 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:57.982 INFO:teuthology.orchestra.run.smithi111.stdout: "throttle-objecter_ops": { 2024-04-21T23:28:57.982 INFO:teuthology.orchestra.run.smithi111.stdout: "val": 0, 2024-04-21T23:28:57.982 INFO:teuthology.orchestra.run.smithi111.stdout: "max": 1024, 2024-04-21T23:28:57.982 INFO:teuthology.orchestra.run.smithi111.stdout: "get_started": 0, 2024-04-21T23:28:57.982 INFO:teuthology.orchestra.run.smithi111.stdout: "get": 0, 2024-04-21T23:28:57.982 INFO:teuthology.orchestra.run.smithi111.stdout: "get_sum": 0, 2024-04-21T23:28:57.982 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_fail": 0, 2024-04-21T23:28:57.983 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_success": 0, 2024-04-21T23:28:57.983 INFO:teuthology.orchestra.run.smithi111.stdout: "take": 10556, 2024-04-21T23:28:57.983 INFO:teuthology.orchestra.run.smithi111.stdout: "take_sum": 10556, 2024-04-21T23:28:57.983 INFO:teuthology.orchestra.run.smithi111.stdout: "put": 10556, 2024-04-21T23:28:57.983 INFO:teuthology.orchestra.run.smithi111.stdout: "put_sum": 10556, 2024-04-21T23:28:57.983 INFO:teuthology.orchestra.run.smithi111.stdout: "wait": { 2024-04-21T23:28:57.984 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:57.984 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:57.984 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:57.984 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-21T23:28:57.984 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:57.984 INFO:teuthology.orchestra.run.smithi111.stdout: "throttle-write_buf_throttle": { 2024-04-21T23:28:57.984 INFO:teuthology.orchestra.run.smithi111.stdout: "val": 0, 2024-04-21T23:28:57.984 INFO:teuthology.orchestra.run.smithi111.stdout: "max": 3758096384, 2024-04-21T23:28:57.985 INFO:teuthology.orchestra.run.smithi111.stdout: "get_started": 0, 2024-04-21T23:28:57.985 INFO:teuthology.orchestra.run.smithi111.stdout: "get": 3219, 2024-04-21T23:28:57.985 INFO:teuthology.orchestra.run.smithi111.stdout: "get_sum": 370759, 2024-04-21T23:28:57.985 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_fail": 0, 2024-04-21T23:28:57.985 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_success": 3219, 2024-04-21T23:28:57.985 INFO:teuthology.orchestra.run.smithi111.stdout: "take": 0, 2024-04-21T23:28:57.985 INFO:teuthology.orchestra.run.smithi111.stdout: "take_sum": 0, 2024-04-21T23:28:57.985 INFO:teuthology.orchestra.run.smithi111.stdout: "put": 25, 2024-04-21T23:28:57.986 INFO:teuthology.orchestra.run.smithi111.stdout: "put_sum": 370759, 2024-04-21T23:28:57.986 INFO:teuthology.orchestra.run.smithi111.stdout: "wait": { 2024-04-21T23:28:57.986 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:57.986 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:57.986 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:57.986 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-21T23:28:57.986 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:28:57.986 INFO:teuthology.orchestra.run.smithi111.stdout: "throttle-write_buf_throttle-0x5572f2dd0500": { 2024-04-21T23:28:57.987 INFO:teuthology.orchestra.run.smithi111.stdout: "val": 0, 2024-04-21T23:28:57.987 INFO:teuthology.orchestra.run.smithi111.stdout: "max": 3758096384, 2024-04-21T23:28:57.987 INFO:teuthology.orchestra.run.smithi111.stdout: "get_started": 0, 2024-04-21T23:28:57.987 INFO:teuthology.orchestra.run.smithi111.stdout: "get": 24888, 2024-04-21T23:28:57.987 INFO:teuthology.orchestra.run.smithi111.stdout: "get_sum": 68308711, 2024-04-21T23:28:57.987 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_fail": 0, 2024-04-21T23:28:57.987 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_success": 24888, 2024-04-21T23:28:57.987 INFO:teuthology.orchestra.run.smithi111.stdout: "take": 0, 2024-04-21T23:28:57.987 INFO:teuthology.orchestra.run.smithi111.stdout: "take_sum": 0, 2024-04-21T23:28:57.988 INFO:teuthology.orchestra.run.smithi111.stdout: "put": 508, 2024-04-21T23:28:57.988 INFO:teuthology.orchestra.run.smithi111.stdout: "put_sum": 68308711, 2024-04-21T23:28:57.988 INFO:teuthology.orchestra.run.smithi111.stdout: "wait": { 2024-04-21T23:28:57.988 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:28:57.988 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:28:57.988 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:28:57.988 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-21T23:28:57.988 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-21T23:28:57.989 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-21T23:28:58.741 INFO:tasks.check_counter:Daemon mds.h mds.exported=9 2024-04-21T23:28:58.742 INFO:tasks.check_counter:Daemon mds.h mds.imported=12 2024-04-21T23:28:58.742 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=2 2024-04-21T23:28:58.742 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=0 2024-04-21T23:28:58.742 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=0 2024-04-21T23:28:58.742 INFO:tasks.check_counter:Daemon mds.h mds.dir_split=19 2024-04-21T23:28:58.742 DEBUG:tasks.check_counter:Getting stats from k 2024-04-21T23:28:58.743 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph daemon mds.k perf dump 2024-04-21T23:28:58.981 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.b/config 2024-04-21T23:28:59.102 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:28:58 smithi111 ceph-mon[31085]: pgmap v328: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:28:59.181 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:28:58 smithi040 ceph-mon[27984]: pgmap v328: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:28:59.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:28:58 smithi174 ceph-mon[31409]: pgmap v328: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:29:00.701 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-21T23:29:00.702 INFO:teuthology.orchestra.run.smithi111.stdout: "AsyncMessenger::Worker-0": { 2024-04-21T23:29:00.702 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_messages": 0, 2024-04-21T23:29:00.702 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages": 2, 2024-04-21T23:29:00.702 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_bytes": 0, 2024-04-21T23:29:00.702 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_bytes": 0, 2024-04-21T23:29:00.702 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_created_connections": 4, 2024-04-21T23:29:00.702 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_active_connections": 0, 2024-04-21T23:29:00.702 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_total_time": 0.000968139, 2024-04-21T23:29:00.703 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_send_time": 0.000000000, 2024-04-21T23:29:00.703 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_recv_time": 0.000201308, 2024-04-21T23:29:00.703 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T23:29:00.703 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T23:29:00.703 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:29:00.703 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:29:00.703 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:29:00.703 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:29:00.704 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_handle_ack_lat": { 2024-04-21T23:29:00.704 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:29:00.704 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:29:00.704 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:29:00.704 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:29:00.704 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-21T23:29:00.704 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_encrypted_bytes": 0 2024-04-21T23:29:00.704 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:29:00.704 INFO:teuthology.orchestra.run.smithi111.stdout: "AsyncMessenger::Worker-1": { 2024-04-21T23:29:00.705 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_messages": 231, 2024-04-21T23:29:00.705 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages": 220, 2024-04-21T23:29:00.705 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_bytes": 1583440, 2024-04-21T23:29:00.705 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_bytes": 135104, 2024-04-21T23:29:00.705 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_created_connections": 2, 2024-04-21T23:29:00.705 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_active_connections": 1, 2024-04-21T23:29:00.705 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_total_time": 0.045188554, 2024-04-21T23:29:00.705 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_send_time": 0.013685620, 2024-04-21T23:29:00.705 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_recv_time": 0.035461748, 2024-04-21T23:29:00.706 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_fast_dispatch_time": 0.005465969, 2024-04-21T23:29:00.706 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T23:29:00.706 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 219, 2024-04-21T23:29:00.706 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.010543558, 2024-04-21T23:29:00.706 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000048144 2024-04-21T23:29:00.706 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:29:00.706 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_handle_ack_lat": { 2024-04-21T23:29:00.707 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:29:00.707 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:29:00.707 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:29:00.707 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:29:00.707 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_encrypted_bytes": 1583440, 2024-04-21T23:29:00.707 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_encrypted_bytes": 135104 2024-04-21T23:29:00.707 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:29:00.708 INFO:teuthology.orchestra.run.smithi111.stdout: "AsyncMessenger::Worker-2": { 2024-04-21T23:29:00.708 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_messages": 6, 2024-04-21T23:29:00.708 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages": 170, 2024-04-21T23:29:00.708 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_bytes": 3344, 2024-04-21T23:29:00.708 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_bytes": 185424, 2024-04-21T23:29:00.708 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_created_connections": 5, 2024-04-21T23:29:00.708 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_active_connections": 3, 2024-04-21T23:29:00.708 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_total_time": 0.011825824, 2024-04-21T23:29:00.709 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_send_time": 0.007795639, 2024-04-21T23:29:00.709 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_recv_time": 0.001823689, 2024-04-21T23:29:00.709 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T23:29:00.709 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T23:29:00.709 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 168, 2024-04-21T23:29:00.709 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.019624870, 2024-04-21T23:29:00.709 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000116814 2024-04-21T23:29:00.709 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:29:00.709 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_handle_ack_lat": { 2024-04-21T23:29:00.710 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:29:00.710 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:29:00.710 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:29:00.710 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:29:00.710 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_encrypted_bytes": 3344, 2024-04-21T23:29:00.710 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_encrypted_bytes": 185424 2024-04-21T23:29:00.710 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:29:00.710 INFO:teuthology.orchestra.run.smithi111.stdout: "cct": { 2024-04-21T23:29:00.711 INFO:teuthology.orchestra.run.smithi111.stdout: "total_workers": 0, 2024-04-21T23:29:00.711 INFO:teuthology.orchestra.run.smithi111.stdout: "unhealthy_workers": 0 2024-04-21T23:29:00.711 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:29:00.711 INFO:teuthology.orchestra.run.smithi111.stdout: "mempool": { 2024-04-21T23:29:00.711 INFO:teuthology.orchestra.run.smithi111.stdout: "bloom_filter_bytes": 0, 2024-04-21T23:29:00.711 INFO:teuthology.orchestra.run.smithi111.stdout: "bloom_filter_items": 0, 2024-04-21T23:29:00.711 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_alloc_bytes": 0, 2024-04-21T23:29:00.711 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_alloc_items": 0, 2024-04-21T23:29:00.711 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_data_bytes": 0, 2024-04-21T23:29:00.712 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_data_items": 0, 2024-04-21T23:29:00.712 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-21T23:29:00.712 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_onode_items": 0, 2024-04-21T23:29:00.712 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-21T23:29:00.712 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_meta_items": 0, 2024-04-21T23:29:00.712 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_other_bytes": 0, 2024-04-21T23:29:00.713 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_other_items": 0, 2024-04-21T23:29:00.713 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-21T23:29:00.713 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_buffer_items": 0, 2024-04-21T23:29:00.713 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_extent_bytes": 0, 2024-04-21T23:29:00.713 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_extent_items": 0, 2024-04-21T23:29:00.713 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_blob_bytes": 0, 2024-04-21T23:29:00.713 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_blob_items": 0, 2024-04-21T23:29:00.713 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-21T23:29:00.714 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_shared_blob_items": 0, 2024-04-21T23:29:00.714 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-21T23:29:00.714 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_inline_bl_items": 0, 2024-04-21T23:29:00.714 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_fsck_bytes": 0, 2024-04-21T23:29:00.714 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_fsck_items": 0, 2024-04-21T23:29:00.714 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_txc_bytes": 0, 2024-04-21T23:29:00.714 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_txc_items": 0, 2024-04-21T23:29:00.714 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-21T23:29:00.714 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_deferred_items": 0, 2024-04-21T23:29:00.715 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_bytes": 0, 2024-04-21T23:29:00.715 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_items": 0, 2024-04-21T23:29:00.715 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_bytes": 0, 2024-04-21T23:29:00.715 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_items": 0, 2024-04-21T23:29:00.715 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_reader_bytes": 0, 2024-04-21T23:29:00.715 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_reader_items": 0, 2024-04-21T23:29:00.715 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_writer_bytes": 0, 2024-04-21T23:29:00.715 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_writer_items": 0, 2024-04-21T23:29:00.716 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_anon_bytes": 148732, 2024-04-21T23:29:00.716 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_anon_items": 31, 2024-04-21T23:29:00.716 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_meta_bytes": 0, 2024-04-21T23:29:00.716 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_meta_items": 0, 2024-04-21T23:29:00.716 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_bytes": 0, 2024-04-21T23:29:00.716 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_items": 0, 2024-04-21T23:29:00.716 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_mapbl_bytes": 0, 2024-04-21T23:29:00.716 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_mapbl_items": 0, 2024-04-21T23:29:00.716 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pglog_bytes": 0, 2024-04-21T23:29:00.717 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pglog_items": 0, 2024-04-21T23:29:00.717 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_bytes": 0, 2024-04-21T23:29:00.717 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_items": 0, 2024-04-21T23:29:00.717 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_mapping_bytes": 0, 2024-04-21T23:29:00.717 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_mapping_items": 0, 2024-04-21T23:29:00.717 INFO:teuthology.orchestra.run.smithi111.stdout: "pgmap_bytes": 0, 2024-04-21T23:29:00.717 INFO:teuthology.orchestra.run.smithi111.stdout: "pgmap_items": 0, 2024-04-21T23:29:00.718 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_co_bytes": 928, 2024-04-21T23:29:00.718 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_co_items": 2, 2024-04-21T23:29:00.718 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_1_bytes": 0, 2024-04-21T23:29:00.718 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_1_items": 0, 2024-04-21T23:29:00.718 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_2_bytes": 0, 2024-04-21T23:29:00.718 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_2_items": 0 2024-04-21T23:29:00.718 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-21T23:29:00.719 INFO:teuthology.orchestra.run.smithi111.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-21T23:29:00.719 INFO:teuthology.orchestra.run.smithi111.stdout: "val": 0, 2024-04-21T23:29:00.719 INFO:teuthology.orchestra.run.smithi111.stdout: "max": 104857600, 2024-04-21T23:29:00.719 INFO:teuthology.orchestra.run.smithi111.stdout: "get_started": 0, 2024-04-21T23:29:00.719 INFO:teuthology.orchestra.run.smithi111.stdout: "get": 233, 2024-04-21T23:29:00.719 INFO:teuthology.orchestra.run.smithi111.stdout: "get_sum": 1550890, 2024-04-21T23:29:00.719 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_fail": 0, 2024-04-21T23:29:00.719 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_success": 233, 2024-04-21T23:29:00.719 INFO:teuthology.orchestra.run.smithi111.stdout: "take": 0, 2024-04-21T23:29:00.720 INFO:teuthology.orchestra.run.smithi111.stdout: "take_sum": 0, 2024-04-21T23:29:00.720 INFO:teuthology.orchestra.run.smithi111.stdout: "put": 233, 2024-04-21T23:29:00.720 INFO:teuthology.orchestra.run.smithi111.stdout: "put_sum": 1550890, 2024-04-21T23:29:00.720 INFO:teuthology.orchestra.run.smithi111.stdout: "wait": { 2024-04-21T23:29:00.720 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-21T23:29:00.720 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-21T23:29:00.720 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-21T23:29:00.720 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-21T23:29:00.720 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-21T23:29:00.721 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-21T23:29:01.127 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:29:00 smithi111 ceph-mon[31085]: pgmap v329: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:29:01.181 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:29:00 smithi040 ceph-mon[27984]: pgmap v329: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:29:01.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:29:00 smithi174 ceph-mon[31409]: pgmap v329: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-21T23:29:01.387 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.k 2024-04-21T23:29:01.387 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.k 2024-04-21T23:29:01.387 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.k 2024-04-21T23:29:01.387 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.k 2024-04-21T23:29:01.387 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.k 2024-04-21T23:29:01.387 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.k 2024-04-21T23:29:01.387 DEBUG:tasks.check_counter:Getting stats from c 2024-04-21T23:29:01.388 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph daemon mds.c perf dump 2024-04-21T23:29:01.640 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.c/config 2024-04-21T23:29:03.048 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:29:02 smithi174 ceph-mon[31409]: pgmap v330: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:29:03.181 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:29:02 smithi040 ceph-mon[27984]: pgmap v330: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:29:03.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:29:02 smithi111 ceph-mon[31085]: pgmap v330: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:29:03.464 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-04-21T23:29:03.464 INFO:teuthology.orchestra.run.smithi174.stdout: "AsyncMessenger::Worker-0": { 2024-04-21T23:29:03.464 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_messages": 1011, 2024-04-21T23:29:03.465 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages": 1013, 2024-04-21T23:29:03.465 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_bytes": 4515509, 2024-04-21T23:29:03.465 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_bytes": 321508, 2024-04-21T23:29:03.465 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_created_connections": 6, 2024-04-21T23:29:03.465 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_active_connections": 2, 2024-04-21T23:29:03.465 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_total_time": 0.120528118, 2024-04-21T23:29:03.465 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_send_time": 0.035904091, 2024-04-21T23:29:03.465 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_recv_time": 0.033761301, 2024-04-21T23:29:03.466 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_fast_dispatch_time": 0.038794225, 2024-04-21T23:29:03.466 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T23:29:03.466 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 1011, 2024-04-21T23:29:03.466 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.050224120, 2024-04-21T23:29:03.466 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000049677 2024-04-21T23:29:03.466 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:03.466 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_handle_ack_lat": { 2024-04-21T23:29:03.466 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:03.466 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:03.467 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:03.467 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:03.467 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-21T23:29:03.467 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_encrypted_bytes": 0 2024-04-21T23:29:03.467 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:03.467 INFO:teuthology.orchestra.run.smithi174.stdout: "AsyncMessenger::Worker-1": { 2024-04-21T23:29:03.467 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_messages": 6, 2024-04-21T23:29:03.467 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages": 169, 2024-04-21T23:29:03.468 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_bytes": 1976, 2024-04-21T23:29:03.468 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_bytes": 446746, 2024-04-21T23:29:03.468 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_created_connections": 7, 2024-04-21T23:29:03.468 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_active_connections": 5, 2024-04-21T23:29:03.468 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_total_time": 0.010692758, 2024-04-21T23:29:03.468 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_send_time": 0.006720691, 2024-04-21T23:29:03.468 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_recv_time": 0.002153924, 2024-04-21T23:29:03.468 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_fast_dispatch_time": 0.000083435, 2024-04-21T23:29:03.468 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T23:29:03.469 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 166, 2024-04-21T23:29:03.469 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.013932500, 2024-04-21T23:29:03.469 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000083930 2024-04-21T23:29:03.469 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:03.469 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_handle_ack_lat": { 2024-04-21T23:29:03.469 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:03.469 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:03.469 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:03.469 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:03.470 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-21T23:29:03.470 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_encrypted_bytes": 445408 2024-04-21T23:29:03.470 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:03.470 INFO:teuthology.orchestra.run.smithi174.stdout: "AsyncMessenger::Worker-2": { 2024-04-21T23:29:03.470 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_messages": 1606, 2024-04-21T23:29:03.470 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages": 1584, 2024-04-21T23:29:03.470 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_bytes": 4693435, 2024-04-21T23:29:03.470 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_bytes": 564941, 2024-04-21T23:29:03.471 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_created_connections": 4, 2024-04-21T23:29:03.471 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_active_connections": 3, 2024-04-21T23:29:03.471 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_total_time": 0.152408025, 2024-04-21T23:29:03.471 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_send_time": 0.053372444, 2024-04-21T23:29:03.471 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_recv_time": 0.060420619, 2024-04-21T23:29:03.471 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_fast_dispatch_time": 0.039216576, 2024-04-21T23:29:03.471 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T23:29:03.471 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 1584, 2024-04-21T23:29:03.471 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.072722582, 2024-04-21T23:29:03.472 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000045910 2024-04-21T23:29:03.472 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:03.472 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_handle_ack_lat": { 2024-04-21T23:29:03.472 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:03.472 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:03.472 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:03.472 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:03.472 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_encrypted_bytes": 1649024, 2024-04-21T23:29:03.473 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_encrypted_bytes": 136912 2024-04-21T23:29:03.473 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:03.473 INFO:teuthology.orchestra.run.smithi174.stdout: "cct": { 2024-04-21T23:29:03.473 INFO:teuthology.orchestra.run.smithi174.stdout: "total_workers": 1, 2024-04-21T23:29:03.473 INFO:teuthology.orchestra.run.smithi174.stdout: "unhealthy_workers": 0 2024-04-21T23:29:03.473 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:03.473 INFO:teuthology.orchestra.run.smithi174.stdout: "finisher-MDSRank": { 2024-04-21T23:29:03.473 INFO:teuthology.orchestra.run.smithi174.stdout: "queue_len": 0, 2024-04-21T23:29:03.474 INFO:teuthology.orchestra.run.smithi174.stdout: "complete_latency": { 2024-04-21T23:29:03.474 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 3942, 2024-04-21T23:29:03.474 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.122475667, 2024-04-21T23:29:03.474 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000031069 2024-04-21T23:29:03.474 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-04-21T23:29:03.474 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:03.474 INFO:teuthology.orchestra.run.smithi174.stdout: "finisher-PurgeQueue": { 2024-04-21T23:29:03.474 INFO:teuthology.orchestra.run.smithi174.stdout: "queue_len": 0, 2024-04-21T23:29:03.474 INFO:teuthology.orchestra.run.smithi174.stdout: "complete_latency": { 2024-04-21T23:29:03.475 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:03.475 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:03.475 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:03.475 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-04-21T23:29:03.475 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:03.475 INFO:teuthology.orchestra.run.smithi174.stdout: "mds": { 2024-04-21T23:29:03.475 INFO:teuthology.orchestra.run.smithi174.stdout: "request": 0, 2024-04-21T23:29:03.475 INFO:teuthology.orchestra.run.smithi174.stdout: "reply": 0, 2024-04-21T23:29:03.475 INFO:teuthology.orchestra.run.smithi174.stdout: "reply_latency": { 2024-04-21T23:29:03.476 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:03.476 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:03.476 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:03.476 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:03.476 INFO:teuthology.orchestra.run.smithi174.stdout: "slow_reply": 0, 2024-04-21T23:29:03.476 INFO:teuthology.orchestra.run.smithi174.stdout: "forward": 0, 2024-04-21T23:29:03.476 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_fetch_complete": 0, 2024-04-21T23:29:03.476 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_fetch_keys": 0, 2024-04-21T23:29:03.477 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_commit": 0, 2024-04-21T23:29:03.477 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_split": 0, 2024-04-21T23:29:03.477 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_merge": 0, 2024-04-21T23:29:03.477 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes": 48, 2024-04-21T23:29:03.477 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_top": 19, 2024-04-21T23:29:03.477 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_bottom": 29, 2024-04-21T23:29:03.477 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_pin_tail": 0, 2024-04-21T23:29:03.477 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_pinned": 13, 2024-04-21T23:29:03.478 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_expired": 0, 2024-04-21T23:29:03.478 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_with_caps": 0, 2024-04-21T23:29:03.478 INFO:teuthology.orchestra.run.smithi174.stdout: "caps": 0, 2024-04-21T23:29:03.478 INFO:teuthology.orchestra.run.smithi174.stdout: "subtrees": 4, 2024-04-21T23:29:03.478 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse": 0, 2024-04-21T23:29:03.478 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_hit": 0, 2024-04-21T23:29:03.478 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_forward": 0, 2024-04-21T23:29:03.478 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_discover": 0, 2024-04-21T23:29:03.478 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_dir_fetch": 0, 2024-04-21T23:29:03.479 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_remote_ino": 0, 2024-04-21T23:29:03.479 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_lock": 0, 2024-04-21T23:29:03.479 INFO:teuthology.orchestra.run.smithi174.stdout: "load_cent": 0, 2024-04-21T23:29:03.479 INFO:teuthology.orchestra.run.smithi174.stdout: "q": 0, 2024-04-21T23:29:03.479 INFO:teuthology.orchestra.run.smithi174.stdout: "exported": 0, 2024-04-21T23:29:03.479 INFO:teuthology.orchestra.run.smithi174.stdout: "exported_inodes": 0, 2024-04-21T23:29:03.479 INFO:teuthology.orchestra.run.smithi174.stdout: "imported": 0, 2024-04-21T23:29:03.479 INFO:teuthology.orchestra.run.smithi174.stdout: "imported_inodes": 0, 2024-04-21T23:29:03.479 INFO:teuthology.orchestra.run.smithi174.stdout: "openino_dir_fetch": 0, 2024-04-21T23:29:03.480 INFO:teuthology.orchestra.run.smithi174.stdout: "openino_backtrace_fetch": 0, 2024-04-21T23:29:03.480 INFO:teuthology.orchestra.run.smithi174.stdout: "openino_peer_discover": 0, 2024-04-21T23:29:03.480 INFO:teuthology.orchestra.run.smithi174.stdout: "root_rfiles": 3454, 2024-04-21T23:29:03.480 INFO:teuthology.orchestra.run.smithi174.stdout: "root_rbytes": 26625020, 2024-04-21T23:29:03.480 INFO:teuthology.orchestra.run.smithi174.stdout: "root_rsnaps": 6, 2024-04-21T23:29:03.480 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_backtrace_fetch": 0, 2024-04-21T23:29:03.480 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_set_tag": 0, 2024-04-21T23:29:03.480 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_backtrace_repaired": 0, 2024-04-21T23:29:03.480 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_inotable_repaired": 0, 2024-04-21T23:29:03.481 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_dir_inodes": 0, 2024-04-21T23:29:03.481 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_dir_base_inodes": 0, 2024-04-21T23:29:03.481 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_dirfrag_rstats": 0, 2024-04-21T23:29:03.481 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_file_inodes": 0, 2024-04-21T23:29:03.481 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_inode_file_caps": 0, 2024-04-21T23:29:03.481 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_revoke": 0, 2024-04-21T23:29:03.481 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_grant": 0, 2024-04-21T23:29:03.481 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_trunc": 0, 2024-04-21T23:29:03.482 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-21T23:29:03.482 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-21T23:29:03.482 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_caps": 0, 2024-04-21T23:29:03.482 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_caps_dirty": 0, 2024-04-21T23:29:03.482 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_cap_release": 0, 2024-04-21T23:29:03.482 INFO:teuthology.orchestra.run.smithi174.stdout: "process_request_cap_release": 0 2024-04-21T23:29:03.482 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:03.482 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_cache": { 2024-04-21T23:29:03.482 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_update": 0, 2024-04-21T23:29:03.483 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_update_receipt": 0, 2024-04-21T23:29:03.483 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_try_discover": 0, 2024-04-21T23:29:03.483 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_send_discover": 0, 2024-04-21T23:29:03.483 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_handle_discover": 0, 2024-04-21T23:29:03.483 INFO:teuthology.orchestra.run.smithi174.stdout: "num_strays": 0, 2024-04-21T23:29:03.483 INFO:teuthology.orchestra.run.smithi174.stdout: "num_strays_delayed": 0, 2024-04-21T23:29:03.483 INFO:teuthology.orchestra.run.smithi174.stdout: "num_strays_enqueuing": 0, 2024-04-21T23:29:03.483 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_created": 0, 2024-04-21T23:29:03.484 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_enqueued": 0, 2024-04-21T23:29:03.484 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_reintegrated": 0, 2024-04-21T23:29:03.484 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_migrated": 0, 2024-04-21T23:29:03.484 INFO:teuthology.orchestra.run.smithi174.stdout: "num_recovering_processing": 0, 2024-04-21T23:29:03.484 INFO:teuthology.orchestra.run.smithi174.stdout: "num_recovering_enqueued": 0, 2024-04-21T23:29:03.484 INFO:teuthology.orchestra.run.smithi174.stdout: "num_recovering_prioritized": 0, 2024-04-21T23:29:03.484 INFO:teuthology.orchestra.run.smithi174.stdout: "recovery_started": 0, 2024-04-21T23:29:03.484 INFO:teuthology.orchestra.run.smithi174.stdout: "recovery_completed": 0, 2024-04-21T23:29:03.484 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_quiesce_path": 0, 2024-04-21T23:29:03.485 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_quiesce_inode": 0, 2024-04-21T23:29:03.485 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_enqueue_scrub": 0, 2024-04-21T23:29:03.485 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_exportdir": 0, 2024-04-21T23:29:03.485 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_flush": 0, 2024-04-21T23:29:03.485 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_fragmentdir": 0, 2024-04-21T23:29:03.485 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_fragstats": 0, 2024-04-21T23:29:03.485 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_inodestats": 0 2024-04-21T23:29:03.485 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:03.485 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_log": { 2024-04-21T23:29:03.486 INFO:teuthology.orchestra.run.smithi174.stdout: "evlrg": 0, 2024-04-21T23:29:03.486 INFO:teuthology.orchestra.run.smithi174.stdout: "evadd": 0, 2024-04-21T23:29:03.486 INFO:teuthology.orchestra.run.smithi174.stdout: "evex": 0, 2024-04-21T23:29:03.486 INFO:teuthology.orchestra.run.smithi174.stdout: "evtrm": 0, 2024-04-21T23:29:03.486 INFO:teuthology.orchestra.run.smithi174.stdout: "ev": 3212, 2024-04-21T23:29:03.486 INFO:teuthology.orchestra.run.smithi174.stdout: "evexg": 0, 2024-04-21T23:29:03.486 INFO:teuthology.orchestra.run.smithi174.stdout: "evexd": 0, 2024-04-21T23:29:03.486 INFO:teuthology.orchestra.run.smithi174.stdout: "segadd": 0, 2024-04-21T23:29:03.487 INFO:teuthology.orchestra.run.smithi174.stdout: "segex": 0, 2024-04-21T23:29:03.487 INFO:teuthology.orchestra.run.smithi174.stdout: "segtrm": 0, 2024-04-21T23:29:03.487 INFO:teuthology.orchestra.run.smithi174.stdout: "seg": 2, 2024-04-21T23:29:03.487 INFO:teuthology.orchestra.run.smithi174.stdout: "segmjr": 11, 2024-04-21T23:29:03.487 INFO:teuthology.orchestra.run.smithi174.stdout: "segexg": 0, 2024-04-21T23:29:03.487 INFO:teuthology.orchestra.run.smithi174.stdout: "segexd": 0, 2024-04-21T23:29:03.487 INFO:teuthology.orchestra.run.smithi174.stdout: "expos": 10953535, 2024-04-21T23:29:03.487 INFO:teuthology.orchestra.run.smithi174.stdout: "wrpos": 11093064, 2024-04-21T23:29:03.488 INFO:teuthology.orchestra.run.smithi174.stdout: "rdpos": 11091743, 2024-04-21T23:29:03.488 INFO:teuthology.orchestra.run.smithi174.stdout: "jlat": { 2024-04-21T23:29:03.488 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:03.488 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:03.488 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:03.488 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:03.488 INFO:teuthology.orchestra.run.smithi174.stdout: "replayed": 3212 2024-04-21T23:29:03.488 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:03.488 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_mem": { 2024-04-21T23:29:03.489 INFO:teuthology.orchestra.run.smithi174.stdout: "ino": 28, 2024-04-21T23:29:03.489 INFO:teuthology.orchestra.run.smithi174.stdout: "ino+": 2961, 2024-04-21T23:29:03.489 INFO:teuthology.orchestra.run.smithi174.stdout: "ino-": 2933, 2024-04-21T23:29:03.489 INFO:teuthology.orchestra.run.smithi174.stdout: "dir": 16, 2024-04-21T23:29:03.489 INFO:teuthology.orchestra.run.smithi174.stdout: "dir+": 158, 2024-04-21T23:29:03.489 INFO:teuthology.orchestra.run.smithi174.stdout: "dir-": 142, 2024-04-21T23:29:03.489 INFO:teuthology.orchestra.run.smithi174.stdout: "dn": 48, 2024-04-21T23:29:03.489 INFO:teuthology.orchestra.run.smithi174.stdout: "dn+": 2977, 2024-04-21T23:29:03.490 INFO:teuthology.orchestra.run.smithi174.stdout: "dn-": 2929, 2024-04-21T23:29:03.490 INFO:teuthology.orchestra.run.smithi174.stdout: "cap": 0, 2024-04-21T23:29:03.490 INFO:teuthology.orchestra.run.smithi174.stdout: "cap+": 0, 2024-04-21T23:29:03.490 INFO:teuthology.orchestra.run.smithi174.stdout: "cap-": 0, 2024-04-21T23:29:03.490 INFO:teuthology.orchestra.run.smithi174.stdout: "rss": 58648, 2024-04-21T23:29:03.490 INFO:teuthology.orchestra.run.smithi174.stdout: "heap": 215420 2024-04-21T23:29:03.490 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:03.490 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_server": { 2024-04-21T23:29:03.490 INFO:teuthology.orchestra.run.smithi174.stdout: "dispatch_client_request": 0, 2024-04-21T23:29:03.491 INFO:teuthology.orchestra.run.smithi174.stdout: "dispatch_server_request": 0, 2024-04-21T23:29:03.491 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_request": 0, 2024-04-21T23:29:03.491 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_session": 0, 2024-04-21T23:29:03.491 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_peer_request": 0, 2024-04-21T23:29:03.491 INFO:teuthology.orchestra.run.smithi174.stdout: "req_create_latency": { 2024-04-21T23:29:03.491 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:03.491 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:03.491 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:03.491 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:03.492 INFO:teuthology.orchestra.run.smithi174.stdout: "req_getattr_latency": { 2024-04-21T23:29:03.492 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:03.492 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:03.492 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:03.492 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:03.492 INFO:teuthology.orchestra.run.smithi174.stdout: "req_getfilelock_latency": { 2024-04-21T23:29:03.492 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:03.492 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:03.493 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:03.493 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:03.493 INFO:teuthology.orchestra.run.smithi174.stdout: "req_link_latency": { 2024-04-21T23:29:03.493 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:03.493 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:03.493 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:03.493 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:03.493 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookup_latency": { 2024-04-21T23:29:03.493 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:03.494 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:03.494 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:03.494 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:03.494 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookuphash_latency": { 2024-04-21T23:29:03.494 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:03.494 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:03.494 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:03.494 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:03.494 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupino_latency": { 2024-04-21T23:29:03.495 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:03.495 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:03.495 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:03.495 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:03.495 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupname_latency": { 2024-04-21T23:29:03.495 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:03.495 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:03.495 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:03.496 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:03.496 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupparent_latency": { 2024-04-21T23:29:03.496 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:03.496 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:03.496 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:03.496 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:03.496 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupsnap_latency": { 2024-04-21T23:29:03.496 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:03.497 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:03.497 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:03.497 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:03.497 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lssnap_latency": { 2024-04-21T23:29:03.497 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:03.497 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:03.497 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:03.497 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:03.497 INFO:teuthology.orchestra.run.smithi174.stdout: "req_mkdir_latency": { 2024-04-21T23:29:03.498 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:03.499 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:03.499 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:03.499 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:03.499 INFO:teuthology.orchestra.run.smithi174.stdout: "req_mknod_latency": { 2024-04-21T23:29:03.499 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:03.499 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:03.499 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:03.499 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:03.499 INFO:teuthology.orchestra.run.smithi174.stdout: "req_mksnap_latency": { 2024-04-21T23:29:03.500 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:03.500 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:03.500 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:03.500 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:03.500 INFO:teuthology.orchestra.run.smithi174.stdout: "req_open_latency": { 2024-04-21T23:29:03.500 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:03.500 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:03.500 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:03.501 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:03.501 INFO:teuthology.orchestra.run.smithi174.stdout: "req_readdir_latency": { 2024-04-21T23:29:03.501 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:03.501 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:03.501 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:03.501 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:03.501 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rename_latency": { 2024-04-21T23:29:03.501 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:03.502 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:03.502 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:03.502 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:03.502 INFO:teuthology.orchestra.run.smithi174.stdout: "req_renamesnap_latency": { 2024-04-21T23:29:03.502 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:03.502 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:03.502 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:03.502 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:03.502 INFO:teuthology.orchestra.run.smithi174.stdout: "req_snapdiff_latency": { 2024-04-21T23:29:03.503 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:03.503 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:03.503 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:03.503 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:03.503 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rmdir_latency": { 2024-04-21T23:29:03.503 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:03.503 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:03.503 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:03.504 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:03.504 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rmsnap_latency": { 2024-04-21T23:29:03.504 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:03.504 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:03.504 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:03.504 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:03.504 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rmxattr_latency": { 2024-04-21T23:29:03.504 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:03.504 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:03.505 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:03.505 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:03.505 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setattr_latency": { 2024-04-21T23:29:03.505 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:03.505 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:03.505 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:03.505 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:03.505 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setdirlayout_latency": { 2024-04-21T23:29:03.505 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:03.506 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:03.506 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:03.506 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:03.506 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setfilelock_latency": { 2024-04-21T23:29:03.506 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:03.506 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:03.506 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:03.506 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:03.507 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setlayout_latency": { 2024-04-21T23:29:03.507 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:03.507 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:03.507 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:03.507 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:03.507 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setxattr_latency": { 2024-04-21T23:29:03.507 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:03.507 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:03.508 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:03.508 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:03.508 INFO:teuthology.orchestra.run.smithi174.stdout: "req_symlink_latency": { 2024-04-21T23:29:03.508 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:03.508 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:03.508 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:03.508 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:03.508 INFO:teuthology.orchestra.run.smithi174.stdout: "req_unlink_latency": { 2024-04-21T23:29:03.509 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:03.509 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:03.509 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:03.509 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:03.509 INFO:teuthology.orchestra.run.smithi174.stdout: "cap_revoke_eviction": 0, 2024-04-21T23:29:03.509 INFO:teuthology.orchestra.run.smithi174.stdout: "cap_acquisition_throttle": 0, 2024-04-21T23:29:03.509 INFO:teuthology.orchestra.run.smithi174.stdout: "req_getvxattr_latency": { 2024-04-21T23:29:03.509 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:03.509 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:03.510 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:03.510 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-04-21T23:29:03.510 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:03.510 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_sessions": { 2024-04-21T23:29:03.510 INFO:teuthology.orchestra.run.smithi174.stdout: "session_count": 2, 2024-04-21T23:29:03.510 INFO:teuthology.orchestra.run.smithi174.stdout: "session_add": 7, 2024-04-21T23:29:03.510 INFO:teuthology.orchestra.run.smithi174.stdout: "session_remove": 5, 2024-04-21T23:29:03.510 INFO:teuthology.orchestra.run.smithi174.stdout: "sessions_open": 5, 2024-04-21T23:29:03.510 INFO:teuthology.orchestra.run.smithi174.stdout: "sessions_stale": 0, 2024-04-21T23:29:03.511 INFO:teuthology.orchestra.run.smithi174.stdout: "total_load": 0, 2024-04-21T23:29:03.511 INFO:teuthology.orchestra.run.smithi174.stdout: "average_load": 0, 2024-04-21T23:29:03.511 INFO:teuthology.orchestra.run.smithi174.stdout: "avg_session_uptime": 3495, 2024-04-21T23:29:03.511 INFO:teuthology.orchestra.run.smithi174.stdout: "mdthresh_evicted": 0 2024-04-21T23:29:03.511 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:03.511 INFO:teuthology.orchestra.run.smithi174.stdout: "mempool": { 2024-04-21T23:29:03.511 INFO:teuthology.orchestra.run.smithi174.stdout: "bloom_filter_bytes": 0, 2024-04-21T23:29:03.511 INFO:teuthology.orchestra.run.smithi174.stdout: "bloom_filter_items": 0, 2024-04-21T23:29:03.512 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_alloc_bytes": 0, 2024-04-21T23:29:03.512 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_alloc_items": 0, 2024-04-21T23:29:03.512 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_data_bytes": 0, 2024-04-21T23:29:03.512 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_data_items": 0, 2024-04-21T23:29:03.512 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-21T23:29:03.512 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_onode_items": 0, 2024-04-21T23:29:03.512 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-21T23:29:03.512 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_meta_items": 0, 2024-04-21T23:29:03.512 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_other_bytes": 0, 2024-04-21T23:29:03.513 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_other_items": 0, 2024-04-21T23:29:03.513 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-21T23:29:03.513 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_buffer_items": 0, 2024-04-21T23:29:03.513 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_extent_bytes": 0, 2024-04-21T23:29:03.513 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_extent_items": 0, 2024-04-21T23:29:03.513 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_blob_bytes": 0, 2024-04-21T23:29:03.513 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_blob_items": 0, 2024-04-21T23:29:03.513 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-21T23:29:03.513 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_shared_blob_items": 0, 2024-04-21T23:29:03.514 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-21T23:29:03.514 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_inline_bl_items": 0, 2024-04-21T23:29:03.514 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_fsck_bytes": 0, 2024-04-21T23:29:03.514 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_fsck_items": 0, 2024-04-21T23:29:03.514 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_txc_bytes": 0, 2024-04-21T23:29:03.514 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_txc_items": 0, 2024-04-21T23:29:03.514 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-21T23:29:03.514 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_deferred_items": 0, 2024-04-21T23:29:03.515 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_bytes": 0, 2024-04-21T23:29:03.515 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_items": 0, 2024-04-21T23:29:03.515 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_bytes": 0, 2024-04-21T23:29:03.515 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_items": 0, 2024-04-21T23:29:03.515 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_reader_bytes": 0, 2024-04-21T23:29:03.515 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_reader_items": 0, 2024-04-21T23:29:03.515 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_writer_bytes": 0, 2024-04-21T23:29:03.515 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_writer_items": 0, 2024-04-21T23:29:03.515 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_anon_bytes": 199917, 2024-04-21T23:29:03.516 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_anon_items": 75, 2024-04-21T23:29:03.516 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_meta_bytes": 0, 2024-04-21T23:29:03.516 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_meta_items": 0, 2024-04-21T23:29:03.516 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_bytes": 0, 2024-04-21T23:29:03.516 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_items": 0, 2024-04-21T23:29:03.516 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_mapbl_bytes": 0, 2024-04-21T23:29:03.516 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_mapbl_items": 0, 2024-04-21T23:29:03.516 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_pglog_bytes": 0, 2024-04-21T23:29:03.516 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_pglog_items": 0, 2024-04-21T23:29:03.517 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_bytes": 9704, 2024-04-21T23:29:03.517 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_items": 186, 2024-04-21T23:29:03.517 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_mapping_bytes": 0, 2024-04-21T23:29:03.517 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_mapping_items": 0, 2024-04-21T23:29:03.517 INFO:teuthology.orchestra.run.smithi174.stdout: "pgmap_bytes": 0, 2024-04-21T23:29:03.517 INFO:teuthology.orchestra.run.smithi174.stdout: "pgmap_items": 0, 2024-04-21T23:29:03.517 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_co_bytes": 131171, 2024-04-21T23:29:03.517 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_co_items": 809, 2024-04-21T23:29:03.517 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_1_bytes": 0, 2024-04-21T23:29:03.518 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_1_items": 0, 2024-04-21T23:29:03.518 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_2_bytes": 0, 2024-04-21T23:29:03.518 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_2_items": 0 2024-04-21T23:29:03.518 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:03.518 INFO:teuthology.orchestra.run.smithi174.stdout: "objecter": { 2024-04-21T23:29:03.518 INFO:teuthology.orchestra.run.smithi174.stdout: "op_active": 0, 2024-04-21T23:29:03.518 INFO:teuthology.orchestra.run.smithi174.stdout: "op_laggy": 0, 2024-04-21T23:29:03.518 INFO:teuthology.orchestra.run.smithi174.stdout: "op_send": 2361, 2024-04-21T23:29:03.519 INFO:teuthology.orchestra.run.smithi174.stdout: "op_send_bytes": 26, 2024-04-21T23:29:03.519 INFO:teuthology.orchestra.run.smithi174.stdout: "op_resend": 4, 2024-04-21T23:29:03.519 INFO:teuthology.orchestra.run.smithi174.stdout: "op_reply": 2361, 2024-04-21T23:29:03.519 INFO:teuthology.orchestra.run.smithi174.stdout: "op_latency": { 2024-04-21T23:29:03.519 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 2361, 2024-04-21T23:29:03.520 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 1.725975855, 2024-04-21T23:29:03.520 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000731035 2024-04-21T23:29:03.520 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:03.520 INFO:teuthology.orchestra.run.smithi174.stdout: "op_inflight": 0, 2024-04-21T23:29:03.520 INFO:teuthology.orchestra.run.smithi174.stdout: "oplen_avg": { 2024-04-21T23:29:03.520 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 2361, 2024-04-21T23:29:03.520 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 2362 2024-04-21T23:29:03.520 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:03.521 INFO:teuthology.orchestra.run.smithi174.stdout: "op": 2361, 2024-04-21T23:29:03.521 INFO:teuthology.orchestra.run.smithi174.stdout: "op_r": 2361, 2024-04-21T23:29:03.521 INFO:teuthology.orchestra.run.smithi174.stdout: "op_w": 0, 2024-04-21T23:29:03.521 INFO:teuthology.orchestra.run.smithi174.stdout: "op_rmw": 0, 2024-04-21T23:29:03.521 INFO:teuthology.orchestra.run.smithi174.stdout: "op_pg": 0, 2024-04-21T23:29:03.521 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_stat": 1518, 2024-04-21T23:29:03.521 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_create": 0, 2024-04-21T23:29:03.521 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_read": 840, 2024-04-21T23:29:03.522 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_write": 0, 2024-04-21T23:29:03.522 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_writefull": 0, 2024-04-21T23:29:03.522 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_writesame": 0, 2024-04-21T23:29:03.522 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_append": 0, 2024-04-21T23:29:03.522 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_zero": 0, 2024-04-21T23:29:03.522 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_truncate": 0, 2024-04-21T23:29:03.522 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_delete": 0, 2024-04-21T23:29:03.522 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_mapext": 0, 2024-04-21T23:29:03.523 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_sparse_read": 0, 2024-04-21T23:29:03.523 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_clonerange": 0, 2024-04-21T23:29:03.523 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_getxattr": 2, 2024-04-21T23:29:03.523 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_setxattr": 0, 2024-04-21T23:29:03.523 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_cmpxattr": 0, 2024-04-21T23:29:03.523 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_rmxattr": 0, 2024-04-21T23:29:03.523 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_resetxattrs": 0, 2024-04-21T23:29:03.523 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_call": 0, 2024-04-21T23:29:03.524 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_watch": 0, 2024-04-21T23:29:03.524 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_notify": 0, 2024-04-21T23:29:03.524 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_src_cmpxattr": 0, 2024-04-21T23:29:03.524 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_pgls": 0, 2024-04-21T23:29:03.524 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_pgls_filter": 0, 2024-04-21T23:29:03.524 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_other": 0, 2024-04-21T23:29:03.524 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_active": 0, 2024-04-21T23:29:03.524 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_send": 0, 2024-04-21T23:29:03.524 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_resend": 0, 2024-04-21T23:29:03.525 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_ping": 0, 2024-04-21T23:29:03.525 INFO:teuthology.orchestra.run.smithi174.stdout: "poolop_active": 0, 2024-04-21T23:29:03.525 INFO:teuthology.orchestra.run.smithi174.stdout: "poolop_send": 0, 2024-04-21T23:29:03.525 INFO:teuthology.orchestra.run.smithi174.stdout: "poolop_resend": 0, 2024-04-21T23:29:03.525 INFO:teuthology.orchestra.run.smithi174.stdout: "poolstat_active": 0, 2024-04-21T23:29:03.525 INFO:teuthology.orchestra.run.smithi174.stdout: "poolstat_send": 0, 2024-04-21T23:29:03.525 INFO:teuthology.orchestra.run.smithi174.stdout: "poolstat_resend": 0, 2024-04-21T23:29:03.525 INFO:teuthology.orchestra.run.smithi174.stdout: "statfs_active": 0, 2024-04-21T23:29:03.525 INFO:teuthology.orchestra.run.smithi174.stdout: "statfs_send": 0, 2024-04-21T23:29:03.526 INFO:teuthology.orchestra.run.smithi174.stdout: "statfs_resend": 0, 2024-04-21T23:29:03.526 INFO:teuthology.orchestra.run.smithi174.stdout: "command_active": 0, 2024-04-21T23:29:03.526 INFO:teuthology.orchestra.run.smithi174.stdout: "command_send": 0, 2024-04-21T23:29:03.526 INFO:teuthology.orchestra.run.smithi174.stdout: "command_resend": 0, 2024-04-21T23:29:03.526 INFO:teuthology.orchestra.run.smithi174.stdout: "map_epoch": 103, 2024-04-21T23:29:03.526 INFO:teuthology.orchestra.run.smithi174.stdout: "map_full": 0, 2024-04-21T23:29:03.526 INFO:teuthology.orchestra.run.smithi174.stdout: "map_inc": 13, 2024-04-21T23:29:03.526 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_sessions": 6, 2024-04-21T23:29:03.527 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_session_open": 6, 2024-04-21T23:29:03.527 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_session_close": 0, 2024-04-21T23:29:03.527 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_laggy": 0, 2024-04-21T23:29:03.527 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_wr": 0, 2024-04-21T23:29:03.527 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_rd": 2, 2024-04-21T23:29:03.527 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_del": 0 2024-04-21T23:29:03.527 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:03.527 INFO:teuthology.orchestra.run.smithi174.stdout: "oft": { 2024-04-21T23:29:03.527 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_objs": 0, 2024-04-21T23:29:03.528 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_kv_pairs": 0, 2024-04-21T23:29:03.528 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_updates": 0, 2024-04-21T23:29:03.528 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_removes": 0 2024-04-21T23:29:03.528 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:03.528 INFO:teuthology.orchestra.run.smithi174.stdout: "osd-slow-ops": { 2024-04-21T23:29:03.528 INFO:teuthology.orchestra.run.smithi174.stdout: "slow_ops_count": 0 2024-04-21T23:29:03.528 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:03.528 INFO:teuthology.orchestra.run.smithi174.stdout: "purge_queue": { 2024-04-21T23:29:03.528 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing_ops": 0, 2024-04-21T23:29:03.529 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing_ops_high_water": 0, 2024-04-21T23:29:03.529 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing": 0, 2024-04-21T23:29:03.529 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing_high_water": 0, 2024-04-21T23:29:03.529 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executed_ops": 0, 2024-04-21T23:29:03.529 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executed": 0, 2024-04-21T23:29:03.529 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_item_in_journal": 0 2024-04-21T23:29:03.529 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:03.529 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-21T23:29:03.529 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-04-21T23:29:03.530 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 104857600, 2024-04-21T23:29:03.530 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-04-21T23:29:03.530 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 2619, 2024-04-21T23:29:03.530 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 8959106, 2024-04-21T23:29:03.530 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-04-21T23:29:03.530 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 2619, 2024-04-21T23:29:03.530 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 0, 2024-04-21T23:29:03.530 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 0, 2024-04-21T23:29:03.531 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 2619, 2024-04-21T23:29:03.531 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 8959106, 2024-04-21T23:29:03.531 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-04-21T23:29:03.531 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:03.531 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:03.531 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:03.531 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-04-21T23:29:03.531 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:03.531 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-objecter_bytes": { 2024-04-21T23:29:03.532 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-04-21T23:29:03.532 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 104857600, 2024-04-21T23:29:03.532 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-04-21T23:29:03.532 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 0, 2024-04-21T23:29:03.532 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 0, 2024-04-21T23:29:03.532 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-04-21T23:29:03.532 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 0, 2024-04-21T23:29:03.532 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 2361, 2024-04-21T23:29:03.532 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 6898770, 2024-04-21T23:29:03.533 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 2361, 2024-04-21T23:29:03.533 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 6898770, 2024-04-21T23:29:03.533 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-04-21T23:29:03.533 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:03.533 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:03.533 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:03.533 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-04-21T23:29:03.533 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:03.534 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-objecter_ops": { 2024-04-21T23:29:03.534 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-04-21T23:29:03.534 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 1024, 2024-04-21T23:29:03.534 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-04-21T23:29:03.534 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 0, 2024-04-21T23:29:03.534 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 0, 2024-04-21T23:29:03.534 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-04-21T23:29:03.534 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 0, 2024-04-21T23:29:03.534 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 2361, 2024-04-21T23:29:03.535 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 2361, 2024-04-21T23:29:03.535 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 2361, 2024-04-21T23:29:03.535 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 2361, 2024-04-21T23:29:03.535 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-04-21T23:29:03.535 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:03.535 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:03.535 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:03.535 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-04-21T23:29:03.536 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:03.536 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-write_buf_throttle": { 2024-04-21T23:29:03.536 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-04-21T23:29:03.536 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 3758096384, 2024-04-21T23:29:03.536 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-04-21T23:29:03.536 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 0, 2024-04-21T23:29:03.536 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 0, 2024-04-21T23:29:03.536 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-04-21T23:29:03.536 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 0, 2024-04-21T23:29:03.537 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 0, 2024-04-21T23:29:03.537 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 0, 2024-04-21T23:29:03.537 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 0, 2024-04-21T23:29:03.537 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 0, 2024-04-21T23:29:03.537 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-04-21T23:29:03.537 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:03.537 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:03.537 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:03.537 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-04-21T23:29:03.538 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:03.538 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-write_buf_throttle-0x55ccfbcac6e0": { 2024-04-21T23:29:03.538 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-04-21T23:29:03.538 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 3758096384, 2024-04-21T23:29:03.538 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-04-21T23:29:03.538 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 0, 2024-04-21T23:29:03.538 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 0, 2024-04-21T23:29:03.538 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-04-21T23:29:03.538 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 0, 2024-04-21T23:29:03.539 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 0, 2024-04-21T23:29:03.539 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 0, 2024-04-21T23:29:03.539 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 0, 2024-04-21T23:29:03.539 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 0, 2024-04-21T23:29:03.539 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-04-21T23:29:03.539 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:03.539 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:03.539 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:03.539 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-04-21T23:29:03.540 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-04-21T23:29:03.540 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-04-21T23:29:04.127 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2024-04-21T23:29:04.127 INFO:tasks.check_counter:Daemon mds.c mds.imported=0 2024-04-21T23:29:04.127 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=6 2024-04-21T23:29:04.128 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2024-04-21T23:29:04.128 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2024-04-21T23:29:04.128 INFO:tasks.check_counter:Daemon mds.c mds.dir_split=0 2024-04-21T23:29:04.128 DEBUG:tasks.check_counter:Getting stats from f 2024-04-21T23:29:04.128 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph daemon mds.f perf dump 2024-04-21T23:29:04.372 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.c/config 2024-04-21T23:29:05.181 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:29:04 smithi040 ceph-mon[27984]: pgmap v331: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:29:05.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:29:04 smithi174 ceph-mon[31409]: pgmap v331: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:29:05.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:29:04 smithi111 ceph-mon[31085]: pgmap v331: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T23:29:06.090 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-04-21T23:29:06.090 INFO:teuthology.orchestra.run.smithi174.stdout: "AsyncMessenger::Worker-0": { 2024-04-21T23:29:06.090 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_messages": 2196, 2024-04-21T23:29:06.091 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages": 2198, 2024-04-21T23:29:06.091 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_bytes": 1199923, 2024-04-21T23:29:06.091 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_bytes": 698335, 2024-04-21T23:29:06.091 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_created_connections": 6, 2024-04-21T23:29:06.091 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_active_connections": 2, 2024-04-21T23:29:06.091 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_total_time": 0.194206780, 2024-04-21T23:29:06.091 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_send_time": 0.065459179, 2024-04-21T23:29:06.091 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_recv_time": 0.048356781, 2024-04-21T23:29:06.092 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_fast_dispatch_time": 0.064954395, 2024-04-21T23:29:06.092 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T23:29:06.092 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 2196, 2024-04-21T23:29:06.092 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.084225638, 2024-04-21T23:29:06.092 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000038354 2024-04-21T23:29:06.092 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:06.092 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_handle_ack_lat": { 2024-04-21T23:29:06.093 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:06.093 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:06.093 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:06.093 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:06.093 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-21T23:29:06.093 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_encrypted_bytes": 0 2024-04-21T23:29:06.093 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:06.093 INFO:teuthology.orchestra.run.smithi174.stdout: "AsyncMessenger::Worker-1": { 2024-04-21T23:29:06.093 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_messages": 8, 2024-04-21T23:29:06.094 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages": 171, 2024-04-21T23:29:06.094 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_bytes": 3900, 2024-04-21T23:29:06.094 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_bytes": 450138, 2024-04-21T23:29:06.094 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_created_connections": 6, 2024-04-21T23:29:06.094 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_active_connections": 4, 2024-04-21T23:29:06.094 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_total_time": 0.011607524, 2024-04-21T23:29:06.094 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_send_time": 0.006667876, 2024-04-21T23:29:06.094 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_recv_time": 0.002084652, 2024-04-21T23:29:06.095 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_fast_dispatch_time": 0.000079695, 2024-04-21T23:29:06.095 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T23:29:06.095 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 169, 2024-04-21T23:29:06.095 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.014294433, 2024-04-21T23:29:06.095 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000084582 2024-04-21T23:29:06.095 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:06.095 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_handle_ack_lat": { 2024-04-21T23:29:06.096 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:06.096 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:06.096 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:06.096 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:06.096 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_encrypted_bytes": 3344, 2024-04-21T23:29:06.096 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_encrypted_bytes": 449440 2024-04-21T23:29:06.096 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:06.096 INFO:teuthology.orchestra.run.smithi174.stdout: "AsyncMessenger::Worker-2": { 2024-04-21T23:29:06.096 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_messages": 258, 2024-04-21T23:29:06.097 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages": 238, 2024-04-21T23:29:06.097 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_bytes": 1646652, 2024-04-21T23:29:06.097 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_bytes": 137902, 2024-04-21T23:29:06.097 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_created_connections": 3, 2024-04-21T23:29:06.097 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_active_connections": 2, 2024-04-21T23:29:06.097 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_total_time": 0.050482736, 2024-04-21T23:29:06.098 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_send_time": 0.014928495, 2024-04-21T23:29:06.098 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_recv_time": 0.039221527, 2024-04-21T23:29:06.098 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_fast_dispatch_time": 0.005653290, 2024-04-21T23:29:06.098 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T23:29:06.098 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 237, 2024-04-21T23:29:06.098 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.013359431, 2024-04-21T23:29:06.098 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000056368 2024-04-21T23:29:06.098 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:06.099 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_handle_ack_lat": { 2024-04-21T23:29:06.099 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:06.099 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:06.099 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:06.099 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:06.099 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_encrypted_bytes": 1646384, 2024-04-21T23:29:06.100 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_encrypted_bytes": 137584 2024-04-21T23:29:06.100 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:06.100 INFO:teuthology.orchestra.run.smithi174.stdout: "cct": { 2024-04-21T23:29:06.100 INFO:teuthology.orchestra.run.smithi174.stdout: "total_workers": 1, 2024-04-21T23:29:06.100 INFO:teuthology.orchestra.run.smithi174.stdout: "unhealthy_workers": 0 2024-04-21T23:29:06.100 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:06.100 INFO:teuthology.orchestra.run.smithi174.stdout: "finisher-MDSRank": { 2024-04-21T23:29:06.100 INFO:teuthology.orchestra.run.smithi174.stdout: "queue_len": 0, 2024-04-21T23:29:06.100 INFO:teuthology.orchestra.run.smithi174.stdout: "complete_latency": { 2024-04-21T23:29:06.101 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 3747, 2024-04-21T23:29:06.101 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.106341274, 2024-04-21T23:29:06.101 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000028380 2024-04-21T23:29:06.101 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-04-21T23:29:06.101 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:06.101 INFO:teuthology.orchestra.run.smithi174.stdout: "finisher-PurgeQueue": { 2024-04-21T23:29:06.101 INFO:teuthology.orchestra.run.smithi174.stdout: "queue_len": 0, 2024-04-21T23:29:06.102 INFO:teuthology.orchestra.run.smithi174.stdout: "complete_latency": { 2024-04-21T23:29:06.102 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:06.102 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:06.102 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:06.102 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-04-21T23:29:06.102 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:06.102 INFO:teuthology.orchestra.run.smithi174.stdout: "mds": { 2024-04-21T23:29:06.102 INFO:teuthology.orchestra.run.smithi174.stdout: "request": 0, 2024-04-21T23:29:06.103 INFO:teuthology.orchestra.run.smithi174.stdout: "reply": 0, 2024-04-21T23:29:06.103 INFO:teuthology.orchestra.run.smithi174.stdout: "reply_latency": { 2024-04-21T23:29:06.103 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:06.103 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:06.103 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:06.103 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:06.103 INFO:teuthology.orchestra.run.smithi174.stdout: "slow_reply": 0, 2024-04-21T23:29:06.103 INFO:teuthology.orchestra.run.smithi174.stdout: "forward": 0, 2024-04-21T23:29:06.104 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_fetch_complete": 0, 2024-04-21T23:29:06.104 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_fetch_keys": 0, 2024-04-21T23:29:06.104 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_commit": 0, 2024-04-21T23:29:06.104 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_split": 0, 2024-04-21T23:29:06.104 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_merge": 0, 2024-04-21T23:29:06.104 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes": 194, 2024-04-21T23:29:06.104 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_top": 10, 2024-04-21T23:29:06.104 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_bottom": 184, 2024-04-21T23:29:06.104 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_pin_tail": 0, 2024-04-21T23:29:06.105 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_pinned": 180, 2024-04-21T23:29:06.105 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_expired": 0, 2024-04-21T23:29:06.105 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_with_caps": 0, 2024-04-21T23:29:06.105 INFO:teuthology.orchestra.run.smithi174.stdout: "caps": 0, 2024-04-21T23:29:06.105 INFO:teuthology.orchestra.run.smithi174.stdout: "subtrees": 10, 2024-04-21T23:29:06.105 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse": 0, 2024-04-21T23:29:06.105 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_hit": 0, 2024-04-21T23:29:06.105 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_forward": 0, 2024-04-21T23:29:06.106 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_discover": 0, 2024-04-21T23:29:06.106 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_dir_fetch": 0, 2024-04-21T23:29:06.106 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_remote_ino": 0, 2024-04-21T23:29:06.106 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_lock": 0, 2024-04-21T23:29:06.106 INFO:teuthology.orchestra.run.smithi174.stdout: "load_cent": 0, 2024-04-21T23:29:06.106 INFO:teuthology.orchestra.run.smithi174.stdout: "q": 0, 2024-04-21T23:29:06.106 INFO:teuthology.orchestra.run.smithi174.stdout: "exported": 0, 2024-04-21T23:29:06.107 INFO:teuthology.orchestra.run.smithi174.stdout: "exported_inodes": 0, 2024-04-21T23:29:06.107 INFO:teuthology.orchestra.run.smithi174.stdout: "imported": 0, 2024-04-21T23:29:06.107 INFO:teuthology.orchestra.run.smithi174.stdout: "imported_inodes": 0, 2024-04-21T23:29:06.107 INFO:teuthology.orchestra.run.smithi174.stdout: "openino_dir_fetch": 0, 2024-04-21T23:29:06.107 INFO:teuthology.orchestra.run.smithi174.stdout: "openino_backtrace_fetch": 0, 2024-04-21T23:29:06.107 INFO:teuthology.orchestra.run.smithi174.stdout: "openino_peer_discover": 0, 2024-04-21T23:29:06.107 INFO:teuthology.orchestra.run.smithi174.stdout: "root_rfiles": 0, 2024-04-21T23:29:06.107 INFO:teuthology.orchestra.run.smithi174.stdout: "root_rbytes": 0, 2024-04-21T23:29:06.107 INFO:teuthology.orchestra.run.smithi174.stdout: "root_rsnaps": 0, 2024-04-21T23:29:06.108 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_backtrace_fetch": 0, 2024-04-21T23:29:06.108 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_set_tag": 0, 2024-04-21T23:29:06.108 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_backtrace_repaired": 0, 2024-04-21T23:29:06.108 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_inotable_repaired": 0, 2024-04-21T23:29:06.108 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_dir_inodes": 0, 2024-04-21T23:29:06.108 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_dir_base_inodes": 0, 2024-04-21T23:29:06.108 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_dirfrag_rstats": 0, 2024-04-21T23:29:06.108 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_file_inodes": 0, 2024-04-21T23:29:06.108 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_inode_file_caps": 0, 2024-04-21T23:29:06.109 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_revoke": 0, 2024-04-21T23:29:06.109 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_grant": 0, 2024-04-21T23:29:06.109 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_trunc": 0, 2024-04-21T23:29:06.109 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-21T23:29:06.109 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-21T23:29:06.109 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_caps": 0, 2024-04-21T23:29:06.109 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_caps_dirty": 0, 2024-04-21T23:29:06.109 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_cap_release": 0, 2024-04-21T23:29:06.110 INFO:teuthology.orchestra.run.smithi174.stdout: "process_request_cap_release": 0 2024-04-21T23:29:06.110 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:06.110 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_cache": { 2024-04-21T23:29:06.110 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_update": 0, 2024-04-21T23:29:06.110 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_update_receipt": 0, 2024-04-21T23:29:06.110 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_try_discover": 0, 2024-04-21T23:29:06.110 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_send_discover": 0, 2024-04-21T23:29:06.110 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_handle_discover": 0, 2024-04-21T23:29:06.111 INFO:teuthology.orchestra.run.smithi174.stdout: "num_strays": 0, 2024-04-21T23:29:06.111 INFO:teuthology.orchestra.run.smithi174.stdout: "num_strays_delayed": 0, 2024-04-21T23:29:06.111 INFO:teuthology.orchestra.run.smithi174.stdout: "num_strays_enqueuing": 0, 2024-04-21T23:29:06.111 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_created": 0, 2024-04-21T23:29:06.111 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_enqueued": 0, 2024-04-21T23:29:06.111 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_reintegrated": 0, 2024-04-21T23:29:06.111 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_migrated": 0, 2024-04-21T23:29:06.111 INFO:teuthology.orchestra.run.smithi174.stdout: "num_recovering_processing": 0, 2024-04-21T23:29:06.111 INFO:teuthology.orchestra.run.smithi174.stdout: "num_recovering_enqueued": 0, 2024-04-21T23:29:06.112 INFO:teuthology.orchestra.run.smithi174.stdout: "num_recovering_prioritized": 0, 2024-04-21T23:29:06.112 INFO:teuthology.orchestra.run.smithi174.stdout: "recovery_started": 0, 2024-04-21T23:29:06.112 INFO:teuthology.orchestra.run.smithi174.stdout: "recovery_completed": 0, 2024-04-21T23:29:06.112 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_quiesce_path": 0, 2024-04-21T23:29:06.112 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_quiesce_inode": 0, 2024-04-21T23:29:06.112 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_enqueue_scrub": 0, 2024-04-21T23:29:06.112 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_exportdir": 0, 2024-04-21T23:29:06.112 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_flush": 0, 2024-04-21T23:29:06.113 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_fragmentdir": 0, 2024-04-21T23:29:06.113 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_fragstats": 0, 2024-04-21T23:29:06.113 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_inodestats": 0 2024-04-21T23:29:06.113 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:06.113 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_log": { 2024-04-21T23:29:06.113 INFO:teuthology.orchestra.run.smithi174.stdout: "evlrg": 0, 2024-04-21T23:29:06.113 INFO:teuthology.orchestra.run.smithi174.stdout: "evadd": 0, 2024-04-21T23:29:06.113 INFO:teuthology.orchestra.run.smithi174.stdout: "evex": 0, 2024-04-21T23:29:06.113 INFO:teuthology.orchestra.run.smithi174.stdout: "evtrm": 0, 2024-04-21T23:29:06.114 INFO:teuthology.orchestra.run.smithi174.stdout: "ev": 234, 2024-04-21T23:29:06.114 INFO:teuthology.orchestra.run.smithi174.stdout: "evexg": 0, 2024-04-21T23:29:06.114 INFO:teuthology.orchestra.run.smithi174.stdout: "evexd": 0, 2024-04-21T23:29:06.114 INFO:teuthology.orchestra.run.smithi174.stdout: "segadd": 0, 2024-04-21T23:29:06.114 INFO:teuthology.orchestra.run.smithi174.stdout: "segex": 0, 2024-04-21T23:29:06.114 INFO:teuthology.orchestra.run.smithi174.stdout: "segtrm": 0, 2024-04-21T23:29:06.114 INFO:teuthology.orchestra.run.smithi174.stdout: "seg": 2, 2024-04-21T23:29:06.114 INFO:teuthology.orchestra.run.smithi174.stdout: "segmjr": 2, 2024-04-21T23:29:06.115 INFO:teuthology.orchestra.run.smithi174.stdout: "segexg": 0, 2024-04-21T23:29:06.115 INFO:teuthology.orchestra.run.smithi174.stdout: "segexd": 0, 2024-04-21T23:29:06.115 INFO:teuthology.orchestra.run.smithi174.stdout: "expos": 4194304, 2024-04-21T23:29:06.115 INFO:teuthology.orchestra.run.smithi174.stdout: "wrpos": 4772510, 2024-04-21T23:29:06.115 INFO:teuthology.orchestra.run.smithi174.stdout: "rdpos": 4771936, 2024-04-21T23:29:06.115 INFO:teuthology.orchestra.run.smithi174.stdout: "jlat": { 2024-04-21T23:29:06.115 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:06.116 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:06.116 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:06.116 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:06.116 INFO:teuthology.orchestra.run.smithi174.stdout: "replayed": 234 2024-04-21T23:29:06.116 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:06.116 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_mem": { 2024-04-21T23:29:06.116 INFO:teuthology.orchestra.run.smithi174.stdout: "ino": 179, 2024-04-21T23:29:06.116 INFO:teuthology.orchestra.run.smithi174.stdout: "ino+": 183, 2024-04-21T23:29:06.117 INFO:teuthology.orchestra.run.smithi174.stdout: "ino-": 4, 2024-04-21T23:29:06.117 INFO:teuthology.orchestra.run.smithi174.stdout: "dir": 36, 2024-04-21T23:29:06.117 INFO:teuthology.orchestra.run.smithi174.stdout: "dir+": 36, 2024-04-21T23:29:06.117 INFO:teuthology.orchestra.run.smithi174.stdout: "dir-": 0, 2024-04-21T23:29:06.117 INFO:teuthology.orchestra.run.smithi174.stdout: "dn": 194, 2024-04-21T23:29:06.117 INFO:teuthology.orchestra.run.smithi174.stdout: "dn+": 194, 2024-04-21T23:29:06.117 INFO:teuthology.orchestra.run.smithi174.stdout: "dn-": 0, 2024-04-21T23:29:06.117 INFO:teuthology.orchestra.run.smithi174.stdout: "cap": 0, 2024-04-21T23:29:06.117 INFO:teuthology.orchestra.run.smithi174.stdout: "cap+": 0, 2024-04-21T23:29:06.118 INFO:teuthology.orchestra.run.smithi174.stdout: "cap-": 0, 2024-04-21T23:29:06.118 INFO:teuthology.orchestra.run.smithi174.stdout: "rss": 51540, 2024-04-21T23:29:06.118 INFO:teuthology.orchestra.run.smithi174.stdout: "heap": 215420 2024-04-21T23:29:06.118 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:06.118 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_server": { 2024-04-21T23:29:06.118 INFO:teuthology.orchestra.run.smithi174.stdout: "dispatch_client_request": 0, 2024-04-21T23:29:06.118 INFO:teuthology.orchestra.run.smithi174.stdout: "dispatch_server_request": 0, 2024-04-21T23:29:06.118 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_request": 0, 2024-04-21T23:29:06.119 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_session": 0, 2024-04-21T23:29:06.119 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_peer_request": 0, 2024-04-21T23:29:06.119 INFO:teuthology.orchestra.run.smithi174.stdout: "req_create_latency": { 2024-04-21T23:29:06.119 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:06.119 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:06.119 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:06.119 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:06.119 INFO:teuthology.orchestra.run.smithi174.stdout: "req_getattr_latency": { 2024-04-21T23:29:06.120 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:06.120 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:06.120 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:06.120 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:06.120 INFO:teuthology.orchestra.run.smithi174.stdout: "req_getfilelock_latency": { 2024-04-21T23:29:06.120 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:06.120 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:06.120 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:06.120 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:06.121 INFO:teuthology.orchestra.run.smithi174.stdout: "req_link_latency": { 2024-04-21T23:29:06.121 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:06.121 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:06.121 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:06.121 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:06.121 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookup_latency": { 2024-04-21T23:29:06.121 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:06.121 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:06.122 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:06.122 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:06.122 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookuphash_latency": { 2024-04-21T23:29:06.122 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:06.122 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:06.122 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:06.122 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:06.122 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupino_latency": { 2024-04-21T23:29:06.123 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:06.123 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:06.123 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:06.123 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:06.123 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupname_latency": { 2024-04-21T23:29:06.123 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:06.123 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:06.123 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:06.123 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:06.124 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupparent_latency": { 2024-04-21T23:29:06.124 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:06.124 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:06.124 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:06.124 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:06.124 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupsnap_latency": { 2024-04-21T23:29:06.124 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:06.124 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:06.125 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:06.125 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:06.125 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lssnap_latency": { 2024-04-21T23:29:06.125 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:06.125 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:06.125 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:06.125 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:06.125 INFO:teuthology.orchestra.run.smithi174.stdout: "req_mkdir_latency": { 2024-04-21T23:29:06.125 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:06.126 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:06.126 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:06.127 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:06.127 INFO:teuthology.orchestra.run.smithi174.stdout: "req_mknod_latency": { 2024-04-21T23:29:06.127 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:06.127 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:06.127 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:06.127 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:06.127 INFO:teuthology.orchestra.run.smithi174.stdout: "req_mksnap_latency": { 2024-04-21T23:29:06.127 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:06.128 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:06.128 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:06.128 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:06.128 INFO:teuthology.orchestra.run.smithi174.stdout: "req_open_latency": { 2024-04-21T23:29:06.128 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:06.128 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:06.128 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:06.128 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:06.128 INFO:teuthology.orchestra.run.smithi174.stdout: "req_readdir_latency": { 2024-04-21T23:29:06.129 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:06.129 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:06.129 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:06.129 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:06.129 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rename_latency": { 2024-04-21T23:29:06.129 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:06.130 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:06.130 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:06.130 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:06.130 INFO:teuthology.orchestra.run.smithi174.stdout: "req_renamesnap_latency": { 2024-04-21T23:29:06.130 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:06.130 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:06.130 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:06.130 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:06.131 INFO:teuthology.orchestra.run.smithi174.stdout: "req_snapdiff_latency": { 2024-04-21T23:29:06.131 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:06.131 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:06.131 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:06.131 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:06.131 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rmdir_latency": { 2024-04-21T23:29:06.131 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:06.131 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:06.131 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:06.132 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:06.132 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rmsnap_latency": { 2024-04-21T23:29:06.132 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:06.132 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:06.132 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:06.132 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:06.132 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rmxattr_latency": { 2024-04-21T23:29:06.132 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:06.132 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:06.133 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:06.133 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:06.133 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setattr_latency": { 2024-04-21T23:29:06.133 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:06.133 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:06.133 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:06.133 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:06.133 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setdirlayout_latency": { 2024-04-21T23:29:06.133 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:06.134 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:06.134 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:06.134 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:06.134 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setfilelock_latency": { 2024-04-21T23:29:06.134 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:06.134 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:06.134 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:06.135 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:06.135 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setlayout_latency": { 2024-04-21T23:29:06.135 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:06.135 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:06.135 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:06.135 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:06.135 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setxattr_latency": { 2024-04-21T23:29:06.136 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:06.136 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:06.136 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:06.136 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:06.136 INFO:teuthology.orchestra.run.smithi174.stdout: "req_symlink_latency": { 2024-04-21T23:29:06.136 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:06.136 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:06.136 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:06.137 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:06.137 INFO:teuthology.orchestra.run.smithi174.stdout: "req_unlink_latency": { 2024-04-21T23:29:06.137 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:06.137 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:06.137 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:06.137 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:06.137 INFO:teuthology.orchestra.run.smithi174.stdout: "cap_revoke_eviction": 0, 2024-04-21T23:29:06.137 INFO:teuthology.orchestra.run.smithi174.stdout: "cap_acquisition_throttle": 0, 2024-04-21T23:29:06.137 INFO:teuthology.orchestra.run.smithi174.stdout: "req_getvxattr_latency": { 2024-04-21T23:29:06.138 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:06.138 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:06.138 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:06.138 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-04-21T23:29:06.138 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:06.138 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_sessions": { 2024-04-21T23:29:06.138 INFO:teuthology.orchestra.run.smithi174.stdout: "session_count": 1, 2024-04-21T23:29:06.138 INFO:teuthology.orchestra.run.smithi174.stdout: "session_add": 3, 2024-04-21T23:29:06.139 INFO:teuthology.orchestra.run.smithi174.stdout: "session_remove": 2, 2024-04-21T23:29:06.139 INFO:teuthology.orchestra.run.smithi174.stdout: "sessions_open": 3, 2024-04-21T23:29:06.139 INFO:teuthology.orchestra.run.smithi174.stdout: "sessions_stale": 0, 2024-04-21T23:29:06.139 INFO:teuthology.orchestra.run.smithi174.stdout: "total_load": 0, 2024-04-21T23:29:06.139 INFO:teuthology.orchestra.run.smithi174.stdout: "average_load": 0, 2024-04-21T23:29:06.139 INFO:teuthology.orchestra.run.smithi174.stdout: "avg_session_uptime": 4069, 2024-04-21T23:29:06.139 INFO:teuthology.orchestra.run.smithi174.stdout: "mdthresh_evicted": 0 2024-04-21T23:29:06.139 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:06.139 INFO:teuthology.orchestra.run.smithi174.stdout: "mempool": { 2024-04-21T23:29:06.140 INFO:teuthology.orchestra.run.smithi174.stdout: "bloom_filter_bytes": 0, 2024-04-21T23:29:06.140 INFO:teuthology.orchestra.run.smithi174.stdout: "bloom_filter_items": 0, 2024-04-21T23:29:06.140 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_alloc_bytes": 0, 2024-04-21T23:29:06.140 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_alloc_items": 0, 2024-04-21T23:29:06.140 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_data_bytes": 0, 2024-04-21T23:29:06.140 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_data_items": 0, 2024-04-21T23:29:06.140 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-21T23:29:06.140 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_onode_items": 0, 2024-04-21T23:29:06.140 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-21T23:29:06.141 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_meta_items": 0, 2024-04-21T23:29:06.141 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_other_bytes": 0, 2024-04-21T23:29:06.141 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_other_items": 0, 2024-04-21T23:29:06.141 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-21T23:29:06.141 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_buffer_items": 0, 2024-04-21T23:29:06.141 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_extent_bytes": 0, 2024-04-21T23:29:06.141 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_extent_items": 0, 2024-04-21T23:29:06.141 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_blob_bytes": 0, 2024-04-21T23:29:06.141 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_blob_items": 0, 2024-04-21T23:29:06.142 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-21T23:29:06.142 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_shared_blob_items": 0, 2024-04-21T23:29:06.142 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-21T23:29:06.142 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_inline_bl_items": 0, 2024-04-21T23:29:06.142 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_fsck_bytes": 0, 2024-04-21T23:29:06.142 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_fsck_items": 0, 2024-04-21T23:29:06.142 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_txc_bytes": 0, 2024-04-21T23:29:06.142 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_txc_items": 0, 2024-04-21T23:29:06.143 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-21T23:29:06.143 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_deferred_items": 0, 2024-04-21T23:29:06.143 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_bytes": 0, 2024-04-21T23:29:06.143 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_items": 0, 2024-04-21T23:29:06.143 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_bytes": 0, 2024-04-21T23:29:06.143 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_items": 0, 2024-04-21T23:29:06.143 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_reader_bytes": 0, 2024-04-21T23:29:06.143 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_reader_items": 0, 2024-04-21T23:29:06.143 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_writer_bytes": 0, 2024-04-21T23:29:06.144 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_writer_items": 0, 2024-04-21T23:29:06.144 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_anon_bytes": 187895, 2024-04-21T23:29:06.144 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_anon_items": 224, 2024-04-21T23:29:06.144 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_meta_bytes": 0, 2024-04-21T23:29:06.144 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_meta_items": 0, 2024-04-21T23:29:06.144 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_bytes": 0, 2024-04-21T23:29:06.144 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_items": 0, 2024-04-21T23:29:06.145 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_mapbl_bytes": 0, 2024-04-21T23:29:06.145 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_mapbl_items": 0, 2024-04-21T23:29:06.145 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_pglog_bytes": 0, 2024-04-21T23:29:06.145 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_pglog_items": 0, 2024-04-21T23:29:06.145 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_bytes": 9704, 2024-04-21T23:29:06.145 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_items": 186, 2024-04-21T23:29:06.145 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_mapping_bytes": 0, 2024-04-21T23:29:06.145 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_mapping_items": 0, 2024-04-21T23:29:06.146 INFO:teuthology.orchestra.run.smithi174.stdout: "pgmap_bytes": 0, 2024-04-21T23:29:06.146 INFO:teuthology.orchestra.run.smithi174.stdout: "pgmap_items": 0, 2024-04-21T23:29:06.146 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_co_bytes": 607246, 2024-04-21T23:29:06.146 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_co_items": 5778, 2024-04-21T23:29:06.146 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_1_bytes": 0, 2024-04-21T23:29:06.146 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_1_items": 0, 2024-04-21T23:29:06.146 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_2_bytes": 0, 2024-04-21T23:29:06.147 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_2_items": 0 2024-04-21T23:29:06.147 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:06.147 INFO:teuthology.orchestra.run.smithi174.stdout: "objecter": { 2024-04-21T23:29:06.147 INFO:teuthology.orchestra.run.smithi174.stdout: "op_active": 0, 2024-04-21T23:29:06.147 INFO:teuthology.orchestra.run.smithi174.stdout: "op_laggy": 0, 2024-04-21T23:29:06.147 INFO:teuthology.orchestra.run.smithi174.stdout: "op_send": 2199, 2024-04-21T23:29:06.147 INFO:teuthology.orchestra.run.smithi174.stdout: "op_send_bytes": 21, 2024-04-21T23:29:06.147 INFO:teuthology.orchestra.run.smithi174.stdout: "op_resend": 3, 2024-04-21T23:29:06.147 INFO:teuthology.orchestra.run.smithi174.stdout: "op_reply": 2199, 2024-04-21T23:29:06.148 INFO:teuthology.orchestra.run.smithi174.stdout: "op_latency": { 2024-04-21T23:29:06.148 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 2199, 2024-04-21T23:29:06.148 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 1.074984917, 2024-04-21T23:29:06.148 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000488851 2024-04-21T23:29:06.148 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:06.148 INFO:teuthology.orchestra.run.smithi174.stdout: "op_inflight": 0, 2024-04-21T23:29:06.148 INFO:teuthology.orchestra.run.smithi174.stdout: "oplen_avg": { 2024-04-21T23:29:06.148 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 2199, 2024-04-21T23:29:06.149 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 2200 2024-04-21T23:29:06.149 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:06.149 INFO:teuthology.orchestra.run.smithi174.stdout: "op": 2199, 2024-04-21T23:29:06.149 INFO:teuthology.orchestra.run.smithi174.stdout: "op_r": 2199, 2024-04-21T23:29:06.149 INFO:teuthology.orchestra.run.smithi174.stdout: "op_w": 0, 2024-04-21T23:29:06.149 INFO:teuthology.orchestra.run.smithi174.stdout: "op_rmw": 0, 2024-04-21T23:29:06.149 INFO:teuthology.orchestra.run.smithi174.stdout: "op_pg": 0, 2024-04-21T23:29:06.149 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_stat": 1405, 2024-04-21T23:29:06.150 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_create": 0, 2024-04-21T23:29:06.150 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_read": 792, 2024-04-21T23:29:06.150 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_write": 0, 2024-04-21T23:29:06.150 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_writefull": 0, 2024-04-21T23:29:06.150 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_writesame": 0, 2024-04-21T23:29:06.150 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_append": 0, 2024-04-21T23:29:06.150 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_zero": 0, 2024-04-21T23:29:06.150 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_truncate": 0, 2024-04-21T23:29:06.150 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_delete": 0, 2024-04-21T23:29:06.151 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_mapext": 0, 2024-04-21T23:29:06.151 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_sparse_read": 0, 2024-04-21T23:29:06.151 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_clonerange": 0, 2024-04-21T23:29:06.151 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_getxattr": 1, 2024-04-21T23:29:06.151 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_setxattr": 0, 2024-04-21T23:29:06.151 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_cmpxattr": 0, 2024-04-21T23:29:06.151 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_rmxattr": 0, 2024-04-21T23:29:06.151 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_resetxattrs": 0, 2024-04-21T23:29:06.152 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_call": 0, 2024-04-21T23:29:06.152 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_watch": 0, 2024-04-21T23:29:06.152 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_notify": 0, 2024-04-21T23:29:06.152 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_src_cmpxattr": 0, 2024-04-21T23:29:06.152 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_pgls": 0, 2024-04-21T23:29:06.152 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_pgls_filter": 0, 2024-04-21T23:29:06.152 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_other": 0, 2024-04-21T23:29:06.152 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_active": 0, 2024-04-21T23:29:06.153 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_send": 0, 2024-04-21T23:29:06.153 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_resend": 0, 2024-04-21T23:29:06.153 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_ping": 0, 2024-04-21T23:29:06.153 INFO:teuthology.orchestra.run.smithi174.stdout: "poolop_active": 0, 2024-04-21T23:29:06.153 INFO:teuthology.orchestra.run.smithi174.stdout: "poolop_send": 0, 2024-04-21T23:29:06.153 INFO:teuthology.orchestra.run.smithi174.stdout: "poolop_resend": 0, 2024-04-21T23:29:06.153 INFO:teuthology.orchestra.run.smithi174.stdout: "poolstat_active": 0, 2024-04-21T23:29:06.153 INFO:teuthology.orchestra.run.smithi174.stdout: "poolstat_send": 0, 2024-04-21T23:29:06.153 INFO:teuthology.orchestra.run.smithi174.stdout: "poolstat_resend": 0, 2024-04-21T23:29:06.154 INFO:teuthology.orchestra.run.smithi174.stdout: "statfs_active": 0, 2024-04-21T23:29:06.154 INFO:teuthology.orchestra.run.smithi174.stdout: "statfs_send": 0, 2024-04-21T23:29:06.154 INFO:teuthology.orchestra.run.smithi174.stdout: "statfs_resend": 0, 2024-04-21T23:29:06.154 INFO:teuthology.orchestra.run.smithi174.stdout: "command_active": 0, 2024-04-21T23:29:06.154 INFO:teuthology.orchestra.run.smithi174.stdout: "command_send": 0, 2024-04-21T23:29:06.154 INFO:teuthology.orchestra.run.smithi174.stdout: "command_resend": 0, 2024-04-21T23:29:06.154 INFO:teuthology.orchestra.run.smithi174.stdout: "map_epoch": 103, 2024-04-21T23:29:06.154 INFO:teuthology.orchestra.run.smithi174.stdout: "map_full": 0, 2024-04-21T23:29:06.155 INFO:teuthology.orchestra.run.smithi174.stdout: "map_inc": 13, 2024-04-21T23:29:06.155 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_sessions": 4, 2024-04-21T23:29:06.155 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_session_open": 4, 2024-04-21T23:29:06.155 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_session_close": 0, 2024-04-21T23:29:06.155 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_laggy": 0, 2024-04-21T23:29:06.155 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_wr": 0, 2024-04-21T23:29:06.155 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_rd": 2, 2024-04-21T23:29:06.155 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_del": 0 2024-04-21T23:29:06.156 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:06.156 INFO:teuthology.orchestra.run.smithi174.stdout: "oft": { 2024-04-21T23:29:06.156 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_objs": 0, 2024-04-21T23:29:06.156 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_kv_pairs": 0, 2024-04-21T23:29:06.156 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_updates": 0, 2024-04-21T23:29:06.156 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_removes": 0 2024-04-21T23:29:06.156 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:06.156 INFO:teuthology.orchestra.run.smithi174.stdout: "osd-slow-ops": { 2024-04-21T23:29:06.157 INFO:teuthology.orchestra.run.smithi174.stdout: "slow_ops_count": 0 2024-04-21T23:29:06.157 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:06.157 INFO:teuthology.orchestra.run.smithi174.stdout: "purge_queue": { 2024-04-21T23:29:06.157 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing_ops": 0, 2024-04-21T23:29:06.157 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing_ops_high_water": 0, 2024-04-21T23:29:06.157 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing": 0, 2024-04-21T23:29:06.157 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing_high_water": 0, 2024-04-21T23:29:06.157 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executed_ops": 0, 2024-04-21T23:29:06.158 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executed": 0, 2024-04-21T23:29:06.158 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_item_in_journal": 0 2024-04-21T23:29:06.158 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:06.158 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-21T23:29:06.158 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-04-21T23:29:06.158 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 104857600, 2024-04-21T23:29:06.158 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-04-21T23:29:06.158 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 2458, 2024-04-21T23:29:06.159 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 2613093, 2024-04-21T23:29:06.159 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-04-21T23:29:06.159 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 2458, 2024-04-21T23:29:06.159 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 0, 2024-04-21T23:29:06.159 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 0, 2024-04-21T23:29:06.159 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 2458, 2024-04-21T23:29:06.159 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 2613093, 2024-04-21T23:29:06.159 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-04-21T23:29:06.159 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:06.160 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:06.160 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:06.160 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-04-21T23:29:06.160 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:06.160 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-objecter_bytes": { 2024-04-21T23:29:06.160 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-04-21T23:29:06.160 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 104857600, 2024-04-21T23:29:06.160 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-04-21T23:29:06.161 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 0, 2024-04-21T23:29:06.161 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 0, 2024-04-21T23:29:06.161 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-04-21T23:29:06.161 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 0, 2024-04-21T23:29:06.162 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 2199, 2024-04-21T23:29:06.162 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 578211, 2024-04-21T23:29:06.163 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 2199, 2024-04-21T23:29:06.163 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 578211, 2024-04-21T23:29:06.163 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-04-21T23:29:06.163 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:06.163 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:06.164 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:06.164 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-04-21T23:29:06.164 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:06.164 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-objecter_ops": { 2024-04-21T23:29:06.164 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-04-21T23:29:06.164 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 1024, 2024-04-21T23:29:06.164 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-04-21T23:29:06.165 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 0, 2024-04-21T23:29:06.165 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 0, 2024-04-21T23:29:06.165 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-04-21T23:29:06.165 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 0, 2024-04-21T23:29:06.165 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 2199, 2024-04-21T23:29:06.165 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 2199, 2024-04-21T23:29:06.165 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 2199, 2024-04-21T23:29:06.165 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 2199, 2024-04-21T23:29:06.165 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-04-21T23:29:06.166 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:06.166 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:06.166 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:06.166 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-04-21T23:29:06.166 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:06.166 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-write_buf_throttle": { 2024-04-21T23:29:06.166 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-04-21T23:29:06.166 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 3758096384, 2024-04-21T23:29:06.167 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-04-21T23:29:06.167 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 0, 2024-04-21T23:29:06.167 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 0, 2024-04-21T23:29:06.167 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-04-21T23:29:06.167 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 0, 2024-04-21T23:29:06.167 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 0, 2024-04-21T23:29:06.167 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 0, 2024-04-21T23:29:06.167 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 0, 2024-04-21T23:29:06.167 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 0, 2024-04-21T23:29:06.168 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-04-21T23:29:06.168 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:06.168 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:06.168 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:06.168 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-04-21T23:29:06.168 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:06.168 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-write_buf_throttle-0x55df7d6d86e0": { 2024-04-21T23:29:06.168 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-04-21T23:29:06.168 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 3758096384, 2024-04-21T23:29:06.169 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-04-21T23:29:06.169 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 0, 2024-04-21T23:29:06.169 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 0, 2024-04-21T23:29:06.169 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-04-21T23:29:06.169 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 0, 2024-04-21T23:29:06.169 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 0, 2024-04-21T23:29:06.169 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 0, 2024-04-21T23:29:06.169 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 0, 2024-04-21T23:29:06.169 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 0, 2024-04-21T23:29:06.169 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-04-21T23:29:06.170 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:06.170 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:06.170 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:06.170 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-04-21T23:29:06.170 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-04-21T23:29:06.170 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-04-21T23:29:06.904 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-04-21T23:29:06.905 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-04-21T23:29:06.905 INFO:tasks.check_counter:Daemon mds.f mds.root_rsnaps=0 2024-04-21T23:29:06.905 INFO:tasks.check_counter:Daemon mds.f mds_server.req_mksnap_latency.avgcount=0 2024-04-21T23:29:06.905 INFO:tasks.check_counter:Daemon mds.f mds_server.req_rmsnap_latency.avgcount=0 2024-04-21T23:29:06.905 INFO:tasks.check_counter:Daemon mds.f mds.dir_split=0 2024-04-21T23:29:06.905 DEBUG:tasks.check_counter:Getting stats from i 2024-04-21T23:29:06.906 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph daemon mds.i perf dump 2024-04-21T23:29:07.142 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.c/config 2024-04-21T23:29:07.181 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:29:06 smithi040 ceph-mon[27984]: pgmap v332: 129 pgs: 129 active+clean; 95 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-21T23:29:07.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:29:06 smithi174 ceph-mon[31409]: pgmap v332: 129 pgs: 129 active+clean; 95 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-21T23:29:07.193 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:29:06 smithi111 ceph-mon[31085]: pgmap v332: 129 pgs: 129 active+clean; 95 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-21T23:29:08.863 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:29:08 smithi174 ceph-mon[31409]: pgmap v333: 129 pgs: 129 active+clean; 95 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-04-21T23:29:08.888 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-04-21T23:29:08.888 INFO:teuthology.orchestra.run.smithi174.stdout: "AsyncMessenger::Worker-0": { 2024-04-21T23:29:08.888 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_messages": 397, 2024-04-21T23:29:08.888 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages": 707, 2024-04-21T23:29:08.888 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_bytes": 113367, 2024-04-21T23:29:08.889 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_bytes": 1194620, 2024-04-21T23:29:08.889 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_created_connections": 16, 2024-04-21T23:29:08.889 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_active_connections": 5, 2024-04-21T23:29:08.889 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_total_time": 0.054541790, 2024-04-21T23:29:08.889 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_send_time": 0.020782290, 2024-04-21T23:29:08.889 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_recv_time": 0.030018508, 2024-04-21T23:29:08.889 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_fast_dispatch_time": 0.002889024, 2024-04-21T23:29:08.889 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T23:29:08.890 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 705, 2024-04-21T23:29:08.890 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.045699024, 2024-04-21T23:29:08.890 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000064821 2024-04-21T23:29:08.890 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:08.890 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_handle_ack_lat": { 2024-04-21T23:29:08.890 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 701, 2024-04-21T23:29:08.890 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000172973, 2024-04-21T23:29:08.890 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000246 2024-04-21T23:29:08.890 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:08.891 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-21T23:29:08.891 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_encrypted_bytes": 318720 2024-04-21T23:29:08.891 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:08.891 INFO:teuthology.orchestra.run.smithi174.stdout: "AsyncMessenger::Worker-1": { 2024-04-21T23:29:08.891 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_messages": 1183, 2024-04-21T23:29:08.891 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages": 747, 2024-04-21T23:29:08.891 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_bytes": 511409, 2024-04-21T23:29:08.891 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_bytes": 376144, 2024-04-21T23:29:08.892 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_created_connections": 18, 2024-04-21T23:29:08.892 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_active_connections": 7, 2024-04-21T23:29:08.892 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_total_time": 0.083787511, 2024-04-21T23:29:08.892 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_send_time": 0.030542043, 2024-04-21T23:29:08.892 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_recv_time": 0.068757179, 2024-04-21T23:29:08.892 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_fast_dispatch_time": 0.001257788, 2024-04-21T23:29:08.892 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T23:29:08.892 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 745, 2024-04-21T23:29:08.893 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.029194992, 2024-04-21T23:29:08.893 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000039187 2024-04-21T23:29:08.893 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:08.893 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_handle_ack_lat": { 2024-04-21T23:29:08.893 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 1743, 2024-04-21T23:29:08.893 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000273759, 2024-04-21T23:29:08.893 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000157 2024-04-21T23:29:08.893 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:08.894 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_encrypted_bytes": 6864, 2024-04-21T23:29:08.894 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_encrypted_bytes": 136240 2024-04-21T23:29:08.894 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:08.894 INFO:teuthology.orchestra.run.smithi174.stdout: "AsyncMessenger::Worker-2": { 2024-04-21T23:29:08.894 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_messages": 727, 2024-04-21T23:29:08.894 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages": 890, 2024-04-21T23:29:08.894 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_bytes": 1803978, 2024-04-21T23:29:08.894 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_bytes": 430432, 2024-04-21T23:29:08.894 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_created_connections": 7, 2024-04-21T23:29:08.895 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_active_connections": 6, 2024-04-21T23:29:08.895 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_total_time": 0.109230615, 2024-04-21T23:29:08.895 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_send_time": 0.039279891, 2024-04-21T23:29:08.895 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_recv_time": 0.063988246, 2024-04-21T23:29:08.895 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_fast_dispatch_time": 0.012815887, 2024-04-21T23:29:08.895 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T23:29:08.895 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 889, 2024-04-21T23:29:08.895 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.052536489, 2024-04-21T23:29:08.895 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000059096 2024-04-21T23:29:08.896 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:08.896 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_handle_ack_lat": { 2024-04-21T23:29:08.896 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 856, 2024-04-21T23:29:08.896 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000240704, 2024-04-21T23:29:08.896 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000281 2024-04-21T23:29:08.896 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:08.896 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_encrypted_bytes": 1659792, 2024-04-21T23:29:08.896 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_encrypted_bytes": 143264 2024-04-21T23:29:08.896 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:08.897 INFO:teuthology.orchestra.run.smithi174.stdout: "cct": { 2024-04-21T23:29:08.897 INFO:teuthology.orchestra.run.smithi174.stdout: "total_workers": 1, 2024-04-21T23:29:08.897 INFO:teuthology.orchestra.run.smithi174.stdout: "unhealthy_workers": 0 2024-04-21T23:29:08.897 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:08.897 INFO:teuthology.orchestra.run.smithi174.stdout: "finisher-MDSRank": { 2024-04-21T23:29:08.897 INFO:teuthology.orchestra.run.smithi174.stdout: "queue_len": 0, 2024-04-21T23:29:08.897 INFO:teuthology.orchestra.run.smithi174.stdout: "complete_latency": { 2024-04-21T23:29:08.897 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 202, 2024-04-21T23:29:08.898 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.100835756, 2024-04-21T23:29:08.898 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000499186 2024-04-21T23:29:08.898 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-04-21T23:29:08.898 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:08.898 INFO:teuthology.orchestra.run.smithi174.stdout: "finisher-PurgeQueue": { 2024-04-21T23:29:08.898 INFO:teuthology.orchestra.run.smithi174.stdout: "queue_len": 0, 2024-04-21T23:29:08.898 INFO:teuthology.orchestra.run.smithi174.stdout: "complete_latency": { 2024-04-21T23:29:08.898 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 19, 2024-04-21T23:29:08.899 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000442603, 2024-04-21T23:29:08.899 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000023294 2024-04-21T23:29:08.899 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-04-21T23:29:08.899 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:08.899 INFO:teuthology.orchestra.run.smithi174.stdout: "mds": { 2024-04-21T23:29:08.899 INFO:teuthology.orchestra.run.smithi174.stdout: "request": 102, 2024-04-21T23:29:08.899 INFO:teuthology.orchestra.run.smithi174.stdout: "reply": 40, 2024-04-21T23:29:08.899 INFO:teuthology.orchestra.run.smithi174.stdout: "reply_latency": { 2024-04-21T23:29:08.899 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 40, 2024-04-21T23:29:08.900 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.035733669, 2024-04-21T23:29:08.900 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000893341 2024-04-21T23:29:08.900 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:08.900 INFO:teuthology.orchestra.run.smithi174.stdout: "slow_reply": 0, 2024-04-21T23:29:08.900 INFO:teuthology.orchestra.run.smithi174.stdout: "forward": 62, 2024-04-21T23:29:08.900 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_fetch_complete": 0, 2024-04-21T23:29:08.900 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_fetch_keys": 0, 2024-04-21T23:29:08.900 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_commit": 11, 2024-04-21T23:29:08.901 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_split": 0, 2024-04-21T23:29:08.901 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_merge": 0, 2024-04-21T23:29:08.901 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes": 201, 2024-04-21T23:29:08.901 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_top": 9, 2024-04-21T23:29:08.901 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_bottom": 192, 2024-04-21T23:29:08.901 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_pin_tail": 0, 2024-04-21T23:29:08.901 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_pinned": 185, 2024-04-21T23:29:08.901 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_expired": 1, 2024-04-21T23:29:08.901 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_with_caps": 114, 2024-04-21T23:29:08.902 INFO:teuthology.orchestra.run.smithi174.stdout: "caps": 114, 2024-04-21T23:29:08.902 INFO:teuthology.orchestra.run.smithi174.stdout: "subtrees": 11, 2024-04-21T23:29:08.902 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse": 160, 2024-04-21T23:29:08.902 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_hit": 49, 2024-04-21T23:29:08.902 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_forward": 61, 2024-04-21T23:29:08.902 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_discover": 16, 2024-04-21T23:29:08.902 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_dir_fetch": 0, 2024-04-21T23:29:08.902 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_remote_ino": 0, 2024-04-21T23:29:08.902 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_lock": 0, 2024-04-21T23:29:08.903 INFO:teuthology.orchestra.run.smithi174.stdout: "load_cent": 0, 2024-04-21T23:29:08.903 INFO:teuthology.orchestra.run.smithi174.stdout: "q": 0, 2024-04-21T23:29:08.903 INFO:teuthology.orchestra.run.smithi174.stdout: "exported": 0, 2024-04-21T23:29:08.903 INFO:teuthology.orchestra.run.smithi174.stdout: "exported_inodes": 0, 2024-04-21T23:29:08.903 INFO:teuthology.orchestra.run.smithi174.stdout: "imported": 9, 2024-04-21T23:29:08.903 INFO:teuthology.orchestra.run.smithi174.stdout: "imported_inodes": 155, 2024-04-21T23:29:08.903 INFO:teuthology.orchestra.run.smithi174.stdout: "openino_dir_fetch": 0, 2024-04-21T23:29:08.903 INFO:teuthology.orchestra.run.smithi174.stdout: "openino_backtrace_fetch": 142, 2024-04-21T23:29:08.904 INFO:teuthology.orchestra.run.smithi174.stdout: "openino_peer_discover": 1, 2024-04-21T23:29:08.904 INFO:teuthology.orchestra.run.smithi174.stdout: "root_rfiles": 1313, 2024-04-21T23:29:08.904 INFO:teuthology.orchestra.run.smithi174.stdout: "root_rbytes": 2097386, 2024-04-21T23:29:08.904 INFO:teuthology.orchestra.run.smithi174.stdout: "root_rsnaps": 2, 2024-04-21T23:29:08.904 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_backtrace_fetch": 142, 2024-04-21T23:29:08.904 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_set_tag": 0, 2024-04-21T23:29:08.904 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_backtrace_repaired": 0, 2024-04-21T23:29:08.904 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_inotable_repaired": 0, 2024-04-21T23:29:08.904 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_dir_inodes": 18, 2024-04-21T23:29:08.905 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_dir_base_inodes": 0, 2024-04-21T23:29:08.905 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_dirfrag_rstats": 18, 2024-04-21T23:29:08.905 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_file_inodes": 124, 2024-04-21T23:29:08.905 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_inode_file_caps": 0, 2024-04-21T23:29:08.905 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_revoke": 2, 2024-04-21T23:29:08.905 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_grant": 24, 2024-04-21T23:29:08.905 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_trunc": 0, 2024-04-21T23:29:08.905 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-21T23:29:08.906 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-21T23:29:08.906 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_caps": 169, 2024-04-21T23:29:08.906 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_caps_dirty": 146, 2024-04-21T23:29:08.906 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_cap_release": 6, 2024-04-21T23:29:08.906 INFO:teuthology.orchestra.run.smithi174.stdout: "process_request_cap_release": 13 2024-04-21T23:29:08.906 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:08.906 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_cache": { 2024-04-21T23:29:08.907 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_update": 0, 2024-04-21T23:29:08.907 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_update_receipt": 0, 2024-04-21T23:29:08.907 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_try_discover": 0, 2024-04-21T23:29:08.907 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_send_discover": 18, 2024-04-21T23:29:08.907 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_handle_discover": 0, 2024-04-21T23:29:08.907 INFO:teuthology.orchestra.run.smithi174.stdout: "num_strays": 0, 2024-04-21T23:29:08.907 INFO:teuthology.orchestra.run.smithi174.stdout: "num_strays_delayed": 0, 2024-04-21T23:29:08.907 INFO:teuthology.orchestra.run.smithi174.stdout: "num_strays_enqueuing": 0, 2024-04-21T23:29:08.907 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_created": 4, 2024-04-21T23:29:08.908 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_enqueued": 4, 2024-04-21T23:29:08.908 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_reintegrated": 0, 2024-04-21T23:29:08.908 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_migrated": 0, 2024-04-21T23:29:08.908 INFO:teuthology.orchestra.run.smithi174.stdout: "num_recovering_processing": 0, 2024-04-21T23:29:08.908 INFO:teuthology.orchestra.run.smithi174.stdout: "num_recovering_enqueued": 0, 2024-04-21T23:29:08.908 INFO:teuthology.orchestra.run.smithi174.stdout: "num_recovering_prioritized": 0, 2024-04-21T23:29:08.908 INFO:teuthology.orchestra.run.smithi174.stdout: "recovery_started": 0, 2024-04-21T23:29:08.908 INFO:teuthology.orchestra.run.smithi174.stdout: "recovery_completed": 0, 2024-04-21T23:29:08.909 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_quiesce_path": 0, 2024-04-21T23:29:08.909 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_quiesce_inode": 0, 2024-04-21T23:29:08.909 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_enqueue_scrub": 0, 2024-04-21T23:29:08.909 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_exportdir": 0, 2024-04-21T23:29:08.909 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_flush": 0, 2024-04-21T23:29:08.909 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_fragmentdir": 0, 2024-04-21T23:29:08.909 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_fragstats": 0, 2024-04-21T23:29:08.909 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_inodestats": 0 2024-04-21T23:29:08.909 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:08.910 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_log": { 2024-04-21T23:29:08.910 INFO:teuthology.orchestra.run.smithi174.stdout: "evlrg": 0, 2024-04-21T23:29:08.910 INFO:teuthology.orchestra.run.smithi174.stdout: "evadd": 234, 2024-04-21T23:29:08.910 INFO:teuthology.orchestra.run.smithi174.stdout: "evex": 0, 2024-04-21T23:29:08.910 INFO:teuthology.orchestra.run.smithi174.stdout: "evtrm": 0, 2024-04-21T23:29:08.911 INFO:teuthology.orchestra.run.smithi174.stdout: "ev": 234, 2024-04-21T23:29:08.911 INFO:teuthology.orchestra.run.smithi174.stdout: "evexg": 0, 2024-04-21T23:29:08.911 INFO:teuthology.orchestra.run.smithi174.stdout: "evexd": 0, 2024-04-21T23:29:08.911 INFO:teuthology.orchestra.run.smithi174.stdout: "segadd": 2, 2024-04-21T23:29:08.911 INFO:teuthology.orchestra.run.smithi174.stdout: "segex": 0, 2024-04-21T23:29:08.911 INFO:teuthology.orchestra.run.smithi174.stdout: "segtrm": 0, 2024-04-21T23:29:08.911 INFO:teuthology.orchestra.run.smithi174.stdout: "seg": 2, 2024-04-21T23:29:08.911 INFO:teuthology.orchestra.run.smithi174.stdout: "segmjr": 2, 2024-04-21T23:29:08.911 INFO:teuthology.orchestra.run.smithi174.stdout: "segexg": 0, 2024-04-21T23:29:08.912 INFO:teuthology.orchestra.run.smithi174.stdout: "segexd": 0, 2024-04-21T23:29:08.912 INFO:teuthology.orchestra.run.smithi174.stdout: "expos": 4194304, 2024-04-21T23:29:08.912 INFO:teuthology.orchestra.run.smithi174.stdout: "wrpos": 4772510, 2024-04-21T23:29:08.912 INFO:teuthology.orchestra.run.smithi174.stdout: "rdpos": 0, 2024-04-21T23:29:08.912 INFO:teuthology.orchestra.run.smithi174.stdout: "jlat": { 2024-04-21T23:29:08.912 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 47, 2024-04-21T23:29:08.912 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.279595167, 2024-04-21T23:29:08.912 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.005948833 2024-04-21T23:29:08.913 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:08.913 INFO:teuthology.orchestra.run.smithi174.stdout: "replayed": 0 2024-04-21T23:29:08.913 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:08.913 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_mem": { 2024-04-21T23:29:08.913 INFO:teuthology.orchestra.run.smithi174.stdout: "ino": 188, 2024-04-21T23:29:08.913 INFO:teuthology.orchestra.run.smithi174.stdout: "ino+": 192, 2024-04-21T23:29:08.913 INFO:teuthology.orchestra.run.smithi174.stdout: "ino-": 4, 2024-04-21T23:29:08.913 INFO:teuthology.orchestra.run.smithi174.stdout: "dir": 45, 2024-04-21T23:29:08.913 INFO:teuthology.orchestra.run.smithi174.stdout: "dir+": 46, 2024-04-21T23:29:08.914 INFO:teuthology.orchestra.run.smithi174.stdout: "dir-": 1, 2024-04-21T23:29:08.914 INFO:teuthology.orchestra.run.smithi174.stdout: "dn": 201, 2024-04-21T23:29:08.914 INFO:teuthology.orchestra.run.smithi174.stdout: "dn+": 207, 2024-04-21T23:29:08.914 INFO:teuthology.orchestra.run.smithi174.stdout: "dn-": 6, 2024-04-21T23:29:08.914 INFO:teuthology.orchestra.run.smithi174.stdout: "cap": 114, 2024-04-21T23:29:08.914 INFO:teuthology.orchestra.run.smithi174.stdout: "cap+": 156, 2024-04-21T23:29:08.914 INFO:teuthology.orchestra.run.smithi174.stdout: "cap-": 42, 2024-04-21T23:29:08.914 INFO:teuthology.orchestra.run.smithi174.stdout: "rss": 55784, 2024-04-21T23:29:08.915 INFO:teuthology.orchestra.run.smithi174.stdout: "heap": 223612 2024-04-21T23:29:08.915 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:08.915 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_server": { 2024-04-21T23:29:08.915 INFO:teuthology.orchestra.run.smithi174.stdout: "dispatch_client_request": 121, 2024-04-21T23:29:08.915 INFO:teuthology.orchestra.run.smithi174.stdout: "dispatch_server_request": 0, 2024-04-21T23:29:08.915 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_request": 102, 2024-04-21T23:29:08.915 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_session": 17, 2024-04-21T23:29:08.915 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_peer_request": 11, 2024-04-21T23:29:08.915 INFO:teuthology.orchestra.run.smithi174.stdout: "req_create_latency": { 2024-04-21T23:29:08.916 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 7, 2024-04-21T23:29:08.916 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.014360121, 2024-04-21T23:29:08.916 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.002051445 2024-04-21T23:29:08.916 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:08.916 INFO:teuthology.orchestra.run.smithi174.stdout: "req_getattr_latency": { 2024-04-21T23:29:08.916 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 2, 2024-04-21T23:29:08.916 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000764304, 2024-04-21T23:29:08.916 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000382152 2024-04-21T23:29:08.917 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:08.917 INFO:teuthology.orchestra.run.smithi174.stdout: "req_getfilelock_latency": { 2024-04-21T23:29:08.917 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:08.917 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:08.917 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:08.917 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:08.917 INFO:teuthology.orchestra.run.smithi174.stdout: "req_link_latency": { 2024-04-21T23:29:08.917 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:08.917 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:08.918 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:08.918 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:08.918 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookup_latency": { 2024-04-21T23:29:08.918 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 24, 2024-04-21T23:29:08.918 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.006425642, 2024-04-21T23:29:08.918 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000267735 2024-04-21T23:29:08.918 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:08.918 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookuphash_latency": { 2024-04-21T23:29:08.918 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:08.919 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:08.919 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:08.919 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:08.919 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupino_latency": { 2024-04-21T23:29:08.919 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:08.919 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:08.919 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:08.919 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:08.919 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupname_latency": { 2024-04-21T23:29:08.920 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:08.920 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:08.920 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:08.920 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:08.920 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupparent_latency": { 2024-04-21T23:29:08.920 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:08.920 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:08.920 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:08.921 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:08.921 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupsnap_latency": { 2024-04-21T23:29:08.921 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:08.921 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:08.921 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:08.921 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:08.921 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lssnap_latency": { 2024-04-21T23:29:08.921 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:08.922 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:08.922 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:08.923 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:08.923 INFO:teuthology.orchestra.run.smithi174.stdout: "req_mkdir_latency": { 2024-04-21T23:29:08.923 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:08.923 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:08.923 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:08.923 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:08.923 INFO:teuthology.orchestra.run.smithi174.stdout: "req_mknod_latency": { 2024-04-21T23:29:08.923 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:08.924 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:08.924 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:08.924 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:08.924 INFO:teuthology.orchestra.run.smithi174.stdout: "req_mksnap_latency": { 2024-04-21T23:29:08.924 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:08.924 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:08.924 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:08.924 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:08.924 INFO:teuthology.orchestra.run.smithi174.stdout: "req_open_latency": { 2024-04-21T23:29:08.925 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 1, 2024-04-21T23:29:08.925 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000252722, 2024-04-21T23:29:08.925 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000252722 2024-04-21T23:29:08.925 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:08.925 INFO:teuthology.orchestra.run.smithi174.stdout: "req_readdir_latency": { 2024-04-21T23:29:08.925 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 2, 2024-04-21T23:29:08.925 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000974175, 2024-04-21T23:29:08.925 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000487087 2024-04-21T23:29:08.926 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:08.926 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rename_latency": { 2024-04-21T23:29:08.926 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:08.926 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:08.926 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:08.926 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:08.926 INFO:teuthology.orchestra.run.smithi174.stdout: "req_renamesnap_latency": { 2024-04-21T23:29:08.926 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:08.926 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:08.927 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:08.927 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:08.927 INFO:teuthology.orchestra.run.smithi174.stdout: "req_snapdiff_latency": { 2024-04-21T23:29:08.927 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:08.927 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:08.927 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:08.927 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:08.927 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rmdir_latency": { 2024-04-21T23:29:08.928 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:08.928 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:08.928 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:08.928 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:08.928 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rmsnap_latency": { 2024-04-21T23:29:08.928 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:08.928 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:08.928 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:08.928 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:08.929 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rmxattr_latency": { 2024-04-21T23:29:08.929 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:08.929 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:08.929 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:08.929 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:08.929 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setattr_latency": { 2024-04-21T23:29:08.929 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:08.929 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:08.930 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:08.930 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:08.930 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setdirlayout_latency": { 2024-04-21T23:29:08.930 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:08.930 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:08.930 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:08.930 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:08.930 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setfilelock_latency": { 2024-04-21T23:29:08.931 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:08.931 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:08.931 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:08.931 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:08.931 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setlayout_latency": { 2024-04-21T23:29:08.931 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:08.931 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:08.931 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:08.931 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:08.932 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setxattr_latency": { 2024-04-21T23:29:08.932 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:08.932 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:08.932 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:08.932 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:08.932 INFO:teuthology.orchestra.run.smithi174.stdout: "req_symlink_latency": { 2024-04-21T23:29:08.932 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:08.932 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:08.933 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:08.933 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:08.933 INFO:teuthology.orchestra.run.smithi174.stdout: "req_unlink_latency": { 2024-04-21T23:29:08.933 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 4, 2024-04-21T23:29:08.933 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.012956705, 2024-04-21T23:29:08.933 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.003239176 2024-04-21T23:29:08.933 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:08.933 INFO:teuthology.orchestra.run.smithi174.stdout: "cap_revoke_eviction": 0, 2024-04-21T23:29:08.933 INFO:teuthology.orchestra.run.smithi174.stdout: "cap_acquisition_throttle": 0, 2024-04-21T23:29:08.934 INFO:teuthology.orchestra.run.smithi174.stdout: "req_getvxattr_latency": { 2024-04-21T23:29:08.934 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:08.934 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:08.934 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:08.934 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-04-21T23:29:08.934 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:08.934 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_sessions": { 2024-04-21T23:29:08.934 INFO:teuthology.orchestra.run.smithi174.stdout: "session_count": 1, 2024-04-21T23:29:08.935 INFO:teuthology.orchestra.run.smithi174.stdout: "session_add": 3, 2024-04-21T23:29:08.935 INFO:teuthology.orchestra.run.smithi174.stdout: "session_remove": 2, 2024-04-21T23:29:08.935 INFO:teuthology.orchestra.run.smithi174.stdout: "sessions_open": 1, 2024-04-21T23:29:08.935 INFO:teuthology.orchestra.run.smithi174.stdout: "sessions_stale": 0, 2024-04-21T23:29:08.935 INFO:teuthology.orchestra.run.smithi174.stdout: "total_load": 28, 2024-04-21T23:29:08.935 INFO:teuthology.orchestra.run.smithi174.stdout: "average_load": 28, 2024-04-21T23:29:08.935 INFO:teuthology.orchestra.run.smithi174.stdout: "avg_session_uptime": 633, 2024-04-21T23:29:08.935 INFO:teuthology.orchestra.run.smithi174.stdout: "mdthresh_evicted": 0 2024-04-21T23:29:08.935 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:08.936 INFO:teuthology.orchestra.run.smithi174.stdout: "mempool": { 2024-04-21T23:29:08.936 INFO:teuthology.orchestra.run.smithi174.stdout: "bloom_filter_bytes": 0, 2024-04-21T23:29:08.936 INFO:teuthology.orchestra.run.smithi174.stdout: "bloom_filter_items": 0, 2024-04-21T23:29:08.936 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_alloc_bytes": 0, 2024-04-21T23:29:08.936 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_alloc_items": 0, 2024-04-21T23:29:08.936 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_data_bytes": 0, 2024-04-21T23:29:08.936 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_data_items": 0, 2024-04-21T23:29:08.936 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-21T23:29:08.936 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_onode_items": 0, 2024-04-21T23:29:08.937 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-21T23:29:08.937 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_meta_items": 0, 2024-04-21T23:29:08.937 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_other_bytes": 0, 2024-04-21T23:29:08.937 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_other_items": 0, 2024-04-21T23:29:08.937 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-21T23:29:08.937 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_buffer_items": 0, 2024-04-21T23:29:08.937 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_extent_bytes": 0, 2024-04-21T23:29:08.937 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_extent_items": 0, 2024-04-21T23:29:08.938 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_blob_bytes": 0, 2024-04-21T23:29:08.938 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_blob_items": 0, 2024-04-21T23:29:08.938 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-21T23:29:08.938 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_shared_blob_items": 0, 2024-04-21T23:29:08.938 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-21T23:29:08.938 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_inline_bl_items": 0, 2024-04-21T23:29:08.938 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_fsck_bytes": 0, 2024-04-21T23:29:08.938 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_fsck_items": 0, 2024-04-21T23:29:08.938 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_txc_bytes": 0, 2024-04-21T23:29:08.939 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_txc_items": 0, 2024-04-21T23:29:08.939 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-21T23:29:08.939 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_deferred_items": 0, 2024-04-21T23:29:08.939 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_bytes": 0, 2024-04-21T23:29:08.939 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_items": 0, 2024-04-21T23:29:08.939 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_bytes": 0, 2024-04-21T23:29:08.939 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_items": 0, 2024-04-21T23:29:08.939 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_reader_bytes": 0, 2024-04-21T23:29:08.940 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_reader_items": 0, 2024-04-21T23:29:08.940 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_writer_bytes": 0, 2024-04-21T23:29:08.940 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_writer_items": 0, 2024-04-21T23:29:08.940 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_anon_bytes": 305709, 2024-04-21T23:29:08.940 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_anon_items": 291, 2024-04-21T23:29:08.940 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_meta_bytes": 616, 2024-04-21T23:29:08.940 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_meta_items": 7, 2024-04-21T23:29:08.940 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_bytes": 0, 2024-04-21T23:29:08.940 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_items": 0, 2024-04-21T23:29:08.941 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_mapbl_bytes": 0, 2024-04-21T23:29:08.941 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_mapbl_items": 0, 2024-04-21T23:29:08.941 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_pglog_bytes": 0, 2024-04-21T23:29:08.941 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_pglog_items": 0, 2024-04-21T23:29:08.941 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_bytes": 9704, 2024-04-21T23:29:08.941 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_items": 186, 2024-04-21T23:29:08.941 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_mapping_bytes": 0, 2024-04-21T23:29:08.941 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_mapping_items": 0, 2024-04-21T23:29:08.942 INFO:teuthology.orchestra.run.smithi174.stdout: "pgmap_bytes": 0, 2024-04-21T23:29:08.942 INFO:teuthology.orchestra.run.smithi174.stdout: "pgmap_items": 0, 2024-04-21T23:29:08.942 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_co_bytes": 698242, 2024-04-21T23:29:08.942 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_co_items": 7483, 2024-04-21T23:29:08.942 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_1_bytes": 0, 2024-04-21T23:29:08.942 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_1_items": 0, 2024-04-21T23:29:08.942 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_2_bytes": 0, 2024-04-21T23:29:08.942 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_2_items": 0 2024-04-21T23:29:08.943 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:08.943 INFO:teuthology.orchestra.run.smithi174.stdout: "objecter": { 2024-04-21T23:29:08.943 INFO:teuthology.orchestra.run.smithi174.stdout: "op_active": 0, 2024-04-21T23:29:08.943 INFO:teuthology.orchestra.run.smithi174.stdout: "op_laggy": 0, 2024-04-21T23:29:08.943 INFO:teuthology.orchestra.run.smithi174.stdout: "op_send": 262, 2024-04-21T23:29:08.943 INFO:teuthology.orchestra.run.smithi174.stdout: "op_send_bytes": 604677, 2024-04-21T23:29:08.943 INFO:teuthology.orchestra.run.smithi174.stdout: "op_resend": 28, 2024-04-21T23:29:08.943 INFO:teuthology.orchestra.run.smithi174.stdout: "op_reply": 262, 2024-04-21T23:29:08.943 INFO:teuthology.orchestra.run.smithi174.stdout: "op_latency": { 2024-04-21T23:29:08.944 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 262, 2024-04-21T23:29:08.944 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.727989881, 2024-04-21T23:29:08.944 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.002778587 2024-04-21T23:29:08.944 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:08.944 INFO:teuthology.orchestra.run.smithi174.stdout: "op_inflight": 0, 2024-04-21T23:29:08.944 INFO:teuthology.orchestra.run.smithi174.stdout: "oplen_avg": { 2024-04-21T23:29:08.944 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 262, 2024-04-21T23:29:08.944 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 300 2024-04-21T23:29:08.944 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:08.945 INFO:teuthology.orchestra.run.smithi174.stdout: "op": 262, 2024-04-21T23:29:08.945 INFO:teuthology.orchestra.run.smithi174.stdout: "op_r": 143, 2024-04-21T23:29:08.945 INFO:teuthology.orchestra.run.smithi174.stdout: "op_w": 119, 2024-04-21T23:29:08.945 INFO:teuthology.orchestra.run.smithi174.stdout: "op_rmw": 0, 2024-04-21T23:29:08.945 INFO:teuthology.orchestra.run.smithi174.stdout: "op_pg": 0, 2024-04-21T23:29:08.945 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_stat": 0, 2024-04-21T23:29:08.945 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_create": 10, 2024-04-21T23:29:08.945 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_read": 1, 2024-04-21T23:29:08.945 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_write": 48, 2024-04-21T23:29:08.946 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_writefull": 13, 2024-04-21T23:29:08.946 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_writesame": 0, 2024-04-21T23:29:08.946 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_append": 0, 2024-04-21T23:29:08.946 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_zero": 0, 2024-04-21T23:29:08.946 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_truncate": 0, 2024-04-21T23:29:08.946 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_delete": 20, 2024-04-21T23:29:08.946 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_mapext": 0, 2024-04-21T23:29:08.946 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_sparse_read": 0, 2024-04-21T23:29:08.947 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_clonerange": 0, 2024-04-21T23:29:08.947 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_getxattr": 142, 2024-04-21T23:29:08.947 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_setxattr": 20, 2024-04-21T23:29:08.947 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_cmpxattr": 0, 2024-04-21T23:29:08.947 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_rmxattr": 0, 2024-04-21T23:29:08.947 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_resetxattrs": 0, 2024-04-21T23:29:08.947 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_call": 0, 2024-04-21T23:29:08.947 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_watch": 0, 2024-04-21T23:29:08.947 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_notify": 0, 2024-04-21T23:29:08.948 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_src_cmpxattr": 0, 2024-04-21T23:29:08.948 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_pgls": 0, 2024-04-21T23:29:08.948 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_pgls_filter": 0, 2024-04-21T23:29:08.948 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_other": 0, 2024-04-21T23:29:08.948 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_active": 0, 2024-04-21T23:29:08.948 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_send": 0, 2024-04-21T23:29:08.948 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_resend": 0, 2024-04-21T23:29:08.948 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_ping": 0, 2024-04-21T23:29:08.948 INFO:teuthology.orchestra.run.smithi174.stdout: "poolop_active": 0, 2024-04-21T23:29:08.949 INFO:teuthology.orchestra.run.smithi174.stdout: "poolop_send": 0, 2024-04-21T23:29:08.949 INFO:teuthology.orchestra.run.smithi174.stdout: "poolop_resend": 0, 2024-04-21T23:29:08.949 INFO:teuthology.orchestra.run.smithi174.stdout: "poolstat_active": 0, 2024-04-21T23:29:08.949 INFO:teuthology.orchestra.run.smithi174.stdout: "poolstat_send": 0, 2024-04-21T23:29:08.949 INFO:teuthology.orchestra.run.smithi174.stdout: "poolstat_resend": 0, 2024-04-21T23:29:08.949 INFO:teuthology.orchestra.run.smithi174.stdout: "statfs_active": 0, 2024-04-21T23:29:08.949 INFO:teuthology.orchestra.run.smithi174.stdout: "statfs_send": 0, 2024-04-21T23:29:08.949 INFO:teuthology.orchestra.run.smithi174.stdout: "statfs_resend": 0, 2024-04-21T23:29:08.950 INFO:teuthology.orchestra.run.smithi174.stdout: "command_active": 0, 2024-04-21T23:29:08.950 INFO:teuthology.orchestra.run.smithi174.stdout: "command_send": 0, 2024-04-21T23:29:08.950 INFO:teuthology.orchestra.run.smithi174.stdout: "command_resend": 0, 2024-04-21T23:29:08.950 INFO:teuthology.orchestra.run.smithi174.stdout: "map_epoch": 103, 2024-04-21T23:29:08.950 INFO:teuthology.orchestra.run.smithi174.stdout: "map_full": 0, 2024-04-21T23:29:08.950 INFO:teuthology.orchestra.run.smithi174.stdout: "map_inc": 13, 2024-04-21T23:29:08.950 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_sessions": 12, 2024-04-21T23:29:08.950 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_session_open": 12, 2024-04-21T23:29:08.950 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_session_close": 0, 2024-04-21T23:29:08.951 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_laggy": 0, 2024-04-21T23:29:08.951 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_wr": 39, 2024-04-21T23:29:08.951 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_rd": 0, 2024-04-21T23:29:08.951 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_del": 7 2024-04-21T23:29:08.951 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:08.951 INFO:teuthology.orchestra.run.smithi174.stdout: "oft": { 2024-04-21T23:29:08.951 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_objs": 1, 2024-04-21T23:29:08.951 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_kv_pairs": 0, 2024-04-21T23:29:08.951 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_updates": 0, 2024-04-21T23:29:08.952 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_removes": 0 2024-04-21T23:29:08.952 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:08.952 INFO:teuthology.orchestra.run.smithi174.stdout: "osd-slow-ops": { 2024-04-21T23:29:08.952 INFO:teuthology.orchestra.run.smithi174.stdout: "slow_ops_count": 0 2024-04-21T23:29:08.952 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:08.952 INFO:teuthology.orchestra.run.smithi174.stdout: "purge_queue": { 2024-04-21T23:29:08.952 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing_ops": 0, 2024-04-21T23:29:08.952 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing_ops_high_water": 8, 2024-04-21T23:29:08.953 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing": 0, 2024-04-21T23:29:08.953 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing_high_water": 4, 2024-04-21T23:29:08.953 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executed_ops": 8, 2024-04-21T23:29:08.953 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executed": 4, 2024-04-21T23:29:08.953 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_item_in_journal": 0 2024-04-21T23:29:08.953 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:08.953 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-21T23:29:08.953 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-04-21T23:29:08.954 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 104857600, 2024-04-21T23:29:08.954 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-04-21T23:29:08.954 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 2303, 2024-04-21T23:29:08.954 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 2203314, 2024-04-21T23:29:08.954 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-04-21T23:29:08.954 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 2303, 2024-04-21T23:29:08.954 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 0, 2024-04-21T23:29:08.954 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 0, 2024-04-21T23:29:08.954 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 2303, 2024-04-21T23:29:08.955 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 2203314, 2024-04-21T23:29:08.955 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-04-21T23:29:08.955 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:08.955 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:08.955 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:08.955 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-04-21T23:29:08.955 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:08.955 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-objecter_bytes": { 2024-04-21T23:29:08.956 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-04-21T23:29:08.956 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 104857600, 2024-04-21T23:29:08.956 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-04-21T23:29:08.956 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 0, 2024-04-21T23:29:08.956 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 0, 2024-04-21T23:29:08.956 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-04-21T23:29:08.956 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 0, 2024-04-21T23:29:08.956 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 262, 2024-04-21T23:29:08.956 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 605145, 2024-04-21T23:29:08.957 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 262, 2024-04-21T23:29:08.957 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 605145, 2024-04-21T23:29:08.957 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-04-21T23:29:08.957 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:08.957 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:08.957 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:08.957 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-04-21T23:29:08.957 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:08.958 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-objecter_ops": { 2024-04-21T23:29:08.958 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-04-21T23:29:08.958 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 1024, 2024-04-21T23:29:08.958 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-04-21T23:29:08.958 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 0, 2024-04-21T23:29:08.958 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 0, 2024-04-21T23:29:08.958 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-04-21T23:29:08.958 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 0, 2024-04-21T23:29:08.959 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 262, 2024-04-21T23:29:08.959 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 262, 2024-04-21T23:29:08.959 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 262, 2024-04-21T23:29:08.959 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 262, 2024-04-21T23:29:08.959 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-04-21T23:29:08.959 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:08.959 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:08.959 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:08.959 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-04-21T23:29:08.960 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:08.960 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-write_buf_throttle": { 2024-04-21T23:29:08.960 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-04-21T23:29:08.960 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 3758096384, 2024-04-21T23:29:08.960 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-04-21T23:29:08.960 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 4, 2024-04-21T23:29:08.960 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 468, 2024-04-21T23:29:08.960 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-04-21T23:29:08.961 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 4, 2024-04-21T23:29:08.961 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 0, 2024-04-21T23:29:08.961 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 0, 2024-04-21T23:29:08.961 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 1, 2024-04-21T23:29:08.961 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 468, 2024-04-21T23:29:08.961 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-04-21T23:29:08.961 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:08.961 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:08.961 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:08.962 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-04-21T23:29:08.962 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:08.962 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-write_buf_throttle-0x557c3d050500": { 2024-04-21T23:29:08.962 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-04-21T23:29:08.962 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 3758096384, 2024-04-21T23:29:08.962 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-04-21T23:29:08.962 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 234, 2024-04-21T23:29:08.962 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 578206, 2024-04-21T23:29:08.963 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-04-21T23:29:08.963 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 234, 2024-04-21T23:29:08.963 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 0, 2024-04-21T23:29:08.963 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 0, 2024-04-21T23:29:08.963 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 47, 2024-04-21T23:29:08.963 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 578206, 2024-04-21T23:29:08.963 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-04-21T23:29:08.963 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:08.963 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:08.964 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:08.964 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-04-21T23:29:08.964 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-04-21T23:29:08.964 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-04-21T23:29:09.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:29:08 smithi040 ceph-mon[27984]: pgmap v333: 129 pgs: 129 active+clean; 95 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-04-21T23:29:09.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:29:08 smithi111 ceph-mon[31085]: pgmap v333: 129 pgs: 129 active+clean; 95 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-04-21T23:29:09.697 INFO:tasks.check_counter:Daemon mds.i mds.exported=0 2024-04-21T23:29:09.697 INFO:tasks.check_counter:Daemon mds.i mds.imported=9 2024-04-21T23:29:09.698 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=2 2024-04-21T23:29:09.698 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2024-04-21T23:29:09.698 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2024-04-21T23:29:09.698 INFO:tasks.check_counter:Daemon mds.i mds.dir_split=0 2024-04-21T23:29:09.698 DEBUG:tasks.check_counter:Getting stats from l 2024-04-21T23:29:09.698 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid d814e4dc-0033-11ef-bc93-c7b262605968 -- ceph daemon mds.l perf dump 2024-04-21T23:29:09.937 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/mon.c/config 2024-04-21T23:29:11.078 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:29:10 smithi174 ceph-mon[31409]: pgmap v334: 129 pgs: 129 active+clean; 95 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-21T23:29:11.181 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:29:10 smithi040 ceph-mon[27984]: pgmap v334: 129 pgs: 129 active+clean; 95 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-21T23:29:11.193 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:29:10 smithi111 ceph-mon[31085]: pgmap v334: 129 pgs: 129 active+clean; 95 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-21T23:29:11.631 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-04-21T23:29:11.632 INFO:teuthology.orchestra.run.smithi174.stdout: "AsyncMessenger::Worker-0": { 2024-04-21T23:29:11.632 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_messages": 235, 2024-04-21T23:29:11.632 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages": 224, 2024-04-21T23:29:11.632 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_bytes": 1584528, 2024-04-21T23:29:11.632 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_bytes": 137664, 2024-04-21T23:29:11.632 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_created_connections": 4, 2024-04-21T23:29:11.632 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_active_connections": 0, 2024-04-21T23:29:11.633 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_total_time": 0.049597026, 2024-04-21T23:29:11.633 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_send_time": 0.014618455, 2024-04-21T23:29:11.633 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_recv_time": 0.036997131, 2024-04-21T23:29:11.633 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_fast_dispatch_time": 0.005807913, 2024-04-21T23:29:11.633 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T23:29:11.633 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 223, 2024-04-21T23:29:11.633 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.013579834, 2024-04-21T23:29:11.633 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000060896 2024-04-21T23:29:11.633 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:11.634 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_handle_ack_lat": { 2024-04-21T23:29:11.634 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:11.634 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:11.634 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:11.634 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:11.634 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_encrypted_bytes": 1584528, 2024-04-21T23:29:11.634 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_encrypted_bytes": 137664 2024-04-21T23:29:11.634 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:11.635 INFO:teuthology.orchestra.run.smithi174.stdout: "AsyncMessenger::Worker-1": { 2024-04-21T23:29:11.635 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_messages": 2, 2024-04-21T23:29:11.635 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages": 171, 2024-04-21T23:29:11.635 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_bytes": 320, 2024-04-21T23:29:11.635 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_bytes": 185504, 2024-04-21T23:29:11.635 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_created_connections": 5, 2024-04-21T23:29:11.635 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_active_connections": 3, 2024-04-21T23:29:11.635 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_total_time": 0.011702236, 2024-04-21T23:29:11.636 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_send_time": 0.007973172, 2024-04-21T23:29:11.636 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_recv_time": 0.001385632, 2024-04-21T23:29:11.636 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T23:29:11.636 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T23:29:11.636 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 168, 2024-04-21T23:29:11.636 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.016981975, 2024-04-21T23:29:11.637 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000101083 2024-04-21T23:29:11.637 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:11.637 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_handle_ack_lat": { 2024-04-21T23:29:11.637 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:11.637 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:11.637 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:11.637 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:11.637 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-21T23:29:11.637 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_encrypted_bytes": 185504 2024-04-21T23:29:11.638 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:11.638 INFO:teuthology.orchestra.run.smithi174.stdout: "AsyncMessenger::Worker-2": { 2024-04-21T23:29:11.638 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_messages": 4, 2024-04-21T23:29:11.638 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages": 4, 2024-04-21T23:29:11.638 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_bytes": 3024, 2024-04-21T23:29:11.638 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_bytes": 448, 2024-04-21T23:29:11.638 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_created_connections": 2, 2024-04-21T23:29:11.638 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_active_connections": 1, 2024-04-21T23:29:11.639 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_total_time": 0.000565197, 2024-04-21T23:29:11.639 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_send_time": 0.000026810, 2024-04-21T23:29:11.639 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_recv_time": 0.000502712, 2024-04-21T23:29:11.639 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T23:29:11.639 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T23:29:11.639 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 3, 2024-04-21T23:29:11.639 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000731222, 2024-04-21T23:29:11.639 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000243740 2024-04-21T23:29:11.639 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:11.640 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_handle_ack_lat": { 2024-04-21T23:29:11.640 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:11.640 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:11.640 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:11.640 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:11.640 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-04-21T23:29:11.640 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_encrypted_bytes": 448 2024-04-21T23:29:11.640 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:11.640 INFO:teuthology.orchestra.run.smithi174.stdout: "cct": { 2024-04-21T23:29:11.641 INFO:teuthology.orchestra.run.smithi174.stdout: "total_workers": 0, 2024-04-21T23:29:11.641 INFO:teuthology.orchestra.run.smithi174.stdout: "unhealthy_workers": 0 2024-04-21T23:29:11.641 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:11.641 INFO:teuthology.orchestra.run.smithi174.stdout: "mempool": { 2024-04-21T23:29:11.641 INFO:teuthology.orchestra.run.smithi174.stdout: "bloom_filter_bytes": 0, 2024-04-21T23:29:11.641 INFO:teuthology.orchestra.run.smithi174.stdout: "bloom_filter_items": 0, 2024-04-21T23:29:11.641 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_alloc_bytes": 0, 2024-04-21T23:29:11.641 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_alloc_items": 0, 2024-04-21T23:29:11.641 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_data_bytes": 0, 2024-04-21T23:29:11.642 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_data_items": 0, 2024-04-21T23:29:11.642 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-21T23:29:11.642 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_onode_items": 0, 2024-04-21T23:29:11.642 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-21T23:29:11.642 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_meta_items": 0, 2024-04-21T23:29:11.642 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_other_bytes": 0, 2024-04-21T23:29:11.642 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_other_items": 0, 2024-04-21T23:29:11.642 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-21T23:29:11.643 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_buffer_items": 0, 2024-04-21T23:29:11.643 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_extent_bytes": 0, 2024-04-21T23:29:11.643 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_extent_items": 0, 2024-04-21T23:29:11.643 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_blob_bytes": 0, 2024-04-21T23:29:11.643 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_blob_items": 0, 2024-04-21T23:29:11.643 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-21T23:29:11.643 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_shared_blob_items": 0, 2024-04-21T23:29:11.643 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-21T23:29:11.644 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_inline_bl_items": 0, 2024-04-21T23:29:11.644 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_fsck_bytes": 0, 2024-04-21T23:29:11.644 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_fsck_items": 0, 2024-04-21T23:29:11.644 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_txc_bytes": 0, 2024-04-21T23:29:11.644 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_txc_items": 0, 2024-04-21T23:29:11.644 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-21T23:29:11.644 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_deferred_items": 0, 2024-04-21T23:29:11.644 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_bytes": 0, 2024-04-21T23:29:11.644 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_items": 0, 2024-04-21T23:29:11.645 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_bytes": 0, 2024-04-21T23:29:11.645 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_items": 0, 2024-04-21T23:29:11.645 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_reader_bytes": 0, 2024-04-21T23:29:11.645 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_reader_items": 0, 2024-04-21T23:29:11.645 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_writer_bytes": 0, 2024-04-21T23:29:11.645 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_writer_items": 0, 2024-04-21T23:29:11.645 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_anon_bytes": 148721, 2024-04-21T23:29:11.645 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_anon_items": 30, 2024-04-21T23:29:11.645 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_meta_bytes": 0, 2024-04-21T23:29:11.646 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_meta_items": 0, 2024-04-21T23:29:11.646 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_bytes": 0, 2024-04-21T23:29:11.646 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_items": 0, 2024-04-21T23:29:11.646 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_mapbl_bytes": 0, 2024-04-21T23:29:11.646 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_mapbl_items": 0, 2024-04-21T23:29:11.646 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_pglog_bytes": 0, 2024-04-21T23:29:11.646 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_pglog_items": 0, 2024-04-21T23:29:11.646 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_bytes": 0, 2024-04-21T23:29:11.646 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_items": 0, 2024-04-21T23:29:11.647 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_mapping_bytes": 0, 2024-04-21T23:29:11.647 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_mapping_items": 0, 2024-04-21T23:29:11.647 INFO:teuthology.orchestra.run.smithi174.stdout: "pgmap_bytes": 0, 2024-04-21T23:29:11.647 INFO:teuthology.orchestra.run.smithi174.stdout: "pgmap_items": 0, 2024-04-21T23:29:11.647 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_co_bytes": 928, 2024-04-21T23:29:11.647 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_co_items": 2, 2024-04-21T23:29:11.647 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_1_bytes": 0, 2024-04-21T23:29:11.647 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_1_items": 0, 2024-04-21T23:29:11.647 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_2_bytes": 0, 2024-04-21T23:29:11.648 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_2_items": 0 2024-04-21T23:29:11.648 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-21T23:29:11.648 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-21T23:29:11.648 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-04-21T23:29:11.648 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 104857600, 2024-04-21T23:29:11.648 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-04-21T23:29:11.648 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 237, 2024-04-21T23:29:11.648 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 1551410, 2024-04-21T23:29:11.648 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-04-21T23:29:11.649 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 237, 2024-04-21T23:29:11.649 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 0, 2024-04-21T23:29:11.649 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 0, 2024-04-21T23:29:11.649 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 237, 2024-04-21T23:29:11.649 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 1551410, 2024-04-21T23:29:11.649 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-04-21T23:29:11.649 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-04-21T23:29:11.649 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-04-21T23:29:11.650 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-04-21T23:29:11.650 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-04-21T23:29:11.650 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-04-21T23:29:11.650 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-04-21T23:29:11.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:29:11 smithi174 ceph-mon[31409]: osdmap e104: 12 total, 12 up, 12 in 2024-04-21T23:29:12.095 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:29:11 smithi111 ceph-mon[31085]: osdmap e104: 12 total, 12 up, 12 in 2024-04-21T23:29:12.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:29:11 smithi040 ceph-mon[27984]: osdmap e104: 12 total, 12 up, 12 in 2024-04-21T23:29:12.269 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.l 2024-04-21T23:29:12.269 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.l 2024-04-21T23:29:12.269 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.l 2024-04-21T23:29:12.269 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.l 2024-04-21T23:29:12.270 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.l 2024-04-21T23:29:12.270 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.l 2024-04-21T23:29:12.270 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-04-21T23:29:12.281 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-21T23:29:12.282 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-21T23:29:12.282 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-21T23:29:12.303 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-21T23:29:12.303 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-21T23:29:12.363 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T23:29:12.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.520+0000 7fb5a899a640 1 -- 172.21.15.40:0/2785205202 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5a4102a20 msgr2=0x7fb5a4102e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:29:12.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.520+0000 7fb5a899a640 1 --2- 172.21.15.40:0/2785205202 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5a4102a20 0x7fb5a4102e00 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fb594009900 tx=0x7fb59402bcd0 comp rx=0 tx=0).stop 2024-04-21T23:29:12.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.520+0000 7fb5a899a640 1 -- 172.21.15.40:0/2785205202 shutdown_connections 2024-04-21T23:29:12.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.521+0000 7fb5a899a640 1 --2- 172.21.15.40:0/2785205202 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5a4103340 0x7fb5a410b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:12.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.521+0000 7fb5a899a640 1 --2- 172.21.15.40:0/2785205202 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb5a410bd60 0x7fb5a410e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:12.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.521+0000 7fb5a899a640 1 --2- 172.21.15.40:0/2785205202 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5a4102a20 0x7fb5a4102e00 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:12.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.521+0000 7fb5a899a640 1 -- 172.21.15.40:0/2785205202 >> 172.21.15.40:0/2785205202 conn(0x7fb5a40777a0 msgr2=0x7fb5a4077bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:29:12.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.521+0000 7fb5a899a640 1 -- 172.21.15.40:0/2785205202 shutdown_connections 2024-04-21T23:29:12.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.521+0000 7fb5a899a640 1 -- 172.21.15.40:0/2785205202 wait complete. 2024-04-21T23:29:12.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.521+0000 7fb5a899a640 1 Processor -- start 2024-04-21T23:29:12.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.521+0000 7fb5a899a640 1 -- start start 2024-04-21T23:29:12.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.522+0000 7fb5a899a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb5a4102a20 0x7fb5a4197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:29:12.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.522+0000 7fb5a899a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5a4103340 0x7fb5a4198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:29:12.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.522+0000 7fb5a899a640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5a410bd60 0x7fb5a419c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:29:12.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.522+0000 7fb5a899a640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb5a4111b30 con 0x7fb5a4103340 2024-04-21T23:29:12.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.522+0000 7fb5a899a640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb5a41119b0 con 0x7fb5a410bd60 2024-04-21T23:29:12.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.522+0000 7fb5a899a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb5a4111cb0 con 0x7fb5a4102a20 2024-04-21T23:29:12.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.522+0000 7fb59bfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5a4103340 0x7fb5a4198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:29:12.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.522+0000 7fb59bfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5a4103340 0x7fb5a4198510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52058/0 (socket says 172.21.15.40:52058) 2024-04-21T23:29:12.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.522+0000 7fb59bfff640 1 -- 172.21.15.40:0/1031735729 learned_addr learned my addr 172.21.15.40:0/1031735729 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:29:12.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.522+0000 7fb5a2d9d640 1 --2- 172.21.15.40:0/1031735729 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5a410bd60 0x7fb5a419c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:29:12.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.522+0000 7fb5a259c640 1 --2- 172.21.15.40:0/1031735729 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb5a4102a20 0x7fb5a4197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:29:12.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.522+0000 7fb59bfff640 1 -- 172.21.15.40:0/1031735729 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb5a4102a20 msgr2=0x7fb5a4197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:29:12.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.522+0000 7fb59bfff640 1 --2- 172.21.15.40:0/1031735729 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb5a4102a20 0x7fb5a4197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:12.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.522+0000 7fb59bfff640 1 -- 172.21.15.40:0/1031735729 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5a410bd60 msgr2=0x7fb5a419c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:29:12.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.522+0000 7fb59bfff640 1 --2- 172.21.15.40:0/1031735729 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5a410bd60 0x7fb5a419c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:12.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.522+0000 7fb59bfff640 1 -- 172.21.15.40:0/1031735729 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5a419cfc0 con 0x7fb5a4103340 2024-04-21T23:29:12.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.522+0000 7fb59bfff640 1 --2- 172.21.15.40:0/1031735729 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5a4103340 0x7fb5a4198510 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fb58c00e9f0 tx=0x7fb58c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:29:12.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.522+0000 7fb59b7fe640 1 -- 172.21.15.40:0/1031735729 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb58c00cd80 con 0x7fb5a4103340 2024-04-21T23:29:12.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.522+0000 7fb59b7fe640 1 -- 172.21.15.40:0/1031735729 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb58c010910 con 0x7fb5a4103340 2024-04-21T23:29:12.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.522+0000 7fb5a899a640 1 -- 172.21.15.40:0/1031735729 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5a419d2b0 con 0x7fb5a4103340 2024-04-21T23:29:12.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.523+0000 7fb59b7fe640 1 -- 172.21.15.40:0/1031735729 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb58c018990 con 0x7fb5a4103340 2024-04-21T23:29:12.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.523+0000 7fb5a899a640 1 -- 172.21.15.40:0/1031735729 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5a41a4b40 con 0x7fb5a4103340 2024-04-21T23:29:12.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.524+0000 7fb5a899a640 1 -- 172.21.15.40:0/1031735729 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5a40689b0 con 0x7fb5a4103340 2024-04-21T23:29:12.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.525+0000 7fb59b7fe640 1 -- 172.21.15.40:0/1031735729 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fb58c010ac0 con 0x7fb5a4103340 2024-04-21T23:29:12.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.526+0000 7fb59b7fe640 1 --2- 172.21.15.40:0/1031735729 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fb5900a2d00 0x7fb5900a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:29:12.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.526+0000 7fb5a259c640 1 --2- 172.21.15.40:0/1031735729 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fb5900a2d00 0x7fb5900a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:29:12.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.526+0000 7fb59b7fe640 1 -- 172.21.15.40:0/1031735729 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9642+0+0 (secure 0 0 0) 0x7fb58c0cfc50 con 0x7fb5a4103340 2024-04-21T23:29:12.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.526+0000 7fb5a259c640 1 --2- 172.21.15.40:0/1031735729 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fb5900a2d00 0x7fb5900a51c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb594000c00 tx=0x7fb59402c710 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:29:12.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.530+0000 7fb59b7fe640 1 -- 172.21.15.40:0/1031735729 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fb58c0d5020 con 0x7fb5a4103340 2024-04-21T23:29:12.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.715+0000 7fb5a899a640 1 -- 172.21.15.40:0/1031735729 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb5a41a4de0 con 0x7fb5a4103340 2024-04-21T23:29:12.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.716+0000 7fb59b7fe640 1 -- 172.21.15.40:0/1031735729 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v104) ==== 74+0+25552 (secure 0 0 0) 0x7fb58c09c410 con 0x7fb5a4103340 2024-04-21T23:29:12.716 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:29:12.716 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":104,"fsid":"d814e4dc-0033-11ef-bc93-c7b262605968","created":"2024-04-21T23:07:52.659207+0000","modified":"2024-04-21T23:29:10.833904+0000","last_up_change":"2024-04-21T23:15:05.308821+0000","last_in_change":"2024-04-21T23:14:45.447491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:11:14.824775+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T23:15:54.322571+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T23:15:55.164493+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"104","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":104,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T23:15:58.838266+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"104","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":104,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"0e8d692a-22fe-4a87-8d99-9151f96cbb02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6803","nonce":2009758784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6805","nonce":2009758784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6809","nonce":2009758784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6807","nonce":2009758784}]},"public_addr":"172.21.15.40:6803/2009758784","cluster_addr":"172.21.15.40:6805/2009758784","heartbeat_back_addr":"172.21.15.40:6809/2009758784","heartbeat_front_addr":"172.21.15.40:6807/2009758784","state":["exists","up"]},{"osd":1,"uuid":"fe5fe11b-0e84-4359-83a7-1ba45105755a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6801","nonce":1763903836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6803","nonce":1763903836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6807","nonce":1763903836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6805","nonce":1763903836}]},"public_addr":"172.21.15.111:6801/1763903836","cluster_addr":"172.21.15.111:6803/1763903836","heartbeat_back_addr":"172.21.15.111:6807/1763903836","heartbeat_front_addr":"172.21.15.111:6805/1763903836","state":["exists","up"]},{"osd":2,"uuid":"813ad735-5cc2-4da0-ae16-6db238395fbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6801","nonce":439695825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6803","nonce":439695825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6807","nonce":439695825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6805","nonce":439695825}]},"public_addr":"172.21.15.174:6801/439695825","cluster_addr":"172.21.15.174:6803/439695825","heartbeat_back_addr":"172.21.15.174:6807/439695825","heartbeat_front_addr":"172.21.15.174:6805/439695825","state":["exists","up"]},{"osd":3,"uuid":"09becb03-b7ce-4ffc-917d-b7900ec37e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6811","nonce":3454312436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6813","nonce":3454312436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6817","nonce":3454312436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6815","nonce":3454312436}]},"public_addr":"172.21.15.40:6811/3454312436","cluster_addr":"172.21.15.40:6813/3454312436","heartbeat_back_addr":"172.21.15.40:6817/3454312436","heartbeat_front_addr":"172.21.15.40:6815/3454312436","state":["exists","up"]},{"osd":4,"uuid":"46eabfd7-d88a-4efd-94eb-04354d3a9d7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6809","nonce":1810866070}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6811","nonce":1810866070}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6815","nonce":1810866070}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6813","nonce":1810866070}]},"public_addr":"172.21.15.111:6809/1810866070","cluster_addr":"172.21.15.111:6811/1810866070","heartbeat_back_addr":"172.21.15.111:6815/1810866070","heartbeat_front_addr":"172.21.15.111:6813/1810866070","state":["exists","up"]},{"osd":5,"uuid":"6f73b582-6c65-41db-aab0-9b360f83d5c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6809","nonce":3752986711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6811","nonce":3752986711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6815","nonce":3752986711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6813","nonce":3752986711}]},"public_addr":"172.21.15.174:6809/3752986711","cluster_addr":"172.21.15.174:6811/3752986711","heartbeat_back_addr":"172.21.15.174:6815/3752986711","heartbeat_front_addr":"172.21.15.174:6813/3752986711","state":["exists","up"]},{"osd":6,"uuid":"39f305f0-12d2-4732-aaa0-cd66700af538","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6819","nonce":1618855427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6821","nonce":1618855427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6825","nonce":1618855427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6823","nonce":1618855427}]},"public_addr":"172.21.15.40:6819/1618855427","cluster_addr":"172.21.15.40:6821/1618855427","heartbeat_back_addr":"172.21.15.40:6825/1618855427","heartbeat_front_addr":"172.21.15.40:6823/1618855427","state":["exists","up"]},{"osd":7,"uuid":"769f2468-5102-490b-89f8-d0444d3169a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6817","nonce":927054162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6819","nonce":927054162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6823","nonce":927054162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6821","nonce":927054162}]},"public_addr":"172.21.15.111:6817/927054162","cluster_addr":"172.21.15.111:6819/927054162","heartbeat_back_addr":"172.21.15.111:6823/927054162","heartbeat_front_addr":"172.21.15.111:6821/927054162","state":["exists","up"]},{"osd":8,"uuid":"1a444e80-007d-473b-b972-4c814a813c22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6817","nonce":3817848746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6819","nonce":3817848746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6823","nonce":3817848746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6821","nonce":3817848746}]},"public_addr":"172.21.15.174:6817/3817848746","cluster_addr":"172.21.15.174:6819/3817848746","heartbeat_back_addr":"172.21.15.174:6823/3817848746","heartbeat_front_addr":"172.21.15.174:6821/3817848746","state":["exists","up"]},{"osd":9,"uuid":"9a6b6a56-6ae5-49fe-9e5b-cb17d686e0dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6827","nonce":1649446531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6829","nonce":1649446531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6833","nonce":1649446531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6831","nonce":1649446531}]},"public_addr":"172.21.15.40:6827/1649446531","cluster_addr":"172.21.15.40:6829/1649446531","heartbeat_back_addr":"172.21.15.40:6833/1649446531","heartbeat_front_addr":"172.21.15.40:6831/1649446531","state":["exists","up"]},{"osd":10,"uuid":"a70038ba-e2ad-4170-85cc-ea0470a2edef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6825","nonce":1227538935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6827","nonce":1227538935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6831","nonce":1227538935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6829","nonce":1227538935}]},"public_addr":"172.21.15.111:6825/1227538935","cluster_addr":"172.21.15.111:6827/1227538935","heartbeat_back_addr":"172.21.15.111:6831/1227538935","heartbeat_front_addr":"172.21.15.111:6829/1227538935","state":["exists","up"]},{"osd":11,"uuid":"781532bb-7a5e-44e3-8eb7-b584b0054666","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6825","nonce":2045865553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6827","nonce":2045865553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6831","nonce":2045865553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6829","nonce":2045865553}]},"public_addr":"172.21.15.174:6825/2045865553","cluster_addr":"172.21.15.174:6827/2045865553","heartbeat_back_addr":"172.21.15.174:6831/2045865553","heartbeat_front_addr":"172.21.15.174:6829/2045865553","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:10:24.761290+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:10:48.565428+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:11:12.307818+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:11:37.951529+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:02.832523+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:27.568879+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:52.997707+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:13:18.937274+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:13:44.708657+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:14:10.833855+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:14:37.078272+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:15:03.878076+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":1}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.2a","mappings":[{"from":3,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"4.9","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":10}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:6801/2500760788":"2024-04-22T23:18:23.389674+0000","172.21.15.40:6800/2500760788":"2024-04-22T23:18:23.389674+0000","172.21.15.40:0/2742002992":"2024-04-22T23:18:23.389674+0000","172.21.15.40:0/2494672676":"2024-04-22T23:18:23.389674+0000","172.21.15.40:6801/526554319":"2024-04-22T23:08:50.305770+0000","172.21.15.40:6800/526554319":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/3178763471":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/4045911364":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/276300080":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/3689649814":"2024-04-22T23:08:23.743030+0000","172.21.15.40:0/2241116715":"2024-04-22T23:08:23.743030+0000","172.21.15.40:0/1406242364":"2024-04-22T23:18:23.389674+0000","172.21.15.40:0/1737227818":"2024-04-22T23:18:23.389674+0000","172.21.15.40:6801/1479040326":"2024-04-22T23:08:23.743030+0000","172.21.15.40:6800/1479040326":"2024-04-22T23:08:23.743030+0000","172.21.15.40:0/681298099":"2024-04-22T23:08:23.743030+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]},{"pool":4,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"new_removed_snaps":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]},{"pool":4,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T23:29:12.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.718+0000 7fb5a899a640 1 -- 172.21.15.40:0/1031735729 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fb5900a2d00 msgr2=0x7fb5900a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:29:12.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.718+0000 7fb5a899a640 1 --2- 172.21.15.40:0/1031735729 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fb5900a2d00 0x7fb5900a51c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb594000c00 tx=0x7fb59402c710 comp rx=0 tx=0).stop 2024-04-21T23:29:12.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.718+0000 7fb5a899a640 1 -- 172.21.15.40:0/1031735729 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5a4103340 msgr2=0x7fb5a4198510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:29:12.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.718+0000 7fb5a899a640 1 --2- 172.21.15.40:0/1031735729 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5a4103340 0x7fb5a4198510 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fb58c00e9f0 tx=0x7fb58c00eeb0 comp rx=0 tx=0).stop 2024-04-21T23:29:12.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.718+0000 7fb5a899a640 1 -- 172.21.15.40:0/1031735729 shutdown_connections 2024-04-21T23:29:12.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.718+0000 7fb5a899a640 1 --2- 172.21.15.40:0/1031735729 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fb5900a2d00 0x7fb5900a51c0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:12.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.718+0000 7fb5a899a640 1 --2- 172.21.15.40:0/1031735729 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5a410bd60 0x7fb5a419c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:12.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.718+0000 7fb5a899a640 1 --2- 172.21.15.40:0/1031735729 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5a4103340 0x7fb5a4198510 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:12.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.718+0000 7fb5a899a640 1 --2- 172.21.15.40:0/1031735729 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb5a4102a20 0x7fb5a4197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:12.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.718+0000 7fb5a899a640 1 -- 172.21.15.40:0/1031735729 >> 172.21.15.40:0/1031735729 conn(0x7fb5a40777a0 msgr2=0x7fb5a40fd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:29:12.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.719+0000 7fb5a899a640 1 -- 172.21.15.40:0/1031735729 shutdown_connections 2024-04-21T23:29:12.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.719+0000 7fb5a899a640 1 -- 172.21.15.40:0/1031735729 wait complete. 2024-04-21T23:29:12.729 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T23:11:14.824775+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T23:15:54.322571+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T23:15:55.164493+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '104', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 17, 'snap_epoch': 104, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T23:15:58.838266+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '104', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 17, 'snap_epoch': 104, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T23:29:12.729 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T23:29:12.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.835+0000 7f3cd49cb640 1 -- 172.21.15.40:0/2989981919 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3cd0069dc0 msgr2=0x7f3cd006a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:29:12.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.835+0000 7f3cd49cb640 1 --2- 172.21.15.40:0/2989981919 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3cd0069dc0 0x7f3cd006a240 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f3cc0009930 tx=0x7f3cc002bc30 comp rx=0 tx=0).stop 2024-04-21T23:29:12.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.835+0000 7f3cd49cb640 1 -- 172.21.15.40:0/2989981919 shutdown_connections 2024-04-21T23:29:12.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.835+0000 7f3cd49cb640 1 --2- 172.21.15.40:0/2989981919 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3cd006a780 0x7f3cd010e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:12.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.835+0000 7f3cd49cb640 1 --2- 172.21.15.40:0/2989981919 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3cd0069dc0 0x7f3cd006a240 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:12.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.835+0000 7f3cd49cb640 1 --2- 172.21.15.40:0/2989981919 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3cd0107730 0x7f3cd0107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:12.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.835+0000 7f3cd49cb640 1 -- 172.21.15.40:0/2989981919 >> 172.21.15.40:0/2989981919 conn(0x7f3cd00776a0 msgr2=0x7f3cd0077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:29:12.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.835+0000 7f3cd49cb640 1 -- 172.21.15.40:0/2989981919 shutdown_connections 2024-04-21T23:29:12.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.835+0000 7f3cd49cb640 1 -- 172.21.15.40:0/2989981919 wait complete. 2024-04-21T23:29:12.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.836+0000 7f3cd49cb640 1 Processor -- start 2024-04-21T23:29:12.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.836+0000 7f3cd49cb640 1 -- start start 2024-04-21T23:29:12.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.836+0000 7f3cd49cb640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3cd0069dc0 0x7f3cd01980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:29:12.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.836+0000 7f3cd49cb640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3cd006a780 0x7f3cd0198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:29:12.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.836+0000 7f3cd49cb640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3cd0107730 0x7f3cd019c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:29:12.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.836+0000 7f3cd49cb640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3cd0111b40 con 0x7f3cd006a780 2024-04-21T23:29:12.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.836+0000 7f3cd49cb640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3cd01119c0 con 0x7f3cd0069dc0 2024-04-21T23:29:12.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.836+0000 7f3cd49cb640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3cd0111cc0 con 0x7f3cd0107730 2024-04-21T23:29:12.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.836+0000 7f3ccdd9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3cd006a780 0x7f3cd0198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:29:12.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.836+0000 7f3ccdd9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3cd006a780 0x7f3cd0198620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52076/0 (socket says 172.21.15.40:52076) 2024-04-21T23:29:12.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.837+0000 7f3ccdd9b640 1 -- 172.21.15.40:0/2091824067 learned_addr learned my addr 172.21.15.40:0/2091824067 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:29:12.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.837+0000 7f3cce59c640 1 --2- 172.21.15.40:0/2091824067 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3cd0069dc0 0x7f3cd01980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:29:12.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.837+0000 7f3cced9d640 1 --2- 172.21.15.40:0/2091824067 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3cd0107730 0x7f3cd019c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:29:12.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.837+0000 7f3ccdd9b640 1 -- 172.21.15.40:0/2091824067 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3cd0107730 msgr2=0x7f3cd019c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:29:12.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.837+0000 7f3ccdd9b640 1 --2- 172.21.15.40:0/2091824067 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3cd0107730 0x7f3cd019c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:12.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.837+0000 7f3ccdd9b640 1 -- 172.21.15.40:0/2091824067 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3cd0069dc0 msgr2=0x7f3cd01980e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:29:12.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.837+0000 7f3ccdd9b640 1 --2- 172.21.15.40:0/2091824067 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3cd0069dc0 0x7f3cd01980e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:12.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.837+0000 7f3ccdd9b640 1 -- 172.21.15.40:0/2091824067 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cd019d080 con 0x7f3cd006a780 2024-04-21T23:29:12.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.837+0000 7f3cced9d640 1 --2- 172.21.15.40:0/2091824067 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3cd0107730 0x7f3cd019c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:29:12.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.837+0000 7f3ccdd9b640 1 --2- 172.21.15.40:0/2091824067 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3cd006a780 0x7f3cd0198620 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f3cc0009230 tx=0x7f3cc002cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:29:12.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.837+0000 7f3cc77fe640 1 -- 172.21.15.40:0/2091824067 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cc0041070 con 0x7f3cd006a780 2024-04-21T23:29:12.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.837+0000 7f3cc77fe640 1 -- 172.21.15.40:0/2091824067 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3cc0004450 con 0x7f3cd006a780 2024-04-21T23:29:12.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.837+0000 7f3cc77fe640 1 -- 172.21.15.40:0/2091824067 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cc0003e50 con 0x7f3cd006a780 2024-04-21T23:29:12.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.837+0000 7f3cd49cb640 1 -- 172.21.15.40:0/2091824067 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cd019d310 con 0x7f3cd006a780 2024-04-21T23:29:12.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.837+0000 7f3cd49cb640 1 -- 172.21.15.40:0/2091824067 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cd01a4c00 con 0x7f3cd006a780 2024-04-21T23:29:12.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.838+0000 7f3cd49cb640 1 -- 172.21.15.40:0/2091824067 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3cd00689b0 con 0x7f3cd006a780 2024-04-21T23:29:12.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.839+0000 7f3cc77fe640 1 -- 172.21.15.40:0/2091824067 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f3cc0044070 con 0x7f3cd006a780 2024-04-21T23:29:12.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.840+0000 7f3cc77fe640 1 --2- 172.21.15.40:0/2091824067 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f3cbc0a2f20 0x7f3cbc0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:29:12.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.840+0000 7f3cc77fe640 1 -- 172.21.15.40:0/2091824067 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9642+0+0 (secure 0 0 0) 0x7f3cc00f1490 con 0x7f3cd006a780 2024-04-21T23:29:12.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.840+0000 7f3cce59c640 1 --2- 172.21.15.40:0/2091824067 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f3cbc0a2f20 0x7f3cbc0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:29:12.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.841+0000 7f3cce59c640 1 --2- 172.21.15.40:0/2091824067 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f3cbc0a2f20 0x7f3cbc0a53e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3cb8009b50 tx=0x7f3cb8005f90 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:29:12.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:12.844+0000 7f3cc77fe640 1 -- 172.21.15.40:0/2091824067 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f3cc00bde20 con 0x7f3cd006a780 2024-04-21T23:29:13.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.028+0000 7f3cd49cb640 1 -- 172.21.15.40:0/2091824067 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3cd0109ce0 con 0x7f3cd006a780 2024-04-21T23:29:13.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.028+0000 7f3cc77fe640 1 -- 172.21.15.40:0/2091824067 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v104) ==== 93+0+10 (secure 0 0 0) 0x7f3cc00bd720 con 0x7f3cd006a780 2024-04-21T23:29:13.027 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-04-21T23:29:13.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.030+0000 7f3cd49cb640 1 -- 172.21.15.40:0/2091824067 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f3cbc0a2f20 msgr2=0x7f3cbc0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:29:13.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.030+0000 7f3cd49cb640 1 --2- 172.21.15.40:0/2091824067 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f3cbc0a2f20 0x7f3cbc0a53e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3cb8009b50 tx=0x7f3cb8005f90 comp rx=0 tx=0).stop 2024-04-21T23:29:13.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.030+0000 7f3cd49cb640 1 -- 172.21.15.40:0/2091824067 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3cd006a780 msgr2=0x7f3cd0198620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:29:13.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.030+0000 7f3cd49cb640 1 --2- 172.21.15.40:0/2091824067 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3cd006a780 0x7f3cd0198620 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f3cc0009230 tx=0x7f3cc002cb30 comp rx=0 tx=0).stop 2024-04-21T23:29:13.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.030+0000 7f3cd49cb640 1 -- 172.21.15.40:0/2091824067 shutdown_connections 2024-04-21T23:29:13.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.030+0000 7f3cd49cb640 1 --2- 172.21.15.40:0/2091824067 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f3cbc0a2f20 0x7f3cbc0a53e0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:13.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.030+0000 7f3cd49cb640 1 --2- 172.21.15.40:0/2091824067 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3cd0107730 0x7f3cd019c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:13.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.030+0000 7f3cd49cb640 1 --2- 172.21.15.40:0/2091824067 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3cd006a780 0x7f3cd0198620 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:13.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.030+0000 7f3cd49cb640 1 --2- 172.21.15.40:0/2091824067 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3cd0069dc0 0x7f3cd01980e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:13.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.030+0000 7f3cd49cb640 1 -- 172.21.15.40:0/2091824067 >> 172.21.15.40:0/2091824067 conn(0x7f3cd00776a0 msgr2=0x7f3cd00fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:29:13.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.030+0000 7f3cd49cb640 1 -- 172.21.15.40:0/2091824067 shutdown_connections 2024-04-21T23:29:13.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.030+0000 7f3cd49cb640 1 -- 172.21.15.40:0/2091824067 wait complete. 2024-04-21T23:29:13.039 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:29:12 smithi040 ceph-mon[27984]: pgmap v336: 129 pgs: 129 active+clean; 95 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-21T23:29:13.039 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:29:12 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/1031735729' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:29:13.040 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T23:29:13.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.147+0000 7fb7a06ec640 1 -- 172.21.15.40:0/1050803162 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb798101330 msgr2=0x7fb79810d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:29:13.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.147+0000 7fb7a06ec640 1 --2- 172.21.15.40:0/1050803162 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb798101330 0x7fb79810d460 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fb790009ca0 tx=0x7fb79002c0f0 comp rx=0 tx=0).stop 2024-04-21T23:29:13.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.147+0000 7fb7a06ec640 1 -- 172.21.15.40:0/1050803162 shutdown_connections 2024-04-21T23:29:13.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.147+0000 7fb7a06ec640 1 --2- 172.21.15.40:0/1050803162 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb798100990 0x7fb798100df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:13.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.147+0000 7fb7a06ec640 1 --2- 172.21.15.40:0/1050803162 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb798101330 0x7fb79810d460 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:13.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.147+0000 7fb7a06ec640 1 --2- 172.21.15.40:0/1050803162 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb798106690 0x7fb798106a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:13.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.147+0000 7fb7a06ec640 1 -- 172.21.15.40:0/1050803162 >> 172.21.15.40:0/1050803162 conn(0x7fb798076440 msgr2=0x7fb798076850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:29:13.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.147+0000 7fb7a06ec640 1 -- 172.21.15.40:0/1050803162 shutdown_connections 2024-04-21T23:29:13.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.147+0000 7fb7a06ec640 1 -- 172.21.15.40:0/1050803162 wait complete. 2024-04-21T23:29:13.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.148+0000 7fb7a06ec640 1 Processor -- start 2024-04-21T23:29:13.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.148+0000 7fb7a06ec640 1 -- start start 2024-04-21T23:29:13.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.148+0000 7fb7a06ec640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb798100990 0x7fb798198d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:29:13.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.148+0000 7fb7a06ec640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb798101330 0x7fb798199240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:29:13.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.148+0000 7fb7a06ec640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb798106690 0x7fb7981a3d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:29:13.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.148+0000 7fb7a06ec640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb7981109e0 con 0x7fb798106690 2024-04-21T23:29:13.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.148+0000 7fb7a06ec640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb798110860 con 0x7fb798100990 2024-04-21T23:29:13.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.148+0000 7fb7a06ec640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb798110b60 con 0x7fb798101330 2024-04-21T23:29:13.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.148+0000 7fb79ec89640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb798106690 0x7fb7981a3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:29:13.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.148+0000 7fb79ec89640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb798106690 0x7fb7981a3d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52108/0 (socket says 172.21.15.40:52108) 2024-04-21T23:29:13.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.148+0000 7fb79ec89640 1 -- 172.21.15.40:0/3215496932 learned_addr learned my addr 172.21.15.40:0/3215496932 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:29:13.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.148+0000 7fb79e488640 1 --2- 172.21.15.40:0/3215496932 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb798100990 0x7fb798198d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:29:13.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.148+0000 7fb79dc87640 1 --2- 172.21.15.40:0/3215496932 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb798101330 0x7fb798199240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:29:13.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.148+0000 7fb79ec89640 1 -- 172.21.15.40:0/3215496932 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb798101330 msgr2=0x7fb798199240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:29:13.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.148+0000 7fb79ec89640 1 --2- 172.21.15.40:0/3215496932 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb798101330 0x7fb798199240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:13.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.148+0000 7fb79ec89640 1 -- 172.21.15.40:0/3215496932 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb798100990 msgr2=0x7fb798198d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:29:13.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.148+0000 7fb79ec89640 1 --2- 172.21.15.40:0/3215496932 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb798100990 0x7fb798198d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:13.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.148+0000 7fb79ec89640 1 -- 172.21.15.40:0/3215496932 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7981a42c0 con 0x7fb798106690 2024-04-21T23:29:13.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.149+0000 7fb79ec89640 1 --2- 172.21.15.40:0/3215496932 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb798106690 0x7fb7981a3d80 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fb79002c700 tx=0x7fb790032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:29:13.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.149+0000 7fb78f7fe640 1 -- 172.21.15.40:0/3215496932 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb790041020 con 0x7fb798106690 2024-04-21T23:29:13.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.149+0000 7fb78f7fe640 1 -- 172.21.15.40:0/3215496932 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb790034cd0 con 0x7fb798106690 2024-04-21T23:29:13.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.149+0000 7fb7a06ec640 1 -- 172.21.15.40:0/3215496932 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7981a4490 con 0x7fb798106690 2024-04-21T23:29:13.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.149+0000 7fb78f7fe640 1 -- 172.21.15.40:0/3215496932 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb790039040 con 0x7fb798106690 2024-04-21T23:29:13.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.149+0000 7fb7a06ec640 1 -- 172.21.15.40:0/3215496932 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7981a48f0 con 0x7fb798106690 2024-04-21T23:29:13.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.150+0000 7fb7a06ec640 1 -- 172.21.15.40:0/3215496932 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb798101d80 con 0x7fb798106690 2024-04-21T23:29:13.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.151+0000 7fb78f7fe640 1 -- 172.21.15.40:0/3215496932 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fb790044020 con 0x7fb798106690 2024-04-21T23:29:13.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.152+0000 7fb78f7fe640 1 --2- 172.21.15.40:0/3215496932 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fb7840a2d00 0x7fb7840a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:29:13.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.152+0000 7fb79e488640 1 --2- 172.21.15.40:0/3215496932 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fb7840a2d00 0x7fb7840a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:29:13.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.152+0000 7fb78f7fe640 1 -- 172.21.15.40:0/3215496932 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9642+0+0 (secure 0 0 0) 0x7fb7900f1950 con 0x7fb798106690 2024-04-21T23:29:13.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.152+0000 7fb79e488640 1 --2- 172.21.15.40:0/3215496932 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fb7840a2d00 0x7fb7840a51c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fb79400ab60 tx=0x7fb79400a2d0 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:29:13.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.155+0000 7fb78f7fe640 1 -- 172.21.15.40:0/3215496932 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fb79004b020 con 0x7fb798106690 2024-04-21T23:29:13.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:29:12 smithi174 ceph-mon[31409]: pgmap v336: 129 pgs: 129 active+clean; 95 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-21T23:29:13.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:29:12 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/1031735729' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:29:13.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:29:12 smithi111 ceph-mon[31085]: pgmap v336: 129 pgs: 129 active+clean; 95 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-21T23:29:13.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:29:12 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/1031735729' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:29:13.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.340+0000 7fb7a06ec640 1 -- 172.21.15.40:0/3215496932 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fb7981a4cd0 con 0x7fb798106690 2024-04-21T23:29:13.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.341+0000 7fb78f7fe640 1 -- 172.21.15.40:0/3215496932 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v104) ==== 104+0+11 (secure 0 0 0) 0x7fb7900be1c0 con 0x7fb798106690 2024-04-21T23:29:13.340 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-21T23:29:13.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.343+0000 7fb7a06ec640 1 -- 172.21.15.40:0/3215496932 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fb7840a2d00 msgr2=0x7fb7840a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:29:13.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.343+0000 7fb7a06ec640 1 --2- 172.21.15.40:0/3215496932 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fb7840a2d00 0x7fb7840a51c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fb79400ab60 tx=0x7fb79400a2d0 comp rx=0 tx=0).stop 2024-04-21T23:29:13.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.343+0000 7fb7a06ec640 1 -- 172.21.15.40:0/3215496932 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb798106690 msgr2=0x7fb7981a3d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:29:13.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.343+0000 7fb7a06ec640 1 --2- 172.21.15.40:0/3215496932 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb798106690 0x7fb7981a3d80 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fb79002c700 tx=0x7fb790032f20 comp rx=0 tx=0).stop 2024-04-21T23:29:13.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.343+0000 7fb7a06ec640 1 -- 172.21.15.40:0/3215496932 shutdown_connections 2024-04-21T23:29:13.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.343+0000 7fb7a06ec640 1 --2- 172.21.15.40:0/3215496932 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fb7840a2d00 0x7fb7840a51c0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:13.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.343+0000 7fb7a06ec640 1 --2- 172.21.15.40:0/3215496932 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb798106690 0x7fb7981a3d80 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:13.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.343+0000 7fb7a06ec640 1 --2- 172.21.15.40:0/3215496932 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb798101330 0x7fb798199240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:13.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.343+0000 7fb7a06ec640 1 --2- 172.21.15.40:0/3215496932 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb798100990 0x7fb798198d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:13.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.343+0000 7fb7a06ec640 1 -- 172.21.15.40:0/3215496932 >> 172.21.15.40:0/3215496932 conn(0x7fb798076440 msgr2=0x7fb7980ff630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:29:13.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.343+0000 7fb7a06ec640 1 -- 172.21.15.40:0/3215496932 shutdown_connections 2024-04-21T23:29:13.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.343+0000 7fb7a06ec640 1 -- 172.21.15.40:0/3215496932 wait complete. 2024-04-21T23:29:13.353 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T23:29:13.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.459+0000 7f901725f640 1 -- 172.21.15.40:0/1596106859 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9010101710 msgr2=0x7f9010101b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:29:13.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.459+0000 7f901725f640 1 --2- 172.21.15.40:0/1596106859 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9010101710 0x7f9010101b90 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f9004009930 tx=0x7f900402bc30 comp rx=0 tx=0).stop 2024-04-21T23:29:13.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.459+0000 7f901725f640 1 -- 172.21.15.40:0/1596106859 shutdown_connections 2024-04-21T23:29:13.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.459+0000 7f901725f640 1 --2- 172.21.15.40:0/1596106859 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90101020d0 0x7f901010e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:13.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.459+0000 7f901725f640 1 --2- 172.21.15.40:0/1596106859 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9010101710 0x7f9010101b90 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:13.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.459+0000 7f901725f640 1 --2- 172.21.15.40:0/1596106859 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9010107720 0x7f9010107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:13.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.459+0000 7f901725f640 1 -- 172.21.15.40:0/1596106859 >> 172.21.15.40:0/1596106859 conn(0x7f9010078b90 msgr2=0x7f9010078fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:29:13.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.459+0000 7f901725f640 1 -- 172.21.15.40:0/1596106859 shutdown_connections 2024-04-21T23:29:13.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.459+0000 7f901725f640 1 -- 172.21.15.40:0/1596106859 wait complete. 2024-04-21T23:29:13.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.460+0000 7f901725f640 1 Processor -- start 2024-04-21T23:29:13.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.460+0000 7f901725f640 1 -- start start 2024-04-21T23:29:13.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.460+0000 7f901725f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9010101710 0x7f9010198190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:29:13.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.460+0000 7f901725f640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90101020d0 0x7f90101986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:29:13.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.460+0000 7f901725f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9010107720 0x7f901019ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:29:13.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.460+0000 7f901725f640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9010111bb0 con 0x7f9010101710 2024-04-21T23:29:13.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.460+0000 7f901725f640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f9010111a30 con 0x7f90101020d0 2024-04-21T23:29:13.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.460+0000 7f901725f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9010111d30 con 0x7f9010107720 2024-04-21T23:29:13.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.460+0000 7f9014ffb640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9010101710 0x7f9010198190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:29:13.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.460+0000 7f9014ffb640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9010101710 0x7f9010198190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52136/0 (socket says 172.21.15.40:52136) 2024-04-21T23:29:13.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.460+0000 7f9014ffb640 1 -- 172.21.15.40:0/1876735274 learned_addr learned my addr 172.21.15.40:0/1876735274 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:29:13.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.460+0000 7f9014ffb640 1 -- 172.21.15.40:0/1876735274 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9010107720 msgr2=0x7f901019ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:29:13.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.460+0000 7f90157fc640 1 --2- 172.21.15.40:0/1876735274 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9010107720 0x7f901019ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:29:13.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.460+0000 7f900ffff640 1 --2- 172.21.15.40:0/1876735274 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90101020d0 0x7f90101986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:29:13.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.460+0000 7f9014ffb640 1 --2- 172.21.15.40:0/1876735274 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9010107720 0x7f901019ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:13.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.460+0000 7f9014ffb640 1 -- 172.21.15.40:0/1876735274 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90101020d0 msgr2=0x7f90101986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:29:13.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.460+0000 7f9014ffb640 1 --2- 172.21.15.40:0/1876735274 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90101020d0 0x7f90101986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:13.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.460+0000 7f9014ffb640 1 -- 172.21.15.40:0/1876735274 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f901019d130 con 0x7f9010101710 2024-04-21T23:29:13.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.460+0000 7f90157fc640 1 --2- 172.21.15.40:0/1876735274 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9010107720 0x7f901019ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:29:13.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.460+0000 7f900ffff640 1 --2- 172.21.15.40:0/1876735274 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90101020d0 0x7f90101986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:29:13.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.460+0000 7f9014ffb640 1 --2- 172.21.15.40:0/1876735274 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9010101710 0x7f9010198190 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f900000b4a0 tx=0x7f900000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:29:13.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.461+0000 7f900dffb640 1 -- 172.21.15.40:0/1876735274 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9000014070 con 0x7f9010101710 2024-04-21T23:29:13.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.461+0000 7f900dffb640 1 -- 172.21.15.40:0/1876735274 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9000010af0 con 0x7f9010101710 2024-04-21T23:29:13.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.461+0000 7f900dffb640 1 -- 172.21.15.40:0/1876735274 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9000017d10 con 0x7f9010101710 2024-04-21T23:29:13.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.461+0000 7f901725f640 1 -- 172.21.15.40:0/1876735274 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f901019d420 con 0x7f9010101710 2024-04-21T23:29:13.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.461+0000 7f901725f640 1 -- 172.21.15.40:0/1876735274 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90101a4cb0 con 0x7f9010101710 2024-04-21T23:29:13.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.462+0000 7f901725f640 1 -- 172.21.15.40:0/1876735274 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90100689b0 con 0x7f9010101710 2024-04-21T23:29:13.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.463+0000 7f900dffb640 1 -- 172.21.15.40:0/1876735274 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f9000017070 con 0x7f9010101710 2024-04-21T23:29:13.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.464+0000 7f900dffb640 1 --2- 172.21.15.40:0/1876735274 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f8ff80a2d00 0x7f8ff80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:29:13.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.464+0000 7f900ffff640 1 --2- 172.21.15.40:0/1876735274 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f8ff80a2d00 0x7f8ff80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:29:13.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.464+0000 7f900dffb640 1 -- 172.21.15.40:0/1876735274 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9642+0+0 (secure 0 0 0) 0x7f90000cec20 con 0x7f9010101710 2024-04-21T23:29:13.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.464+0000 7f900ffff640 1 --2- 172.21.15.40:0/1876735274 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f8ff80a2d00 0x7f8ff80a51c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9004006010 tx=0x7f9004000bc0 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:29:13.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.468+0000 7f900dffb640 1 -- 172.21.15.40:0/1876735274 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f900009b490 con 0x7f9010101710 2024-04-21T23:29:13.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.652+0000 7f901725f640 1 -- 172.21.15.40:0/1876735274 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f901002d170 con 0x7f9010101710 2024-04-21T23:29:13.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.653+0000 7f900dffb640 1 -- 172.21.15.40:0/1876735274 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v104) ==== 100+0+11 (secure 0 0 0) 0x7f900009ad90 con 0x7f9010101710 2024-04-21T23:29:13.652 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-04-21T23:29:13.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.655+0000 7f901725f640 1 -- 172.21.15.40:0/1876735274 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f8ff80a2d00 msgr2=0x7f8ff80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:29:13.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.655+0000 7f901725f640 1 --2- 172.21.15.40:0/1876735274 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f8ff80a2d00 0x7f8ff80a51c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9004006010 tx=0x7f9004000bc0 comp rx=0 tx=0).stop 2024-04-21T23:29:13.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.655+0000 7f901725f640 1 -- 172.21.15.40:0/1876735274 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9010101710 msgr2=0x7f9010198190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:29:13.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.655+0000 7f901725f640 1 --2- 172.21.15.40:0/1876735274 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9010101710 0x7f9010198190 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f900000b4a0 tx=0x7f900000b960 comp rx=0 tx=0).stop 2024-04-21T23:29:13.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.655+0000 7f901725f640 1 -- 172.21.15.40:0/1876735274 shutdown_connections 2024-04-21T23:29:13.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.655+0000 7f901725f640 1 --2- 172.21.15.40:0/1876735274 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f8ff80a2d00 0x7f8ff80a51c0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:13.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.655+0000 7f901725f640 1 --2- 172.21.15.40:0/1876735274 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90101020d0 0x7f90101986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:13.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.655+0000 7f901725f640 1 --2- 172.21.15.40:0/1876735274 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9010107720 0x7f901019ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:13.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.655+0000 7f901725f640 1 --2- 172.21.15.40:0/1876735274 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9010101710 0x7f9010198190 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:13.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.655+0000 7f901725f640 1 -- 172.21.15.40:0/1876735274 >> 172.21.15.40:0/1876735274 conn(0x7f9010078b90 msgr2=0x7f90100ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:29:13.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.655+0000 7f901725f640 1 -- 172.21.15.40:0/1876735274 shutdown_connections 2024-04-21T23:29:13.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.655+0000 7f901725f640 1 -- 172.21.15.40:0/1876735274 wait complete. 2024-04-21T23:29:13.666 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T23:29:13.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.771+0000 7fe2886cd640 1 -- 172.21.15.40:0/3425384259 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2801020d0 msgr2=0x7fe28010e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:29:13.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.771+0000 7fe2886cd640 1 --2- 172.21.15.40:0/3425384259 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2801020d0 0x7fe28010e5e0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fe278009cd0 tx=0x7fe27802c0f0 comp rx=0 tx=0).stop 2024-04-21T23:29:13.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.771+0000 7fe2886cd640 1 -- 172.21.15.40:0/3425384259 shutdown_connections 2024-04-21T23:29:13.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.771+0000 7fe2886cd640 1 --2- 172.21.15.40:0/3425384259 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2801020d0 0x7fe28010e5e0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:13.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.771+0000 7fe2886cd640 1 --2- 172.21.15.40:0/3425384259 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe280101710 0x7fe280101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:13.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.771+0000 7fe2886cd640 1 --2- 172.21.15.40:0/3425384259 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe280107720 0x7fe280107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:13.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.771+0000 7fe2886cd640 1 -- 172.21.15.40:0/3425384259 >> 172.21.15.40:0/3425384259 conn(0x7fe280078b90 msgr2=0x7fe280078fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:29:13.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.771+0000 7fe2886cd640 1 -- 172.21.15.40:0/3425384259 shutdown_connections 2024-04-21T23:29:13.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.771+0000 7fe2886cd640 1 -- 172.21.15.40:0/3425384259 wait complete. 2024-04-21T23:29:13.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.772+0000 7fe2886cd640 1 Processor -- start 2024-04-21T23:29:13.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.772+0000 7fe2886cd640 1 -- start start 2024-04-21T23:29:13.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.772+0000 7fe2886cd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe280101710 0x7fe28019a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:29:13.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.772+0000 7fe2886cd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2801020d0 0x7fe28019a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:29:13.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.772+0000 7fe2886cd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe280107720 0x7fe28019eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:29:13.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.772+0000 7fe2886cd640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe280111b30 con 0x7fe2801020d0 2024-04-21T23:29:13.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.772+0000 7fe2886cd640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe2801119b0 con 0x7fe280107720 2024-04-21T23:29:13.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.772+0000 7fe2886cd640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe280111cb0 con 0x7fe280101710 2024-04-21T23:29:13.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.772+0000 7fe285c68640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2801020d0 0x7fe28019a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:29:13.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.772+0000 7fe285c68640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2801020d0 0x7fe28019a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52156/0 (socket says 172.21.15.40:52156) 2024-04-21T23:29:13.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.772+0000 7fe285c68640 1 -- 172.21.15.40:0/803701965 learned_addr learned my addr 172.21.15.40:0/803701965 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:29:13.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.772+0000 7fe286c6a640 1 --2- 172.21.15.40:0/803701965 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe280107720 0x7fe28019eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:29:13.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.772+0000 7fe286469640 1 --2- 172.21.15.40:0/803701965 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe280101710 0x7fe28019a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:29:13.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.772+0000 7fe285c68640 1 -- 172.21.15.40:0/803701965 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe280101710 msgr2=0x7fe28019a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:29:13.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.772+0000 7fe285c68640 1 --2- 172.21.15.40:0/803701965 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe280101710 0x7fe28019a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:13.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.772+0000 7fe285c68640 1 -- 172.21.15.40:0/803701965 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe280107720 msgr2=0x7fe28019eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:29:13.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.772+0000 7fe285c68640 1 --2- 172.21.15.40:0/803701965 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe280107720 0x7fe28019eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:13.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.772+0000 7fe285c68640 1 -- 172.21.15.40:0/803701965 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe28019f1e0 con 0x7fe2801020d0 2024-04-21T23:29:13.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.773+0000 7fe285c68640 1 --2- 172.21.15.40:0/803701965 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2801020d0 0x7fe28019a730 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fe27000e9f0 tx=0x7fe27000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:29:13.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.773+0000 7fe2777fe640 1 -- 172.21.15.40:0/803701965 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe27000cd80 con 0x7fe2801020d0 2024-04-21T23:29:13.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.773+0000 7fe2777fe640 1 -- 172.21.15.40:0/803701965 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe270005500 con 0x7fe2801020d0 2024-04-21T23:29:13.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.773+0000 7fe2777fe640 1 -- 172.21.15.40:0/803701965 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe270010690 con 0x7fe2801020d0 2024-04-21T23:29:13.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.773+0000 7fe2886cd640 1 -- 172.21.15.40:0/803701965 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe28019f4d0 con 0x7fe2801020d0 2024-04-21T23:29:13.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.773+0000 7fe2886cd640 1 -- 172.21.15.40:0/803701965 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2801a6d60 con 0x7fe2801020d0 2024-04-21T23:29:13.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.774+0000 7fe2886cd640 1 -- 172.21.15.40:0/803701965 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2800689b0 con 0x7fe2801020d0 2024-04-21T23:29:13.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.775+0000 7fe2777fe640 1 -- 172.21.15.40:0/803701965 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fe2700046c0 con 0x7fe2801020d0 2024-04-21T23:29:13.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.775+0000 7fe2777fe640 1 --2- 172.21.15.40:0/803701965 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fe26c0a2d80 0x7fe26c0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:29:13.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.776+0000 7fe2777fe640 1 -- 172.21.15.40:0/803701965 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9642+0+0 (secure 0 0 0) 0x7fe2700cfce0 con 0x7fe2801020d0 2024-04-21T23:29:13.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.776+0000 7fe286469640 1 --2- 172.21.15.40:0/803701965 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fe26c0a2d80 0x7fe26c0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:29:13.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.776+0000 7fe286469640 1 --2- 172.21.15.40:0/803701965 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fe26c0a2d80 0x7fe26c0a5240 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe27c005fd0 tx=0x7fe27c005850 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:29:13.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.780+0000 7fe2777fe640 1 -- 172.21.15.40:0/803701965 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fe27009c4a0 con 0x7fe2801020d0 2024-04-21T23:29:13.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:29:13 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/2091824067' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:29:13.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:29:13 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/3215496932' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T23:29:13.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:29:13 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/1876735274' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T23:29:13.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.964+0000 7fe2886cd640 1 -- 172.21.15.40:0/803701965 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fe2801a7060 con 0x7fe2801020d0 2024-04-21T23:29:13.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.965+0000 7fe2777fe640 1 -- 172.21.15.40:0/803701965 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v104) ==== 103+0+11 (secure 0 0 0) 0x7fe27009bda0 con 0x7fe2801020d0 2024-04-21T23:29:13.964 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-21T23:29:13.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.967+0000 7fe2886cd640 1 -- 172.21.15.40:0/803701965 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fe26c0a2d80 msgr2=0x7fe26c0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:29:13.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.967+0000 7fe2886cd640 1 --2- 172.21.15.40:0/803701965 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fe26c0a2d80 0x7fe26c0a5240 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe27c005fd0 tx=0x7fe27c005850 comp rx=0 tx=0).stop 2024-04-21T23:29:13.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.967+0000 7fe2886cd640 1 -- 172.21.15.40:0/803701965 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2801020d0 msgr2=0x7fe28019a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:29:13.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.967+0000 7fe2886cd640 1 --2- 172.21.15.40:0/803701965 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2801020d0 0x7fe28019a730 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fe27000e9f0 tx=0x7fe27000edf0 comp rx=0 tx=0).stop 2024-04-21T23:29:13.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.967+0000 7fe2886cd640 1 -- 172.21.15.40:0/803701965 shutdown_connections 2024-04-21T23:29:13.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.967+0000 7fe2886cd640 1 --2- 172.21.15.40:0/803701965 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fe26c0a2d80 0x7fe26c0a5240 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:13.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.967+0000 7fe2886cd640 1 --2- 172.21.15.40:0/803701965 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe280107720 0x7fe28019eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:13.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.967+0000 7fe2886cd640 1 --2- 172.21.15.40:0/803701965 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2801020d0 0x7fe28019a730 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:13.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.967+0000 7fe2886cd640 1 --2- 172.21.15.40:0/803701965 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe280101710 0x7fe28019a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:13.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.967+0000 7fe2886cd640 1 -- 172.21.15.40:0/803701965 >> 172.21.15.40:0/803701965 conn(0x7fe280078b90 msgr2=0x7fe28010ce80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:29:13.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.967+0000 7fe2886cd640 1 -- 172.21.15.40:0/803701965 shutdown_connections 2024-04-21T23:29:13.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:13.967+0000 7fe2886cd640 1 -- 172.21.15.40:0/803701965 wait complete. 2024-04-21T23:29:13.976 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-21T23:29:14.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.080+0000 7fe99c177640 1 -- 172.21.15.40:0/2346591646 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe994074400 msgr2=0x7fe9941121e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:29:14.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.080+0000 7fe99c177640 1 --2- 172.21.15.40:0/2346591646 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe994074400 0x7fe9941121e0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fe984009900 tx=0x7fe98402bcd0 comp rx=0 tx=0).stop 2024-04-21T23:29:14.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.080+0000 7fe99c177640 1 -- 172.21.15.40:0/2346591646 shutdown_connections 2024-04-21T23:29:14.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.080+0000 7fe99c177640 1 --2- 172.21.15.40:0/2346591646 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe994112720 0x7fe994114b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:14.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.080+0000 7fe99c177640 1 --2- 172.21.15.40:0/2346591646 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe994074400 0x7fe9941121e0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:14.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.080+0000 7fe99c177640 1 --2- 172.21.15.40:0/2346591646 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe994073ae0 0x7fe994073ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:14.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.080+0000 7fe99c177640 1 -- 172.21.15.40:0/2346591646 >> 172.21.15.40:0/2346591646 conn(0x7fe9940fd180 msgr2=0x7fe9940ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:29:14.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.080+0000 7fe99c177640 1 -- 172.21.15.40:0/2346591646 shutdown_connections 2024-04-21T23:29:14.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.080+0000 7fe99c177640 1 -- 172.21.15.40:0/2346591646 wait complete. 2024-04-21T23:29:14.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.081+0000 7fe99c177640 1 Processor -- start 2024-04-21T23:29:14.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.081+0000 7fe99c177640 1 -- start start 2024-04-21T23:29:14.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.081+0000 7fe99c177640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe994073ae0 0x7fe99419e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:29:14.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.081+0000 7fe99c177640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe994074400 0x7fe99419ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:29:14.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.081+0000 7fe99c177640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe994112720 0x7fe9941a2f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:29:14.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.081+0000 7fe99c177640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe994118110 con 0x7fe994074400 2024-04-21T23:29:14.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.081+0000 7fe99c177640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe994117f90 con 0x7fe994112720 2024-04-21T23:29:14.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.081+0000 7fe99c177640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe994118290 con 0x7fe994073ae0 2024-04-21T23:29:14.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.081+0000 7fe999712640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe994074400 0x7fe99419ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:29:14.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.081+0000 7fe999712640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe994074400 0x7fe99419ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52176/0 (socket says 172.21.15.40:52176) 2024-04-21T23:29:14.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.081+0000 7fe999712640 1 -- 172.21.15.40:0/3155055891 learned_addr learned my addr 172.21.15.40:0/3155055891 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:29:14.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.081+0000 7fe999f13640 1 --2- 172.21.15.40:0/3155055891 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe994073ae0 0x7fe99419e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:29:14.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.082+0000 7fe99a714640 1 --2- 172.21.15.40:0/3155055891 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe994112720 0x7fe9941a2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:29:14.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.082+0000 7fe999712640 1 -- 172.21.15.40:0/3155055891 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe994073ae0 msgr2=0x7fe99419e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:29:14.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.082+0000 7fe999712640 1 --2- 172.21.15.40:0/3155055891 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe994073ae0 0x7fe99419e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:14.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.082+0000 7fe999712640 1 -- 172.21.15.40:0/3155055891 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe994112720 msgr2=0x7fe9941a2f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:29:14.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.082+0000 7fe999712640 1 --2- 172.21.15.40:0/3155055891 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe994112720 0x7fe9941a2f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:14.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.082+0000 7fe999712640 1 -- 172.21.15.40:0/3155055891 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9941a3670 con 0x7fe994074400 2024-04-21T23:29:14.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.082+0000 7fe99a714640 1 --2- 172.21.15.40:0/3155055891 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe994112720 0x7fe9941a2f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:29:14.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.082+0000 7fe999712640 1 --2- 172.21.15.40:0/3155055891 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe994074400 0x7fe99419ec10 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fe984005940 tx=0x7fe98402cd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:29:14.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.082+0000 7fe98affd640 1 -- 172.21.15.40:0/3155055891 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe984041070 con 0x7fe994074400 2024-04-21T23:29:14.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.082+0000 7fe99c177640 1 -- 172.21.15.40:0/3155055891 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9941a3900 con 0x7fe994074400 2024-04-21T23:29:14.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.082+0000 7fe98affd640 1 -- 172.21.15.40:0/3155055891 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe984004450 con 0x7fe994074400 2024-04-21T23:29:14.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.082+0000 7fe98affd640 1 -- 172.21.15.40:0/3155055891 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe984034440 con 0x7fe994074400 2024-04-21T23:29:14.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.082+0000 7fe99c177640 1 -- 172.21.15.40:0/3155055891 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9941ab1f0 con 0x7fe994074400 2024-04-21T23:29:14.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.083+0000 7fe99c177640 1 -- 172.21.15.40:0/3155055891 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9940689b0 con 0x7fe994074400 2024-04-21T23:29:14.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.084+0000 7fe98affd640 1 -- 172.21.15.40:0/3155055891 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fe984044070 con 0x7fe994074400 2024-04-21T23:29:14.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.085+0000 7fe98affd640 1 --2- 172.21.15.40:0/3155055891 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fe9800a2d00 0x7fe9800a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:29:14.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.085+0000 7fe98affd640 1 -- 172.21.15.40:0/3155055891 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9642+0+0 (secure 0 0 0) 0x7fe9840f1710 con 0x7fe994074400 2024-04-21T23:29:14.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.085+0000 7fe999f13640 1 --2- 172.21.15.40:0/3155055891 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fe9800a2d00 0x7fe9800a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:29:14.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.085+0000 7fe999f13640 1 --2- 172.21.15.40:0/3155055891 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fe9800a2d00 0x7fe9800a51c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fe990005fd0 tx=0x7fe990005850 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:29:14.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.088+0000 7fe98affd640 1 -- 172.21.15.40:0/3155055891 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fe9840bdf80 con 0x7fe994074400 2024-04-21T23:29:14.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:29:13 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/2091824067' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:29:14.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:29:13 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/3215496932' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T23:29:14.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:29:13 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/1876735274' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T23:29:14.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:29:13 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/2091824067' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:29:14.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:29:13 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/3215496932' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T23:29:14.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:29:13 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/1876735274' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T23:29:14.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.286+0000 7fe99c177640 1 -- 172.21.15.40:0/3155055891 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fe994104b70 con 0x7fe994074400 2024-04-21T23:29:14.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.286+0000 7fe98affd640 1 -- 172.21.15.40:0/3155055891 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v104) ==== 81+0+942 (secure 0 0 0) 0x7fe9840bd880 con 0x7fe994074400 2024-04-21T23:29:14.286 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:6801/2500760788 2024-04-22T23:18:23.389674+0000 2024-04-21T23:29:14.286 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:6800/2500760788 2024-04-22T23:18:23.389674+0000 2024-04-21T23:29:14.286 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:0/2742002992 2024-04-22T23:18:23.389674+0000 2024-04-21T23:29:14.286 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:0/2494672676 2024-04-22T23:18:23.389674+0000 2024-04-21T23:29:14.286 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:6801/526554319 2024-04-22T23:08:50.305770+0000 2024-04-21T23:29:14.286 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:6800/526554319 2024-04-22T23:08:50.305770+0000 2024-04-21T23:29:14.287 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:0/3178763471 2024-04-22T23:08:50.305770+0000 2024-04-21T23:29:14.287 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:0/4045911364 2024-04-22T23:08:50.305770+0000 2024-04-21T23:29:14.287 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:0/276300080 2024-04-22T23:08:50.305770+0000 2024-04-21T23:29:14.287 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:0/3689649814 2024-04-22T23:08:23.743030+0000 2024-04-21T23:29:14.287 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:0/2241116715 2024-04-22T23:08:23.743030+0000 2024-04-21T23:29:14.287 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:0/1406242364 2024-04-22T23:18:23.389674+0000 2024-04-21T23:29:14.287 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:0/1737227818 2024-04-22T23:18:23.389674+0000 2024-04-21T23:29:14.287 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:6801/1479040326 2024-04-22T23:08:23.743030+0000 2024-04-21T23:29:14.287 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:6800/1479040326 2024-04-22T23:08:23.743030+0000 2024-04-21T23:29:14.288 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:0/681298099 2024-04-22T23:08:23.743030+0000 2024-04-21T23:29:14.288 INFO:teuthology.orchestra.run.smithi040.stderr:listed 16 entries 2024-04-21T23:29:14.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.288+0000 7fe99c177640 1 -- 172.21.15.40:0/3155055891 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fe9800a2d00 msgr2=0x7fe9800a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:29:14.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.288+0000 7fe99c177640 1 --2- 172.21.15.40:0/3155055891 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fe9800a2d00 0x7fe9800a51c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fe990005fd0 tx=0x7fe990005850 comp rx=0 tx=0).stop 2024-04-21T23:29:14.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.288+0000 7fe99c177640 1 -- 172.21.15.40:0/3155055891 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe994074400 msgr2=0x7fe99419ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:29:14.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.288+0000 7fe99c177640 1 --2- 172.21.15.40:0/3155055891 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe994074400 0x7fe99419ec10 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fe984005940 tx=0x7fe98402cd10 comp rx=0 tx=0).stop 2024-04-21T23:29:14.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.288+0000 7fe99c177640 1 -- 172.21.15.40:0/3155055891 shutdown_connections 2024-04-21T23:29:14.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.288+0000 7fe99c177640 1 --2- 172.21.15.40:0/3155055891 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fe9800a2d00 0x7fe9800a51c0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:14.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.288+0000 7fe99c177640 1 --2- 172.21.15.40:0/3155055891 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe994112720 0x7fe9941a2f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:14.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.288+0000 7fe99c177640 1 --2- 172.21.15.40:0/3155055891 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe994074400 0x7fe99419ec10 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:14.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.288+0000 7fe99c177640 1 --2- 172.21.15.40:0/3155055891 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe994073ae0 0x7fe99419e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:14.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.288+0000 7fe99c177640 1 -- 172.21.15.40:0/3155055891 >> 172.21.15.40:0/3155055891 conn(0x7fe9940fd180 msgr2=0x7fe9940fea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:29:14.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.288+0000 7fe99c177640 1 -- 172.21.15.40:0/3155055891 shutdown_connections 2024-04-21T23:29:14.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.288+0000 7fe99c177640 1 -- 172.21.15.40:0/3155055891 wait complete. 2024-04-21T23:29:14.298 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-21T23:29:14.298 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:29:14.298 DEBUG:teuthology.orchestra.run.smithi040:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-21T23:29:14.403 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi040.front.sepia.ceph.com 2024-04-21T23:29:14.403 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:29:14.403 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-21T23:29:14.430 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:29:14.431 DEBUG:teuthology.orchestra.run.smithi040:> ip netns list 2024-04-21T23:29:14.489 INFO:teuthology.orchestra.run.smithi040.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-21T23:29:14.490 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:29:14.490 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-21T23:29:14.555 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:29:14.556 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link delete ceph-brx 2024-04-21T23:29:14.669 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-21T23:29:14.669 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-21T23:29:14.669 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-21T23:29:14.686 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056882,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1576 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/sda1 /var/lib/containers/storage/overlay ext4 rw,seclabel,relatime 0 0 overlay /var/lib/containers/storage/overlay/ee97a42f3a3e584103a65e5401d26f9e90beee3b099774e27956a06f9aae3f04/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/EFQX5Y55JKD55XPWEOWP52ZOQB:/var/lib/containers/storage/overlay/l/LUXYVRUOWUHRECDY5CU37K55RY,upperdir=/var/lib/containers/storage/overlay/ee97a42f3a3e584103a65e5401d26f9e90beee3b099774e27956a06f9aae3f04/diff,workdir=/var/lib/containers/storage/overlay/ee97a42f3a3e584103a65e5401d26f9e90beee3b099774e27956a06f9aae3f04/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/5b578d117dff6c77ff42485355b1f3fbd0ba32be0fd7454e14957896ffde180b/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/EFQX5Y55JKD55XPWEOWP52ZOQB:/var/lib/containers/storage/overlay/l/LUXYVRUOWUHRECDY5CU37K55RY,upperdir=/var/lib/containers/storage/overlay/5b578d117dff6c77ff42485355b1f3fbd0ba32be0fd7454e14957896ffde180b/diff,workdir=/var/lib/containers/storage/overlay/5b578d117dff6c77ff42485355b1f3fbd0ba32be0fd7454e14957896ffde180b/work,metacopy=on,volatile 0 0 tmpfs /run/user/0 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,inode64 0 0 binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,nosuid,nodev,noexec,relatime 0 0 overlay /var/lib/containers/storage/overlay/bf129acafa6e662f9d769e35d1abad3b8a73cce30705229d03896b165f2b4872/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/EFQX5Y55JKD55XPWEOWP52ZOQB:/var/lib/containers/storage/overlay/l/LUXYVRUOWUHRECDY5CU37K55RY,upperdir=/var/lib/containers/storage/overlay/bf129acafa6e662f9d769e35d1abad3b8a73cce30705229d03896b165f2b4872/diff,workdir=/var/lib/containers/storage/overlay/bf129acafa6e662f9d769e35d1abad3b8a73cce30705229d03896b165f2b4872/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/c435ffa4563f8d391453469885f0c52413600a8c901881dca35f7818d9476346/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/EFQX5Y55JKD55XPWEOWP52ZOQB:/var/lib/containers/storage/overlay/l/LUXYVRUOWUHRECDY5CU37K55RY,upperdir=/var/lib/containers/storage/overlay/c435ffa4563f8d391453469885f0c52413600a8c901881dca35f7818d9476346/diff,workdir=/var/lib/containers/storage/overlay/c435ffa4563f8d391453469885f0c52413600a8c901881dca35f7818d9476346/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/12f881f08e65218aacc8a6f86b8d1c811dd8e7658e6e8413801df6999f7b4c19/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/EFQX5Y55JKD55XPWEOWP52ZOQB:/var/lib/containers/storage/overlay/l/LUXYVRUOWUHRECDY5CU37K55RY,upperdir=/var/lib/containers/storage/overlay/12f881f08e65218aacc8a6f86b8d1c811dd8e7658e6e8413801df6999f7b4c19/diff,workdir=/var/lib/containers/storage/overlay/12f881f08e65218aacc8a6f86b8d1c811dd8e7658e6e8413801df6999f7b4c19/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/b0b1a36da5ee61cea069b200bcfca27244b02cdd87683301abbb66cf65f536b5/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/EFQX5Y55JKD55XPWEOWP52ZOQB:/var/lib/containers/storage/overlay/l/LUXYVRUOWUHRECDY5CU37K55RY,upperdir=/var/lib/containers/storage/overlay/b0b1a36da5ee61cea069b200bcfca27244b02cdd87683301abbb66cf65f536b5/diff,workdir=/var/lib/containers/storage/overlay/b0b1a36da5ee61cea069b200bcfca27244b02cdd87683301abbb66cf65f536b5/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/5f6305e2feed35661880eadf1335149f8d71d88c3a6744bb0d69757ff7359b05/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/EFQX5Y55JKD55XPWEOWP52ZOQB:/var/lib/containers/storage/overlay/l/LUXYVRUOWUHRECDY5CU37K55RY,upperdir=/var/lib/containers/storage/overlay/5f6305e2feed35661880eadf1335149f8d71d88c3a6744bb0d69757ff7359b05/diff,workdir=/var/lib/containers/storage/overlay/5f6305e2feed35661880eadf1335149f8d71d88c3a6744bb0d69757ff7359b05/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/63385edd6428bd56f26f165ab67b26b0d9049a9a4de274cf1b6ae99f2abc6c34/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/EFQX5Y55JKD55XPWEOWP52ZOQB:/var/lib/containers/storage/overlay/l/LUXYVRUOWUHRECDY5CU37K55RY,upperdir=/var/lib/containers/storage/overlay/63385edd6428bd56f26f165ab67b26b0d9049a9a4de274cf1b6ae99f2abc6c34/diff,workdir=/var/lib/containers/storage/overlay/63385edd6428bd56f26f165ab67b26b0d9049a9a4de274cf1b6ae99f2abc6c34/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/0d815bb55ee16425c0accc123ecc48ae735c4c714c19e49a953241cdbd8a4711/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/EFQX5Y55JKD55XPWEOWP52ZOQB:/var/lib/containers/storage/overlay/l/LUXYVRUOWUHRECDY5CU37K55RY,upperdir=/var/lib/containers/storage/overlay/0d815bb55ee16425c0accc123ecc48ae735c4c714c19e49a953241cdbd8a4711/diff,workdir=/var/lib/containers/storage/overlay/0d815bb55ee16425c0accc123ecc48ae735c4c714c19e49a953241cdbd8a4711/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/36fed82ac0fd5a4832d3465a289ff98ef2fd709230eff6e76c50acf5cf96d143/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/EFQX5Y55JKD55XPWEOWP52ZOQB:/var/lib/containers/storage/overlay/l/LUXYVRUOWUHRECDY5CU37K55RY,upperdir=/var/lib/containers/storage/overlay/36fed82ac0fd5a4832d3465a289ff98ef2fd709230eff6e76c50acf5cf96d143/diff,workdir=/var/lib/containers/storage/overlay/36fed82ac0fd5a4832d3465a289ff98ef2fd709230eff6e76c50acf5cf96d143/work,metacopy=on,volatile 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 2024-04-21T23:29:14.686 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:29:14.687 DEBUG:teuthology.orchestra.run.smithi040:> ip netns list 2024-04-21T23:29:14.743 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:29:14.744 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link delete ceph-brx 2024-04-21T23:29:14.811 INFO:teuthology.orchestra.run.smithi040.stderr:Cannot find device "ceph-brx" 2024-04-21T23:29:14.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T23:29:14.813 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-21T23:29:14.886 INFO:tasks.cephadm:Teardown begin 2024-04-21T23:29:14.887 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-21T23:29:14.887 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:29:14.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.992+0000 7fb8845be640 1 -- 172.21.15.40:0/4051794988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb87c102160 msgr2=0x7fb87c10e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:29:14.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.992+0000 7fb8845be640 1 --2- 172.21.15.40:0/4051794988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb87c102160 0x7fb87c10e5a0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fb874009ca0 tx=0x7fb87402c180 comp rx=0 tx=0).stop 2024-04-21T23:29:14.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.992+0000 7fb8845be640 1 -- 172.21.15.40:0/4051794988 shutdown_connections 2024-04-21T23:29:14.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.992+0000 7fb8845be640 1 --2- 172.21.15.40:0/4051794988 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb87c1017c0 0x7fb87c101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:14.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.992+0000 7fb8845be640 1 --2- 172.21.15.40:0/4051794988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb87c102160 0x7fb87c10e5a0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:14.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.992+0000 7fb8845be640 1 --2- 172.21.15.40:0/4051794988 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb87c107750 0x7fb87c107b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:14.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.992+0000 7fb8845be640 1 -- 172.21.15.40:0/4051794988 >> 172.21.15.40:0/4051794988 conn(0x7fb87c0788d0 msgr2=0x7fb87c078ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:29:14.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.992+0000 7fb8845be640 1 -- 172.21.15.40:0/4051794988 shutdown_connections 2024-04-21T23:29:14.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.992+0000 7fb8845be640 1 -- 172.21.15.40:0/4051794988 wait complete. 2024-04-21T23:29:14.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.993+0000 7fb8845be640 1 Processor -- start 2024-04-21T23:29:14.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.993+0000 7fb8845be640 1 -- start start 2024-04-21T23:29:14.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.993+0000 7fb8845be640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb87c1017c0 0x7fb87c197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:29:14.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.993+0000 7fb8845be640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb87c102160 0x7fb87c198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:29:14.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.993+0000 7fb8845be640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb87c107750 0x7fb87c19c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:29:14.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.993+0000 7fb8845be640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb87c111b50 con 0x7fb87c102160 2024-04-21T23:29:14.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.993+0000 7fb8845be640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb87c1119d0 con 0x7fb87c107750 2024-04-21T23:29:14.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.993+0000 7fb8845be640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb87c111cd0 con 0x7fb87c1017c0 2024-04-21T23:29:14.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.993+0000 7fb881b59640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb87c102160 0x7fb87c198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:29:14.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.993+0000 7fb881b59640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb87c102160 0x7fb87c198530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52188/0 (socket says 172.21.15.40:52188) 2024-04-21T23:29:14.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.993+0000 7fb881b59640 1 -- 172.21.15.40:0/2381870370 learned_addr learned my addr 172.21.15.40:0/2381870370 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:29:14.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.993+0000 7fb88235a640 1 --2- 172.21.15.40:0/2381870370 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb87c1017c0 0x7fb87c197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:29:14.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.993+0000 7fb882b5b640 1 --2- 172.21.15.40:0/2381870370 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb87c107750 0x7fb87c19c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:29:14.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.993+0000 7fb881b59640 1 -- 172.21.15.40:0/2381870370 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb87c1017c0 msgr2=0x7fb87c197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:29:14.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.993+0000 7fb881b59640 1 --2- 172.21.15.40:0/2381870370 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb87c1017c0 0x7fb87c197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:14.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.993+0000 7fb881b59640 1 -- 172.21.15.40:0/2381870370 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb87c107750 msgr2=0x7fb87c19c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:29:14.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.993+0000 7fb881b59640 1 --2- 172.21.15.40:0/2381870370 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb87c107750 0x7fb87c19c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:14.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.993+0000 7fb881b59640 1 -- 172.21.15.40:0/2381870370 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb87c19cfe0 con 0x7fb87c102160 2024-04-21T23:29:14.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.993+0000 7fb882b5b640 1 --2- 172.21.15.40:0/2381870370 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb87c107750 0x7fb87c19c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:29:14.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.993+0000 7fb881b59640 1 --2- 172.21.15.40:0/2381870370 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb87c102160 0x7fb87c198530 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fb86c00b4a0 tx=0x7fb86c00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:29:14.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.994+0000 7fb8737fe640 1 -- 172.21.15.40:0/2381870370 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb86c014070 con 0x7fb87c102160 2024-04-21T23:29:14.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.994+0000 7fb8845be640 1 -- 172.21.15.40:0/2381870370 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb87c19d2d0 con 0x7fb87c102160 2024-04-21T23:29:14.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.994+0000 7fb8737fe640 1 -- 172.21.15.40:0/2381870370 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb86c010af0 con 0x7fb87c102160 2024-04-21T23:29:14.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.994+0000 7fb8737fe640 1 -- 172.21.15.40:0/2381870370 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb86c017d10 con 0x7fb87c102160 2024-04-21T23:29:14.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.994+0000 7fb8845be640 1 -- 172.21.15.40:0/2381870370 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb87c1a4b60 con 0x7fb87c102160 2024-04-21T23:29:14.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.995+0000 7fb8845be640 1 -- 172.21.15.40:0/2381870370 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb87c0689b0 con 0x7fb87c102160 2024-04-21T23:29:14.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.996+0000 7fb8737fe640 1 -- 172.21.15.40:0/2381870370 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fb86c017070 con 0x7fb87c102160 2024-04-21T23:29:14.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.997+0000 7fb8737fe640 1 --2- 172.21.15.40:0/2381870370 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fb8680a2d00 0x7fb8680a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:29:14.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.997+0000 7fb88235a640 1 --2- 172.21.15.40:0/2381870370 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fb8680a2d00 0x7fb8680a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:29:14.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.997+0000 7fb8737fe640 1 -- 172.21.15.40:0/2381870370 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9642+0+0 (secure 0 0 0) 0x7fb86c0ceb10 con 0x7fb87c102160 2024-04-21T23:29:14.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:14.998+0000 7fb88235a640 1 --2- 172.21.15.40:0/2381870370 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fb8680a2d00 0x7fb8680a51c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb878005d00 tx=0x7fb87800b410 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:29:15.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.002+0000 7fb8737fe640 1 -- 172.21.15.40:0/2381870370 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fb86c09b3e0 con 0x7fb87c102160 2024-04-21T23:29:15.181 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:29:14 smithi040 ceph-mon[27984]: pgmap v337: 129 pgs: 129 active+clean; 95 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-21T23:29:15.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:29:14 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/803701965' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T23:29:15.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:29:14 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/3155055891' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-21T23:29:15.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:29:14 smithi174 ceph-mon[31409]: pgmap v337: 129 pgs: 129 active+clean; 95 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-21T23:29:15.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:29:14 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/803701965' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T23:29:15.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:29:14 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/3155055891' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-21T23:29:15.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:29:14 smithi111 ceph-mon[31085]: pgmap v337: 129 pgs: 129 active+clean; 95 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-21T23:29:15.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:29:14 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/803701965' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T23:29:15.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:29:14 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/3155055891' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-21T23:29:15.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.226+0000 7fb8845be640 1 -- 172.21.15.40:0/2381870370 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb87c02d170 con 0x7fb87c102160 2024-04-21T23:29:15.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.227+0000 7fb8737fe640 1 -- 172.21.15.40:0/2381870370 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11882 (secure 0 0 0) 0x7fb86c09ace0 con 0x7fb87c102160 2024-04-21T23:29:15.226 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:29:15.226 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/1571948830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1571948830},{"type":"v1","addr":"172.21.15.40:6835","nonce":1571948830}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/969356642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":969356642},{"type":"v1","addr":"172.21.15.40:6837","nonce":969356642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6839/3025057845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":3025057845},{"type":"v1","addr":"172.21.15.40:6839","nonce":3025057845}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6841/3350488804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":3350488804},{"type":"v1","addr":"172.21.15.40:6841","nonce":3350488804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/3438493203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":3438493203},{"type":"v1","addr":"172.21.15.174:6833","nonce":3438493203}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/3105752678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":3105752678},{"type":"v1","addr":"172.21.15.111:6833","nonce":3105752678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T23:15:55.742233+0000","modified":"2024-04-21T23:23:05.004549+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24521,"mds_1":24502,"mds_2":24494},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24461":{"gid":24461,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6835/2704462092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2704462092},{"type":"v1","addr":"172.21.15.111:6835","nonce":2704462092}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/3905733045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3905733045},{"type":"v1","addr":"172.21.15.174:6835","nonce":3905733045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6837/465579746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":465579746},{"type":"v1","addr":"172.21.15.174:6837","nonce":465579746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":71,"addr":"172.21.15.111:6837/2703764182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":2703764182},{"type":"v1","addr":"172.21.15.111:6837","nonce":2703764182}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24502":{"gid":24502,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.174:6839/2467509274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":2467509274},{"type":"v1","addr":"172.21.15.174:6839","nonce":2467509274}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3712235548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3712235548},{"type":"v1","addr":"172.21.15.111:6839","nonce":3712235548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24494,24502,24521]},"id":1}]} 2024-04-21T23:29:15.227 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 33 2024-04-21T23:29:15.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.229+0000 7fb8845be640 1 -- 172.21.15.40:0/2381870370 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fb8680a2d00 msgr2=0x7fb8680a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:29:15.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.229+0000 7fb8845be640 1 --2- 172.21.15.40:0/2381870370 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fb8680a2d00 0x7fb8680a51c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb878005d00 tx=0x7fb87800b410 comp rx=0 tx=0).stop 2024-04-21T23:29:15.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.229+0000 7fb8845be640 1 -- 172.21.15.40:0/2381870370 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb87c102160 msgr2=0x7fb87c198530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:29:15.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.229+0000 7fb8845be640 1 --2- 172.21.15.40:0/2381870370 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb87c102160 0x7fb87c198530 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fb86c00b4a0 tx=0x7fb86c00b960 comp rx=0 tx=0).stop 2024-04-21T23:29:15.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.229+0000 7fb8845be640 1 -- 172.21.15.40:0/2381870370 shutdown_connections 2024-04-21T23:29:15.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.229+0000 7fb8845be640 1 --2- 172.21.15.40:0/2381870370 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fb8680a2d00 0x7fb8680a51c0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:15.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.229+0000 7fb8845be640 1 --2- 172.21.15.40:0/2381870370 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb87c107750 0x7fb87c19c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:15.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.229+0000 7fb8845be640 1 --2- 172.21.15.40:0/2381870370 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb87c102160 0x7fb87c198530 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:15.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.229+0000 7fb8845be640 1 --2- 172.21.15.40:0/2381870370 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb87c1017c0 0x7fb87c197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:15.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.229+0000 7fb8845be640 1 -- 172.21.15.40:0/2381870370 >> 172.21.15.40:0/2381870370 conn(0x7fb87c0788d0 msgr2=0x7fb87c100490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:29:15.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.229+0000 7fb8845be640 1 -- 172.21.15.40:0/2381870370 shutdown_connections 2024-04-21T23:29:15.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.229+0000 7fb8845be640 1 -- 172.21.15.40:0/2381870370 wait complete. 2024-04-21T23:29:15.239 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:29:15.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.342+0000 7efd73839640 1 -- 172.21.15.40:0/146374181 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd6c100620 msgr2=0x7efd6c100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:29:15.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.342+0000 7efd73839640 1 --2- 172.21.15.40:0/146374181 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd6c100620 0x7efd6c100aa0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7efd5c009930 tx=0x7efd5c02bc30 comp rx=0 tx=0).stop 2024-04-21T23:29:15.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.342+0000 7efd73839640 1 -- 172.21.15.40:0/146374181 shutdown_connections 2024-04-21T23:29:15.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.342+0000 7efd73839640 1 --2- 172.21.15.40:0/146374181 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efd6c100fe0 0x7efd6c10d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:15.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.342+0000 7efd73839640 1 --2- 172.21.15.40:0/146374181 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd6c100620 0x7efd6c100aa0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:15.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.342+0000 7efd73839640 1 --2- 172.21.15.40:0/146374181 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efd6c106630 0x7efd6c106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:15.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.342+0000 7efd73839640 1 -- 172.21.15.40:0/146374181 >> 172.21.15.40:0/146374181 conn(0x7efd6c077740 msgr2=0x7efd6c077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:29:15.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.342+0000 7efd73839640 1 -- 172.21.15.40:0/146374181 shutdown_connections 2024-04-21T23:29:15.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.342+0000 7efd73839640 1 -- 172.21.15.40:0/146374181 wait complete. 2024-04-21T23:29:15.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.343+0000 7efd73839640 1 Processor -- start 2024-04-21T23:29:15.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.343+0000 7efd73839640 1 -- start start 2024-04-21T23:29:15.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.343+0000 7efd73839640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd6c100620 0x7efd6c19a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:29:15.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.343+0000 7efd73839640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efd6c100fe0 0x7efd6c19a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:29:15.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.343+0000 7efd73839640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efd6c106630 0x7efd6c19ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:29:15.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.343+0000 7efd73839640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7efd6c110a40 con 0x7efd6c100620 2024-04-21T23:29:15.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.343+0000 7efd73839640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7efd6c1108c0 con 0x7efd6c100fe0 2024-04-21T23:29:15.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.343+0000 7efd73839640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7efd6c110bc0 con 0x7efd6c106630 2024-04-21T23:29:15.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.343+0000 7efd715d5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd6c100620 0x7efd6c19a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:29:15.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.343+0000 7efd715d5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd6c100620 0x7efd6c19a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52214/0 (socket says 172.21.15.40:52214) 2024-04-21T23:29:15.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.343+0000 7efd715d5640 1 -- 172.21.15.40:0/944750620 learned_addr learned my addr 172.21.15.40:0/944750620 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:29:15.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.343+0000 7efd71dd6640 1 --2- 172.21.15.40:0/944750620 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efd6c106630 0x7efd6c19ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:29:15.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.343+0000 7efd715d5640 1 -- 172.21.15.40:0/944750620 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efd6c106630 msgr2=0x7efd6c19ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:29:15.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.343+0000 7efd715d5640 1 --2- 172.21.15.40:0/944750620 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efd6c106630 0x7efd6c19ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:15.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.343+0000 7efd70dd4640 1 --2- 172.21.15.40:0/944750620 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efd6c100fe0 0x7efd6c19a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:29:15.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.343+0000 7efd715d5640 1 -- 172.21.15.40:0/944750620 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efd6c100fe0 msgr2=0x7efd6c19a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:29:15.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.343+0000 7efd715d5640 1 --2- 172.21.15.40:0/944750620 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efd6c100fe0 0x7efd6c19a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:15.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.343+0000 7efd715d5640 1 -- 172.21.15.40:0/944750620 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd6c19f190 con 0x7efd6c100620 2024-04-21T23:29:15.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.343+0000 7efd715d5640 1 --2- 172.21.15.40:0/944750620 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd6c100620 0x7efd6c19a1a0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7efd6800c890 tx=0x7efd6800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:29:15.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.344+0000 7efd627fc640 1 -- 172.21.15.40:0/944750620 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd68004220 con 0x7efd6c100620 2024-04-21T23:29:15.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.344+0000 7efd627fc640 1 -- 172.21.15.40:0/944750620 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd680043c0 con 0x7efd6c100620 2024-04-21T23:29:15.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.344+0000 7efd73839640 1 -- 172.21.15.40:0/944750620 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd6c19f480 con 0x7efd6c100620 2024-04-21T23:29:15.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.344+0000 7efd627fc640 1 -- 172.21.15.40:0/944750620 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd680049e0 con 0x7efd6c100620 2024-04-21T23:29:15.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.344+0000 7efd73839640 1 -- 172.21.15.40:0/944750620 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd6c1a6d10 con 0x7efd6c100620 2024-04-21T23:29:15.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.345+0000 7efd73839640 1 -- 172.21.15.40:0/944750620 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd6c068560 con 0x7efd6c100620 2024-04-21T23:29:15.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.346+0000 7efd627fc640 1 -- 172.21.15.40:0/944750620 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7efd680079b0 con 0x7efd6c100620 2024-04-21T23:29:15.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.346+0000 7efd627fc640 1 --2- 172.21.15.40:0/944750620 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7efd580a2d00 0x7efd580a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:29:15.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.347+0000 7efd70dd4640 1 --2- 172.21.15.40:0/944750620 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7efd580a2d00 0x7efd580a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:29:15.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.347+0000 7efd627fc640 1 -- 172.21.15.40:0/944750620 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9642+0+0 (secure 0 0 0) 0x7efd680cf8d0 con 0x7efd6c100620 2024-04-21T23:29:15.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.347+0000 7efd70dd4640 1 --2- 172.21.15.40:0/944750620 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7efd580a2d00 0x7efd580a51c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7efd5c036040 tx=0x7efd5c005fd0 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:29:15.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.351+0000 7efd627fc640 1 -- 172.21.15.40:0/944750620 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7efd68014030 con 0x7efd6c100620 2024-04-21T23:29:15.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.574+0000 7efd73839640 1 -- 172.21.15.40:0/944750620 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efd6c19b4f0 con 0x7efd6c100620 2024-04-21T23:29:15.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.575+0000 7efd627fc640 1 -- 172.21.15.40:0/944750620 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11882 (secure 0 0 0) 0x7efd6809c090 con 0x7efd6c100620 2024-04-21T23:29:15.574 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:29:15.574 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/1571948830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1571948830},{"type":"v1","addr":"172.21.15.40:6835","nonce":1571948830}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/969356642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":969356642},{"type":"v1","addr":"172.21.15.40:6837","nonce":969356642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6839/3025057845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":3025057845},{"type":"v1","addr":"172.21.15.40:6839","nonce":3025057845}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6841/3350488804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":3350488804},{"type":"v1","addr":"172.21.15.40:6841","nonce":3350488804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/3438493203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":3438493203},{"type":"v1","addr":"172.21.15.174:6833","nonce":3438493203}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/3105752678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":3105752678},{"type":"v1","addr":"172.21.15.111:6833","nonce":3105752678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T23:15:55.742233+0000","modified":"2024-04-21T23:23:05.004549+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24521,"mds_1":24502,"mds_2":24494},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24461":{"gid":24461,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6835/2704462092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2704462092},{"type":"v1","addr":"172.21.15.111:6835","nonce":2704462092}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/3905733045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3905733045},{"type":"v1","addr":"172.21.15.174:6835","nonce":3905733045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6837/465579746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":465579746},{"type":"v1","addr":"172.21.15.174:6837","nonce":465579746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":71,"addr":"172.21.15.111:6837/2703764182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":2703764182},{"type":"v1","addr":"172.21.15.111:6837","nonce":2703764182}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24502":{"gid":24502,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.174:6839/2467509274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":2467509274},{"type":"v1","addr":"172.21.15.174:6839","nonce":2467509274}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3712235548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3712235548},{"type":"v1","addr":"172.21.15.111:6839","nonce":3712235548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24521,"qdb_cluster":[24494,24502,24521]},"id":1}]} 2024-04-21T23:29:15.574 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 33 2024-04-21T23:29:15.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.576+0000 7efd73839640 1 -- 172.21.15.40:0/944750620 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7efd580a2d00 msgr2=0x7efd580a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:29:15.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.576+0000 7efd73839640 1 --2- 172.21.15.40:0/944750620 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7efd580a2d00 0x7efd580a51c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7efd5c036040 tx=0x7efd5c005fd0 comp rx=0 tx=0).stop 2024-04-21T23:29:15.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.576+0000 7efd73839640 1 -- 172.21.15.40:0/944750620 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd6c100620 msgr2=0x7efd6c19a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:29:15.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.576+0000 7efd73839640 1 --2- 172.21.15.40:0/944750620 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd6c100620 0x7efd6c19a1a0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7efd6800c890 tx=0x7efd6800cd50 comp rx=0 tx=0).stop 2024-04-21T23:29:15.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.577+0000 7efd73839640 1 -- 172.21.15.40:0/944750620 shutdown_connections 2024-04-21T23:29:15.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.577+0000 7efd73839640 1 --2- 172.21.15.40:0/944750620 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7efd580a2d00 0x7efd580a51c0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:15.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.577+0000 7efd73839640 1 --2- 172.21.15.40:0/944750620 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efd6c100fe0 0x7efd6c19a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:15.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.577+0000 7efd73839640 1 --2- 172.21.15.40:0/944750620 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efd6c106630 0x7efd6c19ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:15.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.577+0000 7efd73839640 1 --2- 172.21.15.40:0/944750620 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd6c100620 0x7efd6c19a1a0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:15.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.577+0000 7efd73839640 1 -- 172.21.15.40:0/944750620 >> 172.21.15.40:0/944750620 conn(0x7efd6c077740 msgr2=0x7efd6c0688d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:29:15.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.577+0000 7efd73839640 1 -- 172.21.15.40:0/944750620 shutdown_connections 2024-04-21T23:29:15.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.577+0000 7efd73839640 1 -- 172.21.15.40:0/944750620 wait complete. 2024-04-21T23:29:15.586 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T23:29:15.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.690+0000 7f8250408640 1 -- 172.21.15.40:0/1580982551 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f824810bd60 msgr2=0x7f824810e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:29:15.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.690+0000 7f8250408640 1 --2- 172.21.15.40:0/1580982551 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f824810bd60 0x7f824810e1d0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f8240009ca0 tx=0x7f824002c180 comp rx=0 tx=0).stop 2024-04-21T23:29:15.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.691+0000 7f8250408640 1 -- 172.21.15.40:0/1580982551 shutdown_connections 2024-04-21T23:29:15.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.691+0000 7f8250408640 1 --2- 172.21.15.40:0/1580982551 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8248103340 0x7f824810b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:15.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.691+0000 7f8250408640 1 --2- 172.21.15.40:0/1580982551 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f824810bd60 0x7f824810e1d0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:15.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.691+0000 7f8250408640 1 --2- 172.21.15.40:0/1580982551 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8248102a20 0x7f8248102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:15.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.691+0000 7f8250408640 1 -- 172.21.15.40:0/1580982551 >> 172.21.15.40:0/1580982551 conn(0x7f82480777a0 msgr2=0x7f8248077bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:29:15.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.691+0000 7f8250408640 1 -- 172.21.15.40:0/1580982551 shutdown_connections 2024-04-21T23:29:15.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.691+0000 7f8250408640 1 -- 172.21.15.40:0/1580982551 wait complete. 2024-04-21T23:29:15.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.691+0000 7f8250408640 1 Processor -- start 2024-04-21T23:29:15.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.691+0000 7f8250408640 1 -- start start 2024-04-21T23:29:15.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.691+0000 7f8250408640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8248102a20 0x7f824819a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:29:15.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.691+0000 7f8250408640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8248103340 0x7f824819a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:29:15.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.691+0000 7f8250408640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f824810bd60 0x7f824819ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:29:15.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.691+0000 7f8250408640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8248111b30 con 0x7f8248103340 2024-04-21T23:29:15.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.691+0000 7f8250408640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f82481119b0 con 0x7f8248102a20 2024-04-21T23:29:15.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.691+0000 7f8250408640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8248111cb0 con 0x7f824810bd60 2024-04-21T23:29:15.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.692+0000 7f824d9a3640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8248103340 0x7f824819a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:29:15.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.692+0000 7f824d9a3640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8248103340 0x7f824819a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52226/0 (socket says 172.21.15.40:52226) 2024-04-21T23:29:15.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.692+0000 7f824d9a3640 1 -- 172.21.15.40:0/3315378241 learned_addr learned my addr 172.21.15.40:0/3315378241 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:29:15.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.692+0000 7f824e1a4640 1 --2- 172.21.15.40:0/3315378241 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8248102a20 0x7f824819a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:29:15.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.692+0000 7f824e9a5640 1 --2- 172.21.15.40:0/3315378241 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f824810bd60 0x7f824819ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:29:15.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.692+0000 7f824d9a3640 1 -- 172.21.15.40:0/3315378241 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f824810bd60 msgr2=0x7f824819ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:29:15.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.692+0000 7f824d9a3640 1 --2- 172.21.15.40:0/3315378241 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f824810bd60 0x7f824819ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:15.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.692+0000 7f824d9a3640 1 -- 172.21.15.40:0/3315378241 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8248102a20 msgr2=0x7f824819a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:29:15.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.692+0000 7f824d9a3640 1 --2- 172.21.15.40:0/3315378241 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8248102a20 0x7f824819a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:15.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.692+0000 7f824d9a3640 1 -- 172.21.15.40:0/3315378241 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f824819f1c0 con 0x7f8248103340 2024-04-21T23:29:15.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.692+0000 7f824e9a5640 1 --2- 172.21.15.40:0/3315378241 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f824810bd60 0x7f824819ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:29:15.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.692+0000 7f824d9a3640 1 --2- 172.21.15.40:0/3315378241 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8248103340 0x7f824819a710 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f823800d950 tx=0x7f823800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:29:15.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.692+0000 7f823f7fe640 1 -- 172.21.15.40:0/3315378241 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8238017070 con 0x7f8248103340 2024-04-21T23:29:15.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.692+0000 7f8250408640 1 -- 172.21.15.40:0/3315378241 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f824819f4b0 con 0x7f8248103340 2024-04-21T23:29:15.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.692+0000 7f823f7fe640 1 -- 172.21.15.40:0/3315378241 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8238009c30 con 0x7f8248103340 2024-04-21T23:29:15.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.692+0000 7f823f7fe640 1 -- 172.21.15.40:0/3315378241 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f823800f6b0 con 0x7f8248103340 2024-04-21T23:29:15.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.692+0000 7f8250408640 1 -- 172.21.15.40:0/3315378241 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82481a6d40 con 0x7f8248103340 2024-04-21T23:29:15.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.693+0000 7f8250408640 1 -- 172.21.15.40:0/3315378241 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82480689b0 con 0x7f8248103340 2024-04-21T23:29:15.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.694+0000 7f823f7fe640 1 -- 172.21.15.40:0/3315378241 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f82380109d0 con 0x7f8248103340 2024-04-21T23:29:15.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.695+0000 7f823f7fe640 1 --2- 172.21.15.40:0/3315378241 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f82340a2d00 0x7f82340a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:29:15.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.695+0000 7f823f7fe640 1 -- 172.21.15.40:0/3315378241 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9642+0+0 (secure 0 0 0) 0x7f82380cf4c0 con 0x7f8248103340 2024-04-21T23:29:15.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.695+0000 7f824e1a4640 1 --2- 172.21.15.40:0/3315378241 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f82340a2d00 0x7f82340a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:29:15.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.696+0000 7f824e1a4640 1 --2- 172.21.15.40:0/3315378241 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f82340a2d00 0x7f82340a51c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f82440058c0 tx=0x7f824400a2d0 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:29:15.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.699+0000 7f823f7fe640 1 -- 172.21.15.40:0/3315378241 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f8238009de0 con 0x7f8248103340 2024-04-21T23:29:15.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.888+0000 7f8250408640 1 -- 172.21.15.40:0/3315378241 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f82481a7030 con 0x7f8248103340 2024-04-21T23:29:15.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.890+0000 7f823f7fe640 1 -- 172.21.15.40:0/3315378241 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v104) ==== 74+0+25552 (secure 0 0 0) 0x7f823809bc20 con 0x7f8248103340 2024-04-21T23:29:15.889 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-21T23:29:15.889 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":104,"fsid":"d814e4dc-0033-11ef-bc93-c7b262605968","created":"2024-04-21T23:07:52.659207+0000","modified":"2024-04-21T23:29:10.833904+0000","last_up_change":"2024-04-21T23:15:05.308821+0000","last_in_change":"2024-04-21T23:14:45.447491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:11:14.824775+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T23:15:54.322571+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T23:15:55.164493+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"104","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":104,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T23:15:58.838266+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"104","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":104,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"0e8d692a-22fe-4a87-8d99-9151f96cbb02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6803","nonce":2009758784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6805","nonce":2009758784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6809","nonce":2009758784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2009758784},{"type":"v1","addr":"172.21.15.40:6807","nonce":2009758784}]},"public_addr":"172.21.15.40:6803/2009758784","cluster_addr":"172.21.15.40:6805/2009758784","heartbeat_back_addr":"172.21.15.40:6809/2009758784","heartbeat_front_addr":"172.21.15.40:6807/2009758784","state":["exists","up"]},{"osd":1,"uuid":"fe5fe11b-0e84-4359-83a7-1ba45105755a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6801","nonce":1763903836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6803","nonce":1763903836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6807","nonce":1763903836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":1763903836},{"type":"v1","addr":"172.21.15.111:6805","nonce":1763903836}]},"public_addr":"172.21.15.111:6801/1763903836","cluster_addr":"172.21.15.111:6803/1763903836","heartbeat_back_addr":"172.21.15.111:6807/1763903836","heartbeat_front_addr":"172.21.15.111:6805/1763903836","state":["exists","up"]},{"osd":2,"uuid":"813ad735-5cc2-4da0-ae16-6db238395fbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6801","nonce":439695825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6803","nonce":439695825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6807","nonce":439695825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":439695825},{"type":"v1","addr":"172.21.15.174:6805","nonce":439695825}]},"public_addr":"172.21.15.174:6801/439695825","cluster_addr":"172.21.15.174:6803/439695825","heartbeat_back_addr":"172.21.15.174:6807/439695825","heartbeat_front_addr":"172.21.15.174:6805/439695825","state":["exists","up"]},{"osd":3,"uuid":"09becb03-b7ce-4ffc-917d-b7900ec37e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6811","nonce":3454312436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6813","nonce":3454312436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6817","nonce":3454312436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3454312436},{"type":"v1","addr":"172.21.15.40:6815","nonce":3454312436}]},"public_addr":"172.21.15.40:6811/3454312436","cluster_addr":"172.21.15.40:6813/3454312436","heartbeat_back_addr":"172.21.15.40:6817/3454312436","heartbeat_front_addr":"172.21.15.40:6815/3454312436","state":["exists","up"]},{"osd":4,"uuid":"46eabfd7-d88a-4efd-94eb-04354d3a9d7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6809","nonce":1810866070}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6811","nonce":1810866070}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6815","nonce":1810866070}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":1810866070},{"type":"v1","addr":"172.21.15.111:6813","nonce":1810866070}]},"public_addr":"172.21.15.111:6809/1810866070","cluster_addr":"172.21.15.111:6811/1810866070","heartbeat_back_addr":"172.21.15.111:6815/1810866070","heartbeat_front_addr":"172.21.15.111:6813/1810866070","state":["exists","up"]},{"osd":5,"uuid":"6f73b582-6c65-41db-aab0-9b360f83d5c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6809","nonce":3752986711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6811","nonce":3752986711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6815","nonce":3752986711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":3752986711},{"type":"v1","addr":"172.21.15.174:6813","nonce":3752986711}]},"public_addr":"172.21.15.174:6809/3752986711","cluster_addr":"172.21.15.174:6811/3752986711","heartbeat_back_addr":"172.21.15.174:6815/3752986711","heartbeat_front_addr":"172.21.15.174:6813/3752986711","state":["exists","up"]},{"osd":6,"uuid":"39f305f0-12d2-4732-aaa0-cd66700af538","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6819","nonce":1618855427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6821","nonce":1618855427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6825","nonce":1618855427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":1618855427},{"type":"v1","addr":"172.21.15.40:6823","nonce":1618855427}]},"public_addr":"172.21.15.40:6819/1618855427","cluster_addr":"172.21.15.40:6821/1618855427","heartbeat_back_addr":"172.21.15.40:6825/1618855427","heartbeat_front_addr":"172.21.15.40:6823/1618855427","state":["exists","up"]},{"osd":7,"uuid":"769f2468-5102-490b-89f8-d0444d3169a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6817","nonce":927054162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6819","nonce":927054162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6823","nonce":927054162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":927054162},{"type":"v1","addr":"172.21.15.111:6821","nonce":927054162}]},"public_addr":"172.21.15.111:6817/927054162","cluster_addr":"172.21.15.111:6819/927054162","heartbeat_back_addr":"172.21.15.111:6823/927054162","heartbeat_front_addr":"172.21.15.111:6821/927054162","state":["exists","up"]},{"osd":8,"uuid":"1a444e80-007d-473b-b972-4c814a813c22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6817","nonce":3817848746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6819","nonce":3817848746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6823","nonce":3817848746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":3817848746},{"type":"v1","addr":"172.21.15.174:6821","nonce":3817848746}]},"public_addr":"172.21.15.174:6817/3817848746","cluster_addr":"172.21.15.174:6819/3817848746","heartbeat_back_addr":"172.21.15.174:6823/3817848746","heartbeat_front_addr":"172.21.15.174:6821/3817848746","state":["exists","up"]},{"osd":9,"uuid":"9a6b6a56-6ae5-49fe-9e5b-cb17d686e0dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6827","nonce":1649446531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6829","nonce":1649446531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6833","nonce":1649446531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":1649446531},{"type":"v1","addr":"172.21.15.40:6831","nonce":1649446531}]},"public_addr":"172.21.15.40:6827/1649446531","cluster_addr":"172.21.15.40:6829/1649446531","heartbeat_back_addr":"172.21.15.40:6833/1649446531","heartbeat_front_addr":"172.21.15.40:6831/1649446531","state":["exists","up"]},{"osd":10,"uuid":"a70038ba-e2ad-4170-85cc-ea0470a2edef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6825","nonce":1227538935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6827","nonce":1227538935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6831","nonce":1227538935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":1227538935},{"type":"v1","addr":"172.21.15.111:6829","nonce":1227538935}]},"public_addr":"172.21.15.111:6825/1227538935","cluster_addr":"172.21.15.111:6827/1227538935","heartbeat_back_addr":"172.21.15.111:6831/1227538935","heartbeat_front_addr":"172.21.15.111:6829/1227538935","state":["exists","up"]},{"osd":11,"uuid":"781532bb-7a5e-44e3-8eb7-b584b0054666","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6825","nonce":2045865553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6827","nonce":2045865553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6831","nonce":2045865553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":2045865553},{"type":"v1","addr":"172.21.15.174:6829","nonce":2045865553}]},"public_addr":"172.21.15.174:6825/2045865553","cluster_addr":"172.21.15.174:6827/2045865553","heartbeat_back_addr":"172.21.15.174:6831/2045865553","heartbeat_front_addr":"172.21.15.174:6829/2045865553","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:10:24.761290+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:10:48.565428+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:11:12.307818+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:11:37.951529+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:02.832523+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:27.568879+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:12:52.997707+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:13:18.937274+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:13:44.708657+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:14:10.833855+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:14:37.078272+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:15:03.878076+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":1}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.2a","mappings":[{"from":3,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"4.9","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":10}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:6801/2500760788":"2024-04-22T23:18:23.389674+0000","172.21.15.40:6800/2500760788":"2024-04-22T23:18:23.389674+0000","172.21.15.40:0/2742002992":"2024-04-22T23:18:23.389674+0000","172.21.15.40:0/2494672676":"2024-04-22T23:18:23.389674+0000","172.21.15.40:6801/526554319":"2024-04-22T23:08:50.305770+0000","172.21.15.40:6800/526554319":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/3178763471":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/4045911364":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/276300080":"2024-04-22T23:08:50.305770+0000","172.21.15.40:0/3689649814":"2024-04-22T23:08:23.743030+0000","172.21.15.40:0/2241116715":"2024-04-22T23:08:23.743030+0000","172.21.15.40:0/1406242364":"2024-04-22T23:18:23.389674+0000","172.21.15.40:0/1737227818":"2024-04-22T23:18:23.389674+0000","172.21.15.40:6801/1479040326":"2024-04-22T23:08:23.743030+0000","172.21.15.40:6800/1479040326":"2024-04-22T23:08:23.743030+0000","172.21.15.40:0/681298099":"2024-04-22T23:08:23.743030+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]},{"pool":4,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"new_removed_snaps":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]},{"pool":4,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T23:29:15.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.892+0000 7f8250408640 1 -- 172.21.15.40:0/3315378241 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f82340a2d00 msgr2=0x7f82340a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:29:15.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.892+0000 7f8250408640 1 --2- 172.21.15.40:0/3315378241 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f82340a2d00 0x7f82340a51c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f82440058c0 tx=0x7f824400a2d0 comp rx=0 tx=0).stop 2024-04-21T23:29:15.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.892+0000 7f8250408640 1 -- 172.21.15.40:0/3315378241 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8248103340 msgr2=0x7f824819a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:29:15.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.892+0000 7f8250408640 1 --2- 172.21.15.40:0/3315378241 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8248103340 0x7f824819a710 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f823800d950 tx=0x7f823800de10 comp rx=0 tx=0).stop 2024-04-21T23:29:15.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.892+0000 7f8250408640 1 -- 172.21.15.40:0/3315378241 shutdown_connections 2024-04-21T23:29:15.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.892+0000 7f8250408640 1 --2- 172.21.15.40:0/3315378241 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f82340a2d00 0x7f82340a51c0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:15.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.892+0000 7f8250408640 1 --2- 172.21.15.40:0/3315378241 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f824810bd60 0x7f824819ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:15.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.892+0000 7f8250408640 1 --2- 172.21.15.40:0/3315378241 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8248103340 0x7f824819a710 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:15.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.892+0000 7f8250408640 1 --2- 172.21.15.40:0/3315378241 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8248102a20 0x7f824819a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:15.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.892+0000 7f8250408640 1 -- 172.21.15.40:0/3315378241 >> 172.21.15.40:0/3315378241 conn(0x7f82480777a0 msgr2=0x7f8248107c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:29:15.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.892+0000 7f8250408640 1 -- 172.21.15.40:0/3315378241 shutdown_connections 2024-04-21T23:29:15.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:15.892+0000 7f8250408640 1 -- 172.21.15.40:0/3315378241 wait complete. 2024-04-21T23:29:15.903 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-21T23:29:16.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.007+0000 7fc971d02640 1 -- 172.21.15.40:0/97901737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc96c101710 msgr2=0x7fc96c101b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:29:16.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.007+0000 7fc971d02640 1 --2- 172.21.15.40:0/97901737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc96c101710 0x7fc96c101b90 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fc960009930 tx=0x7fc96002bc30 comp rx=0 tx=0).stop 2024-04-21T23:29:16.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.007+0000 7fc971d02640 1 -- 172.21.15.40:0/97901737 shutdown_connections 2024-04-21T23:29:16.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.007+0000 7fc971d02640 1 --2- 172.21.15.40:0/97901737 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc96c1020d0 0x7fc96c10e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:16.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.007+0000 7fc971d02640 1 --2- 172.21.15.40:0/97901737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc96c101710 0x7fc96c101b90 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:16.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.007+0000 7fc971d02640 1 --2- 172.21.15.40:0/97901737 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc96c107720 0x7fc96c107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:16.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.007+0000 7fc971d02640 1 -- 172.21.15.40:0/97901737 >> 172.21.15.40:0/97901737 conn(0x7fc96c078b90 msgr2=0x7fc96c078fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:29:16.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.007+0000 7fc971d02640 1 -- 172.21.15.40:0/97901737 shutdown_connections 2024-04-21T23:29:16.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.007+0000 7fc971d02640 1 -- 172.21.15.40:0/97901737 wait complete. 2024-04-21T23:29:16.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.007+0000 7fc971d02640 1 Processor -- start 2024-04-21T23:29:16.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.008+0000 7fc971d02640 1 -- start start 2024-04-21T23:29:16.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.008+0000 7fc971d02640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc96c101710 0x7fc96c198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:29:16.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.008+0000 7fc971d02640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc96c1020d0 0x7fc96c1985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:29:16.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.008+0000 7fc971d02640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc96c107720 0x7fc96c19c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:29:16.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.008+0000 7fc971d02640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc96c111b40 con 0x7fc96c1020d0 2024-04-21T23:29:16.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.008+0000 7fc971d02640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc96c1119c0 con 0x7fc96c101710 2024-04-21T23:29:16.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.008+0000 7fc971d02640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc96c111cc0 con 0x7fc96c107720 2024-04-21T23:29:16.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.008+0000 7fc96affd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc96c1020d0 0x7fc96c1985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:29:16.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.008+0000 7fc96affd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc96c1020d0 0x7fc96c1985a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52244/0 (socket says 172.21.15.40:52244) 2024-04-21T23:29:16.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.008+0000 7fc96affd640 1 -- 172.21.15.40:0/1613429803 learned_addr learned my addr 172.21.15.40:0/1613429803 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:29:16.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.008+0000 7fc96bfff640 1 --2- 172.21.15.40:0/1613429803 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc96c107720 0x7fc96c19c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:29:16.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.008+0000 7fc96b7fe640 1 --2- 172.21.15.40:0/1613429803 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc96c101710 0x7fc96c198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:29:16.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.008+0000 7fc96affd640 1 -- 172.21.15.40:0/1613429803 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc96c107720 msgr2=0x7fc96c19c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:29:16.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.008+0000 7fc96affd640 1 --2- 172.21.15.40:0/1613429803 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc96c107720 0x7fc96c19c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:16.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.008+0000 7fc96affd640 1 -- 172.21.15.40:0/1613429803 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc96c101710 msgr2=0x7fc96c198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:29:16.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.008+0000 7fc96affd640 1 --2- 172.21.15.40:0/1613429803 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc96c101710 0x7fc96c198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:16.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.008+0000 7fc96affd640 1 -- 172.21.15.40:0/1613429803 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc96c19d050 con 0x7fc96c1020d0 2024-04-21T23:29:16.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.008+0000 7fc96affd640 1 --2- 172.21.15.40:0/1613429803 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc96c1020d0 0x7fc96c1985a0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fc960009900 tx=0x7fc960004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:29:16.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.009+0000 7fc968ff9640 1 -- 172.21.15.40:0/1613429803 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc960043020 con 0x7fc96c1020d0 2024-04-21T23:29:16.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.009+0000 7fc968ff9640 1 -- 172.21.15.40:0/1613429803 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc9600054e0 con 0x7fc96c1020d0 2024-04-21T23:29:16.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.009+0000 7fc971d02640 1 -- 172.21.15.40:0/1613429803 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc96c19d2e0 con 0x7fc96c1020d0 2024-04-21T23:29:16.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.009+0000 7fc968ff9640 1 -- 172.21.15.40:0/1613429803 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc960003d00 con 0x7fc96c1020d0 2024-04-21T23:29:16.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.009+0000 7fc971d02640 1 -- 172.21.15.40:0/1613429803 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc96c1a4bd0 con 0x7fc96c1020d0 2024-04-21T23:29:16.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.010+0000 7fc971d02640 1 -- 172.21.15.40:0/1613429803 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc96c0689b0 con 0x7fc96c1020d0 2024-04-21T23:29:16.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.011+0000 7fc968ff9640 1 -- 172.21.15.40:0/1613429803 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fc960046020 con 0x7fc96c1020d0 2024-04-21T23:29:16.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.011+0000 7fc968ff9640 1 --2- 172.21.15.40:0/1613429803 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fc9540a2d00 0x7fc9540a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:29:16.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.012+0000 7fc96b7fe640 1 --2- 172.21.15.40:0/1613429803 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fc9540a2d00 0x7fc9540a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:29:16.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.012+0000 7fc968ff9640 1 -- 172.21.15.40:0/1613429803 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9642+0+0 (secure 0 0 0) 0x7fc9600f1ca0 con 0x7fc96c1020d0 2024-04-21T23:29:16.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.012+0000 7fc96b7fe640 1 --2- 172.21.15.40:0/1613429803 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fc9540a2d00 0x7fc9540a51c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc95c0058c0 tx=0x7fc95c00a2d0 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:29:16.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.015+0000 7fc968ff9640 1 -- 172.21.15.40:0/1613429803 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fc96004d020 con 0x7fc96c1020d0 2024-04-21T23:29:16.181 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:29:15 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/2381870370' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:29:16.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:29:15 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/944750620' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:29:16.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:29:15 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/2381870370' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:29:16.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:29:15 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/944750620' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:29:16.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:29:15 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/2381870370' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:29:16.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:29:15 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/944750620' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:29:16.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.241+0000 7fc971d02640 1 -- 172.21.15.40:0/1613429803 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fc96c109cd0 con 0x7fc96c1020d0 2024-04-21T23:29:16.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.244+0000 7fc968ff9640 1 -- 172.21.15.40:0/1613429803 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v34) ==== 161+0+0 (secure 0 0 0) 0x7fc9600be570 con 0x7fc96c1020d0 2024-04-21T23:29:16.242 INFO:teuthology.orchestra.run.smithi040.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-21T23:29:16.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.246+0000 7fc971d02640 1 -- 172.21.15.40:0/1613429803 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fc9540a2d00 msgr2=0x7fc9540a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:29:16.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.246+0000 7fc971d02640 1 --2- 172.21.15.40:0/1613429803 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fc9540a2d00 0x7fc9540a51c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc95c0058c0 tx=0x7fc95c00a2d0 comp rx=0 tx=0).stop 2024-04-21T23:29:16.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.246+0000 7fc971d02640 1 -- 172.21.15.40:0/1613429803 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc96c1020d0 msgr2=0x7fc96c1985a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:29:16.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.246+0000 7fc971d02640 1 --2- 172.21.15.40:0/1613429803 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc96c1020d0 0x7fc96c1985a0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fc960009900 tx=0x7fc960004430 comp rx=0 tx=0).stop 2024-04-21T23:29:16.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.247+0000 7fc971d02640 1 -- 172.21.15.40:0/1613429803 shutdown_connections 2024-04-21T23:29:16.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.247+0000 7fc971d02640 1 --2- 172.21.15.40:0/1613429803 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7fc9540a2d00 0x7fc9540a51c0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:16.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.247+0000 7fc971d02640 1 --2- 172.21.15.40:0/1613429803 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc96c107720 0x7fc96c19c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:16.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.247+0000 7fc971d02640 1 --2- 172.21.15.40:0/1613429803 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc96c1020d0 0x7fc96c1985a0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:16.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.247+0000 7fc971d02640 1 --2- 172.21.15.40:0/1613429803 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc96c101710 0x7fc96c198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:16.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.247+0000 7fc971d02640 1 -- 172.21.15.40:0/1613429803 >> 172.21.15.40:0/1613429803 conn(0x7fc96c078b90 msgr2=0x7fc96c0ff0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:29:16.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.248+0000 7fc971d02640 1 -- 172.21.15.40:0/1613429803 shutdown_connections 2024-04-21T23:29:16.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.248+0000 7fc971d02640 1 -- 172.21.15.40:0/1613429803 wait complete. 2024-04-21T23:29:16.258 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-21T23:29:16.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.365+0000 7f64b1375640 1 -- 172.21.15.40:0/2016946914 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f64ac1055c0 msgr2=0x7f64ac1059a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:29:16.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.365+0000 7f64b1375640 1 --2- 172.21.15.40:0/2016946914 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f64ac1055c0 0x7f64ac1059a0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f6494009ca0 tx=0x7f649402c0f0 comp rx=0 tx=0).stop 2024-04-21T23:29:16.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.365+0000 7f64b1375640 1 -- 172.21.15.40:0/2016946914 shutdown_connections 2024-04-21T23:29:16.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.365+0000 7f64b1375640 1 --2- 172.21.15.40:0/2016946914 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64ac101250 0x7f64ac10d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:16.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.365+0000 7f64b1375640 1 --2- 172.21.15.40:0/2016946914 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f64ac100890 0x7f64ac100d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:16.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.365+0000 7f64b1375640 1 --2- 172.21.15.40:0/2016946914 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f64ac1055c0 0x7f64ac1059a0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:16.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.365+0000 7f64b1375640 1 -- 172.21.15.40:0/2016946914 >> 172.21.15.40:0/2016946914 conn(0x7f64ac077430 msgr2=0x7f64ac077840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:29:16.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.365+0000 7f64b1375640 1 -- 172.21.15.40:0/2016946914 shutdown_connections 2024-04-21T23:29:16.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.365+0000 7f64b1375640 1 -- 172.21.15.40:0/2016946914 wait complete. 2024-04-21T23:29:16.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.366+0000 7f64b1375640 1 Processor -- start 2024-04-21T23:29:16.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.366+0000 7f64b1375640 1 -- start start 2024-04-21T23:29:16.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.366+0000 7f64b1375640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f64ac100890 0x7f64ac1022f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:29:16.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.366+0000 7f64b1375640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64ac101250 0x7f64ac102830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:29:16.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.366+0000 7f64b1375640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f64ac1055c0 0x7f64ac102d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:29:16.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.366+0000 7f64b1375640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f64ac110ab0 con 0x7f64ac100890 2024-04-21T23:29:16.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.366+0000 7f64b1375640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f64ac110930 con 0x7f64ac101250 2024-04-21T23:29:16.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.366+0000 7f64b1375640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f64ac110c30 con 0x7f64ac1055c0 2024-04-21T23:29:16.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.366+0000 7f64aaffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f64ac100890 0x7f64ac1022f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:29:16.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.366+0000 7f64aaffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f64ac100890 0x7f64ac1022f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52262/0 (socket says 172.21.15.40:52262) 2024-04-21T23:29:16.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.366+0000 7f64aaffd640 1 -- 172.21.15.40:0/4151723345 learned_addr learned my addr 172.21.15.40:0/4151723345 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:29:16.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.366+0000 7f64aa7fc640 1 --2- 172.21.15.40:0/4151723345 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64ac101250 0x7f64ac102830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:29:16.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.367+0000 7f64aaffd640 1 -- 172.21.15.40:0/4151723345 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f64ac1055c0 msgr2=0x7f64ac102d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:29:16.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.367+0000 7f64aaffd640 1 --2- 172.21.15.40:0/4151723345 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f64ac1055c0 0x7f64ac102d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:16.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.367+0000 7f64aaffd640 1 -- 172.21.15.40:0/4151723345 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64ac101250 msgr2=0x7f64ac102830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:29:16.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.367+0000 7f64aaffd640 1 --2- 172.21.15.40:0/4151723345 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64ac101250 0x7f64ac102830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:16.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.367+0000 7f64aaffd640 1 -- 172.21.15.40:0/4151723345 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64ac1a6570 con 0x7f64ac100890 2024-04-21T23:29:16.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.367+0000 7f64aa7fc640 1 --2- 172.21.15.40:0/4151723345 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64ac101250 0x7f64ac102830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:29:16.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.367+0000 7f64aaffd640 1 --2- 172.21.15.40:0/4151723345 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f64ac100890 0x7f64ac1022f0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f6494005e30 tx=0x7f6494032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:29:16.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.367+0000 7f64a3fff640 1 -- 172.21.15.40:0/4151723345 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6494039040 con 0x7f64ac100890 2024-04-21T23:29:16.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.367+0000 7f64b1375640 1 -- 172.21.15.40:0/4151723345 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64ac1a6800 con 0x7f64ac100890 2024-04-21T23:29:16.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.367+0000 7f64a3fff640 1 -- 172.21.15.40:0/4151723345 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6494044070 con 0x7f64ac100890 2024-04-21T23:29:16.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.367+0000 7f64a3fff640 1 -- 172.21.15.40:0/4151723345 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64940042a0 con 0x7f64ac100890 2024-04-21T23:29:16.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.367+0000 7f64b1375640 1 -- 172.21.15.40:0/4151723345 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64ac1a6d40 con 0x7f64ac100890 2024-04-21T23:29:16.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.368+0000 7f64b1375640 1 -- 172.21.15.40:0/4151723345 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64ac101d30 con 0x7f64ac100890 2024-04-21T23:29:16.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.369+0000 7f64a3fff640 1 -- 172.21.15.40:0/4151723345 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f6494004440 con 0x7f64ac100890 2024-04-21T23:29:16.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.370+0000 7f64a3fff640 1 --2- 172.21.15.40:0/4151723345 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f64980a2d00 0x7f64980a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:29:16.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.370+0000 7f64aa7fc640 1 --2- 172.21.15.40:0/4151723345 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f64980a2d00 0x7f64980a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:29:16.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.370+0000 7f64a3fff640 1 -- 172.21.15.40:0/4151723345 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9812+0+0 (secure 0 0 0) 0x7f64940f1c60 con 0x7f64ac100890 2024-04-21T23:29:16.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.371+0000 7f64aa7fc640 1 --2- 172.21.15.40:0/4151723345 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f64980a2d00 0x7f64980a51c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f64ac103d80 tx=0x7f649c00b5f0 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:29:16.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.374+0000 7f64a3fff640 1 -- 172.21.15.40:0/4151723345 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f64940be480 con 0x7f64ac100890 2024-04-21T23:29:16.537 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:29:16 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mon-a[27980]: 2024-04-21T23:29:16.242+0000 7fd81babd700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-21T23:29:16.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:16.602+0000 7f64b1375640 1 -- 172.21.15.40:0/4151723345 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f64ac1a7000 con 0x7f64ac100890 2024-04-21T23:29:16.694 INFO:journalctl@ceph.mds.c.smithi174.stdout:Apr 21 23:29:16 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-c[61803]: ignoring --setuser ceph since I am not root 2024-04-21T23:29:16.694 INFO:journalctl@ceph.mds.c.smithi174.stdout:Apr 21 23:29:16 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-c[61803]: ignoring --setgroup ceph since I am not root 2024-04-21T23:29:16.694 INFO:journalctl@ceph.mds.c.smithi174.stdout:Apr 21 23:29:16 smithi174 ceph-mds[61807]: -- 172.21.15.174:0/2193889355 <== mon.1 v2:172.21.15.174:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55926bc41a40 con 0x55926bc15000 2024-04-21T23:29:16.694 INFO:journalctl@ceph.mds.c.smithi174.stdout:Apr 21 23:29:16 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-c[61803]: starting mds.c at 2024-04-21T23:29:16.695 INFO:journalctl@ceph.mds.i.smithi174.stdout:Apr 21 23:29:16 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-i[62111]: ignoring --setuser ceph since I am not root 2024-04-21T23:29:16.695 INFO:journalctl@ceph.mds.i.smithi174.stdout:Apr 21 23:29:16 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-i[62111]: ignoring --setgroup ceph since I am not root 2024-04-21T23:29:16.695 INFO:journalctl@ceph.mds.i.smithi174.stdout:Apr 21 23:29:16 smithi174 ceph-mds[62115]: -- 172.21.15.174:0/1261859262 <== mon.2 v2:172.21.15.111:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557130dd5a40 con 0x557130da8800 2024-04-21T23:29:16.695 INFO:journalctl@ceph.mds.i.smithi174.stdout:Apr 21 23:29:16 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-i[62111]: starting mds.i at 2024-04-21T23:29:16.695 INFO:journalctl@ceph.mds.f.smithi174.stdout:Apr 21 23:29:16 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-f[61481]: ignoring --setuser ceph since I am not root 2024-04-21T23:29:16.696 INFO:journalctl@ceph.mds.f.smithi174.stdout:Apr 21 23:29:16 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-f[61481]: ignoring --setgroup ceph since I am not root 2024-04-21T23:29:16.696 INFO:journalctl@ceph.mds.f.smithi174.stdout:Apr 21 23:29:16 smithi174 ceph-mds[61485]: -- 172.21.15.174:0/5486908 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55aadca844e0 con 0x55aadca5ac00 2024-04-21T23:29:16.696 INFO:journalctl@ceph.mds.f.smithi174.stdout:Apr 21 23:29:16 smithi174 ceph-mds[61485]: -- 172.21.15.174:0/5486908 <== mon.0 v2:172.21.15.40:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55aadca87a40 con 0x55aadca5ac00 2024-04-21T23:29:16.696 INFO:journalctl@ceph.mds.f.smithi174.stdout:Apr 21 23:29:16 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-f[61481]: starting mds.f at 2024-04-21T23:29:16.697 INFO:journalctl@ceph.mds.e.smithi111.stdout:Apr 21 23:29:16 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-e[60799]: ignoring --setuser ceph since I am not root 2024-04-21T23:29:16.697 INFO:journalctl@ceph.mds.e.smithi111.stdout:Apr 21 23:29:16 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-e[60799]: ignoring --setgroup ceph since I am not root 2024-04-21T23:29:16.697 INFO:journalctl@ceph.mds.e.smithi111.stdout:Apr 21 23:29:16 smithi111 ceph-mds[60803]: -- 172.21.15.111:0/672667755 <== mon.2 v2:172.21.15.111:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55adf1f6da40 con 0x55adf1f41000 2024-04-21T23:29:16.697 INFO:journalctl@ceph.mds.e.smithi111.stdout:Apr 21 23:29:16 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-e[60799]: starting mds.e at 2024-04-21T23:29:16.697 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:29:16 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: ignoring --setuser ceph since I am not root 2024-04-21T23:29:16.697 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:29:16 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: ignoring --setgroup ceph since I am not root 2024-04-21T23:29:16.698 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:29:16 smithi111 ceph-mds[61113]: -- 172.21.15.111:0/2795881712 <== mon.0 v2:172.21.15.40:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555d75241a40 con 0x555d75214800 2024-04-21T23:29:16.698 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:29:16 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: starting mds.h at 2024-04-21T23:29:16.698 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 21 23:29:16 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-b[61419]: ignoring --setuser ceph since I am not root 2024-04-21T23:29:16.698 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 21 23:29:16 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-b[61419]: ignoring --setgroup ceph since I am not root 2024-04-21T23:29:16.698 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 21 23:29:16 smithi111 ceph-mds[61423]: -- 172.21.15.111:0/1815788729 <== mon.1 v2:172.21.15.174:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5578b6639a40 con 0x5578b660cc00 2024-04-21T23:29:16.698 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 21 23:29:16 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-b[61419]: starting mds.b at 2024-04-21T23:29:17.181 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:29:16 smithi040 ceph-mon[27984]: pgmap v338: 129 pgs: 129 active+clean; 95 MiB data, 805 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.1 KiB/s wr, 7 op/s 2024-04-21T23:29:17.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:29:16 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/3315378241' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:29:17.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:29:16 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/1613429803' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-21T23:29:17.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:29:16 smithi040 ceph-mon[27984]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-21T23:29:17.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:29:16 smithi040 ceph-mon[27984]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-21T23:29:17.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:29:16 smithi040 ceph-mon[27984]: Health check cleared: MDS_DAMAGE (was: 1 MDSs report damaged metadata) 2024-04-21T23:29:17.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:29:16 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/1613429803' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-21T23:29:17.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:29:16 smithi040 ceph-mon[27984]: osdmap e105: 12 total, 12 up, 12 in 2024-04-21T23:29:17.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:29:16 smithi040 ceph-mon[27984]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-21T23:29:17.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:29:16 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/4151723345' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-21T23:29:17.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:29:16 smithi174 ceph-mon[31409]: pgmap v338: 129 pgs: 129 active+clean; 95 MiB data, 805 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.1 KiB/s wr, 7 op/s 2024-04-21T23:29:17.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:29:16 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/3315378241' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:29:17.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:29:16 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/1613429803' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-21T23:29:17.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:29:16 smithi174 ceph-mon[31409]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-21T23:29:17.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:29:16 smithi174 ceph-mon[31409]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-21T23:29:17.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:29:16 smithi174 ceph-mon[31409]: Health check cleared: MDS_DAMAGE (was: 1 MDSs report damaged metadata) 2024-04-21T23:29:17.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:29:16 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/1613429803' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-21T23:29:17.195 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:29:16 smithi174 ceph-mon[31409]: osdmap e105: 12 total, 12 up, 12 in 2024-04-21T23:29:17.195 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:29:16 smithi174 ceph-mon[31409]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-21T23:29:17.195 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:29:16 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/4151723345' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-21T23:29:17.195 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:29:16 smithi111 ceph-mon[31085]: pgmap v338: 129 pgs: 129 active+clean; 95 MiB data, 805 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.1 KiB/s wr, 7 op/s 2024-04-21T23:29:17.196 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:29:16 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/3315378241' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:29:17.196 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:29:16 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/1613429803' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-21T23:29:17.196 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:29:16 smithi111 ceph-mon[31085]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-21T23:29:17.196 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:29:16 smithi111 ceph-mon[31085]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-21T23:29:17.196 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:29:16 smithi111 ceph-mon[31085]: Health check cleared: MDS_DAMAGE (was: 1 MDSs report damaged metadata) 2024-04-21T23:29:17.196 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:29:16 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/1613429803' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-21T23:29:17.196 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:29:16 smithi111 ceph-mon[31085]: osdmap e105: 12 total, 12 up, 12 in 2024-04-21T23:29:17.196 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:29:16 smithi111 ceph-mon[31085]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-21T23:29:17.197 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:29:16 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/4151723345' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-21T23:29:17.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:17.245+0000 7f64a3fff640 1 -- 172.21.15.40:0/4151723345 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v35) ==== 104+0+0 (secure 0 0 0) 0x7f64940bdd80 con 0x7f64ac100890 2024-04-21T23:29:17.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:17.247+0000 7f64b1375640 1 -- 172.21.15.40:0/4151723345 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f64980a2d00 msgr2=0x7f64980a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:29:17.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:17.247+0000 7f64b1375640 1 --2- 172.21.15.40:0/4151723345 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f64980a2d00 0x7f64980a51c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f64ac103d80 tx=0x7f649c00b5f0 comp rx=0 tx=0).stop 2024-04-21T23:29:17.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:17.247+0000 7f64b1375640 1 -- 172.21.15.40:0/4151723345 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f64ac100890 msgr2=0x7f64ac1022f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:29:17.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:17.247+0000 7f64b1375640 1 --2- 172.21.15.40:0/4151723345 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f64ac100890 0x7f64ac1022f0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f6494005e30 tx=0x7f6494032f20 comp rx=0 tx=0).stop 2024-04-21T23:29:17.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:17.247+0000 7f64b1375640 1 -- 172.21.15.40:0/4151723345 shutdown_connections 2024-04-21T23:29:17.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:17.247+0000 7f64b1375640 1 --2- 172.21.15.40:0/4151723345 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f64980a2d00 0x7f64980a51c0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:17.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:17.247+0000 7f64b1375640 1 --2- 172.21.15.40:0/4151723345 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64ac101250 0x7f64ac102830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:17.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:17.247+0000 7f64b1375640 1 --2- 172.21.15.40:0/4151723345 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f64ac1055c0 0x7f64ac102d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:17.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:17.247+0000 7f64b1375640 1 --2- 172.21.15.40:0/4151723345 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f64ac100890 0x7f64ac1022f0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:17.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:17.247+0000 7f64b1375640 1 -- 172.21.15.40:0/4151723345 >> 172.21.15.40:0/4151723345 conn(0x7f64ac077430 msgr2=0x7f64ac1044d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:29:17.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:17.247+0000 7f64b1375640 1 -- 172.21.15.40:0/4151723345 shutdown_connections 2024-04-21T23:29:17.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:17.248+0000 7f64b1375640 1 -- 172.21.15.40:0/4151723345 wait complete. 2024-04-21T23:29:17.259 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-21T23:29:17.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:17.367+0000 7f106835b640 1 -- 172.21.15.40:0/2397202809 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10601029c0 msgr2=0x7f1060102da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:29:17.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:17.367+0000 7f106835b640 1 --2- 172.21.15.40:0/2397202809 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10601029c0 0x7f1060102da0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f1050009900 tx=0x7f105002bd60 comp rx=0 tx=0).stop 2024-04-21T23:29:17.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:17.367+0000 7f106835b640 1 -- 172.21.15.40:0/2397202809 shutdown_connections 2024-04-21T23:29:17.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:17.367+0000 7f106835b640 1 --2- 172.21.15.40:0/2397202809 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10601032e0 0x7f106010b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:17.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:17.367+0000 7f106835b640 1 --2- 172.21.15.40:0/2397202809 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f106010bd00 0x7f106010e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:17.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:17.367+0000 7f106835b640 1 --2- 172.21.15.40:0/2397202809 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10601029c0 0x7f1060102da0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:17.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:17.367+0000 7f106835b640 1 -- 172.21.15.40:0/2397202809 >> 172.21.15.40:0/2397202809 conn(0x7f1060077740 msgr2=0x7f1060077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:29:17.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:17.367+0000 7f106835b640 1 -- 172.21.15.40:0/2397202809 shutdown_connections 2024-04-21T23:29:17.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:17.367+0000 7f106835b640 1 -- 172.21.15.40:0/2397202809 wait complete. 2024-04-21T23:29:17.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:17.367+0000 7f106835b640 1 Processor -- start 2024-04-21T23:29:17.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:17.367+0000 7f106835b640 1 -- start start 2024-04-21T23:29:17.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:17.368+0000 7f106835b640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10601029c0 0x7f1060197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:29:17.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:17.368+0000 7f106835b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10601032e0 0x7f1060198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:29:17.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:17.368+0000 7f106835b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f106010bd00 0x7f106019c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:29:17.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:17.368+0000 7f106835b640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1060111b90 con 0x7f10601032e0 2024-04-21T23:29:17.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:17.368+0000 7f106835b640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f1060111a10 con 0x7f10601029c0 2024-04-21T23:29:17.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:17.368+0000 7f106835b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1060111d10 con 0x7f106010bd00 2024-04-21T23:29:17.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:17.368+0000 7f10658f6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10601032e0 0x7f1060198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:29:17.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:17.368+0000 7f10658f6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10601032e0 0x7f1060198510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52288/0 (socket says 172.21.15.40:52288) 2024-04-21T23:29:17.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:17.368+0000 7f10658f6640 1 -- 172.21.15.40:0/1837012047 learned_addr learned my addr 172.21.15.40:0/1837012047 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:29:17.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:17.368+0000 7f10660f7640 1 --2- 172.21.15.40:0/1837012047 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10601029c0 0x7f1060197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:29:17.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:17.368+0000 7f10668f8640 1 --2- 172.21.15.40:0/1837012047 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f106010bd00 0x7f106019c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:29:17.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:17.368+0000 7f10658f6640 1 -- 172.21.15.40:0/1837012047 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f106010bd00 msgr2=0x7f106019c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:29:17.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:17.368+0000 7f10658f6640 1 --2- 172.21.15.40:0/1837012047 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f106010bd00 0x7f106019c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:17.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:17.368+0000 7f10658f6640 1 -- 172.21.15.40:0/1837012047 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10601029c0 msgr2=0x7f1060197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:29:17.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:17.368+0000 7f10658f6640 1 --2- 172.21.15.40:0/1837012047 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10601029c0 0x7f1060197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:17.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:17.368+0000 7f10658f6640 1 -- 172.21.15.40:0/1837012047 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f106019cfc0 con 0x7f10601032e0 2024-04-21T23:29:17.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:17.368+0000 7f10668f8640 1 --2- 172.21.15.40:0/1837012047 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f106010bd00 0x7f106019c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:29:17.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:17.368+0000 7f10660f7640 1 --2- 172.21.15.40:0/1837012047 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10601029c0 0x7f1060197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:29:17.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:17.368+0000 7f10658f6640 1 --2- 172.21.15.40:0/1837012047 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10601032e0 0x7f1060198510 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f105400c920 tx=0x7f105400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:29:17.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:17.368+0000 7f105f7fe640 1 -- 172.21.15.40:0/1837012047 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1054007c70 con 0x7f10601032e0 2024-04-21T23:29:17.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:17.368+0000 7f105f7fe640 1 -- 172.21.15.40:0/1837012047 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1054007e10 con 0x7f10601032e0 2024-04-21T23:29:17.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:17.368+0000 7f106835b640 1 -- 172.21.15.40:0/1837012047 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f106019d2b0 con 0x7f10601032e0 2024-04-21T23:29:17.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:17.368+0000 7f105f7fe640 1 -- 172.21.15.40:0/1837012047 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1054018890 con 0x7f10601032e0 2024-04-21T23:29:17.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:17.369+0000 7f106835b640 1 -- 172.21.15.40:0/1837012047 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10601a4b40 con 0x7f10601032e0 2024-04-21T23:29:17.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:17.370+0000 7f106835b640 1 -- 172.21.15.40:0/1837012047 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1060068560 con 0x7f10601032e0 2024-04-21T23:29:17.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:17.371+0000 7f105f7fe640 1 -- 172.21.15.40:0/1837012047 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f1054018a70 con 0x7f10601032e0 2024-04-21T23:29:17.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:17.372+0000 7f105f7fe640 1 --2- 172.21.15.40:0/1837012047 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f10480a2d00 0x7f10480a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:29:17.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:17.372+0000 7f10660f7640 1 --2- 172.21.15.40:0/1837012047 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f10480a2d00 0x7f10480a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:29:17.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:17.373+0000 7f105f7fe640 1 -- 172.21.15.40:0/1837012047 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9812+0+0 (secure 0 0 0) 0x7f10540cfb60 con 0x7f10601032e0 2024-04-21T23:29:17.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:17.373+0000 7f10660f7640 1 --2- 172.21.15.40:0/1837012047 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f10480a2d00 0x7f10480a51c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f1050006010 tx=0x7f1050005a20 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:29:17.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:17.377+0000 7f105f7fe640 1 -- 172.21.15.40:0/1837012047 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f1054014030 con 0x7f10601032e0 2024-04-21T23:29:17.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:17.562+0000 7f106835b640 1 -- 172.21.15.40:0/1837012047 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f10601a4e40 con 0x7f10601032e0 2024-04-21T23:29:18.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:18.027+0000 7f105f7fe640 1 -- 172.21.15.40:0/1837012047 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v106) ==== 181+0+0 (secure 0 0 0) 0x7f105400de20 con 0x7f10601032e0 2024-04-21T23:29:18.026 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_metadata' removed 2024-04-21T23:29:18.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:18.032+0000 7f106835b640 1 -- 172.21.15.40:0/1837012047 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f10480a2d00 msgr2=0x7f10480a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:29:18.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:18.032+0000 7f106835b640 1 --2- 172.21.15.40:0/1837012047 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f10480a2d00 0x7f10480a51c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f1050006010 tx=0x7f1050005a20 comp rx=0 tx=0).stop 2024-04-21T23:29:18.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:18.032+0000 7f106835b640 1 -- 172.21.15.40:0/1837012047 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10601032e0 msgr2=0x7f1060198510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:29:18.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:18.032+0000 7f106835b640 1 --2- 172.21.15.40:0/1837012047 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10601032e0 0x7f1060198510 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f105400c920 tx=0x7f105400cde0 comp rx=0 tx=0).stop 2024-04-21T23:29:18.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:18.032+0000 7f106835b640 1 -- 172.21.15.40:0/1837012047 shutdown_connections 2024-04-21T23:29:18.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:18.032+0000 7f106835b640 1 --2- 172.21.15.40:0/1837012047 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f10480a2d00 0x7f10480a51c0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:18.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:18.032+0000 7f106835b640 1 --2- 172.21.15.40:0/1837012047 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f106010bd00 0x7f106019c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:18.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:18.032+0000 7f106835b640 1 --2- 172.21.15.40:0/1837012047 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10601032e0 0x7f1060198510 secure :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f105400c920 tx=0x7f105400cde0 comp rx=0 tx=0).stop 2024-04-21T23:29:18.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:18.032+0000 7f106835b640 1 --2- 172.21.15.40:0/1837012047 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10601029c0 0x7f1060197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:18.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:18.032+0000 7f106835b640 1 -- 172.21.15.40:0/1837012047 >> 172.21.15.40:0/1837012047 conn(0x7f1060077740 msgr2=0x7f10600fd6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:29:18.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:18.033+0000 7f106835b640 1 -- 172.21.15.40:0/1837012047 shutdown_connections 2024-04-21T23:29:18.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:18.033+0000 7f106835b640 1 -- 172.21.15.40:0/1837012047 wait complete. 2024-04-21T23:29:18.045 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-21T23:29:18.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:18.157+0000 7f32429fe640 1 -- 172.21.15.40:0/3185463869 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f323c10bd00 msgr2=0x7f323c10e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:29:18.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:18.157+0000 7f32429fe640 1 --2- 172.21.15.40:0/3185463869 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f323c10bd00 0x7f323c10e1a0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f3234009ca0 tx=0x7f323402c230 comp rx=0 tx=0).stop 2024-04-21T23:29:18.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:18.157+0000 7f32429fe640 1 -- 172.21.15.40:0/3185463869 shutdown_connections 2024-04-21T23:29:18.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:18.157+0000 7f32429fe640 1 --2- 172.21.15.40:0/3185463869 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f323c10bd00 0x7f323c10e1a0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:18.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:18.157+0000 7f32429fe640 1 --2- 172.21.15.40:0/3185463869 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f323c1032e0 0x7f323c10b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:18.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:18.157+0000 7f32429fe640 1 --2- 172.21.15.40:0/3185463869 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f323c1029c0 0x7f323c102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:18.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:18.157+0000 7f32429fe640 1 -- 172.21.15.40:0/3185463869 >> 172.21.15.40:0/3185463869 conn(0x7f323c077740 msgr2=0x7f323c077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:29:18.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:18.157+0000 7f32429fe640 1 -- 172.21.15.40:0/3185463869 shutdown_connections 2024-04-21T23:29:18.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:18.157+0000 7f32429fe640 1 -- 172.21.15.40:0/3185463869 wait complete. 2024-04-21T23:29:18.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:18.157+0000 7f32429fe640 1 Processor -- start 2024-04-21T23:29:18.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:18.158+0000 7f32429fe640 1 -- start start 2024-04-21T23:29:18.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:18.158+0000 7f32429fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f323c1029c0 0x7f323c198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:29:18.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:18.158+0000 7f32429fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f323c1032e0 0x7f323c198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:29:18.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:18.158+0000 7f32429fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f323c10bd00 0x7f323c19c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:29:18.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:18.158+0000 7f32429fe640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f323c111b00 con 0x7f323c1032e0 2024-04-21T23:29:18.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:18.158+0000 7f32429fe640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f323c111980 con 0x7f323c1029c0 2024-04-21T23:29:18.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:18.158+0000 7f32429fe640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f323c111c80 con 0x7f323c10bd00 2024-04-21T23:29:18.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:18.158+0000 7f323b7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f323c1032e0 0x7f323c198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:29:18.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:18.158+0000 7f323b7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f323c1032e0 0x7f323c198540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52296/0 (socket says 172.21.15.40:52296) 2024-04-21T23:29:18.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:18.158+0000 7f323b7fe640 1 -- 172.21.15.40:0/1069248446 learned_addr learned my addr 172.21.15.40:0/1069248446 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:29:18.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:18.158+0000 7f3240f9b640 1 --2- 172.21.15.40:0/1069248446 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f323c10bd00 0x7f323c19c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:29:18.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:18.158+0000 7f323bfff640 1 --2- 172.21.15.40:0/1069248446 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f323c1029c0 0x7f323c198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:29:18.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:18.158+0000 7f323b7fe640 1 -- 172.21.15.40:0/1069248446 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f323c10bd00 msgr2=0x7f323c19c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:29:18.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:18.158+0000 7f323b7fe640 1 --2- 172.21.15.40:0/1069248446 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f323c10bd00 0x7f323c19c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:18.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:18.158+0000 7f323b7fe640 1 -- 172.21.15.40:0/1069248446 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f323c1029c0 msgr2=0x7f323c198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:29:18.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:18.158+0000 7f323b7fe640 1 --2- 172.21.15.40:0/1069248446 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f323c1029c0 0x7f323c198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:18.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:18.158+0000 7f323b7fe640 1 -- 172.21.15.40:0/1069248446 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f323c19cfa0 con 0x7f323c1032e0 2024-04-21T23:29:18.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:18.158+0000 7f323bfff640 1 --2- 172.21.15.40:0/1069248446 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f323c1029c0 0x7f323c198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:29:18.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:18.159+0000 7f323b7fe640 1 --2- 172.21.15.40:0/1069248446 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f323c1032e0 0x7f323c198540 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f323000d770 tx=0x7f323000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:29:18.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:18.159+0000 7f32397fa640 1 -- 172.21.15.40:0/1069248446 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3230017070 con 0x7f323c1032e0 2024-04-21T23:29:18.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:18.159+0000 7f32429fe640 1 -- 172.21.15.40:0/1069248446 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f323c19d290 con 0x7f323c1032e0 2024-04-21T23:29:18.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:18.159+0000 7f32397fa640 1 -- 172.21.15.40:0/1069248446 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3230005500 con 0x7f323c1032e0 2024-04-21T23:29:18.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:18.159+0000 7f32397fa640 1 -- 172.21.15.40:0/1069248446 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3230004e00 con 0x7f323c1032e0 2024-04-21T23:29:18.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:18.159+0000 7f32429fe640 1 -- 172.21.15.40:0/1069248446 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f323c1a4b20 con 0x7f323c1032e0 2024-04-21T23:29:18.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:18.160+0000 7f32429fe640 1 -- 172.21.15.40:0/1069248446 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f323c068560 con 0x7f323c1032e0 2024-04-21T23:29:18.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:18.161+0000 7f32397fa640 1 -- 172.21.15.40:0/1069248446 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f3230004fa0 con 0x7f323c1032e0 2024-04-21T23:29:18.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:18.161+0000 7f32397fa640 1 --2- 172.21.15.40:0/1069248446 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f32240a2d00 0x7f32240a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:29:18.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:18.162+0000 7f323bfff640 1 --2- 172.21.15.40:0/1069248446 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f32240a2d00 0x7f32240a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:29:18.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:18.162+0000 7f32397fa640 1 -- 172.21.15.40:0/1069248446 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9258+0+0 (secure 0 0 0) 0x7f32300cf1b0 con 0x7f323c1032e0 2024-04-21T23:29:18.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:18.162+0000 7f323bfff640 1 --2- 172.21.15.40:0/1069248446 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f32240a2d00 0x7f32240a51c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f322c0058c0 tx=0x7f322c005d30 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:29:18.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:18.166+0000 7f32397fa640 1 -- 172.21.15.40:0/1069248446 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f323009baf0 con 0x7f323c1032e0 2024-04-21T23:29:18.181 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:29:17 smithi040 ceph-mon[27984]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-21T23:29:18.181 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:29:17 smithi040 ceph-mon[27984]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-21T23:29:18.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:29:17 smithi040 ceph-mon[27984]: Cluster is now healthy 2024-04-21T23:29:18.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:29:17 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/4151723345' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-21T23:29:18.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:29:17 smithi040 ceph-mon[27984]: mds.? [v2:172.21.15.174:6834/3553467784,v1:172.21.15.174:6837/3553467784] up:boot 2024-04-21T23:29:18.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:29:17 smithi040 ceph-mon[27984]: mds.? [v2:172.21.15.174:6836/3687414558,v1:172.21.15.174:6839/3687414558] up:boot 2024-04-21T23:29:18.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:29:17 smithi040 ceph-mon[27984]: mds.? [v2:172.21.15.174:6835/831022406,v1:172.21.15.174:6838/831022406] up:boot 2024-04-21T23:29:18.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:29:17 smithi040 ceph-mon[27984]: mds.? [v2:172.21.15.111:6834/2128970958,v1:172.21.15.111:6836/2128970958] up:boot 2024-04-21T23:29:18.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:29:17 smithi040 ceph-mon[27984]: mds.? [v2:172.21.15.111:6835/890653121,v1:172.21.15.111:6837/890653121] up:boot 2024-04-21T23:29:18.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:29:17 smithi040 ceph-mon[27984]: mds.? [v2:172.21.15.111:6838/223500367,v1:172.21.15.111:6839/223500367] up:boot 2024-04-21T23:29:18.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:29:17 smithi040 ceph-mon[27984]: fsmap 12 up:standby 2024-04-21T23:29:18.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:29:17 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-21T23:29:18.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:29:17 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-21T23:29:18.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:29:17 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-21T23:29:18.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:29:17 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-21T23:29:18.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:29:17 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-21T23:29:18.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:29:17 smithi040 ceph-mon[27984]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-21T23:29:18.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:29:17 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/1837012047' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-21T23:29:18.193 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:29:17 smithi174 ceph-mon[31409]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-21T23:29:18.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:29:17 smithi174 ceph-mon[31409]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-21T23:29:18.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:29:17 smithi174 ceph-mon[31409]: Cluster is now healthy 2024-04-21T23:29:18.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:29:17 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/4151723345' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-21T23:29:18.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:29:17 smithi174 ceph-mon[31409]: mds.? [v2:172.21.15.174:6834/3553467784,v1:172.21.15.174:6837/3553467784] up:boot 2024-04-21T23:29:18.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:29:17 smithi174 ceph-mon[31409]: mds.? [v2:172.21.15.174:6836/3687414558,v1:172.21.15.174:6839/3687414558] up:boot 2024-04-21T23:29:18.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:29:17 smithi174 ceph-mon[31409]: mds.? [v2:172.21.15.174:6835/831022406,v1:172.21.15.174:6838/831022406] up:boot 2024-04-21T23:29:18.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:29:17 smithi174 ceph-mon[31409]: mds.? [v2:172.21.15.111:6834/2128970958,v1:172.21.15.111:6836/2128970958] up:boot 2024-04-21T23:29:18.194 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:29:17 smithi174 ceph-mon[31409]: mds.? [v2:172.21.15.111:6835/890653121,v1:172.21.15.111:6837/890653121] up:boot 2024-04-21T23:29:18.195 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:29:17 smithi174 ceph-mon[31409]: mds.? [v2:172.21.15.111:6838/223500367,v1:172.21.15.111:6839/223500367] up:boot 2024-04-21T23:29:18.195 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:29:17 smithi174 ceph-mon[31409]: fsmap 12 up:standby 2024-04-21T23:29:18.195 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:29:17 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-21T23:29:18.195 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:29:17 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-21T23:29:18.195 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:29:17 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-21T23:29:18.195 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:29:17 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-21T23:29:18.195 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:29:17 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-21T23:29:18.195 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:29:17 smithi174 ceph-mon[31409]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-21T23:29:18.196 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:29:17 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/1837012047' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-21T23:29:18.196 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:29:17 smithi111 ceph-mon[31085]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-21T23:29:18.196 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:29:17 smithi111 ceph-mon[31085]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-21T23:29:18.196 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:29:17 smithi111 ceph-mon[31085]: Cluster is now healthy 2024-04-21T23:29:18.196 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:29:17 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/4151723345' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-21T23:29:18.196 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:29:17 smithi111 ceph-mon[31085]: mds.? [v2:172.21.15.174:6834/3553467784,v1:172.21.15.174:6837/3553467784] up:boot 2024-04-21T23:29:18.197 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:29:17 smithi111 ceph-mon[31085]: mds.? [v2:172.21.15.174:6836/3687414558,v1:172.21.15.174:6839/3687414558] up:boot 2024-04-21T23:29:18.197 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:29:17 smithi111 ceph-mon[31085]: mds.? [v2:172.21.15.174:6835/831022406,v1:172.21.15.174:6838/831022406] up:boot 2024-04-21T23:29:18.197 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:29:17 smithi111 ceph-mon[31085]: mds.? [v2:172.21.15.111:6834/2128970958,v1:172.21.15.111:6836/2128970958] up:boot 2024-04-21T23:29:18.197 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:29:17 smithi111 ceph-mon[31085]: mds.? [v2:172.21.15.111:6835/890653121,v1:172.21.15.111:6837/890653121] up:boot 2024-04-21T23:29:18.197 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:29:17 smithi111 ceph-mon[31085]: mds.? [v2:172.21.15.111:6838/223500367,v1:172.21.15.111:6839/223500367] up:boot 2024-04-21T23:29:18.197 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:29:17 smithi111 ceph-mon[31085]: fsmap 12 up:standby 2024-04-21T23:29:18.197 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:29:17 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-21T23:29:18.197 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:29:17 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-21T23:29:18.198 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:29:17 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-21T23:29:18.198 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:29:17 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-21T23:29:18.198 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:29:17 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-21T23:29:18.198 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:29:17 smithi111 ceph-mon[31085]: from='mgr.25111 172.21.15.40:0/340996279' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-21T23:29:18.198 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:29:17 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/1837012047' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-21T23:29:18.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:18.360+0000 7f32429fe640 1 -- 172.21.15.40:0/1069248446 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f323c1068f0 con 0x7f323c1032e0 2024-04-21T23:29:19.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:19.247+0000 7f32397fa640 1 -- 172.21.15.40:0/1069248446 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v107) ==== 169+0+0 (secure 0 0 0) 0x7f323009b3f0 con 0x7f323c1032e0 2024-04-21T23:29:19.248 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_data' removed 2024-04-21T23:29:19.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:19.251+0000 7f32429fe640 1 -- 172.21.15.40:0/1069248446 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f32240a2d00 msgr2=0x7f32240a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:29:19.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:19.251+0000 7f32429fe640 1 --2- 172.21.15.40:0/1069248446 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f32240a2d00 0x7f32240a51c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f322c0058c0 tx=0x7f322c005d30 comp rx=0 tx=0).stop 2024-04-21T23:29:19.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:19.251+0000 7f32429fe640 1 -- 172.21.15.40:0/1069248446 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f323c1032e0 msgr2=0x7f323c198540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:29:19.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:19.251+0000 7f32429fe640 1 --2- 172.21.15.40:0/1069248446 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f323c1032e0 0x7f323c198540 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f323000d770 tx=0x7f323000dc30 comp rx=0 tx=0).stop 2024-04-21T23:29:19.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:19.251+0000 7f32429fe640 1 -- 172.21.15.40:0/1069248446 shutdown_connections 2024-04-21T23:29:19.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:19.251+0000 7f32429fe640 1 --2- 172.21.15.40:0/1069248446 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f32240a2d00 0x7f32240a51c0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:19.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:19.251+0000 7f32429fe640 1 --2- 172.21.15.40:0/1069248446 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f323c10bd00 0x7f323c19c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:19.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:19.251+0000 7f32429fe640 1 --2- 172.21.15.40:0/1069248446 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f323c1032e0 0x7f323c198540 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:19.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:19.251+0000 7f32429fe640 1 --2- 172.21.15.40:0/1069248446 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f323c1029c0 0x7f323c198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:19.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:19.251+0000 7f32429fe640 1 -- 172.21.15.40:0/1069248446 >> 172.21.15.40:0/1069248446 conn(0x7f323c077740 msgr2=0x7f323c0fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:29:19.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:19.251+0000 7f32429fe640 1 -- 172.21.15.40:0/1069248446 shutdown_connections 2024-04-21T23:29:19.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:19.252+0000 7f32429fe640 1 -- 172.21.15.40:0/1069248446 wait complete. 2024-04-21T23:29:19.268 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-04-21T23:29:19.291 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:29:19 smithi040 ceph-mon[27984]: pgmap v340: 129 pgs: 129 active+clean; 95 MiB data, 805 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-04-21T23:29:19.291 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:29:19 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/1837012047' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-21T23:29:19.292 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:29:19 smithi040 ceph-mon[27984]: osdmap e106: 12 total, 12 up, 12 in 2024-04-21T23:29:19.292 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:29:19 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/1069248446' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-21T23:29:19.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:19.381+0000 7f1d1a896640 1 -- 172.21.15.40:0/1060731732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d14100610 msgr2=0x7f1d14100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:29:19.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:19.381+0000 7f1d1a896640 1 --2- 172.21.15.40:0/1060731732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d14100610 0x7f1d14100a90 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f1cfc009930 tx=0x7f1cfc02bc20 comp rx=0 tx=0).stop 2024-04-21T23:29:19.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:19.381+0000 7f1d1a896640 1 -- 172.21.15.40:0/1060731732 shutdown_connections 2024-04-21T23:29:19.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:19.381+0000 7f1d1a896640 1 --2- 172.21.15.40:0/1060731732 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1d14100fd0 0x7f1d1410d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:19.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:19.381+0000 7f1d1a896640 1 --2- 172.21.15.40:0/1060731732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d14100610 0x7f1d14100a90 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:19.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:19.381+0000 7f1d1a896640 1 --2- 172.21.15.40:0/1060731732 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1d141066b0 0x7f1d14106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:19.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:19.381+0000 7f1d1a896640 1 -- 172.21.15.40:0/1060731732 >> 172.21.15.40:0/1060731732 conn(0x7f1d14077740 msgr2=0x7f1d14077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:29:19.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:19.381+0000 7f1d1a896640 1 -- 172.21.15.40:0/1060731732 shutdown_connections 2024-04-21T23:29:19.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:19.381+0000 7f1d1a896640 1 -- 172.21.15.40:0/1060731732 wait complete. 2024-04-21T23:29:19.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:19.381+0000 7f1d1a896640 1 Processor -- start 2024-04-21T23:29:19.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:19.381+0000 7f1d1a896640 1 -- start start 2024-04-21T23:29:19.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:19.382+0000 7f1d1a896640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d14100610 0x7f1d14102540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:29:19.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:19.382+0000 7f1d1a896640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1d14100fd0 0x7f1d14102a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:29:19.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:19.382+0000 7f1d1a896640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1d141066b0 0x7f1d14102fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:29:19.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:19.382+0000 7f1d1a896640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1d141109f0 con 0x7f1d14100610 2024-04-21T23:29:19.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:19.382+0000 7f1d1a896640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f1d14110870 con 0x7f1d14100fd0 2024-04-21T23:29:19.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:19.382+0000 7f1d1a896640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1d14110b70 con 0x7f1d141066b0 2024-04-21T23:29:19.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:19.382+0000 7f1d13fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d14100610 0x7f1d14102540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:29:19.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:19.382+0000 7f1d13fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d14100610 0x7f1d14102540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52314/0 (socket says 172.21.15.40:52314) 2024-04-21T23:29:19.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:19.382+0000 7f1d13fff640 1 -- 172.21.15.40:0/3359313866 learned_addr learned my addr 172.21.15.40:0/3359313866 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-21T23:29:19.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:19.382+0000 7f1d18e33640 1 --2- 172.21.15.40:0/3359313866 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1d141066b0 0x7f1d14102fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:29:19.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:19.382+0000 7f1d137fe640 1 --2- 172.21.15.40:0/3359313866 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1d14100fd0 0x7f1d14102a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:29:19.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:19.382+0000 7f1d13fff640 1 -- 172.21.15.40:0/3359313866 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1d141066b0 msgr2=0x7f1d14102fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:29:19.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:19.382+0000 7f1d13fff640 1 --2- 172.21.15.40:0/3359313866 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1d141066b0 0x7f1d14102fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:19.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:19.382+0000 7f1d13fff640 1 -- 172.21.15.40:0/3359313866 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1d14100fd0 msgr2=0x7f1d14102a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:29:19.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:19.382+0000 7f1d13fff640 1 --2- 172.21.15.40:0/3359313866 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1d14100fd0 0x7f1d14102a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:19.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:19.382+0000 7f1d13fff640 1 -- 172.21.15.40:0/3359313866 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d141a6430 con 0x7f1d14100610 2024-04-21T23:29:19.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:19.383+0000 7f1d13fff640 1 --2- 172.21.15.40:0/3359313866 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d14100610 0x7f1d14102540 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f1d0400c970 tx=0x7f1d0400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:29:19.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:19.383+0000 7f1d117fa640 1 -- 172.21.15.40:0/3359313866 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d04004220 con 0x7f1d14100610 2024-04-21T23:29:19.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:19.383+0000 7f1d117fa640 1 -- 172.21.15.40:0/3359313866 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1d040043c0 con 0x7f1d14100610 2024-04-21T23:29:19.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:19.383+0000 7f1d1a896640 1 -- 172.21.15.40:0/3359313866 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d141a6720 con 0x7f1d14100610 2024-04-21T23:29:19.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:19.383+0000 7f1d117fa640 1 -- 172.21.15.40:0/3359313866 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d04004a00 con 0x7f1d14100610 2024-04-21T23:29:19.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:19.383+0000 7f1d1a896640 1 -- 172.21.15.40:0/3359313866 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d141a6bf0 con 0x7f1d14100610 2024-04-21T23:29:19.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:19.384+0000 7f1d1a896640 1 -- 172.21.15.40:0/3359313866 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d14101d50 con 0x7f1d14100610 2024-04-21T23:29:19.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:19.385+0000 7f1d117fa640 1 -- 172.21.15.40:0/3359313866 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f1d040079b0 con 0x7f1d14100610 2024-04-21T23:29:19.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:19.385+0000 7f1d117fa640 1 --2- 172.21.15.40:0/3359313866 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f1d000a2d00 0x7f1d000a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:29:19.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:19.386+0000 7f1d137fe640 1 --2- 172.21.15.40:0/3359313866 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f1d000a2d00 0x7f1d000a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:29:19.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:19.386+0000 7f1d117fa640 1 -- 172.21.15.40:0/3359313866 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 8582+0+0 (secure 0 0 0) 0x7f1d040cf110 con 0x7f1d14100610 2024-04-21T23:29:19.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:19.386+0000 7f1d137fe640 1 --2- 172.21.15.40:0/3359313866 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f1d000a2d00 0x7f1d000a51c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1cfc009900 tx=0x7f1cfc000bc0 comp rx=0 tx=0).ready entity=mgr.25111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:29:19.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:19.390+0000 7f1d117fa640 1 -- 172.21.15.40:0/3359313866 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f1d04014030 con 0x7f1d14100610 2024-04-21T23:29:19.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:29:19 smithi174 ceph-mon[31409]: pgmap v340: 129 pgs: 129 active+clean; 95 MiB data, 805 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-04-21T23:29:19.443 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:29:19 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/1837012047' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-21T23:29:19.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:29:19 smithi174 ceph-mon[31409]: osdmap e106: 12 total, 12 up, 12 in 2024-04-21T23:29:19.444 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:29:19 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/1069248446' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-21T23:29:19.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:29:19 smithi111 ceph-mon[31085]: pgmap v340: 129 pgs: 129 active+clean; 95 MiB data, 805 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-04-21T23:29:19.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:29:19 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/1837012047' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-21T23:29:19.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:29:19 smithi111 ceph-mon[31085]: osdmap e106: 12 total, 12 up, 12 in 2024-04-21T23:29:19.445 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:29:19 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/1069248446' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-21T23:29:19.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:19.575+0000 7f1d1a896640 1 -- 172.21.15.40:0/3359313866 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7f1d141a7000 con 0x7f1d14100610 2024-04-21T23:29:20.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:20.253+0000 7f1d117fa640 1 -- 172.21.15.40:0/3359313866 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v108) ==== 178+0+0 (secure 0 0 0) 0x7f1d0409b6a0 con 0x7f1d14100610 2024-04-21T23:29:20.253 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_data_ec' removed 2024-04-21T23:29:20.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:20.257+0000 7f1d1a896640 1 -- 172.21.15.40:0/3359313866 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f1d000a2d00 msgr2=0x7f1d000a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:29:20.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:20.257+0000 7f1d1a896640 1 --2- 172.21.15.40:0/3359313866 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f1d000a2d00 0x7f1d000a51c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1cfc009900 tx=0x7f1cfc000bc0 comp rx=0 tx=0).stop 2024-04-21T23:29:20.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:20.257+0000 7f1d1a896640 1 -- 172.21.15.40:0/3359313866 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d14100610 msgr2=0x7f1d14102540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:29:20.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:20.257+0000 7f1d1a896640 1 --2- 172.21.15.40:0/3359313866 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d14100610 0x7f1d14102540 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f1d0400c970 tx=0x7f1d0400ce30 comp rx=0 tx=0).stop 2024-04-21T23:29:20.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:20.257+0000 7f1d1a896640 1 -- 172.21.15.40:0/3359313866 shutdown_connections 2024-04-21T23:29:20.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:20.257+0000 7f1d1a896640 1 --2- 172.21.15.40:0/3359313866 >> [v2:172.21.15.40:6800/478611259,v1:172.21.15.40:6801/478611259] conn(0x7f1d000a2d00 0x7f1d000a51c0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:20.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:20.257+0000 7f1d1a896640 1 --2- 172.21.15.40:0/3359313866 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1d14100fd0 0x7f1d14102a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:20.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:20.257+0000 7f1d1a896640 1 --2- 172.21.15.40:0/3359313866 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1d141066b0 0x7f1d14102fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:20.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:20.257+0000 7f1d1a896640 1 --2- 172.21.15.40:0/3359313866 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d14100610 0x7f1d14102540 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:29:20.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:20.257+0000 7f1d1a896640 1 -- 172.21.15.40:0/3359313866 >> 172.21.15.40:0/3359313866 conn(0x7f1d14077740 msgr2=0x7f1d141046e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:29:20.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:20.257+0000 7f1d1a896640 1 -- 172.21.15.40:0/3359313866 shutdown_connections 2024-04-21T23:29:20.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-21T23:29:20.258+0000 7f1d1a896640 1 -- 172.21.15.40:0/3359313866 wait complete. 2024-04-21T23:29:20.274 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-21T23:29:20.341 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-21T23:29:20.374 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-21T23:29:20.403 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-21T23:29:20.403 DEBUG:teuthology.orchestra.run.smithi040:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-21T23:29:20.420 DEBUG:teuthology.orchestra.run.smithi111:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-21T23:29:20.437 DEBUG:teuthology.orchestra.run.smithi174:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-21T23:29:20.458 INFO:tasks.cephadm:Stopping all daemons... 2024-04-21T23:29:20.458 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-04-21T23:29:20.459 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-d814e4dc-0033-11ef-bc93-c7b262605968@mon.a 2024-04-21T23:29:20.510 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:29:20 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/1069248446' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-21T23:29:20.510 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:29:20 smithi040 ceph-mon[27984]: osdmap e107: 12 total, 12 up, 12 in 2024-04-21T23:29:20.510 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:29:20 smithi040 ceph-mon[27984]: from='client.? 172.21.15.40:0/3359313866' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-21T23:29:20.510 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:29:20 smithi040 ceph-mon[27984]: pgmap v343: 33 pgs: 33 active+clean; 26 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:29:20.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:29:20 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/1069248446' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-21T23:29:20.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:29:20 smithi174 ceph-mon[31409]: osdmap e107: 12 total, 12 up, 12 in 2024-04-21T23:29:20.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:29:20 smithi174 ceph-mon[31409]: from='client.? 172.21.15.40:0/3359313866' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-21T23:29:20.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:29:20 smithi174 ceph-mon[31409]: pgmap v343: 33 pgs: 33 active+clean; 26 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:29:20.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:29:20 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/1069248446' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-21T23:29:20.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:29:20 smithi111 ceph-mon[31085]: osdmap e107: 12 total, 12 up, 12 in 2024-04-21T23:29:20.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:29:20 smithi111 ceph-mon[31085]: from='client.? 172.21.15.40:0/3359313866' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-21T23:29:20.695 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:29:20 smithi111 ceph-mon[31085]: pgmap v343: 33 pgs: 33 active+clean; 26 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:29:20.785 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:29:20 smithi040 systemd[1]: Stopping Ceph mon.a for d814e4dc-0033-11ef-bc93-c7b262605968... 2024-04-21T23:29:21.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:29:20 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mon-a[27980]: 2024-04-21T23:29:20.785+0000 7fd8212c8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T23:29:21.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:29:20 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mon-a[27980]: 2024-04-21T23:29:20.785+0000 7fd8212c8700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-04-21T23:29:21.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:29:21 smithi040 podman[94610]: 2024-04-21 23:29:21.000703403 +0000 UTC m=+0.339878981 container died 5fb73014130c79a2b3ca1e60ee21c825a0f1f4d38399689ce2bc910dfe6881aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mon-a, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, ceph=True, RELEASE=squid-820a2f8, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=) 2024-04-21T23:29:21.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:29:21 smithi040 podman[94610]: 2024-04-21 23:29:21.379214216 +0000 UTC m=+0.718389777 container cleanup 5fb73014130c79a2b3ca1e60ee21c825a0f1f4d38399689ce2bc910dfe6881aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mon-a, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=squid-820a2f8, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240416) 2024-04-21T23:29:21.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 21 23:29:21 smithi040 bash[94610]: ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mon-a 2024-04-21T23:29:21.859 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d814e4dc-0033-11ef-bc93-c7b262605968@mon.a.service' 2024-04-21T23:29:21.905 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T23:29:21.905 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-04-21T23:29:21.905 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-04-21T23:29:21.906 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-d814e4dc-0033-11ef-bc93-c7b262605968@mon.b 2024-04-21T23:29:22.193 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:29:21 smithi111 systemd[1]: Stopping Ceph mon.b for d814e4dc-0033-11ef-bc93-c7b262605968... 2024-04-21T23:29:22.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:29:22 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mon-b[31081]: 2024-04-21T23:29:22.228+0000 7f9f90c56700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T23:29:22.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:29:22 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mon-b[31081]: 2024-04-21T23:29:22.228+0000 7f9f90c56700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-04-21T23:29:22.694 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:29:22 smithi111 podman[66804]: 2024-04-21 23:29:22.311636466 +0000 UTC m=+0.211916453 container died 7f9ded09855079fd92f89602e3f31fdaff233dbb1cc7b76c28e68c6a209ab4c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mon-b, RELEASE=squid-820a2f8, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-21T23:29:23.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:29:22 smithi111 podman[66804]: 2024-04-21 23:29:22.814117583 +0000 UTC m=+0.714397569 container cleanup 7f9ded09855079fd92f89602e3f31fdaff233dbb1cc7b76c28e68c6a209ab4c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mon-b, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-820a2f8) 2024-04-21T23:29:23.194 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:29:22 smithi111 bash[66804]: ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mon-b 2024-04-21T23:29:23.317 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d814e4dc-0033-11ef-bc93-c7b262605968@mon.b.service' 2024-04-21T23:29:23.363 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T23:29:23.364 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-04-21T23:29:23.364 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-04-21T23:29:23.365 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ceph-d814e4dc-0033-11ef-bc93-c7b262605968@mon.c 2024-04-21T23:29:23.693 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:29:23 smithi174 systemd[1]: Stopping Ceph mon.c for d814e4dc-0033-11ef-bc93-c7b262605968... 2024-04-21T23:29:23.943 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:29:23 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mon-c[31389]: 2024-04-21T23:29:23.693+0000 7f87e86ca700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T23:29:23.944 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:29:23 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mon-c[31389]: 2024-04-21T23:29:23.693+0000 7f87e86ca700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-04-21T23:29:24.344 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:29:23 smithi174 podman[67645]: 2024-04-21 23:29:23.959631006 +0000 UTC m=+0.399234846 container died 2500b95b86b5aa92e16def89b40df69f8ef4f99398a445434e904402392d284d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-820a2f8, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-21T23:29:24.694 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:29:24 smithi174 podman[67645]: 2024-04-21 23:29:24.436313741 +0000 UTC m=+0.875917571 container cleanup 2500b95b86b5aa92e16def89b40df69f8ef4f99398a445434e904402392d284d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mon-c, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-820a2f8, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T23:29:24.695 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 21 23:29:24 smithi174 bash[67645]: ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mon-c 2024-04-21T23:29:24.845 DEBUG:teuthology.orchestra.run.smithi174:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d814e4dc-0033-11ef-bc93-c7b262605968@mon.c.service' 2024-04-21T23:29:24.894 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T23:29:24.894 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-04-21T23:29:24.895 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-04-21T23:29:24.895 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-d814e4dc-0033-11ef-bc93-c7b262605968@mgr.x 2024-04-21T23:29:25.181 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:29:24 smithi040 systemd[1]: Stopping Ceph mgr.x for d814e4dc-0033-11ef-bc93-c7b262605968... 2024-04-21T23:29:25.618 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:29:25 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x[28215]: 2024-04-21T23:29:25.308+0000 7f2f174c0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T23:29:25.618 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:29:25 smithi040 podman[94735]: 2024-04-21 23:29:25.34035689 +0000 UTC m=+0.243638892 container died 7b8d28516472063ffc63f57bbf7e41b8b6030acb159c557b367597f51489171c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, RELEASE=squid-820a2f8, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T23:29:25.931 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:29:25 smithi040 podman[94735]: 2024-04-21 23:29:25.726885312 +0000 UTC m=+0.630167321 container cleanup 7b8d28516472063ffc63f57bbf7e41b8b6030acb159c557b367597f51489171c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x, RELEASE=squid-820a2f8, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-21T23:29:25.932 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 21 23:29:25 smithi040 bash[94735]: ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-x 2024-04-21T23:29:26.114 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d814e4dc-0033-11ef-bc93-c7b262605968@mgr.x.service' 2024-04-21T23:29:26.149 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T23:29:26.149 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-04-21T23:29:26.149 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-04-21T23:29:26.149 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-d814e4dc-0033-11ef-bc93-c7b262605968@mgr.y 2024-04-21T23:29:26.444 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:29:26 smithi111 systemd[1]: Stopping Ceph mgr.y for d814e4dc-0033-11ef-bc93-c7b262605968... 2024-04-21T23:29:26.767 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:29:26 smithi111 podman[66928]: 2024-04-21 23:29:26.48963827 +0000 UTC m=+0.142038676 container died 8f392e362b15b39584278d2989fb8cecadf326bbfad9fe4d4bc8fb1b4a2823a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-y, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-820a2f8, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T23:29:27.177 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:29:26 smithi111 podman[66928]: 2024-04-21 23:29:26.852610737 +0000 UTC m=+0.505011126 container cleanup 8f392e362b15b39584278d2989fb8cecadf326bbfad9fe4d4bc8fb1b4a2823a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-y, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-820a2f8, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-21T23:29:27.178 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:29:26 smithi111 bash[66928]: ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-y 2024-04-21T23:29:27.417 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d814e4dc-0033-11ef-bc93-c7b262605968@mgr.y.service' 2024-04-21T23:29:27.437 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:29:27 smithi111 podman[66940]: 2024-04-21 23:29:27.179539804 +0000 UTC m=+0.685698803 container remove 8f392e362b15b39584278d2989fb8cecadf326bbfad9fe4d4bc8fb1b4a2823a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-820a2f8, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-21T23:29:27.437 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:29:27 smithi111 systemd[1]: ceph-d814e4dc-0033-11ef-bc93-c7b262605968@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-04-21T23:29:27.438 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:29:27 smithi111 systemd[1]: ceph-d814e4dc-0033-11ef-bc93-c7b262605968@mgr.y.service: Failed with result 'exit-code'. 2024-04-21T23:29:27.438 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:29:27 smithi111 systemd[1]: Stopped Ceph mgr.y for d814e4dc-0033-11ef-bc93-c7b262605968. 2024-04-21T23:29:27.438 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:29:27 smithi111 systemd[1]: ceph-d814e4dc-0033-11ef-bc93-c7b262605968@mgr.y.service: Consumed 20.121s CPU time. 2024-04-21T23:29:27.450 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T23:29:27.450 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-04-21T23:29:27.450 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-04-21T23:29:27.450 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ceph-d814e4dc-0033-11ef-bc93-c7b262605968@mgr.z 2024-04-21T23:29:27.794 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Apr 21 23:29:27 smithi174 systemd[1]: Stopping Ceph mgr.z for d814e4dc-0033-11ef-bc93-c7b262605968... 2024-04-21T23:29:28.082 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Apr 21 23:29:27 smithi174 podman[67766]: 2024-04-21 23:29:27.79551588 +0000 UTC m=+0.154345380 container died a7eda681942db0f41a0b1ada86e558a5271dff1767c39119e57f24cf1c8b11b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-z, RELEASE=squid-820a2f8, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T23:29:28.433 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Apr 21 23:29:28 smithi174 podman[67766]: 2024-04-21 23:29:28.175483246 +0000 UTC m=+0.534312735 container cleanup a7eda681942db0f41a0b1ada86e558a5271dff1767c39119e57f24cf1c8b11b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-z, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-820a2f8, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, ceph=True) 2024-04-21T23:29:28.433 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Apr 21 23:29:28 smithi174 bash[67766]: ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mgr-z 2024-04-21T23:29:28.600 DEBUG:teuthology.orchestra.run.smithi174:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d814e4dc-0033-11ef-bc93-c7b262605968@mgr.z.service' 2024-04-21T23:29:28.634 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T23:29:28.634 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-04-21T23:29:28.634 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-21T23:29:28.635 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-d814e4dc-0033-11ef-bc93-c7b262605968@osd.0 2024-04-21T23:29:28.931 INFO:journalctl@ceph.osd.0.smithi040.stdout:Apr 21 23:29:28 smithi040 systemd[1]: Stopping Ceph osd.0 for d814e4dc-0033-11ef-bc93-c7b262605968... 2024-04-21T23:29:29.432 INFO:journalctl@ceph.osd.0.smithi040.stdout:Apr 21 23:29:28 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-0[40139]: 2024-04-21T23:29:28.956+0000 7f29a79a2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T23:29:29.432 INFO:journalctl@ceph.osd.0.smithi040.stdout:Apr 21 23:29:28 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-0[40139]: 2024-04-21T23:29:28.956+0000 7f29a79a2700 -1 osd.0 108 *** Got signal Terminated *** 2024-04-21T23:29:29.432 INFO:journalctl@ceph.osd.0.smithi040.stdout:Apr 21 23:29:28 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-0[40139]: 2024-04-21T23:29:28.956+0000 7f29a79a2700 -1 osd.0 108 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-21T23:29:34.565 INFO:journalctl@ceph.osd.0.smithi040.stdout:Apr 21 23:29:34 smithi040 podman[94861]: 2024-04-21 23:29:34.275809562 +0000 UTC m=+5.441848228 container died bb691923bf7ab75150a3d5e26a371705e208c35030121488061c9cb64fd389c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=squid-820a2f8, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T23:29:34.882 INFO:journalctl@ceph.osd.0.smithi040.stdout:Apr 21 23:29:34 smithi040 podman[94861]: 2024-04-21 23:29:34.665861039 +0000 UTC m=+5.831899701 container cleanup bb691923bf7ab75150a3d5e26a371705e208c35030121488061c9cb64fd389c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-820a2f8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-21T23:29:34.882 INFO:journalctl@ceph.osd.0.smithi040.stdout:Apr 21 23:29:34 smithi040 bash[94861]: ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-0 2024-04-21T23:29:35.181 INFO:journalctl@ceph.osd.0.smithi040.stdout:Apr 21 23:29:34 smithi040 podman[94879]: 2024-04-21 23:29:34.883980596 +0000 UTC m=+0.605945034 container remove bb691923bf7ab75150a3d5e26a371705e208c35030121488061c9cb64fd389c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_CLEAN=True, RELEASE=squid-820a2f8, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-21T23:29:35.578 INFO:journalctl@ceph.osd.0.smithi040.stdout:Apr 21 23:29:35 smithi040 podman[94974]: 2024-04-21 23:29:35.201668916 +0000 UTC m=+0.018701275 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-21T23:29:35.579 INFO:journalctl@ceph.osd.0.smithi040.stdout:Apr 21 23:29:35 smithi040 podman[94974]: 2024-04-21 23:29:35.326771788 +0000 UTC m=+0.143804138 container create 0046e30365822273cc4b074d3a216c13c1cdafda9a22abf21f37bb43ba3e2316 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-820a2f8, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, io.buildah.version=1.33.5) 2024-04-21T23:29:35.896 INFO:journalctl@ceph.osd.0.smithi040.stdout:Apr 21 23:29:35 smithi040 podman[94974]: 2024-04-21 23:29:35.580196834 +0000 UTC m=+0.397229192 container init 0046e30365822273cc4b074d3a216c13c1cdafda9a22abf21f37bb43ba3e2316 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=squid-820a2f8, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-21T23:29:35.896 INFO:journalctl@ceph.osd.0.smithi040.stdout:Apr 21 23:29:35 smithi040 podman[94974]: 2024-04-21 23:29:35.58405931 +0000 UTC m=+0.401091660 container start 0046e30365822273cc4b074d3a216c13c1cdafda9a22abf21f37bb43ba3e2316 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-0-deactivate, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-820a2f8, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-04-21T23:29:35.896 INFO:journalctl@ceph.osd.0.smithi040.stdout:Apr 21 23:29:35 smithi040 podman[94974]: 2024-04-21 23:29:35.680123013 +0000 UTC m=+0.497155373 container attach 0046e30365822273cc4b074d3a216c13c1cdafda9a22abf21f37bb43ba3e2316 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-820a2f8, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-21T23:29:36.181 INFO:journalctl@ceph.osd.0.smithi040.stdout:Apr 21 23:29:35 smithi040 podman[94974]: 2024-04-21 23:29:35.898346571 +0000 UTC m=+0.715378922 container died 0046e30365822273cc4b074d3a216c13c1cdafda9a22abf21f37bb43ba3e2316 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-0-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=squid-820a2f8, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T23:29:36.607 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d814e4dc-0033-11ef-bc93-c7b262605968@osd.0.service' 2024-04-21T23:29:36.651 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T23:29:36.652 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-21T23:29:36.652 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-21T23:29:36.653 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-d814e4dc-0033-11ef-bc93-c7b262605968@osd.1 2024-04-21T23:29:36.943 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 21 23:29:36 smithi111 systemd[1]: Stopping Ceph osd.1 for d814e4dc-0033-11ef-bc93-c7b262605968... 2024-04-21T23:29:37.694 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 21 23:29:37 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-1[35676]: 2024-04-21T23:29:37.256+0000 7f3779bff700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T23:29:37.694 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 21 23:29:37 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-1[35676]: 2024-04-21T23:29:37.256+0000 7f3779bff700 -1 osd.1 108 *** Got signal Terminated *** 2024-04-21T23:29:37.694 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 21 23:29:37 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-1[35676]: 2024-04-21T23:29:37.256+0000 7f3779bff700 -1 osd.1 108 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-21T23:29:43.064 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 21 23:29:42 smithi111 podman[67052]: 2024-04-21 23:29:42.770942265 +0000 UTC m=+5.932107850 container died 6838042f57c3ecd26d1afcbb31c7fca27941095fe0cdcfb350fe843a7949c164 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-1, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, RELEASE=squid-820a2f8, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T23:29:43.390 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 21 23:29:43 smithi111 podman[67052]: 2024-04-21 23:29:43.15739656 +0000 UTC m=+6.318562156 container cleanup 6838042f57c3ecd26d1afcbb31c7fca27941095fe0cdcfb350fe843a7949c164 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-1, RELEASE=squid-820a2f8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-21T23:29:43.390 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 21 23:29:43 smithi111 bash[67052]: ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-1 2024-04-21T23:29:43.693 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 21 23:29:43 smithi111 podman[67070]: 2024-04-21 23:29:43.392191074 +0000 UTC m=+0.623217409 container remove 6838042f57c3ecd26d1afcbb31c7fca27941095fe0cdcfb350fe843a7949c164 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-1, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=squid-820a2f8, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-04-21T23:29:44.094 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 21 23:29:43 smithi111 podman[67158]: 2024-04-21 23:29:43.732968873 +0000 UTC m=+0.017261695 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-21T23:29:44.095 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 21 23:29:43 smithi111 podman[67158]: 2024-04-21 23:29:43.851853789 +0000 UTC m=+0.136146595 container create e4702ce5a87698c0700b939ac89b2e61f82dc419dc71f6d06d6483ee26452a8d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-820a2f8, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS) 2024-04-21T23:29:44.364 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 21 23:29:44 smithi111 podman[67158]: 2024-04-21 23:29:44.096613766 +0000 UTC m=+0.380906585 container init e4702ce5a87698c0700b939ac89b2e61f82dc419dc71f6d06d6483ee26452a8d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-1-deactivate, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=squid-820a2f8, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-21T23:29:44.364 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 21 23:29:44 smithi111 podman[67158]: 2024-04-21 23:29:44.100484225 +0000 UTC m=+0.384777043 container start e4702ce5a87698c0700b939ac89b2e61f82dc419dc71f6d06d6483ee26452a8d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-820a2f8, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T23:29:44.364 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 21 23:29:44 smithi111 podman[67158]: 2024-04-21 23:29:44.196424346 +0000 UTC m=+0.480717155 container attach e4702ce5a87698c0700b939ac89b2e61f82dc419dc71f6d06d6483ee26452a8d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-820a2f8, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-21T23:29:44.694 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 21 23:29:44 smithi111 podman[67158]: 2024-04-21 23:29:44.406549628 +0000 UTC m=+0.690842429 container died e4702ce5a87698c0700b939ac89b2e61f82dc419dc71f6d06d6483ee26452a8d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-820a2f8, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T23:29:45.038 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d814e4dc-0033-11ef-bc93-c7b262605968@osd.1.service' 2024-04-21T23:29:45.075 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T23:29:45.076 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-21T23:29:45.076 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-21T23:29:45.076 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ceph-d814e4dc-0033-11ef-bc93-c7b262605968@osd.2 2024-04-21T23:29:45.403 INFO:journalctl@ceph.osd.2.smithi174.stdout:Apr 21 23:29:45 smithi174 systemd[1]: Stopping Ceph osd.2 for d814e4dc-0033-11ef-bc93-c7b262605968... 2024-04-21T23:29:45.667 INFO:journalctl@ceph.osd.2.smithi174.stdout:Apr 21 23:29:45 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-2[36550]: 2024-04-21T23:29:45.403+0000 7f0f45c1b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T23:29:45.667 INFO:journalctl@ceph.osd.2.smithi174.stdout:Apr 21 23:29:45 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-2[36550]: 2024-04-21T23:29:45.403+0000 7f0f45c1b700 -1 osd.2 108 *** Got signal Terminated *** 2024-04-21T23:29:45.667 INFO:journalctl@ceph.osd.2.smithi174.stdout:Apr 21 23:29:45 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-2[36550]: 2024-04-21T23:29:45.403+0000 7f0f45c1b700 -1 osd.2 108 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-21T23:29:51.193 INFO:journalctl@ceph.osd.2.smithi174.stdout:Apr 21 23:29:50 smithi174 podman[67890]: 2024-04-21 23:29:50.921588037 +0000 UTC m=+5.650835540 container died f08f149613884746c150d06e6f339421d0795d3926848027a2b7dbfa2f556496 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-820a2f8, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416) 2024-04-21T23:29:51.579 INFO:journalctl@ceph.osd.2.smithi174.stdout:Apr 21 23:29:51 smithi174 podman[67890]: 2024-04-21 23:29:51.304654811 +0000 UTC m=+6.033902310 container cleanup f08f149613884746c150d06e6f339421d0795d3926848027a2b7dbfa2f556496 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-820a2f8, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-04-21T23:29:51.579 INFO:journalctl@ceph.osd.2.smithi174.stdout:Apr 21 23:29:51 smithi174 bash[67890]: ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-2 2024-04-21T23:29:51.931 INFO:journalctl@ceph.osd.2.smithi174.stdout:Apr 21 23:29:51 smithi174 podman[67920]: 2024-04-21 23:29:51.581117568 +0000 UTC m=+0.658941541 container remove f08f149613884746c150d06e6f339421d0795d3926848027a2b7dbfa2f556496 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-2, RELEASE=squid-820a2f8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-21T23:29:52.193 INFO:journalctl@ceph.osd.2.smithi174.stdout:Apr 21 23:29:51 smithi174 podman[68013]: 2024-04-21 23:29:51.835856952 +0000 UTC m=+0.017869407 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-21T23:29:52.194 INFO:journalctl@ceph.osd.2.smithi174.stdout:Apr 21 23:29:51 smithi174 podman[68013]: 2024-04-21 23:29:51.9994838 +0000 UTC m=+0.181496259 container create e29dfb2a3010da3706431c95ddf251958c630ad07578fda59b28ebc2f9e8e9bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-820a2f8, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-21T23:29:52.524 INFO:journalctl@ceph.osd.2.smithi174.stdout:Apr 21 23:29:52 smithi174 podman[68013]: 2024-04-21 23:29:52.234840297 +0000 UTC m=+0.416852753 container init e29dfb2a3010da3706431c95ddf251958c630ad07578fda59b28ebc2f9e8e9bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-820a2f8, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True) 2024-04-21T23:29:52.524 INFO:journalctl@ceph.osd.2.smithi174.stdout:Apr 21 23:29:52 smithi174 podman[68013]: 2024-04-21 23:29:52.238009556 +0000 UTC m=+0.420022013 container start e29dfb2a3010da3706431c95ddf251958c630ad07578fda59b28ebc2f9e8e9bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-820a2f8, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0) 2024-04-21T23:29:52.525 INFO:journalctl@ceph.osd.2.smithi174.stdout:Apr 21 23:29:52 smithi174 podman[68013]: 2024-04-21 23:29:52.318506691 +0000 UTC m=+0.500519159 container attach e29dfb2a3010da3706431c95ddf251958c630ad07578fda59b28ebc2f9e8e9bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-2-deactivate, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-820a2f8, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-21T23:29:52.821 INFO:journalctl@ceph.osd.2.smithi174.stdout:Apr 21 23:29:52 smithi174 podman[68013]: 2024-04-21 23:29:52.52629887 +0000 UTC m=+0.708311332 container died e29dfb2a3010da3706431c95ddf251958c630ad07578fda59b28ebc2f9e8e9bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, RELEASE=squid-820a2f8, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-21T23:29:53.110 DEBUG:teuthology.orchestra.run.smithi174:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d814e4dc-0033-11ef-bc93-c7b262605968@osd.2.service' 2024-04-21T23:29:53.150 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T23:29:53.153 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-21T23:29:53.154 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-21T23:29:53.154 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-d814e4dc-0033-11ef-bc93-c7b262605968@osd.3 2024-04-21T23:29:53.432 INFO:journalctl@ceph.osd.3.smithi040.stdout:Apr 21 23:29:53 smithi040 systemd[1]: Stopping Ceph osd.3 for d814e4dc-0033-11ef-bc93-c7b262605968... 2024-04-21T23:29:53.706 INFO:journalctl@ceph.osd.3.smithi040.stdout:Apr 21 23:29:53 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-3[47087]: 2024-04-21T23:29:53.477+0000 7fe7a7444700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T23:29:53.706 INFO:journalctl@ceph.osd.3.smithi040.stdout:Apr 21 23:29:53 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-3[47087]: 2024-04-21T23:29:53.477+0000 7fe7a7444700 -1 osd.3 108 *** Got signal Terminated *** 2024-04-21T23:29:53.706 INFO:journalctl@ceph.osd.3.smithi040.stdout:Apr 21 23:29:53 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-3[47087]: 2024-04-21T23:29:53.477+0000 7fe7a7444700 -1 osd.3 108 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-21T23:29:54.932 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 21 23:29:54 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-6[53641]: 2024-04-21T23:29:54.743+0000 7f4fdab4a700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:32.252857+0000 front 2024-04-21T23:29:32.253034+0000 (oldest deadline 2024-04-21T23:29:54.553015+0000) 2024-04-21T23:29:56.181 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 21 23:29:55 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-6[53641]: 2024-04-21T23:29:55.738+0000 7f4fdab4a700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:32.252857+0000 front 2024-04-21T23:29:32.253034+0000 (oldest deadline 2024-04-21T23:29:54.553015+0000) 2024-04-21T23:29:57.181 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 21 23:29:56 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-6[53641]: 2024-04-21T23:29:56.718+0000 7f4fdab4a700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:32.252857+0000 front 2024-04-21T23:29:32.253034+0000 (oldest deadline 2024-04-21T23:29:54.553015+0000) 2024-04-21T23:29:57.943 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:29:57 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:29:57.676+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:31.573404+0000 front 2024-04-21T23:29:31.573351+0000 (oldest deadline 2024-04-21T23:29:56.873350+0000) 2024-04-21T23:29:58.182 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 21 23:29:57 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-6[53641]: 2024-04-21T23:29:57.702+0000 7f4fdab4a700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:32.252857+0000 front 2024-04-21T23:29:32.253034+0000 (oldest deadline 2024-04-21T23:29:54.553015+0000) 2024-04-21T23:29:58.182 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:29:57 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:29:57.869+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.927390+0000 front 2024-04-21T23:29:33.927500+0000 (oldest deadline 2024-04-21T23:29:56.827517+0000) 2024-04-21T23:29:58.682 INFO:journalctl@ceph.osd.3.smithi040.stdout:Apr 21 23:29:58 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-3[47087]: 2024-04-21T23:29:58.396+0000 7fe79fa30700 -1 osd.3 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.009922+0000 front 2024-04-21T23:29:33.009986+0000 (oldest deadline 2024-04-21T23:29:57.710136+0000) 2024-04-21T23:29:58.943 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:29:58 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:29:58.693+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:31.573404+0000 front 2024-04-21T23:29:31.573351+0000 (oldest deadline 2024-04-21T23:29:56.873350+0000) 2024-04-21T23:29:58.944 INFO:journalctl@ceph.osd.5.smithi174.stdout:Apr 21 23:29:58 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-5[43337]: 2024-04-21T23:29:58.707+0000 7f5a73535700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.108794+0000 front 2024-04-21T23:29:33.108670+0000 (oldest deadline 2024-04-21T23:29:57.808697+0000) 2024-04-21T23:29:59.182 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 21 23:29:58 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-6[53641]: 2024-04-21T23:29:58.735+0000 7f4fdab4a700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:32.252857+0000 front 2024-04-21T23:29:32.253034+0000 (oldest deadline 2024-04-21T23:29:54.553015+0000) 2024-04-21T23:29:59.182 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:29:58 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:29:58.836+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.927390+0000 front 2024-04-21T23:29:33.927500+0000 (oldest deadline 2024-04-21T23:29:56.827517+0000) 2024-04-21T23:29:59.183 INFO:journalctl@ceph.osd.3.smithi040.stdout:Apr 21 23:29:58 smithi040 podman[95084]: 2024-04-21 23:29:58.952995569 +0000 UTC m=+5.599464370 container died 095368d70c2ffed297340290d06b3370bebc175cac1360f1ad534b4793d00e28 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-3, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=squid-820a2f8, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-21T23:29:59.595 INFO:journalctl@ceph.osd.3.smithi040.stdout:Apr 21 23:29:59 smithi040 podman[95084]: 2024-04-21 23:29:59.337193571 +0000 UTC m=+5.983662369 container cleanup 095368d70c2ffed297340290d06b3370bebc175cac1360f1ad534b4793d00e28 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-3, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=squid-820a2f8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True) 2024-04-21T23:29:59.595 INFO:journalctl@ceph.osd.3.smithi040.stdout:Apr 21 23:29:59 smithi040 bash[95084]: ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-3 2024-04-21T23:29:59.859 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 21 23:29:59 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-6[53641]: 2024-04-21T23:29:59.780+0000 7f4fdab4a700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:32.252857+0000 front 2024-04-21T23:29:32.253034+0000 (oldest deadline 2024-04-21T23:29:54.553015+0000) 2024-04-21T23:29:59.860 INFO:journalctl@ceph.osd.3.smithi040.stdout:Apr 21 23:29:59 smithi040 podman[95113]: 2024-04-21 23:29:59.596843942 +0000 UTC m=+0.646582110 container remove 095368d70c2ffed297340290d06b3370bebc175cac1360f1ad534b4793d00e28 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-3, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=squid-820a2f8, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-21T23:29:59.943 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:29:59 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:29:59.734+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:31.573404+0000 front 2024-04-21T23:29:31.573351+0000 (oldest deadline 2024-04-21T23:29:56.873350+0000) 2024-04-21T23:29:59.944 INFO:journalctl@ceph.osd.5.smithi174.stdout:Apr 21 23:29:59 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-5[43337]: 2024-04-21T23:29:59.690+0000 7f5a73535700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.108794+0000 front 2024-04-21T23:29:33.108670+0000 (oldest deadline 2024-04-21T23:29:57.808697+0000) 2024-04-21T23:29:59.944 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:29:59 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:29:59.477+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.197048+0000 front 2024-04-21T23:29:33.197157+0000 (oldest deadline 2024-04-21T23:29:59.097025+0000) 2024-04-21T23:30:00.110 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:29:59 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:29:59.860+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.927390+0000 front 2024-04-21T23:29:33.927500+0000 (oldest deadline 2024-04-21T23:29:56.827517+0000) 2024-04-21T23:30:00.110 INFO:journalctl@ceph.osd.3.smithi040.stdout:Apr 21 23:29:59 smithi040 podman[95204]: 2024-04-21 23:29:59.862208915 +0000 UTC m=+0.018197906 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-21T23:30:00.110 INFO:journalctl@ceph.osd.3.smithi040.stdout:Apr 21 23:29:59 smithi040 podman[95204]: 2024-04-21 23:29:59.989679344 +0000 UTC m=+0.145668334 container create 33df914760d4430aa10d5f23d11e8cf0bc0c4a4d42c6c0c13573320339e16d83 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-820a2f8, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0) 2024-04-21T23:30:00.432 INFO:journalctl@ceph.osd.3.smithi040.stdout:Apr 21 23:30:00 smithi040 podman[95204]: 2024-04-21 23:30:00.217434987 +0000 UTC m=+0.373423979 container init 33df914760d4430aa10d5f23d11e8cf0bc0c4a4d42c6c0c13573320339e16d83 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-820a2f8, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T23:30:00.432 INFO:journalctl@ceph.osd.3.smithi040.stdout:Apr 21 23:30:00 smithi040 podman[95204]: 2024-04-21 23:30:00.220822436 +0000 UTC m=+0.376811428 container start 33df914760d4430aa10d5f23d11e8cf0bc0c4a4d42c6c0c13573320339e16d83 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, RELEASE=squid-820a2f8, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-04-21T23:30:00.432 INFO:journalctl@ceph.osd.3.smithi040.stdout:Apr 21 23:30:00 smithi040 podman[95204]: 2024-04-21 23:30:00.292141347 +0000 UTC m=+0.448130340 container attach 33df914760d4430aa10d5f23d11e8cf0bc0c4a4d42c6c0c13573320339e16d83 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-820a2f8) 2024-04-21T23:30:00.766 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:00 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:00.459+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.197048+0000 front 2024-04-21T23:29:33.197157+0000 (oldest deadline 2024-04-21T23:29:59.097025+0000) 2024-04-21T23:30:00.815 INFO:journalctl@ceph.osd.3.smithi040.stdout:Apr 21 23:30:00 smithi040 podman[95204]: 2024-04-21 23:30:00.534840726 +0000 UTC m=+0.690829717 container died 33df914760d4430aa10d5f23d11e8cf0bc0c4a4d42c6c0c13573320339e16d83 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-3-deactivate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-820a2f8, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-21T23:30:00.943 INFO:journalctl@ceph.osd.5.smithi174.stdout:Apr 21 23:30:00 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-5[43337]: 2024-04-21T23:30:00.646+0000 7f5a73535700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.108794+0000 front 2024-04-21T23:29:33.108670+0000 (oldest deadline 2024-04-21T23:29:57.808697+0000) 2024-04-21T23:30:00.944 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:00 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:00.685+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:31.573404+0000 front 2024-04-21T23:29:31.573351+0000 (oldest deadline 2024-04-21T23:29:56.873350+0000) 2024-04-21T23:30:01.144 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 21 23:30:00 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-6[53641]: 2024-04-21T23:30:00.816+0000 7f4fdab4a700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:32.252857+0000 front 2024-04-21T23:29:32.253034+0000 (oldest deadline 2024-04-21T23:29:54.553015+0000) 2024-04-21T23:30:01.145 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:00 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:00.898+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.927390+0000 front 2024-04-21T23:29:33.927500+0000 (oldest deadline 2024-04-21T23:29:56.827517+0000) 2024-04-21T23:30:01.159 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d814e4dc-0033-11ef-bc93-c7b262605968@osd.3.service' 2024-04-21T23:30:01.200 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T23:30:01.200 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-21T23:30:01.200 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-21T23:30:01.200 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-d814e4dc-0033-11ef-bc93-c7b262605968@osd.4 2024-04-21T23:30:01.491 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 21 23:30:01 smithi111 systemd[1]: Stopping Ceph osd.4 for d814e4dc-0033-11ef-bc93-c7b262605968... 2024-04-21T23:30:01.943 INFO:journalctl@ceph.osd.5.smithi174.stdout:Apr 21 23:30:01 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-5[43337]: 2024-04-21T23:30:01.603+0000 7f5a73535700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.108794+0000 front 2024-04-21T23:29:33.108670+0000 (oldest deadline 2024-04-21T23:29:57.808697+0000) 2024-04-21T23:30:01.944 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:01 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:01.653+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:31.573404+0000 front 2024-04-21T23:29:31.573351+0000 (oldest deadline 2024-04-21T23:29:56.873350+0000) 2024-04-21T23:30:01.944 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 21 23:30:01 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-4[42582]: 2024-04-21T23:30:01.519+0000 7fa4a74e8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T23:30:01.944 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 21 23:30:01 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-4[42582]: 2024-04-21T23:30:01.519+0000 7fa4a74e8700 -1 osd.4 108 *** Got signal Terminated *** 2024-04-21T23:30:01.945 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 21 23:30:01 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-4[42582]: 2024-04-21T23:30:01.519+0000 7fa4a74e8700 -1 osd.4 108 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-21T23:30:01.945 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:01 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:01.493+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.197048+0000 front 2024-04-21T23:29:33.197157+0000 (oldest deadline 2024-04-21T23:29:59.097025+0000) 2024-04-21T23:30:02.181 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 21 23:30:01 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-6[53641]: 2024-04-21T23:30:01.852+0000 7f4fdab4a700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:32.252857+0000 front 2024-04-21T23:29:32.253034+0000 (oldest deadline 2024-04-21T23:29:54.553015+0000) 2024-04-21T23:30:02.182 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:01 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:01.906+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.927390+0000 front 2024-04-21T23:29:33.927500+0000 (oldest deadline 2024-04-21T23:29:56.827517+0000) 2024-04-21T23:30:02.943 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:02 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:02.652+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:31.573404+0000 front 2024-04-21T23:29:31.573351+0000 (oldest deadline 2024-04-21T23:29:56.873350+0000) 2024-04-21T23:30:02.944 INFO:journalctl@ceph.osd.5.smithi174.stdout:Apr 21 23:30:02 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-5[43337]: 2024-04-21T23:30:02.581+0000 7f5a73535700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.108794+0000 front 2024-04-21T23:29:33.108670+0000 (oldest deadline 2024-04-21T23:29:57.808697+0000) 2024-04-21T23:30:02.945 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:02 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:02.469+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.197048+0000 front 2024-04-21T23:29:33.197157+0000 (oldest deadline 2024-04-21T23:29:59.097025+0000) 2024-04-21T23:30:03.181 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:02 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:02.940+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.927390+0000 front 2024-04-21T23:29:33.927500+0000 (oldest deadline 2024-04-21T23:29:56.827517+0000) 2024-04-21T23:30:03.182 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 21 23:30:02 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-6[53641]: 2024-04-21T23:30:02.847+0000 7f4fdab4a700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:32.252857+0000 front 2024-04-21T23:29:32.253034+0000 (oldest deadline 2024-04-21T23:29:54.553015+0000) 2024-04-21T23:30:03.943 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:03 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:03.628+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:31.573404+0000 front 2024-04-21T23:29:31.573351+0000 (oldest deadline 2024-04-21T23:29:56.873350+0000) 2024-04-21T23:30:03.944 INFO:journalctl@ceph.osd.5.smithi174.stdout:Apr 21 23:30:03 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-5[43337]: 2024-04-21T23:30:03.540+0000 7f5a73535700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.108794+0000 front 2024-04-21T23:29:33.108670+0000 (oldest deadline 2024-04-21T23:29:57.808697+0000) 2024-04-21T23:30:03.944 INFO:journalctl@ceph.osd.5.smithi174.stdout:Apr 21 23:30:03 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-5[43337]: 2024-04-21T23:30:03.540+0000 7f5a73535700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:41.910133+0000 front 2024-04-21T23:29:41.909972+0000 (oldest deadline 2024-04-21T23:30:03.009857+0000) 2024-04-21T23:30:03.944 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:03 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:03.454+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.197048+0000 front 2024-04-21T23:29:33.197157+0000 (oldest deadline 2024-04-21T23:29:59.097025+0000) 2024-04-21T23:30:04.181 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:03 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:03.989+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.927390+0000 front 2024-04-21T23:29:33.927500+0000 (oldest deadline 2024-04-21T23:29:56.827517+0000) 2024-04-21T23:30:04.182 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:03 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:03.989+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:36.828037+0000 front 2024-04-21T23:29:36.827973+0000 (oldest deadline 2024-04-21T23:30:03.228609+0000) 2024-04-21T23:30:04.182 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 21 23:30:03 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-6[53641]: 2024-04-21T23:30:03.891+0000 7f4fdab4a700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:32.252857+0000 front 2024-04-21T23:29:32.253034+0000 (oldest deadline 2024-04-21T23:29:54.553015+0000) 2024-04-21T23:30:04.182 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 21 23:30:03 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-6[53641]: 2024-04-21T23:30:03.891+0000 7f4fdab4a700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:39.254040+0000 front 2024-04-21T23:29:39.254010+0000 (oldest deadline 2024-04-21T23:30:03.353822+0000) 2024-04-21T23:30:04.693 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:04 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:04.435+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.197048+0000 front 2024-04-21T23:29:33.197157+0000 (oldest deadline 2024-04-21T23:29:59.097025+0000) 2024-04-21T23:30:04.943 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:04 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:04.615+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:31.573404+0000 front 2024-04-21T23:29:31.573351+0000 (oldest deadline 2024-04-21T23:29:56.873350+0000) 2024-04-21T23:30:04.944 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:04 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:04.615+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.374892+0000 front 2024-04-21T23:29:40.374834+0000 (oldest deadline 2024-04-21T23:30:04.474654+0000) 2024-04-21T23:30:04.944 INFO:journalctl@ceph.osd.5.smithi174.stdout:Apr 21 23:30:04 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-5[43337]: 2024-04-21T23:30:04.510+0000 7f5a73535700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.108794+0000 front 2024-04-21T23:29:33.108670+0000 (oldest deadline 2024-04-21T23:29:57.808697+0000) 2024-04-21T23:30:04.944 INFO:journalctl@ceph.osd.5.smithi174.stdout:Apr 21 23:30:04 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-5[43337]: 2024-04-21T23:30:04.510+0000 7f5a73535700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:41.910133+0000 front 2024-04-21T23:29:41.909972+0000 (oldest deadline 2024-04-21T23:30:03.009857+0000) 2024-04-21T23:30:05.181 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:05 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:05.011+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.927390+0000 front 2024-04-21T23:29:33.927500+0000 (oldest deadline 2024-04-21T23:29:56.827517+0000) 2024-04-21T23:30:05.182 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:05 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:05.011+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:36.828037+0000 front 2024-04-21T23:29:36.827973+0000 (oldest deadline 2024-04-21T23:30:03.228609+0000) 2024-04-21T23:30:05.182 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 21 23:30:04 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-6[53641]: 2024-04-21T23:30:04.888+0000 7f4fdab4a700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:32.252857+0000 front 2024-04-21T23:29:32.253034+0000 (oldest deadline 2024-04-21T23:29:54.553015+0000) 2024-04-21T23:30:05.182 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 21 23:30:04 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-6[53641]: 2024-04-21T23:30:04.888+0000 7f4fdab4a700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:39.254040+0000 front 2024-04-21T23:29:39.254010+0000 (oldest deadline 2024-04-21T23:30:03.353822+0000) 2024-04-21T23:30:05.693 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:05 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:05.412+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.197048+0000 front 2024-04-21T23:29:33.197157+0000 (oldest deadline 2024-04-21T23:29:59.097025+0000) 2024-04-21T23:30:05.694 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 23:30:05 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-7[49121]: 2024-04-21T23:30:05.516+0000 7ff225f50700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.650347+0000 front 2024-04-21T23:29:40.650470+0000 (oldest deadline 2024-04-21T23:30:05.350454+0000) 2024-04-21T23:30:05.943 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:05 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:05.648+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:31.573404+0000 front 2024-04-21T23:29:31.573351+0000 (oldest deadline 2024-04-21T23:29:56.873350+0000) 2024-04-21T23:30:05.944 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:05 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:05.648+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.374892+0000 front 2024-04-21T23:29:40.374834+0000 (oldest deadline 2024-04-21T23:30:04.474654+0000) 2024-04-21T23:30:05.944 INFO:journalctl@ceph.osd.5.smithi174.stdout:Apr 21 23:30:05 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-5[43337]: 2024-04-21T23:30:05.507+0000 7f5a73535700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.108794+0000 front 2024-04-21T23:29:33.108670+0000 (oldest deadline 2024-04-21T23:29:57.808697+0000) 2024-04-21T23:30:05.944 INFO:journalctl@ceph.osd.5.smithi174.stdout:Apr 21 23:30:05 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-5[43337]: 2024-04-21T23:30:05.507+0000 7f5a73535700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:41.910133+0000 front 2024-04-21T23:29:41.909972+0000 (oldest deadline 2024-04-21T23:30:03.009857+0000) 2024-04-21T23:30:06.181 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 21 23:30:05 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-6[53641]: 2024-04-21T23:30:05.921+0000 7f4fdab4a700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:32.252857+0000 front 2024-04-21T23:29:32.253034+0000 (oldest deadline 2024-04-21T23:29:54.553015+0000) 2024-04-21T23:30:06.182 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 21 23:30:05 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-6[53641]: 2024-04-21T23:30:05.921+0000 7f4fdab4a700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:39.254040+0000 front 2024-04-21T23:29:39.254010+0000 (oldest deadline 2024-04-21T23:30:03.353822+0000) 2024-04-21T23:30:06.182 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:05 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:05.993+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.927390+0000 front 2024-04-21T23:29:33.927500+0000 (oldest deadline 2024-04-21T23:29:56.827517+0000) 2024-04-21T23:30:06.182 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:05 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:05.993+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:36.828037+0000 front 2024-04-21T23:29:36.827973+0000 (oldest deadline 2024-04-21T23:30:03.228609+0000) 2024-04-21T23:30:06.693 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 23:30:06 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-7[49121]: 2024-04-21T23:30:06.497+0000 7ff225f50700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.650347+0000 front 2024-04-21T23:29:40.650470+0000 (oldest deadline 2024-04-21T23:30:05.350454+0000) 2024-04-21T23:30:06.694 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:06 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:06.374+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.197048+0000 front 2024-04-21T23:29:33.197157+0000 (oldest deadline 2024-04-21T23:29:59.097025+0000) 2024-04-21T23:30:06.810 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:06 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:06.648+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:31.573404+0000 front 2024-04-21T23:29:31.573351+0000 (oldest deadline 2024-04-21T23:29:56.873350+0000) 2024-04-21T23:30:06.810 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:06 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:06.648+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.374892+0000 front 2024-04-21T23:29:40.374834+0000 (oldest deadline 2024-04-21T23:30:04.474654+0000) 2024-04-21T23:30:06.811 INFO:journalctl@ceph.osd.5.smithi174.stdout:Apr 21 23:30:06 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-5[43337]: 2024-04-21T23:30:06.552+0000 7f5a73535700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.108794+0000 front 2024-04-21T23:29:33.108670+0000 (oldest deadline 2024-04-21T23:29:57.808697+0000) 2024-04-21T23:30:06.811 INFO:journalctl@ceph.osd.5.smithi174.stdout:Apr 21 23:30:06 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-5[43337]: 2024-04-21T23:30:06.552+0000 7f5a73535700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:41.910133+0000 front 2024-04-21T23:29:41.909972+0000 (oldest deadline 2024-04-21T23:30:03.009857+0000) 2024-04-21T23:30:07.181 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 21 23:30:06 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-6[53641]: 2024-04-21T23:30:06.913+0000 7f4fdab4a700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:32.252857+0000 front 2024-04-21T23:29:32.253034+0000 (oldest deadline 2024-04-21T23:29:54.553015+0000) 2024-04-21T23:30:07.182 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 21 23:30:06 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-6[53641]: 2024-04-21T23:30:06.913+0000 7f4fdab4a700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:39.254040+0000 front 2024-04-21T23:29:39.254010+0000 (oldest deadline 2024-04-21T23:30:03.353822+0000) 2024-04-21T23:30:07.182 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:07 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:07.012+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.927390+0000 front 2024-04-21T23:29:33.927500+0000 (oldest deadline 2024-04-21T23:29:56.827517+0000) 2024-04-21T23:30:07.182 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:07 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:07.012+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:36.828037+0000 front 2024-04-21T23:29:36.827973+0000 (oldest deadline 2024-04-21T23:30:03.228609+0000) 2024-04-21T23:30:07.193 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:06 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:06.811+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:42.041804+0000 front 2024-04-21T23:29:42.041700+0000 (oldest deadline 2024-04-21T23:30:06.741665+0000) 2024-04-21T23:30:07.335 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 21 23:30:07 smithi111 podman[67279]: 2024-04-21 23:30:07.056447059 +0000 UTC m=+5.664086919 container died 214e5d90d6d26724fa0a19b8b6f05b97f736424e6de0b7296025c813b8091c89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-4, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, RELEASE=squid-820a2f8, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2) 2024-04-21T23:30:07.693 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 23:30:07 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-7[49121]: 2024-04-21T23:30:07.495+0000 7ff225f50700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.650347+0000 front 2024-04-21T23:29:40.650470+0000 (oldest deadline 2024-04-21T23:30:05.350454+0000) 2024-04-21T23:30:07.694 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:07 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:07.365+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.197048+0000 front 2024-04-21T23:29:33.197157+0000 (oldest deadline 2024-04-21T23:29:59.097025+0000) 2024-04-21T23:30:07.694 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 21 23:30:07 smithi111 podman[67279]: 2024-04-21 23:30:07.436363576 +0000 UTC m=+6.044003436 container cleanup 214e5d90d6d26724fa0a19b8b6f05b97f736424e6de0b7296025c813b8091c89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-4, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-820a2f8, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-04-21T23:30:07.694 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 21 23:30:07 smithi111 bash[67279]: ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-4 2024-04-21T23:30:07.825 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:07 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:07.657+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:31.573404+0000 front 2024-04-21T23:29:31.573351+0000 (oldest deadline 2024-04-21T23:29:56.873350+0000) 2024-04-21T23:30:07.825 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:07 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:07.657+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.374892+0000 front 2024-04-21T23:29:40.374834+0000 (oldest deadline 2024-04-21T23:30:04.474654+0000) 2024-04-21T23:30:07.826 INFO:journalctl@ceph.osd.5.smithi174.stdout:Apr 21 23:30:07 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-5[43337]: 2024-04-21T23:30:07.519+0000 7f5a73535700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.108794+0000 front 2024-04-21T23:29:33.108670+0000 (oldest deadline 2024-04-21T23:29:57.808697+0000) 2024-04-21T23:30:07.826 INFO:journalctl@ceph.osd.5.smithi174.stdout:Apr 21 23:30:07 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-5[43337]: 2024-04-21T23:30:07.519+0000 7f5a73535700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:41.910133+0000 front 2024-04-21T23:29:41.909972+0000 (oldest deadline 2024-04-21T23:30:03.009857+0000) 2024-04-21T23:30:08.046 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 21 23:30:07 smithi111 podman[67296]: 2024-04-21 23:30:07.696115784 +0000 UTC m=+0.641608531 container remove 214e5d90d6d26724fa0a19b8b6f05b97f736424e6de0b7296025c813b8091c89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-4, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-820a2f8, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2) 2024-04-21T23:30:08.181 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 21 23:30:07 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-6[53641]: 2024-04-21T23:30:07.903+0000 7f4fdab4a700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:32.252857+0000 front 2024-04-21T23:29:32.253034+0000 (oldest deadline 2024-04-21T23:29:54.553015+0000) 2024-04-21T23:30:08.182 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 21 23:30:07 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-6[53641]: 2024-04-21T23:30:07.903+0000 7f4fdab4a700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:39.254040+0000 front 2024-04-21T23:29:39.254010+0000 (oldest deadline 2024-04-21T23:30:03.353822+0000) 2024-04-21T23:30:08.182 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:08 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:08.049+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.927390+0000 front 2024-04-21T23:29:33.927500+0000 (oldest deadline 2024-04-21T23:29:56.827517+0000) 2024-04-21T23:30:08.182 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:08 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:08.049+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:36.828037+0000 front 2024-04-21T23:29:36.827973+0000 (oldest deadline 2024-04-21T23:30:03.228609+0000) 2024-04-21T23:30:08.193 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:07 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:07.827+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:42.041804+0000 front 2024-04-21T23:29:42.041700+0000 (oldest deadline 2024-04-21T23:30:06.741665+0000) 2024-04-21T23:30:08.315 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 21 23:30:08 smithi111 podman[67384]: 2024-04-21 23:30:07.951365049 +0000 UTC m=+0.017690835 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-21T23:30:08.315 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 21 23:30:08 smithi111 podman[67384]: 2024-04-21 23:30:08.063851931 +0000 UTC m=+0.130177709 container create f45aca49b50aadde95c95969e5945019130c5efa0b87eddd1d6bebdd561f4ddb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, ceph=True, GIT_CLEAN=True, RELEASE=squid-820a2f8, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-21T23:30:08.315 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 21 23:30:08 smithi111 podman[67384]: 2024-04-21 23:30:08.316609893 +0000 UTC m=+0.382935672 container init f45aca49b50aadde95c95969e5945019130c5efa0b87eddd1d6bebdd561f4ddb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-4-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-820a2f8, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-21T23:30:08.609 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 23:30:08 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-7[49121]: 2024-04-21T23:30:08.534+0000 7ff225f50700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.650347+0000 front 2024-04-21T23:29:40.650470+0000 (oldest deadline 2024-04-21T23:30:05.350454+0000) 2024-04-21T23:30:08.610 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 21 23:30:08 smithi111 podman[67384]: 2024-04-21 23:30:08.319873873 +0000 UTC m=+0.386199658 container start f45aca49b50aadde95c95969e5945019130c5efa0b87eddd1d6bebdd561f4ddb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, RELEASE=squid-820a2f8, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-21T23:30:08.610 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 21 23:30:08 smithi111 podman[67384]: 2024-04-21 23:30:08.41669523 +0000 UTC m=+0.483021026 container attach f45aca49b50aadde95c95969e5945019130c5efa0b87eddd1d6bebdd561f4ddb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-820a2f8, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T23:30:08.610 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:08 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:08.324+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.197048+0000 front 2024-04-21T23:29:33.197157+0000 (oldest deadline 2024-04-21T23:29:59.097025+0000) 2024-04-21T23:30:08.854 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:08 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:08.627+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:31.573404+0000 front 2024-04-21T23:29:31.573351+0000 (oldest deadline 2024-04-21T23:29:56.873350+0000) 2024-04-21T23:30:08.855 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:08 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:08.627+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.374892+0000 front 2024-04-21T23:29:40.374834+0000 (oldest deadline 2024-04-21T23:30:04.474654+0000) 2024-04-21T23:30:08.855 INFO:journalctl@ceph.osd.5.smithi174.stdout:Apr 21 23:30:08 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-5[43337]: 2024-04-21T23:30:08.536+0000 7f5a73535700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.108794+0000 front 2024-04-21T23:29:33.108670+0000 (oldest deadline 2024-04-21T23:29:57.808697+0000) 2024-04-21T23:30:08.855 INFO:journalctl@ceph.osd.5.smithi174.stdout:Apr 21 23:30:08 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-5[43337]: 2024-04-21T23:30:08.536+0000 7f5a73535700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:41.910133+0000 front 2024-04-21T23:29:41.909972+0000 (oldest deadline 2024-04-21T23:30:03.009857+0000) 2024-04-21T23:30:08.901 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 21 23:30:08 smithi111 podman[67384]: 2024-04-21 23:30:08.61203073 +0000 UTC m=+0.678356513 container died f45aca49b50aadde95c95969e5945019130c5efa0b87eddd1d6bebdd561f4ddb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-4-deactivate, RELEASE=squid-820a2f8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T23:30:09.181 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 21 23:30:08 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-6[53641]: 2024-04-21T23:30:08.867+0000 7f4fdab4a700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:32.252857+0000 front 2024-04-21T23:29:32.253034+0000 (oldest deadline 2024-04-21T23:29:54.553015+0000) 2024-04-21T23:30:09.182 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 21 23:30:08 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-6[53641]: 2024-04-21T23:30:08.867+0000 7f4fdab4a700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:39.254040+0000 front 2024-04-21T23:29:39.254010+0000 (oldest deadline 2024-04-21T23:30:03.353822+0000) 2024-04-21T23:30:09.182 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:09 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:09.069+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.927390+0000 front 2024-04-21T23:29:33.927500+0000 (oldest deadline 2024-04-21T23:29:56.827517+0000) 2024-04-21T23:30:09.182 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:09 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:09.069+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:36.828037+0000 front 2024-04-21T23:29:36.827973+0000 (oldest deadline 2024-04-21T23:30:03.228609+0000) 2024-04-21T23:30:09.193 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:08 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:08.855+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:42.041804+0000 front 2024-04-21T23:29:42.041700+0000 (oldest deadline 2024-04-21T23:30:06.741665+0000) 2024-04-21T23:30:09.235 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d814e4dc-0033-11ef-bc93-c7b262605968@osd.4.service' 2024-04-21T23:30:09.273 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T23:30:09.274 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-21T23:30:09.274 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-21T23:30:09.274 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ceph-d814e4dc-0033-11ef-bc93-c7b262605968@osd.5 2024-04-21T23:30:09.549 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:09 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:09.357+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.197048+0000 front 2024-04-21T23:29:33.197157+0000 (oldest deadline 2024-04-21T23:29:59.097025+0000) 2024-04-21T23:30:09.574 INFO:journalctl@ceph.osd.5.smithi174.stdout:Apr 21 23:30:09 smithi174 systemd[1]: Stopping Ceph osd.5 for d814e4dc-0033-11ef-bc93-c7b262605968... 2024-04-21T23:30:09.899 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:09 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:09.627+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:31.573404+0000 front 2024-04-21T23:29:31.573351+0000 (oldest deadline 2024-04-21T23:29:56.873350+0000) 2024-04-21T23:30:09.899 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:09 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:09.627+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.374892+0000 front 2024-04-21T23:29:40.374834+0000 (oldest deadline 2024-04-21T23:30:04.474654+0000) 2024-04-21T23:30:09.900 INFO:journalctl@ceph.osd.5.smithi174.stdout:Apr 21 23:30:09 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-5[43337]: 2024-04-21T23:30:09.575+0000 7f5a73535700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.108794+0000 front 2024-04-21T23:29:33.108670+0000 (oldest deadline 2024-04-21T23:29:57.808697+0000) 2024-04-21T23:30:09.900 INFO:journalctl@ceph.osd.5.smithi174.stdout:Apr 21 23:30:09 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-5[43337]: 2024-04-21T23:30:09.575+0000 7f5a73535700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:41.910133+0000 front 2024-04-21T23:29:41.909972+0000 (oldest deadline 2024-04-21T23:30:03.009857+0000) 2024-04-21T23:30:09.900 INFO:journalctl@ceph.osd.5.smithi174.stdout:Apr 21 23:30:09 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-5[43337]: 2024-04-21T23:30:09.616+0000 7f5a7af49700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T23:30:09.900 INFO:journalctl@ceph.osd.5.smithi174.stdout:Apr 21 23:30:09 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-5[43337]: 2024-04-21T23:30:09.616+0000 7f5a7af49700 -1 osd.5 108 *** Got signal Terminated *** 2024-04-21T23:30:09.900 INFO:journalctl@ceph.osd.5.smithi174.stdout:Apr 21 23:30:09 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-5[43337]: 2024-04-21T23:30:09.616+0000 7f5a7af49700 -1 osd.5 108 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-21T23:30:09.944 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 23:30:09 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-7[49121]: 2024-04-21T23:30:09.550+0000 7ff225f50700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.650347+0000 front 2024-04-21T23:29:40.650470+0000 (oldest deadline 2024-04-21T23:30:05.350454+0000) 2024-04-21T23:30:10.181 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 21 23:30:09 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-6[53641]: 2024-04-21T23:30:09.846+0000 7f4fdab4a700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:32.252857+0000 front 2024-04-21T23:29:32.253034+0000 (oldest deadline 2024-04-21T23:29:54.553015+0000) 2024-04-21T23:30:10.182 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 21 23:30:09 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-6[53641]: 2024-04-21T23:30:09.846+0000 7f4fdab4a700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:39.254040+0000 front 2024-04-21T23:29:39.254010+0000 (oldest deadline 2024-04-21T23:30:03.353822+0000) 2024-04-21T23:30:10.182 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:10 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:10.053+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.927390+0000 front 2024-04-21T23:29:33.927500+0000 (oldest deadline 2024-04-21T23:29:56.827517+0000) 2024-04-21T23:30:10.182 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:10 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:10.053+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:36.828037+0000 front 2024-04-21T23:29:36.827973+0000 (oldest deadline 2024-04-21T23:30:03.228609+0000) 2024-04-21T23:30:10.193 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:09 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:09.900+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:42.041804+0000 front 2024-04-21T23:29:42.041700+0000 (oldest deadline 2024-04-21T23:30:06.741665+0000) 2024-04-21T23:30:10.693 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:10 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:10.345+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.197048+0000 front 2024-04-21T23:29:33.197157+0000 (oldest deadline 2024-04-21T23:29:59.097025+0000) 2024-04-21T23:30:10.694 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 23:30:10 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-7[49121]: 2024-04-21T23:30:10.593+0000 7ff225f50700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.650347+0000 front 2024-04-21T23:29:40.650470+0000 (oldest deadline 2024-04-21T23:30:05.350454+0000) 2024-04-21T23:30:10.865 INFO:journalctl@ceph.osd.5.smithi174.stdout:Apr 21 23:30:10 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-5[43337]: 2024-04-21T23:30:10.603+0000 7f5a73535700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.108794+0000 front 2024-04-21T23:29:33.108670+0000 (oldest deadline 2024-04-21T23:29:57.808697+0000) 2024-04-21T23:30:10.866 INFO:journalctl@ceph.osd.5.smithi174.stdout:Apr 21 23:30:10 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-5[43337]: 2024-04-21T23:30:10.603+0000 7f5a73535700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:41.910133+0000 front 2024-04-21T23:29:41.909972+0000 (oldest deadline 2024-04-21T23:30:03.009857+0000) 2024-04-21T23:30:10.866 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:10 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:10.658+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:31.573404+0000 front 2024-04-21T23:29:31.573351+0000 (oldest deadline 2024-04-21T23:29:56.873350+0000) 2024-04-21T23:30:10.866 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:10 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:10.658+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.374892+0000 front 2024-04-21T23:29:40.374834+0000 (oldest deadline 2024-04-21T23:30:04.474654+0000) 2024-04-21T23:30:11.181 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 21 23:30:10 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-6[53641]: 2024-04-21T23:30:10.800+0000 7f4fdab4a700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:32.252857+0000 front 2024-04-21T23:29:32.253034+0000 (oldest deadline 2024-04-21T23:29:54.553015+0000) 2024-04-21T23:30:11.182 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 21 23:30:10 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-6[53641]: 2024-04-21T23:30:10.800+0000 7f4fdab4a700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:39.254040+0000 front 2024-04-21T23:29:39.254010+0000 (oldest deadline 2024-04-21T23:30:03.353822+0000) 2024-04-21T23:30:11.182 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:11 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:11.015+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.927390+0000 front 2024-04-21T23:29:33.927500+0000 (oldest deadline 2024-04-21T23:29:56.827517+0000) 2024-04-21T23:30:11.182 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:11 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:11.015+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:36.828037+0000 front 2024-04-21T23:29:36.827973+0000 (oldest deadline 2024-04-21T23:30:03.228609+0000) 2024-04-21T23:30:11.193 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:10 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:10.866+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:42.041804+0000 front 2024-04-21T23:29:42.041700+0000 (oldest deadline 2024-04-21T23:30:06.741665+0000) 2024-04-21T23:30:11.693 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:11 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:11.395+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.197048+0000 front 2024-04-21T23:29:33.197157+0000 (oldest deadline 2024-04-21T23:29:59.097025+0000) 2024-04-21T23:30:11.694 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 23:30:11 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-7[49121]: 2024-04-21T23:30:11.570+0000 7ff225f50700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.650347+0000 front 2024-04-21T23:29:40.650470+0000 (oldest deadline 2024-04-21T23:30:05.350454+0000) 2024-04-21T23:30:11.694 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 23:30:11 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-7[49121]: 2024-04-21T23:30:11.570+0000 7ff225f50700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:48.751995+0000 front 2024-04-21T23:29:48.752022+0000 (oldest deadline 2024-04-21T23:30:11.052223+0000) 2024-04-21T23:30:11.878 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:11 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:11.682+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:31.573404+0000 front 2024-04-21T23:29:31.573351+0000 (oldest deadline 2024-04-21T23:29:56.873350+0000) 2024-04-21T23:30:11.879 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:11 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:11.682+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.374892+0000 front 2024-04-21T23:29:40.374834+0000 (oldest deadline 2024-04-21T23:30:04.474654+0000) 2024-04-21T23:30:11.879 INFO:journalctl@ceph.osd.5.smithi174.stdout:Apr 21 23:30:11 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-5[43337]: 2024-04-21T23:30:11.626+0000 7f5a73535700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.108794+0000 front 2024-04-21T23:29:33.108670+0000 (oldest deadline 2024-04-21T23:29:57.808697+0000) 2024-04-21T23:30:11.879 INFO:journalctl@ceph.osd.5.smithi174.stdout:Apr 21 23:30:11 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-5[43337]: 2024-04-21T23:30:11.626+0000 7f5a73535700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:41.910133+0000 front 2024-04-21T23:29:41.909972+0000 (oldest deadline 2024-04-21T23:30:03.009857+0000) 2024-04-21T23:30:11.879 INFO:journalctl@ceph.osd.5.smithi174.stdout:Apr 21 23:30:11 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-5[43337]: 2024-04-21T23:30:11.626+0000 7f5a73535700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:49.911980+0000 front 2024-04-21T23:29:49.911923+0000 (oldest deadline 2024-04-21T23:30:11.012146+0000) 2024-04-21T23:30:12.181 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 21 23:30:11 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-6[53641]: 2024-04-21T23:30:11.790+0000 7f4fdab4a700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:32.252857+0000 front 2024-04-21T23:29:32.253034+0000 (oldest deadline 2024-04-21T23:29:54.553015+0000) 2024-04-21T23:30:12.182 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 21 23:30:11 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-6[53641]: 2024-04-21T23:30:11.790+0000 7f4fdab4a700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:39.254040+0000 front 2024-04-21T23:29:39.254010+0000 (oldest deadline 2024-04-21T23:30:03.353822+0000) 2024-04-21T23:30:12.182 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:12 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:12.008+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.927390+0000 front 2024-04-21T23:29:33.927500+0000 (oldest deadline 2024-04-21T23:29:56.827517+0000) 2024-04-21T23:30:12.182 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:12 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:12.008+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:36.828037+0000 front 2024-04-21T23:29:36.827973+0000 (oldest deadline 2024-04-21T23:30:03.228609+0000) 2024-04-21T23:30:12.182 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:12 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:12.008+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:46.730173+0000 front 2024-04-21T23:29:46.729971+0000 (oldest deadline 2024-04-21T23:30:11.330259+0000) 2024-04-21T23:30:12.193 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:11 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:11.879+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:42.041804+0000 front 2024-04-21T23:29:42.041700+0000 (oldest deadline 2024-04-21T23:30:06.741665+0000) 2024-04-21T23:30:12.693 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:12 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:12.438+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.197048+0000 front 2024-04-21T23:29:33.197157+0000 (oldest deadline 2024-04-21T23:29:59.097025+0000) 2024-04-21T23:30:12.694 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 23:30:12 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-7[49121]: 2024-04-21T23:30:12.535+0000 7ff225f50700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.650347+0000 front 2024-04-21T23:29:40.650470+0000 (oldest deadline 2024-04-21T23:30:05.350454+0000) 2024-04-21T23:30:12.694 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 23:30:12 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-7[49121]: 2024-04-21T23:30:12.535+0000 7ff225f50700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:48.751995+0000 front 2024-04-21T23:29:48.752022+0000 (oldest deadline 2024-04-21T23:30:11.052223+0000) 2024-04-21T23:30:12.893 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:12 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:12.641+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:31.573404+0000 front 2024-04-21T23:29:31.573351+0000 (oldest deadline 2024-04-21T23:29:56.873350+0000) 2024-04-21T23:30:12.893 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:12 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:12.641+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.374892+0000 front 2024-04-21T23:29:40.374834+0000 (oldest deadline 2024-04-21T23:30:04.474654+0000) 2024-04-21T23:30:12.894 INFO:journalctl@ceph.osd.5.smithi174.stdout:Apr 21 23:30:12 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-5[43337]: 2024-04-21T23:30:12.579+0000 7f5a73535700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.108794+0000 front 2024-04-21T23:29:33.108670+0000 (oldest deadline 2024-04-21T23:29:57.808697+0000) 2024-04-21T23:30:12.894 INFO:journalctl@ceph.osd.5.smithi174.stdout:Apr 21 23:30:12 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-5[43337]: 2024-04-21T23:30:12.579+0000 7f5a73535700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:41.910133+0000 front 2024-04-21T23:29:41.909972+0000 (oldest deadline 2024-04-21T23:30:03.009857+0000) 2024-04-21T23:30:12.894 INFO:journalctl@ceph.osd.5.smithi174.stdout:Apr 21 23:30:12 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-5[43337]: 2024-04-21T23:30:12.579+0000 7f5a73535700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:49.911980+0000 front 2024-04-21T23:29:49.911923+0000 (oldest deadline 2024-04-21T23:30:11.012146+0000) 2024-04-21T23:30:13.182 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 21 23:30:12 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-6[53641]: 2024-04-21T23:30:12.792+0000 7f4fdab4a700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:32.252857+0000 front 2024-04-21T23:29:32.253034+0000 (oldest deadline 2024-04-21T23:29:54.553015+0000) 2024-04-21T23:30:13.182 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 21 23:30:12 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-6[53641]: 2024-04-21T23:30:12.792+0000 7f4fdab4a700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:39.254040+0000 front 2024-04-21T23:29:39.254010+0000 (oldest deadline 2024-04-21T23:30:03.353822+0000) 2024-04-21T23:30:13.182 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 21 23:30:12 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-6[53641]: 2024-04-21T23:30:12.792+0000 7f4fdab4a700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:50.355743+0000 front 2024-04-21T23:29:50.355516+0000 (oldest deadline 2024-04-21T23:30:12.055457+0000) 2024-04-21T23:30:13.182 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:13 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:13.002+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.927390+0000 front 2024-04-21T23:29:33.927500+0000 (oldest deadline 2024-04-21T23:29:56.827517+0000) 2024-04-21T23:30:13.182 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:13 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:13.002+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:36.828037+0000 front 2024-04-21T23:29:36.827973+0000 (oldest deadline 2024-04-21T23:30:03.228609+0000) 2024-04-21T23:30:13.183 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:13 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:13.002+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:46.730173+0000 front 2024-04-21T23:29:46.729971+0000 (oldest deadline 2024-04-21T23:30:11.330259+0000) 2024-04-21T23:30:13.193 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:12 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:12.894+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:42.041804+0000 front 2024-04-21T23:29:42.041700+0000 (oldest deadline 2024-04-21T23:30:06.741665+0000) 2024-04-21T23:30:13.193 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:12 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:12.894+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:49.642677+0000 front 2024-04-21T23:29:49.642702+0000 (oldest deadline 2024-04-21T23:30:11.942839+0000) 2024-04-21T23:30:13.693 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:13 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:13.438+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.197048+0000 front 2024-04-21T23:29:33.197157+0000 (oldest deadline 2024-04-21T23:29:59.097025+0000) 2024-04-21T23:30:13.694 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:13 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:13.438+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:47.298757+0000 front 2024-04-21T23:29:47.298709+0000 (oldest deadline 2024-04-21T23:30:13.198900+0000) 2024-04-21T23:30:13.694 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 23:30:13 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-7[49121]: 2024-04-21T23:30:13.543+0000 7ff225f50700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.650347+0000 front 2024-04-21T23:29:40.650470+0000 (oldest deadline 2024-04-21T23:30:05.350454+0000) 2024-04-21T23:30:13.694 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 23:30:13 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-7[49121]: 2024-04-21T23:30:13.543+0000 7ff225f50700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:48.751995+0000 front 2024-04-21T23:29:48.752022+0000 (oldest deadline 2024-04-21T23:30:11.052223+0000) 2024-04-21T23:30:13.898 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:13 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:13.657+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:31.573404+0000 front 2024-04-21T23:29:31.573351+0000 (oldest deadline 2024-04-21T23:29:56.873350+0000) 2024-04-21T23:30:13.898 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:13 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:13.657+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.374892+0000 front 2024-04-21T23:29:40.374834+0000 (oldest deadline 2024-04-21T23:30:04.474654+0000) 2024-04-21T23:30:13.898 INFO:journalctl@ceph.osd.5.smithi174.stdout:Apr 21 23:30:13 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-5[43337]: 2024-04-21T23:30:13.620+0000 7f5a73535700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.108794+0000 front 2024-04-21T23:29:33.108670+0000 (oldest deadline 2024-04-21T23:29:57.808697+0000) 2024-04-21T23:30:13.899 INFO:journalctl@ceph.osd.5.smithi174.stdout:Apr 21 23:30:13 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-5[43337]: 2024-04-21T23:30:13.620+0000 7f5a73535700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:41.910133+0000 front 2024-04-21T23:29:41.909972+0000 (oldest deadline 2024-04-21T23:30:03.009857+0000) 2024-04-21T23:30:13.899 INFO:journalctl@ceph.osd.5.smithi174.stdout:Apr 21 23:30:13 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-5[43337]: 2024-04-21T23:30:13.620+0000 7f5a73535700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:49.911980+0000 front 2024-04-21T23:29:49.911923+0000 (oldest deadline 2024-04-21T23:30:11.012146+0000) 2024-04-21T23:30:14.181 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 21 23:30:13 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-6[53641]: 2024-04-21T23:30:13.834+0000 7f4fdab4a700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:32.252857+0000 front 2024-04-21T23:29:32.253034+0000 (oldest deadline 2024-04-21T23:29:54.553015+0000) 2024-04-21T23:30:14.182 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 21 23:30:13 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-6[53641]: 2024-04-21T23:30:13.834+0000 7f4fdab4a700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:39.254040+0000 front 2024-04-21T23:29:39.254010+0000 (oldest deadline 2024-04-21T23:30:03.353822+0000) 2024-04-21T23:30:14.182 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 21 23:30:13 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-6[53641]: 2024-04-21T23:30:13.834+0000 7f4fdab4a700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:50.355743+0000 front 2024-04-21T23:29:50.355516+0000 (oldest deadline 2024-04-21T23:30:12.055457+0000) 2024-04-21T23:30:14.182 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:14 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:14.046+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.927390+0000 front 2024-04-21T23:29:33.927500+0000 (oldest deadline 2024-04-21T23:29:56.827517+0000) 2024-04-21T23:30:14.182 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:14 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:14.046+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:36.828037+0000 front 2024-04-21T23:29:36.827973+0000 (oldest deadline 2024-04-21T23:30:03.228609+0000) 2024-04-21T23:30:14.183 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:14 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:14.046+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:46.730173+0000 front 2024-04-21T23:29:46.729971+0000 (oldest deadline 2024-04-21T23:30:11.330259+0000) 2024-04-21T23:30:14.193 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:13 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:13.899+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:42.041804+0000 front 2024-04-21T23:29:42.041700+0000 (oldest deadline 2024-04-21T23:30:06.741665+0000) 2024-04-21T23:30:14.193 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:13 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:13.899+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:49.642677+0000 front 2024-04-21T23:29:49.642702+0000 (oldest deadline 2024-04-21T23:30:11.942839+0000) 2024-04-21T23:30:14.693 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 23:30:14 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-7[49121]: 2024-04-21T23:30:14.512+0000 7ff225f50700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.650347+0000 front 2024-04-21T23:29:40.650470+0000 (oldest deadline 2024-04-21T23:30:05.350454+0000) 2024-04-21T23:30:14.694 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 23:30:14 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-7[49121]: 2024-04-21T23:30:14.512+0000 7ff225f50700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:48.751995+0000 front 2024-04-21T23:29:48.752022+0000 (oldest deadline 2024-04-21T23:30:11.052223+0000) 2024-04-21T23:30:14.694 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:14 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:14.400+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.197048+0000 front 2024-04-21T23:29:33.197157+0000 (oldest deadline 2024-04-21T23:29:59.097025+0000) 2024-04-21T23:30:14.694 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:14 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:14.400+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:47.298757+0000 front 2024-04-21T23:29:47.298709+0000 (oldest deadline 2024-04-21T23:30:13.198900+0000) 2024-04-21T23:30:14.943 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:14 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:14.850+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:42.041804+0000 front 2024-04-21T23:29:42.041700+0000 (oldest deadline 2024-04-21T23:30:06.741665+0000) 2024-04-21T23:30:14.943 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:14 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:14.850+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:49.642677+0000 front 2024-04-21T23:29:49.642702+0000 (oldest deadline 2024-04-21T23:30:11.942839+0000) 2024-04-21T23:30:14.944 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:14 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:14.632+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:31.573404+0000 front 2024-04-21T23:29:31.573351+0000 (oldest deadline 2024-04-21T23:29:56.873350+0000) 2024-04-21T23:30:14.944 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:14 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:14.632+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.374892+0000 front 2024-04-21T23:29:40.374834+0000 (oldest deadline 2024-04-21T23:30:04.474654+0000) 2024-04-21T23:30:14.944 INFO:journalctl@ceph.osd.5.smithi174.stdout:Apr 21 23:30:14 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-5[43337]: 2024-04-21T23:30:14.616+0000 7f5a73535700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.108794+0000 front 2024-04-21T23:29:33.108670+0000 (oldest deadline 2024-04-21T23:29:57.808697+0000) 2024-04-21T23:30:14.944 INFO:journalctl@ceph.osd.5.smithi174.stdout:Apr 21 23:30:14 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-5[43337]: 2024-04-21T23:30:14.616+0000 7f5a73535700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:41.910133+0000 front 2024-04-21T23:29:41.909972+0000 (oldest deadline 2024-04-21T23:30:03.009857+0000) 2024-04-21T23:30:14.944 INFO:journalctl@ceph.osd.5.smithi174.stdout:Apr 21 23:30:14 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-5[43337]: 2024-04-21T23:30:14.616+0000 7f5a73535700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:49.911980+0000 front 2024-04-21T23:29:49.911923+0000 (oldest deadline 2024-04-21T23:30:11.012146+0000) 2024-04-21T23:30:15.181 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 21 23:30:14 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-6[53641]: 2024-04-21T23:30:14.839+0000 7f4fdab4a700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:32.252857+0000 front 2024-04-21T23:29:32.253034+0000 (oldest deadline 2024-04-21T23:29:54.553015+0000) 2024-04-21T23:30:15.182 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 21 23:30:14 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-6[53641]: 2024-04-21T23:30:14.839+0000 7f4fdab4a700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:39.254040+0000 front 2024-04-21T23:29:39.254010+0000 (oldest deadline 2024-04-21T23:30:03.353822+0000) 2024-04-21T23:30:15.182 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 21 23:30:14 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-6[53641]: 2024-04-21T23:30:14.839+0000 7f4fdab4a700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:50.355743+0000 front 2024-04-21T23:29:50.355516+0000 (oldest deadline 2024-04-21T23:30:12.055457+0000) 2024-04-21T23:30:15.182 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:15 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:15.081+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.927390+0000 front 2024-04-21T23:29:33.927500+0000 (oldest deadline 2024-04-21T23:29:56.827517+0000) 2024-04-21T23:30:15.182 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:15 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:15.081+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:36.828037+0000 front 2024-04-21T23:29:36.827973+0000 (oldest deadline 2024-04-21T23:30:03.228609+0000) 2024-04-21T23:30:15.182 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:15 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:15.081+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:46.730173+0000 front 2024-04-21T23:29:46.729971+0000 (oldest deadline 2024-04-21T23:30:11.330259+0000) 2024-04-21T23:30:15.258 INFO:journalctl@ceph.osd.5.smithi174.stdout:Apr 21 23:30:14 smithi174 podman[68119]: 2024-04-21 23:30:14.960414531 +0000 UTC m=+5.488672557 container died 1752207a0d70fcd68c62a7d082c2a3010e0563cefdd6886ea232232434bcc3e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-5, RELEASE=squid-820a2f8, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T23:30:15.566 INFO:journalctl@ceph.osd.5.smithi174.stdout:Apr 21 23:30:15 smithi174 podman[68119]: 2024-04-21 23:30:15.359182862 +0000 UTC m=+5.887440893 container cleanup 1752207a0d70fcd68c62a7d082c2a3010e0563cefdd6886ea232232434bcc3e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-5, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, RELEASE=squid-820a2f8, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-21T23:30:15.566 INFO:journalctl@ceph.osd.5.smithi174.stdout:Apr 21 23:30:15 smithi174 bash[68119]: ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-5 2024-04-21T23:30:15.694 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 23:30:15 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-7[49121]: 2024-04-21T23:30:15.530+0000 7ff225f50700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.650347+0000 front 2024-04-21T23:29:40.650470+0000 (oldest deadline 2024-04-21T23:30:05.350454+0000) 2024-04-21T23:30:15.694 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 23:30:15 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-7[49121]: 2024-04-21T23:30:15.530+0000 7ff225f50700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:48.751995+0000 front 2024-04-21T23:29:48.752022+0000 (oldest deadline 2024-04-21T23:30:11.052223+0000) 2024-04-21T23:30:15.694 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:15 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:15.356+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.197048+0000 front 2024-04-21T23:29:33.197157+0000 (oldest deadline 2024-04-21T23:29:59.097025+0000) 2024-04-21T23:30:15.694 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:15 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:15.356+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:47.298757+0000 front 2024-04-21T23:29:47.298709+0000 (oldest deadline 2024-04-21T23:30:13.198900+0000) 2024-04-21T23:30:15.849 INFO:journalctl@ceph.osd.5.smithi174.stdout:Apr 21 23:30:15 smithi174 podman[68136]: 2024-04-21 23:30:15.568633229 +0000 UTC m=+0.603401475 container remove 1752207a0d70fcd68c62a7d082c2a3010e0563cefdd6886ea232232434bcc3e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-820a2f8) 2024-04-21T23:30:15.849 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:15 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:15.600+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:31.573404+0000 front 2024-04-21T23:29:31.573351+0000 (oldest deadline 2024-04-21T23:29:56.873350+0000) 2024-04-21T23:30:15.849 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:15 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:15.600+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.374892+0000 front 2024-04-21T23:29:40.374834+0000 (oldest deadline 2024-04-21T23:30:04.474654+0000) 2024-04-21T23:30:16.109 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:15 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:15.849+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:42.041804+0000 front 2024-04-21T23:29:42.041700+0000 (oldest deadline 2024-04-21T23:30:06.741665+0000) 2024-04-21T23:30:16.109 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:15 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:15.849+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:49.642677+0000 front 2024-04-21T23:29:49.642702+0000 (oldest deadline 2024-04-21T23:30:11.942839+0000) 2024-04-21T23:30:16.109 INFO:journalctl@ceph.osd.5.smithi174.stdout:Apr 21 23:30:15 smithi174 podman[68226]: 2024-04-21 23:30:15.885289117 +0000 UTC m=+0.017808968 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-21T23:30:16.109 INFO:journalctl@ceph.osd.5.smithi174.stdout:Apr 21 23:30:15 smithi174 podman[68226]: 2024-04-21 23:30:15.994450055 +0000 UTC m=+0.126969893 container create f05048f1ff2c553935034f20b59213f6fa2c0cd94587e95ecbd9a78b97ab88ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-820a2f8, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T23:30:16.182 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 21 23:30:15 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-6[53641]: 2024-04-21T23:30:15.857+0000 7f4fdab4a700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:32.252857+0000 front 2024-04-21T23:29:32.253034+0000 (oldest deadline 2024-04-21T23:29:54.553015+0000) 2024-04-21T23:30:16.182 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 21 23:30:15 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-6[53641]: 2024-04-21T23:30:15.857+0000 7f4fdab4a700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:39.254040+0000 front 2024-04-21T23:29:39.254010+0000 (oldest deadline 2024-04-21T23:30:03.353822+0000) 2024-04-21T23:30:16.182 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 21 23:30:15 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-6[53641]: 2024-04-21T23:30:15.857+0000 7f4fdab4a700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:50.355743+0000 front 2024-04-21T23:29:50.355516+0000 (oldest deadline 2024-04-21T23:30:12.055457+0000) 2024-04-21T23:30:16.182 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:16 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:16.033+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.927390+0000 front 2024-04-21T23:29:33.927500+0000 (oldest deadline 2024-04-21T23:29:56.827517+0000) 2024-04-21T23:30:16.183 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:16 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:16.033+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:36.828037+0000 front 2024-04-21T23:29:36.827973+0000 (oldest deadline 2024-04-21T23:30:03.228609+0000) 2024-04-21T23:30:16.183 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:16 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:16.033+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:46.730173+0000 front 2024-04-21T23:29:46.729971+0000 (oldest deadline 2024-04-21T23:30:11.330259+0000) 2024-04-21T23:30:16.444 INFO:journalctl@ceph.osd.5.smithi174.stdout:Apr 21 23:30:16 smithi174 podman[68226]: 2024-04-21 23:30:16.23993438 +0000 UTC m=+0.372454226 container init f05048f1ff2c553935034f20b59213f6fa2c0cd94587e95ecbd9a78b97ab88ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-5-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, RELEASE=squid-820a2f8, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T23:30:16.444 INFO:journalctl@ceph.osd.5.smithi174.stdout:Apr 21 23:30:16 smithi174 podman[68226]: 2024-04-21 23:30:16.243068466 +0000 UTC m=+0.375588312 container start f05048f1ff2c553935034f20b59213f6fa2c0cd94587e95ecbd9a78b97ab88ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, RELEASE=squid-820a2f8) 2024-04-21T23:30:16.444 INFO:journalctl@ceph.osd.5.smithi174.stdout:Apr 21 23:30:16 smithi174 podman[68226]: 2024-04-21 23:30:16.347104916 +0000 UTC m=+0.479624770 container attach f05048f1ff2c553935034f20b59213f6fa2c0cd94587e95ecbd9a78b97ab88ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-5-deactivate, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, RELEASE=squid-820a2f8, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-21T23:30:16.693 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:16 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:16.402+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.197048+0000 front 2024-04-21T23:29:33.197157+0000 (oldest deadline 2024-04-21T23:29:59.097025+0000) 2024-04-21T23:30:16.694 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:16 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:16.402+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:47.298757+0000 front 2024-04-21T23:29:47.298709+0000 (oldest deadline 2024-04-21T23:30:13.198900+0000) 2024-04-21T23:30:16.694 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 23:30:16 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-7[49121]: 2024-04-21T23:30:16.484+0000 7ff225f50700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.650347+0000 front 2024-04-21T23:29:40.650470+0000 (oldest deadline 2024-04-21T23:30:05.350454+0000) 2024-04-21T23:30:16.694 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 23:30:16 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-7[49121]: 2024-04-21T23:30:16.484+0000 7ff225f50700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:48.751995+0000 front 2024-04-21T23:29:48.752022+0000 (oldest deadline 2024-04-21T23:30:11.052223+0000) 2024-04-21T23:30:16.821 INFO:journalctl@ceph.osd.5.smithi174.stdout:Apr 21 23:30:16 smithi174 podman[68226]: 2024-04-21 23:30:16.542697965 +0000 UTC m=+0.675217805 container died f05048f1ff2c553935034f20b59213f6fa2c0cd94587e95ecbd9a78b97ab88ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-5-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-820a2f8, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-21T23:30:16.821 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:16 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:16.573+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:31.573404+0000 front 2024-04-21T23:29:31.573351+0000 (oldest deadline 2024-04-21T23:29:56.873350+0000) 2024-04-21T23:30:16.822 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:16 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:16.574+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.374892+0000 front 2024-04-21T23:29:40.374834+0000 (oldest deadline 2024-04-21T23:30:04.474654+0000) 2024-04-21T23:30:17.148 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:16 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:16.822+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:42.041804+0000 front 2024-04-21T23:29:42.041700+0000 (oldest deadline 2024-04-21T23:30:06.741665+0000) 2024-04-21T23:30:17.149 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:16 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:16.822+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:49.642677+0000 front 2024-04-21T23:29:49.642702+0000 (oldest deadline 2024-04-21T23:30:11.942839+0000) 2024-04-21T23:30:17.172 DEBUG:teuthology.orchestra.run.smithi174:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d814e4dc-0033-11ef-bc93-c7b262605968@osd.5.service' 2024-04-21T23:30:17.181 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 21 23:30:16 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-6[53641]: 2024-04-21T23:30:16.877+0000 7f4fdab4a700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:32.252857+0000 front 2024-04-21T23:29:32.253034+0000 (oldest deadline 2024-04-21T23:29:54.553015+0000) 2024-04-21T23:30:17.181 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 21 23:30:16 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-6[53641]: 2024-04-21T23:30:16.877+0000 7f4fdab4a700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:39.254040+0000 front 2024-04-21T23:29:39.254010+0000 (oldest deadline 2024-04-21T23:30:03.353822+0000) 2024-04-21T23:30:17.181 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 21 23:30:16 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-6[53641]: 2024-04-21T23:30:16.877+0000 7f4fdab4a700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:50.355743+0000 front 2024-04-21T23:29:50.355516+0000 (oldest deadline 2024-04-21T23:30:12.055457+0000) 2024-04-21T23:30:17.182 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:17 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:17.045+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.927390+0000 front 2024-04-21T23:29:33.927500+0000 (oldest deadline 2024-04-21T23:29:56.827517+0000) 2024-04-21T23:30:17.182 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:17 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:17.045+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:36.828037+0000 front 2024-04-21T23:29:36.827973+0000 (oldest deadline 2024-04-21T23:30:03.228609+0000) 2024-04-21T23:30:17.182 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:17 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:17.045+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:46.730173+0000 front 2024-04-21T23:29:46.729971+0000 (oldest deadline 2024-04-21T23:30:11.330259+0000) 2024-04-21T23:30:17.208 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T23:30:17.209 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-21T23:30:17.209 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-21T23:30:17.209 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-d814e4dc-0033-11ef-bc93-c7b262605968@osd.6 2024-04-21T23:30:17.531 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 21 23:30:17 smithi040 systemd[1]: Stopping Ceph osd.6 for d814e4dc-0033-11ef-bc93-c7b262605968... 2024-04-21T23:30:17.693 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:17 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:17.357+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.197048+0000 front 2024-04-21T23:29:33.197157+0000 (oldest deadline 2024-04-21T23:29:59.097025+0000) 2024-04-21T23:30:17.694 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:17 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:17.357+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:47.298757+0000 front 2024-04-21T23:29:47.298709+0000 (oldest deadline 2024-04-21T23:30:13.198900+0000) 2024-04-21T23:30:17.694 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 23:30:17 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-7[49121]: 2024-04-21T23:30:17.528+0000 7ff225f50700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.650347+0000 front 2024-04-21T23:29:40.650470+0000 (oldest deadline 2024-04-21T23:30:05.350454+0000) 2024-04-21T23:30:17.694 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 23:30:17 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-7[49121]: 2024-04-21T23:30:17.528+0000 7ff225f50700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:48.751995+0000 front 2024-04-21T23:29:48.752022+0000 (oldest deadline 2024-04-21T23:30:11.052223+0000) 2024-04-21T23:30:17.854 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 21 23:30:17 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-6[53641]: 2024-04-21T23:30:17.532+0000 7f4fde556700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T23:30:17.855 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 21 23:30:17 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-6[53641]: 2024-04-21T23:30:17.532+0000 7f4fde556700 -1 osd.6 108 *** Got signal Terminated *** 2024-04-21T23:30:17.855 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 21 23:30:17 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-6[53641]: 2024-04-21T23:30:17.532+0000 7f4fde556700 -1 osd.6 108 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-21T23:30:17.855 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 21 23:30:17 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-6[53641]: 2024-04-21T23:30:17.856+0000 7f4fdab4a700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:32.252857+0000 front 2024-04-21T23:29:32.253034+0000 (oldest deadline 2024-04-21T23:29:54.553015+0000) 2024-04-21T23:30:17.855 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 21 23:30:17 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-6[53641]: 2024-04-21T23:30:17.856+0000 7f4fdab4a700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:39.254040+0000 front 2024-04-21T23:29:39.254010+0000 (oldest deadline 2024-04-21T23:30:03.353822+0000) 2024-04-21T23:30:17.855 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 21 23:30:17 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-6[53641]: 2024-04-21T23:30:17.856+0000 7f4fdab4a700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:50.355743+0000 front 2024-04-21T23:29:50.355516+0000 (oldest deadline 2024-04-21T23:30:12.055457+0000) 2024-04-21T23:30:17.867 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:17 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:17.526+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:31.573404+0000 front 2024-04-21T23:29:31.573351+0000 (oldest deadline 2024-04-21T23:29:56.873350+0000) 2024-04-21T23:30:17.867 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:17 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:17.526+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.374892+0000 front 2024-04-21T23:29:40.374834+0000 (oldest deadline 2024-04-21T23:30:04.474654+0000) 2024-04-21T23:30:17.867 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:17 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:17.526+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:50.176921+0000 front 2024-04-21T23:29:50.176955+0000 (oldest deadline 2024-04-21T23:30:16.577589+0000) 2024-04-21T23:30:18.181 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:18 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:18.092+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.927390+0000 front 2024-04-21T23:29:33.927500+0000 (oldest deadline 2024-04-21T23:29:56.827517+0000) 2024-04-21T23:30:18.181 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:18 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:18.092+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:36.828037+0000 front 2024-04-21T23:29:36.827973+0000 (oldest deadline 2024-04-21T23:30:03.228609+0000) 2024-04-21T23:30:18.181 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:18 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:18.092+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:46.730173+0000 front 2024-04-21T23:29:46.729971+0000 (oldest deadline 2024-04-21T23:30:11.330259+0000) 2024-04-21T23:30:18.193 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:17 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:17.869+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:42.041804+0000 front 2024-04-21T23:29:42.041700+0000 (oldest deadline 2024-04-21T23:30:06.741665+0000) 2024-04-21T23:30:18.193 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:17 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:17.869+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:49.642677+0000 front 2024-04-21T23:29:49.642702+0000 (oldest deadline 2024-04-21T23:30:11.942839+0000) 2024-04-21T23:30:18.693 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:18 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:18.374+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.197048+0000 front 2024-04-21T23:29:33.197157+0000 (oldest deadline 2024-04-21T23:29:59.097025+0000) 2024-04-21T23:30:18.694 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:18 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:18.374+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:47.298757+0000 front 2024-04-21T23:29:47.298709+0000 (oldest deadline 2024-04-21T23:30:13.198900+0000) 2024-04-21T23:30:18.694 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 23:30:18 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-7[49121]: 2024-04-21T23:30:18.568+0000 7ff225f50700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.650347+0000 front 2024-04-21T23:29:40.650470+0000 (oldest deadline 2024-04-21T23:30:05.350454+0000) 2024-04-21T23:30:18.694 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 23:30:18 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-7[49121]: 2024-04-21T23:30:18.568+0000 7ff225f50700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:48.751995+0000 front 2024-04-21T23:29:48.752022+0000 (oldest deadline 2024-04-21T23:30:11.052223+0000) 2024-04-21T23:30:18.822 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:18 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:18.525+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:31.573404+0000 front 2024-04-21T23:29:31.573351+0000 (oldest deadline 2024-04-21T23:29:56.873350+0000) 2024-04-21T23:30:18.823 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:18 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:18.525+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.374892+0000 front 2024-04-21T23:29:40.374834+0000 (oldest deadline 2024-04-21T23:30:04.474654+0000) 2024-04-21T23:30:18.823 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:18 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:18.525+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:50.176921+0000 front 2024-04-21T23:29:50.176955+0000 (oldest deadline 2024-04-21T23:30:16.577589+0000) 2024-04-21T23:30:19.113 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 21 23:30:18 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-6[53641]: 2024-04-21T23:30:18.827+0000 7f4fdab4a700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:32.252857+0000 front 2024-04-21T23:29:32.253034+0000 (oldest deadline 2024-04-21T23:29:54.553015+0000) 2024-04-21T23:30:19.113 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 21 23:30:18 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-6[53641]: 2024-04-21T23:30:18.827+0000 7f4fdab4a700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:39.254040+0000 front 2024-04-21T23:29:39.254010+0000 (oldest deadline 2024-04-21T23:30:03.353822+0000) 2024-04-21T23:30:19.113 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 21 23:30:18 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-6[53641]: 2024-04-21T23:30:18.827+0000 7f4fdab4a700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:50.355743+0000 front 2024-04-21T23:29:50.355516+0000 (oldest deadline 2024-04-21T23:30:12.055457+0000) 2024-04-21T23:30:19.193 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:18 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:18.824+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:42.041804+0000 front 2024-04-21T23:29:42.041700+0000 (oldest deadline 2024-04-21T23:30:06.741665+0000) 2024-04-21T23:30:19.193 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:18 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:18.824+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:49.642677+0000 front 2024-04-21T23:29:49.642702+0000 (oldest deadline 2024-04-21T23:30:11.942839+0000) 2024-04-21T23:30:19.431 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:19 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:19.115+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.927390+0000 front 2024-04-21T23:29:33.927500+0000 (oldest deadline 2024-04-21T23:29:56.827517+0000) 2024-04-21T23:30:19.431 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:19 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:19.115+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:36.828037+0000 front 2024-04-21T23:29:36.827973+0000 (oldest deadline 2024-04-21T23:30:03.228609+0000) 2024-04-21T23:30:19.432 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:19 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:19.115+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:46.730173+0000 front 2024-04-21T23:29:46.729971+0000 (oldest deadline 2024-04-21T23:30:11.330259+0000) 2024-04-21T23:30:19.693 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:19 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:19.404+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.197048+0000 front 2024-04-21T23:29:33.197157+0000 (oldest deadline 2024-04-21T23:29:59.097025+0000) 2024-04-21T23:30:19.694 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:19 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:19.404+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:47.298757+0000 front 2024-04-21T23:29:47.298709+0000 (oldest deadline 2024-04-21T23:30:13.198900+0000) 2024-04-21T23:30:19.694 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 23:30:19 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-7[49121]: 2024-04-21T23:30:19.582+0000 7ff225f50700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.650347+0000 front 2024-04-21T23:29:40.650470+0000 (oldest deadline 2024-04-21T23:30:05.350454+0000) 2024-04-21T23:30:19.694 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 23:30:19 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-7[49121]: 2024-04-21T23:30:19.582+0000 7ff225f50700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:48.751995+0000 front 2024-04-21T23:29:48.752022+0000 (oldest deadline 2024-04-21T23:30:11.052223+0000) 2024-04-21T23:30:19.943 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:19 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:19.782+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:42.041804+0000 front 2024-04-21T23:29:42.041700+0000 (oldest deadline 2024-04-21T23:30:06.741665+0000) 2024-04-21T23:30:19.943 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:19 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:19.782+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:49.642677+0000 front 2024-04-21T23:29:49.642702+0000 (oldest deadline 2024-04-21T23:30:11.942839+0000) 2024-04-21T23:30:19.944 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:19 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:19.535+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:31.573404+0000 front 2024-04-21T23:29:31.573351+0000 (oldest deadline 2024-04-21T23:29:56.873350+0000) 2024-04-21T23:30:19.944 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:19 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:19.535+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.374892+0000 front 2024-04-21T23:29:40.374834+0000 (oldest deadline 2024-04-21T23:30:04.474654+0000) 2024-04-21T23:30:19.944 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:19 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:19.535+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:50.176921+0000 front 2024-04-21T23:29:50.176955+0000 (oldest deadline 2024-04-21T23:30:16.577589+0000) 2024-04-21T23:30:20.143 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 21 23:30:19 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-6[53641]: 2024-04-21T23:30:19.788+0000 7f4fdab4a700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:32.252857+0000 front 2024-04-21T23:29:32.253034+0000 (oldest deadline 2024-04-21T23:29:54.553015+0000) 2024-04-21T23:30:20.145 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 21 23:30:19 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-6[53641]: 2024-04-21T23:30:19.788+0000 7f4fdab4a700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:39.254040+0000 front 2024-04-21T23:29:39.254010+0000 (oldest deadline 2024-04-21T23:30:03.353822+0000) 2024-04-21T23:30:20.146 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 21 23:30:19 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-6[53641]: 2024-04-21T23:30:19.788+0000 7f4fdab4a700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:50.355743+0000 front 2024-04-21T23:29:50.355516+0000 (oldest deadline 2024-04-21T23:30:12.055457+0000) 2024-04-21T23:30:20.431 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:20 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:20.145+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.927390+0000 front 2024-04-21T23:29:33.927500+0000 (oldest deadline 2024-04-21T23:29:56.827517+0000) 2024-04-21T23:30:20.431 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:20 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:20.145+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:36.828037+0000 front 2024-04-21T23:29:36.827973+0000 (oldest deadline 2024-04-21T23:30:03.228609+0000) 2024-04-21T23:30:20.432 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:20 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:20.145+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:46.730173+0000 front 2024-04-21T23:29:46.729971+0000 (oldest deadline 2024-04-21T23:30:11.330259+0000) 2024-04-21T23:30:20.432 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:20 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:20.145+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:57.731564+0000 front 2024-04-21T23:29:57.731525+0000 (oldest deadline 2024-04-21T23:30:20.031550+0000) 2024-04-21T23:30:20.693 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:20 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:20.367+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.197048+0000 front 2024-04-21T23:29:33.197157+0000 (oldest deadline 2024-04-21T23:29:59.097025+0000) 2024-04-21T23:30:20.694 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:20 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:20.367+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:47.298757+0000 front 2024-04-21T23:29:47.298709+0000 (oldest deadline 2024-04-21T23:30:13.198900+0000) 2024-04-21T23:30:20.694 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:20 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:20.367+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:57.900160+0000 front 2024-04-21T23:29:57.900044+0000 (oldest deadline 2024-04-21T23:30:20.199812+0000) 2024-04-21T23:30:20.694 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 23:30:20 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-7[49121]: 2024-04-21T23:30:20.612+0000 7ff225f50700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.650347+0000 front 2024-04-21T23:29:40.650470+0000 (oldest deadline 2024-04-21T23:30:05.350454+0000) 2024-04-21T23:30:20.694 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 23:30:20 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-7[49121]: 2024-04-21T23:30:20.612+0000 7ff225f50700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:48.751995+0000 front 2024-04-21T23:29:48.752022+0000 (oldest deadline 2024-04-21T23:30:11.052223+0000) 2024-04-21T23:30:20.943 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:20 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:20.794+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:42.041804+0000 front 2024-04-21T23:29:42.041700+0000 (oldest deadline 2024-04-21T23:30:06.741665+0000) 2024-04-21T23:30:20.943 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:20 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:20.794+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:49.642677+0000 front 2024-04-21T23:29:49.642702+0000 (oldest deadline 2024-04-21T23:30:11.942839+0000) 2024-04-21T23:30:20.944 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:20 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:20.548+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:31.573404+0000 front 2024-04-21T23:29:31.573351+0000 (oldest deadline 2024-04-21T23:29:56.873350+0000) 2024-04-21T23:30:20.944 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:20 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:20.548+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.374892+0000 front 2024-04-21T23:29:40.374834+0000 (oldest deadline 2024-04-21T23:30:04.474654+0000) 2024-04-21T23:30:20.944 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:20 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:20.548+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:50.176921+0000 front 2024-04-21T23:29:50.176955+0000 (oldest deadline 2024-04-21T23:30:16.577589+0000) 2024-04-21T23:30:21.156 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 21 23:30:20 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-6[53641]: 2024-04-21T23:30:20.740+0000 7f4fdab4a700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:32.252857+0000 front 2024-04-21T23:29:32.253034+0000 (oldest deadline 2024-04-21T23:29:54.553015+0000) 2024-04-21T23:30:21.156 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 21 23:30:20 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-6[53641]: 2024-04-21T23:30:20.740+0000 7f4fdab4a700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:39.254040+0000 front 2024-04-21T23:29:39.254010+0000 (oldest deadline 2024-04-21T23:30:03.353822+0000) 2024-04-21T23:30:21.156 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 21 23:30:20 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-6[53641]: 2024-04-21T23:30:20.740+0000 7f4fdab4a700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:50.355743+0000 front 2024-04-21T23:29:50.355516+0000 (oldest deadline 2024-04-21T23:30:12.055457+0000) 2024-04-21T23:30:21.431 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:21 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:21.157+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.927390+0000 front 2024-04-21T23:29:33.927500+0000 (oldest deadline 2024-04-21T23:29:56.827517+0000) 2024-04-21T23:30:21.432 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:21 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:21.157+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:36.828037+0000 front 2024-04-21T23:29:36.827973+0000 (oldest deadline 2024-04-21T23:30:03.228609+0000) 2024-04-21T23:30:21.432 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:21 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:21.157+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:46.730173+0000 front 2024-04-21T23:29:46.729971+0000 (oldest deadline 2024-04-21T23:30:11.330259+0000) 2024-04-21T23:30:21.432 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:21 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:21.157+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:57.731564+0000 front 2024-04-21T23:29:57.731525+0000 (oldest deadline 2024-04-21T23:30:20.031550+0000) 2024-04-21T23:30:21.694 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:21.354+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.197048+0000 front 2024-04-21T23:29:33.197157+0000 (oldest deadline 2024-04-21T23:29:59.097025+0000) 2024-04-21T23:30:21.694 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:21.354+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:47.298757+0000 front 2024-04-21T23:29:47.298709+0000 (oldest deadline 2024-04-21T23:30:13.198900+0000) 2024-04-21T23:30:21.694 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:21.354+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:57.900160+0000 front 2024-04-21T23:29:57.900044+0000 (oldest deadline 2024-04-21T23:30:20.199812+0000) 2024-04-21T23:30:21.694 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 23:30:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-7[49121]: 2024-04-21T23:30:21.600+0000 7ff225f50700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.650347+0000 front 2024-04-21T23:29:40.650470+0000 (oldest deadline 2024-04-21T23:30:05.350454+0000) 2024-04-21T23:30:21.694 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 23:30:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-7[49121]: 2024-04-21T23:30:21.600+0000 7ff225f50700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:48.751995+0000 front 2024-04-21T23:29:48.752022+0000 (oldest deadline 2024-04-21T23:30:11.052223+0000) 2024-04-21T23:30:21.943 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:21 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:21.565+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:31.573404+0000 front 2024-04-21T23:29:31.573351+0000 (oldest deadline 2024-04-21T23:29:56.873350+0000) 2024-04-21T23:30:21.944 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:21 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:21.565+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.374892+0000 front 2024-04-21T23:29:40.374834+0000 (oldest deadline 2024-04-21T23:30:04.474654+0000) 2024-04-21T23:30:21.944 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:21 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:21.565+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:50.176921+0000 front 2024-04-21T23:29:50.176955+0000 (oldest deadline 2024-04-21T23:30:16.577589+0000) 2024-04-21T23:30:21.944 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:21 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:21.760+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:42.041804+0000 front 2024-04-21T23:29:42.041700+0000 (oldest deadline 2024-04-21T23:30:06.741665+0000) 2024-04-21T23:30:21.944 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:21 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:21.760+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:49.642677+0000 front 2024-04-21T23:29:49.642702+0000 (oldest deadline 2024-04-21T23:30:11.942839+0000) 2024-04-21T23:30:21.944 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:21 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:21.760+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:56.544455+0000 front 2024-04-21T23:29:56.544332+0000 (oldest deadline 2024-04-21T23:30:21.244522+0000) 2024-04-21T23:30:22.181 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 21 23:30:21 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-6[53641]: 2024-04-21T23:30:21.703+0000 7f4fdab4a700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:32.252857+0000 front 2024-04-21T23:29:32.253034+0000 (oldest deadline 2024-04-21T23:29:54.553015+0000) 2024-04-21T23:30:22.182 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 21 23:30:21 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-6[53641]: 2024-04-21T23:30:21.703+0000 7f4fdab4a700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:39.254040+0000 front 2024-04-21T23:29:39.254010+0000 (oldest deadline 2024-04-21T23:30:03.353822+0000) 2024-04-21T23:30:22.182 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 21 23:30:21 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-6[53641]: 2024-04-21T23:30:21.703+0000 7f4fdab4a700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:50.355743+0000 front 2024-04-21T23:29:50.355516+0000 (oldest deadline 2024-04-21T23:30:12.055457+0000) 2024-04-21T23:30:22.681 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:22 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:22.205+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.927390+0000 front 2024-04-21T23:29:33.927500+0000 (oldest deadline 2024-04-21T23:29:56.827517+0000) 2024-04-21T23:30:22.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:22 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:22.205+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:36.828037+0000 front 2024-04-21T23:29:36.827973+0000 (oldest deadline 2024-04-21T23:30:03.228609+0000) 2024-04-21T23:30:22.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:22 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:22.205+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:46.730173+0000 front 2024-04-21T23:29:46.729971+0000 (oldest deadline 2024-04-21T23:30:11.330259+0000) 2024-04-21T23:30:22.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:22 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:22.205+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:57.731564+0000 front 2024-04-21T23:29:57.731525+0000 (oldest deadline 2024-04-21T23:30:20.031550+0000) 2024-04-21T23:30:22.693 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 23:30:22 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-7[49121]: 2024-04-21T23:30:22.587+0000 7ff225f50700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.650347+0000 front 2024-04-21T23:29:40.650470+0000 (oldest deadline 2024-04-21T23:30:05.350454+0000) 2024-04-21T23:30:22.694 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 23:30:22 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-7[49121]: 2024-04-21T23:30:22.587+0000 7ff225f50700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:48.751995+0000 front 2024-04-21T23:29:48.752022+0000 (oldest deadline 2024-04-21T23:30:11.052223+0000) 2024-04-21T23:30:22.694 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:22 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:22.395+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.197048+0000 front 2024-04-21T23:29:33.197157+0000 (oldest deadline 2024-04-21T23:29:59.097025+0000) 2024-04-21T23:30:22.694 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:22 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:22.395+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:47.298757+0000 front 2024-04-21T23:29:47.298709+0000 (oldest deadline 2024-04-21T23:30:13.198900+0000) 2024-04-21T23:30:22.694 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:22 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:22.395+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:57.900160+0000 front 2024-04-21T23:29:57.900044+0000 (oldest deadline 2024-04-21T23:30:20.199812+0000) 2024-04-21T23:30:22.943 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:22 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:22.739+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:42.041804+0000 front 2024-04-21T23:29:42.041700+0000 (oldest deadline 2024-04-21T23:30:06.741665+0000) 2024-04-21T23:30:22.943 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:22 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:22.739+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:49.642677+0000 front 2024-04-21T23:29:49.642702+0000 (oldest deadline 2024-04-21T23:30:11.942839+0000) 2024-04-21T23:30:22.944 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:22 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:22.739+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:56.544455+0000 front 2024-04-21T23:29:56.544332+0000 (oldest deadline 2024-04-21T23:30:21.244522+0000) 2024-04-21T23:30:22.944 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:22 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:22.570+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:31.573404+0000 front 2024-04-21T23:29:31.573351+0000 (oldest deadline 2024-04-21T23:29:56.873350+0000) 2024-04-21T23:30:22.944 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:22 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:22.570+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.374892+0000 front 2024-04-21T23:29:40.374834+0000 (oldest deadline 2024-04-21T23:30:04.474654+0000) 2024-04-21T23:30:22.944 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:22 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:22.570+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:50.176921+0000 front 2024-04-21T23:29:50.176955+0000 (oldest deadline 2024-04-21T23:30:16.577589+0000) 2024-04-21T23:30:23.157 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 21 23:30:22 smithi040 podman[95310]: 2024-04-21 23:30:22.853243032 +0000 UTC m=+5.448834447 container died 67014dd8f2263e80981ea725844e1e6f6cf020736e5e7f7d495b93ccd92a8670 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-6, RELEASE=squid-820a2f8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-21T23:30:23.432 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 21 23:30:23 smithi040 podman[95310]: 2024-04-21 23:30:23.249611667 +0000 UTC m=+5.845203113 container cleanup 67014dd8f2263e80981ea725844e1e6f6cf020736e5e7f7d495b93ccd92a8670 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-6, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-820a2f8, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-21T23:30:23.432 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 21 23:30:23 smithi040 bash[95310]: ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-6 2024-04-21T23:30:23.432 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:23 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:23.233+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.927390+0000 front 2024-04-21T23:29:33.927500+0000 (oldest deadline 2024-04-21T23:29:56.827517+0000) 2024-04-21T23:30:23.432 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:23 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:23.233+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:36.828037+0000 front 2024-04-21T23:29:36.827973+0000 (oldest deadline 2024-04-21T23:30:03.228609+0000) 2024-04-21T23:30:23.432 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:23 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:23.233+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:46.730173+0000 front 2024-04-21T23:29:46.729971+0000 (oldest deadline 2024-04-21T23:30:11.330259+0000) 2024-04-21T23:30:23.433 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:23 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:23.233+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:57.731564+0000 front 2024-04-21T23:29:57.731525+0000 (oldest deadline 2024-04-21T23:30:20.031550+0000) 2024-04-21T23:30:23.610 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:23 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:23.352+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.197048+0000 front 2024-04-21T23:29:33.197157+0000 (oldest deadline 2024-04-21T23:29:59.097025+0000) 2024-04-21T23:30:23.611 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:23 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:23.352+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:47.298757+0000 front 2024-04-21T23:29:47.298709+0000 (oldest deadline 2024-04-21T23:30:13.198900+0000) 2024-04-21T23:30:23.611 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:23 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:23.352+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:57.900160+0000 front 2024-04-21T23:29:57.900044+0000 (oldest deadline 2024-04-21T23:30:20.199812+0000) 2024-04-21T23:30:23.884 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 21 23:30:23 smithi040 podman[95326]: 2024-04-21 23:30:23.551193358 +0000 UTC m=+0.695697118 container remove 67014dd8f2263e80981ea725844e1e6f6cf020736e5e7f7d495b93ccd92a8670 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-820a2f8, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-21T23:30:23.943 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:23 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:23.530+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:31.573404+0000 front 2024-04-21T23:29:31.573351+0000 (oldest deadline 2024-04-21T23:29:56.873350+0000) 2024-04-21T23:30:23.943 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:23 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:23.530+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.374892+0000 front 2024-04-21T23:29:40.374834+0000 (oldest deadline 2024-04-21T23:30:04.474654+0000) 2024-04-21T23:30:23.944 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:23 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:23.530+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:50.176921+0000 front 2024-04-21T23:29:50.176955+0000 (oldest deadline 2024-04-21T23:30:16.577589+0000) 2024-04-21T23:30:23.944 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:23 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:23.775+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:42.041804+0000 front 2024-04-21T23:29:42.041700+0000 (oldest deadline 2024-04-21T23:30:06.741665+0000) 2024-04-21T23:30:23.944 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:23 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:23.775+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:49.642677+0000 front 2024-04-21T23:29:49.642702+0000 (oldest deadline 2024-04-21T23:30:11.942839+0000) 2024-04-21T23:30:23.944 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:23 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:23.775+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:56.544455+0000 front 2024-04-21T23:29:56.544332+0000 (oldest deadline 2024-04-21T23:30:21.244522+0000) 2024-04-21T23:30:23.945 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 23:30:23 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-7[49121]: 2024-04-21T23:30:23.611+0000 7ff225f50700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.650347+0000 front 2024-04-21T23:29:40.650470+0000 (oldest deadline 2024-04-21T23:30:05.350454+0000) 2024-04-21T23:30:23.945 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 23:30:23 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-7[49121]: 2024-04-21T23:30:23.611+0000 7ff225f50700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:48.751995+0000 front 2024-04-21T23:29:48.752022+0000 (oldest deadline 2024-04-21T23:30:11.052223+0000) 2024-04-21T23:30:24.181 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 21 23:30:23 smithi040 podman[95419]: 2024-04-21 23:30:23.789877028 +0000 UTC m=+0.018845293 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-21T23:30:24.182 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 21 23:30:23 smithi040 podman[95419]: 2024-04-21 23:30:23.91044547 +0000 UTC m=+0.139413719 container create 83c4365af5878258c4042394dab15e0bcd3bd858e87d301f22ed346f44831c95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-820a2f8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0) 2024-04-21T23:30:24.509 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:24 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:24.268+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.927390+0000 front 2024-04-21T23:29:33.927500+0000 (oldest deadline 2024-04-21T23:29:56.827517+0000) 2024-04-21T23:30:24.510 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:24 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:24.268+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:36.828037+0000 front 2024-04-21T23:29:36.827973+0000 (oldest deadline 2024-04-21T23:30:03.228609+0000) 2024-04-21T23:30:24.510 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:24 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:24.268+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:46.730173+0000 front 2024-04-21T23:29:46.729971+0000 (oldest deadline 2024-04-21T23:30:11.330259+0000) 2024-04-21T23:30:24.510 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:24 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:24.268+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:57.731564+0000 front 2024-04-21T23:29:57.731525+0000 (oldest deadline 2024-04-21T23:30:20.031550+0000) 2024-04-21T23:30:24.511 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 21 23:30:24 smithi040 podman[95419]: 2024-04-21 23:30:24.196419823 +0000 UTC m=+0.425388074 container init 83c4365af5878258c4042394dab15e0bcd3bd858e87d301f22ed346f44831c95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-6-deactivate, RELEASE=squid-820a2f8, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-21T23:30:24.511 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 21 23:30:24 smithi040 podman[95419]: 2024-04-21 23:30:24.199585668 +0000 UTC m=+0.428553919 container start 83c4365af5878258c4042394dab15e0bcd3bd858e87d301f22ed346f44831c95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-6-deactivate, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-820a2f8, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-21T23:30:24.511 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 21 23:30:24 smithi040 podman[95419]: 2024-04-21 23:30:24.287909551 +0000 UTC m=+0.516877811 container attach 83c4365af5878258c4042394dab15e0bcd3bd858e87d301f22ed346f44831c95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-6-deactivate, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=squid-820a2f8, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-21T23:30:24.657 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:24 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:24.354+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.197048+0000 front 2024-04-21T23:29:33.197157+0000 (oldest deadline 2024-04-21T23:29:59.097025+0000) 2024-04-21T23:30:24.657 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:24 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:24.354+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:47.298757+0000 front 2024-04-21T23:29:47.298709+0000 (oldest deadline 2024-04-21T23:30:13.198900+0000) 2024-04-21T23:30:24.657 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:24 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:24.354+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:57.900160+0000 front 2024-04-21T23:29:57.900044+0000 (oldest deadline 2024-04-21T23:30:20.199812+0000) 2024-04-21T23:30:24.769 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:24 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:24.518+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:31.573404+0000 front 2024-04-21T23:29:31.573351+0000 (oldest deadline 2024-04-21T23:29:56.873350+0000) 2024-04-21T23:30:24.769 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:24 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:24.518+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.374892+0000 front 2024-04-21T23:29:40.374834+0000 (oldest deadline 2024-04-21T23:30:04.474654+0000) 2024-04-21T23:30:24.770 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:24 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:24.518+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:50.176921+0000 front 2024-04-21T23:29:50.176955+0000 (oldest deadline 2024-04-21T23:30:16.577589+0000) 2024-04-21T23:30:24.822 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 21 23:30:24 smithi040 podman[95419]: 2024-04-21 23:30:24.511306468 +0000 UTC m=+0.740274711 container died 83c4365af5878258c4042394dab15e0bcd3bd858e87d301f22ed346f44831c95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-6-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.build-date=20240416, RELEASE=squid-820a2f8, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-21T23:30:24.943 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 23:30:24 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-7[49121]: 2024-04-21T23:30:24.658+0000 7ff225f50700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.650347+0000 front 2024-04-21T23:29:40.650470+0000 (oldest deadline 2024-04-21T23:30:05.350454+0000) 2024-04-21T23:30:24.944 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 23:30:24 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-7[49121]: 2024-04-21T23:30:24.658+0000 7ff225f50700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:48.751995+0000 front 2024-04-21T23:29:48.752022+0000 (oldest deadline 2024-04-21T23:30:11.052223+0000) 2024-04-21T23:30:24.944 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 23:30:24 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-7[49121]: 2024-04-21T23:30:24.658+0000 7ff225f50700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:54.454028+0000 front 2024-04-21T23:29:54.454113+0000 (oldest deadline 2024-04-21T23:30:24.454419+0000) 2024-04-21T23:30:25.123 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d814e4dc-0033-11ef-bc93-c7b262605968@osd.6.service' 2024-04-21T23:30:25.167 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T23:30:25.167 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-21T23:30:25.167 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-21T23:30:25.167 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-d814e4dc-0033-11ef-bc93-c7b262605968@osd.7 2024-04-21T23:30:25.193 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:24 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:24.771+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:42.041804+0000 front 2024-04-21T23:29:42.041700+0000 (oldest deadline 2024-04-21T23:30:06.741665+0000) 2024-04-21T23:30:25.193 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:24 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:24.771+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:49.642677+0000 front 2024-04-21T23:29:49.642702+0000 (oldest deadline 2024-04-21T23:30:11.942839+0000) 2024-04-21T23:30:25.193 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:24 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:24.771+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:56.544455+0000 front 2024-04-21T23:29:56.544332+0000 (oldest deadline 2024-04-21T23:30:21.244522+0000) 2024-04-21T23:30:25.392 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:25 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:25.273+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.927390+0000 front 2024-04-21T23:29:33.927500+0000 (oldest deadline 2024-04-21T23:29:56.827517+0000) 2024-04-21T23:30:25.392 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:25 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:25.273+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:36.828037+0000 front 2024-04-21T23:29:36.827973+0000 (oldest deadline 2024-04-21T23:30:03.228609+0000) 2024-04-21T23:30:25.392 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:25 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:25.273+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:46.730173+0000 front 2024-04-21T23:29:46.729971+0000 (oldest deadline 2024-04-21T23:30:11.330259+0000) 2024-04-21T23:30:25.392 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:25 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:25.273+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:57.731564+0000 front 2024-04-21T23:29:57.731525+0000 (oldest deadline 2024-04-21T23:30:20.031550+0000) 2024-04-21T23:30:25.487 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:25 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:25.376+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.197048+0000 front 2024-04-21T23:29:33.197157+0000 (oldest deadline 2024-04-21T23:29:59.097025+0000) 2024-04-21T23:30:25.488 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:25 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:25.376+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:47.298757+0000 front 2024-04-21T23:29:47.298709+0000 (oldest deadline 2024-04-21T23:30:13.198900+0000) 2024-04-21T23:30:25.488 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:25 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:25.376+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:57.900160+0000 front 2024-04-21T23:29:57.900044+0000 (oldest deadline 2024-04-21T23:30:20.199812+0000) 2024-04-21T23:30:25.488 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 23:30:25 smithi111 systemd[1]: Stopping Ceph osd.7 for d814e4dc-0033-11ef-bc93-c7b262605968... 2024-04-21T23:30:25.488 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 23:30:25 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-7[49121]: 2024-04-21T23:30:25.488+0000 7ff22995c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T23:30:25.488 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 23:30:25 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-7[49121]: 2024-04-21T23:30:25.488+0000 7ff22995c700 -1 osd.7 108 *** Got signal Terminated *** 2024-04-21T23:30:25.488 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 23:30:25 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-7[49121]: 2024-04-21T23:30:25.488+0000 7ff22995c700 -1 osd.7 108 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-21T23:30:25.782 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:25 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:25.519+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:31.573404+0000 front 2024-04-21T23:29:31.573351+0000 (oldest deadline 2024-04-21T23:29:56.873350+0000) 2024-04-21T23:30:25.782 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:25 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:25.519+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.374892+0000 front 2024-04-21T23:29:40.374834+0000 (oldest deadline 2024-04-21T23:30:04.474654+0000) 2024-04-21T23:30:25.782 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:25 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:25.519+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:50.176921+0000 front 2024-04-21T23:29:50.176955+0000 (oldest deadline 2024-04-21T23:30:16.577589+0000) 2024-04-21T23:30:25.858 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 23:30:25 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-7[49121]: 2024-04-21T23:30:25.704+0000 7ff225f50700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.650347+0000 front 2024-04-21T23:29:40.650470+0000 (oldest deadline 2024-04-21T23:30:05.350454+0000) 2024-04-21T23:30:25.858 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 23:30:25 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-7[49121]: 2024-04-21T23:30:25.704+0000 7ff225f50700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:48.751995+0000 front 2024-04-21T23:29:48.752022+0000 (oldest deadline 2024-04-21T23:30:11.052223+0000) 2024-04-21T23:30:25.858 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 23:30:25 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-7[49121]: 2024-04-21T23:30:25.704+0000 7ff225f50700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:54.454028+0000 front 2024-04-21T23:29:54.454113+0000 (oldest deadline 2024-04-21T23:30:24.454419+0000) 2024-04-21T23:30:26.193 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:25 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:25.783+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:42.041804+0000 front 2024-04-21T23:29:42.041700+0000 (oldest deadline 2024-04-21T23:30:06.741665+0000) 2024-04-21T23:30:26.193 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:25 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:25.783+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:49.642677+0000 front 2024-04-21T23:29:49.642702+0000 (oldest deadline 2024-04-21T23:30:11.942839+0000) 2024-04-21T23:30:26.193 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:25 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:25.783+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:56.544455+0000 front 2024-04-21T23:29:56.544332+0000 (oldest deadline 2024-04-21T23:30:21.244522+0000) 2024-04-21T23:30:26.681 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:26 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:26.256+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.927390+0000 front 2024-04-21T23:29:33.927500+0000 (oldest deadline 2024-04-21T23:29:56.827517+0000) 2024-04-21T23:30:26.681 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:26 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:26.256+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:36.828037+0000 front 2024-04-21T23:29:36.827973+0000 (oldest deadline 2024-04-21T23:30:03.228609+0000) 2024-04-21T23:30:26.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:26 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:26.256+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:46.730173+0000 front 2024-04-21T23:29:46.729971+0000 (oldest deadline 2024-04-21T23:30:11.330259+0000) 2024-04-21T23:30:26.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:26 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:26.256+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:57.731564+0000 front 2024-04-21T23:29:57.731525+0000 (oldest deadline 2024-04-21T23:30:20.031550+0000) 2024-04-21T23:30:26.693 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:26 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:26.424+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.197048+0000 front 2024-04-21T23:29:33.197157+0000 (oldest deadline 2024-04-21T23:29:59.097025+0000) 2024-04-21T23:30:26.693 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:26 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:26.424+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:47.298757+0000 front 2024-04-21T23:29:47.298709+0000 (oldest deadline 2024-04-21T23:30:13.198900+0000) 2024-04-21T23:30:26.693 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:26 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:26.424+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:57.900160+0000 front 2024-04-21T23:29:57.900044+0000 (oldest deadline 2024-04-21T23:30:20.199812+0000) 2024-04-21T23:30:26.943 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:26 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:26.742+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:42.041804+0000 front 2024-04-21T23:29:42.041700+0000 (oldest deadline 2024-04-21T23:30:06.741665+0000) 2024-04-21T23:30:26.944 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:26 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:26.742+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:49.642677+0000 front 2024-04-21T23:29:49.642702+0000 (oldest deadline 2024-04-21T23:30:11.942839+0000) 2024-04-21T23:30:26.944 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:26 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:26.742+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:56.544455+0000 front 2024-04-21T23:29:56.544332+0000 (oldest deadline 2024-04-21T23:30:21.244522+0000) 2024-04-21T23:30:26.944 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:26 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:26.558+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:31.573404+0000 front 2024-04-21T23:29:31.573351+0000 (oldest deadline 2024-04-21T23:29:56.873350+0000) 2024-04-21T23:30:26.944 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:26 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:26.558+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.374892+0000 front 2024-04-21T23:29:40.374834+0000 (oldest deadline 2024-04-21T23:30:04.474654+0000) 2024-04-21T23:30:26.944 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:26 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:26.558+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:50.176921+0000 front 2024-04-21T23:29:50.176955+0000 (oldest deadline 2024-04-21T23:30:16.577589+0000) 2024-04-21T23:30:27.194 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 23:30:26 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-7[49121]: 2024-04-21T23:30:26.737+0000 7ff225f50700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.650347+0000 front 2024-04-21T23:29:40.650470+0000 (oldest deadline 2024-04-21T23:30:05.350454+0000) 2024-04-21T23:30:27.194 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 23:30:26 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-7[49121]: 2024-04-21T23:30:26.737+0000 7ff225f50700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:48.751995+0000 front 2024-04-21T23:29:48.752022+0000 (oldest deadline 2024-04-21T23:30:11.052223+0000) 2024-04-21T23:30:27.194 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 23:30:26 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-7[49121]: 2024-04-21T23:30:26.737+0000 7ff225f50700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:54.454028+0000 front 2024-04-21T23:29:54.454113+0000 (oldest deadline 2024-04-21T23:30:24.454419+0000) 2024-04-21T23:30:27.681 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:27 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:27.282+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.927390+0000 front 2024-04-21T23:29:33.927500+0000 (oldest deadline 2024-04-21T23:29:56.827517+0000) 2024-04-21T23:30:27.681 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:27 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:27.282+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:36.828037+0000 front 2024-04-21T23:29:36.827973+0000 (oldest deadline 2024-04-21T23:30:03.228609+0000) 2024-04-21T23:30:27.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:27 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:27.282+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:46.730173+0000 front 2024-04-21T23:29:46.729971+0000 (oldest deadline 2024-04-21T23:30:11.330259+0000) 2024-04-21T23:30:27.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:27 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:27.282+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:57.731564+0000 front 2024-04-21T23:29:57.731525+0000 (oldest deadline 2024-04-21T23:30:20.031550+0000) 2024-04-21T23:30:27.693 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:27 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:27.425+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.197048+0000 front 2024-04-21T23:29:33.197157+0000 (oldest deadline 2024-04-21T23:29:59.097025+0000) 2024-04-21T23:30:27.694 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:27 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:27.425+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:47.298757+0000 front 2024-04-21T23:29:47.298709+0000 (oldest deadline 2024-04-21T23:30:13.198900+0000) 2024-04-21T23:30:27.694 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:27 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:27.425+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:57.900160+0000 front 2024-04-21T23:29:57.900044+0000 (oldest deadline 2024-04-21T23:30:20.199812+0000) 2024-04-21T23:30:27.943 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:27 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:27.734+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:42.041804+0000 front 2024-04-21T23:29:42.041700+0000 (oldest deadline 2024-04-21T23:30:06.741665+0000) 2024-04-21T23:30:27.943 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:27 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:27.734+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:49.642677+0000 front 2024-04-21T23:29:49.642702+0000 (oldest deadline 2024-04-21T23:30:11.942839+0000) 2024-04-21T23:30:27.944 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:27 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:27.734+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:56.544455+0000 front 2024-04-21T23:29:56.544332+0000 (oldest deadline 2024-04-21T23:30:21.244522+0000) 2024-04-21T23:30:27.944 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:27 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:27.734+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:01.745868+0000 front 2024-04-21T23:30:01.745850+0000 (oldest deadline 2024-04-21T23:30:27.645830+0000) 2024-04-21T23:30:27.944 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:27 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:27.574+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:31.573404+0000 front 2024-04-21T23:29:31.573351+0000 (oldest deadline 2024-04-21T23:29:56.873350+0000) 2024-04-21T23:30:27.944 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:27 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:27.574+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.374892+0000 front 2024-04-21T23:29:40.374834+0000 (oldest deadline 2024-04-21T23:30:04.474654+0000) 2024-04-21T23:30:27.944 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:27 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:27.574+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:50.176921+0000 front 2024-04-21T23:29:50.176955+0000 (oldest deadline 2024-04-21T23:30:16.577589+0000) 2024-04-21T23:30:27.944 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:27 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:27.574+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:02.478617+0000 front 2024-04-21T23:30:02.478436+0000 (oldest deadline 2024-04-21T23:30:27.178522+0000) 2024-04-21T23:30:28.193 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 23:30:27 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-7[49121]: 2024-04-21T23:30:27.745+0000 7ff225f50700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.650347+0000 front 2024-04-21T23:29:40.650470+0000 (oldest deadline 2024-04-21T23:30:05.350454+0000) 2024-04-21T23:30:28.193 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 23:30:27 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-7[49121]: 2024-04-21T23:30:27.745+0000 7ff225f50700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:48.751995+0000 front 2024-04-21T23:29:48.752022+0000 (oldest deadline 2024-04-21T23:30:11.052223+0000) 2024-04-21T23:30:28.194 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 23:30:27 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-7[49121]: 2024-04-21T23:30:27.745+0000 7ff225f50700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:54.454028+0000 front 2024-04-21T23:29:54.454113+0000 (oldest deadline 2024-04-21T23:30:24.454419+0000) 2024-04-21T23:30:28.681 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:28 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:28.310+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.927390+0000 front 2024-04-21T23:29:33.927500+0000 (oldest deadline 2024-04-21T23:29:56.827517+0000) 2024-04-21T23:30:28.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:28 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:28.310+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:36.828037+0000 front 2024-04-21T23:29:36.827973+0000 (oldest deadline 2024-04-21T23:30:03.228609+0000) 2024-04-21T23:30:28.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:28 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:28.310+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:46.730173+0000 front 2024-04-21T23:29:46.729971+0000 (oldest deadline 2024-04-21T23:30:11.330259+0000) 2024-04-21T23:30:28.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:28 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:28.310+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:57.731564+0000 front 2024-04-21T23:29:57.731525+0000 (oldest deadline 2024-04-21T23:30:20.031550+0000) 2024-04-21T23:30:28.693 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:28 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:28.413+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.197048+0000 front 2024-04-21T23:29:33.197157+0000 (oldest deadline 2024-04-21T23:29:59.097025+0000) 2024-04-21T23:30:28.693 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:28 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:28.413+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:47.298757+0000 front 2024-04-21T23:29:47.298709+0000 (oldest deadline 2024-04-21T23:30:13.198900+0000) 2024-04-21T23:30:28.694 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:28 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:28.413+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:57.900160+0000 front 2024-04-21T23:29:57.900044+0000 (oldest deadline 2024-04-21T23:30:20.199812+0000) 2024-04-21T23:30:28.694 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:28 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:28.413+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:03.700683+0000 front 2024-04-21T23:30:03.700711+0000 (oldest deadline 2024-04-21T23:30:27.800810+0000) 2024-04-21T23:30:28.943 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:28 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:28.761+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:42.041804+0000 front 2024-04-21T23:29:42.041700+0000 (oldest deadline 2024-04-21T23:30:06.741665+0000) 2024-04-21T23:30:28.944 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:28 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:28.761+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:49.642677+0000 front 2024-04-21T23:29:49.642702+0000 (oldest deadline 2024-04-21T23:30:11.942839+0000) 2024-04-21T23:30:28.944 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:28 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:28.761+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:56.544455+0000 front 2024-04-21T23:29:56.544332+0000 (oldest deadline 2024-04-21T23:30:21.244522+0000) 2024-04-21T23:30:28.944 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:28 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:28.761+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:01.745868+0000 front 2024-04-21T23:30:01.745850+0000 (oldest deadline 2024-04-21T23:30:27.645830+0000) 2024-04-21T23:30:28.944 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:28 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:28.616+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:31.573404+0000 front 2024-04-21T23:29:31.573351+0000 (oldest deadline 2024-04-21T23:29:56.873350+0000) 2024-04-21T23:30:28.944 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:28 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:28.616+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.374892+0000 front 2024-04-21T23:29:40.374834+0000 (oldest deadline 2024-04-21T23:30:04.474654+0000) 2024-04-21T23:30:28.944 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:28 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:28.616+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:50.176921+0000 front 2024-04-21T23:29:50.176955+0000 (oldest deadline 2024-04-21T23:30:16.577589+0000) 2024-04-21T23:30:28.944 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:28 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:28.616+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:02.478617+0000 front 2024-04-21T23:30:02.478436+0000 (oldest deadline 2024-04-21T23:30:27.178522+0000) 2024-04-21T23:30:29.193 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 23:30:28 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-7[49121]: 2024-04-21T23:30:28.704+0000 7ff225f50700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.650347+0000 front 2024-04-21T23:29:40.650470+0000 (oldest deadline 2024-04-21T23:30:05.350454+0000) 2024-04-21T23:30:29.193 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 23:30:28 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-7[49121]: 2024-04-21T23:30:28.704+0000 7ff225f50700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:48.751995+0000 front 2024-04-21T23:29:48.752022+0000 (oldest deadline 2024-04-21T23:30:11.052223+0000) 2024-04-21T23:30:29.194 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 23:30:28 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-7[49121]: 2024-04-21T23:30:28.704+0000 7ff225f50700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:54.454028+0000 front 2024-04-21T23:29:54.454113+0000 (oldest deadline 2024-04-21T23:30:24.454419+0000) 2024-04-21T23:30:29.681 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:29 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:29.314+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.927390+0000 front 2024-04-21T23:29:33.927500+0000 (oldest deadline 2024-04-21T23:29:56.827517+0000) 2024-04-21T23:30:29.681 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:29 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:29.314+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:36.828037+0000 front 2024-04-21T23:29:36.827973+0000 (oldest deadline 2024-04-21T23:30:03.228609+0000) 2024-04-21T23:30:29.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:29 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:29.314+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:46.730173+0000 front 2024-04-21T23:29:46.729971+0000 (oldest deadline 2024-04-21T23:30:11.330259+0000) 2024-04-21T23:30:29.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:29 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:29.314+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:57.731564+0000 front 2024-04-21T23:29:57.731525+0000 (oldest deadline 2024-04-21T23:30:20.031550+0000) 2024-04-21T23:30:29.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:29 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:29.314+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:06.333794+0000 front 2024-04-21T23:30:06.333742+0000 (oldest deadline 2024-04-21T23:30:29.233758+0000) 2024-04-21T23:30:29.693 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:29 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:29.366+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.197048+0000 front 2024-04-21T23:29:33.197157+0000 (oldest deadline 2024-04-21T23:29:59.097025+0000) 2024-04-21T23:30:29.693 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:29 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:29.366+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:47.298757+0000 front 2024-04-21T23:29:47.298709+0000 (oldest deadline 2024-04-21T23:30:13.198900+0000) 2024-04-21T23:30:29.693 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:29 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:29.366+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:57.900160+0000 front 2024-04-21T23:29:57.900044+0000 (oldest deadline 2024-04-21T23:30:20.199812+0000) 2024-04-21T23:30:29.694 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:29 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:29.366+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:03.700683+0000 front 2024-04-21T23:30:03.700711+0000 (oldest deadline 2024-04-21T23:30:27.800810+0000) 2024-04-21T23:30:29.943 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:29 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:29.779+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:42.041804+0000 front 2024-04-21T23:29:42.041700+0000 (oldest deadline 2024-04-21T23:30:06.741665+0000) 2024-04-21T23:30:29.943 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:29 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:29.779+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:49.642677+0000 front 2024-04-21T23:29:49.642702+0000 (oldest deadline 2024-04-21T23:30:11.942839+0000) 2024-04-21T23:30:29.944 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:29 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:29.779+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:56.544455+0000 front 2024-04-21T23:29:56.544332+0000 (oldest deadline 2024-04-21T23:30:21.244522+0000) 2024-04-21T23:30:29.944 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:29 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:29.779+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:01.745868+0000 front 2024-04-21T23:30:01.745850+0000 (oldest deadline 2024-04-21T23:30:27.645830+0000) 2024-04-21T23:30:29.944 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:29 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:29.661+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:31.573404+0000 front 2024-04-21T23:29:31.573351+0000 (oldest deadline 2024-04-21T23:29:56.873350+0000) 2024-04-21T23:30:29.944 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:29 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:29.661+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.374892+0000 front 2024-04-21T23:29:40.374834+0000 (oldest deadline 2024-04-21T23:30:04.474654+0000) 2024-04-21T23:30:29.944 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:29 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:29.661+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:50.176921+0000 front 2024-04-21T23:29:50.176955+0000 (oldest deadline 2024-04-21T23:30:16.577589+0000) 2024-04-21T23:30:29.944 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:29 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:29.661+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:02.478617+0000 front 2024-04-21T23:30:02.478436+0000 (oldest deadline 2024-04-21T23:30:27.178522+0000) 2024-04-21T23:30:30.193 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 23:30:29 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-7[49121]: 2024-04-21T23:30:29.720+0000 7ff225f50700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.650347+0000 front 2024-04-21T23:29:40.650470+0000 (oldest deadline 2024-04-21T23:30:05.350454+0000) 2024-04-21T23:30:30.194 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 23:30:29 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-7[49121]: 2024-04-21T23:30:29.720+0000 7ff225f50700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:48.751995+0000 front 2024-04-21T23:29:48.752022+0000 (oldest deadline 2024-04-21T23:30:11.052223+0000) 2024-04-21T23:30:30.194 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 23:30:29 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-7[49121]: 2024-04-21T23:30:29.720+0000 7ff225f50700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:54.454028+0000 front 2024-04-21T23:29:54.454113+0000 (oldest deadline 2024-04-21T23:30:24.454419+0000) 2024-04-21T23:30:30.681 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:30 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:30.353+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.927390+0000 front 2024-04-21T23:29:33.927500+0000 (oldest deadline 2024-04-21T23:29:56.827517+0000) 2024-04-21T23:30:30.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:30 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:30.353+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:36.828037+0000 front 2024-04-21T23:29:36.827973+0000 (oldest deadline 2024-04-21T23:30:03.228609+0000) 2024-04-21T23:30:30.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:30 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:30.353+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:46.730173+0000 front 2024-04-21T23:29:46.729971+0000 (oldest deadline 2024-04-21T23:30:11.330259+0000) 2024-04-21T23:30:30.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:30 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:30.353+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:57.731564+0000 front 2024-04-21T23:29:57.731525+0000 (oldest deadline 2024-04-21T23:30:20.031550+0000) 2024-04-21T23:30:30.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:30 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:30.353+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:06.333794+0000 front 2024-04-21T23:30:06.333742+0000 (oldest deadline 2024-04-21T23:30:29.233758+0000) 2024-04-21T23:30:30.693 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:30 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:30.390+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.197048+0000 front 2024-04-21T23:29:33.197157+0000 (oldest deadline 2024-04-21T23:29:59.097025+0000) 2024-04-21T23:30:30.693 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:30 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:30.390+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:47.298757+0000 front 2024-04-21T23:29:47.298709+0000 (oldest deadline 2024-04-21T23:30:13.198900+0000) 2024-04-21T23:30:30.693 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:30 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:30.390+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:57.900160+0000 front 2024-04-21T23:29:57.900044+0000 (oldest deadline 2024-04-21T23:30:20.199812+0000) 2024-04-21T23:30:30.694 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:30 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:30.390+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:03.700683+0000 front 2024-04-21T23:30:03.700711+0000 (oldest deadline 2024-04-21T23:30:27.800810+0000) 2024-04-21T23:30:30.943 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:30 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:30.748+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:42.041804+0000 front 2024-04-21T23:29:42.041700+0000 (oldest deadline 2024-04-21T23:30:06.741665+0000) 2024-04-21T23:30:30.944 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:30 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:30.748+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:49.642677+0000 front 2024-04-21T23:29:49.642702+0000 (oldest deadline 2024-04-21T23:30:11.942839+0000) 2024-04-21T23:30:30.944 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:30 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:30.748+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:56.544455+0000 front 2024-04-21T23:29:56.544332+0000 (oldest deadline 2024-04-21T23:30:21.244522+0000) 2024-04-21T23:30:30.944 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:30 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:30.748+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:01.745868+0000 front 2024-04-21T23:30:01.745850+0000 (oldest deadline 2024-04-21T23:30:27.645830+0000) 2024-04-21T23:30:30.944 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:30 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:30.670+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:31.573404+0000 front 2024-04-21T23:29:31.573351+0000 (oldest deadline 2024-04-21T23:29:56.873350+0000) 2024-04-21T23:30:30.944 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:30 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:30.670+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.374892+0000 front 2024-04-21T23:29:40.374834+0000 (oldest deadline 2024-04-21T23:30:04.474654+0000) 2024-04-21T23:30:30.944 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:30 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:30.670+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:50.176921+0000 front 2024-04-21T23:29:50.176955+0000 (oldest deadline 2024-04-21T23:30:16.577589+0000) 2024-04-21T23:30:30.945 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:30 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:30.670+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:02.478617+0000 front 2024-04-21T23:30:02.478436+0000 (oldest deadline 2024-04-21T23:30:27.178522+0000) 2024-04-21T23:30:31.305 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 23:30:31 smithi111 podman[67503]: 2024-04-21 23:30:31.015274323 +0000 UTC m=+5.641065261 container died 38069fd11dfdc0b22860b65055003dfe92d59bbf23fef31cfe800e1e349b1c18 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-7, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=squid-820a2f8, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T23:30:31.681 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:31 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:31.350+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.927390+0000 front 2024-04-21T23:29:33.927500+0000 (oldest deadline 2024-04-21T23:29:56.827517+0000) 2024-04-21T23:30:31.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:31 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:31.350+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:36.828037+0000 front 2024-04-21T23:29:36.827973+0000 (oldest deadline 2024-04-21T23:30:03.228609+0000) 2024-04-21T23:30:31.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:31 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:31.350+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:46.730173+0000 front 2024-04-21T23:29:46.729971+0000 (oldest deadline 2024-04-21T23:30:11.330259+0000) 2024-04-21T23:30:31.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:31 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:31.350+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:57.731564+0000 front 2024-04-21T23:29:57.731525+0000 (oldest deadline 2024-04-21T23:30:20.031550+0000) 2024-04-21T23:30:31.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:31 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:31.350+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:06.333794+0000 front 2024-04-21T23:30:06.333742+0000 (oldest deadline 2024-04-21T23:30:29.233758+0000) 2024-04-21T23:30:31.689 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 23:30:31 smithi111 podman[67503]: 2024-04-21 23:30:31.397887672 +0000 UTC m=+6.023678602 container cleanup 38069fd11dfdc0b22860b65055003dfe92d59bbf23fef31cfe800e1e349b1c18 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-7, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-820a2f8, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T23:30:31.689 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 23:30:31 smithi111 bash[67503]: ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-7 2024-04-21T23:30:31.689 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:31 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:31.431+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.197048+0000 front 2024-04-21T23:29:33.197157+0000 (oldest deadline 2024-04-21T23:29:59.097025+0000) 2024-04-21T23:30:31.689 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:31 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:31.431+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:47.298757+0000 front 2024-04-21T23:29:47.298709+0000 (oldest deadline 2024-04-21T23:30:13.198900+0000) 2024-04-21T23:30:31.690 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:31 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:31.431+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:57.900160+0000 front 2024-04-21T23:29:57.900044+0000 (oldest deadline 2024-04-21T23:30:20.199812+0000) 2024-04-21T23:30:31.690 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:31 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:31.431+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:03.700683+0000 front 2024-04-21T23:30:03.700711+0000 (oldest deadline 2024-04-21T23:30:27.800810+0000) 2024-04-21T23:30:31.943 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:31 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:31.774+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:42.041804+0000 front 2024-04-21T23:29:42.041700+0000 (oldest deadline 2024-04-21T23:30:06.741665+0000) 2024-04-21T23:30:31.944 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:31 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:31.774+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:49.642677+0000 front 2024-04-21T23:29:49.642702+0000 (oldest deadline 2024-04-21T23:30:11.942839+0000) 2024-04-21T23:30:31.944 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:31 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:31.774+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:56.544455+0000 front 2024-04-21T23:29:56.544332+0000 (oldest deadline 2024-04-21T23:30:21.244522+0000) 2024-04-21T23:30:31.944 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:31 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:31.774+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:01.745868+0000 front 2024-04-21T23:30:01.745850+0000 (oldest deadline 2024-04-21T23:30:27.645830+0000) 2024-04-21T23:30:31.944 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:31 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:31.693+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:31.573404+0000 front 2024-04-21T23:29:31.573351+0000 (oldest deadline 2024-04-21T23:29:56.873350+0000) 2024-04-21T23:30:31.944 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:31 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:31.693+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.374892+0000 front 2024-04-21T23:29:40.374834+0000 (oldest deadline 2024-04-21T23:30:04.474654+0000) 2024-04-21T23:30:31.944 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:31 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:31.693+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:50.176921+0000 front 2024-04-21T23:29:50.176955+0000 (oldest deadline 2024-04-21T23:30:16.577589+0000) 2024-04-21T23:30:31.945 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:31 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:31.693+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:02.478617+0000 front 2024-04-21T23:30:02.478436+0000 (oldest deadline 2024-04-21T23:30:27.178522+0000) 2024-04-21T23:30:31.945 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 23:30:31 smithi111 podman[67521]: 2024-04-21 23:30:31.690747327 +0000 UTC m=+0.672463047 container remove 38069fd11dfdc0b22860b65055003dfe92d59bbf23fef31cfe800e1e349b1c18 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-7, RELEASE=squid-820a2f8, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416) 2024-04-21T23:30:32.275 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 23:30:32 smithi111 podman[67610]: 2024-04-21 23:30:31.916397889 +0000 UTC m=+0.017773201 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-21T23:30:32.275 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 23:30:32 smithi111 podman[67610]: 2024-04-21 23:30:32.016827262 +0000 UTC m=+0.118202566 container create eccd68634c16a14aa2862f5368d74bfe9b7c4bcbaf636f2ebc78efcdae676347 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-7-deactivate, RELEASE=squid-820a2f8, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-21T23:30:32.275 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 23:30:32 smithi111 podman[67610]: 2024-04-21 23:30:32.202371721 +0000 UTC m=+0.303747019 container init eccd68634c16a14aa2862f5368d74bfe9b7c4bcbaf636f2ebc78efcdae676347 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=squid-820a2f8, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T23:30:32.275 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 23:30:32 smithi111 podman[67610]: 2024-04-21 23:30:32.205575785 +0000 UTC m=+0.306951096 container start eccd68634c16a14aa2862f5368d74bfe9b7c4bcbaf636f2ebc78efcdae676347 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-7-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-820a2f8, org.label-schema.vendor=CentOS) 2024-04-21T23:30:32.276 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 23:30:32 smithi111 podman[67610]: 2024-04-21 23:30:32.277355326 +0000 UTC m=+0.378730624 container attach eccd68634c16a14aa2862f5368d74bfe9b7c4bcbaf636f2ebc78efcdae676347 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=squid-820a2f8, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T23:30:32.681 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:32 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:32.306+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.927390+0000 front 2024-04-21T23:29:33.927500+0000 (oldest deadline 2024-04-21T23:29:56.827517+0000) 2024-04-21T23:30:32.681 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:32 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:32.306+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:36.828037+0000 front 2024-04-21T23:29:36.827973+0000 (oldest deadline 2024-04-21T23:30:03.228609+0000) 2024-04-21T23:30:32.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:32 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:32.306+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:46.730173+0000 front 2024-04-21T23:29:46.729971+0000 (oldest deadline 2024-04-21T23:30:11.330259+0000) 2024-04-21T23:30:32.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:32 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:32.306+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:57.731564+0000 front 2024-04-21T23:29:57.731525+0000 (oldest deadline 2024-04-21T23:30:20.031550+0000) 2024-04-21T23:30:32.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:32 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:32.306+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:06.333794+0000 front 2024-04-21T23:30:06.333742+0000 (oldest deadline 2024-04-21T23:30:29.233758+0000) 2024-04-21T23:30:32.693 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 23:30:32 smithi111 podman[67610]: 2024-04-21 23:30:32.492594311 +0000 UTC m=+0.593969601 container died eccd68634c16a14aa2862f5368d74bfe9b7c4bcbaf636f2ebc78efcdae676347 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-820a2f8) 2024-04-21T23:30:32.694 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:32 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:32.439+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.197048+0000 front 2024-04-21T23:29:33.197157+0000 (oldest deadline 2024-04-21T23:29:59.097025+0000) 2024-04-21T23:30:32.694 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:32 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:32.439+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:47.298757+0000 front 2024-04-21T23:29:47.298709+0000 (oldest deadline 2024-04-21T23:30:13.198900+0000) 2024-04-21T23:30:32.694 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:32 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:32.439+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:57.900160+0000 front 2024-04-21T23:29:57.900044+0000 (oldest deadline 2024-04-21T23:30:20.199812+0000) 2024-04-21T23:30:32.694 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:32 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:32.439+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:03.700683+0000 front 2024-04-21T23:30:03.700711+0000 (oldest deadline 2024-04-21T23:30:27.800810+0000) 2024-04-21T23:30:33.136 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d814e4dc-0033-11ef-bc93-c7b262605968@osd.7.service' 2024-04-21T23:30:33.174 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T23:30:33.174 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-21T23:30:33.175 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-04-21T23:30:33.175 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ceph-d814e4dc-0033-11ef-bc93-c7b262605968@osd.8 2024-04-21T23:30:33.193 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:32 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:32.795+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:42.041804+0000 front 2024-04-21T23:29:42.041700+0000 (oldest deadline 2024-04-21T23:30:06.741665+0000) 2024-04-21T23:30:33.193 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:32 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:32.795+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:49.642677+0000 front 2024-04-21T23:29:49.642702+0000 (oldest deadline 2024-04-21T23:30:11.942839+0000) 2024-04-21T23:30:33.193 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:32 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:32.795+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:56.544455+0000 front 2024-04-21T23:29:56.544332+0000 (oldest deadline 2024-04-21T23:30:21.244522+0000) 2024-04-21T23:30:33.193 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:32 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:32.795+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:01.745868+0000 front 2024-04-21T23:30:01.745850+0000 (oldest deadline 2024-04-21T23:30:27.645830+0000) 2024-04-21T23:30:33.194 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:32 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:32.738+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:31.573404+0000 front 2024-04-21T23:29:31.573351+0000 (oldest deadline 2024-04-21T23:29:56.873350+0000) 2024-04-21T23:30:33.194 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:32 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:32.738+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.374892+0000 front 2024-04-21T23:29:40.374834+0000 (oldest deadline 2024-04-21T23:30:04.474654+0000) 2024-04-21T23:30:33.194 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:32 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:32.738+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:50.176921+0000 front 2024-04-21T23:29:50.176955+0000 (oldest deadline 2024-04-21T23:30:16.577589+0000) 2024-04-21T23:30:33.194 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:32 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:32.738+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:02.478617+0000 front 2024-04-21T23:30:02.478436+0000 (oldest deadline 2024-04-21T23:30:27.178522+0000) 2024-04-21T23:30:33.601 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:33 smithi174 systemd[1]: Stopping Ceph osd.8 for d814e4dc-0033-11ef-bc93-c7b262605968... 2024-04-21T23:30:33.681 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:33 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:33.285+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.927390+0000 front 2024-04-21T23:29:33.927500+0000 (oldest deadline 2024-04-21T23:29:56.827517+0000) 2024-04-21T23:30:33.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:33 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:33.285+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:36.828037+0000 front 2024-04-21T23:29:36.827973+0000 (oldest deadline 2024-04-21T23:30:03.228609+0000) 2024-04-21T23:30:33.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:33 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:33.285+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:46.730173+0000 front 2024-04-21T23:29:46.729971+0000 (oldest deadline 2024-04-21T23:30:11.330259+0000) 2024-04-21T23:30:33.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:33 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:33.285+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:57.731564+0000 front 2024-04-21T23:29:57.731525+0000 (oldest deadline 2024-04-21T23:30:20.031550+0000) 2024-04-21T23:30:33.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:33 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:33.285+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:06.333794+0000 front 2024-04-21T23:30:06.333742+0000 (oldest deadline 2024-04-21T23:30:29.233758+0000) 2024-04-21T23:30:33.943 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:33 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:33.747+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:31.573404+0000 front 2024-04-21T23:29:31.573351+0000 (oldest deadline 2024-04-21T23:29:56.873350+0000) 2024-04-21T23:30:33.944 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:33 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:33.747+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.374892+0000 front 2024-04-21T23:29:40.374834+0000 (oldest deadline 2024-04-21T23:30:04.474654+0000) 2024-04-21T23:30:33.944 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:33 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:33.747+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:50.176921+0000 front 2024-04-21T23:29:50.176955+0000 (oldest deadline 2024-04-21T23:30:16.577589+0000) 2024-04-21T23:30:33.944 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:33 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:33.747+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:02.478617+0000 front 2024-04-21T23:30:02.478436+0000 (oldest deadline 2024-04-21T23:30:27.178522+0000) 2024-04-21T23:30:33.944 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:33 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:33.602+0000 7fcab9825700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T23:30:33.944 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:33 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:33.602+0000 7fcab9825700 -1 osd.8 108 *** Got signal Terminated *** 2024-04-21T23:30:33.944 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:33 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:33.602+0000 7fcab9825700 -1 osd.8 108 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-21T23:30:33.944 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:33 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:33.781+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:42.041804+0000 front 2024-04-21T23:29:42.041700+0000 (oldest deadline 2024-04-21T23:30:06.741665+0000) 2024-04-21T23:30:33.945 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:33 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:33.781+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:49.642677+0000 front 2024-04-21T23:29:49.642702+0000 (oldest deadline 2024-04-21T23:30:11.942839+0000) 2024-04-21T23:30:33.945 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:33 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:33.781+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:56.544455+0000 front 2024-04-21T23:29:56.544332+0000 (oldest deadline 2024-04-21T23:30:21.244522+0000) 2024-04-21T23:30:33.945 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:33 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:33.781+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:01.745868+0000 front 2024-04-21T23:30:01.745850+0000 (oldest deadline 2024-04-21T23:30:27.645830+0000) 2024-04-21T23:30:33.945 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:33 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:33.458+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.197048+0000 front 2024-04-21T23:29:33.197157+0000 (oldest deadline 2024-04-21T23:29:59.097025+0000) 2024-04-21T23:30:33.945 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:33 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:33.458+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:47.298757+0000 front 2024-04-21T23:29:47.298709+0000 (oldest deadline 2024-04-21T23:30:13.198900+0000) 2024-04-21T23:30:33.945 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:33 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:33.458+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:57.900160+0000 front 2024-04-21T23:29:57.900044+0000 (oldest deadline 2024-04-21T23:30:20.199812+0000) 2024-04-21T23:30:33.946 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:33 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:33.458+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:03.700683+0000 front 2024-04-21T23:30:03.700711+0000 (oldest deadline 2024-04-21T23:30:27.800810+0000) 2024-04-21T23:30:34.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:34 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:34.238+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.927390+0000 front 2024-04-21T23:29:33.927500+0000 (oldest deadline 2024-04-21T23:29:56.827517+0000) 2024-04-21T23:30:34.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:34 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:34.238+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:36.828037+0000 front 2024-04-21T23:29:36.827973+0000 (oldest deadline 2024-04-21T23:30:03.228609+0000) 2024-04-21T23:30:34.683 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:34 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:34.238+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:46.730173+0000 front 2024-04-21T23:29:46.729971+0000 (oldest deadline 2024-04-21T23:30:11.330259+0000) 2024-04-21T23:30:34.683 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:34 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:34.238+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:57.731564+0000 front 2024-04-21T23:29:57.731525+0000 (oldest deadline 2024-04-21T23:30:20.031550+0000) 2024-04-21T23:30:34.683 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:34 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:34.238+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:06.333794+0000 front 2024-04-21T23:30:06.333742+0000 (oldest deadline 2024-04-21T23:30:29.233758+0000) 2024-04-21T23:30:34.943 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:34 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:34.465+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.197048+0000 front 2024-04-21T23:29:33.197157+0000 (oldest deadline 2024-04-21T23:29:59.097025+0000) 2024-04-21T23:30:34.944 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:34 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:34.465+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:47.298757+0000 front 2024-04-21T23:29:47.298709+0000 (oldest deadline 2024-04-21T23:30:13.198900+0000) 2024-04-21T23:30:34.944 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:34 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:34.465+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:57.900160+0000 front 2024-04-21T23:29:57.900044+0000 (oldest deadline 2024-04-21T23:30:20.199812+0000) 2024-04-21T23:30:34.944 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:34 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:34.465+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:03.700683+0000 front 2024-04-21T23:30:03.700711+0000 (oldest deadline 2024-04-21T23:30:27.800810+0000) 2024-04-21T23:30:35.193 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:34 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:34.818+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:42.041804+0000 front 2024-04-21T23:29:42.041700+0000 (oldest deadline 2024-04-21T23:30:06.741665+0000) 2024-04-21T23:30:35.193 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:34 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:34.818+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:49.642677+0000 front 2024-04-21T23:29:49.642702+0000 (oldest deadline 2024-04-21T23:30:11.942839+0000) 2024-04-21T23:30:35.194 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:34 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:34.818+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:56.544455+0000 front 2024-04-21T23:29:56.544332+0000 (oldest deadline 2024-04-21T23:30:21.244522+0000) 2024-04-21T23:30:35.194 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:34 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:34.818+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:01.745868+0000 front 2024-04-21T23:30:01.745850+0000 (oldest deadline 2024-04-21T23:30:27.645830+0000) 2024-04-21T23:30:35.194 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:34 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:34.778+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:31.573404+0000 front 2024-04-21T23:29:31.573351+0000 (oldest deadline 2024-04-21T23:29:56.873350+0000) 2024-04-21T23:30:35.194 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:34 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:34.778+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.374892+0000 front 2024-04-21T23:29:40.374834+0000 (oldest deadline 2024-04-21T23:30:04.474654+0000) 2024-04-21T23:30:35.195 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:34 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:34.778+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:50.176921+0000 front 2024-04-21T23:29:50.176955+0000 (oldest deadline 2024-04-21T23:30:16.577589+0000) 2024-04-21T23:30:35.195 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:34 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:34.778+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:02.478617+0000 front 2024-04-21T23:30:02.478436+0000 (oldest deadline 2024-04-21T23:30:27.178522+0000) 2024-04-21T23:30:35.681 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:35 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:35.225+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.927390+0000 front 2024-04-21T23:29:33.927500+0000 (oldest deadline 2024-04-21T23:29:56.827517+0000) 2024-04-21T23:30:35.681 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:35 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:35.225+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:36.828037+0000 front 2024-04-21T23:29:36.827973+0000 (oldest deadline 2024-04-21T23:30:03.228609+0000) 2024-04-21T23:30:35.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:35 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:35.225+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:46.730173+0000 front 2024-04-21T23:29:46.729971+0000 (oldest deadline 2024-04-21T23:30:11.330259+0000) 2024-04-21T23:30:35.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:35 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:35.225+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:57.731564+0000 front 2024-04-21T23:29:57.731525+0000 (oldest deadline 2024-04-21T23:30:20.031550+0000) 2024-04-21T23:30:35.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:35 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:35.225+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:06.333794+0000 front 2024-04-21T23:30:06.333742+0000 (oldest deadline 2024-04-21T23:30:29.233758+0000) 2024-04-21T23:30:35.683 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:35 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:35.225+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-04-21T23:30:13.335027+0000 front 2024-04-21T23:30:13.334985+0000 (oldest deadline 2024-04-21T23:30:35.034821+0000) 2024-04-21T23:30:35.945 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:35 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:35.514+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.197048+0000 front 2024-04-21T23:29:33.197157+0000 (oldest deadline 2024-04-21T23:29:59.097025+0000) 2024-04-21T23:30:35.945 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:35 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:35.514+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:47.298757+0000 front 2024-04-21T23:29:47.298709+0000 (oldest deadline 2024-04-21T23:30:13.198900+0000) 2024-04-21T23:30:35.945 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:35 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:35.514+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:57.900160+0000 front 2024-04-21T23:29:57.900044+0000 (oldest deadline 2024-04-21T23:30:20.199812+0000) 2024-04-21T23:30:35.945 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:35 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:35.514+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:03.700683+0000 front 2024-04-21T23:30:03.700711+0000 (oldest deadline 2024-04-21T23:30:27.800810+0000) 2024-04-21T23:30:35.945 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:35 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:35.514+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-04-21T23:30:12.802926+0000 front 2024-04-21T23:30:12.802986+0000 (oldest deadline 2024-04-21T23:30:35.102863+0000) 2024-04-21T23:30:36.193 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:35 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:35.775+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:31.573404+0000 front 2024-04-21T23:29:31.573351+0000 (oldest deadline 2024-04-21T23:29:56.873350+0000) 2024-04-21T23:30:36.193 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:35 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:35.775+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.374892+0000 front 2024-04-21T23:29:40.374834+0000 (oldest deadline 2024-04-21T23:30:04.474654+0000) 2024-04-21T23:30:36.193 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:35 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:35.775+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:50.176921+0000 front 2024-04-21T23:29:50.176955+0000 (oldest deadline 2024-04-21T23:30:16.577589+0000) 2024-04-21T23:30:36.193 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:35 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:35.775+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:02.478617+0000 front 2024-04-21T23:30:02.478436+0000 (oldest deadline 2024-04-21T23:30:27.178522+0000) 2024-04-21T23:30:36.194 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:35 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:35.795+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:42.041804+0000 front 2024-04-21T23:29:42.041700+0000 (oldest deadline 2024-04-21T23:30:06.741665+0000) 2024-04-21T23:30:36.194 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:35 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:35.795+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:49.642677+0000 front 2024-04-21T23:29:49.642702+0000 (oldest deadline 2024-04-21T23:30:11.942839+0000) 2024-04-21T23:30:36.194 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:35 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:35.795+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:56.544455+0000 front 2024-04-21T23:29:56.544332+0000 (oldest deadline 2024-04-21T23:30:21.244522+0000) 2024-04-21T23:30:36.194 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:35 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:35.795+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:01.745868+0000 front 2024-04-21T23:30:01.745850+0000 (oldest deadline 2024-04-21T23:30:27.645830+0000) 2024-04-21T23:30:36.194 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:35 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:35.795+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-04-21T23:30:13.947641+0000 front 2024-04-21T23:30:13.947594+0000 (oldest deadline 2024-04-21T23:30:35.047902+0000) 2024-04-21T23:30:36.681 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:36 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:36.254+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.927390+0000 front 2024-04-21T23:29:33.927500+0000 (oldest deadline 2024-04-21T23:29:56.827517+0000) 2024-04-21T23:30:36.681 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:36 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:36.254+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:36.828037+0000 front 2024-04-21T23:29:36.827973+0000 (oldest deadline 2024-04-21T23:30:03.228609+0000) 2024-04-21T23:30:36.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:36 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:36.254+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:46.730173+0000 front 2024-04-21T23:29:46.729971+0000 (oldest deadline 2024-04-21T23:30:11.330259+0000) 2024-04-21T23:30:36.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:36 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:36.254+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:57.731564+0000 front 2024-04-21T23:29:57.731525+0000 (oldest deadline 2024-04-21T23:30:20.031550+0000) 2024-04-21T23:30:36.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:36 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:36.254+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:06.333794+0000 front 2024-04-21T23:30:06.333742+0000 (oldest deadline 2024-04-21T23:30:29.233758+0000) 2024-04-21T23:30:36.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:36 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:36.254+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-04-21T23:30:13.335027+0000 front 2024-04-21T23:30:13.334985+0000 (oldest deadline 2024-04-21T23:30:35.034821+0000) 2024-04-21T23:30:36.943 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:36 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:36.562+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.197048+0000 front 2024-04-21T23:29:33.197157+0000 (oldest deadline 2024-04-21T23:29:59.097025+0000) 2024-04-21T23:30:36.944 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:36 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:36.562+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:47.298757+0000 front 2024-04-21T23:29:47.298709+0000 (oldest deadline 2024-04-21T23:30:13.198900+0000) 2024-04-21T23:30:36.944 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:36 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:36.562+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:57.900160+0000 front 2024-04-21T23:29:57.900044+0000 (oldest deadline 2024-04-21T23:30:20.199812+0000) 2024-04-21T23:30:36.944 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:36 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:36.562+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:03.700683+0000 front 2024-04-21T23:30:03.700711+0000 (oldest deadline 2024-04-21T23:30:27.800810+0000) 2024-04-21T23:30:36.944 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:36 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:36.562+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-04-21T23:30:12.802926+0000 front 2024-04-21T23:30:12.802986+0000 (oldest deadline 2024-04-21T23:30:35.102863+0000) 2024-04-21T23:30:37.193 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:36 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:36.795+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:42.041804+0000 front 2024-04-21T23:29:42.041700+0000 (oldest deadline 2024-04-21T23:30:06.741665+0000) 2024-04-21T23:30:37.193 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:36 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:36.795+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:49.642677+0000 front 2024-04-21T23:29:49.642702+0000 (oldest deadline 2024-04-21T23:30:11.942839+0000) 2024-04-21T23:30:37.194 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:36 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:36.795+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:56.544455+0000 front 2024-04-21T23:29:56.544332+0000 (oldest deadline 2024-04-21T23:30:21.244522+0000) 2024-04-21T23:30:37.194 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:36 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:36.795+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:01.745868+0000 front 2024-04-21T23:30:01.745850+0000 (oldest deadline 2024-04-21T23:30:27.645830+0000) 2024-04-21T23:30:37.194 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:36 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:36.795+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-04-21T23:30:13.947641+0000 front 2024-04-21T23:30:13.947594+0000 (oldest deadline 2024-04-21T23:30:35.047902+0000) 2024-04-21T23:30:37.194 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:36 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:36.767+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:31.573404+0000 front 2024-04-21T23:29:31.573351+0000 (oldest deadline 2024-04-21T23:29:56.873350+0000) 2024-04-21T23:30:37.194 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:36 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:36.767+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.374892+0000 front 2024-04-21T23:29:40.374834+0000 (oldest deadline 2024-04-21T23:30:04.474654+0000) 2024-04-21T23:30:37.194 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:36 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:36.767+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:50.176921+0000 front 2024-04-21T23:29:50.176955+0000 (oldest deadline 2024-04-21T23:30:16.577589+0000) 2024-04-21T23:30:37.194 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:36 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:36.767+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:02.478617+0000 front 2024-04-21T23:30:02.478436+0000 (oldest deadline 2024-04-21T23:30:27.178522+0000) 2024-04-21T23:30:37.195 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:36 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:36.767+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-04-21T23:30:10.980975+0000 front 2024-04-21T23:30:10.980934+0000 (oldest deadline 2024-04-21T23:30:36.280923+0000) 2024-04-21T23:30:37.681 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:37 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:37.276+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.927390+0000 front 2024-04-21T23:29:33.927500+0000 (oldest deadline 2024-04-21T23:29:56.827517+0000) 2024-04-21T23:30:37.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:37 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:37.276+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:36.828037+0000 front 2024-04-21T23:29:36.827973+0000 (oldest deadline 2024-04-21T23:30:03.228609+0000) 2024-04-21T23:30:37.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:37 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:37.276+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:46.730173+0000 front 2024-04-21T23:29:46.729971+0000 (oldest deadline 2024-04-21T23:30:11.330259+0000) 2024-04-21T23:30:37.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:37 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:37.276+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:57.731564+0000 front 2024-04-21T23:29:57.731525+0000 (oldest deadline 2024-04-21T23:30:20.031550+0000) 2024-04-21T23:30:37.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:37 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:37.276+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:06.333794+0000 front 2024-04-21T23:30:06.333742+0000 (oldest deadline 2024-04-21T23:30:29.233758+0000) 2024-04-21T23:30:37.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:37 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:37.276+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-04-21T23:30:13.335027+0000 front 2024-04-21T23:30:13.334985+0000 (oldest deadline 2024-04-21T23:30:35.034821+0000) 2024-04-21T23:30:37.943 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:37 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:37.558+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.197048+0000 front 2024-04-21T23:29:33.197157+0000 (oldest deadline 2024-04-21T23:29:59.097025+0000) 2024-04-21T23:30:37.944 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:37 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:37.558+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:47.298757+0000 front 2024-04-21T23:29:47.298709+0000 (oldest deadline 2024-04-21T23:30:13.198900+0000) 2024-04-21T23:30:37.944 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:37 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:37.558+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:57.900160+0000 front 2024-04-21T23:29:57.900044+0000 (oldest deadline 2024-04-21T23:30:20.199812+0000) 2024-04-21T23:30:37.944 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:37 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:37.558+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:03.700683+0000 front 2024-04-21T23:30:03.700711+0000 (oldest deadline 2024-04-21T23:30:27.800810+0000) 2024-04-21T23:30:37.944 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:37 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:37.558+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-04-21T23:30:12.802926+0000 front 2024-04-21T23:30:12.802986+0000 (oldest deadline 2024-04-21T23:30:35.102863+0000) 2024-04-21T23:30:38.193 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:37 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:37.829+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:42.041804+0000 front 2024-04-21T23:29:42.041700+0000 (oldest deadline 2024-04-21T23:30:06.741665+0000) 2024-04-21T23:30:38.193 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:37 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:37.829+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:49.642677+0000 front 2024-04-21T23:29:49.642702+0000 (oldest deadline 2024-04-21T23:30:11.942839+0000) 2024-04-21T23:30:38.193 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:37 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:37.829+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:56.544455+0000 front 2024-04-21T23:29:56.544332+0000 (oldest deadline 2024-04-21T23:30:21.244522+0000) 2024-04-21T23:30:38.194 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:37 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:37.829+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:01.745868+0000 front 2024-04-21T23:30:01.745850+0000 (oldest deadline 2024-04-21T23:30:27.645830+0000) 2024-04-21T23:30:38.194 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:37 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8[50309]: 2024-04-21T23:30:37.829+0000 7fcab5e19700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-04-21T23:30:13.947641+0000 front 2024-04-21T23:30:13.947594+0000 (oldest deadline 2024-04-21T23:30:35.047902+0000) 2024-04-21T23:30:38.194 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:37 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:37.782+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:31.573404+0000 front 2024-04-21T23:29:31.573351+0000 (oldest deadline 2024-04-21T23:29:56.873350+0000) 2024-04-21T23:30:38.194 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:37 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:37.782+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.374892+0000 front 2024-04-21T23:29:40.374834+0000 (oldest deadline 2024-04-21T23:30:04.474654+0000) 2024-04-21T23:30:38.194 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:37 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:37.782+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:50.176921+0000 front 2024-04-21T23:29:50.176955+0000 (oldest deadline 2024-04-21T23:30:16.577589+0000) 2024-04-21T23:30:38.194 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:37 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:37.782+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:02.478617+0000 front 2024-04-21T23:30:02.478436+0000 (oldest deadline 2024-04-21T23:30:27.178522+0000) 2024-04-21T23:30:38.195 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:37 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:37.783+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-04-21T23:30:10.980975+0000 front 2024-04-21T23:30:10.980934+0000 (oldest deadline 2024-04-21T23:30:36.280923+0000) 2024-04-21T23:30:38.681 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:38 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:38.280+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.927390+0000 front 2024-04-21T23:29:33.927500+0000 (oldest deadline 2024-04-21T23:29:56.827517+0000) 2024-04-21T23:30:38.681 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:38 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:38.280+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:36.828037+0000 front 2024-04-21T23:29:36.827973+0000 (oldest deadline 2024-04-21T23:30:03.228609+0000) 2024-04-21T23:30:38.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:38 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:38.280+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:46.730173+0000 front 2024-04-21T23:29:46.729971+0000 (oldest deadline 2024-04-21T23:30:11.330259+0000) 2024-04-21T23:30:38.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:38 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:38.280+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:57.731564+0000 front 2024-04-21T23:29:57.731525+0000 (oldest deadline 2024-04-21T23:30:20.031550+0000) 2024-04-21T23:30:38.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:38 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:38.280+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:06.333794+0000 front 2024-04-21T23:30:06.333742+0000 (oldest deadline 2024-04-21T23:30:29.233758+0000) 2024-04-21T23:30:38.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:38 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:38.280+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-04-21T23:30:13.335027+0000 front 2024-04-21T23:30:13.334985+0000 (oldest deadline 2024-04-21T23:30:35.034821+0000) 2024-04-21T23:30:38.943 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:38 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:38.514+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.197048+0000 front 2024-04-21T23:29:33.197157+0000 (oldest deadline 2024-04-21T23:29:59.097025+0000) 2024-04-21T23:30:38.944 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:38 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:38.514+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:47.298757+0000 front 2024-04-21T23:29:47.298709+0000 (oldest deadline 2024-04-21T23:30:13.198900+0000) 2024-04-21T23:30:38.944 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:38 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:38.514+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:57.900160+0000 front 2024-04-21T23:29:57.900044+0000 (oldest deadline 2024-04-21T23:30:20.199812+0000) 2024-04-21T23:30:38.944 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:38 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:38.514+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:03.700683+0000 front 2024-04-21T23:30:03.700711+0000 (oldest deadline 2024-04-21T23:30:27.800810+0000) 2024-04-21T23:30:38.944 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:38 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:38.514+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-04-21T23:30:12.802926+0000 front 2024-04-21T23:30:12.802986+0000 (oldest deadline 2024-04-21T23:30:35.102863+0000) 2024-04-21T23:30:39.132 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:38 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:38.806+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:31.573404+0000 front 2024-04-21T23:29:31.573351+0000 (oldest deadline 2024-04-21T23:29:56.873350+0000) 2024-04-21T23:30:39.133 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:38 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:38.806+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.374892+0000 front 2024-04-21T23:29:40.374834+0000 (oldest deadline 2024-04-21T23:30:04.474654+0000) 2024-04-21T23:30:39.133 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:38 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:38.806+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:50.176921+0000 front 2024-04-21T23:29:50.176955+0000 (oldest deadline 2024-04-21T23:30:16.577589+0000) 2024-04-21T23:30:39.133 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:38 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:38.806+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:02.478617+0000 front 2024-04-21T23:30:02.478436+0000 (oldest deadline 2024-04-21T23:30:27.178522+0000) 2024-04-21T23:30:39.133 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:38 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:38.806+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-04-21T23:30:10.980975+0000 front 2024-04-21T23:30:10.980934+0000 (oldest deadline 2024-04-21T23:30:36.280923+0000) 2024-04-21T23:30:39.426 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:39 smithi174 podman[68332]: 2024-04-21 23:30:39.134821162 +0000 UTC m=+5.766579666 container died c129400dfb96a7329b1da802a51dccc1067fb49fa5841cbfe4479ecf9b480d37 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=squid-820a2f8, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.license=GPLv2) 2024-04-21T23:30:39.681 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:39 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:39.245+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.927390+0000 front 2024-04-21T23:29:33.927500+0000 (oldest deadline 2024-04-21T23:29:56.827517+0000) 2024-04-21T23:30:39.681 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:39 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:39.245+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:36.828037+0000 front 2024-04-21T23:29:36.827973+0000 (oldest deadline 2024-04-21T23:30:03.228609+0000) 2024-04-21T23:30:39.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:39 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:39.245+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:46.730173+0000 front 2024-04-21T23:29:46.729971+0000 (oldest deadline 2024-04-21T23:30:11.330259+0000) 2024-04-21T23:30:39.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:39 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:39.246+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:57.731564+0000 front 2024-04-21T23:29:57.731525+0000 (oldest deadline 2024-04-21T23:30:20.031550+0000) 2024-04-21T23:30:39.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:39 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:39.246+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:06.333794+0000 front 2024-04-21T23:30:06.333742+0000 (oldest deadline 2024-04-21T23:30:29.233758+0000) 2024-04-21T23:30:39.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:39 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:39.246+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-04-21T23:30:13.335027+0000 front 2024-04-21T23:30:13.334985+0000 (oldest deadline 2024-04-21T23:30:35.034821+0000) 2024-04-21T23:30:39.693 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:39 smithi174 podman[68332]: 2024-04-21 23:30:39.52737003 +0000 UTC m=+6.159128525 container cleanup c129400dfb96a7329b1da802a51dccc1067fb49fa5841cbfe4479ecf9b480d37 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-820a2f8, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-21T23:30:39.693 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:39 smithi174 bash[68332]: ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8 2024-04-21T23:30:39.943 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:39 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:39.548+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.197048+0000 front 2024-04-21T23:29:33.197157+0000 (oldest deadline 2024-04-21T23:29:59.097025+0000) 2024-04-21T23:30:39.944 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:39 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:39.548+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:47.298757+0000 front 2024-04-21T23:29:47.298709+0000 (oldest deadline 2024-04-21T23:30:13.198900+0000) 2024-04-21T23:30:39.944 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:39 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:39.548+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:57.900160+0000 front 2024-04-21T23:29:57.900044+0000 (oldest deadline 2024-04-21T23:30:20.199812+0000) 2024-04-21T23:30:39.944 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:39 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:39.548+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:03.700683+0000 front 2024-04-21T23:30:03.700711+0000 (oldest deadline 2024-04-21T23:30:27.800810+0000) 2024-04-21T23:30:39.944 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:39 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:39.548+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-04-21T23:30:12.802926+0000 front 2024-04-21T23:30:12.802986+0000 (oldest deadline 2024-04-21T23:30:35.102863+0000) 2024-04-21T23:30:40.129 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:39 smithi174 podman[68350]: 2024-04-21 23:30:39.722324541 +0000 UTC m=+0.589562526 container remove c129400dfb96a7329b1da802a51dccc1067fb49fa5841cbfe4479ecf9b480d37 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8, RELEASE=squid-820a2f8, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416) 2024-04-21T23:30:40.130 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:39 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:39.850+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:31.573404+0000 front 2024-04-21T23:29:31.573351+0000 (oldest deadline 2024-04-21T23:29:56.873350+0000) 2024-04-21T23:30:40.130 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:39 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:39.850+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.374892+0000 front 2024-04-21T23:29:40.374834+0000 (oldest deadline 2024-04-21T23:30:04.474654+0000) 2024-04-21T23:30:40.130 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:39 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:39.850+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:50.176921+0000 front 2024-04-21T23:29:50.176955+0000 (oldest deadline 2024-04-21T23:30:16.577589+0000) 2024-04-21T23:30:40.130 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:39 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:39.850+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:02.478617+0000 front 2024-04-21T23:30:02.478436+0000 (oldest deadline 2024-04-21T23:30:27.178522+0000) 2024-04-21T23:30:40.130 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:39 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:39.850+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-04-21T23:30:10.980975+0000 front 2024-04-21T23:30:10.980934+0000 (oldest deadline 2024-04-21T23:30:36.280923+0000) 2024-04-21T23:30:40.418 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:40 smithi174 podman[68440]: 2024-04-21 23:30:40.131700277 +0000 UTC m=+0.112169318 container create 6e5ba73fee09ebfc48664ec779a08275be77b0c714e1655b74168ad2e5a839a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8-deactivate, RELEASE=squid-820a2f8, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T23:30:40.418 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:40 smithi174 podman[68440]: 2024-04-21 23:30:40.037410966 +0000 UTC m=+0.017880016 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-21T23:30:40.681 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:40 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:40.217+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.927390+0000 front 2024-04-21T23:29:33.927500+0000 (oldest deadline 2024-04-21T23:29:56.827517+0000) 2024-04-21T23:30:40.681 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:40 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:40.217+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:36.828037+0000 front 2024-04-21T23:29:36.827973+0000 (oldest deadline 2024-04-21T23:30:03.228609+0000) 2024-04-21T23:30:40.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:40 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:40.217+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:46.730173+0000 front 2024-04-21T23:29:46.729971+0000 (oldest deadline 2024-04-21T23:30:11.330259+0000) 2024-04-21T23:30:40.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:40 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:40.217+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:57.731564+0000 front 2024-04-21T23:29:57.731525+0000 (oldest deadline 2024-04-21T23:30:20.031550+0000) 2024-04-21T23:30:40.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:40 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:40.217+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:06.333794+0000 front 2024-04-21T23:30:06.333742+0000 (oldest deadline 2024-04-21T23:30:29.233758+0000) 2024-04-21T23:30:40.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:40 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:40.217+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-04-21T23:30:13.335027+0000 front 2024-04-21T23:30:13.334985+0000 (oldest deadline 2024-04-21T23:30:35.034821+0000) 2024-04-21T23:30:40.693 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:40 smithi174 podman[68440]: 2024-04-21 23:30:40.420191221 +0000 UTC m=+0.400660268 container init 6e5ba73fee09ebfc48664ec779a08275be77b0c714e1655b74168ad2e5a839a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-820a2f8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T23:30:40.693 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:40 smithi174 podman[68440]: 2024-04-21 23:30:40.423659914 +0000 UTC m=+0.404128956 container start 6e5ba73fee09ebfc48664ec779a08275be77b0c714e1655b74168ad2e5a839a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-820a2f8, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-21T23:30:40.693 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:40 smithi174 podman[68440]: 2024-04-21 23:30:40.517381078 +0000 UTC m=+0.497850121 container attach 6e5ba73fee09ebfc48664ec779a08275be77b0c714e1655b74168ad2e5a839a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-820a2f8) 2024-04-21T23:30:40.943 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:40 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:40.556+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.197048+0000 front 2024-04-21T23:29:33.197157+0000 (oldest deadline 2024-04-21T23:29:59.097025+0000) 2024-04-21T23:30:40.943 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:40 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:40.556+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:47.298757+0000 front 2024-04-21T23:29:47.298709+0000 (oldest deadline 2024-04-21T23:30:13.198900+0000) 2024-04-21T23:30:40.944 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:40 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:40.556+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:57.900160+0000 front 2024-04-21T23:29:57.900044+0000 (oldest deadline 2024-04-21T23:30:20.199812+0000) 2024-04-21T23:30:40.944 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:40 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:40.556+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:03.700683+0000 front 2024-04-21T23:30:03.700711+0000 (oldest deadline 2024-04-21T23:30:27.800810+0000) 2024-04-21T23:30:40.944 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:40 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:40.556+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-04-21T23:30:12.802926+0000 front 2024-04-21T23:30:12.802986+0000 (oldest deadline 2024-04-21T23:30:35.102863+0000) 2024-04-21T23:30:41.001 INFO:journalctl@ceph.osd.8.smithi174.stdout:Apr 21 23:30:40 smithi174 podman[68440]: 2024-04-21 23:30:40.709785943 +0000 UTC m=+0.690254988 container died 6e5ba73fee09ebfc48664ec779a08275be77b0c714e1655b74168ad2e5a839a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-8-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-820a2f8, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T23:30:41.001 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:40 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:40.847+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:31.573404+0000 front 2024-04-21T23:29:31.573351+0000 (oldest deadline 2024-04-21T23:29:56.873350+0000) 2024-04-21T23:30:41.001 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:40 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:40.847+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.374892+0000 front 2024-04-21T23:29:40.374834+0000 (oldest deadline 2024-04-21T23:30:04.474654+0000) 2024-04-21T23:30:41.001 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:40 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:40.847+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:50.176921+0000 front 2024-04-21T23:29:50.176955+0000 (oldest deadline 2024-04-21T23:30:16.577589+0000) 2024-04-21T23:30:41.001 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:40 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:40.847+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:02.478617+0000 front 2024-04-21T23:30:02.478436+0000 (oldest deadline 2024-04-21T23:30:27.178522+0000) 2024-04-21T23:30:41.002 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:40 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:40.847+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-04-21T23:30:10.980975+0000 front 2024-04-21T23:30:10.980934+0000 (oldest deadline 2024-04-21T23:30:36.280923+0000) 2024-04-21T23:30:41.314 DEBUG:teuthology.orchestra.run.smithi174:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d814e4dc-0033-11ef-bc93-c7b262605968@osd.8.service' 2024-04-21T23:30:41.392 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T23:30:41.392 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-04-21T23:30:41.392 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-04-21T23:30:41.392 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-d814e4dc-0033-11ef-bc93-c7b262605968@osd.9 2024-04-21T23:30:41.681 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:41 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:41.229+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.927390+0000 front 2024-04-21T23:29:33.927500+0000 (oldest deadline 2024-04-21T23:29:56.827517+0000) 2024-04-21T23:30:41.681 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:41 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:41.229+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:36.828037+0000 front 2024-04-21T23:29:36.827973+0000 (oldest deadline 2024-04-21T23:30:03.228609+0000) 2024-04-21T23:30:41.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:41 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:41.229+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:46.730173+0000 front 2024-04-21T23:29:46.729971+0000 (oldest deadline 2024-04-21T23:30:11.330259+0000) 2024-04-21T23:30:41.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:41 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:41.229+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:57.731564+0000 front 2024-04-21T23:29:57.731525+0000 (oldest deadline 2024-04-21T23:30:20.031550+0000) 2024-04-21T23:30:41.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:41 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:41.229+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:06.333794+0000 front 2024-04-21T23:30:06.333742+0000 (oldest deadline 2024-04-21T23:30:29.233758+0000) 2024-04-21T23:30:41.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:41 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:41.229+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-04-21T23:30:13.335027+0000 front 2024-04-21T23:30:13.334985+0000 (oldest deadline 2024-04-21T23:30:35.034821+0000) 2024-04-21T23:30:41.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:41 smithi040 systemd[1]: Stopping Ceph osd.9 for d814e4dc-0033-11ef-bc93-c7b262605968... 2024-04-21T23:30:41.943 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:41 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:41.507+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.197048+0000 front 2024-04-21T23:29:33.197157+0000 (oldest deadline 2024-04-21T23:29:59.097025+0000) 2024-04-21T23:30:41.943 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:41 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:41.507+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:47.298757+0000 front 2024-04-21T23:29:47.298709+0000 (oldest deadline 2024-04-21T23:30:13.198900+0000) 2024-04-21T23:30:41.944 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:41 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:41.507+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:57.900160+0000 front 2024-04-21T23:29:57.900044+0000 (oldest deadline 2024-04-21T23:30:20.199812+0000) 2024-04-21T23:30:41.944 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:41 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:41.507+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:03.700683+0000 front 2024-04-21T23:30:03.700711+0000 (oldest deadline 2024-04-21T23:30:27.800810+0000) 2024-04-21T23:30:41.944 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:41 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:41.507+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-04-21T23:30:12.802926+0000 front 2024-04-21T23:30:12.802986+0000 (oldest deadline 2024-04-21T23:30:35.102863+0000) 2024-04-21T23:30:42.181 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:41 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:41.710+0000 7fef21e2d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T23:30:42.181 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:41 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:41.710+0000 7fef21e2d700 -1 osd.9 108 *** Got signal Terminated *** 2024-04-21T23:30:42.181 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:41 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:41.710+0000 7fef21e2d700 -1 osd.9 108 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-21T23:30:42.193 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:41 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:41.874+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:31.573404+0000 front 2024-04-21T23:29:31.573351+0000 (oldest deadline 2024-04-21T23:29:56.873350+0000) 2024-04-21T23:30:42.193 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:41 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:41.874+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.374892+0000 front 2024-04-21T23:29:40.374834+0000 (oldest deadline 2024-04-21T23:30:04.474654+0000) 2024-04-21T23:30:42.193 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:41 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:41.874+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:50.176921+0000 front 2024-04-21T23:29:50.176955+0000 (oldest deadline 2024-04-21T23:30:16.577589+0000) 2024-04-21T23:30:42.193 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:41 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:41.874+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:02.478617+0000 front 2024-04-21T23:30:02.478436+0000 (oldest deadline 2024-04-21T23:30:27.178522+0000) 2024-04-21T23:30:42.193 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:41 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:41.874+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-04-21T23:30:10.980975+0000 front 2024-04-21T23:30:10.980934+0000 (oldest deadline 2024-04-21T23:30:36.280923+0000) 2024-04-21T23:30:42.681 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:42 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:42.248+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.927390+0000 front 2024-04-21T23:29:33.927500+0000 (oldest deadline 2024-04-21T23:29:56.827517+0000) 2024-04-21T23:30:42.681 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:42 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:42.248+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:36.828037+0000 front 2024-04-21T23:29:36.827973+0000 (oldest deadline 2024-04-21T23:30:03.228609+0000) 2024-04-21T23:30:42.681 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:42 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:42.248+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:46.730173+0000 front 2024-04-21T23:29:46.729971+0000 (oldest deadline 2024-04-21T23:30:11.330259+0000) 2024-04-21T23:30:42.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:42 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:42.248+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:57.731564+0000 front 2024-04-21T23:29:57.731525+0000 (oldest deadline 2024-04-21T23:30:20.031550+0000) 2024-04-21T23:30:42.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:42 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:42.248+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:06.333794+0000 front 2024-04-21T23:30:06.333742+0000 (oldest deadline 2024-04-21T23:30:29.233758+0000) 2024-04-21T23:30:42.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:42 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:42.248+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-04-21T23:30:13.335027+0000 front 2024-04-21T23:30:13.334985+0000 (oldest deadline 2024-04-21T23:30:35.034821+0000) 2024-04-21T23:30:42.943 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:42 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:42.512+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.197048+0000 front 2024-04-21T23:29:33.197157+0000 (oldest deadline 2024-04-21T23:29:59.097025+0000) 2024-04-21T23:30:42.943 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:42 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:42.512+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:47.298757+0000 front 2024-04-21T23:29:47.298709+0000 (oldest deadline 2024-04-21T23:30:13.198900+0000) 2024-04-21T23:30:42.944 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:42 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:42.512+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:57.900160+0000 front 2024-04-21T23:29:57.900044+0000 (oldest deadline 2024-04-21T23:30:20.199812+0000) 2024-04-21T23:30:42.944 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:42 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:42.512+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:03.700683+0000 front 2024-04-21T23:30:03.700711+0000 (oldest deadline 2024-04-21T23:30:27.800810+0000) 2024-04-21T23:30:42.944 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:42 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:42.512+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-04-21T23:30:12.802926+0000 front 2024-04-21T23:30:12.802986+0000 (oldest deadline 2024-04-21T23:30:35.102863+0000) 2024-04-21T23:30:43.193 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:42 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:42.845+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:31.573404+0000 front 2024-04-21T23:29:31.573351+0000 (oldest deadline 2024-04-21T23:29:56.873350+0000) 2024-04-21T23:30:43.193 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:42 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:42.845+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.374892+0000 front 2024-04-21T23:29:40.374834+0000 (oldest deadline 2024-04-21T23:30:04.474654+0000) 2024-04-21T23:30:43.193 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:42 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:42.845+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:50.176921+0000 front 2024-04-21T23:29:50.176955+0000 (oldest deadline 2024-04-21T23:30:16.577589+0000) 2024-04-21T23:30:43.194 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:42 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:42.845+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:02.478617+0000 front 2024-04-21T23:30:02.478436+0000 (oldest deadline 2024-04-21T23:30:27.178522+0000) 2024-04-21T23:30:43.194 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:42 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:42.845+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-04-21T23:30:10.980975+0000 front 2024-04-21T23:30:10.980934+0000 (oldest deadline 2024-04-21T23:30:36.280923+0000) 2024-04-21T23:30:43.681 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:43 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:43.204+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.927390+0000 front 2024-04-21T23:29:33.927500+0000 (oldest deadline 2024-04-21T23:29:56.827517+0000) 2024-04-21T23:30:43.681 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:43 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:43.204+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:36.828037+0000 front 2024-04-21T23:29:36.827973+0000 (oldest deadline 2024-04-21T23:30:03.228609+0000) 2024-04-21T23:30:43.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:43 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:43.204+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:46.730173+0000 front 2024-04-21T23:29:46.729971+0000 (oldest deadline 2024-04-21T23:30:11.330259+0000) 2024-04-21T23:30:43.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:43 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:43.204+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:57.731564+0000 front 2024-04-21T23:29:57.731525+0000 (oldest deadline 2024-04-21T23:30:20.031550+0000) 2024-04-21T23:30:43.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:43 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:43.204+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:06.333794+0000 front 2024-04-21T23:30:06.333742+0000 (oldest deadline 2024-04-21T23:30:29.233758+0000) 2024-04-21T23:30:43.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:43 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:43.204+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-04-21T23:30:13.335027+0000 front 2024-04-21T23:30:13.334985+0000 (oldest deadline 2024-04-21T23:30:35.034821+0000) 2024-04-21T23:30:43.844 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:43 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:43.522+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.197048+0000 front 2024-04-21T23:29:33.197157+0000 (oldest deadline 2024-04-21T23:29:59.097025+0000) 2024-04-21T23:30:43.845 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:43 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:43.522+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:47.298757+0000 front 2024-04-21T23:29:47.298709+0000 (oldest deadline 2024-04-21T23:30:13.198900+0000) 2024-04-21T23:30:43.845 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:43 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:43.522+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:57.900160+0000 front 2024-04-21T23:29:57.900044+0000 (oldest deadline 2024-04-21T23:30:20.199812+0000) 2024-04-21T23:30:43.845 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:43 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:43.522+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:03.700683+0000 front 2024-04-21T23:30:03.700711+0000 (oldest deadline 2024-04-21T23:30:27.800810+0000) 2024-04-21T23:30:43.845 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:43 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:43.522+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-04-21T23:30:12.802926+0000 front 2024-04-21T23:30:12.802986+0000 (oldest deadline 2024-04-21T23:30:35.102863+0000) 2024-04-21T23:30:44.193 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:43 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:43.810+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:31.573404+0000 front 2024-04-21T23:29:31.573351+0000 (oldest deadline 2024-04-21T23:29:56.873350+0000) 2024-04-21T23:30:44.193 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:43 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:43.810+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.374892+0000 front 2024-04-21T23:29:40.374834+0000 (oldest deadline 2024-04-21T23:30:04.474654+0000) 2024-04-21T23:30:44.193 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:43 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:43.810+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:50.176921+0000 front 2024-04-21T23:29:50.176955+0000 (oldest deadline 2024-04-21T23:30:16.577589+0000) 2024-04-21T23:30:44.193 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:43 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:43.810+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:02.478617+0000 front 2024-04-21T23:30:02.478436+0000 (oldest deadline 2024-04-21T23:30:27.178522+0000) 2024-04-21T23:30:44.194 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:43 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:43.810+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-04-21T23:30:10.980975+0000 front 2024-04-21T23:30:10.980934+0000 (oldest deadline 2024-04-21T23:30:36.280923+0000) 2024-04-21T23:30:44.681 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:44 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:44.254+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.927390+0000 front 2024-04-21T23:29:33.927500+0000 (oldest deadline 2024-04-21T23:29:56.827517+0000) 2024-04-21T23:30:44.681 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:44 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:44.254+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:36.828037+0000 front 2024-04-21T23:29:36.827973+0000 (oldest deadline 2024-04-21T23:30:03.228609+0000) 2024-04-21T23:30:44.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:44 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:44.254+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:46.730173+0000 front 2024-04-21T23:29:46.729971+0000 (oldest deadline 2024-04-21T23:30:11.330259+0000) 2024-04-21T23:30:44.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:44 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:44.254+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:57.731564+0000 front 2024-04-21T23:29:57.731525+0000 (oldest deadline 2024-04-21T23:30:20.031550+0000) 2024-04-21T23:30:44.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:44 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:44.254+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:06.333794+0000 front 2024-04-21T23:30:06.333742+0000 (oldest deadline 2024-04-21T23:30:29.233758+0000) 2024-04-21T23:30:44.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:44 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:44.254+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-04-21T23:30:13.335027+0000 front 2024-04-21T23:30:13.334985+0000 (oldest deadline 2024-04-21T23:30:35.034821+0000) 2024-04-21T23:30:44.943 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:44 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:44.500+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.197048+0000 front 2024-04-21T23:29:33.197157+0000 (oldest deadline 2024-04-21T23:29:59.097025+0000) 2024-04-21T23:30:44.944 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:44 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:44.500+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:47.298757+0000 front 2024-04-21T23:29:47.298709+0000 (oldest deadline 2024-04-21T23:30:13.198900+0000) 2024-04-21T23:30:44.944 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:44 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:44.500+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:57.900160+0000 front 2024-04-21T23:29:57.900044+0000 (oldest deadline 2024-04-21T23:30:20.199812+0000) 2024-04-21T23:30:44.944 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:44 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:44.500+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:03.700683+0000 front 2024-04-21T23:30:03.700711+0000 (oldest deadline 2024-04-21T23:30:27.800810+0000) 2024-04-21T23:30:44.944 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:44 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:44.500+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-04-21T23:30:12.802926+0000 front 2024-04-21T23:30:12.802986+0000 (oldest deadline 2024-04-21T23:30:35.102863+0000) 2024-04-21T23:30:45.094 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:44 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:44.832+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:31.573404+0000 front 2024-04-21T23:29:31.573351+0000 (oldest deadline 2024-04-21T23:29:56.873350+0000) 2024-04-21T23:30:45.095 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:44 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:44.832+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.374892+0000 front 2024-04-21T23:29:40.374834+0000 (oldest deadline 2024-04-21T23:30:04.474654+0000) 2024-04-21T23:30:45.095 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:44 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:44.832+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:50.176921+0000 front 2024-04-21T23:29:50.176955+0000 (oldest deadline 2024-04-21T23:30:16.577589+0000) 2024-04-21T23:30:45.095 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:44 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:44.832+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:02.478617+0000 front 2024-04-21T23:30:02.478436+0000 (oldest deadline 2024-04-21T23:30:27.178522+0000) 2024-04-21T23:30:45.095 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:44 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:44.832+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-04-21T23:30:10.980975+0000 front 2024-04-21T23:30:10.980934+0000 (oldest deadline 2024-04-21T23:30:36.280923+0000) 2024-04-21T23:30:45.681 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:45 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:45.214+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.927390+0000 front 2024-04-21T23:29:33.927500+0000 (oldest deadline 2024-04-21T23:29:56.827517+0000) 2024-04-21T23:30:45.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:45 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:45.214+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:36.828037+0000 front 2024-04-21T23:29:36.827973+0000 (oldest deadline 2024-04-21T23:30:03.228609+0000) 2024-04-21T23:30:45.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:45 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:45.214+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:46.730173+0000 front 2024-04-21T23:29:46.729971+0000 (oldest deadline 2024-04-21T23:30:11.330259+0000) 2024-04-21T23:30:45.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:45 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:45.214+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:57.731564+0000 front 2024-04-21T23:29:57.731525+0000 (oldest deadline 2024-04-21T23:30:20.031550+0000) 2024-04-21T23:30:45.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:45 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:45.214+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:06.333794+0000 front 2024-04-21T23:30:06.333742+0000 (oldest deadline 2024-04-21T23:30:29.233758+0000) 2024-04-21T23:30:45.683 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:45 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:45.214+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-04-21T23:30:13.335027+0000 front 2024-04-21T23:30:13.334985+0000 (oldest deadline 2024-04-21T23:30:35.034821+0000) 2024-04-21T23:30:45.943 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:45 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:45.502+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.197048+0000 front 2024-04-21T23:29:33.197157+0000 (oldest deadline 2024-04-21T23:29:59.097025+0000) 2024-04-21T23:30:45.944 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:45 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:45.502+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:47.298757+0000 front 2024-04-21T23:29:47.298709+0000 (oldest deadline 2024-04-21T23:30:13.198900+0000) 2024-04-21T23:30:45.944 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:45 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:45.502+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:57.900160+0000 front 2024-04-21T23:29:57.900044+0000 (oldest deadline 2024-04-21T23:30:20.199812+0000) 2024-04-21T23:30:45.944 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:45 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:45.502+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:03.700683+0000 front 2024-04-21T23:30:03.700711+0000 (oldest deadline 2024-04-21T23:30:27.800810+0000) 2024-04-21T23:30:45.945 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:45 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:45.502+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-04-21T23:30:12.802926+0000 front 2024-04-21T23:30:12.802986+0000 (oldest deadline 2024-04-21T23:30:35.102863+0000) 2024-04-21T23:30:46.193 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:45 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:45.867+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:31.573404+0000 front 2024-04-21T23:29:31.573351+0000 (oldest deadline 2024-04-21T23:29:56.873350+0000) 2024-04-21T23:30:46.193 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:45 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:45.867+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.374892+0000 front 2024-04-21T23:29:40.374834+0000 (oldest deadline 2024-04-21T23:30:04.474654+0000) 2024-04-21T23:30:46.193 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:45 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:45.867+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:50.176921+0000 front 2024-04-21T23:29:50.176955+0000 (oldest deadline 2024-04-21T23:30:16.577589+0000) 2024-04-21T23:30:46.193 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:45 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:45.867+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:02.478617+0000 front 2024-04-21T23:30:02.478436+0000 (oldest deadline 2024-04-21T23:30:27.178522+0000) 2024-04-21T23:30:46.193 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:45 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:45.867+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-04-21T23:30:10.980975+0000 front 2024-04-21T23:30:10.980934+0000 (oldest deadline 2024-04-21T23:30:36.280923+0000) 2024-04-21T23:30:46.681 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:46 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:46.231+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.927390+0000 front 2024-04-21T23:29:33.927500+0000 (oldest deadline 2024-04-21T23:29:56.827517+0000) 2024-04-21T23:30:46.681 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:46 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:46.231+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:36.828037+0000 front 2024-04-21T23:29:36.827973+0000 (oldest deadline 2024-04-21T23:30:03.228609+0000) 2024-04-21T23:30:46.681 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:46 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:46.231+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:46.730173+0000 front 2024-04-21T23:29:46.729971+0000 (oldest deadline 2024-04-21T23:30:11.330259+0000) 2024-04-21T23:30:46.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:46 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:46.231+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:57.731564+0000 front 2024-04-21T23:29:57.731525+0000 (oldest deadline 2024-04-21T23:30:20.031550+0000) 2024-04-21T23:30:46.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:46 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:46.231+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:06.333794+0000 front 2024-04-21T23:30:06.333742+0000 (oldest deadline 2024-04-21T23:30:29.233758+0000) 2024-04-21T23:30:46.682 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:46 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9[60667]: 2024-04-21T23:30:46.231+0000 7fef1a419700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-04-21T23:30:13.335027+0000 front 2024-04-21T23:30:13.334985+0000 (oldest deadline 2024-04-21T23:30:35.034821+0000) 2024-04-21T23:30:46.944 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:46 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:46.512+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.197048+0000 front 2024-04-21T23:29:33.197157+0000 (oldest deadline 2024-04-21T23:29:59.097025+0000) 2024-04-21T23:30:46.944 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:46 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:46.512+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:47.298757+0000 front 2024-04-21T23:29:47.298709+0000 (oldest deadline 2024-04-21T23:30:13.198900+0000) 2024-04-21T23:30:46.944 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:46 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:46.512+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:57.900160+0000 front 2024-04-21T23:29:57.900044+0000 (oldest deadline 2024-04-21T23:30:20.199812+0000) 2024-04-21T23:30:46.944 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:46 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:46.512+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:03.700683+0000 front 2024-04-21T23:30:03.700711+0000 (oldest deadline 2024-04-21T23:30:27.800810+0000) 2024-04-21T23:30:46.945 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:46 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:46.512+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-04-21T23:30:12.802926+0000 front 2024-04-21T23:30:12.802986+0000 (oldest deadline 2024-04-21T23:30:35.102863+0000) 2024-04-21T23:30:47.193 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:46 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:46.879+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:31.573404+0000 front 2024-04-21T23:29:31.573351+0000 (oldest deadline 2024-04-21T23:29:56.873350+0000) 2024-04-21T23:30:47.193 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:46 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:46.879+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.374892+0000 front 2024-04-21T23:29:40.374834+0000 (oldest deadline 2024-04-21T23:30:04.474654+0000) 2024-04-21T23:30:47.193 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:46 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:46.879+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:50.176921+0000 front 2024-04-21T23:29:50.176955+0000 (oldest deadline 2024-04-21T23:30:16.577589+0000) 2024-04-21T23:30:47.193 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:46 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:46.879+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:02.478617+0000 front 2024-04-21T23:30:02.478436+0000 (oldest deadline 2024-04-21T23:30:27.178522+0000) 2024-04-21T23:30:47.193 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:46 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:46.879+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-04-21T23:30:10.980975+0000 front 2024-04-21T23:30:10.980934+0000 (oldest deadline 2024-04-21T23:30:36.280923+0000) 2024-04-21T23:30:47.543 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:47 smithi040 podman[95537]: 2024-04-21 23:30:47.253977836 +0000 UTC m=+5.660088415 container died 556ebc8fe58e05b8b71b2e479f6eb69f30ca8212bc067309deae2291ab25e01f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9, RELEASE=squid-820a2f8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-21T23:30:47.893 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:47 smithi040 podman[95537]: 2024-04-21 23:30:47.652735563 +0000 UTC m=+6.058846142 container cleanup 556ebc8fe58e05b8b71b2e479f6eb69f30ca8212bc067309deae2291ab25e01f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-820a2f8) 2024-04-21T23:30:47.893 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:47 smithi040 bash[95537]: ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9 2024-04-21T23:30:47.943 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:47 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:47.527+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.197048+0000 front 2024-04-21T23:29:33.197157+0000 (oldest deadline 2024-04-21T23:29:59.097025+0000) 2024-04-21T23:30:47.943 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:47 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:47.527+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:47.298757+0000 front 2024-04-21T23:29:47.298709+0000 (oldest deadline 2024-04-21T23:30:13.198900+0000) 2024-04-21T23:30:47.944 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:47 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:47.527+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:57.900160+0000 front 2024-04-21T23:29:57.900044+0000 (oldest deadline 2024-04-21T23:30:20.199812+0000) 2024-04-21T23:30:47.944 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:47 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:47.527+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:03.700683+0000 front 2024-04-21T23:30:03.700711+0000 (oldest deadline 2024-04-21T23:30:27.800810+0000) 2024-04-21T23:30:47.944 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:47 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:47.527+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-04-21T23:30:12.802926+0000 front 2024-04-21T23:30:12.802986+0000 (oldest deadline 2024-04-21T23:30:35.102863+0000) 2024-04-21T23:30:47.944 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:47 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:47.527+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6822 osd.6 since back 2024-04-21T23:30:21.004026+0000 front 2024-04-21T23:30:21.003952+0000 (oldest deadline 2024-04-21T23:30:46.903860+0000) 2024-04-21T23:30:48.181 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:47 smithi040 podman[95555]: 2024-04-21 23:30:47.895656016 +0000 UTC m=+0.638859671 container remove 556ebc8fe58e05b8b71b2e479f6eb69f30ca8212bc067309deae2291ab25e01f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, RELEASE=squid-820a2f8, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-21T23:30:48.193 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:47 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:47.879+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:31.573404+0000 front 2024-04-21T23:29:31.573351+0000 (oldest deadline 2024-04-21T23:29:56.873350+0000) 2024-04-21T23:30:48.193 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:47 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:47.879+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.374892+0000 front 2024-04-21T23:29:40.374834+0000 (oldest deadline 2024-04-21T23:30:04.474654+0000) 2024-04-21T23:30:48.193 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:47 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:47.879+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:50.176921+0000 front 2024-04-21T23:29:50.176955+0000 (oldest deadline 2024-04-21T23:30:16.577589+0000) 2024-04-21T23:30:48.193 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:47 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:47.879+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:02.478617+0000 front 2024-04-21T23:30:02.478436+0000 (oldest deadline 2024-04-21T23:30:27.178522+0000) 2024-04-21T23:30:48.194 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:47 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:47.879+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-04-21T23:30:10.980975+0000 front 2024-04-21T23:30:10.980934+0000 (oldest deadline 2024-04-21T23:30:36.280923+0000) 2024-04-21T23:30:48.194 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:47 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:47.879+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6822 osd.6 since back 2024-04-21T23:30:21.582122+0000 front 2024-04-21T23:30:21.582137+0000 (oldest deadline 2024-04-21T23:30:46.881984+0000) 2024-04-21T23:30:48.541 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:48 smithi040 podman[95646]: 2024-04-21 23:30:48.131699945 +0000 UTC m=+0.018570819 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-21T23:30:48.541 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:48 smithi040 podman[95646]: 2024-04-21 23:30:48.246549603 +0000 UTC m=+0.133420499 container create 97a9a95a1df0d5239dbaf9c291ef155cf8007e3e14273b765fd0fbad55161e3b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=squid-820a2f8, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5) 2024-04-21T23:30:48.541 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:48 smithi040 podman[95646]: 2024-04-21 23:30:48.457074728 +0000 UTC m=+0.343945589 container init 97a9a95a1df0d5239dbaf9c291ef155cf8007e3e14273b765fd0fbad55161e3b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9-deactivate, RELEASE=squid-820a2f8, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-21T23:30:48.541 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:48 smithi040 podman[95646]: 2024-04-21 23:30:48.460517816 +0000 UTC m=+0.347388669 container start 97a9a95a1df0d5239dbaf9c291ef155cf8007e3e14273b765fd0fbad55161e3b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=squid-820a2f8, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T23:30:48.542 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:48 smithi040 podman[95646]: 2024-04-21 23:30:48.543246543 +0000 UTC m=+0.430117425 container attach 97a9a95a1df0d5239dbaf9c291ef155cf8007e3e14273b765fd0fbad55161e3b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=squid-820a2f8, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T23:30:48.931 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:48 smithi040 podman[95646]: 2024-04-21 23:30:48.76051068 +0000 UTC m=+0.647381539 container died 97a9a95a1df0d5239dbaf9c291ef155cf8007e3e14273b765fd0fbad55161e3b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-820a2f8, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-21T23:30:48.943 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:48 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:48.520+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.197048+0000 front 2024-04-21T23:29:33.197157+0000 (oldest deadline 2024-04-21T23:29:59.097025+0000) 2024-04-21T23:30:48.943 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:48 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:48.520+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:47.298757+0000 front 2024-04-21T23:29:47.298709+0000 (oldest deadline 2024-04-21T23:30:13.198900+0000) 2024-04-21T23:30:48.943 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:48 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:48.520+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:57.900160+0000 front 2024-04-21T23:29:57.900044+0000 (oldest deadline 2024-04-21T23:30:20.199812+0000) 2024-04-21T23:30:48.943 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:48 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:48.520+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:03.700683+0000 front 2024-04-21T23:30:03.700711+0000 (oldest deadline 2024-04-21T23:30:27.800810+0000) 2024-04-21T23:30:48.943 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:48 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:48.520+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-04-21T23:30:12.802926+0000 front 2024-04-21T23:30:12.802986+0000 (oldest deadline 2024-04-21T23:30:35.102863+0000) 2024-04-21T23:30:48.944 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:48 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:48.520+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6822 osd.6 since back 2024-04-21T23:30:21.004026+0000 front 2024-04-21T23:30:21.003952+0000 (oldest deadline 2024-04-21T23:30:46.903860+0000) 2024-04-21T23:30:49.193 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:48 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:48.830+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:31.573404+0000 front 2024-04-21T23:29:31.573351+0000 (oldest deadline 2024-04-21T23:29:56.873350+0000) 2024-04-21T23:30:49.193 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:48 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:48.830+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.374892+0000 front 2024-04-21T23:29:40.374834+0000 (oldest deadline 2024-04-21T23:30:04.474654+0000) 2024-04-21T23:30:49.193 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:48 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:48.830+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:50.176921+0000 front 2024-04-21T23:29:50.176955+0000 (oldest deadline 2024-04-21T23:30:16.577589+0000) 2024-04-21T23:30:49.193 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:48 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:48.830+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:02.478617+0000 front 2024-04-21T23:30:02.478436+0000 (oldest deadline 2024-04-21T23:30:27.178522+0000) 2024-04-21T23:30:49.193 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:48 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:48.830+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-04-21T23:30:10.980975+0000 front 2024-04-21T23:30:10.980934+0000 (oldest deadline 2024-04-21T23:30:36.280923+0000) 2024-04-21T23:30:49.194 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:48 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:48.830+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6822 osd.6 since back 2024-04-21T23:30:21.582122+0000 front 2024-04-21T23:30:21.582137+0000 (oldest deadline 2024-04-21T23:30:46.881984+0000) 2024-04-21T23:30:49.343 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 21 23:30:49 smithi040 podman[95672]: 2024-04-21 23:30:49.326912427 +0000 UTC m=+0.560521215 container remove 97a9a95a1df0d5239dbaf9c291ef155cf8007e3e14273b765fd0fbad55161e3b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-9-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-820a2f8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True) 2024-04-21T23:30:49.348 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d814e4dc-0033-11ef-bc93-c7b262605968@osd.9.service' 2024-04-21T23:30:49.389 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T23:30:49.389 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-04-21T23:30:49.389 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-04-21T23:30:49.389 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-d814e4dc-0033-11ef-bc93-c7b262605968@osd.10 2024-04-21T23:30:49.689 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:49 smithi111 systemd[1]: Stopping Ceph osd.10 for d814e4dc-0033-11ef-bc93-c7b262605968... 2024-04-21T23:30:49.690 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:49 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:49.500+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.197048+0000 front 2024-04-21T23:29:33.197157+0000 (oldest deadline 2024-04-21T23:29:59.097025+0000) 2024-04-21T23:30:49.690 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:49 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:49.501+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:47.298757+0000 front 2024-04-21T23:29:47.298709+0000 (oldest deadline 2024-04-21T23:30:13.198900+0000) 2024-04-21T23:30:49.690 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:49 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:49.501+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:57.900160+0000 front 2024-04-21T23:29:57.900044+0000 (oldest deadline 2024-04-21T23:30:20.199812+0000) 2024-04-21T23:30:49.690 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:49 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:49.501+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:03.700683+0000 front 2024-04-21T23:30:03.700711+0000 (oldest deadline 2024-04-21T23:30:27.800810+0000) 2024-04-21T23:30:49.690 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:49 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:49.501+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-04-21T23:30:12.802926+0000 front 2024-04-21T23:30:12.802986+0000 (oldest deadline 2024-04-21T23:30:35.102863+0000) 2024-04-21T23:30:49.691 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:49 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:49.501+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6822 osd.6 since back 2024-04-21T23:30:21.004026+0000 front 2024-04-21T23:30:21.003952+0000 (oldest deadline 2024-04-21T23:30:46.903860+0000) 2024-04-21T23:30:49.691 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:49 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:49.691+0000 7f982024c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T23:30:49.691 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:49 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:49.691+0000 7f982024c700 -1 osd.10 108 *** Got signal Terminated *** 2024-04-21T23:30:49.691 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:49 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:49.691+0000 7f982024c700 -1 osd.10 108 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-21T23:30:50.193 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:49 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:49.784+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:31.573404+0000 front 2024-04-21T23:29:31.573351+0000 (oldest deadline 2024-04-21T23:29:56.873350+0000) 2024-04-21T23:30:50.193 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:49 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:49.784+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.374892+0000 front 2024-04-21T23:29:40.374834+0000 (oldest deadline 2024-04-21T23:30:04.474654+0000) 2024-04-21T23:30:50.194 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:49 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:49.784+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:50.176921+0000 front 2024-04-21T23:29:50.176955+0000 (oldest deadline 2024-04-21T23:30:16.577589+0000) 2024-04-21T23:30:50.194 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:49 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:49.784+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:02.478617+0000 front 2024-04-21T23:30:02.478436+0000 (oldest deadline 2024-04-21T23:30:27.178522+0000) 2024-04-21T23:30:50.194 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:49 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:49.784+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-04-21T23:30:10.980975+0000 front 2024-04-21T23:30:10.980934+0000 (oldest deadline 2024-04-21T23:30:36.280923+0000) 2024-04-21T23:30:50.194 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:49 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:49.784+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6822 osd.6 since back 2024-04-21T23:30:21.582122+0000 front 2024-04-21T23:30:21.582137+0000 (oldest deadline 2024-04-21T23:30:46.881984+0000) 2024-04-21T23:30:50.947 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:50 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:50.471+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.197048+0000 front 2024-04-21T23:29:33.197157+0000 (oldest deadline 2024-04-21T23:29:59.097025+0000) 2024-04-21T23:30:50.947 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:50 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:50.471+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:47.298757+0000 front 2024-04-21T23:29:47.298709+0000 (oldest deadline 2024-04-21T23:30:13.198900+0000) 2024-04-21T23:30:50.947 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:50 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:50.471+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:57.900160+0000 front 2024-04-21T23:29:57.900044+0000 (oldest deadline 2024-04-21T23:30:20.199812+0000) 2024-04-21T23:30:50.948 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:50 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:50.471+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:03.700683+0000 front 2024-04-21T23:30:03.700711+0000 (oldest deadline 2024-04-21T23:30:27.800810+0000) 2024-04-21T23:30:50.948 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:50 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:50.471+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-04-21T23:30:12.802926+0000 front 2024-04-21T23:30:12.802986+0000 (oldest deadline 2024-04-21T23:30:35.102863+0000) 2024-04-21T23:30:50.948 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:50 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:50.471+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6822 osd.6 since back 2024-04-21T23:30:21.004026+0000 front 2024-04-21T23:30:21.003952+0000 (oldest deadline 2024-04-21T23:30:46.903860+0000) 2024-04-21T23:30:51.193 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:50 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:50.758+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:31.573404+0000 front 2024-04-21T23:29:31.573351+0000 (oldest deadline 2024-04-21T23:29:56.873350+0000) 2024-04-21T23:30:51.193 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:50 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:50.758+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.374892+0000 front 2024-04-21T23:29:40.374834+0000 (oldest deadline 2024-04-21T23:30:04.474654+0000) 2024-04-21T23:30:51.193 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:50 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:50.758+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:50.176921+0000 front 2024-04-21T23:29:50.176955+0000 (oldest deadline 2024-04-21T23:30:16.577589+0000) 2024-04-21T23:30:51.193 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:50 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:50.758+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:02.478617+0000 front 2024-04-21T23:30:02.478436+0000 (oldest deadline 2024-04-21T23:30:27.178522+0000) 2024-04-21T23:30:51.193 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:50 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:50.758+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-04-21T23:30:10.980975+0000 front 2024-04-21T23:30:10.980934+0000 (oldest deadline 2024-04-21T23:30:36.280923+0000) 2024-04-21T23:30:51.194 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:50 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:50.758+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6822 osd.6 since back 2024-04-21T23:30:21.582122+0000 front 2024-04-21T23:30:21.582137+0000 (oldest deadline 2024-04-21T23:30:46.881984+0000) 2024-04-21T23:30:51.943 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:51 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:51.483+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.197048+0000 front 2024-04-21T23:29:33.197157+0000 (oldest deadline 2024-04-21T23:29:59.097025+0000) 2024-04-21T23:30:51.943 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:51 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:51.483+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:47.298757+0000 front 2024-04-21T23:29:47.298709+0000 (oldest deadline 2024-04-21T23:30:13.198900+0000) 2024-04-21T23:30:51.944 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:51 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:51.483+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:57.900160+0000 front 2024-04-21T23:29:57.900044+0000 (oldest deadline 2024-04-21T23:30:20.199812+0000) 2024-04-21T23:30:51.944 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:51 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:51.483+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:03.700683+0000 front 2024-04-21T23:30:03.700711+0000 (oldest deadline 2024-04-21T23:30:27.800810+0000) 2024-04-21T23:30:51.944 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:51 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:51.483+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-04-21T23:30:12.802926+0000 front 2024-04-21T23:30:12.802986+0000 (oldest deadline 2024-04-21T23:30:35.102863+0000) 2024-04-21T23:30:51.944 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:51 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:51.483+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6822 osd.6 since back 2024-04-21T23:30:21.004026+0000 front 2024-04-21T23:30:21.003952+0000 (oldest deadline 2024-04-21T23:30:46.903860+0000) 2024-04-21T23:30:51.995 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:51 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:51.741+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:31.573404+0000 front 2024-04-21T23:29:31.573351+0000 (oldest deadline 2024-04-21T23:29:56.873350+0000) 2024-04-21T23:30:51.995 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:51 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:51.741+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.374892+0000 front 2024-04-21T23:29:40.374834+0000 (oldest deadline 2024-04-21T23:30:04.474654+0000) 2024-04-21T23:30:51.995 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:51 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:51.741+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:50.176921+0000 front 2024-04-21T23:29:50.176955+0000 (oldest deadline 2024-04-21T23:30:16.577589+0000) 2024-04-21T23:30:51.995 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:51 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:51.741+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:02.478617+0000 front 2024-04-21T23:30:02.478436+0000 (oldest deadline 2024-04-21T23:30:27.178522+0000) 2024-04-21T23:30:51.995 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:51 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:51.741+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-04-21T23:30:10.980975+0000 front 2024-04-21T23:30:10.980934+0000 (oldest deadline 2024-04-21T23:30:36.280923+0000) 2024-04-21T23:30:51.995 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:51 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:51.741+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6822 osd.6 since back 2024-04-21T23:30:21.582122+0000 front 2024-04-21T23:30:21.582137+0000 (oldest deadline 2024-04-21T23:30:46.881984+0000) 2024-04-21T23:30:52.943 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:52 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:52.731+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:31.573404+0000 front 2024-04-21T23:29:31.573351+0000 (oldest deadline 2024-04-21T23:29:56.873350+0000) 2024-04-21T23:30:52.943 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:52 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:52.731+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.374892+0000 front 2024-04-21T23:29:40.374834+0000 (oldest deadline 2024-04-21T23:30:04.474654+0000) 2024-04-21T23:30:52.943 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:52 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:52.731+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:50.176921+0000 front 2024-04-21T23:29:50.176955+0000 (oldest deadline 2024-04-21T23:30:16.577589+0000) 2024-04-21T23:30:52.943 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:52 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:52.731+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:02.478617+0000 front 2024-04-21T23:30:02.478436+0000 (oldest deadline 2024-04-21T23:30:27.178522+0000) 2024-04-21T23:30:52.944 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:52 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:52.731+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-04-21T23:30:10.980975+0000 front 2024-04-21T23:30:10.980934+0000 (oldest deadline 2024-04-21T23:30:36.280923+0000) 2024-04-21T23:30:52.944 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:52 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:52.731+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6822 osd.6 since back 2024-04-21T23:30:21.582122+0000 front 2024-04-21T23:30:21.582137+0000 (oldest deadline 2024-04-21T23:30:46.881984+0000) 2024-04-21T23:30:52.944 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:52 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:52.731+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6820 osd.7 since back 2024-04-21T23:30:28.583132+0000 front 2024-04-21T23:30:28.583055+0000 (oldest deadline 2024-04-21T23:30:52.082915+0000) 2024-04-21T23:30:52.944 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:52 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:52.457+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.197048+0000 front 2024-04-21T23:29:33.197157+0000 (oldest deadline 2024-04-21T23:29:59.097025+0000) 2024-04-21T23:30:52.944 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:52 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:52.457+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:47.298757+0000 front 2024-04-21T23:29:47.298709+0000 (oldest deadline 2024-04-21T23:30:13.198900+0000) 2024-04-21T23:30:52.944 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:52 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:52.457+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:57.900160+0000 front 2024-04-21T23:29:57.900044+0000 (oldest deadline 2024-04-21T23:30:20.199812+0000) 2024-04-21T23:30:52.945 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:52 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:52.457+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:03.700683+0000 front 2024-04-21T23:30:03.700711+0000 (oldest deadline 2024-04-21T23:30:27.800810+0000) 2024-04-21T23:30:52.945 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:52 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:52.457+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-04-21T23:30:12.802926+0000 front 2024-04-21T23:30:12.802986+0000 (oldest deadline 2024-04-21T23:30:35.102863+0000) 2024-04-21T23:30:52.945 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:52 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:52.457+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6822 osd.6 since back 2024-04-21T23:30:21.004026+0000 front 2024-04-21T23:30:21.003952+0000 (oldest deadline 2024-04-21T23:30:46.903860+0000) 2024-04-21T23:30:53.943 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:53 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:53.466+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.197048+0000 front 2024-04-21T23:29:33.197157+0000 (oldest deadline 2024-04-21T23:29:59.097025+0000) 2024-04-21T23:30:53.944 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:53 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:53.466+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:47.298757+0000 front 2024-04-21T23:29:47.298709+0000 (oldest deadline 2024-04-21T23:30:13.198900+0000) 2024-04-21T23:30:53.944 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:53 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:53.466+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:57.900160+0000 front 2024-04-21T23:29:57.900044+0000 (oldest deadline 2024-04-21T23:30:20.199812+0000) 2024-04-21T23:30:53.944 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:53 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:53.466+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:03.700683+0000 front 2024-04-21T23:30:03.700711+0000 (oldest deadline 2024-04-21T23:30:27.800810+0000) 2024-04-21T23:30:53.944 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:53 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:53.466+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-04-21T23:30:12.802926+0000 front 2024-04-21T23:30:12.802986+0000 (oldest deadline 2024-04-21T23:30:35.102863+0000) 2024-04-21T23:30:53.944 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:53 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:53.466+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6822 osd.6 since back 2024-04-21T23:30:21.004026+0000 front 2024-04-21T23:30:21.003952+0000 (oldest deadline 2024-04-21T23:30:46.903860+0000) 2024-04-21T23:30:53.944 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:53 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:53.466+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.111:6820 osd.7 since back 2024-04-21T23:30:30.404985+0000 front 2024-04-21T23:30:30.404945+0000 (oldest deadline 2024-04-21T23:30:53.205546+0000) 2024-04-21T23:30:54.193 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:53 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:53.695+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:31.573404+0000 front 2024-04-21T23:29:31.573351+0000 (oldest deadline 2024-04-21T23:29:56.873350+0000) 2024-04-21T23:30:54.193 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:53 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:53.695+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.374892+0000 front 2024-04-21T23:29:40.374834+0000 (oldest deadline 2024-04-21T23:30:04.474654+0000) 2024-04-21T23:30:54.193 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:53 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:53.695+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:50.176921+0000 front 2024-04-21T23:29:50.176955+0000 (oldest deadline 2024-04-21T23:30:16.577589+0000) 2024-04-21T23:30:54.194 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:53 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:53.695+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:02.478617+0000 front 2024-04-21T23:30:02.478436+0000 (oldest deadline 2024-04-21T23:30:27.178522+0000) 2024-04-21T23:30:54.194 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:53 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:53.695+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-04-21T23:30:10.980975+0000 front 2024-04-21T23:30:10.980934+0000 (oldest deadline 2024-04-21T23:30:36.280923+0000) 2024-04-21T23:30:54.194 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:53 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:53.695+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6822 osd.6 since back 2024-04-21T23:30:21.582122+0000 front 2024-04-21T23:30:21.582137+0000 (oldest deadline 2024-04-21T23:30:46.881984+0000) 2024-04-21T23:30:54.194 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:53 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:53.695+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6820 osd.7 since back 2024-04-21T23:30:28.583132+0000 front 2024-04-21T23:30:28.583055+0000 (oldest deadline 2024-04-21T23:30:52.082915+0000) 2024-04-21T23:30:54.693 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:54 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:54.441+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:33.197048+0000 front 2024-04-21T23:29:33.197157+0000 (oldest deadline 2024-04-21T23:29:59.097025+0000) 2024-04-21T23:30:54.693 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:54 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:54.441+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:47.298757+0000 front 2024-04-21T23:29:47.298709+0000 (oldest deadline 2024-04-21T23:30:13.198900+0000) 2024-04-21T23:30:54.694 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:54 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:54.441+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-21T23:29:57.900160+0000 front 2024-04-21T23:29:57.900044+0000 (oldest deadline 2024-04-21T23:30:20.199812+0000) 2024-04-21T23:30:54.694 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:54 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:54.441+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:03.700683+0000 front 2024-04-21T23:30:03.700711+0000 (oldest deadline 2024-04-21T23:30:27.800810+0000) 2024-04-21T23:30:54.694 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:54 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:54.441+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-04-21T23:30:12.802926+0000 front 2024-04-21T23:30:12.802986+0000 (oldest deadline 2024-04-21T23:30:35.102863+0000) 2024-04-21T23:30:54.694 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:54 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:54.441+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.40:6822 osd.6 since back 2024-04-21T23:30:21.004026+0000 front 2024-04-21T23:30:21.003952+0000 (oldest deadline 2024-04-21T23:30:46.903860+0000) 2024-04-21T23:30:54.694 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:54 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10[56011]: 2024-04-21T23:30:54.441+0000 7f9818838700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.111:6820 osd.7 since back 2024-04-21T23:30:30.404985+0000 front 2024-04-21T23:30:30.404945+0000 (oldest deadline 2024-04-21T23:30:53.205546+0000) 2024-04-21T23:30:55.193 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:54 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:54.741+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:31.573404+0000 front 2024-04-21T23:29:31.573351+0000 (oldest deadline 2024-04-21T23:29:56.873350+0000) 2024-04-21T23:30:55.193 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:54 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:54.741+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.374892+0000 front 2024-04-21T23:29:40.374834+0000 (oldest deadline 2024-04-21T23:30:04.474654+0000) 2024-04-21T23:30:55.193 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:54 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:54.741+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:50.176921+0000 front 2024-04-21T23:29:50.176955+0000 (oldest deadline 2024-04-21T23:30:16.577589+0000) 2024-04-21T23:30:55.193 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:54 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:54.741+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:02.478617+0000 front 2024-04-21T23:30:02.478436+0000 (oldest deadline 2024-04-21T23:30:27.178522+0000) 2024-04-21T23:30:55.193 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:54 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:54.741+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-04-21T23:30:10.980975+0000 front 2024-04-21T23:30:10.980934+0000 (oldest deadline 2024-04-21T23:30:36.280923+0000) 2024-04-21T23:30:55.194 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:54 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:54.741+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6822 osd.6 since back 2024-04-21T23:30:21.582122+0000 front 2024-04-21T23:30:21.582137+0000 (oldest deadline 2024-04-21T23:30:46.881984+0000) 2024-04-21T23:30:55.194 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:54 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:54.741+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6820 osd.7 since back 2024-04-21T23:30:28.583132+0000 front 2024-04-21T23:30:28.583055+0000 (oldest deadline 2024-04-21T23:30:52.082915+0000) 2024-04-21T23:30:55.389 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:55 smithi111 podman[67726]: 2024-04-21 23:30:55.190944184 +0000 UTC m=+5.604705105 container died 177f8972b44fcf7a7e27f000a147000eae1a91edc24899e1a6227cd5e86d2fbc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, RELEASE=squid-820a2f8, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T23:30:55.693 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:55 smithi111 podman[67726]: 2024-04-21 23:30:55.516147877 +0000 UTC m=+5.929908805 container cleanup 177f8972b44fcf7a7e27f000a147000eae1a91edc24899e1a6227cd5e86d2fbc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-820a2f8, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T23:30:55.693 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:55 smithi111 bash[67726]: ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10 2024-04-21T23:30:56.123 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:55 smithi111 podman[67743]: 2024-04-21 23:30:55.717584709 +0000 UTC m=+0.526259892 container remove 177f8972b44fcf7a7e27f000a147000eae1a91edc24899e1a6227cd5e86d2fbc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-820a2f8, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-21T23:30:56.193 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:55 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:55.716+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:31.573404+0000 front 2024-04-21T23:29:31.573351+0000 (oldest deadline 2024-04-21T23:29:56.873350+0000) 2024-04-21T23:30:56.193 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:55 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:55.716+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.374892+0000 front 2024-04-21T23:29:40.374834+0000 (oldest deadline 2024-04-21T23:30:04.474654+0000) 2024-04-21T23:30:56.193 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:55 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:55.716+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:50.176921+0000 front 2024-04-21T23:29:50.176955+0000 (oldest deadline 2024-04-21T23:30:16.577589+0000) 2024-04-21T23:30:56.193 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:55 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:55.716+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:02.478617+0000 front 2024-04-21T23:30:02.478436+0000 (oldest deadline 2024-04-21T23:30:27.178522+0000) 2024-04-21T23:30:56.194 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:55 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:55.716+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-04-21T23:30:10.980975+0000 front 2024-04-21T23:30:10.980934+0000 (oldest deadline 2024-04-21T23:30:36.280923+0000) 2024-04-21T23:30:56.194 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:55 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:55.716+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6822 osd.6 since back 2024-04-21T23:30:21.582122+0000 front 2024-04-21T23:30:21.582137+0000 (oldest deadline 2024-04-21T23:30:46.881984+0000) 2024-04-21T23:30:56.194 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:55 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:55.716+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6820 osd.7 since back 2024-04-21T23:30:28.583132+0000 front 2024-04-21T23:30:28.583055+0000 (oldest deadline 2024-04-21T23:30:52.082915+0000) 2024-04-21T23:30:56.443 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:56 smithi111 podman[67836]: 2024-04-21 23:30:56.028814943 +0000 UTC m=+0.017893188 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-21T23:30:56.443 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:56 smithi111 podman[67836]: 2024-04-21 23:30:56.141589658 +0000 UTC m=+0.130667930 container create 7c08242a0000580743d7efd95550c506dc1d96b5031d874fc79ef737c5ad44e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-820a2f8, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-21T23:30:56.444 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:56 smithi111 podman[67836]: 2024-04-21 23:30:56.370673942 +0000 UTC m=+0.359752159 container init 7c08242a0000580743d7efd95550c506dc1d96b5031d874fc79ef737c5ad44e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=squid-820a2f8, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416) 2024-04-21T23:30:56.444 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:56 smithi111 podman[67836]: 2024-04-21 23:30:56.37369429 +0000 UTC m=+0.362772517 container start 7c08242a0000580743d7efd95550c506dc1d96b5031d874fc79ef737c5ad44e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240416, RELEASE=squid-820a2f8, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T23:30:56.938 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:56 smithi111 podman[67836]: 2024-04-21 23:30:56.453972284 +0000 UTC m=+0.443050527 container attach 7c08242a0000580743d7efd95550c506dc1d96b5031d874fc79ef737c5ad44e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, RELEASE=squid-820a2f8, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-21T23:30:56.938 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:56 smithi111 podman[67836]: 2024-04-21 23:30:56.661658084 +0000 UTC m=+0.650736314 container died 7c08242a0000580743d7efd95550c506dc1d96b5031d874fc79ef737c5ad44e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, RELEASE=squid-820a2f8, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-21T23:30:57.170 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d814e4dc-0033-11ef-bc93-c7b262605968@osd.10.service' 2024-04-21T23:30:57.192 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:57 smithi111 podman[67862]: 2024-04-21 23:30:57.154647629 +0000 UTC m=+0.487759685 container remove 7c08242a0000580743d7efd95550c506dc1d96b5031d874fc79ef737c5ad44e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-820a2f8, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T23:30:57.192 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:57 smithi111 systemd[1]: ceph-d814e4dc-0033-11ef-bc93-c7b262605968@osd.10.service: Deactivated successfully. 2024-04-21T23:30:57.192 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:57 smithi111 systemd[1]: Stopped Ceph osd.10 for d814e4dc-0033-11ef-bc93-c7b262605968. 2024-04-21T23:30:57.192 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:30:57 smithi111 systemd[1]: ceph-d814e4dc-0033-11ef-bc93-c7b262605968@osd.10.service: Consumed 17.606s CPU time. 2024-04-21T23:30:57.193 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:56 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:56.698+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:31.573404+0000 front 2024-04-21T23:29:31.573351+0000 (oldest deadline 2024-04-21T23:29:56.873350+0000) 2024-04-21T23:30:57.193 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:56 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:56.698+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.374892+0000 front 2024-04-21T23:29:40.374834+0000 (oldest deadline 2024-04-21T23:30:04.474654+0000) 2024-04-21T23:30:57.193 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:56 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:56.698+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:50.176921+0000 front 2024-04-21T23:29:50.176955+0000 (oldest deadline 2024-04-21T23:30:16.577589+0000) 2024-04-21T23:30:57.193 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:56 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:56.698+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:02.478617+0000 front 2024-04-21T23:30:02.478436+0000 (oldest deadline 2024-04-21T23:30:27.178522+0000) 2024-04-21T23:30:57.193 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:56 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:56.698+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-04-21T23:30:10.980975+0000 front 2024-04-21T23:30:10.980934+0000 (oldest deadline 2024-04-21T23:30:36.280923+0000) 2024-04-21T23:30:57.193 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:56 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:56.698+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6822 osd.6 since back 2024-04-21T23:30:21.582122+0000 front 2024-04-21T23:30:21.582137+0000 (oldest deadline 2024-04-21T23:30:46.881984+0000) 2024-04-21T23:30:57.193 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:56 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:56.698+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6820 osd.7 since back 2024-04-21T23:30:28.583132+0000 front 2024-04-21T23:30:28.583055+0000 (oldest deadline 2024-04-21T23:30:52.082915+0000) 2024-04-21T23:30:57.207 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T23:30:57.208 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-04-21T23:30:57.208 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-04-21T23:30:57.208 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ceph-d814e4dc-0033-11ef-bc93-c7b262605968@osd.11 2024-04-21T23:30:57.488 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:57 smithi174 systemd[1]: Stopping Ceph osd.11 for d814e4dc-0033-11ef-bc93-c7b262605968... 2024-04-21T23:30:57.943 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:57 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:57.489+0000 7f4e01693700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T23:30:57.944 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:57 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:57.489+0000 7f4e01693700 -1 osd.11 108 *** Got signal Terminated *** 2024-04-21T23:30:57.944 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:57 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:57.489+0000 7f4e01693700 -1 osd.11 108 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-21T23:30:57.944 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:57 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:57.655+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:31.573404+0000 front 2024-04-21T23:29:31.573351+0000 (oldest deadline 2024-04-21T23:29:56.873350+0000) 2024-04-21T23:30:57.945 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:57 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:57.655+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.374892+0000 front 2024-04-21T23:29:40.374834+0000 (oldest deadline 2024-04-21T23:30:04.474654+0000) 2024-04-21T23:30:57.945 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:57 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:57.655+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:50.176921+0000 front 2024-04-21T23:29:50.176955+0000 (oldest deadline 2024-04-21T23:30:16.577589+0000) 2024-04-21T23:30:57.945 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:57 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:57.655+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:02.478617+0000 front 2024-04-21T23:30:02.478436+0000 (oldest deadline 2024-04-21T23:30:27.178522+0000) 2024-04-21T23:30:57.946 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:57 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:57.655+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-04-21T23:30:10.980975+0000 front 2024-04-21T23:30:10.980934+0000 (oldest deadline 2024-04-21T23:30:36.280923+0000) 2024-04-21T23:30:57.946 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:57 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:57.655+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6822 osd.6 since back 2024-04-21T23:30:21.582122+0000 front 2024-04-21T23:30:21.582137+0000 (oldest deadline 2024-04-21T23:30:46.881984+0000) 2024-04-21T23:30:57.946 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:57 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:57.655+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6820 osd.7 since back 2024-04-21T23:30:28.583132+0000 front 2024-04-21T23:30:28.583055+0000 (oldest deadline 2024-04-21T23:30:52.082915+0000) 2024-04-21T23:30:59.193 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:58 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:58.701+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:31.573404+0000 front 2024-04-21T23:29:31.573351+0000 (oldest deadline 2024-04-21T23:29:56.873350+0000) 2024-04-21T23:30:59.193 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:58 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:58.701+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.374892+0000 front 2024-04-21T23:29:40.374834+0000 (oldest deadline 2024-04-21T23:30:04.474654+0000) 2024-04-21T23:30:59.193 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:58 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:58.701+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:50.176921+0000 front 2024-04-21T23:29:50.176955+0000 (oldest deadline 2024-04-21T23:30:16.577589+0000) 2024-04-21T23:30:59.193 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:58 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:58.701+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:02.478617+0000 front 2024-04-21T23:30:02.478436+0000 (oldest deadline 2024-04-21T23:30:27.178522+0000) 2024-04-21T23:30:59.193 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:58 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:58.701+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-04-21T23:30:10.980975+0000 front 2024-04-21T23:30:10.980934+0000 (oldest deadline 2024-04-21T23:30:36.280923+0000) 2024-04-21T23:30:59.194 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:58 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:58.701+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6822 osd.6 since back 2024-04-21T23:30:21.582122+0000 front 2024-04-21T23:30:21.582137+0000 (oldest deadline 2024-04-21T23:30:46.881984+0000) 2024-04-21T23:30:59.194 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:58 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:58.701+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6820 osd.7 since back 2024-04-21T23:30:28.583132+0000 front 2024-04-21T23:30:28.583055+0000 (oldest deadline 2024-04-21T23:30:52.082915+0000) 2024-04-21T23:30:59.943 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:59 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:59.654+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:31.573404+0000 front 2024-04-21T23:29:31.573351+0000 (oldest deadline 2024-04-21T23:29:56.873350+0000) 2024-04-21T23:30:59.943 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:59 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:59.654+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.374892+0000 front 2024-04-21T23:29:40.374834+0000 (oldest deadline 2024-04-21T23:30:04.474654+0000) 2024-04-21T23:30:59.943 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:59 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:59.654+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:50.176921+0000 front 2024-04-21T23:29:50.176955+0000 (oldest deadline 2024-04-21T23:30:16.577589+0000) 2024-04-21T23:30:59.943 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:59 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:59.654+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:02.478617+0000 front 2024-04-21T23:30:02.478436+0000 (oldest deadline 2024-04-21T23:30:27.178522+0000) 2024-04-21T23:30:59.943 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:59 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:59.654+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-04-21T23:30:10.980975+0000 front 2024-04-21T23:30:10.980934+0000 (oldest deadline 2024-04-21T23:30:36.280923+0000) 2024-04-21T23:30:59.944 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:59 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:59.654+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6822 osd.6 since back 2024-04-21T23:30:21.582122+0000 front 2024-04-21T23:30:21.582137+0000 (oldest deadline 2024-04-21T23:30:46.881984+0000) 2024-04-21T23:30:59.944 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:30:59 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:30:59.654+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6820 osd.7 since back 2024-04-21T23:30:28.583132+0000 front 2024-04-21T23:30:28.583055+0000 (oldest deadline 2024-04-21T23:30:52.082915+0000) 2024-04-21T23:31:00.943 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:31:00 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:31:00.605+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:31.573404+0000 front 2024-04-21T23:29:31.573351+0000 (oldest deadline 2024-04-21T23:29:56.873350+0000) 2024-04-21T23:31:00.943 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:31:00 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:31:00.605+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.374892+0000 front 2024-04-21T23:29:40.374834+0000 (oldest deadline 2024-04-21T23:30:04.474654+0000) 2024-04-21T23:31:00.943 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:31:00 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:31:00.605+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:50.176921+0000 front 2024-04-21T23:29:50.176955+0000 (oldest deadline 2024-04-21T23:30:16.577589+0000) 2024-04-21T23:31:00.944 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:31:00 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:31:00.605+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:02.478617+0000 front 2024-04-21T23:30:02.478436+0000 (oldest deadline 2024-04-21T23:30:27.178522+0000) 2024-04-21T23:31:00.944 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:31:00 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:31:00.605+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-04-21T23:30:10.980975+0000 front 2024-04-21T23:30:10.980934+0000 (oldest deadline 2024-04-21T23:30:36.280923+0000) 2024-04-21T23:31:00.944 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:31:00 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:31:00.605+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6822 osd.6 since back 2024-04-21T23:30:21.582122+0000 front 2024-04-21T23:30:21.582137+0000 (oldest deadline 2024-04-21T23:30:46.881984+0000) 2024-04-21T23:31:00.944 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:31:00 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:31:00.605+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6820 osd.7 since back 2024-04-21T23:30:28.583132+0000 front 2024-04-21T23:30:28.583055+0000 (oldest deadline 2024-04-21T23:30:52.082915+0000) 2024-04-21T23:31:01.943 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:31:01 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:31:01.574+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-21T23:29:31.573404+0000 front 2024-04-21T23:29:31.573351+0000 (oldest deadline 2024-04-21T23:29:56.873350+0000) 2024-04-21T23:31:01.943 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:31:01 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:31:01.574+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-21T23:29:40.374892+0000 front 2024-04-21T23:29:40.374834+0000 (oldest deadline 2024-04-21T23:30:04.474654+0000) 2024-04-21T23:31:01.943 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:31:01 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:31:01.574+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-04-21T23:29:50.176921+0000 front 2024-04-21T23:29:50.176955+0000 (oldest deadline 2024-04-21T23:30:16.577589+0000) 2024-04-21T23:31:01.943 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:31:01 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:31:01.574+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-21T23:30:02.478617+0000 front 2024-04-21T23:30:02.478436+0000 (oldest deadline 2024-04-21T23:30:27.178522+0000) 2024-04-21T23:31:01.943 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:31:01 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:31:01.574+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-04-21T23:30:10.980975+0000 front 2024-04-21T23:30:10.980934+0000 (oldest deadline 2024-04-21T23:30:36.280923+0000) 2024-04-21T23:31:01.944 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:31:01 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:31:01.574+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.40:6822 osd.6 since back 2024-04-21T23:30:21.582122+0000 front 2024-04-21T23:30:21.582137+0000 (oldest deadline 2024-04-21T23:30:46.881984+0000) 2024-04-21T23:31:01.944 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:31:01 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11[56900]: 2024-04-21T23:31:01.574+0000 7f4dfdc87700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.111:6820 osd.7 since back 2024-04-21T23:30:28.583132+0000 front 2024-04-21T23:30:28.583055+0000 (oldest deadline 2024-04-21T23:30:52.082915+0000) 2024-04-21T23:31:03.229 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:31:02 smithi174 podman[68557]: 2024-04-21 23:31:02.954615477 +0000 UTC m=+5.554300137 container died 3fda8950fec4c28fa6ac85ca0a9919c90044739bfe0bdcfff11dd8a9097bb40a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=squid-820a2f8, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0) 2024-04-21T23:31:03.587 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:31:03 smithi174 podman[68557]: 2024-04-21 23:31:03.330378618 +0000 UTC m=+5.930063283 container cleanup 3fda8950fec4c28fa6ac85ca0a9919c90044739bfe0bdcfff11dd8a9097bb40a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-820a2f8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T23:31:03.587 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:31:03 smithi174 bash[68557]: ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11 2024-04-21T23:31:03.943 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:31:03 smithi174 podman[68574]: 2024-04-21 23:31:03.589563208 +0000 UTC m=+0.637754025 container remove 3fda8950fec4c28fa6ac85ca0a9919c90044739bfe0bdcfff11dd8a9097bb40a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=squid-820a2f8, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, ceph=True) 2024-04-21T23:31:04.240 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:31:03 smithi174 podman[68663]: 2024-04-21 23:31:03.851420469 +0000 UTC m=+0.017615684 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-21T23:31:04.240 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:31:03 smithi174 podman[68663]: 2024-04-21 23:31:03.965165962 +0000 UTC m=+0.131361162 container create c66bdcf41573e99c161edc6a4218b2dd40f10fc23171845b5c7b58ff85e71e2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11-deactivate, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-820a2f8, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-04-21T23:31:04.240 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:31:04 smithi174 podman[68663]: 2024-04-21 23:31:04.175716476 +0000 UTC m=+0.341911676 container init c66bdcf41573e99c161edc6a4218b2dd40f10fc23171845b5c7b58ff85e71e2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-820a2f8, GIT_BRANCH=HEAD) 2024-04-21T23:31:04.240 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:31:04 smithi174 podman[68663]: 2024-04-21 23:31:04.178894628 +0000 UTC m=+0.345089832 container start c66bdcf41573e99c161edc6a4218b2dd40f10fc23171845b5c7b58ff85e71e2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-820a2f8, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2) 2024-04-21T23:31:04.240 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:31:04 smithi174 podman[68663]: 2024-04-21 23:31:04.242149026 +0000 UTC m=+0.408344236 container attach c66bdcf41573e99c161edc6a4218b2dd40f10fc23171845b5c7b58ff85e71e2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-820a2f8, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True) 2024-04-21T23:31:04.693 INFO:journalctl@ceph.osd.11.smithi174.stdout:Apr 21 23:31:04 smithi174 podman[68663]: 2024-04-21 23:31:04.484329736 +0000 UTC m=+0.650524935 container died c66bdcf41573e99c161edc6a4218b2dd40f10fc23171845b5c7b58ff85e71e2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-osd-11-deactivate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, RELEASE=squid-820a2f8, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T23:31:05.092 DEBUG:teuthology.orchestra.run.smithi174:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d814e4dc-0033-11ef-bc93-c7b262605968@osd.11.service' 2024-04-21T23:31:05.129 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T23:31:05.130 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-04-21T23:31:05.130 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-04-21T23:31:05.130 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-d814e4dc-0033-11ef-bc93-c7b262605968@mds.a 2024-04-21T23:31:05.428 INFO:journalctl@ceph.mds.a.smithi040.stdout:Apr 21 23:31:05 smithi040 systemd[1]: Stopping Ceph mds.a for d814e4dc-0033-11ef-bc93-c7b262605968... 2024-04-21T23:31:05.681 INFO:journalctl@ceph.mds.a.smithi040.stdout:Apr 21 23:31:05 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-a[67376]: 2024-04-21T23:31:05.429+0000 7fa79db8f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T23:31:05.681 INFO:journalctl@ceph.mds.a.smithi040.stdout:Apr 21 23:31:05 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-a[67376]: 2024-04-21T23:31:05.429+0000 7fa79db8f700 -1 mds.a *** got signal Terminated *** 2024-04-21T23:31:07.527 INFO:journalctl@ceph.mds.a.smithi040.stdout:Apr 21 23:31:07 smithi040 podman[95766]: 2024-04-21 23:31:07.243717434 +0000 UTC m=+1.908933365 container died dfb13ba672d111681c92978fa4b27449e3f65cca3556499c23dbfae6d18a5c41 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-a, ceph=True, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=squid-820a2f8, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-21T23:31:07.885 INFO:journalctl@ceph.mds.a.smithi040.stdout:Apr 21 23:31:07 smithi040 podman[95766]: 2024-04-21 23:31:07.604110659 +0000 UTC m=+2.269326596 container cleanup dfb13ba672d111681c92978fa4b27449e3f65cca3556499c23dbfae6d18a5c41 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-a, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-820a2f8, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , ceph=True) 2024-04-21T23:31:07.886 INFO:journalctl@ceph.mds.a.smithi040.stdout:Apr 21 23:31:07 smithi040 bash[95766]: ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-a 2024-04-21T23:31:08.030 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d814e4dc-0033-11ef-bc93-c7b262605968@mds.a.service' 2024-04-21T23:31:08.070 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T23:31:08.071 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-04-21T23:31:08.072 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-04-21T23:31:08.072 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-d814e4dc-0033-11ef-bc93-c7b262605968@mds.d 2024-04-21T23:31:08.431 INFO:journalctl@ceph.mds.d.smithi040.stdout:Apr 21 23:31:08 smithi040 systemd[1]: Stopping Ceph mds.d for d814e4dc-0033-11ef-bc93-c7b262605968... 2024-04-21T23:31:08.431 INFO:journalctl@ceph.mds.d.smithi040.stdout:Apr 21 23:31:08 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-d[66422]: 2024-04-21T23:31:08.399+0000 7f56b5f2e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T23:31:08.431 INFO:journalctl@ceph.mds.d.smithi040.stdout:Apr 21 23:31:08 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-d[66422]: 2024-04-21T23:31:08.399+0000 7f56b5f2e700 -1 mds.d *** got signal Terminated *** 2024-04-21T23:31:11.547 INFO:journalctl@ceph.mds.d.smithi040.stdout:Apr 21 23:31:11 smithi040 podman[95886]: 2024-04-21 23:31:11.274814687 +0000 UTC m=+2.983679250 container died 968c7523f37cff51b1f6600ce8560f57046bf95d68b56d96b26cb4bacc0d3900 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-d, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-820a2f8, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T23:31:11.906 INFO:journalctl@ceph.mds.d.smithi040.stdout:Apr 21 23:31:11 smithi040 podman[95886]: 2024-04-21 23:31:11.666215632 +0000 UTC m=+3.375080195 container cleanup 968c7523f37cff51b1f6600ce8560f57046bf95d68b56d96b26cb4bacc0d3900 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=squid-820a2f8, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240416) 2024-04-21T23:31:11.906 INFO:journalctl@ceph.mds.d.smithi040.stdout:Apr 21 23:31:11 smithi040 bash[95886]: ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-d 2024-04-21T23:31:12.040 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d814e4dc-0033-11ef-bc93-c7b262605968@mds.d.service' 2024-04-21T23:31:12.073 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T23:31:12.074 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-04-21T23:31:12.074 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-04-21T23:31:12.074 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-d814e4dc-0033-11ef-bc93-c7b262605968@mds.g 2024-04-21T23:31:12.181 INFO:journalctl@ceph.mds.g.smithi040.stdout:Apr 21 23:31:12 smithi040 systemd[1]: Stopping Ceph mds.g for d814e4dc-0033-11ef-bc93-c7b262605968... 2024-04-21T23:31:12.681 INFO:journalctl@ceph.mds.g.smithi040.stdout:Apr 21 23:31:12 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-g[67028]: 2024-04-21T23:31:12.394+0000 7f304c9d5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T23:31:12.681 INFO:journalctl@ceph.mds.g.smithi040.stdout:Apr 21 23:31:12 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-g[67028]: 2024-04-21T23:31:12.394+0000 7f304c9d5700 -1 mds.g *** got signal Terminated *** 2024-04-21T23:31:16.677 INFO:journalctl@ceph.mds.g.smithi040.stdout:Apr 21 23:31:16 smithi040 podman[96006]: 2024-04-21 23:31:16.410024968 +0000 UTC m=+4.114355403 container died 780ba2a634dc4f2403855e8edbca892dc0504172a27563315143969bb66c2357 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-g, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, ceph=True, io.buildah.version=1.33.5, RELEASE=squid-820a2f8, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-21T23:31:16.678 INFO:journalctl@ceph.mds.g.smithi040.stdout:Apr 21 23:31:16 smithi040 podman[96006]: 2024-04-21 23:31:16.680278435 +0000 UTC m=+4.384608870 container cleanup 780ba2a634dc4f2403855e8edbca892dc0504172a27563315143969bb66c2357 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-g, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=squid-820a2f8, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T23:31:16.931 INFO:journalctl@ceph.mds.g.smithi040.stdout:Apr 21 23:31:16 smithi040 bash[96006]: ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-g 2024-04-21T23:31:17.108 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d814e4dc-0033-11ef-bc93-c7b262605968@mds.g.service' 2024-04-21T23:31:17.140 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T23:31:17.141 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-04-21T23:31:17.141 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-04-21T23:31:17.141 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-d814e4dc-0033-11ef-bc93-c7b262605968@mds.j 2024-04-21T23:31:17.431 INFO:journalctl@ceph.mds.j.smithi040.stdout:Apr 21 23:31:17 smithi040 systemd[1]: Stopping Ceph mds.j for d814e4dc-0033-11ef-bc93-c7b262605968... 2024-04-21T23:31:17.931 INFO:journalctl@ceph.mds.j.smithi040.stdout:Apr 21 23:31:17 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-j[66073]: 2024-04-21T23:31:17.469+0000 7f92a6cef700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T23:31:17.931 INFO:journalctl@ceph.mds.j.smithi040.stdout:Apr 21 23:31:17 smithi040 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-j[66073]: 2024-04-21T23:31:17.469+0000 7f92a6cef700 -1 mds.j *** got signal Terminated *** 2024-04-21T23:31:20.681 INFO:journalctl@ceph.mds.j.smithi040.stdout:Apr 21 23:31:20 smithi040 podman[96126]: 2024-04-21 23:31:20.428144124 +0000 UTC m=+3.065557287 container died bf807801f4277f168efad1a744c587a9718715fb561e58f1401eb4647962aa03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-j, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-820a2f8, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-21T23:31:21.055 INFO:journalctl@ceph.mds.j.smithi040.stdout:Apr 21 23:31:20 smithi040 podman[96126]: 2024-04-21 23:31:20.798255152 +0000 UTC m=+3.435668321 container cleanup bf807801f4277f168efad1a744c587a9718715fb561e58f1401eb4647962aa03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-j, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-820a2f8, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-21T23:31:21.055 INFO:journalctl@ceph.mds.j.smithi040.stdout:Apr 21 23:31:20 smithi040 bash[96126]: ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-j 2024-04-21T23:31:21.221 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d814e4dc-0033-11ef-bc93-c7b262605968@mds.j.service' 2024-04-21T23:31:21.253 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T23:31:21.253 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-04-21T23:31:21.253 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-04-21T23:31:21.253 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-d814e4dc-0033-11ef-bc93-c7b262605968@mds.b 2024-04-21T23:31:21.564 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 21 23:31:21 smithi111 systemd[1]: Stopping Ceph mds.b for d814e4dc-0033-11ef-bc93-c7b262605968... 2024-04-21T23:31:21.943 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 21 23:31:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-b[61419]: 2024-04-21T23:31:21.564+0000 7f840bf17700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T23:31:21.943 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 21 23:31:21 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-b[61419]: 2024-04-21T23:31:21.564+0000 7f840bf17700 -1 mds.b *** got signal Terminated *** 2024-04-21T23:31:24.597 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 21 23:31:24 smithi111 podman[67942]: 2024-04-21 23:31:24.310597263 +0000 UTC m=+2.868068230 container died 7eaf129c2e3cc6c7e248344c6a3f3b8d88d071059f4e693da76a16b8a6281bf8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-b, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-820a2f8, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-21T23:31:24.943 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 21 23:31:24 smithi111 podman[67942]: 2024-04-21 23:31:24.699001053 +0000 UTC m=+3.256472018 container cleanup 7eaf129c2e3cc6c7e248344c6a3f3b8d88d071059f4e693da76a16b8a6281bf8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-b, io.buildah.version=1.33.5, RELEASE=squid-820a2f8, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T23:31:24.943 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 21 23:31:24 smithi111 bash[67942]: ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-b 2024-04-21T23:31:25.114 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d814e4dc-0033-11ef-bc93-c7b262605968@mds.b.service' 2024-04-21T23:31:25.149 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T23:31:25.149 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-04-21T23:31:25.150 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-04-21T23:31:25.150 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-d814e4dc-0033-11ef-bc93-c7b262605968@mds.e 2024-04-21T23:31:25.693 INFO:journalctl@ceph.mds.e.smithi111.stdout:Apr 21 23:31:25 smithi111 systemd[1]: Stopping Ceph mds.e for d814e4dc-0033-11ef-bc93-c7b262605968... 2024-04-21T23:31:25.693 INFO:journalctl@ceph.mds.e.smithi111.stdout:Apr 21 23:31:25 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-e[60799]: 2024-04-21T23:31:25.461+0000 7f8900d10700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T23:31:25.693 INFO:journalctl@ceph.mds.e.smithi111.stdout:Apr 21 23:31:25 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-e[60799]: 2024-04-21T23:31:25.461+0000 7f8900d10700 -1 mds.e *** got signal Terminated *** 2024-04-21T23:31:28.594 INFO:journalctl@ceph.mds.e.smithi111.stdout:Apr 21 23:31:28 smithi111 podman[68076]: 2024-04-21 23:31:28.323299474 +0000 UTC m=+2.949267783 container died e9670c79289b5d48e3b42be2cb7d3c250276be3dbc2deefbe339eb2b213a4647 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-e, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=squid-820a2f8, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-21T23:31:28.878 INFO:journalctl@ceph.mds.e.smithi111.stdout:Apr 21 23:31:28 smithi111 podman[68076]: 2024-04-21 23:31:28.66313486 +0000 UTC m=+3.289103174 container cleanup e9670c79289b5d48e3b42be2cb7d3c250276be3dbc2deefbe339eb2b213a4647 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-e, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-820a2f8, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T23:31:28.879 INFO:journalctl@ceph.mds.e.smithi111.stdout:Apr 21 23:31:28 smithi111 bash[68076]: ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-e 2024-04-21T23:31:29.131 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d814e4dc-0033-11ef-bc93-c7b262605968@mds.e.service' 2024-04-21T23:31:29.151 INFO:journalctl@ceph.mds.e.smithi111.stdout:Apr 21 23:31:28 smithi111 podman[68090]: 2024-04-21 23:31:28.881047467 +0000 UTC m=+0.558037579 container remove e9670c79289b5d48e3b42be2cb7d3c250276be3dbc2deefbe339eb2b213a4647 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-e, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, RELEASE=squid-820a2f8) 2024-04-21T23:31:29.151 INFO:journalctl@ceph.mds.e.smithi111.stdout:Apr 21 23:31:29 smithi111 systemd[1]: ceph-d814e4dc-0033-11ef-bc93-c7b262605968@mds.e.service: Deactivated successfully. 2024-04-21T23:31:29.152 INFO:journalctl@ceph.mds.e.smithi111.stdout:Apr 21 23:31:29 smithi111 systemd[1]: Stopped Ceph mds.e for d814e4dc-0033-11ef-bc93-c7b262605968. 2024-04-21T23:31:29.152 INFO:journalctl@ceph.mds.e.smithi111.stdout:Apr 21 23:31:29 smithi111 systemd[1]: ceph-d814e4dc-0033-11ef-bc93-c7b262605968@mds.e.service: Consumed 6.876s CPU time. 2024-04-21T23:31:29.163 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T23:31:29.163 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-04-21T23:31:29.163 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-04-21T23:31:29.164 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-d814e4dc-0033-11ef-bc93-c7b262605968@mds.h 2024-04-21T23:31:29.443 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:31:29 smithi111 systemd[1]: Stopping Ceph mds.h for d814e4dc-0033-11ef-bc93-c7b262605968... 2024-04-21T23:31:29.946 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:31:29 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:31:29.475+0000 7f7ed53d0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T23:31:29.946 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:31:29 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h[61109]: 2024-04-21T23:31:29.475+0000 7f7ed53d0700 -1 mds.h *** got signal Terminated *** 2024-04-21T23:31:32.598 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:31:32 smithi111 podman[68198]: 2024-04-21 23:31:32.325165673 +0000 UTC m=+2.950005239 container died c4271d3bd94d138b8e00f8cca0e2be7e9c461a91d439be0039946cb7c1d34b64 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-820a2f8) 2024-04-21T23:31:32.943 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:31:32 smithi111 podman[68198]: 2024-04-21 23:31:32.691858359 +0000 UTC m=+3.316697925 container cleanup c4271d3bd94d138b8e00f8cca0e2be7e9c461a91d439be0039946cb7c1d34b64 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-820a2f8, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T23:31:32.943 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:31:32 smithi111 bash[68198]: ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-h 2024-04-21T23:31:33.129 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d814e4dc-0033-11ef-bc93-c7b262605968@mds.h.service' 2024-04-21T23:31:33.160 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T23:31:33.160 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-04-21T23:31:33.160 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-04-21T23:31:33.160 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-d814e4dc-0033-11ef-bc93-c7b262605968@mds.k 2024-04-21T23:31:33.476 INFO:journalctl@ceph.mds.k.smithi111.stdout:Apr 21 23:31:33 smithi111 systemd[1]: Stopping Ceph mds.k for d814e4dc-0033-11ef-bc93-c7b262605968... 2024-04-21T23:31:33.824 INFO:journalctl@ceph.mds.k.smithi111.stdout:Apr 21 23:31:33 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-k[60475]: 2024-04-21T23:31:33.478+0000 7fbd0f2a5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T23:31:33.825 INFO:journalctl@ceph.mds.k.smithi111.stdout:Apr 21 23:31:33 smithi111 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-k[60475]: 2024-04-21T23:31:33.478+0000 7fbd0f2a5700 -1 mds.k *** got signal Terminated *** 2024-04-21T23:31:37.743 INFO:journalctl@ceph.mds.k.smithi111.stdout:Apr 21 23:31:37 smithi111 podman[68318]: 2024-04-21 23:31:37.484329142 +0000 UTC m=+4.108044462 container died 2fd4155c0bf06458b1c6ec9d9b05037039cd48499eaff7e109a62961e2b1f86b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-k, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, RELEASE=squid-820a2f8, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-21T23:31:38.102 INFO:journalctl@ceph.mds.k.smithi111.stdout:Apr 21 23:31:37 smithi111 podman[68318]: 2024-04-21 23:31:37.837444209 +0000 UTC m=+4.461159526 container cleanup 2fd4155c0bf06458b1c6ec9d9b05037039cd48499eaff7e109a62961e2b1f86b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-820a2f8, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-21T23:31:38.102 INFO:journalctl@ceph.mds.k.smithi111.stdout:Apr 21 23:31:37 smithi111 bash[68318]: ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-k 2024-04-21T23:31:38.261 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d814e4dc-0033-11ef-bc93-c7b262605968@mds.k.service' 2024-04-21T23:31:38.292 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T23:31:38.292 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-04-21T23:31:38.293 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-04-21T23:31:38.293 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ceph-d814e4dc-0033-11ef-bc93-c7b262605968@mds.c 2024-04-21T23:31:38.584 INFO:journalctl@ceph.mds.c.smithi174.stdout:Apr 21 23:31:38 smithi174 systemd[1]: Stopping Ceph mds.c for d814e4dc-0033-11ef-bc93-c7b262605968... 2024-04-21T23:31:38.942 INFO:journalctl@ceph.mds.c.smithi174.stdout:Apr 21 23:31:38 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-c[61803]: 2024-04-21T23:31:38.585+0000 7f7fe8204700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T23:31:38.943 INFO:journalctl@ceph.mds.c.smithi174.stdout:Apr 21 23:31:38 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-c[61803]: 2024-04-21T23:31:38.585+0000 7f7fe8204700 -1 mds.c *** got signal Terminated *** 2024-04-21T23:31:40.583 INFO:journalctl@ceph.mds.c.smithi174.stdout:Apr 21 23:31:40 smithi174 podman[68770]: 2024-04-21 23:31:40.313068825 +0000 UTC m=+1.832133589 container died 92e760e59bf0ab44bb316b8ce80313b265f5d4e50eef9e03560f68294cd3bd9b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-c, RELEASE=squid-820a2f8, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T23:31:40.583 INFO:journalctl@ceph.mds.c.smithi174.stdout:Apr 21 23:31:40 smithi174 podman[68770]: 2024-04-21 23:31:40.585644378 +0000 UTC m=+2.104709140 container cleanup 92e760e59bf0ab44bb316b8ce80313b265f5d4e50eef9e03560f68294cd3bd9b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-c, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-820a2f8, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240416, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T23:31:40.931 INFO:journalctl@ceph.mds.c.smithi174.stdout:Apr 21 23:31:40 smithi174 bash[68770]: ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-c 2024-04-21T23:31:40.931 INFO:journalctl@ceph.mds.c.smithi174.stdout:Apr 21 23:31:40 smithi174 podman[68795]: 2024-04-21 23:31:40.828387371 +0000 UTC m=+0.515553653 container remove 92e760e59bf0ab44bb316b8ce80313b265f5d4e50eef9e03560f68294cd3bd9b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-c, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-820a2f8, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-04-21T23:31:40.953 DEBUG:teuthology.orchestra.run.smithi174:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d814e4dc-0033-11ef-bc93-c7b262605968@mds.c.service' 2024-04-21T23:31:40.987 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T23:31:40.987 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-04-21T23:31:40.988 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-04-21T23:31:40.988 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ceph-d814e4dc-0033-11ef-bc93-c7b262605968@mds.f 2024-04-21T23:31:41.295 INFO:journalctl@ceph.mds.f.smithi174.stdout:Apr 21 23:31:41 smithi174 systemd[1]: Stopping Ceph mds.f for d814e4dc-0033-11ef-bc93-c7b262605968... 2024-04-21T23:31:41.692 INFO:journalctl@ceph.mds.f.smithi174.stdout:Apr 21 23:31:41 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-f[61481]: 2024-04-21T23:31:41.296+0000 7f4d67326700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T23:31:41.693 INFO:journalctl@ceph.mds.f.smithi174.stdout:Apr 21 23:31:41 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-f[61481]: 2024-04-21T23:31:41.296+0000 7f4d67326700 -1 mds.f *** got signal Terminated *** 2024-04-21T23:31:44.570 INFO:journalctl@ceph.mds.f.smithi174.stdout:Apr 21 23:31:44 smithi174 podman[68902]: 2024-04-21 23:31:44.309429586 +0000 UTC m=+3.109130549 container died b1cc187a796df3f83505468a095f75c5f78cf2a4ff907606f3b7d4d542004542 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-f, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-820a2f8) 2024-04-21T23:31:44.845 INFO:journalctl@ceph.mds.f.smithi174.stdout:Apr 21 23:31:44 smithi174 podman[68902]: 2024-04-21 23:31:44.655655507 +0000 UTC m=+3.455356472 container cleanup b1cc187a796df3f83505468a095f75c5f78cf2a4ff907606f3b7d4d542004542 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-f, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, RELEASE=squid-820a2f8, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-21T23:31:44.846 INFO:journalctl@ceph.mds.f.smithi174.stdout:Apr 21 23:31:44 smithi174 bash[68902]: ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-f 2024-04-21T23:31:45.075 DEBUG:teuthology.orchestra.run.smithi174:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d814e4dc-0033-11ef-bc93-c7b262605968@mds.f.service' 2024-04-21T23:31:45.095 INFO:journalctl@ceph.mds.f.smithi174.stdout:Apr 21 23:31:44 smithi174 podman[68916]: 2024-04-21 23:31:44.848638631 +0000 UTC m=+0.535106735 container remove b1cc187a796df3f83505468a095f75c5f78cf2a4ff907606f3b7d4d542004542 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-f, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-820a2f8, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-21T23:31:45.096 INFO:journalctl@ceph.mds.f.smithi174.stdout:Apr 21 23:31:45 smithi174 systemd[1]: ceph-d814e4dc-0033-11ef-bc93-c7b262605968@mds.f.service: Deactivated successfully. 2024-04-21T23:31:45.096 INFO:journalctl@ceph.mds.f.smithi174.stdout:Apr 21 23:31:45 smithi174 systemd[1]: Stopped Ceph mds.f for d814e4dc-0033-11ef-bc93-c7b262605968. 2024-04-21T23:31:45.096 INFO:journalctl@ceph.mds.f.smithi174.stdout:Apr 21 23:31:45 smithi174 systemd[1]: ceph-d814e4dc-0033-11ef-bc93-c7b262605968@mds.f.service: Consumed 2.181s CPU time. 2024-04-21T23:31:45.108 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T23:31:45.108 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-04-21T23:31:45.108 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-04-21T23:31:45.108 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ceph-d814e4dc-0033-11ef-bc93-c7b262605968@mds.i 2024-04-21T23:31:45.431 INFO:journalctl@ceph.mds.i.smithi174.stdout:Apr 21 23:31:45 smithi174 systemd[1]: Stopping Ceph mds.i for d814e4dc-0033-11ef-bc93-c7b262605968... 2024-04-21T23:31:45.692 INFO:journalctl@ceph.mds.i.smithi174.stdout:Apr 21 23:31:45 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-i[62111]: 2024-04-21T23:31:45.434+0000 7f1920b23700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T23:31:45.693 INFO:journalctl@ceph.mds.i.smithi174.stdout:Apr 21 23:31:45 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-i[62111]: 2024-04-21T23:31:45.434+0000 7f1920b23700 -1 mds.i *** got signal Terminated *** 2024-04-21T23:31:48.590 INFO:journalctl@ceph.mds.i.smithi174.stdout:Apr 21 23:31:48 smithi174 podman[69025]: 2024-04-21 23:31:48.312347042 +0000 UTC m=+2.985181045 container died 27bf130082a01f74839f7b092e89104f9ecd21aa215be6f0c5c2d0b38eccd22f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-i, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-820a2f8, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240416) 2024-04-21T23:31:48.940 INFO:journalctl@ceph.mds.i.smithi174.stdout:Apr 21 23:31:48 smithi174 podman[69025]: 2024-04-21 23:31:48.667114289 +0000 UTC m=+3.339948291 container cleanup 27bf130082a01f74839f7b092e89104f9ecd21aa215be6f0c5c2d0b38eccd22f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-i, RELEASE=squid-820a2f8, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-04-21T23:31:48.940 INFO:journalctl@ceph.mds.i.smithi174.stdout:Apr 21 23:31:48 smithi174 bash[69025]: ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-i 2024-04-21T23:31:49.046 DEBUG:teuthology.orchestra.run.smithi174:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d814e4dc-0033-11ef-bc93-c7b262605968@mds.i.service' 2024-04-21T23:31:49.077 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T23:31:49.077 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-04-21T23:31:49.077 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-04-21T23:31:49.078 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ceph-d814e4dc-0033-11ef-bc93-c7b262605968@mds.l 2024-04-21T23:31:49.192 INFO:journalctl@ceph.mds.l.smithi174.stdout:Apr 21 23:31:49 smithi174 systemd[1]: Stopping Ceph mds.l for d814e4dc-0033-11ef-bc93-c7b262605968... 2024-04-21T23:31:49.692 INFO:journalctl@ceph.mds.l.smithi174.stdout:Apr 21 23:31:49 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-l[61177]: 2024-04-21T23:31:49.396+0000 7f8d16799700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T23:31:49.693 INFO:journalctl@ceph.mds.l.smithi174.stdout:Apr 21 23:31:49 smithi174 ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-l[61177]: 2024-04-21T23:31:49.396+0000 7f8d16799700 -1 mds.l *** got signal Terminated *** 2024-04-21T23:31:50.660 INFO:journalctl@ceph.mds.l.smithi174.stdout:Apr 21 23:31:50 smithi174 podman[69147]: 2024-04-21 23:31:50.368623006 +0000 UTC m=+1.074091214 container died 816c950d8d17ddb7b2cd1b3bde65e09c161fb657cbad216be5cd018b91a1834f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-l, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=squid-820a2f8, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-21T23:31:50.942 INFO:journalctl@ceph.mds.l.smithi174.stdout:Apr 21 23:31:50 smithi174 podman[69147]: 2024-04-21 23:31:50.737738536 +0000 UTC m=+1.443206745 container cleanup 816c950d8d17ddb7b2cd1b3bde65e09c161fb657cbad216be5cd018b91a1834f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-l, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, RELEASE=squid-820a2f8, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-04-21T23:31:50.943 INFO:journalctl@ceph.mds.l.smithi174.stdout:Apr 21 23:31:50 smithi174 bash[69147]: ceph-d814e4dc-0033-11ef-bc93-c7b262605968-mds-l 2024-04-21T23:31:51.211 DEBUG:teuthology.orchestra.run.smithi174:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d814e4dc-0033-11ef-bc93-c7b262605968@mds.l.service' 2024-04-21T23:31:51.241 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T23:31:51.242 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-04-21T23:31:51.242 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d814e4dc-0033-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-21T23:31:51.443 INFO:teuthology.orchestra.run.smithi040.stdout:Deleting cluster with fsid: d814e4dc-0033-11ef-bc93-c7b262605968 2024-04-21T23:31:53.949 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d814e4dc-0033-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-21T23:31:54.138 INFO:teuthology.orchestra.run.smithi111.stdout:Deleting cluster with fsid: d814e4dc-0033-11ef-bc93-c7b262605968 2024-04-21T23:31:56.821 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d814e4dc-0033-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-21T23:31:57.010 INFO:teuthology.orchestra.run.smithi174.stdout:Deleting cluster with fsid: d814e4dc-0033-11ef-bc93-c7b262605968 2024-04-21T23:31:59.364 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-21T23:31:59.397 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-21T23:31:59.429 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-21T23:31:59.457 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-21T23:31:59.459 DEBUG:teuthology.misc:Transferring archived files from smithi040:/var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667597/remote/smithi040/crash 2024-04-21T23:31:59.460 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar c -f - -C /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/crash -- . 2024-04-21T23:31:59.523 INFO:teuthology.orchestra.run.smithi040.stderr:tar: /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-21T23:31:59.524 INFO:teuthology.orchestra.run.smithi040.stderr:tar: Error is not recoverable: exiting now 2024-04-21T23:31:59.526 DEBUG:teuthology.misc:Transferring archived files from smithi111:/var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667597/remote/smithi111/crash 2024-04-21T23:31:59.527 DEBUG:teuthology.orchestra.run.smithi111:> sudo tar c -f - -C /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/crash -- . 2024-04-21T23:31:59.564 INFO:teuthology.orchestra.run.smithi111.stderr:tar: /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-21T23:31:59.564 INFO:teuthology.orchestra.run.smithi111.stderr:tar: Error is not recoverable: exiting now 2024-04-21T23:31:59.566 DEBUG:teuthology.misc:Transferring archived files from smithi174:/var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667597/remote/smithi174/crash 2024-04-21T23:31:59.566 DEBUG:teuthology.orchestra.run.smithi174:> sudo tar c -f - -C /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/crash -- . 2024-04-21T23:31:59.624 INFO:teuthology.orchestra.run.smithi174.stderr:tar: /var/lib/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-21T23:31:59.624 INFO:teuthology.orchestra.run.smithi174.stderr:tar: Error is not recoverable: exiting now 2024-04-21T23:31:59.625 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-21T23:31:59.625 DEBUG:teuthology.orchestra.run.smithi040:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-04-21T23:31:59.663 INFO:tasks.cephadm:Compressing logs... 2024-04-21T23:31:59.664 DEBUG:teuthology.orchestra.run.smithi040:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-21T23:31:59.706 DEBUG:teuthology.orchestra.run.smithi111:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-21T23:31:59.709 DEBUG:teuthology.orchestra.run.smithi174:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-21T23:31:59.731 INFO:teuthology.orchestra.run.smithi040.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph.log 2024-04-21T23:31:59.731 INFO:teuthology.orchestra.run.smithi040.stderr:: No such file or directory 2024-04-21T23:31:59.732 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-21T23:31:59.732 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-mds.g.log 2024-04-21T23:31:59.732 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph.log: /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-mds.j.log 2024-04-21T23:31:59.732 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-osd.3.log 2024-04-21T23:31:59.741 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-osd.0.log 2024-04-21T23:31:59.741 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-mds.j.log: /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-mgr.x.log 2024-04-21T23:31:59.741 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-mon.a.log 2024-04-21T23:31:59.741 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-osd.6.log 2024-04-21T23:31:59.742 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-mon.a.log: 89.4% -- replaced with /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-21T23:31:59.742 INFO:teuthology.orchestra.run.smithi040.stderr: 90.1% -- replaced with /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-21T23:31:59.742 INFO:teuthology.orchestra.run.smithi040.stderr: 94.6% -- replaced with /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-mds.j.log.gz 2024-04-21T23:31:59.742 INFO:teuthology.orchestra.run.smithi040.stderr: 94.6% -- replaced with /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-mds.g.log.gz 2024-04-21T23:31:59.742 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-21T23:31:59.742 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-mds.a.log 2024-04-21T23:31:59.742 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-21T23:31:59.742 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-mds.d.log 2024-04-21T23:31:59.742 INFO:teuthology.orchestra.run.smithi040.stderr: 84.1% -- replaced with /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-21T23:31:59.743 INFO:teuthology.orchestra.run.smithi174.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph.log: No such file or directory 2024-04-21T23:31:59.743 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-04-21T23:31:59.743 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-osd.2.log 2024-04-21T23:31:59.743 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-mds.f.log 2024-04-21T23:31:59.743 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-21T23:31:59.743 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-osd.5.log 2024-04-21T23:31:59.744 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-mds.c.log 2024-04-21T23:31:59.744 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-mon.c.log 2024-04-21T23:31:59.744 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-mgr.z.log 2024-04-21T23:31:59.744 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-mds.i.log 2024-04-21T23:31:59.744 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-mgr.z.log: 89.4% -- replaced with /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-21T23:31:59.744 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-21T23:31:59.744 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-mds.i.log: 90.2% -- replaced with /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-21T23:31:59.749 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-volume.log: 94.5% -- replaced with /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-mds.a.log.gz 2024-04-21T23:31:59.749 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-osd.9.log 2024-04-21T23:31:59.749 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-21T23:31:59.750 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-osd.9.log: 94.7% -- replaced with /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-mds.d.log.gz 2024-04-21T23:31:59.751 INFO:teuthology.orchestra.run.smithi174.stderr: 92.0% -- replaced with /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-mgr.z.log.gz 2024-04-21T23:31:59.751 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-osd.11.log 2024-04-21T23:31:59.751 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-osd.8.log 2024-04-21T23:31:59.751 INFO:teuthology.orchestra.run.smithi174.stderr: 83.7% -- replaced with /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-21T23:31:59.753 INFO:teuthology.orchestra.run.smithi111.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph.log 2024-04-21T23:31:59.753 INFO:teuthology.orchestra.run.smithi111.stderr:: No such file or directory 2024-04-21T23:31:59.753 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-21T23:31:59.753 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-osd.7.log 2024-04-21T23:31:59.753 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph.log: /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-mds.b.log 2024-04-21T23:31:59.754 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-osd.4.log 2024-04-21T23:31:59.754 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-osd.1.log 2024-04-21T23:31:59.754 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-mds.b.log: /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-mgr.y.log 2024-04-21T23:31:59.754 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-mds.h.log 2024-04-21T23:31:59.754 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-osd.10.log 2024-04-21T23:31:59.754 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-mds.h.log: 90.2% -- replaced with /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-21T23:31:59.754 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 89.4% --verbose -- /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-21T23:31:59.754 INFO:teuthology.orchestra.run.smithi111.stderr: -- replaced with /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-21T23:31:59.754 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-osd.10.log: 91.9% -- replaced with /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-mgr.y.log.gz 2024-04-21T23:31:59.755 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-mds.e.log 2024-04-21T23:31:59.755 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-21T23:31:59.755 INFO:teuthology.orchestra.run.smithi111.stderr: 83.7% -- replaced with /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-21T23:31:59.756 INFO:teuthology.orchestra.run.smithi040.stderr: 91.1% -- replaced with /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-21T23:31:59.756 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2024-04-21T23:31:59.756 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/ceph-client.admin.log: 92.6% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2024-04-21T23:31:59.760 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-21T23:31:59.762 INFO:teuthology.orchestra.run.smithi040.stderr: 93.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-21T23:31:59.765 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-mon.b.log 2024-04-21T23:31:59.773 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-mds.l.log 2024-04-21T23:31:59.777 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-mds.k.log 2024-04-21T23:31:59.778 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-mon.b.log: 91.1% -- replaced with /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-21T23:31:59.779 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-21T23:31:59.780 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-21T23:31:59.784 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-mds.k.log: 94.5% -- replaced with /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-mds.k.log.gz 2024-04-21T23:31:59.787 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cephadm.log: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-21T23:31:59.792 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-mds.l.log: /var/log/ceph/cephadm.log: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-21T23:31:59.795 INFO:teuthology.orchestra.run.smithi174.stderr: 94.6% -- replaced with /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-mds.l.log.gz 2024-04-21T23:31:59.797 INFO:teuthology.orchestra.run.smithi174.stderr: 91.0% -- replaced with /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-21T23:31:59.818 INFO:teuthology.orchestra.run.smithi174.stderr: 94.3% -- replaced with /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-mds.f.log.gz 2024-04-21T23:32:00.012 INFO:teuthology.orchestra.run.smithi174.stderr: 94.2% -- replaced with /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-mds.i.log.gz 2024-04-21T23:32:00.109 INFO:teuthology.orchestra.run.smithi040.stderr: 90.3% -- replaced with /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-mgr.x.log.gz 2024-04-21T23:32:00.111 INFO:teuthology.orchestra.run.smithi174.stderr: 95.1% -- replaced with /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-mds.c.log.gz 2024-04-21T23:32:00.124 INFO:teuthology.orchestra.run.smithi174.stderr: 91.9% -- replaced with /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-mon.c.log.gz 2024-04-21T23:32:00.168 INFO:teuthology.orchestra.run.smithi111.stderr: 92.0% -- replaced with /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-mon.b.log.gz 2024-04-21T23:32:00.573 INFO:teuthology.orchestra.run.smithi040.stderr: 90.5% -- replaced with /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-mon.a.log.gz 2024-04-21T23:32:01.659 INFO:teuthology.orchestra.run.smithi111.stderr: 95.3% -- replaced with /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-mds.e.log.gz 2024-04-21T23:32:02.331 INFO:teuthology.orchestra.run.smithi174.stderr: 91.9% -- replaced with /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-osd.11.log.gz 2024-04-21T23:32:02.514 INFO:teuthology.orchestra.run.smithi174.stderr: 92.0% -- replaced with /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-osd.8.log.gz 2024-04-21T23:32:02.573 INFO:teuthology.orchestra.run.smithi111.stderr: 91.9% -- replaced with /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-osd.1.log.gz 2024-04-21T23:32:02.601 INFO:teuthology.orchestra.run.smithi040.stderr: 91.7% -- replaced with /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-osd.0.log.gz 2024-04-21T23:32:02.834 INFO:teuthology.orchestra.run.smithi174.stderr: 91.8% -- replaced with /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-osd.2.log.gz 2024-04-21T23:32:03.316 INFO:teuthology.orchestra.run.smithi040.stderr: 91.5% -- replaced with /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-osd.6.log.gz 2024-04-21T23:32:03.397 INFO:teuthology.orchestra.run.smithi111.stderr: 91.8% -- replaced with /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-osd.7.log.gz 2024-04-21T23:32:03.583 INFO:teuthology.orchestra.run.smithi040.stderr: 92.2% -- replaced with /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-osd.3.log.gz 2024-04-21T23:32:03.651 INFO:teuthology.orchestra.run.smithi111.stderr: 93.9% -- replaced with /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-mds.b.log.gz 2024-04-21T23:32:03.728 INFO:teuthology.orchestra.run.smithi111.stderr: 92.4% -- replaced with /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-osd.10.log.gz 2024-04-21T23:32:03.876 INFO:teuthology.orchestra.run.smithi040.stderr: 92.2% -- replaced with /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-osd.9.log.gz 2024-04-21T23:32:03.877 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-21T23:32:03.878 INFO:teuthology.orchestra.run.smithi040.stderr:real 0m4.157s 2024-04-21T23:32:03.878 INFO:teuthology.orchestra.run.smithi040.stderr:user 0m15.193s 2024-04-21T23:32:03.878 INFO:teuthology.orchestra.run.smithi040.stderr:sys 0m0.506s 2024-04-21T23:32:04.013 INFO:teuthology.orchestra.run.smithi174.stderr: 92.1% -- replaced with /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-osd.5.log.gz 2024-04-21T23:32:04.014 INFO:teuthology.orchestra.run.smithi111.stderr: 91.8% -- replaced with /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-osd.4.log.gz 2024-04-21T23:32:04.015 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-04-21T23:32:04.015 INFO:teuthology.orchestra.run.smithi174.stderr:real 0m4.292s 2024-04-21T23:32:04.016 INFO:teuthology.orchestra.run.smithi174.stderr:user 0m13.431s 2024-04-21T23:32:04.016 INFO:teuthology.orchestra.run.smithi174.stderr:sys 0m0.437s 2024-04-21T23:32:16.531 INFO:teuthology.orchestra.run.smithi111.stderr: 92.9% -- replaced with /var/log/ceph/d814e4dc-0033-11ef-bc93-c7b262605968/ceph-mds.h.log.gz 2024-04-21T23:32:16.533 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-04-21T23:32:16.533 INFO:teuthology.orchestra.run.smithi111.stderr:real 0m16.810s 2024-04-21T23:32:16.533 INFO:teuthology.orchestra.run.smithi111.stderr:user 0m36.688s 2024-04-21T23:32:16.533 INFO:teuthology.orchestra.run.smithi111.stderr:sys 0m1.067s 2024-04-21T23:32:16.534 INFO:tasks.cephadm:Archiving logs... 2024-04-21T23:32:16.534 DEBUG:teuthology.misc:Transferring archived files from smithi040:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667597/remote/smithi040/log 2024-04-21T23:32:16.535 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-21T23:32:18.853 DEBUG:teuthology.misc:Transferring archived files from smithi111:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667597/remote/smithi111/log 2024-04-21T23:32:18.854 DEBUG:teuthology.orchestra.run.smithi111:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-21T23:32:23.671 DEBUG:teuthology.misc:Transferring archived files from smithi174:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667597/remote/smithi174/log 2024-04-21T23:32:23.672 DEBUG:teuthology.orchestra.run.smithi174:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-21T23:32:25.891 INFO:tasks.cephadm:Removing cluster... 2024-04-21T23:32:25.892 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d814e4dc-0033-11ef-bc93-c7b262605968 --force 2024-04-21T23:32:26.090 INFO:teuthology.orchestra.run.smithi040.stdout:Deleting cluster with fsid: d814e4dc-0033-11ef-bc93-c7b262605968 2024-04-21T23:32:26.370 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d814e4dc-0033-11ef-bc93-c7b262605968 --force 2024-04-21T23:32:26.560 INFO:teuthology.orchestra.run.smithi111.stdout:Deleting cluster with fsid: d814e4dc-0033-11ef-bc93-c7b262605968 2024-04-21T23:32:26.869 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d814e4dc-0033-11ef-bc93-c7b262605968 --force 2024-04-21T23:32:27.060 INFO:teuthology.orchestra.run.smithi174.stdout:Deleting cluster with fsid: d814e4dc-0033-11ef-bc93-c7b262605968 2024-04-21T23:32:27.390 INFO:tasks.cephadm:Removing cephadm ... 2024-04-21T23:32:27.390 DEBUG:teuthology.orchestra.run.smithi040:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-21T23:32:27.406 DEBUG:teuthology.orchestra.run.smithi111:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-21T23:32:27.421 DEBUG:teuthology.orchestra.run.smithi174:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-21T23:32:27.436 INFO:tasks.cephadm:Teardown complete 2024-04-21T23:32:27.437 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-21T23:32:27.448 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-21T23:32:27.448 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-21T23:32:27.450 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-21T23:32:27.463 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-21T23:32:27.573 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-21T23:32:27.584 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-21T23:32:27.584 DEBUG:teuthology.orchestra.run.smithi040:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-21T23:32:27.586 DEBUG:teuthology.orchestra.run.smithi111:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-21T23:32:27.588 DEBUG:teuthology.orchestra.run.smithi174:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-21T23:32:27.598 INFO:teuthology.orchestra.run.smithi040.stderr:bash: line 1: ntpq: command not found 2024-04-21T23:32:27.599 INFO:teuthology.orchestra.run.smithi111.stderr:bash: line 1: ntpq: command not found 2024-04-21T23:32:27.602 INFO:teuthology.orchestra.run.smithi174.stderr:bash: line 1: ntpq: command not found 2024-04-21T23:32:27.603 INFO:teuthology.orchestra.run.smithi040.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-21T23:32:27.603 INFO:teuthology.orchestra.run.smithi040.stdout:=============================================================================== 2024-04-21T23:32:27.603 INFO:teuthology.orchestra.run.smithi040.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 19 -575us[ -575us] +/- 77ms 2024-04-21T23:32:27.603 INFO:teuthology.orchestra.run.smithi040.stdout:^* hv02.front.sepia.ceph.com 3 6 377 23 +340us[ +343us] +/- 71ms 2024-04-21T23:32:27.604 INFO:teuthology.orchestra.run.smithi040.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 20 +220us[ +220us] +/- 81ms 2024-04-21T23:32:27.604 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T23:32:27.605 INFO:teuthology.orchestra.run.smithi111.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-21T23:32:27.605 INFO:teuthology.orchestra.run.smithi111.stdout:=============================================================================== 2024-04-21T23:32:27.605 INFO:teuthology.orchestra.run.smithi111.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 24 -585us[ -580us] +/- 77ms 2024-04-21T23:32:27.605 INFO:teuthology.orchestra.run.smithi111.stdout:^* hv02.front.sepia.ceph.com 3 6 377 21 +331us[ +336us] +/- 71ms 2024-04-21T23:32:27.605 INFO:teuthology.orchestra.run.smithi111.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 22 +212us[ +217us] +/- 81ms 2024-04-21T23:32:27.605 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T23:32:27.606 INFO:teuthology.orchestra.run.smithi174.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-21T23:32:27.606 INFO:teuthology.orchestra.run.smithi174.stdout:=============================================================================== 2024-04-21T23:32:27.607 INFO:teuthology.orchestra.run.smithi174.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 23 -574us[ -574us] +/- 77ms 2024-04-21T23:32:27.607 INFO:teuthology.orchestra.run.smithi174.stdout:^* hv02.front.sepia.ceph.com 3 6 377 27 +336us[ +328us] +/- 71ms 2024-04-21T23:32:27.607 INFO:teuthology.orchestra.run.smithi174.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 23 +211us[ +211us] +/- 81ms 2024-04-21T23:32:27.607 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T23:32:27.607 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-21T23:32:27.617 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-21T23:32:27.618 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-21T23:32:27.639 DEBUG:teuthology.orchestra.run.smithi040:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-21T23:32:27.719 DEBUG:teuthology.orchestra.run.smithi111:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-21T23:32:27.781 DEBUG:teuthology.orchestra.run.smithi174:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-21T23:32:27.845 DEBUG:teuthology.orchestra.run.smithi040:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-21T23:32:27.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T23:32:27.872 DEBUG:teuthology.orchestra.run.smithi111:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-21T23:32:27.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T23:32:27.899 DEBUG:teuthology.orchestra.run.smithi174:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-21T23:32:27.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T23:32:27.926 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-21T23:32:27.937 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-21T23:32:27.981 INFO:teuthology.task.internal:Duration was 1877.481428 seconds 2024-04-21T23:32:27.981 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-21T23:32:28.014 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-21T23:32:28.015 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-21T23:32:28.017 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-21T23:32:28.019 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-21T23:32:28.052 INFO:teuthology.orchestra.run.smithi040.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-21T23:32:28.053 INFO:teuthology.orchestra.run.smithi111.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-21T23:32:28.056 INFO:teuthology.orchestra.run.smithi174.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-21T23:32:28.482 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-21T23:32:28.482 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi040.front.sepia.ceph.com 2024-04-21T23:32:28.482 DEBUG:teuthology.orchestra.run.smithi040:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-21T23:32:28.503 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi111.front.sepia.ceph.com 2024-04-21T23:32:28.503 DEBUG:teuthology.orchestra.run.smithi111:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-21T23:32:28.549 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi174.front.sepia.ceph.com 2024-04-21T23:32:28.549 DEBUG:teuthology.orchestra.run.smithi174:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-21T23:32:28.573 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-21T23:32:28.573 DEBUG:teuthology.orchestra.run.smithi040:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-21T23:32:28.575 DEBUG:teuthology.orchestra.run.smithi111:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-21T23:32:28.592 DEBUG:teuthology.orchestra.run.smithi174:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-21T23:32:28.639 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-21T23:32:28.639 DEBUG:teuthology.orchestra.run.smithi040:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-21T23:32:28.643 DEBUG:teuthology.orchestra.run.smithi111:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-21T23:32:28.657 DEBUG:teuthology.orchestra.run.smithi174:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-21T23:32:29.028 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-21T23:32:29.040 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-21T23:32:29.040 DEBUG:teuthology.orchestra.run.smithi040:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-21T23:32:29.064 DEBUG:teuthology.orchestra.run.smithi111:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-21T23:32:29.087 DEBUG:teuthology.orchestra.run.smithi174:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-21T23:32:29.111 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-21T23:32:29.122 DEBUG:teuthology.orchestra.run.smithi040:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-21T23:32:29.125 DEBUG:teuthology.orchestra.run.smithi111:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-21T23:32:29.130 DEBUG:teuthology.orchestra.run.smithi174:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-21T23:32:29.145 INFO:teuthology.orchestra.run.smithi040.stdout:kernel.core_pattern = core 2024-04-21T23:32:29.150 INFO:teuthology.orchestra.run.smithi111.stdout:kernel.core_pattern = core 2024-04-21T23:32:29.172 INFO:teuthology.orchestra.run.smithi174.stdout:kernel.core_pattern = core 2024-04-21T23:32:29.194 DEBUG:teuthology.orchestra.run.smithi040:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-21T23:32:29.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T23:32:29.213 DEBUG:teuthology.orchestra.run.smithi111:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-21T23:32:29.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T23:32:29.227 DEBUG:teuthology.orchestra.run.smithi174:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-21T23:32:29.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T23:32:29.249 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-21T23:32:29.261 INFO:teuthology.task.internal:Transferring archived files... 2024-04-21T23:32:29.261 DEBUG:teuthology.misc:Transferring archived files from smithi040:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667597/remote/smithi040 2024-04-21T23:32:29.261 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-21T23:32:29.303 DEBUG:teuthology.misc:Transferring archived files from smithi111:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667597/remote/smithi111 2024-04-21T23:32:29.304 DEBUG:teuthology.orchestra.run.smithi111:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-21T23:32:29.338 DEBUG:teuthology.misc:Transferring archived files from smithi174:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667597/remote/smithi174 2024-04-21T23:32:29.338 DEBUG:teuthology.orchestra.run.smithi174:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-21T23:32:29.373 INFO:teuthology.task.internal:Removing archive directory... 2024-04-21T23:32:29.373 DEBUG:teuthology.orchestra.run.smithi040:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-21T23:32:29.376 DEBUG:teuthology.orchestra.run.smithi111:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-21T23:32:29.378 DEBUG:teuthology.orchestra.run.smithi174:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-21T23:32:29.423 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-21T23:32:29.437 INFO:teuthology.task.internal:Not uploading archives. 2024-04-21T23:32:29.438 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-21T23:32:29.481 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-21T23:32:29.482 DEBUG:teuthology.orchestra.run.smithi040:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-21T23:32:29.484 DEBUG:teuthology.orchestra.run.smithi111:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-21T23:32:29.486 DEBUG:teuthology.orchestra.run.smithi174:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-21T23:32:29.497 INFO:teuthology.orchestra.run.smithi040.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 21 23:32 /home/ubuntu/cephtest 2024-04-21T23:32:29.498 INFO:teuthology.orchestra.run.smithi111.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 21 23:32 /home/ubuntu/cephtest 2024-04-21T23:32:29.501 INFO:teuthology.orchestra.run.smithi174.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 21 23:32 /home/ubuntu/cephtest 2024-04-21T23:32:29.502 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-21T23:32:29.514 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-21T23:32:29.581 INFO:teuthology.nuke:Checking targets against current locks 2024-04-21T23:32:29.608 DEBUG:teuthology.nuke:shortname: smithi040 2024-04-21T23:32:29.608 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-21T23:32:29.630 DEBUG:teuthology.nuke:shortname: smithi111 2024-04-21T23:32:29.631 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-21T23:32:29.637 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi040.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667597', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-21 22:55:06.386214', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNT+hzD/V5Q4zEFRYEAvJrUNAYM0ssmmtHkEPvHajAavMOIliQ1IVX1WGg1jcHBZZTXml+Le35lkmhPf97rHsS0='} 2024-04-21T23:32:29.652 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi111.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667597', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-21 22:55:06.387758', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEqzOEpC4iF6MYQz/A+OjWQlHSxszVS/H13hLtt9oQaTUBzQRs9yeoDkxt5lbrSeDflEC6Haneqcu+tNpxds2D0='} 2024-04-21T23:32:29.656 DEBUG:teuthology.nuke:shortname: smithi174 2024-04-21T23:32:29.656 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-21T23:32:29.686 INFO:teuthology.orchestra.console.smithi040:Power off 2024-04-21T23:32:29.686 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-21T23:32:29.700 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi174.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667597', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-21 22:55:06.384410', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDr2h/lwQ+5x1MozouN3F0vf5TFKKT9TfI0KrG+lrE4mK3z0Uoia+My2pbUQR3QkptY/XHF4CVxNVHlrmQGcA40='} 2024-04-21T23:32:29.713 DEBUG:teuthology.orchestra.console.smithi040:power off output: Chassis Power Control: Down/Off 2024-04-21T23:32:29.713 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-21T23:32:29.729 INFO:teuthology.orchestra.console.smithi111:Power off 2024-04-21T23:32:29.730 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-21T23:32:29.741 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is on 2024-04-21T23:32:29.752 DEBUG:teuthology.orchestra.console.smithi111:power off output: Chassis Power Control: Down/Off 2024-04-21T23:32:29.753 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-21T23:32:29.768 INFO:teuthology.orchestra.console.smithi174:Power off 2024-04-21T23:32:29.769 DEBUG:teuthology.orchestra.console.smithi174:pexpect command: ipmitool -H smithi174.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-21T23:32:29.778 DEBUG:teuthology.orchestra.console.smithi111:check power output: Chassis Power is on 2024-04-21T23:32:29.791 DEBUG:teuthology.orchestra.console.smithi174:power off output: Chassis Power Control: Down/Off 2024-04-21T23:32:29.791 DEBUG:teuthology.orchestra.console.smithi174:pexpect command: ipmitool -H smithi174.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-21T23:32:29.816 DEBUG:teuthology.orchestra.console.smithi174:check power output: Chassis Power is on 2024-04-21T23:32:33.743 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-21T23:32:33.779 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-21T23:32:33.818 DEBUG:teuthology.orchestra.console.smithi174:pexpect command: ipmitool -H smithi174.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-21T23:32:33.853 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is on 2024-04-21T23:32:33.888 DEBUG:teuthology.orchestra.console.smithi111:check power output: Chassis Power is on 2024-04-21T23:32:33.928 DEBUG:teuthology.orchestra.console.smithi174:check power output: Chassis Power is on 2024-04-21T23:32:37.854 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-21T23:32:37.888 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-21T23:32:37.929 DEBUG:teuthology.orchestra.console.smithi174:pexpect command: ipmitool -H smithi174.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-21T23:32:37.966 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is on 2024-04-21T23:32:37.997 DEBUG:teuthology.orchestra.console.smithi111:check power output: Chassis Power is off 2024-04-21T23:32:38.040 DEBUG:teuthology.orchestra.console.smithi174:check power output: Chassis Power is on 2024-04-21T23:32:38.098 INFO:teuthology.orchestra.console.smithi111:Power off completed 2024-04-21T23:32:41.970 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-21T23:32:42.041 DEBUG:teuthology.orchestra.console.smithi174:pexpect command: ipmitool -H smithi174.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-21T23:32:42.081 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is off 2024-04-21T23:32:42.150 DEBUG:teuthology.orchestra.console.smithi174:check power output: Chassis Power is off 2024-04-21T23:32:42.183 INFO:teuthology.orchestra.console.smithi040:Power off completed 2024-04-21T23:32:42.251 INFO:teuthology.orchestra.console.smithi174:Power off completed 2024-04-21T23:32:42.358 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/fsx}} duration: 1877.4814276695251 failure_reason: 'Command failed (workunit test suites/fsx.sh) on smithi040 with status 2: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=430e09df97c8fc7dc2b2ae424f68ed11366c540f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh''' flavor: default owner: scheduled_teuthology@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=44d0a6eb22c24cdbae7c8e078c780657 status: fail success: false 2024-04-21T23:32:42.359 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-21T23:32:42.448 INFO:teuthology.run:FAIL